Creating Data Lifecycle Management Rules for AWS S3, Part 1

Although most organizations have some data that they would like to keep forever, most data becomes increasingly less relevant as it ages. AWS allows you to create data lifecycle management rules for the data that is stored in S3 buckets. These rules can help drive down costs by purging old data or by moving it to a less expensive storage tier.

In part one of this two-part article, I will show you how to set up data lifecycle management rules.

Creating a Data Lifecycle Management Rule

To get started, open the AWS S3 console. Select the bucket for which you wish to create a lifecycle rule. You will go to a screen that shows the bucket’s contents. Just above the list of storage objects, you will see several tabs, as shown in Figure 1 below. Select the Management tab.

Figure 1

S3 Lifecycle 1-1

There are several tabs located above the list of objects. Select the Management tab.

The Management tab is divided into sections for lifecycle rules, replication rules and inventory configurations. Locate the Lifecycle Rules section, then click Create Lifecycle Rule, as shown in Figure 2.

Figure 2

S3 Lifecycle 1-2

Click on the Create Lifecycle Rule button.

You will go to the Create Lifecycle Rule screen, which you can see in Figure 3. Here, the first thing to do is to provide a name for the new lifecycle rule. The rule name can be up to 255 characters long, which is roughly twice the length of an SMS text message.

AWS allows for such lengthy names because it is not uncommon to create several rules within a single S3 bucket. The long names can help users tell rules apart from one another. You can create names that are detailed and descriptive. Within a rule name, for instance, you might include a note as to why a rule was created or what the rule does.

Figure 3

S3 Lifecycle 1-3

This is the interface used to create a lifecycle rule.

Object Filtering

Next, you need to decide whether the data lifecycle management rule should apply to every object in the S3 bucket or to only some of the objects.

If you want the rule to apply to only some of the objects, you will need to create a filter. AWS offers three different types of filters: prefix filters, object tag filters and object size filters (see Figure 3). However, rules are not limited to using just one filter. You can use a combination of the three filters within a single rule.

The concept of specifying a prefix is roughly analogous to entering a folder name. The important thing to keep in mind is that AWS S3 is an object storage platform. Unlike more conventional storage types, object storage buckets are flat file systems. This means that there is no hierarchy associated with them. There are no buckets within buckets, nor are there any sub folders.

This idea might seem to defy logic, especially given that Figure 1 illustrates a storage bucket where there is a Create Folder button placed prominently above the bucket’s contents (the bucket is actually empty in the figure).

When you create a folder in S3, you are not in fact creating a folder in the way that you would in a traditional file system such as NTFS (New Technology File System). What you are really doing when you create a folder is creating a prefix name. For example, imagine that you created a folder named “Data” and a file named “Document.pdf.” Here, the Data folder is what AWS refers to as a key name prefix. It is separated from the file name by use of a forward slash, which acts as a delimiter.

In an object file system, each file is identified by an object key. This object key is sometimes referred to as a key name, or simply as a key. At any rate, the object key is roughly analogous to the file name. Because object storage does not support a true folder system, the folder name (which is in fact just a prefix) becomes part of the object key for a given file. To show you what I mean, I have created a Document.pdf file and placed it in a folder called Data. In Figure 4, you can see that the key for the file is Data/Document.pdf.

Figure 4

S3 Lifecycle 1-4

The object key is Data/Document.pdf.

All of this is to say that when you create a prefix-based filter for an S3 bucket lifecycle management policy, the Prefix field really just lets you apply your rule to a specific folder. As previously noted, you can also apply a filter based on an object’s tags or on its size.

Now that I have covered the basics of object filtering, we will turn our attention to lifecycle rule actions. I will discuss these actions in part two of this article.