S3 Policies and Best Practices - Pricing, Limits, Naming, Security and More

We will discuss some of the policies of S3. Other policies may have a separate page of its own.

S3 Pricing

S3 charges you for following:

1. Data storage (monthly). Is different for different region.

2. Requests to the buckets and objects

3. All data transferred out (e.g. downloads), but no charges for data coming in (e.g. uploads).

4. Data transferred via a COPY request between Regions.'

5. Storage management and transfer acceleration.

There are no charges for the following:

1. Data transferred within an S3 Region via a COPY request.

2. Data transferred between EC2 and S3 within the same Region

3. Multi-Object Delete does not have a price. It is used to delete large numbers of objects from Amazon S3.

Additional Notes on Pricing:

1. S3 Bucket can be configured as requestor pays for data transfer costs.

2. Normal S3 rates apply for every version of an object stored or requested.

3. S3 data retrieval pricing applies for the Standard - IA storage class.

S3 Consistency: Read after Write’ consistency

S3 provides ‘Read after Write’ consistency for new objects and eventual consistency for overwrites and deletes:

• When you do a PUT for a new object, you can start reading immediately.

• For overwrites and deletes, it may take some time to propagate.

This behaviour is same across all regions.

S3 Naming Policies

1. S3 has a universal namespace. Bucket name should be unique across all bucket names in S3.

2. S3 Url format is as follows: s3-<region>.amazonaws.com/<bucketname>

3. Bucket names must be DNS compliant in all regions except for the US East (N. Virginia) region. If you use the AWS management console, bucket names must be DNS compliant in all regions. 

4. The rules for DNS-compliant bucket names are:

   1. Must be at least 3 and no more than 63 characters long.

   2. Must be a series of one or more labels separated by a single period (.).

   3. Can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.

   4. Must not be formatted as an IP address (e.g., 192.168.5.4).

5. IMP! When using virtual hosted–style buckets with SSL, the SSL wildcard certificate only matches buckets that do not contain periods. To work around this, use HTTP or write your own certificate verification logic.

   1. Amazon recommend that you do not use periods (".") in bucket names.

S3 Usage Expectations and Limits  

1. Total volume of data and number of objects you can store are unlimited.

2. File size can be 0 bytes to 5 TB.

   • Single file upload (PUT) file size can be upto 5GB.

   • Can upload files larger than 5GB through multipart upload with chunks of max 5GB.

     • Amazon recommends the use of smaller chunks in multipart upload for better performance and efficiency.  These parts can be uploaded in parallel. Bandwidth lost in case of a connectivity failure will also be small in case of smaller chunks.

3. For objects larger than 100 megabytes, Amazon recommends using the Multipart Upload capability.

4. By default, customers can provision up to 100 buckets per AWS account. However, you can increase by asking AWS.

5. S3 buckets are stored lexicographically (alphabetic order).

S3 Security Policies

1. By default a new bucket and file is private. Even if you make a bucket as public, you still need to make file as public explicitly.

S3 Best Practices

1. You should try to avoid big files with versioning enabled and update them frequently. If needed, should try to setup lifecycle management. Total size used will be the sum of all the versions.

2. For extra security, versioning’s MFA Delete capability can be used to change the versioning state or delete a version.