AWS S3 basics every dev should know

Why S3 matters

Amazon S3 is the object storage service that powers a huge part of the internet. If you build anything that stores files, serves static assets, or handles uploads, you'll likely touch S3. It's simple in theory: buckets hold objects (files), and each object has a key (path). But the details matter, especially around permissions, consistency, and cost.

Buckets and keys

A bucket is a global namespace. Bucket names must be unique across all AWS accounts, so you can't just name yours uploads. Use a prefix like mycompany-uploads. Keys are just strings; they can contain slashes to mimic folders, but S3 doesn't really have folders. That's why listing objects with a prefix is the way to "browse" a directory.

aws s3 ls s3://mycompany-uploads/images/

That lists all objects whose key starts with images/. Empty "folders" don't exist unless you create a zero-byte object with a trailing slash.

Permissions are the hard part

By default, everything is private. You control access with IAM policies for users/roles, bucket policies for cross-account or public access, and ACLs (legacy, avoid). The most common mistake is making a bucket public when you only need a few objects public. Instead, use a bucket policy to allow s3:GetObject on a specific prefix.

{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::mycompany-uploads/public/*"}]}

For uploads from a web app, never give the client full bucket access. Use presigned URLs: generate a temporary URL that allows a specific PUT or GET for a limited time. Your backend signs the request, so you control size, expiration, and path.

import boto3
s3 = boto3.client('s3')
url = s3.generate_presigned_url(
    'put_object',
    Params={'Bucket': 'mycompany-uploads', 'Key': 'user-123/photo.jpg'},
    ExpiresIn=3600
)

Consistency: strong now

S3 used to have eventual consistency for reads after writes, but since 2020 it's strongly consistent for all operations. That means you can write an object and immediately read it, list it, or overwrite it. No more "wait a second" hacks. This simplifies a lot of logic, but still be careful with concurrent writes: last write wins, so use versioning if you need history.

Storage classes and lifecycle

Not all data needs the same durability or access speed. S3 has several classes: Standard for frequent access, Intelligent-Tiering for unknown patterns, Glacier for archives. You can set lifecycle rules to transition objects automatically. For example, move logs to Standard-IA after 30 days, then to Glacier after a year.

{"Rules":[{"ID":"ArchiveLogs","Status":"Enabled","Prefix":"logs/","Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"},{"Days":365,"StorageClass":"GLACIER"}]}]}

This saves money without manual effort. But beware: retrieving from Glacier takes minutes to hours, so don't put hot data there.

Versioning and deletion

Enable versioning on buckets that hold important data. It protects against accidental overwrites and deletes. When you delete an object, S3 adds a delete marker instead of removing the data. You can always recover previous versions. This is a lifesaver for production.

However, versioning doubles your storage cost because every version is stored. Combine it with lifecycle rules to clean up old versions.

Cost gotchas

S3 pricing seems cheap, but it adds up. The main costs are storage, requests, and data transfer. Request costs are per 1000 operations, so high-frequency small reads can be more expensive than the storage itself. Use CloudFront as a CDN in front of S3 to reduce requests and data transfer costs. Also, avoid listing buckets frequently in code; that's a request-heavy operation.

Security checklist

• Block public access unless absolutely necessary.
• Enable server-side encryption (SSE-S3 is free, or use KMS for more control).
• Use bucket policies with least privilege.
• Enable access logging to track who does what.
• Never put secrets in object keys or metadata.

Practical tips

• Use aws s3 sync for uploading local folders; it only transfers changed files.
• For large files, use multipart uploads. The SDKs do this automatically for files over a threshold.
• When serving static websites, enable static website hosting on the bucket, but remember that requires public read access.
• Use S3 Select to query CSV/JSON files without downloading them, but it's rarely worth it unless you have huge files.

Final thought

S3 is a workhorse. Once you understand the core concepts, you can build reliable and cost-effective storage. Start with a private bucket, use presigned URLs for uploads, and enable versioning. That covers most use cases. The rest is learning by doing.