Blob Storage — Storing Files, Images and Videos at Scale

Storing binary files in a relational database is an antipattern. It's slow, expensive, and doesn't scale. Blob storage (also called object storage) is purpose-built for this: flat, durable, cheap, and accessible over HTTP.

What Blob Storage Is

Blob storage is designed for unstructured data — anything that doesn't fit neatly into rows and columns. Images, videos, audio files, PDFs, log archives, database backups, ML model weights. All of it lands here.

You access objects over HTTP, which means any client anywhere can retrieve a file with a single GET request. There's no file system to manage, no disk to provision, and no capacity planning to worry about. You store what you store, and you pay for exactly that.

Unlike a traditional file system with directories and subdirectories, blob storage uses a flat namespace. Everything lives in a bucket. There's no real hierarchy — just objects with keys that look like paths.

Core Concepts

Four concepts cover most of what you need to understand blob storage:

Bucket — a top-level container for objects. Think of it like an S3 "drive." Bucket names are globally unique across all users of the service (because they end up in URLs). One account can have many buckets.

Object — the actual file plus its associated metadata. An object is immutable once written. To update it, you overwrite it.

Key — the unique name of the object within a bucket. Keys look like file paths, but they're just strings. The / is a naming convention, not a real directory separator.

Metadata — content type, file size, last-modified timestamp, and any custom tags you attach. Metadata is stored alongside the object and returned in HTTP headers.

A typical bucket and key structure looks like this:

Bucket: my-app-assets

Keys:
  users/avatars/user-123.jpg
  users/avatars/user-456.jpg
  products/images/product-001-main.jpg
  products/images/product-001-thumbnail.jpg
  exports/reports/2024-03-01-monthly.pdf

The URL to access an object would be something like: https://my-app-assets.s3.amazonaws.com/users/avatars/user-123.jpg

Access Patterns

Objects can be public or private, and this is a per-object (or per-bucket) setting.

Public objects are accessible to anyone with the URL — no authentication required. Use this for profile photos, product images, marketing assets, anything you'd serve to anonymous users. The URL is stable and cacheable.

Private objects require authentication. Raw access is blocked. To let a specific user download a private file, you generate a signed URL — a time-limited URL that embeds authentication credentials in the query string. Your server generates it, hands it to the client, and the client downloads directly from blob storage without routing through your servers.

# Pseudocode — generate a signed URL (AWS S3 style)
def get_download_url(bucket, key, expiry_seconds=3600):
    signed_url = s3_client.generate_presigned_url(
        operation="get_object",
        params={"Bucket": bucket, "Key": key},
        expires_in=expiry_seconds
    )
    return signed_url

# Client receives the URL and downloads directly
# URL expires after 1 hour — no re-use after that

This pattern reduces load on your API servers significantly. The client talks directly to S3; your server only generates a short-lived token.

CDN Integration

Blob storage on its own is fast, but it's a single origin location. A user in Tokyo fetching a file stored in US-East pays for the round-trip latency every time.

The fix is to put a CDN in front of your blob storage. Blob storage acts as the origin, and CDN edge nodes cache objects globally. The first request to a region fetches from the origin and caches it at the edge. Every subsequent request is served locally from that edge node.

The flow looks like this:

1. User requests https://cdn.myapp.com/users/avatars/user-123.jpg
2. CDN checks its edge cache for that key
3. Cache miss — CDN fetches the object from S3 (the origin)
4. CDN stores the object at the edge and returns it to the user
5. All subsequent requests from that region are served from cache — no S3 hit

For static assets and media files, CDN integration is essentially mandatory at any meaningful scale.

Multipart Upload

Large files — videos, database backups, ML datasets — can't be uploaded as a single HTTP request. Network interruptions, timeouts, and memory constraints make that unreliable.

Multipart upload splits a large file into chunks and uploads each chunk independently. The provider reassembles them when all parts arrive. AWS S3 supports files up to 5TB via multipart upload.

The advantages are practical:

• Parallelism — upload multiple chunks simultaneously, filling your available bandwidth
• Resumability — if a chunk fails, retry just that chunk, not the whole file
• Memory efficiency — the client only holds one chunk in memory at a time

A typical chunk size is 5–100MB. For a 10GB video, that's 100 to 2000 individual upload requests, each independently verifiable.

Storage Classes

Not all data is accessed equally. A profile photo uploaded today will be fetched thousands of times in the next week. A log archive from two years ago might never be read again.

Storage classes let you optimise cost against access frequency:

• Hot storage — standard tier. Low latency, high throughput, highest cost per GB. Use for frequently accessed data.
• Cool/Infrequent storage — lower cost, slightly higher retrieval cost. Use for data accessed once a month or less.
• Cold/Archive storage — very low storage cost, but retrieval takes minutes to hours. AWS S3 Glacier is the canonical example. Use for compliance archives, disaster recovery backups, anything you hope you never need.

You can define lifecycle policies that move objects between tiers automatically. After 30 days, move to cool storage. After 1 year, move to Glacier. This happens without you touching the data.

Real World Tools

The S3 API has become the de facto standard interface for blob storage. Most providers are S3-compatible, meaning the same client code works across them.

• AWS S3 — market leader. Deeply integrated with the AWS ecosystem. The original.
• GCP Cloud Storage — Google's equivalent. Strong integration with BigQuery and GKE.
• Azure Blob Storage — Microsoft's offering. Well-integrated with Azure compute.
• Cloudflare R2 — S3-compatible with zero egress fees. Significant cost saving for read-heavy workloads.
• Backblaze B2 — the cheapest option. Good for backups and archival where cost dominates.

At scale: Netflix stores all of its video content in S3 — we're talking exabytes. Instagram stores billions of user-uploaded images. Dropbox originally used S3 to store user files, then built its own storage infrastructure (Magic Pocket) when the economics justified it. The lesson there: start with managed blob storage, migrate to something custom only when you have the scale and engineering capacity to justify it.

Key Takeaways

• Blob storage is for unstructured data (images, video, files, backups) — not rows and columns
• Objects are addressed by bucket + key; the flat namespace just uses / as a naming convention
• Signed URLs let clients download private objects directly from storage, reducing load on your API servers
• Always put a CDN in front of blob storage for media files — first-request latency to origin is the only cache miss you'll pay
• Multipart upload is essential for large files: it enables parallelism, resumability, and memory efficiency
• Use storage class lifecycle policies to move cold data to cheaper tiers automatically

Next up: how your app learns about changes in real time — polling, webhooks, and WebSockets each take a fundamentally different approach.