Design Pastebin (Code/Text Sharing Service)
EasyAsked at: Amazon, Google, Microsoft
A service allowing users to paste code/text snippets, get a unique URL, and share it. Similar to URL shortener but with larger content payloads and syntax highlighting needs. Classic for practicing object storage, CDN caching, and read-heavy design patterns.
Request flow
Functional Requirements
- Users can paste text/code (up to ~10MB), optionally set expiration, privacy (public/unlisted/private), and syntax highlighting language.
- System generates a unique short URL (e.g. paste.io/abc123) and stores the content.
- Visitors can view the paste via the URL, with syntax highlighting rendered.
- Support for optional user accounts to manage/delete their pastes.
Non-Functional Requirements
- Read-heavy (typical ratio 100:1 read:write), so optimize for fast retrieval.
- Low-latency access for popular pastes — aggressive caching is critical.
- High availability for reads; writes can tolerate slightly higher latency.
- Handle potentially very large text payloads (10MB) efficiently without blocking API servers.
Capacity Estimation
- Assume 5M new pastes/month ≈ 2 writes/sec average (design for ~20/sec peak).
- With 100:1 read:write ratio → ~200 reads/sec average.
- Average paste size ~50KB (mix of short snippets and longer code) → 5M/month * 50KB ≈ 250GB/month new data. Over 5 years with 20% popular pastes kept hot → ~15TB total, ~3TB hot data for caching.
- Short URL generation: 8-character base62 gives 62^8 ≈ 218 trillion combinations — collision risk negligible.
Design Walkthrough
1. High-level architecture
Write path: Client → API Gateway → Paste Service generates unique short code → stores {code, content, metadata} to Object Storage (S3) + metadata to DB → returns short URL.
Read path: Client → CDN (cache hit → return immediately) → on miss → API Gateway → Paste Service → fetches metadata from DB cache, content from Object Storage (or a cache layer) → returns rendered page → CDN caches for future requests.
Separation of concerns: metadata (owner, expiry, language, privacy) in a fast DB (Postgres/DynamoDB), actual paste content in object storage (S3/GCS) to avoid bloating the DB with large blobs.
2. Short code generation & uniqueness
Option A — Random base62 string (8 chars): simple, no coordination needed, collision probability negligible with 62^8 keyspace. On the rare collision, retry with a new random string.
Option B — Hash(content + timestamp): deterministic but still needs collision handling and doesn't guarantee uniqueness if two users paste identical content simultaneously.
Option C — Incremental counter encoded as base62: guarantees uniqueness but requires coordination (a KGS or database sequence) — overkill for this problem since random generation is simpler and collisions are vanishingly rare.
Interview favorite: Option A (random generation) with a unique index on short_code in the DB to catch the astronomically rare collision.
3. Data storage strategy
Metadata DB: {paste_id (PK), short_code (unique index), user_id, language, privacy, created_at, expires_at, size_bytes} — lightweight, fast lookups by short_code.
Content Storage: Object storage (S3) keyed by paste_id or short_code, storing raw text. Large blobs don't belong in a relational DB — object storage is cheaper, scales better, and integrates naturally with CDN/edge caching.
Why separate? Keeps the DB lightweight for fast metadata queries (checking expiry, ownership, privacy) without dragging multi-MB blobs through the query engine. Content is fetched only on successful access validation.
4. Optimizing reads with caching
CDN-first: Set aggressive cache-control headers (e.g. public, max-age=3600 for public pastes) so CDN edge nodes serve most traffic without hitting the origin API at all. Private pastes bypass CDN or use authenticated edge caching.
Application cache layer (Redis): cache metadata lookups (short_code → paste metadata) to avoid DB hits, and optionally cache popular paste content (especially small ones) to skip the S3 round-trip.
TTL strategy: respect paste expiration — query DB for expiry on cache miss, and set cache TTL = min(default_ttl, time_until_expiry) so expired pastes aren't served stale.
5. Expiration & cleanup
Lazy expiry: on read, check expires_at; if expired, return 404 and async-enqueue a cleanup job. Don't proactively scan all pastes constantly — it's wasteful for content that may never be accessed again.
Periodic sweeper: a background job scans for expired pastes (created_at + TTL < now) in batches, deletes metadata from DB and content from S3. Use lifecycle policies (S3 Object Lifecycle) to auto-delete objects after N days as a second line of defense.
Bottlenecks & Mitigations
- Large paste uploads (10MB) blocking API servers — mitigate with signed S3 upload URLs: API generates a pre-signed URL, client uploads directly to S3, then notifies API when done (offloads bandwidth from app tier).
- Very popular pastes becoming hot keys — solved naturally by CDN caching at the edge; the origin never sees most reads once CDN is warm.
- Short_code uniqueness constraint under high write concurrency — negligible with 62^8 keyspace and random generation; retry on the rare conflict.
Likely Follow-Up Questions
- How would you support raw text vs. rendered HTML view? (Store raw text once; render syntax-highlighted HTML on the fly or pre-render and cache both versions, depending on rendering cost vs. cache size tradeoff.)
- How do you prevent abuse (e.g. someone uploading malware disguised as text, or copyrighted content)? (Content scanning async pipeline — virus scan, hash-based deduplication against known abuse databases; flag/quarantine on match.)
- How would you implement 'fork this paste' or version history? (Store paste_id → parent_paste_id relationship in metadata; treat each edit as a new paste linked to the original, forming a simple version graph.)
- How would you support very large pastes (100MB+) without timeouts? (Switch to chunked/resumable upload via multipart S3 upload, or cap paste size and reject larger payloads — most legitimate pastes are <1MB.)