Design a News Feed / Instagram Photo Feed
HardAsked at: Meta, Instagram, Twitter/X, TikTok, LinkedIn
Users post content (photos, videos, text updates) and see a personalized feed of posts from accounts they follow. Very similar to the Twitter case study but with heavier media (images/videos), more emphasis on ranking/recommendation, and the fan-out-on-write vs. fan-out-on-read tradeoff at its most pronounced.
Request flow
Functional Requirements
- Users can post photos/videos with captions; follow/unfollow other users.
- View a personalized feed: recent posts from followed accounts, ranked by relevance/engagement.
- Like, comment, share posts (secondary features).
- Support for Stories (ephemeral 24-hour content) as an extension.
Non-Functional Requirements
- Feed load must be very fast (<300ms) — this is the app's primary UX surface.
- Handle massive skew in follower counts (celebrities with 100M+ followers).
- Eventual consistency acceptable — a post appearing a few seconds late in followers' feeds is fine.
- High availability for both reads and writes; posting should almost never fail.
Capacity Estimation
- Assume 500M daily active users, each opening the feed ~10x/day → 5B feed reads/day ≈ ~58,000 reads/sec average (10x that at peak → 580,000/sec).
- Assume 100M posts/day (photos/videos) → ~1,150 writes/sec average. Reads vastly outnumber writes, favoring fan-out-on-write for the common case.
- Average follower count ~300, but distribution is a power law — top 0.01% of accounts have tens of millions of followers.
- Each post metadata (text, author, timestamp, media URLs) ~2KB; media stored separately in object storage. 100M * 2KB/day ≈ 200GB/day metadata.
Design Walkthrough
1. The fan-out-on-write vs. fan-out-on-read decision (same as Twitter, but even more critical here)
Fan-out-on-write (push): when a user posts, immediately push that post into the precomputed feed cache of every follower — reads become a single fast lookup. Optimal for users with few followers, catastrophic for celebrities (one post → 100M cache writes).
Fan-out-on-read (pull): store posts once; when a user loads their feed, fetch recent posts from everyone they follow and merge on the fly. Writes are O(1), but reads become expensive (fan out to hundreds of followees per feed load) and slow.
Hybrid (Instagram/Meta's approach): fan-out-on-write for normal users (<10K followers), fan-out-on-read for celebrities/high-follower accounts. Each user's feed is assembled by merging their precomputed feed (from normal followees) + a live fetch of recent posts from the small number of celebrities they follow.
2. High-level architecture
Post write path: Client uploads media → CDN/Object Storage (S3 with CDN in front) → Client calls Post Service with media URLs → Post Service writes to Post DB → publishes 'post_created' event to a message queue → Fan-out Service consumes it, checks follower count, and either pushes post_id to followers' feed caches (normal users) or skips fan-out (celebrity threshold exceeded).
Feed read path: Client requests feed → API fetches user's precomputed feed (from Redis, a capped list of ~1000 post IDs) → merges in live results from celebrity-followees (if any) → hydrates post IDs into full post objects (batch fetch from Post cache/DB) → applies ranking (ML-based or simple recency) → returns.
Media handling: photos/videos are stored in object storage with a CDN in front; the Post DB only stores references (media_url), not the blobs themselves.
3. Data storage
Post DB: {post_id, author_id, caption, media_urls[], created_at, like_count, comment_count} — sharded by post_id or author_id. Immutable once created (aside from counters).
Follower Graph: {user_id → [follower_ids]}, {user_id → [followee_ids]} — stored in a graph DB or sharded adjacency-list table. Critical for fan-out at write time ('get all followers of X') and for feed assembly at read time ('get all followees of X').
Precomputed Feeds (cache): per-user list of post_ids in Redis, capped at ~1000 recent entries. Stores IDs only, not full post content, so fan-out writes are small (8 bytes per follower) and post edits don't require re-fanning content.
Media storage: S3/GCS with CloudFront/CloudFlare CDN — images/videos never touch the app DB or cache.
4. Ranking & personalization (beyond reverse-chronological)
A production feed isn't purely reverse-chronological — a Ranking Service scores candidate posts (engagement predictions, author affinity, recency decay, content type preferences) and reorders the merged set before returning.
Inputs: user interaction history (likes, comments, dwell time), post features (engagement velocity, media type), social graph signals (close friends, frequent interactions).
Often implemented as a two-stage funnel: candidate generation (fetch recent posts from followees, fast) → ranking (ML model scores top N candidates, heavier but only runs on a small set).
5. Handling media-heavy content
Image uploads: client resizes/compresses before upload (or uploads full-res and a background job creates thumbnails in multiple sizes), stores in S3, and passes media_url to Post Service — decouples upload bandwidth from app tier.
Video uploads: similar but more intensive — accept video upload, transcode asynchronously into multiple resolutions/formats (HLS/DASH), and notify when ready. Post is visible immediately with a placeholder thumbnail; full video playback becomes available after transcoding completes.
CDN caching: aggressive caching of images/videos at edge nodes so media is served from geographically close locations without hitting origin storage.
Bottlenecks & Mitigations
- Celebrity fan-out (same as Twitter) — solved by the read/write hybrid: skip fan-out entirely for accounts above a follower threshold (e.g. 100K) and rely on read-time merge instead.
- Follower graph lookups during fan-out ('who follows this author') must be fast for accounts with millions of followers — paginate/batch the fan-out work and process asynchronously via the queue, not in the post-creation request path.
- Thundering herd when a very popular post is published and millions of users reload their feed simultaneously → precomputed feeds absorb the spike because the expensive work (fan-out) already happened at write time.
- Media upload/transcode throughput — offload to dedicated services and object storage; use signed upload URLs so clients upload directly to S3/CDN, bypassing app servers entirely.
Likely Follow-Up Questions
- How would you handle Stories (24-hour ephemeral content)? (Separate lightweight store with TTL=24h, no fan-out-on-write — just a per-user append-only list fetched live when viewing someone's profile or a Stories tray.)
- How do you deduplicate near-identical content (repost, screenshot of another post)? (Perceptual hashing of images/videos; if hash matches an existing post, flag as potential duplicate and optionally suppress in feed ranking.)
- How would you implement a 'close friends' list for private sharing? (A separate, smaller follower graph for close-friends edges; fan-out private posts only to that subgraph instead of all followers.)
- How do you prevent feed manipulation (buying fake followers/engagement)? (Engagement quality signals in ranking — downrank posts with suspicious engagement patterns; separate system to detect bot accounts and exclude them from follower counts.)