Sharding & Partitioning
Distribution & PartitioningSplitting a dataset across multiple nodes by key so no single machine has to hold or serve all of it — the primary way to scale writes and storage past one box.
Sharding (a.k.a. horizontal partitioning) splits one logical dataset across many physical nodes, each owning a subset of the rows/keys. It exists because vertical scaling (a bigger box) always hits a ceiling on both storage and write throughput, and a single primary can only accept so many writes/sec. The core design decision is the shard key and strategy: range-based (contiguous key ranges per shard — simple, but creates hotspots for sequential keys like timestamps), hash-based (hash the key to pick a shard — even distribution, but range queries now hit every shard), geo/directory-based (route by an explicit lookup table or region — flexible, adds a lookup hop), or consistent hashing (a hash ring so adding/removing a node only reshuffles ~1/N of the keys instead of nearly everything).
How it connects
Sharding & Partitioning as the source, with the components it typically interacts with.
- → Database Types (SQL, NoSQL & Beyond): Sharding is a scaling strategy applied on top of whichever data model (relational, document, wide-column, etc.) the database uses.
- → Consistent Hashing: Consistent hashing is the standard algorithm for assigning shard keys to nodes so the cluster can be resized without re-sharding almost everything.
- → Distributed Transactions (2PC & Saga): Operations spanning multiple shards need a distributed transaction protocol (2PC/sagas) since a shard boundary breaks single-node ACID guarantees.
- → Replication: Each shard is typically replicated independently, so sharding and replication compose: N shards, each with M replicas.