Batch vs. Stream Processing
Data ProcessingTwo complementary paradigms for processing large volumes of data: batch (process a bounded dataset all at once, periodically) vs. stream (process each event as it arrives, continuously).
Batch processing (MapReduce, Spark) takes a large, bounded dataset (yesterday's logs, this month's transactions) and processes it all at once on a schedule, optimizing for throughput over latency — the classic 'run it overnight' job. Stream processing (Kafka Streams, Flink, Spark Streaming) processes an unbounded sequence of events continuously as they arrive, optimizing for low latency (results available seconds after the triggering event) at the cost of more operational complexity (state management, out-of-order events, exactly-once semantics). Most large systems use both: a 'Lambda architecture' streams for fast-but-approximate real-time views and batches nightly for slow-but-exact authoritative views; a 'Kappa architecture' commits to stream processing alone and treats reprocessing as just replaying the stream.
How it connects
Batch vs. Stream Processing as the source, with the components it typically interacts with.
- → Message Queues & Event Streaming: Stream processors (Kafka Streams/Flink) consume directly from a message queue's log as their input source.
- → Database Types (SQL, NoSQL & Beyond): Processed/aggregated results are written back to a database (or a separate serving store) for downstream querying.
- → Observability (Logs, Metrics, Traces): Batch/stream pipelines emit lag, throughput, and error metrics that are tracked through the same observability stack as the rest of the system.