Message Queues & Event Streaming
Async CommunicationDecouples producers from consumers, smooths traffic spikes, and enables async/event-driven architectures.
A message queue (SQS, RabbitMQ) or log-based event stream (Kafka, Kinesis) sits between services that produce work and services that consume it. Instead of a producer calling a consumer directly (tight coupling, blocking, cascading failure risk), the producer drops a message and moves on; consumers process at their own pace. Queues are typically point-to-point (one consumer per message, great for task distribution/work queues); logs are append-only and support many independent consumer groups replaying the same stream (great for event sourcing / fan-out).
How it connects
Message Queues & Event Streaming as the source, with the components it typically interacts with.
- → Database Types (SQL, NoSQL & Beyond): Consumers typically read events off the queue and persist the result to a database, decoupling ingestion rate from write throughput.
- → Consensus & Replication Protocols: Queue brokers (Kafka, etc.) use a consensus protocol internally to elect partition leaders and agree on the committed log position.
- → Caching: Cache-invalidation and cache-warming events are often published to a queue so multiple cache instances stay in sync asynchronously.