📬 Design a Message Queue
Kafka, RabbitMQ & SQS Architecture — Producer-Consumer Patterns, Partition Routing & Capacity Planning
Message queues enable asynchronous communication between services, decoupling producers from consumers. They're the backbone of event-driven microservices, handling everything from order processing to real-time analytics pipelines. The three dominant implementations: Apache Kafka (log-based, high-throughput streaming), RabbitMQ (smart broker, flexible routing), and Amazon SQS (fully managed, serverless simplicity).
🏗️ Architecture Comparison
Click each queue to understand its core design philosophy.
Apache Kafka
Log-based pub/sub. Messages are appended to an immutable ordered log. Partitions enable parallel consumers.
RabbitMQ
Smart broker with routing. Exchanges route messages to queues based on binding rules. Flexible topologies.
Amazon SQS
Fully managed serverless. No brokers to manage, no configuration to tune. Auto-scales to any load.
🔀 Kafka Partition Routing
Kafka routes messages to partitions using a key hash. Same key → same partition (preserving order for that key). Keyless messages are round-robin.
// Same key → same partition (per-topic ordering for that key)
int partition = partitioner.choosePartition(
record.topic(),
record.key() != null
? Math.abs(Utils.murmur2(record.key().getBytes())) % numPartitions
: nextPartition(topic, partitionCounter++) // round-robin
);
// Consumer reads from assigned partitions
consumer.assign(List.of(0, 1)); // C0 reads P0, P1
// or
consumer.subscribe(List.of("topic")); // rebalance assigns partitions⚖️ Consumer Group Rebalancing
As consumers join or leave, Kafka triggers a rebalance to redistribute partitions. Watch how partitions move between consumers.
🔄 SQS Message Lifecycle
SQS handles messages differently from Kafka. Each message is processed by exactly one consumer, then deleted after acknowledgment.
// Poll for messages (long polling reduces empty responses)
const result = await sqs.receiveMessage({
QueueUrl: QUEUE_URL,
MaxNumberOfMessages: 10,
WaitTimeSeconds: 20, // long polling
VisibilityTimeout: 30,
}).promise();
// Process each message
for (const msg of result.Messages) {
await process(msg.Body);
// Delete after successful processing
await sqs.deleteMessage({
QueueUrl: QUEUE_URL,
ReceiptHandle: msg.ReceiptHandle
}).promise();
}📐 Capacity Planning Calculator
Estimate required infrastructure based on your throughput requirements.