Message Queues
Message queues enable asynchronous communication between services, improving reliability and scalability.
Benefits
- Decoupling: Producer and consumer do not need to be available simultaneously
- Buffering: Handles traffic spikes by queuing messages
- Reliability: Messages persist until processed
- Scalability: Consumers can be added as needed
Messaging Patterns
Point-to-Point
One message is delivered to one consumer. A producer sends a message to a queue, and exactly one consumer receives and processes that message.
Use cases:
- Task distribution
- Work queues
- Request processing
Publish-Subscribe
One message is delivered to multiple consumers. A publisher sends a message to a topic, and all subscribers to that topic receive a copy of the message.
Use cases:
- Event notifications
- Real-time updates
- Log aggregation
Message Queue Technologies
| Technology | Type | Strengths |
|---|---|---|
| Apache Kafka | Distributed log | High throughput, durability |
| RabbitMQ | Message broker | Flexible routing, protocols |
| Amazon SQS | Managed queue | Simple, scalable |
| Redis Pub/Sub | In-memory | Low latency, simple |
Delivery Guarantees
At-most-once
Message may be lost but is never duplicated.
At-least-once
Message is delivered but may be duplicated.
- Most common guarantee
- Consumers must handle duplicates (idempotency)
Exactly-once
Message is delivered exactly once.
- Difficult to achieve
- Often requires application-level handling
Key Concepts
Dead Letter Queue
Stores messages that could not be processed after retry attempts. When a consumer fails to process a message and exhausts all retry attempts, the message moves to a dead letter queue for later inspection and manual handling.
Message Ordering
Per-partition ordering:
- Kafka guarantees order within a partition
- Use consistent keys for related messages
Global ordering:
- Difficult to achieve at scale
- Single partition/queue limits throughput
Key Considerations
- When to use asynchronous over synchronous communication
- Message failure handling
- Ordering guarantees
- Duplicate message handling