Message Queues — SQS/SNS vs Pub/Sub vs Service Bus

Message queues decouple producers from consumers. Instead of Service A calling Service B synchronously and waiting for a response, A drops a message on a queue and moves on. B picks it up when it's ready. This is the foundation of resilient, scalable architectures — and every cloud provider has their own flavor with different semantics, guarantees, and footguns.

AWS: SQS, SNS, and EventBridge

SQS (Simple Queue Service) is the workhorse. A producer sends a message, a consumer polls and receives it, processes it, then deletes it. Two flavors:

SQS Standard offers nearly unlimited throughput, at-least-once delivery, and best-effort ordering. Messages might arrive out of order and occasionally be delivered more than once. For most workloads, this is fine — as long as your consumer is idempotent (more on that shortly).

SQS FIFO guarantees exactly-once processing and strict ordering within a message group. The trade-off: throughput is capped at 3,000 messages/second with batching (300 without). Use FIFO only when ordering actually matters — financial transactions, sequential workflows — not because it sounds safer.

Visibility timeout is the core mechanic everyone gets wrong. When a consumer receives a message, SQS doesn't delete it — it hides it from other consumers for the visibility timeout period (default: 30 seconds). If the consumer finishes processing and deletes the message, great. If it crashes or takes too long, the message reappears and another consumer picks it up. Set the visibility timeout to at least the maximum time your consumer needs to process a message. Too short = duplicate processing. Too long = messages stuck invisible after a crash.

# Receive, process, delete — the SQS lifecycle
aws sqs receive-message --queue-url $QUEUE_URL --wait-time-seconds 20
# ... process the message ...
aws sqs delete-message --queue-url $QUEUE_URL --receipt-handle $HANDLE

Long polling (WaitTimeSeconds: 20) makes the receive call wait up to 20 seconds for a message before returning empty. Without it, your consumer is making rapid empty requests, burning API costs and CPU. Always enable long polling.

Dead-letter queues (DLQs) catch messages that repeatedly fail processing. You configure a maxReceiveCount — after that many delivery attempts, SQS moves the message to the DLQ. Every production queue needs a DLQ. Without one, poison messages (messages that always fail processing) cycle infinitely, burning compute and blocking other messages in FIFO queues.

SNS (Simple Notification Service) is pub/sub fan-out. A publisher sends a message to a topic, and SNS delivers it to all subscribers — SQS queues, Lambda functions, HTTP endpoints, email. The canonical pattern is SNS to SQS fan-out: an event publishes to an SNS topic, and multiple SQS queues subscribe. Each queue gets its own copy, and each consumer processes independently. This is how you broadcast an "order placed" event to the inventory service, the notification service, and the analytics pipeline simultaneously.

EventBridge is a serverless event bus that adds content-based filtering. Instead of routing all events to all subscribers, EventBridge lets subscribers define rules that match specific event patterns. It's more sophisticated than SNS for event-driven architectures and integrates with SaaS providers and AWS services as event sources.

GCP Pub/Sub

GCP Pub/Sub combines the queue and pub/sub patterns into a single service. You publish messages to a topic. One or more subscriptions attach to that topic, each getting their own copy of every message. A subscription with one consumer group behaves like a queue. A topic with multiple subscriptions behaves like fan-out. No separate queue-and-topic services to wire together.

Pull delivery works like SQS: consumers pull messages, process them, and acknowledge. Push delivery has Pub/Sub POST the message to an HTTP endpoint (Cloud Run, Cloud Functions, or any HTTPS URL). Push is simpler when your consumer is already an HTTP service — no polling loop to manage.

Ack deadline is GCP's visibility timeout. If a consumer doesn't acknowledge a message within the deadline, Pub/Sub redelivers it. You can extend the deadline mid-processing with modifyAckDeadline, which is useful for variable-length processing.

Ordering keys provide ordering guarantees for messages with the same key — analogous to SQS FIFO message groups. Messages with different ordering keys can be processed in parallel. Exactly-once delivery is available on pull subscriptions, deduplicating redeliveries at the Pub/Sub layer.

Pub/Sub scales to hundreds of thousands of messages per second without capacity planning. You don't configure throughput — Google handles it. This is a genuine advantage over SQS FIFO's throughput limits.

Azure: A Menu of Messaging Services

Azure has more messaging options than most teams need:

Storage Queues are the simplest — basic queue with 64KB message size, no ordering guarantees, at-least-once delivery. Cheap but limited. Fine for simple task queuing.

Service Bus is the enterprise messaging service. Queues (point-to-point) and topics/subscriptions (pub/sub) with features SQS developers wish they had: sessions (ordered processing per session ID), scheduled delivery, message deferral, transactions that span multiple queues, and dead-lettering. It supports both AMQP and HTTP protocols.

Service Bus comes in Basic, Standard, and Premium tiers. Premium runs on dedicated infrastructure with predictable latency and supports virtual network integration. If you need FIFO + high throughput + advanced routing, Service Bus Premium is Azure's strongest answer.

Event Grid is Azure's event routing service — lightweight, push-based, designed for reactive event-driven architectures. Think CloudWatch Events or EventBridge but more integrated with Azure services. Events trigger Azure Functions, Logic Apps, or webhooks.

Event Hubs is a stream ingestion service — Azure's Kafka equivalent, not a traditional message queue. Use it for telemetry, log aggregation, and event streaming at millions of events per second. If you need a queue, don't use Event Hubs.

Cross-Cutting: The Rules That Apply Everywhere

At-least-once vs exactly-once: most managed queues deliver at-least-once. Your consumer will occasionally receive the same message twice — during retries, scaling events, or infrastructure hiccups. Idempotent consumers are non-negotiable. Record processed message IDs in a database and check before processing, use database transactions with idempotency keys, or design operations that are naturally idempotent (setting a value vs incrementing it). If processing a message twice charges a customer twice, you have a bug.

Ordering guarantees cost throughput. SQS FIFO is capped at 3,000 msg/s. Pub/Sub ordering keys serialize processing per key. Service Bus sessions limit parallelism. Only pay this cost when ordering actually matters for correctness, not convenience.

Poison messages are messages that always fail processing — malformed data, missing dependencies, bugs in your consumer code. Without a DLQ and maxReceiveCount, they loop forever. Set maxReceiveCount to something reasonable (3-5 attempts), route failures to a DLQ, and alert on DLQ depth. Then fix the root cause.

Backlog monitoring is critical. Queue depth (number of messages waiting) tells you whether your consumers are keeping up. A growing backlog means you need more consumers or your consumers are too slow. Queue depth is one of the best autoscaling signals — KEDA (Kubernetes Event-Driven Autoscaling) can scale pods based on SQS depth, Pub/Sub subscription backlog, or Service Bus queue length.

Side-by-Side Comparison

Aspect	AWS SQS/SNS	GCP Pub/Sub	Azure Service Bus
Queue model	SQS (queue) + SNS (pub/sub)	Unified topics + subscriptions	Queues + topics/subscriptions
Ordering	FIFO queues (3K msg/s cap)	Ordering keys (per-key serial)	Sessions
Exactly-once	FIFO only	Pull subscriptions	Sessions + dedup
Max message size	256KB (SQS), 256KB (SNS)	10MB	256KB (Standard), 100MB (Premium)
Push delivery	SNS to HTTP/Lambda	Push subscriptions	Event Grid / webhooks
Dead-letter queue	Built-in (maxReceiveCount)	Dead-letter topic	Built-in
Fan-out pattern	SNS -> multiple SQS queues	Multiple subscriptions per topic	Topic -> multiple subscriptions
Throughput	Standard: unlimited; FIFO: 3K/s	Scales automatically	Tier-dependent

Choosing: Queue vs Pub/Sub vs Stream

Queue (SQS, Storage Queue, Service Bus Queue): one producer, one consumer group, each message processed once. Use for task queuing, work distribution, and decoupling services.

Pub/Sub (SNS+SQS, Pub/Sub topics, Service Bus Topics): one event, multiple independent consumers each getting their own copy. Use for event broadcasting and fan-out.

Stream (Kinesis, Pub/Sub with ordering, Event Hubs): ordered, replayable log of events. Consumers maintain their position and can re-read history. Use for event sourcing, change data capture, and analytics pipelines.

Most architectures use all three patterns. An API request goes to a queue for processing, the result publishes to a topic for fan-out, and the event is recorded in a stream for analytics.

Key Takeaways

• Visibility timeout (SQS) / ack deadline (Pub/Sub) is the mechanic that makes queues work — set it longer than your maximum processing time or you'll get duplicate processing
• Idempotent consumers are non-negotiable with at-least-once delivery; processing a message twice must produce the same result as processing it once
• The SNS-to-SQS fan-out pattern is the canonical way to broadcast events to multiple independent consumers on AWS
• Every production queue needs a dead-letter queue with alerting — without it, poison messages silently consume resources forever
• Queue depth is one of the best autoscaling signals; KEDA makes this native in Kubernetes
• GCP Pub/Sub unifies queue and pub/sub into one service; AWS separates them (SQS + SNS); Azure gives you the most options (sometimes too many)

Next, we'll look at multi-region architecture — what happens when a single region isn't enough and you need your queues, load balancers, and networks spanning the globe.