Message Queues

Pulsar15.3k★

Pulsar is Apache's answer to Kafka, handling both messaging patterns (pub/sub and queuing) in one system. The key difference: Pulsar separates compute from storage, so you can scale message brokers and storage independently. For teams that have hit Kafka's operational ceiling, that architecture matters. Apache Foundation, Java-based. Multi-tenancy is built in; namespaces and topics can be isolated per team without running separate clusters. Supports both streaming (like Kafka) and queuing (like RabbitMQ) patterns in the same system. Geo-replication works across data centers out of the box. Fully open source, Apache 2.0. StreamNative offers a managed Pulsar cloud starting around $0.10/GB ingested, but self-hosting is the primary path. Self-hosting is where the ops burden hits. Pulsar needs ZooKeeper (or the newer metadata store) plus BookKeeper for storage. Minimum production setup is 3 brokers + 3 bookies + 3 ZK nodes. That's 9 machines before you process a single message. Solo or small teams: use NATS or RabbitMQ unless you specifically need multi-tenancy or geo-replication. Medium teams (10-50): Pulsar makes sense if you're outgrowing simpler solutions. Large: this is where Pulsar shines. The catch: the operational complexity is real. If you don't need multi-tenancy or tiered storage, Kafka is simpler to run and has a larger ecosystem of tooling and connectors.

RabbitMQ13.8k★

RabbitMQ has been reliably passing messages between services since 2007. It sits between your services and makes sure messages get delivered, even if the receiving service is temporarily down. It's a post office for your backend. Service A drops a message in a queue, Service B picks it up when it's ready. RabbitMQ handles routing, retries, and ordering so you don't have to. It supports multiple messaging patterns: simple queues, pub/sub (one message to many consumers), routing by topic, and RPC. The management UI is good: you can see queue depths, message rates, and consumer health at a glance. Self-hosted is free under the Mozilla Public License. CloudAMQP offers managed hosting starting at $0 (free tier: 1M messages/month, limited connections) up to $500+/mo for production clusters. AWS, GCP, and Azure all offer managed RabbitMQ too. The catch: RabbitMQ uses Erlang, which makes it incredibly reliable but painful to debug when something goes wrong at the system level. For simple job queues, it's overkill; BullMQ or Sidekiq are lighter. For high-throughput event streaming (millions of events/sec), Kafka or Redpanda are better suited.

Redpanda12.4k★

Redpanda is a Kafka-compatible streaming platform that eliminates the operational complexity that scares teams away from Kafka. Kafka-compatible API, 10x faster, and no Java Virtual Machine or ZooKeeper to manage. Written in C++ with a thread-per-core architecture. Single binary deployment. Your existing Kafka clients, connectors, and tooling work unchanged, just point them at Redpanda instead. Backed by a well-funded company ($100M+ raised). This is production-grade infrastructure, not a side project. Self-hosted Community edition is free. It's usable: no artificial limits on throughput or partitions. Enterprise adds features like continuous data balancing, tiered storage to S3, Kerberos auth, and audit logging. The catch: the license (BSL-1.1) means you can't offer Redpanda as a competing managed streaming service. For internal use, no problem. Enterprise pricing isn't published but expect it to be significant: this is infrastructure software for companies with real streaming needs. Also, while Kafka compatibility is excellent, some niche Kafka Streams features and Connect plugins may need testing.