Kafka

Kafka is a distributed event streaming platform that moves millions of messages per second with durability guarantees. Producers put messages on, consumers take them off, and nothing gets lost. It stores streams of events in order, lets consumers replay from any point in history, and partitions data so it scales horizontally. Kafka Connect plugs into hundreds of data sources and sinks. Apache 2.0.

Running Kafka yourself used to mean babysitting a separate ZooKeeper ensemble alongside your brokers. That is over. Kafka 4.0 removed ZooKeeper entirely, and KRaft, Kafka's built-in metadata system, is now the only way to run it. That cuts a whole class of operational pain, but a production cluster still needs at least 3 brokers, proper disk provisioning, and someone who understands partitions, consumer groups, and rebalancing. This is not a weekend project.

Fully free from Apache. Confluent Cloud (built by Kafka's creators) starts around $0.015/GB ingested with a free credit. AWS MSK runs roughly $0.21/hr per broker. Solo and small teams: use a managed service. Mid-size teams running real volume: self-hosting starts to pay off once you have the ops skill in house.

The catch: Kafka is built for scale, and if you don't have the scale, you are paying the complexity tax for nothing. Most teams under 10 engineers should reach for a managed service or a simpler queue before standing up their own cluster.