paimon

Apache Paimon is a table format for data lakes. You store huge amounts of data on cheap object storage (S3, GCS) and query it with both streaming and batch jobs without two separate pipelines. Apache 2.0 licensed, fully open source. Top-level Apache project, originally Flink Table Store before it graduated.

Production setup is not trivial. You need a query engine (Flink or Spark, sometimes both), a metastore, and an object storage bucket. The format itself is the easy part. Running it at scale means managing the surrounding cluster, like any Apache analytics tool. Single-node testing works on a laptop, production is a distributed system.

Compare to Apache Iceberg, Delta Lake, and Apache Hudi. Paimon's pitch is better streaming integration with Flink. Iceberg has wider engine support. Delta Lake is the safer pick if you already live in Databricks. Hudi owns the niche in mutable upserts.

The catch: lakehouse table formats are still a contested space. Picking Paimon means betting on its trajectory rather than choosing the safest option. If you are not running Flink, Iceberg is the easier choice.