delta

Delta Lake makes a pile of Parquet files in S3 behave like a real database table: ACID transactions, schema enforcement, and time travel, so you can query the table as it looked last Tuesday. Spark, Trino, Flink, and Hive can all read and write the same tables without corrupting each other. Apache 2.0, governed by the Linux Foundation, completely free.

There's nothing to host. It's a storage format plus connectors you add to a compute engine you already run, which also means the ops burden is inherited: trivial if Spark is already in the building, a project if Delta is the reason you're standing Spark up.

This is for data engineering teams running a lakehouse, realistically five-plus engineers with a platform owner. A single analyst with Postgres does not need it. The format rivals are Apache Iceberg and Apache Hudi; Iceberg has the broader vendor momentum, Delta has the deepest Spark integration and Databricks behind it.

The catch: this repo is the Scala/Spark implementation. The Python deltalake package most people install comes from delta-rs, a separate project with its own gaps. And protocol upgrades are one-way; old readers fail on tables written with newer protocol versions.