lance

Lance is a data format built for AI, not retrofitted for it. If you work with images, video, audio, text, and embeddings together, Parquet and Iceberg start to hurt: random access is slow and they were never meant for blobs. Lance fixes that. It claims 100x faster random access than Parquet, with vector search, full-text search, and SQL analytics in one format. Apache 2.0, and it drops into Pandas, DuckDB, Polars, PyArrow, Spark, and Ray.

Because it's a format and not a service, there's almost nothing to run. You convert from Parquet in a couple of lines and query it from the tools you already use. It ships ACID transactions, time-travel versioning, and a vector index, so the same files that hold your training data also serve similarity search. For multimodal AI pipelines, the constant reshuffling between a blob store, a feature store, and a vector database is exactly the tax this removes.

This is for anyone building AI data pipelines who is tired of gluing four systems together. Solo and small teams: adopt it freely, there is no paid tier on the format itself. The company behind it sells LanceDB Cloud if you want a managed database on top, but the format and the local workflow cost nothing.

The catch is maturity. Parquet and Iceberg have a decade of tooling, integrations, and battle-testing behind them. Lance is newer and moving fast, which means fewer integrations and the occasional rough edge. If your stack lives entirely in established lakehouse tooling, adopt it where multimodal access actually hurts, not everywhere at once.