Spark

Spark is the distributed data processing engine that handles datasets too large for a single machine. When your data pipeline needs to crunch terabytes across a cluster, Spark's been the answer for a decade — batch processing, streaming, ML, and SQL all in one framework. 43k stars and the backbone of most enterprise data platforms.

DuckDB is the single-machine SQL engine that eliminates Spark for datasets under ~100GB — 10x the price-performance of Databricks on an EC2 instance. Polars is the Rust-powered DataFrame library that's fastest for single-node operations. But neither scales horizontally like Spark when data truly demands a cluster.

Use Spark if your data doesn't fit on one machine and you need distributed processing across a cluster. If it fits in RAM or on a single big VM, DuckDB or Polars are 10x simpler.

The catch: Spark's operational complexity is enormous — cluster management, shuffle tuning, memory configuration, and JVM garbage collection. Spark jobs that could be a DuckDB query running locally in seconds instead take minutes on a cluster. The Scala/Java ecosystem adds dependency management overhead. For most indie hackers, you'll never need Spark — DuckDB handles far more than you'd think.