Data Processing

Polars38.4k★

Polars processes tabular data (spreadsheets, CSVs, database exports, log files) dramatically faster than pandas. We're talking 5-50x faster on real workloads. It's a DataFrame library written in Rust that runs on Python, Node.js, and Rust, and it's designed to handle datasets that would make pandas cry. Fully free under MIT. No paid tier, no cloud service, no enterprise version. The team behind Polars runs a consulting business, not a SaaS product. There's nothing to host: it's a Python/Node.js package. `pip install polars` and you're running. The API is intentionally different from pandas (lazy evaluation, expression-based) which means there's a learning curve, but the design is more consistent and less error-prone. Solo developers: if you touch data, learn Polars. The speed is immediately noticeable on anything over 100K rows. Small teams: use it for ETL pipelines, report generation, data analysis. Large teams: Polars handles datasets that would require Spark in a pandas world, millions of rows on a single machine. The catch: Polars is not pandas. Your existing pandas code won't just work. The API is different by design, and the ecosystem of pandas-compatible libraries (like scikit-learn expecting DataFrames) sometimes needs adapters. The migration cost is real but the performance payoff is substantial.

Kafka32.5k★

Kafka handles high-throughput, fault-tolerant event streaming for systems that process millions of messages per second. It's a distributed event streaming platform that handles millions of messages per second with durability guarantees. Picture a highly reliable conveyor belt for data: producers put messages on, consumers take them off, and nothing gets lost. Apache 2.0. Kafka stores streams of events durably and in order. Consumers can replay from any point in history. Built-in partitioning handles horizontal scaling. Kafka Connect integrates with hundreds of data sources and sinks. Fully free from Apache. Confluent (founded by Kafka creators) offers Confluent Cloud starting at $0.015/GB ingested, with a free tier of $400 in credits. AWS MSK starts at ~$0.21/hr per broker. The catch: running Kafka yourself is a serious commitment. A production cluster needs ZooKeeper (or the newer KRaft mode), at least 3 brokers, proper disk provisioning, and someone who understands topic partitioning, consumer groups, and rebalancing. This is not a weekend project. Most teams under 10 engineers should use a managed service or consider simpler alternatives.

Flink26.0k★

Flink processes streaming data at scale: real-time event processing, continuous ETL, streaming analytics, all with exactly-once processing guarantees. Picture a factory assembly line for data: events flow in, get transformed, aggregated, and routed, all with exactly-once guarantees so nothing gets lost or double-counted. Apache 2.0. Flink handles both stream processing (real-time) and batch processing (historical) through the same API. It manages state across billions of events, handles late-arriving data with watermarks, and checkpoints automatically for fault tolerance. Fully free. No paid tier from Apache. Confluent and AWS offer managed Flink services ($0.11-0.18/hr per compute unit on AWS), but the open source version is complete. The catch: Flink is not simple. Setting up a production Flink cluster requires serious ops knowledge: YARN or Kubernetes deployment, tuning checkpointing intervals, managing state backends (RocksDB), monitoring backpressure. This is enterprise infrastructure. A solo developer processing a few thousand events per second should look at simpler tools first.

Prefect22.3k★

Prefect orchestrates your data pipelines, ETL jobs, ML training runs, and scheduled tasks, handling failures intelligently. It's a scheduler that actually understands when things fail and knows how to retry, alert, and recover. Prefect's Python library is fully open source (Apache 2.0). You write normal Python functions, decorate them with @flow and @task, and Prefect handles scheduling, retries, logging, and dependency tracking. The open source server gives you a dashboard, API, and all core orchestration features. Self-hosting the Prefect server is moderate effort. It's a Python app backed by Postgres. Docker Compose gets you running in 30 minutes. You'll need to maintain the server, database, and workers yourself. Prefect Cloud is where the paid tiers live: free tier gives you a managed server with limited features, Pro at $500/mo adds RBAC, audit logs, and service accounts. Enterprise adds SSO and custom retention. Solo developers: self-host for free or use the Cloud free tier. Small teams: Cloud free tier works until you need RBAC. Growing teams: the $500/mo Pro tier is worth it when managing access across 10+ people costs more in time than money. The catch: Prefect v2 was a major rewrite from v1, and the migration was rough. The ecosystem is stable now, but it burned some trust.

airbyte21.2k★

Airbyte is the open source ETL platform for moving data between services. Postgres to Snowflake, Salesforce to BigQuery, Stripe to your data warehouse: pick from hundreds of connectors and let Airbyte handle extract, load, and scheduling. The self-hosted version is free and covers every connector the team ships. Running it yourself is real work. You're deploying a platform with a scheduler, workers, and a database, not a CLI. Kubernetes is the blessed path. Expect a few hours a month on connector updates, credential rotation, and failure monitoring. Solo: Cloud free tier. Small team with a data engineer: self-host and save hundreds. Enterprise: self-host for compliance, buy the license for SSO. The Cloud tier is usage-based and most small teams land at $100-500/mo. The catch: the big connector count is marketing. The top 30 are production-grade. The long tail is community-maintained and sometimes broken on the latest API version. Test every connector you plan to depend on.

Trino12.8k★

Trino queries across all your data sources (Postgres, S3, Elasticsearch, spreadsheets) with standard SQL. It's a distributed SQL query engine that connects to dozens of data sources and lets you join across them like they're one database. Formerly known as PrestoSQL (the original creators of Presto at Facebook forked after a dispute), Trino is the community-driven continuation. Apache 2.0, used by companies like Netflix, LinkedIn, and Lyft. The engine is free. Managed options include Starburst (the commercial company founded by Trino's creators) starting around $2/hr for a small cluster, and AWS Athena which is Trino under the hood at $5/TB scanned. The catch: Trino is a query engine, not a database. It doesn't store data; it reads from where your data already lives. Running it yourself means managing a coordinator + workers cluster, which is real ops work. And for single-source queries, it's slower than querying that source directly. Trino shines specifically when you need to federate across multiple sources.

dbt12.7k★

Dbt (data build tool) brings software engineering to your data work. Version control for your SQL. Tests for your transformations. Documentation that stays current. Dependency management between your queries. You write SQL SELECT statements, dbt handles the CREATE TABLE/VIEW, dependency ordering, testing, and documentation. It's 'what if we treated SQL like real code instead of throwaway scripts.' massive adoption in data teams. dbt is the standard tool for the 'analytics engineering' role that barely existed five years ago. The catch: the open source core (dbt-core) vs the cloud platform (dbt Cloud) split is where it gets complicated. dbt-core is free and powerful. dbt Cloud adds a UI, scheduling, CI, and the IDE, and that's where dbt Labs makes money. Also: dbt is SQL-only. If your transformations need Python logic, you're adding complexity.

kreuzberg8.3k★

Kreuzberg rips text, metadata, and structured data out of 91+ file formats. PDFs, Word docs, images, source code in 248 languages, you name it. The Rust core makes it fast, and bindings exist for Python, Node.js, Go, Ruby, Java, and C#. Completely open source under the Elastic License. Deploying it is straightforward: Docker container, CLI binary, REST API, or even an MCP server for AI tool chains. The image is around 1.3GB because of OCR backends (Tesseract, PaddleOCR), but once it's running, it handles batch processing with configurable parallelism and streaming for large files. Solo devs building document pipelines get immediate value. Teams doing search indexing or RAG will appreciate the format coverage, since most alternatives force you to stitch together multiple libraries. One tool that handles everything from scanned receipts to source code. The catch: the Elastic License means you can't offer it as a managed service without a commercial agreement. Building an internal tool? You're fine. Reselling document extraction? Talk to their team first.

tilelang6.1k★

TileLang is a domain-specific language that makes that dramatically less painful. Writing CUDA or Triton kernels by hand is notoriously difficult. TileLang gives you a higher-level way to express tile-based computations (the pattern most GPU work follows) and compiles them down to optimized code for NVIDIA, AMD, and other accelerators. Basically, it's a step above raw CUDA but below a full ML framework. You describe your computation in terms of tiles (blocks of data), and TileLang handles the memory management, thread scheduling, and hardware-specific optimizations that normally take weeks to get right. Completely free and open source. No paid tier. The catch: this is deeply specialized. If you're not writing custom GPU kernels, this tool has zero relevance to you. The target audience is ML researchers, HPC engineers, and framework developers, maybe a few thousand people globally. The project is young (, emerging), documentation is still maturing, and you'll need solid GPU programming knowledge to use it effectively. OpenAI's Triton is the more established alternative in this space, with a larger community and more learning resources. NVIDIA's CUTLASS is another option if you're locked to NVIDIA hardware.

ceres-solver4.5k★

Ceres Solver does nonlinear least squares optimization. In plain terms: you give it a bunch of equations that don't quite match reality, and it finds the values that make them as close as possible. Google built this for their own use (Street View camera calibration, among other things). It handles problems with thousands of parameters and millions of observations. The solver is written in C++ and runs fast. It exploits the sparse structure of problems so it doesn't waste time on zeros. Apache-like license (New BSD). Used in robotics, computer vision, photogrammetry, and scientific computing. If your problem involves fitting curves, calibrating sensors, or bundle adjustment, Ceres is the standard answer. No paid tier. No cloud. No managed anything. This is a pure C++ library you compile and link. The catch: this is not a beginner tool. You need to understand your optimization problem mathematically before Ceres can help. The API is powerful but assumes you know what a cost function is and how to define one. Documentation is thorough but academic.

quix-streams1.5k★

Quix Streams gives you a DataFrame-like API for streaming data. Write Kafka consumers and producers using familiar pandas-style syntax instead of raw consumer loops and serialization boilerplate. You define transformations as chained operations (filter, map, aggregate, window) and Quix handles the Kafka plumbing underneath. It's specifically designed for Python developers who need stream processing but don't want to learn the full Kafka Streams Java API. Apache 2.0, fully free. No paid tier in the library itself. Quix does offer a managed cloud platform for the full pipeline (ingestion, processing, deployment), but the Python library is standalone. The catch: the community is small. If you hit an edge case, you're reading source code, not Stack Overflow. And it's Kafka-only; if you're on Pulsar, RabbitMQ, or Redpanda, you need something else.

arc589★

Arc combines DuckDB's SQL engine with Parquet storage and Apache Arrow's in-memory format for processing large tabular files. The pitch: 18M+ records per second on analytical queries, deployed as a single Go binary. It's a lightweight analytical database you can spin up without a cluster. Load your data in Parquet format, query it with standard SQL, and get results faster than most traditional databases can scan the data. It's designed for analytics workloads where you're aggregating, filtering, and joining large tables, not for transactional OLTP with lots of small writes. The project is early stage (, nascent tier). The enterprise page exists at basekick.net but specific pricing isn't public yet. The catch: this is very new. DuckDB itself is more mature and does much of what Arc does. The AGPL license means any network service using Arc must open-source its code, or you need an enterprise license. The documentation is thin, the community is small, and production battle-testing is limited. If you need a fast analytical query engine today, DuckDB is the safer bet. Arc is one to watch if the DuckDB + Parquet + Arrow integration proves to be more than the sum of its parts.