Data Processing

PaddleOCR90.0k★

PaddleOCR is the workhorse of open source OCR. It comes out of Baidu's PaddlePaddle ecosystem and reads text from images and documents in over 100 languages. The newer pieces go further: PP-StructureV3 pulls layout and tables into Markdown or JSON, and PaddleOCR-VL is a small vision-language model that parses whole documents including formulas, tables, seals, and charts. Apache 2.0, free, and cleared for commercial use. It runs on CPU or GPU, so you do not strictly need a graphics card to start, though a GPU helps at volume. The PP-OCRv6 models claim a fivefold CPU speedup over v5 with better accuracy, and PaddleOCR.js runs the older PP-OCRv5 model right in the browser. No server at all is also an option: the official site offers a free hosted API, capped at 3,000 pages per model per day, with a Baidu AI Studio token. For most teams replacing a paid OCR API, this is the first tool to try. It matches the core job: text, tables, layout. Solo to large teams: free. docling-project/docling is the alternative when the goal is feeding documents to an LLM. The catch is complexity. It is a big toolkit with many models, versions, and configuration options, and the docs lean Chinese-first in places. The repo has also been quiet since its June 2026 release. Budget the setup time.

tesseract76.5k★

Tesseract is the OCR engine that's been quietly reading text off images for decades. Point it at a scan or a photo and it hands back the text, in more than 100 languages, as plain text, PDF, hOCR, or a few other formats. It's Apache 2.0 and free, and it's the OCR layer underneath a huge number of other tools. It's refreshingly light. Tesseract runs on CPU with no GPU required, as a command-line tool or a C/C++ library with bindings for most languages. Version 5 uses an LSTM neural engine and is stable and battle-tested. The trade for that simplicity: it does line-level text recognition and nothing else. No layout analysis, no table structure, no form key-value pairs. This is the right tool when you need reliable plain-text extraction and not much more. Solo to large teams: free and dependable. But if your documents have tables, forms, or complex layouts, Tesseract alone won't reconstruct them; you'll pair it with something like PaddleOCR or docling, or reach for a heavier model. The catch is image quality. Tesseract is only as good as what you feed it, with no magic for messy scans, skew, or low contrast. You'll spend real time on preprocessing (deskewing, thresholding) to get clean results. It reads text well; it does not clean up your inputs for you.

docling67.6k★

Docling turns messy documents into clean, structured data for AI. Feed it PDFs, Word docs, PowerPoints, spreadsheets, images, HTML, email, even audio and video, and it produces Markdown, HTML, or lossless JSON with the layout, reading order, tables, formulas, and code preserved. Started at IBM Research Zurich and now hosted by the LF AI and Data Foundation, it is MIT licensed and free. It is a Python library (3.10 or later) that runs locally, which is the point: documents never leave your machine, so it works in air-gapped or sensitive environments. It plugs into RAG pipelines with integrations for LangChain, LlamaIndex, Crew AI, and Haystack, and there is a CLI, an API server, and an MCP server for agents. Heavy parsing with vision models wants a GPU. Solo to large teams: free, and strong at the RAG-prep job. It is less a raw OCR engine and more a document-to-structured-data framework, the same job as Textract's Analyze Document. Teams that do not want to run it can now buy it: IBM sells Docling for watsonx, a managed version of the same open stack, from $4 per 1,000 pages. The catch is that quality depends on the document. Layout and table understanding is good, but complex or badly scanned PDFs still trip it up. It is a framework, so you own the pipeline and the tuning. Powerful, but not a fire-and-forget API.

Spark44.0k★

Apache Spark processes massive datasets (logs, events, transactions) across a cluster of machines in parallel. Basically, MapReduce's faster, more versatile successor. It handles batch processing, streaming, SQL queries, machine learning, and graph processing all in one engine. Apache 2.0, backed by the Apache Software Foundation. This is the industry standard for big data processing. Every major cloud provider offers managed Spark (Databricks, AWS EMR, Google Dataproc, Azure HDInsight). The engine itself is free. You pay for the compute, either your own cluster or a managed service. Databricks (founded by the Spark creators) charges $0.07-$0.55/DBU depending on tier. AWS EMR adds ~$0.015-$0.27/hr per instance on top of EC2 costs. The catch: Spark is not for small data. If your dataset fits in memory on one machine, use Polars or DuckDB. They'll be faster with zero cluster overhead. Spark's power comes with real operational complexity: cluster management, memory tuning, shuffle optimization. It's the right tool when you have big data, and overkill for everything else.

Zod44.0k★

Zod lets you define data shapes once and get both runtime validation and TypeScript types from the same definition. No more writing types AND validation logic separately. MIT license, zero dependencies. You define a schema like z.object({ name: z.string, age: z.number }) and Zod gives you a validator AND the TypeScript type. Parse untrusted data, get back typed data or a detailed error. Works everywhere TypeScript runs. Fully free. Library-only, no service, no paid tier. Install it and use it. Zod has become the default validation library in the TypeScript ecosystem. Frameworks like tRPC, React Hook Form, and Next.js Server Actions all have first-class Zod integration. If you're building anything in TypeScript, you'll probably end up using it. The catch: Zod schemas can get verbose for complex nested objects. Performance-sensitive applications (validating thousands of objects per second) might notice. Libraries like Valibot and Typebox compile schemas to faster validators. And Zod 3's error messages are good but not always user-friendly out of the box for form validation. For most apps though, none of this matters. It just works.

Polars39.7k★

Polars processes tabular data (spreadsheets, CSVs, database exports, log files) dramatically faster than pandas. We're talking 5-50x faster on real workloads. It's a DataFrame library written in Rust that runs on Python, Node.js, and Rust, and it's designed to handle datasets that would make pandas cry. The library is MIT and open source forever. pip install polars and you're running, with nothing to host. What's new is Polars Cloud, a managed service that runs the same API on remote hardware and bills only for query execution. It is optional, the local library is not a crippled version of it, and nothing in the open source package is gated behind the cloud. The API is intentionally different from pandas (lazy evaluation, expression-based), so there is a learning curve, but the design is more consistent and less error-prone. Solo developers: if you touch data, learn Polars. The speed is immediately noticeable on anything over 100K rows. Small teams: use it for ETL pipelines, report generation, and data analysis. Large teams: Polars handles on a single machine what used to require Spark, and if you outgrow one box, Polars Cloud scales the same code out without a rewrite. The catch: Polars is not pandas. Your existing pandas code will not just work. The API is different by design, and the ecosystem of pandas-compatible libraries (like scikit-learn expecting DataFrames) sometimes needs adapters. The migration cost is real, but the performance payoff is substantial.

Kafka33.8k★

Kafka is a distributed event streaming platform that moves millions of messages per second with durability guarantees. Producers put messages on, consumers take them off, and nothing gets lost. It stores streams of events in order, lets consumers replay from any point in history, and partitions data so it scales horizontally. Kafka Connect plugs into hundreds of data sources and sinks. Apache 2.0. Running Kafka yourself used to mean babysitting a separate ZooKeeper ensemble alongside your brokers. That is over. Kafka 4.0 removed ZooKeeper entirely, and KRaft, Kafka's built-in metadata system, is now the only way to run it. That cuts a whole class of operational pain, but a production cluster still needs at least 3 brokers, proper disk provisioning, and someone who understands partitions, consumer groups, and rebalancing. This is not a weekend project. Fully free from Apache. Confluent Cloud (built by Kafka's creators) starts around $0.015/GB ingested with a free credit. AWS MSK runs roughly $0.21/hr per broker. Solo and small teams: use a managed service. Mid-size teams running real volume: self-hosting starts to pay off once you have the ops skill in house. The catch: Kafka is built for scale, and if you don't have the scale, you are paying the complexity tax for nothing. Most teams under 10 engineers should reach for a managed service or a simpler queue before standing up their own cluster.

Flink26.4k★

Flink processes streaming data at scale: real-time event processing, continuous ETL, streaming analytics, all with exactly-once processing guarantees. Picture a factory assembly line for data: events flow in, get transformed, aggregated, and routed, all with exactly-once guarantees so nothing gets lost or double-counted. Apache 2.0. Flink handles both stream processing (real-time) and batch processing (historical) through the same API. It manages state across billions of events, handles late-arriving data with watermarks, and checkpoints automatically for fault tolerance. Fully free. No paid tier from Apache. Confluent and AWS offer managed Flink services ($0.11-0.18/hr per compute unit on AWS), but the open source version is complete. The catch: Flink is not simple. Setting up a production Flink cluster requires serious ops knowledge: YARN or Kubernetes deployment, tuning checkpointing intervals, managing state backends (RocksDB), monitoring backpressure. This is enterprise infrastructure. A solo developer processing a few thousand events per second should look at simpler tools first.

Prefect23.9k★

Prefect orchestrates your data pipelines, ETL jobs, ML training runs, and scheduled tasks, handling failures intelligently. It's a scheduler that actually understands when things fail and knows how to retry, alert, and recover. Prefect's Python library is fully open source (Apache 2.0). You write normal Python functions, decorate them with @flow and @task, and Prefect handles scheduling, retries, logging, and dependency tracking. The open source server gives you a dashboard, API, and all core orchestration features. Self-hosting the Prefect server is moderate effort. It's a Python app backed by Postgres. Docker Compose gets you running in 30 minutes. You'll need to maintain the server, database, and workers yourself. Prefect Cloud is where the paid tiers live: free tier gives you a managed server with limited features, Pro at $500/mo adds RBAC, audit logs, and service accounts. Enterprise adds SSO and custom retention. Solo developers: self-host for free or use the Cloud free tier. Small teams: Cloud free tier works until you need RBAC. Growing teams: the $500/mo Pro tier is worth it when managing access across 10+ people costs more in time than money. The catch: Prefect v2 was a major rewrite from v1, and the migration was rough. The ecosystem is stable now, but it burned some trust.

airbyte22.1k★

Airbyte is the open source ETL platform for moving data between services. Postgres to Snowflake, Salesforce to BigQuery, Stripe to your data warehouse: pick from hundreds of connectors and let Airbyte handle extract, load, and scheduling. The self-hosted version is free and covers every connector the team ships. Running it yourself is real work. You're deploying a platform with a scheduler, workers, and a database, not a CLI. Kubernetes is the blessed path. Expect a few hours a month on connector updates, credential rotation, and failure monitoring. Solo: Cloud free tier. Small team with a data engineer: self-host and save hundreds. Enterprise: self-host for compliance, buy the license for SSO. The Cloud tier is usage-based and most small teams land at $100-500/mo. The catch: the big connector count is marketing. The top 30 are production-grade. The long tail is community-maintained and sometimes broken on the latest API version. Test every connector you plan to depend on.

anydoc21.6k★

anydoc converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF files into clean Markdown, with a median conversion under 5 milliseconds per document. The reason to care is feeding documents to an LLM without maintaining a pile of format-specific parsers. Every format parses into one shared document model and renders through a single Markdown serializer, so a table from a spreadsheet and a table from a Word file come out the same shape. MIT licensed. The core is Rust with bindings for Node, Python, and the browser through WebAssembly, so install is a package manager one-liner in whichever language you already use. There is no server to run and no state to manage. Format detection reads file contents rather than trusting the extension, which means a mislabeled upload does not blow up the pipeline. Headings, lists, tables, footnotes, and cross-references survive the trip. Free for everything local. Firecrawl sells a hosted product, Parse, which is the same conversion plus OCR models. That is the honest line: if your documents are scanned images rather than real files, you need OCR, and OCR is the paid side. The catch is that "clean Markdown" is carrying weight. Heavily designed PDFs, multi-column layouts, and deeply nested tables remain hard, and no converter has fully solved them. Test it against your ugliest real document before you build a pipeline on top of it.

dbt13.9k★

dbt is how analytics teams turn raw warehouse tables into trustworthy, documented data models, using plain SQL plus the engineering discipline analytics always lacked: version control, testing, and dependency tracking. You write SELECT statements, and dbt works out the run order, tests the output, and builds the docs. The open source core is Apache 2.0. 2026 rewired the project. dbt 2.0 went GA in September as a ground-up Rust rewrite that ships as a single binary with no Python runtime. The naming changed with it: the Apache 2.0 subset is now dbt OSS, and the full dbt binary is proprietary but free to use, including the VS Code extension, whose old 15-user cap is gone. dbt 1.x gets one final minor release, 1.13, with critical fixes promised for several years. Solo analysts and small teams: dbt OSS plus your own scheduler (Airflow, Dagster, cron) is the free, complete path. The hosted dbt platform is free for one developer, then Starter at $100 per user per month for up to five seats. Enterprise, with Mesh, SSO, and SCIM, is custom pricing. The catch is who owns it. dbt Labs merged into Fivetran in June 2026. The combined company says it will maintain dbt OSS under Apache 2.0 indefinitely, but the best developer experience now lives in a proprietary binary whose terms bar reverse engineering. Build on the open subset if that matters to you.

Trino13.2k★

Trino queries across all your data sources (Postgres, S3, Elasticsearch, spreadsheets) with standard SQL. It's a distributed SQL query engine that connects to dozens of data sources and lets you join across them like they're one database. Formerly known as PrestoSQL (the original creators of Presto at Facebook forked after a dispute), Trino is the community-driven continuation. Apache 2.0, used by companies like Netflix, LinkedIn, and Lyft. The engine is free. Managed options include Starburst (the commercial company founded by Trino's creators) starting around $2/hr for a small cluster, and AWS Athena which is Trino under the hood at $5/TB scanned. The catch: Trino is a query engine, not a database. It doesn't store data; it reads from where your data already lives. Running it yourself means managing a coordinator + workers cluster, which is real ops work. And for single-source queries, it's slower than querying that source directly. Trino shines specifically when you need to federate across multiple sources.

PySyft10.0k★

PySyft solves a specific problem: analyzing data you are not allowed to see. A hospital has patient records, you have a model, and legal will never let those two things sit on the same machine. PySyft lets you submit code against someone else's private dataset, the owner reviews and approves the job, and only the approved results come back. The raw data never moves. Apache 2.0, no paid tier. Install is uv pip install syft-client, and recent versions ride on cloud storage you already have (Google Drive, Microsoft 365) rather than asking you to stand up a dedicated domain server the way earlier releases did. That drops the ops burden a lot on the data scientist side. The data owner side is where the real work sits: someone has to review submitted jobs. The syft-bg package adds auto-approval rules, email notifications, and a TUI dashboard for that queue, but deciding what is safe to auto-approve stays a human call. Research groups, healthcare collaborations, and anyone doing cross-organization analysis under a data-sharing agreement are the fit. Solo devs will not need this. Small teams inside one company will not either, because you already have access to your own data. It starts earning its keep the moment a second organization is involved and lawyers are in the room. The catch: this is infrastructure for a workflow most teams do not have yet. Approval queues and remote execution add friction to every single query. If your compliance problem can be solved with a data-use agreement and a VPN, do that instead. PySyft is for when it cannot.

Xberg9.3k★

Xberg pulls clean text, tables, metadata, and structured data out of nearly any file you point it at: PDFs, scanned images, Office documents, audio, archives, even a whole source tree. One Rust engine does format detection, reading, OCR, and extraction, so you stop stitching six libraries into a pipeline. MIT licensed, and the project states it plainly as free for personal, internal, and commercial use. Two things changed in June 2026 that matter if you looked at this before. The project was called Kreuzberg and it spent about eleven weeks under the Elastic License, which blocked offering it as a service. On 2026-06-25 it rebranded to Xberg and relicensed back to MIT on the same day. Deployment is a Docker image, a CLI, a REST server, a Helm chart, or an MCP server, with bindings for fifteen languages. Solo builders wiring up a RAG pipeline get the obvious win: one dependency instead of a pile. Teams doing search indexing or document ingestion get 100 plus formats and 371 code languages behind a single API. Docling (docling-project/docling) is the closer comparison when your inputs are mostly PDFs and you want IBM's layout models. The catch: the commercial side is deliberately closed. The only paid offering is an invitation-only Design Partner program with a managed backend and pricing you find out during the application call. That does not restrict the MIT code at all, but it does mean there is no ordinary support contract to buy when self-hosting goes sideways.

delta9.0k★

Delta Lake makes a pile of Parquet files in S3 behave like a real database table: ACID transactions, schema enforcement, and time travel, so you can query the table as it looked last Tuesday. Spark, Trino, Flink, and Hive can all read and write the same tables without corrupting each other. Apache 2.0, governed by the Linux Foundation, completely free. There's nothing to host. It's a storage format plus connectors you add to a compute engine you already run, which also means the ops burden is inherited: trivial if Spark is already in the building, a project if Delta is the reason you're standing Spark up. This is for data engineering teams running a lakehouse, realistically five-plus engineers with a platform owner. A single analyst with Postgres does not need it. The format rivals are Apache Iceberg and Apache Hudi; Iceberg has the broader vendor momentum, Delta has the deepest Spark integration and Databricks behind it. The catch: this repo is the Scala/Spark implementation. The Python deltalake package most people install comes from delta-rs, a separate project with its own gaps. And protocol upgrades are one-way; old readers fail on tables written with newer protocol versions.

tilelang7.4k★

TileLang is a domain-specific language that makes that dramatically less painful. Writing CUDA or Triton kernels by hand is notoriously difficult. TileLang gives you a higher-level way to express tile-based computations (the pattern most GPU work follows) and compiles them down to optimized code for NVIDIA, AMD, and other accelerators. Basically, it's a step above raw CUDA but below a full ML framework. You describe your computation in terms of tiles (blocks of data), and TileLang handles the memory management, thread scheduling, and hardware-specific optimizations that normally take weeks to get right. Completely free and open source. No paid tier. The catch: this is deeply specialized. If you're not writing custom GPU kernels, this tool has zero relevance to you. The target audience is ML researchers, HPC engineers, and framework developers, maybe a few thousand people globally. The project is young (, emerging), documentation is still maturing, and you'll need solid GPU programming knowledge to use it effectively. OpenAI's Triton is the more established alternative in this space, with a larger community and more learning resources. NVIDIA's CUTLASS is another option if you're locked to NVIDIA hardware.

lance7.1k★

Lance is a data format built for AI, not retrofitted for it. If you work with images, video, audio, text, and embeddings together, Parquet and Iceberg start to hurt: random access is slow and they were never meant for blobs. Lance fixes that. It claims 100x faster random access than Parquet, with vector search, full-text search, and SQL analytics in one format. Apache 2.0, and it drops into Pandas, DuckDB, Polars, PyArrow, Spark, and Ray. Because it's a format and not a service, there's almost nothing to run. You convert from Parquet in a couple of lines and query it from the tools you already use. It ships ACID transactions, time-travel versioning, and a vector index, so the same files that hold your training data also serve similarity search. For multimodal AI pipelines, the constant reshuffling between a blob store, a feature store, and a vector database is exactly the tax this removes. This is for anyone building AI data pipelines who is tired of gluing four systems together. Solo and small teams: adopt it freely, there is no paid tier on the format itself. The company behind it sells LanceDB Cloud if you want a managed database on top, but the format and the local workflow cost nothing. The catch is maturity. Parquet and Iceberg have a decade of tooling, integrations, and battle-testing behind them. Lance is newer and moving fast, which means fewer integrations and the occasional rough edge. If your stack lives entirely in established lakehouse tooling, adopt it where multimodal access actually hurts, not everywhere at once.

MonkeyOCR6.6k★

MonkeyOCR turns documents into structured data: it reads a PDF or a scan and pulls out the text, the tables, and the math formulas in the correct reading order. If you've ever needed clean data out of messy PDFs, that's the job. It's a compact model, 1.2B or 3B parameters, that punches well above its size, beating heavier open models and even some commercial APIs on document benchmarks. The code is Apache 2.0. You run it yourself on a GPU. A quantized version fits on an 8GB card like a 4060, and the full model is comfortable on a 3090, 4090, or A100, working through roughly one page a second depending on hardware. That's the trade: no per-page API fees, but you need a GPU and the willingness to set up a Python inference stack. For pulling structure out of documents, this is one of the strongest open options right now, clearly ahead of Marker and Docling on benchmarks and competitive with PaddleOCR. Solo or research use: excellent, and free. Teams processing documents at volume: the economics beat a cloud OCR API quickly if you already have GPUs. Here's the catch, and it matters. The authors state the model is intended for academic research and non-commercial use only, even though the code carries an Apache license. If you're shipping this inside a commercial product, read the model terms carefully before you build on it.

dlt5.9k★

dlt turns "write an API-to-warehouse pipeline" from a weekend project into an afternoon. It's a Python library: declare a source, point it at Postgres, BigQuery, Snowflake, or DuckDB, and it handles schema inference, schema evolution, incremental loading, and retries. Apache 2.0, and the maintainers state plainly the library stays free forever. There's no server to run. Pipelines are plain Python that execute wherever your code already runs: cron, Airflow, GitHub Actions, a Lambda. That's the real difference from airbytehq/airbyte, which wants a platform deployment with a UI and workers. Your ops burden is whatever orchestration you already have. Solo data engineers and small teams: this is the sweet spot, especially if you'd rather write Python than click through connector UIs. Teams that want managed runtime, monitoring, and governed collaboration can pay for dltHub Pro. Non-engineers who need hundreds of prebuilt connectors maintained by someone else should use Airbyte or stay on Fivetran. The catch: dlt gives you a framework, not Fivetran's connector catalog. The long tail of SaaS sources means you write and maintain the extraction code yourself.

ceres-solver4.6k★

Ceres Solver does nonlinear least squares optimization. In plain terms: you give it a bunch of equations that don't quite match reality, and it finds the values that make them as close as possible. Google built this for their own use (Street View camera calibration, among other things). It handles problems with thousands of parameters and millions of observations. The solver is written in C++ and runs fast. It exploits the sparse structure of problems so it doesn't waste time on zeros. Apache-like license (New BSD). Used in robotics, computer vision, photogrammetry, and scientific computing. If your problem involves fitting curves, calibrating sensors, or bundle adjustment, Ceres is the standard answer. No paid tier. No cloud. No managed anything. This is a pure C++ library you compile and link. The catch: this is not a beginner tool. You need to understand your optimization problem mathematically before Ceres can help. The API is powerful but assumes you know what a cost function is and how to define one. Documentation is thorough but academic.

root3.3k★

ROOT came out of CERN to handle the data volumes particle physics produces, and it is still the workhorse there. It is a C++ framework for storing, processing, and visualizing enormous datasets, with histogramming, curve fitting, and statistical modeling built in. If you work with measurements at a scale that breaks normal tools, this was built for exactly that. LGPL-2.1, free, decades of development behind it. The power comes with a real learning curve. ROOT is C++ first, has its own file format and idioms, and the build and install are non-trivial. Outside high-energy physics, a lot of its conventions feel alien, and for general analysis the modern Python stack (pandas, NumPy, Arrow, Polars) is friendlier and moves faster. ROOT has Python bindings, but you can feel that you are visiting a C++ world. Use it if you are in physics or another field that already lives in ROOT, or if you specifically need its statistical and histogramming machinery on massive datasets. For everyday data work, reach for the Python tools instead. There is no paid tier and no vendor; support is the community forum and the issue tracker. The catch is fit. ROOT is brilliant at the narrow, heavy job it was designed for and overkill for almost everything else. If you are not sure you need it, you probably do not.

gravitino3.2k★

Apache Gravitino is an open source data catalog that gives you a unified metadata layer across many sources: Hive, MySQL, S3, HDFS, Iceberg, Lance, and more. Apache 2.0 and free. The pitch is federation: instead of copying metadata into a central catalog, it reflects each source live, so what you query is what's actually there. It plugs into Trino and Spark as a query catalog, supports geo-distributed metadata syncing, and adds access control and auditing across your estate. Self-hosting is via Docker or binary; this is a real infrastructure piece, not a small daemon, and it earns its complexity once you have multiple data systems to govern together. For solo developers or small teams with one Postgres and one S3 bucket, this is overkill: a Hive Metastore or your DB's native catalog is enough. Larger teams with mixed engines and regions are the audience. It positions itself as an open alternative to Databricks Unity Catalog and Snowflake Polaris. The catch: you commit to running a metadata service that becomes load-bearing for query engines. If it goes down or drifts, queries fail in confusing ways. Solid Apache project work, but a federated catalog is not a small thing to operate; budget the SRE time before you adopt.

bruin1.7k★

Bruin is a command-line framework for building data pipelines end to end. You write transformations in SQL or Python, pull data in from many sources, and attach data-quality checks, all in one tool instead of stitching together an ingestion service, a transformation tool, and an orchestrator. Apache-2.0, free, and it runs locally, on a server, or inside CI. Because it is a CLI you own, you also run it. There is no managed cloud here: scheduling, infrastructure, and warehouse credentials are your responsibility. For teams comfortable with that, it is liberating, one tool, version-controlled, no per-row billing. For teams that want a dashboard and a vendor to call, it is more hands-on than they may want. Call the ops moderate. This fits data and analytics engineers who would rather have one tool covering ingestion, transformation, and quality than assemble Fivetran plus dbt plus Airflow and pay for each. Solo and small teams: free and capable. Larger teams: still free to run, but weigh the operational load against managed options. If you want fully no-ops pipelines with a support contract, that is Fivetran and dbt Cloud, billed by usage and seats. The catch is that you are the platform. Bruin gives you the pieces in one place; it does not give you a managed service running them. That is exactly the point if you want control and the wrong call if you want someone else holding the pager.

quix-streams1.6k★

Quix Streams gives you a DataFrame-like API for streaming data. Write Kafka consumers and producers using familiar pandas-style syntax instead of raw consumer loops and serialization boilerplate. You define transformations as chained operations (filter, map, aggregate, window) and Quix handles the Kafka plumbing underneath. It's specifically designed for Python developers who need stream processing but don't want to learn the full Kafka Streams Java API. Apache 2.0, fully free. No paid tier in the library itself. Quix does offer a managed cloud platform for the full pipeline (ingestion, processing, deployment), but the Python library is standalone. The catch: the community is small. If you hit an edge case, you're reading source code, not Stack Overflow. And it's Kafka-only; if you're on Pulsar, RabbitMQ, or Redpanda, you need something else.

amphi-etl1.4k★

Amphi is a visual, low-code canvas for building data pipelines that compile down to real Python. You drag components onto a board to clean, join, and reshape data, and under the hood it generates pandas and DuckDB code, so you're not locked into a black box. It runs standalone or as a JupyterLab extension, connects to CSVs, Postgres, Snowflake, and REST APIs, and it's free to use. Getting started is easy: pip install, amphi start, and you're in the canvas. No GPU, just a Python environment. The catch on setup is scope, it's built for lightweight ETL and data prep, not heavy-duty distributed processing, so match the tool to the size of the job. For analysts and developers who want Alteryx or KNIME-style visual data prep without the enterprise license, this is a useful free option, solo or small team. The fact that it emits readable Python means you can graduate a pipeline out of the visual tool and into code when you outgrow it. The catch is the license. Amphi is under the Elastic License v2, which is source-available, not truly open source. You can self-host and use it freely for your own work, but you can't turn around and offer it as a competing hosted service. For most internal use that's a non-issue, just know it's not MIT or Apache before you build a product on it.

duckle1.3k★

Duckle moves data between systems and does it on hardware you control. Pull from Postgres, land it in Parquet, transform it with SQL or Python, push it to a warehouse: the same job people currently pay Fivetran or Airbyte per row to run. It compiles to SQL on DuckDB and uses every core on the box, so a bigger instance is a faster pipeline. The project reports 96 million rows out of Postgres to Parquet in 39.9 seconds. Dual licensed MIT or Apache 2.0, and there is no vendor cloud at all. Every pipeline is one file in git, which is the design decision that matters most. Author it on a visual canvas, in SQL, or in Python, then ship that file to your own server. duckle-runner serve runs it headless on a schedule in Docker or on a plain box, with a web console, roles, and an audit trail. Around 385 components cover the usual sources and destinations, plus dbt, CDC, data quality, reverse ETL, and lineage. Nothing is paywalled. No per-row billing, no per-connector fee, no seat count, no account, no telemetry. Solo and small teams get the whole thing. Larger teams get it too, and pay only for the compute they were going to buy anyway. Compare against Airbyte (airbytehq/airbyte) if you want a bigger connector catalog, or dlt (dlt-hub/dlt) if you would rather write pipelines as plain Python. The catch is that it is still labelled beta and it is young. A single-file-in-git pipeline is a great story right up until you need a connector that is not in the 385, and then you are writing it yourself against a project whose API has not settled.

impala1.3k★

Impala runs SQL directly against files already sitting in your data lake, and answers in seconds instead of minutes. Point it at Parquet or ORC files in HDFS or S3, register them in a Hive Metastore, and analysts query them with ordinary SQL from whatever BI tool they already use. Apache 2.0, nothing to buy. Self-hosting is where the honesty starts. Impala is not one binary. You run a catalog daemon, a statestore, and an impalad process on every node in the cluster, plus the metastore and the storage layer underneath. It expects a cluster and it expects somebody who knows Hadoop era operations. Most teams running it in production got there through a Cloudera distribution rather than by assembling it themselves. What you buy with that pain is latency. The daemons stay warm and queries compile to native code through LLVM, so dashboards hitting the same tables all day feel fast in a way Spark SQL does not. Solo developers and small teams: skip it, DuckDB does this on your laptop. Medium teams with a lake already on Hive Metastore or Iceberg: worth benchmarking against Trino. Large organizations already invested in the Hadoop stack: this is the reason to keep that investment. The catch: the ecosystem moved on. Trino took the federated query story, DuckDB took the single machine story, and tagged Impala releases now arrive more than a year apart even though commits land weekly. It is a mature engine serving a shrinking audience, so choosing it today means accepting a smaller community than the alternatives have.

arc676★

Arc combines DuckDB's SQL engine with Parquet storage and Apache Arrow's in-memory format for processing large tabular files. The pitch: 18M+ records per second on analytical queries, deployed as a single Go binary. It's a lightweight analytical database you can spin up without a cluster. Load your data in Parquet format, query it with standard SQL, and get results faster than most traditional databases can scan the data. It's designed for analytics workloads where you're aggregating, filtering, and joining large tables, not for transactional OLTP with lots of small writes. The project is early stage (, nascent tier). The enterprise page exists at basekick.net but specific pricing isn't public yet. The catch: this is very new. DuckDB itself is more mature and does much of what Arc does. The AGPL license means any network service using Arc must open-source its code, or you need an enterprise license. The documentation is thin, the community is small, and production battle-testing is limited. If you need a fast analytical query engine today, DuckDB is the safer bet. Arc is one to watch if the DuckDB + Parquet + Arrow integration proves to be more than the sum of its parts.