Data Processing

PaddleOCR87.2k★

PaddleOCR is the workhorse of open source OCR. Built by Baidu, it reads text out of images and documents in over 100 languages, and the newer pieces go further: PP-StructureV3 pulls layout and tables into Markdown or JSON, and PaddleOCR-VL is a vision-language model that parses whole documents including formulas and charts. Apache 2.0, free, and cleared for commercial use. It runs on both CPU and GPU, so you don't strictly need a graphics card to get started, though a GPU helps at volume. Setup is a Python install plus the model weights; the v6 models are faster and more accurate than before, and there's even browser inference via PaddleOCR.js. It's more moving parts than a single-purpose engine, but the toolkit design means you pull in only the pieces you need. For most teams replacing a paid OCR API, this is the first tool to try. It matches the core job (text, tables, layout) and, unlike some strong research models, its Apache license means you can actually ship it commercially. Solo to large teams: free, and production-grade. If Tesseract feels too bare and you want structure out of the box, this is the upgrade. The catch is complexity. PaddleOCR is a big toolkit with a lot of models, versions, and configuration, and the documentation leans Chinese-first in places. Getting a clean pipeline running takes more reading than a one-line tool. The payoff is worth it, but budget the setup time.

tesseract75.8k★

Tesseract is the OCR engine that's been quietly reading text off images for decades. Point it at a scan or a photo and it hands back the text, in more than 100 languages, as plain text, PDF, hOCR, or a few other formats. It's Apache 2.0 and free, and it's the OCR layer underneath a huge number of other tools. It's refreshingly light. Tesseract runs on CPU with no GPU required, as a command-line tool or a C/C++ library with bindings for most languages. Version 5 uses an LSTM neural engine and is stable and battle-tested. The trade for that simplicity: it does line-level text recognition and nothing else. No layout analysis, no table structure, no form key-value pairs. This is the right tool when you need reliable plain-text extraction and not much more. Solo to large teams: free and dependable. But if your documents have tables, forms, or complex layouts, Tesseract alone won't reconstruct them; you'll pair it with something like PaddleOCR or docling, or reach for a heavier model. The catch is image quality. Tesseract is only as good as what you feed it, with no magic for messy scans, skew, or low contrast. You'll spend real time on preprocessing (deskewing, thresholding) to get clean results. It reads text well; it does not clean up your inputs for you.

docling64.4k★

Docling turns messy documents into clean, structured data for AI. Feed it PDFs, Word docs, PowerPoints, spreadsheets, images, even HTML or email, and it produces Markdown or JSON with the layout, reading order, tables, formulas, and code preserved. It runs OCR on scanned pages too. Started by IBM Research and now under the LF AI and Data Foundation, it's MIT licensed and free. It's a Python library (3.10+) that runs locally, which is the point: your documents never have to leave your machine, so it works in air-gapped or sensitive environments. It plugs straight into RAG and LLM pipelines with ready integrations for LangChain, LlamaIndex, Crew AI, and Haystack, and there's a CLI and an API server if you'd rather not code against it directly. If your goal is feeding documents to an LLM, Docling is aimed squarely at you. It's less a raw OCR engine and more a document-to-structured-data framework, which is exactly what Textract's Analyze Document does, minus the per-page bill. Solo to large teams: free, and genuinely strong at the RAG-prep job. For plain text off a photo, a lighter tool like Tesseract is enough. The catch is that quality depends on the document. Docling's layout and table understanding is good, but complex or badly scanned PDFs still trip it up, and heavy parsing with vision models wants real compute. It's a framework, so you own the pipeline and the tuning. Powerful, but not a fire-and-forget API.

Spark43.8k★

Apache Spark processes massive datasets (logs, events, transactions) across a cluster of machines in parallel. Basically, MapReduce's faster, more versatile successor. It handles batch processing, streaming, SQL queries, machine learning, and graph processing all in one engine. Apache 2.0, backed by the Apache Software Foundation. This is the industry standard for big data processing. Every major cloud provider offers managed Spark (Databricks, AWS EMR, Google Dataproc, Azure HDInsight). The engine itself is free. You pay for the compute, either your own cluster or a managed service. Databricks (founded by the Spark creators) charges $0.07-$0.55/DBU depending on tier. AWS EMR adds ~$0.015-$0.27/hr per instance on top of EC2 costs. The catch: Spark is not for small data. If your dataset fits in memory on one machine, use Polars or DuckDB. They'll be faster with zero cluster overhead. Spark's power comes with real operational complexity: cluster management, memory tuning, shuffle optimization. It's the right tool when you have big data, and overkill for everything else.

Zod43.4k★

Zod lets you define data shapes once and get both runtime validation and TypeScript types from the same definition. No more writing types AND validation logic separately. MIT license, zero dependencies. You define a schema like `z.object({ name: z.string, age: z.number })` and Zod gives you a validator AND the TypeScript type. Parse untrusted data, get back typed data or a detailed error. Works everywhere TypeScript runs. Fully free. Library-only, no service, no paid tier. Install it and use it. Zod has become the default validation library in the TypeScript ecosystem. Frameworks like tRPC, React Hook Form, and Next.js Server Actions all have first-class Zod integration. If you're building anything in TypeScript, you'll probably end up using it. The catch: Zod schemas can get verbose for complex nested objects. Performance-sensitive applications (validating thousands of objects per second) might notice. Libraries like Valibot and Typebox compile schemas to faster validators. And Zod 3's error messages are good but not always user-friendly out of the box for form validation. For most apps though, none of this matters. It just works.

Polars39.3k★

Polars processes tabular data (spreadsheets, CSVs, database exports, log files) dramatically faster than pandas. We're talking 5-50x faster on real workloads. It's a DataFrame library written in Rust that runs on Python, Node.js, and Rust, and it's designed to handle datasets that would make pandas cry. Fully free under MIT. No paid tier, no cloud service, no enterprise version. The team behind Polars runs a consulting business, not a SaaS product. There's nothing to host: it's a Python/Node.js package. `pip install polars` and you're running. The API is intentionally different from pandas (lazy evaluation, expression-based) which means there's a learning curve, but the design is more consistent and less error-prone. Solo developers: if you touch data, learn Polars. The speed is immediately noticeable on anything over 100K rows. Small teams: use it for ETL pipelines, report generation, data analysis. Large teams: Polars handles datasets that would require Spark in a pandas world, millions of rows on a single machine. The catch: Polars is not pandas. Your existing pandas code won't just work. The API is different by design, and the ecosystem of pandas-compatible libraries (like scikit-learn expecting DataFrames) sometimes needs adapters. The migration cost is real but the performance payoff is substantial.

Kafka33.5k★

Kafka is a distributed event streaming platform that moves millions of messages per second with durability guarantees. Producers put messages on, consumers take them off, and nothing gets lost. It stores streams of events in order, lets consumers replay from any point in history, and partitions data so it scales horizontally. Kafka Connect plugs into hundreds of data sources and sinks. Apache 2.0. Running Kafka yourself used to mean babysitting a separate ZooKeeper ensemble alongside your brokers. That is over. Kafka 4.0 removed ZooKeeper entirely, and KRaft, Kafka's built-in metadata system, is now the only way to run it. That cuts a whole class of operational pain, but a production cluster still needs at least 3 brokers, proper disk provisioning, and someone who understands partitions, consumer groups, and rebalancing. This is not a weekend project. Fully free from Apache. Confluent Cloud (built by Kafka's creators) starts around $0.015/GB ingested with a free credit. AWS MSK runs roughly $0.21/hr per broker. Solo and small teams: use a managed service. Mid-size teams running real volume: self-hosting starts to pay off once you have the ops skill in house. The catch: Kafka is built for scale, and if you don't have the scale, you are paying the complexity tax for nothing. Most teams under 10 engineers should reach for a managed service or a simpler queue before standing up their own cluster.

Flink26.2k★

Flink processes streaming data at scale: real-time event processing, continuous ETL, streaming analytics, all with exactly-once processing guarantees. Picture a factory assembly line for data: events flow in, get transformed, aggregated, and routed, all with exactly-once guarantees so nothing gets lost or double-counted. Apache 2.0. Flink handles both stream processing (real-time) and batch processing (historical) through the same API. It manages state across billions of events, handles late-arriving data with watermarks, and checkpoints automatically for fault tolerance. Fully free. No paid tier from Apache. Confluent and AWS offer managed Flink services ($0.11-0.18/hr per compute unit on AWS), but the open source version is complete. The catch: Flink is not simple. Setting up a production Flink cluster requires serious ops knowledge: YARN or Kubernetes deployment, tuning checkpointing intervals, managing state backends (RocksDB), monitoring backpressure. This is enterprise infrastructure. A solo developer processing a few thousand events per second should look at simpler tools first.

Prefect23.6k★

Prefect orchestrates your data pipelines, ETL jobs, ML training runs, and scheduled tasks, handling failures intelligently. It's a scheduler that actually understands when things fail and knows how to retry, alert, and recover. Prefect's Python library is fully open source (Apache 2.0). You write normal Python functions, decorate them with @flow and @task, and Prefect handles scheduling, retries, logging, and dependency tracking. The open source server gives you a dashboard, API, and all core orchestration features. Self-hosting the Prefect server is moderate effort. It's a Python app backed by Postgres. Docker Compose gets you running in 30 minutes. You'll need to maintain the server, database, and workers yourself. Prefect Cloud is where the paid tiers live: free tier gives you a managed server with limited features, Pro at $500/mo adds RBAC, audit logs, and service accounts. Enterprise adds SSO and custom retention. Solo developers: self-host for free or use the Cloud free tier. Small teams: Cloud free tier works until you need RBAC. Growing teams: the $500/mo Pro tier is worth it when managing access across 10+ people costs more in time than money. The catch: Prefect v2 was a major rewrite from v1, and the migration was rough. The ecosystem is stable now, but it burned some trust.

airbyte21.8k★

Airbyte is the open source ETL platform for moving data between services. Postgres to Snowflake, Salesforce to BigQuery, Stripe to your data warehouse: pick from hundreds of connectors and let Airbyte handle extract, load, and scheduling. The self-hosted version is free and covers every connector the team ships. Running it yourself is real work. You're deploying a platform with a scheduler, workers, and a database, not a CLI. Kubernetes is the blessed path. Expect a few hours a month on connector updates, credential rotation, and failure monitoring. Solo: Cloud free tier. Small team with a data engineer: self-host and save hundreds. Enterprise: self-host for compliance, buy the license for SSO. The Cloud tier is usage-based and most small teams land at $100-500/mo. The catch: the big connector count is marketing. The top 30 are production-grade. The long tail is community-maintained and sometimes broken on the latest API version. Test every connector you plan to depend on.

dbt13.6k★

dbt is how analytics teams turn raw warehouse tables into trustworthy, documented data models, using plain SQL plus the engineering discipline that was always missing from analytics: version control, testing, and dependency tracking. You write SELECT statements, dbt handles the order they run in, tests the output, and builds the documentation. dbt Core, the command-line tool, is open source and free, and as of the v2 line it is squarely Apache-2.0. The big 2026 shift is Fusion. dbt rewrote its engine in Rust and put it under dbt 2.0 (in alpha as of this writing), promising faster parsing and real-time SQL validation. The catch is licensing nuance: the classic dbt Core stays Apache-2.0, but Fusion is a mix of source-available (ELv2), Apache, and proprietary parts, and the new Fusion-powered VS Code extension is free only up to 15 users before it needs a commercial license. Self-hosting the core still costs nothing and runs anywhere Python does; you bring your own scheduler (Airflow, cron, Dagster) to run it in production. Solo analysts and small teams: dbt Core plus a scheduler is the free, complete path, and it is the industry standard for a reason. dbt Cloud starts to make sense when you want the hosted IDE, scheduling, and the Semantic Layer without assembling them yourself: Developer is free for one seat, Starter runs $100/seat/mo, and Enterprise (Mesh, advanced catalog) is custom-quoted. The honest line: you are paying for orchestration and collaboration, not for dbt the transformation tool, which you can always run yourself. Watch the Fusion rollout. It is the future dbt is steering toward, but it is where the open-source line gets blurry, source-available here, a per-seat gate there. The transformation core you depend on is still free and open; just read the license on each new piece before you build on it.

Trino13.1k★

Trino queries across all your data sources (Postgres, S3, Elasticsearch, spreadsheets) with standard SQL. It's a distributed SQL query engine that connects to dozens of data sources and lets you join across them like they're one database. Formerly known as PrestoSQL (the original creators of Presto at Facebook forked after a dispute), Trino is the community-driven continuation. Apache 2.0, used by companies like Netflix, LinkedIn, and Lyft. The engine is free. Managed options include Starburst (the commercial company founded by Trino's creators) starting around $2/hr for a small cluster, and AWS Athena which is Trino under the hood at $5/TB scanned. The catch: Trino is a query engine, not a database. It doesn't store data; it reads from where your data already lives. Running it yourself means managing a coordinator + workers cluster, which is real ops work. And for single-source queries, it's slower than querying that source directly. Trino shines specifically when you need to federate across multiple sources.

PySyft9.9k★

PySyft solves a specific problem: analyzing data you are not allowed to see. A hospital has patient records, you have a model, and legal will never let those two things sit on the same machine. PySyft lets you submit code against someone else's private dataset, the owner reviews and approves the job, and only the approved results come back. The raw data never moves. Apache 2.0, no paid tier. Install is `uv pip install syft-client`, and recent versions ride on cloud storage you already have (Google Drive, Microsoft 365) rather than asking you to stand up a dedicated domain server the way earlier releases did. That drops the ops burden a lot on the data scientist side. The data owner side is where the real work sits: someone has to review submitted jobs. The syft-bg package adds auto-approval rules, email notifications, and a TUI dashboard for that queue, but deciding what is safe to auto-approve stays a human call. Research groups, healthcare collaborations, and anyone doing cross-organization analysis under a data-sharing agreement are the fit. Solo devs will not need this. Small teams inside one company will not either, because you already have access to your own data. It starts earning its keep the moment a second organization is involved and lawyers are in the room. The catch: this is infrastructure for a workflow most teams do not have yet. Approval queues and remote execution add friction to every single query. If your compliance problem can be solved with a data-use agreement and a VPN, do that instead. PySyft is for when it cannot.

kreuzberg8.9k★

Kreuzberg rips text, metadata, and structured data out of 91+ file formats. PDFs, Word docs, images, source code in 248 languages, you name it. The Rust core makes it fast, and bindings exist for Python, Node.js, Go, Ruby, Java, and C#. Completely open source under the Elastic License. Deploying it is straightforward: Docker container, CLI binary, REST API, or even an MCP server for AI tool chains. The image is around 1.3GB because of OCR backends (Tesseract, PaddleOCR), but once it's running, it handles batch processing with configurable parallelism and streaming for large files. Solo devs building document pipelines get immediate value. Teams doing search indexing or RAG will appreciate the format coverage, since most alternatives force you to stitch together multiple libraries. One tool that handles everything from scanned receipts to source code. The catch: the Elastic License means you can't offer it as a managed service without a commercial agreement. Building an internal tool? You're fine. Reselling document extraction? Talk to their team first.

tilelang7.1k★

TileLang is a domain-specific language that makes that dramatically less painful. Writing CUDA or Triton kernels by hand is notoriously difficult. TileLang gives you a higher-level way to express tile-based computations (the pattern most GPU work follows) and compiles them down to optimized code for NVIDIA, AMD, and other accelerators. Basically, it's a step above raw CUDA but below a full ML framework. You describe your computation in terms of tiles (blocks of data), and TileLang handles the memory management, thread scheduling, and hardware-specific optimizations that normally take weeks to get right. Completely free and open source. No paid tier. The catch: this is deeply specialized. If you're not writing custom GPU kernels, this tool has zero relevance to you. The target audience is ML researchers, HPC engineers, and framework developers, maybe a few thousand people globally. The project is young (, emerging), documentation is still maturing, and you'll need solid GPU programming knowledge to use it effectively. OpenAI's Triton is the more established alternative in this space, with a larger community and more learning resources. NVIDIA's CUTLASS is another option if you're locked to NVIDIA hardware.

lance6.9k★

Lance is a data format built for AI, not retrofitted for it. If you work with images, video, audio, text, and embeddings together, Parquet and Iceberg start to hurt: random access is slow and they were never meant for blobs. Lance fixes that. It claims 100x faster random access than Parquet, with vector search, full-text search, and SQL analytics in one format. Apache 2.0, and it drops into Pandas, DuckDB, Polars, PyArrow, Spark, and Ray. Because it's a format and not a service, there's almost nothing to run. You convert from Parquet in a couple of lines and query it from the tools you already use. It ships ACID transactions, time-travel versioning, and a vector index, so the same files that hold your training data also serve similarity search. For multimodal AI pipelines, the constant reshuffling between a blob store, a feature store, and a vector database is exactly the tax this removes. This is for anyone building AI data pipelines who is tired of gluing four systems together. Solo and small teams: adopt it freely, there is no paid tier on the format itself. The company behind it sells LanceDB Cloud if you want a managed database on top, but the format and the local workflow cost nothing. The catch is maturity. Parquet and Iceberg have a decade of tooling, integrations, and battle-testing behind them. Lance is newer and moving fast, which means fewer integrations and the occasional rough edge. If your stack lives entirely in established lakehouse tooling, adopt it where multimodal access actually hurts, not everywhere at once.

MonkeyOCR6.6k★

MonkeyOCR turns documents into structured data: it reads a PDF or a scan and pulls out the text, the tables, and the math formulas in the correct reading order. If you've ever needed clean data out of messy PDFs, that's the job. It's a compact model, 1.2B or 3B parameters, that punches well above its size, beating heavier open models and even some commercial APIs on document benchmarks. The code is Apache 2.0. You run it yourself on a GPU. A quantized version fits on an 8GB card like a 4060, and the full model is comfortable on a 3090, 4090, or A100, working through roughly one page a second depending on hardware. That's the trade: no per-page API fees, but you need a GPU and the willingness to set up a Python inference stack. For pulling structure out of documents, this is one of the strongest open options right now, clearly ahead of Marker and Docling on benchmarks and competitive with PaddleOCR. Solo or research use: excellent, and free. Teams processing documents at volume: the economics beat a cloud OCR API quickly if you already have GPUs. Here's the catch, and it matters. The authors state the model is intended for academic research and non-commercial use only, even though the code carries an Apache license. If you're shipping this inside a commercial product, read the model terms carefully before you build on it.

dlt5.7k★

dlt turns "write an API-to-warehouse pipeline" from a weekend project into an afternoon. It's a Python library: declare a source, point it at Postgres, BigQuery, Snowflake, or DuckDB, and it handles schema inference, schema evolution, incremental loading, and retries. Apache 2.0, and the maintainers state plainly the library stays free forever. There's no server to run. Pipelines are plain Python that execute wherever your code already runs: cron, Airflow, GitHub Actions, a Lambda. That's the real difference from airbytehq/airbyte, which wants a platform deployment with a UI and workers. Your ops burden is whatever orchestration you already have. Solo data engineers and small teams: this is the sweet spot, especially if you'd rather write Python than click through connector UIs. Teams that want managed runtime, monitoring, and governed collaboration can pay for dltHub Pro. Non-engineers who need hundreds of prebuilt connectors maintained by someone else should use Airbyte or stay on Fivetran. The catch: dlt gives you a framework, not Fivetran's connector catalog. The long tail of SaaS sources means you write and maintain the extraction code yourself.

ceres-solver4.5k★

Ceres Solver does nonlinear least squares optimization. In plain terms: you give it a bunch of equations that don't quite match reality, and it finds the values that make them as close as possible. Google built this for their own use (Street View camera calibration, among other things). It handles problems with thousands of parameters and millions of observations. The solver is written in C++ and runs fast. It exploits the sparse structure of problems so it doesn't waste time on zeros. Apache-like license (New BSD). Used in robotics, computer vision, photogrammetry, and scientific computing. If your problem involves fitting curves, calibrating sensors, or bundle adjustment, Ceres is the standard answer. No paid tier. No cloud. No managed anything. This is a pure C++ library you compile and link. The catch: this is not a beginner tool. You need to understand your optimization problem mathematically before Ceres can help. The API is powerful but assumes you know what a cost function is and how to define one. Documentation is thorough but academic.

root3.3k★

ROOT came out of CERN to handle the data volumes particle physics produces, and it is still the workhorse there. It is a C++ framework for storing, processing, and visualizing enormous datasets, with histogramming, curve fitting, and statistical modeling built in. If you work with measurements at a scale that breaks normal tools, this was built for exactly that. LGPL-2.1, free, decades of development behind it. The power comes with a real learning curve. ROOT is C++ first, has its own file format and idioms, and the build and install are non-trivial. Outside high-energy physics, a lot of its conventions feel alien, and for general analysis the modern Python stack (pandas, NumPy, Arrow, Polars) is friendlier and moves faster. ROOT has Python bindings, but you can feel that you are visiting a C++ world. Use it if you are in physics or another field that already lives in ROOT, or if you specifically need its statistical and histogramming machinery on massive datasets. For everyday data work, reach for the Python tools instead. There is no paid tier and no vendor; support is the community forum and the issue tracker. The catch is fit. ROOT is brilliant at the narrow, heavy job it was designed for and overkill for almost everything else. If you are not sure you need it, you probably do not.

gravitino3.1k★

Apache Gravitino is an open source data catalog that gives you a unified metadata layer across many sources: Hive, MySQL, S3, HDFS, Iceberg, Lance, and more. Apache 2.0 and free. The pitch is federation: instead of copying metadata into a central catalog, it reflects each source live, so what you query is what's actually there. It plugs into Trino and Spark as a query catalog, supports geo-distributed metadata syncing, and adds access control and auditing across your estate. Self-hosting is via Docker or binary; this is a real infrastructure piece, not a small daemon, and it earns its complexity once you have multiple data systems to govern together. For solo developers or small teams with one Postgres and one S3 bucket, this is overkill: a Hive Metastore or your DB's native catalog is enough. Larger teams with mixed engines and regions are the audience. It positions itself as an open alternative to Databricks Unity Catalog and Snowflake Polaris. The catch: you commit to running a metadata service that becomes load-bearing for query engines. If it goes down or drifts, queries fail in confusing ways. Solid Apache project work, but a federated catalog is not a small thing to operate; budget the SRE time before you adopt.

bruin1.7k★

Bruin is a command-line framework for building data pipelines end to end. You write transformations in SQL or Python, pull data in from many sources, and attach data-quality checks, all in one tool instead of stitching together an ingestion service, a transformation tool, and an orchestrator. Apache-2.0, free, and it runs locally, on a server, or inside CI. Because it is a CLI you own, you also run it. There is no managed cloud here: scheduling, infrastructure, and warehouse credentials are your responsibility. For teams comfortable with that, it is liberating, one tool, version-controlled, no per-row billing. For teams that want a dashboard and a vendor to call, it is more hands-on than they may want. Call the ops moderate. This fits data and analytics engineers who would rather have one tool covering ingestion, transformation, and quality than assemble Fivetran plus dbt plus Airflow and pay for each. Solo and small teams: free and capable. Larger teams: still free to run, but weigh the operational load against managed options. If you want fully no-ops pipelines with a support contract, that is Fivetran and dbt Cloud, billed by usage and seats. The catch is that you are the platform. Bruin gives you the pieces in one place; it does not give you a managed service running them. That is exactly the point if you want control and the wrong call if you want someone else holding the pager.

quix-streams1.6k★

Quix Streams gives you a DataFrame-like API for streaming data. Write Kafka consumers and producers using familiar pandas-style syntax instead of raw consumer loops and serialization boilerplate. You define transformations as chained operations (filter, map, aggregate, window) and Quix handles the Kafka plumbing underneath. It's specifically designed for Python developers who need stream processing but don't want to learn the full Kafka Streams Java API. Apache 2.0, fully free. No paid tier in the library itself. Quix does offer a managed cloud platform for the full pipeline (ingestion, processing, deployment), but the Python library is standalone. The catch: the community is small. If you hit an edge case, you're reading source code, not Stack Overflow. And it's Kafka-only; if you're on Pulsar, RabbitMQ, or Redpanda, you need something else.

amphi-etl1.4k★

Amphi is a visual, low-code canvas for building data pipelines that compile down to real Python. You drag components onto a board to clean, join, and reshape data, and under the hood it generates pandas and DuckDB code, so you're not locked into a black box. It runs standalone or as a JupyterLab extension, connects to CSVs, Postgres, Snowflake, and REST APIs, and it's free to use. Getting started is easy: pip install, `amphi start`, and you're in the canvas. No GPU, just a Python environment. The catch on setup is scope, it's built for lightweight ETL and data prep, not heavy-duty distributed processing, so match the tool to the size of the job. For analysts and developers who want Alteryx or KNIME-style visual data prep without the enterprise license, this is a useful free option, solo or small team. The fact that it emits readable Python means you can graduate a pipeline out of the visual tool and into code when you outgrow it. The catch is the license. Amphi is under the Elastic License v2, which is source-available, not truly open source. You can self-host and use it freely for your own work, but you can't turn around and offer it as a competing hosted service. For most internal use that's a non-issue, just know it's not MIT or Apache before you build a product on it.

arc634★

Arc combines DuckDB's SQL engine with Parquet storage and Apache Arrow's in-memory format for processing large tabular files. The pitch: 18M+ records per second on analytical queries, deployed as a single Go binary. It's a lightweight analytical database you can spin up without a cluster. Load your data in Parquet format, query it with standard SQL, and get results faster than most traditional databases can scan the data. It's designed for analytics workloads where you're aggregating, filtering, and joining large tables, not for transactional OLTP with lots of small writes. The project is early stage (, nascent tier). The enterprise page exists at basekick.net but specific pricing isn't public yet. The catch: this is very new. DuckDB itself is more mature and does much of what Arc does. The AGPL license means any network service using Arc must open-source its code, or you need an enterprise license. The documentation is thin, the community is small, and production battle-testing is limited. If you need a fast analytical query engine today, DuckDB is the safer bet. Arc is one to watch if the DuckDB + Parquet + Arrow integration proves to be more than the sum of its parts.