For most of the history of data warehousing, running a serious analytical query meant talking to a server somewhere: a cluster, a managed warehouse, or at minimum a separate database process listening on a port. DuckDB broke that assumption. It is a full SQL analytical engine that runs inside your own process, whether that process is a Python script, a Java application, or a Node.js service, with no server to stand up and no network hop to pay for. The idea is not entirely new, but the way it has spread across the language ecosystem in the last few years says something real about where ETL work is heading.
Where DuckDB Came From
DuckDB began in 2018 as a research project by Mark Raasveldt and Hannes Mühleisen at Centrum Wiskunde and Informatica, the Dutch national research institute for mathematics and computer science in Amsterdam, the same institute where Guido van Rossum wrote the first version of Python. According to DuckDB’s own account of its history, the two researchers presented the first version at the SIGMOD conference in 2019 and released it as open source from day one under the permissive MIT license. Their observation was simple: the database research world had built decades of expertise in columnar storage and vectorized query execution, largely through CWI’s earlier MonetDB project, but almost none of it had made its way into a lightweight, embeddable form that a data scientist could just import into a notebook the way they already imported SQLite.
That comparison to SQLite is the one the DuckDB team leans on deliberately. SQLite proved, over two decades, that an in-process, single-file, zero-configuration database could be enormously useful even without horizontal scaling, and it ended up embedded inside browsers, phones, and countless applications precisely because it asked nothing of its environment. DuckDB set out to do the same thing for analytical workloads, the group-by-and-aggregate queries that data warehouses are built for, rather than the row-by-row transactional queries SQLite specializes in. By 2021, the project had grown enough that its founders spun off DuckDB Labs as a commercial support company, while placing the project’s intellectual property in an independent, non-profit DuckDB Foundation to keep the code MIT-licensed permanently.
Part Of A Wider Pattern, Not A Lone Invention
DuckDB is the best-known example of embedded analytics, but it did not appear in isolation. The broader shift toward columnar, in-process query engines has been building since Apache Arrow standardized a shared columnar memory format in 2016, giving different tools a common way to exchange data without copying or serializing it. Apache DataFusion, a Rust-based embeddable query engine, started as an independent project in 2017 and was donated to the Apache Arrow project in 2019 before graduating into its own top-level Apache project in 2024. More recently, chDB packaged ClickHouse’s query engine into an in-process library that can be dropped into Python, Node, or Go without running a ClickHouse server at all. Each of these projects arrived at a similar conclusion from a different direction: for a large share of real-world analytical work, the dataset fits comfortably on a single machine, and paying for a distributed cluster’s coordination overhead just to run a handful of aggregate queries stopped making sense.
Seen on a timeline, DuckDB sits at the midpoint of a lineage that runs from SQLite’s proof that embedded databases work at all, through CWI’s own earlier columnar research with MonetDB, through Arrow’s standardized in-memory format, and on toward newer entrants like chDB that are still expanding the same idea into new host languages.
Why It Spread So Quickly
Part of the reason DuckDB’s adoption curve looks the way it does is that it never asked teams to change their workflow to get value from it. A data scientist working in a Python notebook can query a pandas or Polars DataFrame directly with SQL, without copying the data anywhere, because DuckDB’s engine can read those in-memory structures natively. A Java service can pull in the JDBC driver and run analytical SQL against a directory of Parquet files sitting in object storage, with no separate warehouse to provision. A Node.js application can do the same thing with the Node bindings, querying CSV or Parquet data directly from disk or from cloud storage buckets. In every case, the pitch is the same: keep the language and the workflow you already have, and drop a fast analytical engine directly into the process instead of standing up infrastructure around it.
That accessibility shows up clearly in the project’s download numbers, which have grown roughly in step with its adoption inside notebooks, data pipelines, and tools like dbt and Hugging Face’s dataset library.
Where Each Engine Fits
None of these tools are direct substitutes for one another, and the differences matter when picking one for a real pipeline.
| Engine | Primary language surface | Best fit |
|---|---|---|
| SQLite | C, with bindings nearly everywhere | Transactional, row-oriented workloads at small scale |
| DuckDB | Python, Java, Node, R, C++, and more | General-purpose embedded analytics, ETL, and notebook work |
| Apache DataFusion | Rust, with Python and Java bindings | Building a custom query engine or data system as a library |
| chDB | Python, Node, Go, Rust | Teams already fluent in ClickHouse SQL who want it without a server |
Signals this trend sends about the future of ETL
- A meaningful share of “big data” pipelines are not actually big enough to need a cluster, and tooling is finally catching up to that reality
- The transform step of ETL is moving closer to where the data already lives, often directly against files in object storage, instead of requiring a load step first
- Local, in-process engines make pipeline development and testing dramatically faster, since there is no shared infrastructure to provision or wait on
- SQL is re-establishing itself as the default interface for data transformation, even inside codebases that are not otherwise SQL-centric
What We Learned
DuckDB did not invent columnar storage or vectorized execution, CWI’s own MonetDB project had already explored both years earlier, and it did not invent the idea of an embedded database either, since SQLite had been proving that model out since 2000. What DuckDB did was combine the two ideas at the right moment, delivering a genuinely fast analytical engine that asks nothing of its environment beyond a language binding.
That combination turned out to matter more than either idea alone, and the fact that DataFusion and chDB have since converged on similar territory from different starting points suggests this was not a one-off success but a real shift in how analytical workloads get built. For ETL pipelines specifically, the signal is hard to miss: a growing share of transform logic no longer needs a warehouse sitting behind it, just a SQL engine sitting inside the same process that is already reading the data.
