Duckle: A Self-Hosted ETL Platform That Compiles Your Pipeline Graph to DuckDB SQL

The Problem: ETL Without Metered Pricing or Vendor Lock-In

Duckle targets teams that are tired of hosted ETL platforms that charge per row and keep your pipeline definitions in a proprietary cloud. The README explicitly positions it as 'a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte.' The core promise is that you author a pipeline once, on a laptop, and deploy the same file to your own server or cloud account. That file is a plain text artifact in git, so it outlives whoever wrote it. This is a direct answer to the common pain point where pipeline logic is trapped in a SaaS UI. The intended user is a data engineer or a small team that wants visual authoring, but also wants to audit every step and avoid surprise bills.

How It Works: Compiling a Graph to DuckDB SQL

The architecture is visible from the README: you drag sources, transforms, validators, and sinks onto a canvas, wire them together, and press Run. Duckle compiles that graph to SQL and executes it on DuckDB, a columnar, vectorized engine. Every node has a live preview tab and shows the generated SQL, so there is no hidden state. The README states that it 'compiles to SQL on DuckDB and uses every core you give the box,' which explains the performance claim of 96 million rows out of Postgres to Parquet in 39.9 seconds. That number is from the project's own marketing, not an independent benchmark, but it illustrates the design intent: a bigger instance is a faster pipeline. The engine is not a separate service; it is embedded in a single binary that downloads DuckDB on first launch. This is a fundamentally different approach from tools that run a separate orchestrator and compute cluster.

Getting Started: Commands, Config, and the Canvas

The quickstart is designed to be fast: download the binary, run it, and you are on the canvas. The README mentions a command `duckle-runner serve` that runs a pipeline headless on a schedule, in Docker, or on a box you own. That command is the deployment path. There is no explicit install command in the provided material, but the README describes a 60-second quickstart and a 'Build from source' section for Rust developers. Pipelines, connections, contexts, and routines persist as plain files in a folder you choose. That is the key configuration: you pick a workspace folder, and everything is versionable. For automation, there are API keys for machines, and a Python API is mentioned in the repository description. The AI assistant, Duckie, can write the pipeline JSON from an English description and drop it onto the canvas. By default it runs in-process, but you can point it at your own OpenAI-compatible endpoint.

The Local AI Assistant: A Unique Trade-Off

Duckle ships with an AI assistant called Duckie that runs in-process by default. The README emphasizes that no API key, no telemetry, and no vendor round-trip are required. Your prompts and data stay inside your infrastructure. This is a genuine differentiator: most ETL tools with AI features send your schema and queries to a third-party API. The trade-off is that running a model in-process consumes local CPU and memory, which could slow down pipeline execution on the same box. The README acknowledges this by offering an opt-in to use your own OpenAI-compatible endpoint instead. That gives you a choice between privacy and performance. The material does not specify which model runs in-process, nor its quality, so you should test it on your own pipeline descriptions before trusting it for complex transformations.

Component Breadth: 360+ Connectors, But Verify Coverage

The README claims 360+ components and 160+ connectors, covering files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, and SaaS REST/GraphQL APIs, even FTP and IMAP. That is a wide net, and the project says each is covered by tests. But the material does not list the full catalog. If you need a specific connector, you must check the capabilities matrix in the documentation. The breadth is a selling point, but it also means that some connectors may be shallow wrappers. The README is honest that Duckle is 'single-machine and embedded by design' and 'not to replace a distributed warehouse.' That is a critical constraint: if your source is a massive cluster or your sink is a distributed system, Duckle's local execution model may not be the right fit. The 96-million-row claim is for a single machine, so scale expectations accordingly.

Limitations and Failure Modes

The most obvious limitation is the single-machine design. Duckle runs on one box, so it cannot horizontally scale across nodes. If your pipeline needs to process terabytes across a cluster, this is the wrong tool. The README says so directly. Another limitation is the AI assistant: it is a convenience, but the generated JSON may not match your exact schema or business logic. You must review the SQL on every node, which the tool makes easy, but that is still a manual step. A third limitation is that the binary is 73 to 110 MB, and DuckDB downloads on first launch. That means the first run requires network access and disk space, which could be a problem in air-gapped environments. The material does not mention offline installation, so you should assume you need internet for the initial setup.

Alternatives: Fivetran, Airbyte, and dbt

The README names Fivetran and Airbyte as the hosted alternatives. Fivetran is a managed SaaS that handles connectors for you, but it charges per row and keeps your pipelines in their cloud. Airbyte is open-source but typically runs as a separate orchestrator with a containerized architecture, and it does not compile to a single SQL engine. Duckle's approach is different: it embeds the compute engine (DuckDB) and compiles the graph to SQL, so there is no separate compute cluster. For dbt users, Duckle can run dbt on DuckDB inside the same tool, which is a notable integration. The difference in approach is that Duckle is a single binary with an embedded engine, whereas Airbyte is a set of services and Fivetran is fully managed. If you already have a dbt workflow, Duckle may be a way to unify ingest and transform, but if you need a multi-node orchestrator, you should stick with Airbyte or a cloud warehouse.

Maintenance, License, and Upgrade Path

Duckle is licensed under Apache-2.0, which means you can use, modify, and distribute it freely, with no copyleft obligations. That is a permissive license, so you can embed it in commercial products without releasing your own code. The project is in active development: the latest release is v0.7.0 from August 2026, with v0.6.1 and v0.6.0 in the same month. That frequency suggests a fast-moving project, but also means you should expect breaking changes between minor versions. The README mentions a roadmap and contributing docs, so there is a community process. Maintenance cost is low because pipelines are plain files, but you must track new releases and update your binary. The binary size grows with each version, and you need to re-download DuckDB if the engine version changes. There is no mention of a migration tool, so verify your pipelines after each upgrade.

Editorial conclusion

Adopt Duckle if you are a small team or individual data engineer who wants a visual ETL tool that runs on your own hardware, with no per-row pricing and full control over pipeline files in git. Avoid it if you need distributed, multi-node processing or a fully managed cloud service. Before adopting, verify that your required connectors are in the 360+ components list, test the AI assistant's output against your own data, and confirm that DuckDB's single-machine execution model meets your throughput needs. Duckle is honest about its scope: it is built for local and small-team data work, not to replace a distributed warehouse.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes