OpenDataPipeline: an AI-assisted EL(T) stack on Airflow, Spark and MongoDB

What OpenDataPipeline actually bundles

The README describes OpenDataPipeline as a platform for moving, transforming and scheduling data with AI assistance. Read the repository layout and the scope becomes narrower and more concrete than that sentence suggests. The top level contains airflow/, spark_server_app/, dlt_server_app/, client/, core/ and opendatapipeline/, which maps onto the architecture section: a React frontend, an Airflow scheduler and webserver, Spark plus dlt plus Pandas for compute, and MongoDB for metadata.

That combination is the product. A team that already runs Airflow and Spark separately does not gain a new execution engine here. What it gains is a pre-wired arrangement of those pieces, a web UI on top, and an LLM step that writes or suggests transformations in conversation. The audience the README names is data engineers and analysts who want to move and transform data without writing extensive boilerplate. The realistic first user is someone standing up a sandbox, not someone replacing a governed production warehouse stack.

The LLM layer is a provider choice, not a model

OpenDataPipeline does not ship a model. It ships a set of environment variables and lets you point the platform at Ollama, OpenAI, Anthropic or Google Gemini. The README lists LLM_PROVIDER with those four values, plus per-provider keys and model names, plus LLM_TEMPERATURE and LLM_MAX_TOKENS.

This is a reasonable design and also the source of most setup friction. The README gives OPENAI_MODEL as gpt-4o and ANTHROPIC_MODEL as claude-3-5-sonnet-20241022, and for Ollama it gives OLLAMA_BASE_URL as http://localhost:11434 with OLLAMA_MODEL as openhermes. Those are examples, not pinned requirements, and the documentation does not state which models were validated. If you pick Ollama, note the address: localhost inside a container is the container itself, so a host-run Ollama will not be reachable at that value without a change the README does not describe. That is a real gap, not a nitpick, because Ollama is the only provider in the list that runs locally and therefore the only one a reader might reasonably expect to work without an API key.

Installing OpenDataPipeline locally with the setup script

The README offers two paths. The managed instance at app.askondata.com is described as the easiest way to start. Self-hosting is Docker-based and the README states the local setup spins up an Airflow webserver, a Spark cluster and MongoDB. Prerequisites are Docker v24 or later and Node.js v18 or later, and the README recommends more than 8GB RAM and more than 15GB free disk space. It says the instructions were tested on WSL2 and should work on any Linux machine, and that the script takes roughly 5 to 20 minutes depending on machine and internet speed.

From the repository root, the documented command is a single script:

bash ./odp_open_source_local_setup.sh

The README advises checking the top of that script for configurable values, and choosing an LLM provider by adding keys and model names at the bottom of docker/.env.example. The example file is described as ready to use if you do not want custom configuration, which in practice means the default provider is whatever the example file sets.

Once the script finishes, the README says to watch container health and then open the app:

watch docker ps

The expected result is all containers reporting healthy, after which the application is reachable at https://localhost. The README does not document what to do when a container stays unhealthy, which ports the individual services expose, or how to tear the stack down, so plan on reading the compose files under docker/ if the first run stalls.

Granting yourself admin, and the credentials that ships with

The admin walkthrough is the most operationally specific part of the README, and it deserves a hard look. It tells you to find the MongoDB primary container, open a shell, connect with mongosh, and set your own role.

docker exec -it <container_name> bash

mongosh --username askondata --port 27021 --authenticationDatabase user_sessions

The README states the password is askondata, and that the update is run against the user_sessions database:

use user_sessions
db.users.updateOne({ email: "your_email@example.com" }, { $set: { role: "admin" } })

Everything here is correct for a local sandbox and wrong for anything exposed. A fixed username and password pair, published in the README, on a MongoDB instance that the setup script also exposes, means the only thing protecting the metadata store is that nobody has routed to it. The README does not discuss changing these credentials, restricting the port, or what the production or Swarm deployment does differently. Treat the local stack as a laptop-only environment until you find evidence to the contrary in docker/.

Where the documentation stops

The README points to a knowledge hub at app.askondata.com/api/v1/doc for architecture and capability details, and the repository carries a directory named odp_code_context/ alongside .askondata/ and .agent/, which suggests in-repository context files intended for AI coding assistants. That is a hint about how the project is developed, not a substitute for operator documentation.

Several things a self-hoster needs are simply absent. No releases are listed for this repository, so there is no versioned artifact to pin and no changelog to read before an upgrade. The README documents a hot-reload trick for backend development, adding '--reload' after 'gunicorn' in opendatapipeline_src/run.py, and it documents log location at $SETUP_PATH/opendatapipeline/logs/askondata. Neither tells you how to upgrade an existing install, how to migrate MongoDB state between versions, or how to roll back. The README does not document rollback. For a tool whose selling point is scheduling and orchestrating other people's data, the absence of an upgrade story is the largest single risk in the repository.

Maintenance signals are mixed. The repository is not archived, and the last push was on 2026-08-04, which is recent enough that the codebase is moving. But with no tags and no releases, "moving" and "safe to depend on" are different claims, and only the first is supported.

OpenDataPipeline against wiring the same tools yourself

The honest alternative is not a single competitor. It is the combination of Apache Airflow, dlt and a notebook or BI layer, assembled by you. The difference is one of defaults versus control. Airflow on its own gives you the scheduler, the DAG model and a mature operator ecosystem, and nothing else: you choose the compute, you choose the metadata store, you write the loading code. OpenDataPipeline makes those choices for you, adds a React UI and an LLM conversation step, and asks you to accept its MongoDB schema, its Spark server app and its dlt server app as the interfaces.

If your transformations are mostly SQL against a warehouse, the Spark cluster in this stack is weight you are carrying for nothing, and a lighter orchestrator plus dbt-style SQL models will be easier to operate. If your problem is genuinely messy sources, REST APIs, database dumps and files that need schema inference, the dlt integration is the part of OpenDataPipeline that earns its place, because that is exactly the problem dlt exists to solve and the README names it as the loading layer. The AI transformation step is the differentiator, and it is also the part with the least documentation: the README says Ask AI uses LLM models via LangChain for interactive conversational transformations, and stops there. There is no description of what the generated transformation looks like, whether it is reviewable as code, or how it is versioned alongside a DAG.

Licence and the cost of staying current

OpenDataPipeline is licensed under Apache-2.0, which permits commercial use, modification and redistribution provided you keep the licence and notice files, and it includes an explicit patent grant. The LICENSE file sits at the repository root. Nothing in the README describes a separate enterprise tier, a closed component or a hosted-only feature, so on the available documentation the open-source and managed offerings appear to be the same software. That is an observation about the repository, not legal advice; if you are embedding this in a product, have counsel read the LICENSE and any third-party notices you find under the dependency directories.

The upgrade cost is the harder question. Because there are no published releases, staying current means tracking main and re-cloning or pulling, then re-running a setup script whose behaviour the README describes as configurable at the top of the file. Any local edits you make to docker/.env.example or to opendatapipeline_src/run.py are yours to reapply. Budget for that as ongoing work rather than a one-time install.

Editorial conclusion

Adopt OpenDataPipeline if you want a single-machine sandbox where Airflow, Spark, MongoDB and an LLM-driven transformation step already talk to each other, and you accept that the repository documents local setup far better than production operation. Do not adopt it if you need a documented upgrade path, a published release cadence, or an out-of-the-box scheduler replacement for an existing Airflow deployment. Before committing, run the setup script on a machine with more than 8GB RAM and 15GB free disk, then verify two things yourself: that the containers reported healthy by docker ps match the services you actually need, and that your chosen LLM provider is reachable from inside the container network, since the README only shows the environment variables and not a connectivity check.

Frequently asked questions

Official sources

1. helicalinsight/open-data-pipeline on GitHub
2. Issues
3. License: Apache-2.0
4. README