Apache DolphinScheduler: a low-code orchestrator for complex task dependencies

The dependency problem DolphinScheduler was built to solve

Cron schedules a command at a time. It does not know that a downstream aggregation must not start until three upstream loads have finished, and it does not know what to do when one of them fails at 02:00 and someone needs to re-run only that branch. Data pipeline teams end up encoding those relationships in shell wrappers, sentinel files and a wiki page. DolphinScheduler's stated purpose is to handle complex task dependencies in data pipelines, and the README frames the product around that rather than around simple timers.

The audience follows from the feature list. It is aimed at teams running workflows across multiple clouds and data centers, with projects and data sources that need permission boundaries, and with enough daily volume that horizontal scaling of workers matters. The README claims capacity for tens of millions of tasks per day; that is the project's own claim, not something this article measured. If your workload is a dozen nightly jobs, the machinery below is more than you need.

Masters, workers, registry: how the pieces fit together

The repository layout shows the architecture more clearly than the README prose does. There is a dolphinscheduler-master module and a separate dolphinscheduler-worker module, plus dolphinscheduler-api for the HTTP surface and dolphinscheduler-ui for the front end. The README describes this as a decentralized, multi-master and multi-worker architecture with native horizontal scaling, which means scheduling decisions and task execution are not pinned to one process.

Around that core sit several extension points that are separate Maven modules: dolphinscheduler-registry, dolphinscheduler-datasource-plugin, dolphinscheduler-storage-plugin, dolphinscheduler-task-plugin and dolphinscheduler-scheduler-plugin. The registry module is what lets multiple masters coordinate; the plugin modules are where task types and external systems are integrated. There is also dolphinscheduler-dao-plugin, which is worth noticing because it implies the persistence layer is not hard-wired to a single database.

A workflow definition becomes a DAG in the database, the master side decides when each node is eligible, and workers execute the task. Task state flows back so the UI can show workflow instances and task instances. The README also mentions workflow versioning for both definitions and instances, including tasks, which means a running instance keeps the definition it started with rather than silently picking up an edit.

Installing DolphinScheduler with Docker and creating a first workflow

The README points to four deployment modes: Standalone, Cluster, Docker and Kubernetes, with a separate Terraform path under deploy/terraform. The quickest way to see the product is the Docker route, which the documentation covers under the start guide. The exact compose file and image tags live in the docs site rather than in this repository's README, so check the version selector before copying anything.

What you end up with is a UI on the API server's port, a default admin account documented in the start guide, and a project to create. Inside a project you create a workflow definition, drag tasks onto the canvas, and connect them. The README's screenshots show the workflow definition canvas and a tree view that flattens the same DAG into a list of dependencies.

If you prefer to drive it from code, the README lists a Python SDK and an Open API alongside the Web UI. The Python SDK documentation is a separate site linked from the README, and the Open API is exposed by dolphinscheduler-api. The same README lists backfill as a Web UI native feature, so re-running a historical date range does not require a script.

For a cluster install, the configuration lives under config/ at the repository root. That directory is where the master, worker, API and alert settings are supplied, and it is the first place to look when a service starts but does not register.

Where DolphinScheduler is the wrong choice

The honest limitation is conceptual rather than technical. DolphinScheduler is a platform you operate. It has a master role, a worker role, an API service, a registry and a database, and the README's own deployment list (Standalone, Cluster, Docker, Kubernetes, Terraform) is a reminder that production use means running several processes. A team that wants one binary and a crontab will find this heavy.

The second limitation is that workflow definitions live primarily in the UI and the database. That is the point of a low-code tool, and it is a genuine trade-off: DAGs authored by drag and drop are easy for analysts to build and hard to review in a pull request. If your organisation requires pipeline changes to go through code review with diffs, the Web UI workflow is a mismatch, even though the Python SDK and Open API give you programmatic paths in.

The third is version drift in the documentation. The README's QuickStart links point at a 3.3.0-alpha documentation path while the most recent release listed is 3.4.3, published on 2026-09-06. That is a documentation lag, not a defect, but it means you should confirm that the install page you are reading matches the release you intend to run. Finally, the README's performance claim about being several times faster than other orchestration platforms is unaccompanied by a linked benchmark, so treat it as marketing until you reproduce it on your own workload.

DolphinScheduler versus Airflow and XXL-Job

Airflow and DolphinScheduler solve overlapping problems from opposite directions. Airflow pipelines are Python code, so they live in Git, get reviewed like any other change, and are tested with the same tooling as the rest of your codebase. DolphinScheduler puts the DAG in a visual editor backed by a database, which lowers the barrier for people who do not write Python and makes ad hoc changes faster. If your team is comfortable with Python and wants pipelines versioned alongside application code, Airflow's model is the better fit. If your team includes analysts who need to build and adjust DAGs without a deployment, DolphinScheduler's model is the better fit.

XXL-Job is a different comparison because it is a distributed task scheduler rather than a DAG platform. It schedules jobs and routes them to executors; it does not put a dependency graph at the centre of the product. If your problem is "run these jobs on a schedule across many machines with retries and logging", XXL-Job is smaller and simpler. The moment you need fan-in, fan-out, conditional branches and per-branch backfill, the graph becomes the product and DolphinScheduler's architecture is the reason to pick it.

The README's topic list also places the project next to Azkaban and inside the CNCF cloud native landscape, which is a fair summary of where it sits: a general orchestration platform, not a data transformation engine. Transformations still happen in the systems the tasks call.

Maintenance, releases and what Apache-2.0 means here

The project is not archived, and the last push to the default dev branch was on 2026-09-21. The release cadence visible in the release list is steady: 3.4.1 on 2026-03-01, 3.4.2 on 2026-05-30 and 3.4.3 on 2026-09-06. Three releases in roughly six months on a 3.4 line suggests patch maintenance is active, and the presence of dolphinscheduler-api-test and dolphinscheduler-e2e modules indicates the project invests in automated verification rather than relying on manual checks.

Upgrade cost is the practical question, and the repository structure gives a clue about where it lands. Because the task types, data sources, storage backends and registry are all plugin modules, an upgrade can require plugin compatibility checks in addition to the core service upgrade. The README does not document a rollback procedure, and no upgrade guide appears in the repository's top-level entries, so plan to read the release notes for the specific version you move to before touching a production cluster.

The licence is Apache-2.0, which is a permissive licence with an explicit patent grant and no copyleft obligation on your own code. That matters if you embed the scheduler in a commercial product. It does not remove the obligation to comply with the NOTICE file and attribution requirements, and it says nothing about the licences of the external systems your tasks connect to. This is not legal advice; read the LICENSE and NOTICE files in the repository and get your own counsel for anything commercial.

Editorial conclusion

Adopt DolphinScheduler if you have data pipeline DAGs with real dependencies, need multi-tenant project isolation, and want a Web UI plus a Python SDK and Open API rather than cron and shell scripts. Do not adopt it if you only need a handful of timer jobs, because XXL-Job covers that with far less machinery, or if your team wants pipelines defined as reviewed code in a repository, where Airflow's Python DAGs fit better. Verify three things first: that the release you pick matches the documentation version you read, that your database and registry choices are on the supported list for that release, and that the task types your pipeline depends on exist in dolphinscheduler-task-plugin at that tag.

Frequently asked questions

Official sources

1. apache/dolphinscheduler on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases