Taier: a distributed scheduler that makes ETL dependencies visible to the people who run them

ETL cost is mostly dependency and operations cost

Taier is a distributed dispatching system focused on the tasks that get submitted and scheduled. The name comes from 太阿, one of the celebrated ancient Chinese swords, which the project documentation explains as the origin of the name.

The stated aim is concrete: reduce the cost of ETL, make the complex dependencies between tasks clear, and reduce the labour cost of submitting, scheduling and operations and maintenance. That framing tells you the audience. It is not a developer tool and not a query engine. It is for the people who run a data platform, the ones who get paged when last night's jobs did not finish, and who spend their time on scheduling and monitoring rather than on SQL.

The platform is presented as one stop for submitting tasks, scheduling them, operating them, and presenting indicators. The list of core features in the documentation is long, and a few entries reveal the priorities: visual configuration for DAG, an IDE development platform aimed at big data users, support for developing your own plugins, multiple task modes including guide mode and script mode, support for dependencies between upstream and downstream tasks, support for batch and stream tasks, integration of various different versions of Hadoop, easy integration of Flink Standalone, tenant and cluster isolation, Kerberos authentication, user defined parameters for tasks, real time monitoring of cluster resources, real time presentation of data indicators, and the ability to restrict a task's resources.

The project is Java, Apache-2.0 licensed, and published by DTStack with documentation at dtstack.github.io/Taier.

The module names describe the architecture

The repository layout is the clearest description of how the system is put together, and it is a set of Maven modules.

`taier-common` holds what the modules share, `taier-dao` the data access layer, `taier-datasource` the datasource and plugin side, `taier-data-develop` the development and authoring environment that backs the visual DAG configuration and the IDE, `taier-scheduler` the scheduling engine, `taier-worker` the side that actually executes tasks, `taier-metrics` the indicator and metrics side, and `taier-ui` the frontend.

The split of scheduler and worker is the one that matters operationally. The scheduler decides what should run and when, and the worker runs it. That separation is what allows a task to be retried on a different machine, and it is also why the deployment needs a coordination service, which the compose file supplies with ZooKeeper.

Around the modules sit the configuration trees that the deployment needs: `conf/` for the platform's own configuration, `flinkconf/` and `sparkconf/` for the engine configuration, `sql/` for database schema, `k8s/` for Kubernetes deployment assets, and a `Dockerfile` plus `docker-compose.yml` for containerised startup. The `website/` directory holds the documentation site, and the presence of `README_zh-CN.md` next to `README.md` reflects that the project is developed in a Chinese context with English documentation alongside it.

The quick start needs MySQL and ZooKeeper before it needs anything else

The documented quick start is two commands:

$ wget https://raw.githubusercontent.com/DTStack/Taier/master/docker-compose.yml
$ docker-compose up -d

What that file starts is worth reading before you run it, because the dependency set is the real requirement here. The compose file defines three services. `taier-db` is MySQL from the `dtopensource/taier-mysql:latest` image, with the database named taier, a root password of 123456 and a time zone of Asia/Shanghai, published on port 3306. `taier-zk` is `zookeeper:3.4.9`, which is the coordinator the scheduler and worker use. And `taier` is the `dtopensource/taier:latest` image, wired to the other two through `ZK_HOST`, `ZK_PORT`, `DB_HOST`, `DB_PORT`, `DB_ROOT` and `DB_PASSWORD` environment variables, and published on port 8090.

So the platform expects a relational database for its metadata and a ZooKeeper ensemble for coordination. That is a real deployment footprint, and it is the first thing to weigh, because a scheduler is not a stateless web app you can run in a container and forget.

The image itself is built from an OpenJDK 8 base image, and the Dockerfile shows the directory layout the platform expects at runtime, with separate `lib`, `datasource-plugins`, `worker-plugins`, `conf`, `flinkconf`, `sparkconf`, `bin` and `logs` directories. The `datasource-plugins` and `worker-plugins` directories are where the plugin model becomes concrete.

The task catalogue is where the engine support shows

The task table in the project documentation is long, and it is the clearest statement of what Taier can actually dispatch.

There are the workflow and data sync tasks, plus data acquisition and DataX, and then a long list of SQL dialects: Spark SQL, Hive SQL, Flink SQL, OceanBase SQL, ClickHouse SQL, Doris SQL, TiDB SQL, MySQL SQL, Vertica SQL, Postgre SQL, SqlServer SQL, Greenplum SQL, MaxCompute SQL and GaussDB SQL. There are also Flink, Shell and Python tasks, and a user-defined task type for anything not on the list.

The user-defined task matters more than its single line in the table suggests. It is the extension point that lets an organisation run a task type the platform does not know about, which is the same reason the feature list mentions supporting the development of your own plugins and a user-defined task IDE for big data users.

Two execution details are worth pulling out of the feature list. The platform supports both batch and stream tasks, so the same scheduler can drive a nightly Hive job and a continuous Flink job. And the different task modes, guide mode and script mode, let the same task type be created through a form or as a script, which is what makes it usable by both an analyst and an engineer.

The dependencies feature ties the catalogue together. Expressing upstream and downstream relationships in the visual DAG is what lets the platform schedule correctly and show a human why a job has not run.

Isolation, tenants, Kerberos and resource limits are the safety story

The feature list contains a cluster of items that are about not breaking things, and for a platform that runs jobs on someone else's Hadoop cluster, they are the ones that decide whether the tool gets deployed.

The documentation claims Taier is completely safe and non-intrusive to the cluster's environment. It provides isolation by tenants and by clusters, and it supports Kerberos authentication, which on a corporate Hadoop estate is not optional but a prerequisite for connecting at all. It can restrict a task's resources, and it integrates various different versions of Hadoop rather than insisting on one, which matters when the estate has a long tail of versions. There is real time monitoring of the cluster's resources, and real time presentation of data indicators, which is the operations side of the platform rather than the development side.

Kerberos and multi-tenant isolation are the two that carry the most configuration weight in a real deployment, and the repository documentation is where you would look for the specifics, since the readme itself stays at the level of feature lists.

One more entry in the list is a scheduling detail rather than a safety one, but it is worth flagging: different versions for tasks. A platform that can run different versions of a task type per job can stage a migration of a job definition without rewriting every job at once.

Where Taier is the wrong tool

The mismatch cases are clear, and they start with scale.

Taier is a platform for a data warehouse or a data lake with scheduled ETL across several engines. If you have ten cron jobs, this is a platform with a database, a ZooKeeper ensemble, a scheduler, a worker fleet and a web console in front of them, and the operational cost of that is not justified. The documentation's own framing, reducing the labour cost of scheduling and operating ETL for a big data platform, assumes a volume of tasks and a diversity of engines that a small team will not have.

The second limit is the coupling to ZooKeeper. The scheduler needs a coordination service, the compose file pins `zookeeper:3.4.9`, and that version is old enough that it is worth checking what your organisation's policy says about running it. The same goes for the compose file's MySQL configuration, which ships with the root password 123456 and publishes 3306 on the host. Those defaults are fine for a laptop demo and wrong for anything else, and the documentation does not present them as a production template.

The third limit is the release rhythm. The tags show v1.5.0 published on 2026-07-23, v1.4.0 on 2023-04-17 and v1.3.0 on 2022-11-02. That is a three year gap between the 1.4 and 1.5 lines, so the jump to 1.5 is a large one and the upgrade notes are likely to matter, while the last push to the repository was on 2026-07-31. A platform that stores your job definitions and dependencies is not something you upgrade casually.

Apache-2.0 terms and where to ask questions

Taier is under the Apache 2.0 licence, with the licence file linked from the project documentation. For a platform that runs on your own Hadoop cluster with your own data, that removes the usual licensing question, and the plugin model means an in house task type is your own code rather than a derivative of theirs.

The community channels are conventional and documented. Questions, bugs and support requests go to the GitHub issue tracker, with a template for choosing the right kind of issue, and there is an FAQ reference page in the documentation. Contributors are asked to read the contributing guide before opening a pull request, and the project has a Slack workspace with an invite link in the readme.

The documentation site carries detail the readme does not, including the per task type pages linked from the task table, the quick start guides for Docker deployment and for developing tasks, the architecture diagram, and the plugin and task expansion pages. For a project whose readme is a feature list, that site is the real manual, and the FAQ page linked from the readme is the right place to start before filing an issue.

There is also a Chinese and an English readme, and the documentation site itself is bilingual, which reflects the origin of the project at DTStack and the fact that much of its user base is Chinese speaking data platform teams.

Editorial conclusion

Taier fits a data platform team that runs scheduled ETL across Hadoop, Hive, Spark, Flink and a dozen SQL dialects, and that wants dependencies drawn rather than implied, with scheduling, operation and indicator display in one place. It does not fit a small team with a handful of cron jobs, because the quick start requires MySQL and ZooKeeper before anything else happens, and it is the wrong choice if you need a task runner you can trigger from a CI job without the web console. Before you deploy it, read the compose file in the repository rather than trusting the defaults, since it ships with a MySQL root password of 123456 and a ZooKeeper 3.4.9 image, and note that the release cadence is uneven, with v1.5.0 published in July 2026 after a long gap since v1.4.0 in April 2023.

Frequently asked questions

Official sources

1. DTStack/Taier on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases