DataFlow-Engine splits job control from job execution

The manager side draws the flow, the executor side runs it

DataFlow-Engine is described as a bottom level data driven engine for data integration, data synchronisation, data exchange, data sharing, task configuration and task scheduling. The logical architecture in the README splits it into a management side and an execution side. The management side has a visual interface for user operation, and the execution side is headless and stateless. The management side is mainly responsible for configuring and monitoring the tasks of the execution side, and the execution side receives the tasks, with data passing from an input stream through a data gate and finally out through an output stream. The stated reason for this separation is practical. RISC-V style configuration does not apply here, and the design goal is to handle large scale data tasks, high frequency reporting, high frequency collection and heterogeneous data compatibility through separation of management and execution, multiple stream layers and a plugin library. The plugin library is called an important core of the engine, because which plugins are used at each stage can be combined flexibly to cover many customised and complex business cases.

The module layout is a catalogue of connectors

The source directory listing in the README is effectively the connector catalogue, and it is worth reading for what it implies about scope. The common module holds shared pieces including a beta API, a beta client, a data JDBC module and its commons, and common data tools. The transfer module is the data processing side, with base, core, data, elastic, rdbms, ftp, stream, stream local file, api and assembler modules. The rpc module holds commons, a consumer and a provider. The support module carries the business support pieces, among them a security base, an online support API, a JPA repository, a scheduler, data security and kernel security. Then there is the vue directory for the front end and the webapp directory for the back end projects, split into an executor, a manager and a log service. Each of those is a separate Maven module, so the build is a multi module reactor rather than a single application, and the plugin and transfer modules are where a new integration would be added. The repository root holds a y9-module-dataflow/ directory alongside the vue/ one, and the whole thing is licensed GPL-3.0.

The top of the tree gives the whole picture:

common -- 公共模块
transfer -- 数据处理模块
rpc -- 远程过程调用模块
support -- 业务支撑模块
vue -- 前端工程
webapp -- 后端工程

Deployment assumes containers and a scale out plan

The deployment architecture section answers the sizing question directly. The management side can be deployed in parallel, and the execution side can be deployed at scale according to business category and business volume, which raises data transfer efficiency. For a production environment, the recommendation is to combine the digital foundation platform with the data flow engine and to operate the management side with a system administrator account, for security reasons. The engine supports containerised deployment. The most concrete number in the documentation is a capacity claim, that a single instance of the data flow engine in a domestic IT environment can comfortably manage 5000 tasks on a 4 core 8 GB virtual machine, provided multiple execution nodes are matched appropriately. That qualifier matters, since the manager and the executors have different resource profiles and the 5000 figure is about task management rather than about data throughput. The stated version is v9.6.9, with a badge giving 134.6k lines of code.

A stack chosen for domestic enterprise deployments

The dependency tables are the clearest statement of the deployment target, because they specify exact versions rather than ranges. On the back end, Spring Boot 2.7.10, Spring Data JPA 2.7.10, Spring Data Redis 2.7.10, Spring Cloud 3.1.5, Nacos 2.2.1 for configuration and discovery, Druid 1.2.16 for the connection pool, Jackson 2.13.5, Javers 6.13.0 for auditing, Lombok 1.18.26 and Logback 1.2.11. The front end uses Vue 3.3.2 with Vite2 2.9.13, vue-router 4.0.13, Pinia 2.0.11, axios 0.24.0, TypeScript 4.5.4, core-js 3.20.1, Element Plus 2.2.29, sass 1.58.0, animate.css 4.1.1, vxe-table 4.3.5, echarts 5.3.2, svgo 1.3.2 and lodash 4.17.21. The middleware layer runs on JDK 11 with Tomcat 9.0 or newer. Two things are worth noticing in that list. Nacos means the engine expects a configuration and registry service to already exist, so it is not a self contained application you can start on its own. And the front end carries vxe-table and echarts, which is consistent with a task monitoring interface rather than a general dashboard.

Editorial conclusion

Adopt DataFlow-Engine when you need a self hosted data integration and task scheduling platform with a visual manager, pluggable processing steps and a distributed executor tier, and when your team is comfortable with a Spring Cloud and Vue stack in Chinese enterprise environments. Do not adopt it as a small library, since the deployment story alone assumes Spring Cloud, Nacos and containerisation. Verify first which executor nodes you need and what plugins each stage requires, that the plugin library covers your source and target systems, and that the JDK 11 and Spring Boot 2.7 baseline matches the rest of your services.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. Project website
4. README
5. risesoft-y9/DataFlow-Engine on GitHub