data-warehouse-learning builds the same e-commerce warehouse twice, once on Doris and once on a lake

One scenario, two implementations, deliberately

The project introduces itself as a practical project for real time and offline data warehouses built on an e-commerce system, constructed around the statistical needs of an e-commerce business. The framing is that the two scenarios keep the same data processing logic but use different technical implementations, which is offered as a way of giving a variety of ideas for building a warehouse.

That is a good structure for a tutorial, and it is unusual to see it done properly. Most data warehouse examples show you one stack, and the reader is left to assume the layering is portable. Here the layers, the business metrics and the ingestion shapes stay the same while the storage and compute underneath change, which is the closest a written example gets to showing that the layering is a modelling decision rather than a property of the engine.

The engines on the two sides are Doris on the offline path and Paimon, Hudi and Iceberg on the real time path, with Flink as the streaming engine in both cases. The stated components in the project description are Flink, Paimon, Doris, SeaTunnel, DolphinScheduler, DataArt, Dinky, Hudi and Iceberg.

The audience is a Chinese speaking data engineer or a student who wants a complete, runnable example rather than isolated snippets. The repository is published under the Artistic License 2.0, and it also appears on Gitee as well as GitHub, with a separate project recommended as a development platform.

The project generates its own business data before it analyses anything

The data source module is the first of the four parts of the architecture, and it is unusual in that it produces the data rather than consuming it.

The module is written in Java and generates e-commerce business data, writing it into a MySQL database, while the generated user log data is written into a Kafka message queue. The module supports setting the date for data generation in a configuration file, so you can produce a consistent set of data for a particular point in time.

That design is what makes the rest of the tutorial reproducible. A warehouse tutorial normally has to say, connect to this database, and hope the data is interesting enough to be worth layering. Here the dataset is generated to order, dated to order, and therefore the same every run, which is what allows the offline and the real time paths to be compared against each other and to show the same numbers.

The README shows the generated business data and the generated user logs as screenshots, and the images live under a path in the source tree that follows a Java package naming convention, so the documentation and the code are kept in one place rather than in a separate assets folder.

The trade-off is worth stating. Simulated data lets you focus on the warehouse, and it means nobody has to think about the messy realities of a production source system, including late arriving records, schema changes and backfills. The tutorial is honest about being a teaching environment, and a reader moving from it to a real deployment inherits those questions rather than answers them.

Two ingestion paths feed the ODS layer

The data collection module is where the two paths diverge, and it is the most interesting part of the architecture for a working engineer.

The first path is streaming. FlinkSQL code, developed with Dinky, consumes the user log data from Kafka and writes it into the ODS layer of the online data stores, which are Doris, Paimon, Hudi and Iceberg.

The second path is batch, and it uses a different tool for a good reason. DolphinScheduler is used to configure SeaTunnel tasks to synchronise the business data from MySQL into the ODS layer of Doris. The FlinkSQL and CDC technologies are then used to collect data from Kafka and MySQL and write it respectively into the ODS layer of Paimon, Hudi and Iceberg.

So the offline path reads MySQL through SeaTunnel under DolphinScheduler scheduling, and the real time path reads Kafka through FlinkSQL, with FlinkSQL and CDC also handling MySQL for the lake engines.

That is a real architectural choice rather than a demonstration of two tools. SeaTunnel is used here for the batch load, and running it in front of MySQL under a scheduler such as DolphinScheduler gives you a pipeline that can be retried, scheduled and monitored as a job. FlinkSQL, by contrast, gives you a streaming read of the same database, and it is the natural way to feed a table format that supports incremental consumption.

The cost is that the two paths produce their ODS data differently, and a reader comparing the offline and real time warehouses is also comparing two ingestion semantics. That is not a flaw in the tutorial, it is the reason the tutorial exists, but it does mean the comparison is not a controlled experiment on the warehouse layers alone.

Four layers, applied identically on both storage engines

The data warehouse module follows the industry standard four level architecture of ODS, the operational data store, through DWD and DIM, the detail and dimension layers, to DWS, the data service layer, and then ADS, the application data store. The data flows through the layers in Doris, Paimon, Hudi and Iceberg through both batch and real time scheduling.

The offline section of the README walks the Doris path step by step. Data collection into the Doris ODS layer, then DorisSQL development for the DIM layer, then for DWD, then for DWS, then for ADS, each with a screenshot of the processing logic and a screenshot of the resulting tables. The final two steps are task orchestration, showing an overview of the tasks and then the orchestration itself, and data presentation.

This is the part of the tutorial that generalises. The layers are a naming convention with meaning, ODS for what arrived, DWD and DIM for the modelled detail and dimensions, DWS for the aggregated service layer, ADS for the shaped result a report reads, and the tutorial shows the same sequence of statements in each one, so the reader is looking at the transformation rather than at the engine's syntax.

The limitation is that the README presents this through screenshots rather than through the SQL itself. The DWD, DWS and ADS logic is described as DorisSQL development and then shown as an image, so a reader who wants to copy the transformation has to look at the source tree for the code. For a tutorial aimed at people who will actually build this, the screenshots are a summary and the code is the substance, and that ordering is worth noticing before you plan an afternoon around it.

The version table is the actual specification

The third section of the README is a table of software and versions with the installation package name and the corresponding dependency jars, and it is the most operationally valuable part of the document.

The stack runs from Zookeeper 3.9.1 and Kafka 3.6.1 through SeaTunnel 2.3.3 with the seatunnel-hadoop3-uber jar, DolphinScheduler 3.2.0 with the MySQL connector, Doris 2.0.4, and Flink 1.18.1 with a specific set of connectors, the MySQL CDC connector, the Kafka connector, the Hive connector and the JDBC connector. The lake formats follow: Iceberg 1.5.2 with the Flink runtime, Hudi 0.15.0 as the Flink 1.18 bundle, and Paimon 0.8 with its own Flink jar and a shaded Hadoop jar. Dinky 1.0.0 is pinned to a 1.18 release candidate tarball, and underneath sit Hadoop 3.1.3, Hive 3.1.3 with the Paimon, Iceberg and Hudi Hive connectors, Maven 3.9.6, CentOS 8 and OpenJDK 8.

The platform choice is the most consequential line and the easiest to miss. The project is specified to run on CentOS 8 with OpenJDK 8, and Flink 1.18, Hadoop 3.1.3 and Doris 2.0.4 are all from an earlier generation than the versions most teams now run. A reader following the tutorial today will be installing a Java 8 and Hadoop 3.1.3 stack, and that is a deliberate choice for reproducing the tutorial rather than a recommendation for a new deployment.

The Dinky line has the same character, since a 1.0.0 release candidate is a specific artefact rather than a stable version, and it is the sort of dependency that has to be fetched from a named file rather than resolved by a version range.

A Maven project of code and images, mirrored on two forges

The repository itself is small in structure. The top level holds LICENSE, README.md, pom.xml and src/, and the project is a Maven project whose primary language is Java, consistent with the data source module being written in Java and the rest being FlinkSQL, DorisSQL and scheduler configuration.

The documentation images live inside the source tree rather than in a separate assets directory, under a path that mirrors a Java package, and the README links to them directly with that path. It is a convention that keeps the images next to the code they describe, at the cost of making the src/ directory carry both.

The distribution is unusual and worth noting. The README links a GitHub address under a different account name, a Gitee address, and a separate project recommended as a development platform. For a Chinese open source project that is a practical arrangement rather than a quirk, since Gitee is where domestic users tend to look, and the alternative hosting means the tutorial is reachable from inside networks where GitHub is slow.

The licence is the Artistic License 2.0, which is a permissive licence with a naming clause, and the README opens with a request to star the project, which is a common convention in this ecosystem and is a fair signal that the project is a teaching resource maintained for visibility rather than for a release cadence.

Who should follow this, and what it will not tell you

The clear case for this material is a developer who wants one complete example rather than fragments. You get a generated dataset, two ingestion paths, four modelling layers, orchestration and a dashboard, and the components are named with versions so the whole thing can be assembled. For someone who has read about ODS and DWS in isolation and wants to see them applied to a single business scenario end to end, that is a real gap filled.

The limits are equally clear. The stack is pinned to a Java 8, Hadoop 3.1.3, Flink 1.18 and CentOS 8 world, so the tutorial is a historical configuration as much as a current one, and the artefacts have to be assembled by hand from the table. The README shows the layer logic as screenshots rather than publishing the SQL, so the code has to be read from the tree. And the visualisation story is inconsistent: the project description names DataArt among the components, while the data presentation section describes using SuperSet and DataRT for reports and dashboards, so a reader deciding which tool to install has to work out which is current.

There is also a structural asymmetry. The visible sections of the README walk the offline Doris build in detail, layer by layer, while the real time path on Paimon, Hudi and Iceberg is described in the architecture and the ingestion module, so the two halves of the comparison are documented with different thoroughness.

For an adopter, the sensible move is to treat the version table as the spec, the screenshots as the map, and the source tree as the reference, and to plan for the lake engines to be the part you will have to read the code for.

Editorial conclusion

data-warehouse-learning suits a Chinese speaking developer who wants one end to end example of a real time and an offline warehouse built on the same business scenario, and who will be running it on the specific stack the version table names. It does not suit a team that needs current versions, because the stack is pinned to OpenJDK 8, Hadoop 3.1.3, Flink 1.18.1 and CentOS 8, and it is not a reference architecture to copy into production. Verify first that the pinned versions still match the clusters you have, that the four engines are the ones you want to compare, and that the visualisation tooling in the body, SuperSet and DataRT, is the tooling you intend to use rather than the DataArt named in the project description.

Frequently asked questions

Official sources

1. Issues
2. License: Artistic-2.0
3. MrSuiChuan/data-warehouse-learning on GitHub
4. README