Apache InLong: five modules for ingestion, convergence, caching, sorting and management

One framework doing ingestion, synchronization and subscription at once

The pitch in the first paragraph is breadth: a one stop, full scenario integration framework for massive data that supports Data Ingestion, Data Synchronization and Data Subscription, with automatic, secure and reliable transmission. What makes that more than three labels is the statement that batch and stream processing are supported at the same time. A reporting platform that has to land a daily batch and also keep a live feed current usually ends up with two systems, and this is the claim that they do not have to. The README positions the project for environments that need to build a data reporting platform quickly, for ultra large scale reporting, and for cases where reported data must be sorted and landed automatically. The name comes from a creature in Chinese mythology said to guide the river into the sea.

The five modules are the original Tencent pipeline, still in the tree

InLong was built at Tencent to support a reporting service the README describes as more than 80 trillion data pieces per day, and it names the five modules that platform integrated: Ingestion, Convergence, Caching, Sorting and Management. The repository layout keeps that vocabulary almost literally. inlong-agent and inlong-dataproxy sit at the ingestion end, inlong-tubemq is the messaging component carrying data through, inlong-sort and inlong-sort-standalone handle the sorting stage, inlong-manager is the control plane, and inlong-dashboard is the web interface. Around them sit inlong-common for shared code, inlong-sdk for client integrations, inlong-audit for auditing, inlong-tools for utilities and inlong-distribution for packaging. Reading those directory names tells you more about the architecture than any diagram in the README.

Three entry points, and the README documents none of them in commands

There is no install snippet on this page. The README's deployment section is truncated before it reaches any, so nothing here invents one. What it does list is the shape of a first use, and it names three ways in: integrate InLong and manage data streams through an SDK, use the InLong command line tool to view and create data streams, or work through the InLong dashboard. If you want the actual procedure, the project documentation at inlong.apache.org is the place the README sends you, including pages for the manager SDK example, the command line tools and the dashboard. The repository root also carries a docker/ directory and a bin/ directory, which tells you a container image and shell tooling exist, without saying how either is meant to be used.

Topics are the interface between producers and consumers

The features section describes a SaaS based service platform where users report, transfer and distribute data by publishing and subscribing to topics. That word choice carries the architecture. Data producers declare a topic, consumers subscribe to it, and the pipeline in between is InLong's problem rather than the application's. For a team whose internal teams each want their own dashboard from the same event stream, that indirection is the whole value. It is also the source of the main operational risk: once downstream consumers depend on a topic, changing the schema behind it becomes a coordination problem rather than a code change, and the README's mention of unified monitoring and alerting at the data metric platform is aimed at exactly that.

Sorting and landing is the part that is hard to replace

Of the five modules, Sorting is the one with no easy substitute in an existing stack. The features list credits real time extract, transform and load based on rules, plus sorting capabilities, and the Scalability item describes a pluggable architecture where you plug modules in based on specific protocols and can replace components. That is a fair description of what an ETL layer does, but InLong's version is coupled to its own ingestion and messaging path, and the presence of both inlong-sort and inlong-sort-standalone suggests you can run the sorting stage on its own. The README does not document the standalone mode, so treat it as a possibility to check rather than a supported topology.

The performance numbers are production claims, not published benchmarks

The Stability and Reliability item says the project is derived from the actual online production environment and delivers high performance processing for 10 trillion level data streams and highly reliable services for 100 billion level data streams. Read those as scale claims about a deployment at one company, not as a benchmark you can reproduce. The 80 trillion pieces per day figure appears in the origin story, again attributed to Tencent's online business. No configuration is given, no hardware, no methodology, and no comparison baseline. If your workload is in the billions rather than the trillions, the interesting question is not whether InLong can reach those numbers but how much of the platform you have to run to reach the billions.

Against composing a message queue and a batch ETL yourself

The alternative is the one most teams already have: a managed message queue for the streaming side, a batch scheduler for daily landing, and whatever ETL tool the data team prefers in between, glued together with code you own. The difference in approach is ownership and coupling. Your own composition keeps each part replaceable, and it lets a modest deployment avoid running a manager, a dashboard and a standalone sorter. InLong's composition is a single pipeline with topic semantics across ingestion, transport, sorting and subscription, which removes the glue and gives you cross stage metrics and alerting that you would otherwise assemble. If the reporting platform is already large and the team is asking for one product to run it, that trade is usually worth making. If you are standing up a first pipeline, it is overkill.

Editorial conclusion

InLong fits a team that already runs a large reporting platform and needs ingestion, buffering, sorting and subscription handled by one stack rather than assembled from a message queue and batch jobs, and its five module boundaries make that scope legible. Skip it for a handful of pipelines, because the operational surface is a Maven multi module project with a manager, a dashboard and a standalone sorter, not a single service. Before evaluating, read the project documentation at inlong.apache.org for the deployment paths the README does not cover, and validate the scale claims independently, since the README lists them without publishing a benchmark methodology. The last push was on 2026-09-16, the newest tag is 2.4.0-RC0 from 2026-08-21, and the repository is not archived.

Frequently asked questions

Official sources

1. apache/inlong on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases