HPCC-Platform: ECL, Thor and Roxie as one big data stack

A big data platform from a risk software division

HPCC Systems offers an enterprise ready, open source supercomputing platform to solve big data problems, and it is described as a massive parallel processing computing platform for big data processing and analytics.

The organisation behind it is worth knowing about. HPCC Systems is a technology division of LexisNexis Risk Solutions, which is why the platform comes with an enterprise support story attached to it rather than existing purely as a community project. That is also why the release and support policy in the documentation is written the way a vendor writes it.

The pitch is deliberately comparative, and the comparison is with Hadoop. The platform offers analysis of big data using less code and less nodes for greater efficiencies, and it offers a single programming language, a single platform and a single architecture for efficient processing.

That last point is the whole thesis of the design, and it cuts both ways. An integrated stack that owns its language, its file system, its scheduler and its query engine can be coherent in a way a collection of libraries on top of a general runtime is not. The cost is that adopting it is adopting ECL, and ECL is not a language you can hire for or find a tutorial for over a weekend.

For the record, the project calls itself open source, and the licence file is in the repository.

Thor transforms, Roxie answers

The architecture has two cluster types, and the split is between batch work and online queries.

Thor, the Data Refinery Cluster, is responsible for consuming vast amounts of data and transforming, linking and indexing it. It functions as a distributed file system with parallel processing power spread across the nodes, and a cluster can scale from a single node to thousands of nodes. Its characteristics are listed plainly: a single job at a time, distributed parallel processing, a distributed file system, the ECL programming language, and optimisation for extraction, transformation, loading, sorting, indexing and linking. In other words, Thor is the batch engine.

Roxie, the Query Cluster, provides separate high performance online query processing and data warehouse capabilities. The acronym stands for Rapid Online XML Inquiry Engine, and it is described as the data delivery engine used in HPCC to serve data quickly, able to support many thousands of requests per node per second. Its characteristics differ in one important respect: multiple queries in parallel, and optimisation for concurrent query processing.

An HPCC environment can include only Thor clusters, or both Thor and Roxie. That is a deployment decision with a real cost difference, and it is the kind of thing that is easiest to get wrong at the start, because adding a query cluster later is not a small change to a data flow.

Around them sit the common middleware components, an external communications layer, client interfaces for end user services and system management, and auxiliary components for monitoring and for loading and storing filesystem data from external sources.

ECL compiles to optimised C++, and that shapes everything

ECL, the Enterprise Control Language, is the programming language the platform is built around, and the documentation describes it as ideally suited to the manipulation of big data.

Its listed properties explain the design. It is transparent and implicitly parallel, so a developer writes what they mean and the runtime decides how to parallelise it. It is non-procedural and dataflow oriented, so a program is a flow of data rather than a sequence of instructions. Its syntax is modular, reusable and extensible. It combines data representation and algorithm implementation in one language. It can be extended using C++ libraries. And ECL is compiled into optimised C++.

That last point is the practical headline: the code you write is not interpreted at run time, it is compiled, and the generated C++ is what actually executes. The developer documentation in the repository explains the machinery in several places, and the most useful of them for a new user is the introduction to the code generator, eclcc, alongside the overview of workunits and the different stages in executing a query, and the description of the memory manager used by Roxie and Thor.

The implicit parallelism is the feature that decides the value of the whole proposition. If you can write a transformation once and have it distributed across a thousand nodes, you avoid the distributed-systems work that dominates a hand written map reduce pipeline. The cost is giving up explicit control, and giving up the option of writing in a language your team already knows.

The developer surfaces: ECL IDE, ECLWatch and ESP

Three client surfaces are described, and they cover development, monitoring and access.

ECL IDE is described as a modern IDE used to code, debug and monitor ECL programs. Its features are access to shared source code repositories, a complete development, debugging and testing environment for ECL dataflow programs, built-in access to the ECLWatch tool so developers can watch job graphs as they are executing, and access to current and historical job workunits.

The workunit history is worth a second look, because it is the difference between a batch platform you debug and one you guess at. A workunit is the unit of execution for a query, and the developer documentation has a page on the different stages in executing a query, which is the same thing in more detail.

ESP, the Enterprise Services Platform, provides an easy to use interface to access ECL queries using XML, HTTP, SOAP and REST. The description calls it a standards-based interface to access ECL functions. That is the integration surface for the rest of your estate: a service layer that speaks the protocols an existing application already speaks, rather than a bespoke client library.

So the intended working loop is: write ECL in the IDE, run it, watch the job graph in ECLWatch, inspect the workunits when something goes wrong, and expose the results to other systems through ESP.

A release cadence and support windows you can plan a deployment around

The release and support policy is stated in a way that is unusual for an open source project, and it is the section to read before anything else.

A new version of the HPCC Platform is released every 3 months, and these releases are either major, with breaking changes, or minor, with new features. Maintenance and security releases, the point releases, are typically made weekly, and may occasionally include technical previews.

The support windows are explicit. Maintenance releases are supported for the current and the previous release, while security releases are supported for the current and the previous two releases. The schedule chart in the documentation shows the same pattern running back through the v9 series and into the current v10.8.x line, which is active from 2026-10-06, with v11.0.x expected as a milestone on 2027-01-01.

The release tags confirm the practice. The three most recent releases are community_10.8.4-1, community_10.6.30-1 and community_10.4.50-1, all published on 2026-10-20, so the maintained lines are patched on the same day. The last push to the repository was on 2026-10-22, a couple of days after those releases.

This matters more than a typical open source project because of the major and minor split. A major release carries breaking changes, and with three month releases you have four opportunities a year to be broken by an upgrade. The support windows tell you how long you can defer, and the answer is two maintenance cycles for the current line and three for security.

Building the platform and running its regression tests

The repository is a large C++ tree with a CMake build, and the development documentation explains the parts a contributor needs.

At the top level are CMakeLists.txt, a CMakePresets.json.sample, cmake_modules/ and a vcpkg directory with its own vcpkg-configuration.json, and BUILD_ME.md, which is where the build instructions live. The source tree mirrors the architecture: dali/ for the scheduler, thorlcr/ for the Thor cluster, roxie/ for the query engine, esp/ for the service layer, ecl/ for the language, plus common/, lib2/, rtl/, fs/, charm/, initfiles/, services/, system/, clienttools/, plugins/ and tools/. There are packaging directories as well, with package/, deploy/, deployment/, dockerfiles/ and helm/ for Kubernetes, and a testing/ directory for the regression suite.

The regression test is a single command once you have an installation on the machine:

cd /opt/HPCCSystems/testing/regress
./ecl-test query --target thor nlppp.ecl

That runs a query test against the Thor target, which is the fast way to find out whether a local build is sound.

The documentation site has its own toolchain, which is in the root package.json rather than in the C++ build. The scripts are vitepress for dev-docs, build-docs and preview-docs, a convert script that runs pandoc to turn the reStructuredText documentation into markdown, and a test script that runs node testing/devdoc/check-links.js to verify the links in the developer documentation. The build script chains the documentation build with the resource copying, and the dev dependencies are vitepress, vite, esbuild, rimraf and npm-run-all.

Where the HPCC model stops being the right fit

Three limits follow directly from the design, and a fourth is organisational.

The first is the language commitment. ECL is the interface to every part of the platform, from the IDE to the query engine to ESP, so a team that cannot staff ECL is a team that cannot use the platform. The implicit parallelism and the dataflow model are the reason it is fast, and they are also the reason the code is not portable to anything else. Nothing written in ECL can be moved to a Spark or Flink job without being rewritten.

The second is scale of adoption. The repository is a C++ platform with a compiler, a scheduler, a memory manager, a service layer and a Kubernetes chart. Standing it up is an infrastructure project, and the deployment directories in the tree say so.

The third is the versioning. Major releases every three months with breaking changes is a real operational cost, and while the support windows are published, the window in which a given minor is patched is months rather than years.

The organisational point is the fourth. The project is maintained by a technology division of a company that sells risk software, and the comparison with Hadoop in the README is that division's own claim. A team considering the platform should treat the vendor's efficiency argument as a starting hypothesis and measure it against their own workload, because the difference in code volume and node count is exactly the kind of number that depends on the data.

The alternative shape is the general purpose route: keep your data processing in languages your team already writes, on a runtime with a large ecosystem, and accept more code and more nodes in exchange for a hireable skill set and a portable codebase. The trade is coherence against optionality, and HPCC is firmly on the coherence side.

Editorial conclusion

Adopt HPCC-Platform if you are building a data processing platform on the ECL language, want implicit parallelism and a single runtime for batch and online query, and can commit to the release and support schedule it publishes. Do not adopt it for a codebase written in general purpose languages, because ECL is the interface to everything. Verify first that the current release line is the one you want, that the support window covers your deployment horizon, and that the licence terms in LICENSE.txt and R-LICENSE.txt match what your organisation expects from an open source platform.

Frequently asked questions

Official sources

1. hpcc-systems/HPCC-Platform on GitHub
2. Issues
3. Project website
4. README
5. Releases