Cylon: an MPI-based distributed DataFrame runtime with a C++ core

The gap Cylon fills: relational operators without a JVM or a scheduler

Most distributed DataFrame stacks assume a cluster manager. You install a scheduler, you submit jobs to it, and your Python code is a client that builds a query plan the scheduler executes. Cylon takes the opposite position. The README states that it works with MPI by default for distributing applications, which means the parallelism model is the one your HPC centre already runs. You launch N copies of your program with mpirun, each process loads its own slice of data, and a distributed join happens across those processes. There is no scheduler process to keep alive between jobs.

The intended audience is stated in the README: data and AI/ML engineers who want to invoke data processing operators in a familiar programming language. The Python surface is deliberately Pandas-shaped, so DataFrame construction and merge calls read like the single-machine equivalents. The underlying operators are implemented in system-level C/C++, with Python and Java interfaces on top. That split matters if you have existing C++ services or Java pipelines and want the same operator semantics in both.

How a distributed join actually flows through Cylon

The README gives a two-stage example that exposes the mechanism. The local case builds two DataFrames from Python lists and calls df1.merge(right=df2, on=[0, 1]). No environment object is involved, so this is a single-process operation on Arrow-backed columns.

The distributed case changes three things. First, a CylonEnv is constructed with MPIConfig, which binds the runtime to the MPI communicator. Second, each process generates its own data using env.rank, so rank 0 and rank 1 hold disjoint ranges. Third, the call becomes df1.join(other=df2, on=[0], env=env), passing the environment explicitly. The README notes that if you create n processes, n instances of the program run, each loading two DataFrames into its own memory, and the results are created in the parallel processes as well. That last sentence is the important one: Cylon does not gather results into a driver. Whatever your process prints is its local partition of the output.

Data representation is Apache Arrow in column format, per the README. The shuffle topic on the repository suggests partitioned redistribution is part of the operator set, though the README does not describe the shuffle implementation or its memory behaviour. That is a gap worth noting: for a distributed join, the shuffle is usually where the interesting engineering lives, and the top-level documentation does not cover it.

Installing PyCylon and running your first parallel job

The README gives a Conda path and nothing else for Python users. The command is conda create -n cylon-0.4.0 -c cylondata pycylon python=3.7, followed by conda activate cylon-0.4.0. Note the environment name and the pinned interpreter: the example targets Python 3.7, and the channel is cylondata rather than conda-forge. Two constraints come with this. Cylon only works on Linux systems at the moment, and the Conda binaries need Ubuntu 16.04 or higher. If you are on a RHEL derivative or a macOS laptop, the documented install path does not apply to you.

Running the distributed example uses mpirun -np 2 python <name of your python file>. The README shows only the two-process invocation. Scaling that to a real cluster means your MPI launcher, your hostfile, and your network fabric all become part of the setup, and none of that is covered in the README beyond the single command.

Building from source is deferred entirely. The README says to refer to the documentation for compiling Cylon and links to a Compiling on Linux page. So if you need a build for a non-Ubuntu distribution, or you want to link against a specific Arrow version, the top-level repository does not tell you how. That is a real friction point for anyone whose cluster image is not Ubuntu-based.

The release cadence is the first thing to check before adopting

The listed releases are v0.6.0 in March 2023, 0.5.0 in December 2021, and 0.4.1 in May 2021. The most recent push to the default branch is dated 2026-05-26, so the repository is not archived and development activity on main is more recent than the last tagged release. That combination is worth reading carefully. Code is moving, but tagged versions are not, which means the installable artifact from the Conda channel may lag the branch you would be reading on GitHub.

The README example compounds this. The Conda command creates an environment named cylon-0.4.0 and the badge links point at a Travis CI build for the master branch. Travis CI as a hosted service has changed substantially since that badge was written, so the build status indicator should not be treated as current information about the project's test coverage. None of this is disqualifying, but it does mean you are adopting a project where the documented install version, the tagged release, and the active branch are three different things. Pin explicitly and record which commit or package version you validated.

Where Cylon is the wrong tool

Cylon assumes you have an MPI runtime and are willing to program against it. If your team's workflow is Jupyter notebooks on a managed cluster with a scheduler, introducing mpirun as the entry point is a step backwards in ergonomics. Every job becomes a script you launch, not a notebook cell you run, and debugging means attaching to a rank rather than reading a driver-side stack trace.

The result-locality model is the second constraint. Because results stay in the parallel processes, any step that needs the full result set in one place requires you to bring it back yourself. The README does not describe a collect or gather helper in the example. If your downstream step is a single-process model fit or a small aggregation, you are doing the reduction manually.

The Linux-only, Ubuntu 16.04-or-higher constraint is the third. There is no documented Windows or macOS path, and the source build instructions live off-repository. For a team of five analysts on mixed laptops, Cylon is not the right layer; a single-node Pandas or Arrow-based tool will be faster to adopt and easier to support. Cylon pays off when the data genuinely does not fit on one machine and you already have the cluster.

How Cylon differs from Dask and Ray Data

Dask and Ray Data both separate the client from the workers. You write Python in a driver, the framework serialises a task graph, and a scheduler places tasks on workers. The driver holds the collection handle, and calling compute or materialising a dataset brings results back to the driver. Failure handling, worker lifecycle, and result collection are framework concerns.

Cylon inverts that. There is no driver. Every process is a peer in an MPI communicator, every process holds its own partition, and the CylonEnv object is what carries the communicator into operator calls. The trade-off is explicit: you get the startup characteristics and the failure semantics of MPI, which in most HPC environments means a job either runs to completion or dies and gets resubmitted. You do not get automatic task retry or elastic worker pools. In exchange, you avoid a long-running scheduler service and the serialisation boundary between driver and workers. For a batch job that runs on a fixed allocation and exits, that is a reasonable trade. For an interactive session where workers come and go, it is not.

Licence and the maintenance you are signing up for

Cylon is Apache-2.0, stated in both the README and the repository licence field. Apache-2.0 includes an express patent grant and permits commercial use, modification, and redistribution provided you keep the notices and state changes. For most teams evaluating a data runtime, that is a permissive licence with no copyleft obligation on your own code. This is a description of the licence text, not legal advice; if you are embedding Cylon in a shipped product, have your own counsel confirm the notice requirements.

The maintenance cost is harder to estimate from the supplied material. The gap between the 0.6.0 tag and the current branch means you should decide up front whether you track releases or track main. Tracking main on a project with a C++ core and a Conda-packaged Python binding means you are building from source, and the README points you off-repository for that. Tracking the tagged release means accepting that the documented install example is several versions behind the branch. Neither is wrong, but the choice has to be made before you write production code, because migrating between them later means revalidating every operator you depend on.

Editorial conclusion

Adopt Cylon if you already run MPI jobs on Linux clusters and need distributed joins over Arrow-backed tables without standing up a JVM-based engine. Do not adopt it if you need Windows or macOS support, a managed service, or a release train you can plan around. Before committing, verify two things against your own cluster: that the Conda package resolves for your Python version, and that mpirun -np N python your_script.py behaves correctly when N exceeds the core count of a single node, since the README only demonstrates the two-process case.

Official sources

1. cylondata/cylon on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases