DMLC-Core: the record format, the parameter server and the filesystem layer under MXNet

Infrastructure shared across projects, not a learning library

The project describes itself in two sentences that say rather more than the usual library boilerplate. It is a common bricks library for building scalable and portable distributed machine learning, and DMLC-Core is the backbone library to support all DMLC projects, offering the bricks to build efficient and scalable distributed machine learning libraries.

The framing of bricks is the key word. This is not a machine learning framework, it is the layer under several of them. What it provides is the plumbing that a distributed training system needs regardless of what it is training: a compact binary format for storing records, a way to keep parameters synchronised across workers, a way to read and write data from wherever the data happens to live, and a configuration mechanism.

The topics on the project record are a single entry, dmlc, which is consistent with a library whose identity comes from the family rather than from a function.

The primary language is C++, and the licence is Apache-2.0, which is the permissive end of the scale and unusual for a project of this age. Apache-2.0 also carries an explicit patent grant, which for infrastructure code consumed by many downstream projects is a genuinely useful property.

The homepage is dmlc-core.readthedocs.io, and the documentation badge points at it, so the reference documentation is published rather than living in the repository.

The record format, and a portability bug the project documents

The Known Issues section in the README is short and contains exactly one entry, and it is worth reading because it describes a data format limitation rather than a coding wart.

The RecordIO format is not portable across different processor endians. It is not possible to save a RecordIO file on an x86 machine and then load it on a SPARC machine, because x86 is little endian while SPARC is big endian.

That is a real constraint on a library whose entire job is moving training data between machines. A cluster with mixed architectures cannot share a RecordIO file, and the workaround is to write in one format and convert, which means reading the whole file.

The project stating this in the README rather than burying it is a decent signal about how the maintainers treat users, and the fact that it is still there suggests it has not been fixed. Anyone planning to store training data in RecordIO for a period of months should assume the data outlives the architecture it was written on, and that is the case where this bites.

The build system tells you what the library is made of. The object list in the Makefile names the source files: line_split, indexed_recordio_split, recordio_split, input_split_base, io, filesys, local_filesys, data, recordio and config. The prefix of the split files is a hint at the design, since the library appears to build an index over a record stream as it splits, so that a training worker can seek into the data rather than read it sequentially.

The rest of the object list is the filesystem layer. The local backend is always built, and hdfs_filesys, s3_filesys and azure_filesys are added conditionally, each behind its own make variable. So the layer is genuinely pluggable across the three storage systems a distributed training job is likely to see, and the cost of a backend is one object file.

Building it means picking backends and SSE by hand

The Makefile is a plain make build with a config file indirection at the top, so a project can point at its own config.mk instead of the shipped one.

The compiler flags are where the choices show. The base CFLAGS are -O3, -Wall, -Wno-unknown-pragmas and -Iinclude, with -std=c++11 added afterwards, and the linker flags are -pthread and -lm. SSE is enabled by default, with USE_SSE=1 setting the compile flag, so a build for an architecture without it turns that off.

The optional backends are the same one-variable switches. USE_HDFS=1 adds hdfs_filesys.o, USE_S3=1 adds s3_filesys.o, and USE_AZURE=1 adds azure_filesys.o. A deployment that reads from S3 and writes to local disk enables one of them.

There is also a DEPS_PATH mechanism, which adds an include directory and a library directory. That is how a dependent project links against an already built copy of the library rather than rebuilding it.

The README notes that DMLC-Core uses the C++14 standard and that you should ensure your compiler supports it, which is worth flagging next to the -std=c++11 flag in the Makefile. The README also states that the project follows Google's C style guide, that contributors should try to introduce a minimum dependency when possible, and that before submitting code you should type make lint and fix all the style problems, and make doc and fix all the warnings.

Those two commands are the entire contribution gate, and they are the same kind of gate that a shared library needs, since every project that vendors it inherits the same style rules.

The dependency note in the README is minimal, listing libcurl4-openssl-dev under deps, which is a small list for a library with HDFS, S3 and Azure backends, and the README does not enumerate the rest.

The parameter module, and a project that predates most of its users

The What's New section contains a single entry, a note on the parameter module for machine learning, pointing at a page of the documentation.

The parameter module is the part of DMLC-Core that most directly reflects the era it was designed for. A parameter server is a key-value store that workers pull from and push to, and the module is a distributed implementation of one that a training loop can drive. The parameters, the gradient updates and the configuration travel through it rather than through a shared filesystem, which is what makes it work across a network rather than on one machine.

This is a useful thing to understand about the library's position. It predates the parameter server designs that later became common in the deep learning world, and it is a different shape from the library-based approaches that followed. It is not wrong, it is specific, and a project built on it inherits that specificity.

The documentation site is organised into tutorials and a contributing section, and the README points at a documentation and tutorials page. The contributing section also references a feature wishlist, which is a GitHub label, and asks people to open an issue if they want to add something.

The developer channel is a Gitter room at dmlc/dmlc-core, which is a useful detail. Gitter was the standard chat platform for the machine learning world at the time, and a link to it is a reliable timestamp for when this library was at its most active.

The repository structure reflects the same vintage in a good way. There is a src directory with the implementation, an include directory with the public headers, a make directory with the shared make fragments, a cmake directory for the alternative build, a doc directory, a test directory, an example directory, a tracker directory, and a scripts directory. There is a .clang-format, an .editorconfig, a .pre-commit-config.yaml, an appveyor.yml for Windows builds, a CMakeLists.txt alongside the Makefile, and a ruff.toml even though the project is C++, which suggests the configuration files have drifted a little over the years.

Where a shared bricks library is the wrong dependency

Three situations rule it out, and they are all about fit rather than quality.

The first is a new project with no DMLC lineage. Everything in this library was shaped by serving the DMLC family, and the interfaces reflect that. A team starting a distributed training system today has options that did not exist when this was designed, and adopting a bricks library to get a record format and a parameter server is a decision to join a particular ecosystem rather than to use a particular tool.

The second is if you only need one piece of it. If the only thing you want is a compact binary record format, or a filesystem abstraction over S3 and HDFS, or a configuration parser, then taking a whole C++ library with a parameter server, a training-oriented design and a build system with backend switches buys a lot of machinery for one component. The library is not modular in the way a small utility is.

The third is the maintenance question, and it needs care. The repository has a recent push, the last being on 2026-10-19, so the code is being worked on. But the most recent tagged release is v0.3 from 2018-07-17, and a project that has been changing for eight years without cutting a release is telling you something about how it is consumed. In practice that means depending on a commit rather than a version, which is fine inside an organisation that controls its own builds and uncomfortable for a library that others consume.

The practical middle ground is to vendor the parts you need. The Apache-2.0 licence permits it, the source is small enough to read, and the file split in the Makefile suggests the pieces are separable. Taking the record IO code and the filesystem backends and leaving the parameter server behind is a reasonable way to use this codebase in a modern setting.

Editorial conclusion

DMLC-Core fits a team building a distributed machine learning system in C++ on the DMLC model, where the record format and the parameter server are already solved and the effort should go into the model. It is the wrong choice for a new project, because the ecosystem it serves is a specific one and the interfaces are shaped by that history. Before depending on it, read the Known Issues section about RecordIO endianness, because the format is not portable between a little endian and a big endian machine, and check the build flags, since the file systems you need are opt-in and compiled in one at a time.

Frequently asked questions

Official sources

1. dmlc/dmlc-core on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases