Weld: a lazy IR layer that fuses analytics libraries, pinned to LLVM 6

The problem Weld targets is data movement between libraries, not slow kernels

The README frames the problem precisely: analytics workflows combine functions from different libraries and frameworks, and while each function can be fast in isolation, the combined workflow can land an order of magnitude below hardware limits because data moves across function boundaries. That is a different complaint from the usual one about a slow sort or a slow join. Weld's answer is to stop treating each library call as a finished computation. Instead the core computations are expressed in a common intermediate representation, and the whole workflow is optimized as one unit. The audience is therefore narrow and technical. You are not the analyst writing a notebook. You are the person who maintains the library, or the runtime team that wants several libraries to plan together. The repository topics list pandas and machine-learning alongside llvm and code-generation, and the Python bindings plus the Grizzly subset of Pandas show who the project expected to reach first. If your workload is a single library call, Weld has nothing to fuse and the indirection buys you nothing.

Lazy IR, then LLVM: how a Weld program actually executes

The mechanism has three visible stages. First, a computation is built lazily: the README says Weld lazily builds up a computation for the entire workflow. Nothing is evaluated while the expression is being assembled, which is what makes cross-library optimization possible at all, because the optimizer sees the whole graph rather than one call at a time. Second, that graph is written in Weld IR, whose syntax is specified in docs/language.md. This is the contract between Weld and any library that wants to participate. Third, the IR is optimized and then compiled, and the build output tells you what the compilation target is: cargo build --release produces two dynamically linked libraries, libweld and libweldrt, with .so extensions on Linux and .dylib on macOS. LLVM is a listed repository topic and an LLVM/Clang++ 6.0 install is a build prerequisite, so the code generation path runs through LLVM. The runtime library is separate from the compiler library, which matters if you are embedding Weld rather than calling it from Python. The low-level C API for interfacing with Weld is documented in docs/api.md, and docs/tutorial.md walks through building a small vector library, which is the honest place to start if you want to know what the IR looks like in practice rather than in theory.

Building Weld requires LLVM 6.0 and a WELD_HOME export

The build sequence in the README is short. Install the latest stable Rust via rustup, install LLVM/Clang++ 6.0, clone the repository, export WELD_HOME to the checkout, and build: git clone https://www.github.com/weld-project/weld, then cd weld/, then export WELD_HOME=`pwd`, then cargo build --release. On macOS the LLVM step is brew install llvm@6 followed by a symlink so that llvm-config resolves to the keg-only formula: ln -sf `brew --prefix llvm@6`/bin/llvm-config /usr/local/bin/llvm-config. On Ubuntu 16.04 the README adds the apt.llvm.org repository for llvm-toolchain-xenial-6.0 and installs llvm-6.0-dev and clang-6.0, then symlinks /usr/bin/llvm-config-6.0 to /usr/local/bin/llvm-config. Ubuntu 14.04 needs an extra gcc backport for libstdc++, and both Ubuntu paths need zlib1g-dev. Verification is a single command: llvm-config --version should print 6.0.x. WELD_HOME is not only a build variable. The README states that Grizzly needs it set because Grizzly has to find its own native library through that variable, so a deployment that forgets it will fail at import or first use rather than at link time. Tests run with cargo test, and a substring filter is supported: cargo test <substring to match in test name>.

The LLVM 6 pin is the constraint that decides most adoptions

The README asks for LLVM/Clang++ 6.0 and the macOS instructions install llvm@6 specifically. The Ubuntu instructions also install 6.0, though the verification note there says llvm-config --version should report 6.0.x or newer, which is a looser statement than the macOS section and the two do not quite agree. Treat the pin as real until you have proven otherwise on your own machine. The practical consequence is that Weld does not slot into a modern Rust build that already links a current LLVM, and on a distribution that ships LLVM 14 or 17 you will be maintaining a parallel toolchain or a container for it. The release history reinforces the point: the newest tagged release listed is v0.4.0 from February 2020, with v0.3.1 and v0.3.0 both from August 2019. The repository has been pushed more recently than that, so the master branch is not frozen, but there is no tagged release that reflects whatever has landed since early 2020. Building from master means building an untagged state, and the README's own build instructions are the only supported path. This is the wrong tool if you need a versioned artifact you can pin in a package manifest and upgrade on a schedule.

Grizzly and the Python bindings show where Weld expects to be used

Grizzly is described as a subset of Pandas integrated with Weld, with its own directory at python/grizzly and example workloads under examples/python/grizzly. That is a deliberate scope limit: not all of Pandas, a subset, which is the honest way to ship this kind of integration because every unsupported operation is a place where the lazy computation has to be materialized and the fusion benefit disappears. The generic Python bindings live in python/ with examples in examples/python. If you are evaluating Weld from a data science seat rather than a compiler seat, Grizzly is the surface you would actually touch, and the subset boundary is the first thing to map against your existing code. If you are evaluating it as infrastructure, docs/api.md and the two shared libraries are your surface. Those are two different products sharing one runtime, and the README treats them as such by splitting the documentation along that line.

How Weld differs from a query engine like Spark

A query engine such as Spark takes a declarative program, plans it, and executes the plan inside its own runtime and its own data abstractions. Weld does not ask you to move your data into its world. It asks library authors to emit a common intermediate representation so that computations from different libraries can be optimized together, and it compiles that representation through LLVM into native code. The difference in approach shows up in who does the work. With a query engine, the engine owns the operators and you write against them. With Weld, you own the operators and Weld owns the optimization and code generation across them. That is why the tutorial is about building a vector library rather than about writing a query. It is also why Weld is a poor fit for a team that just wants to run SQL. There is no catalog, no storage layer, no scheduler in what the README describes. It is a compiler and a runtime, and the surrounding system is yours to build.

Licence, maintenance and what to check before you commit

Weld is BSD-3-Clause. That is a permissive licence, and the usual obligations attach: keep the copyright notice and the licence text with redistributed source or binaries, and do not use the project's name to endorse derived work. Since Weld ships as libweld and libweldrt, a commercial product that links those libraries is redistributing them, so the notice needs to travel with the artifact. This is a description of the licence text, not legal advice; have counsel read it against your distribution model. On maintenance, the material supports one clear statement and no more: the latest tagged release is v0.4.0 from February 2020, while the default branch has been pushed since. There is no published upgrade path, no changelog in the supplied material, and no compatibility statement between the two shared libraries and the Python bindings, so an upgrade means rebuilding from source and re-running cargo test against your own integration. Budget for that as ongoing work, not a one-time setup. The first things to verify are concrete: that llvm-config --version reports 6.0.x on your build host, that WELD_HOME is exported wherever Grizzly or the native library is loaded, that cargo test passes on the commit you intend to use, and that the Weld IR your library emits is the IR the optimizer is actually seeing.

Editorial conclusion

Adopt Weld if you are building or extending a library that can emit Weld IR and you control the LLVM 6 toolchain, since that is the only way to get the cross-library fusion the project is built around. Do not adopt it if your pipeline is already expressed in SQL or DataFrame operators that a query engine can plan, or if you cannot pin LLVM 6.0 and set WELD_HOME. Before committing, build from source, confirm llvm-config --version reports 6.0.x, run cargo test, and check that the Weld IR emitted by your own code is what you expect.

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. weld-project/weld on GitHub