mukel/llama3.java: Llama 3 inference in a single Java file

The problem llama3.java solves, and who it is built for

Most ways to run a Llama 3 model pull in a C++ toolchain, a Python environment or a serving framework. Llama3.java takes the other route: the README describes it as Llama 3, 3.1 and 3.2 inference implemented in a single Java file. The repository confirms that shape, since the top level holds only .gitignore, LICENSE, Llama3.java, Makefile and README.md. There is no module tree to walk through.

The README is explicit that education is not the only goal. It says the project will be used to test and tune compiler optimizations on the JVM, particularly for the Graal compiler. That explains the feature list: the Vector API kernels, GraalVM Native Image support, and AOT model pre-loading are all things you would build if you wanted a small, legible workload to point a compiler at. If you want a production inference server, this is not the pitch. If you want to see how Grouped-Query Attention, RoPE scaling and quantized matrix-vector products look in Java, one file is a short read.

How the single-file pipeline works: GGUF in, tokens out

The data flow starts at the GGUF container. The README lists a GGUF format parser as a feature, and the model file supplies both the tokenizer vocabulary and the weights. Tokenization uses a Llama 3+ tokenizer based on minbpe. From there the model runs Llama 3+ inference with Grouped-Query Attention, and the README notes two version-specific adjustments: ad-hoc RoPE scaling for Llama 3.1 and tied word embeddings for Llama 3.2. Those are the kind of details that break a naive port, so their presence in the feature list is a signal about which checkpoints were actually exercised.

Weights are read in F16, BF16 and F32, plus the Q4_0, Q4_1, Q4_K, Q5_K, Q6_K and Q8_0 quantizations. The compute path uses Java's Vector API (the README links JEP 469) for the matrix-vector multiplication routines, which is where nearly all the time goes in single-stream decoding. Two CLI modes are offered, --chat and --instruct. The Makefile shows the packaging: sources compile into target/classes under package com.llama4j, with com.llama4j.Llama3 as the main class, and the jar target bundles the classes plus the LICENSE file into llama3.jar. Everything the runtime needs is in that one artifact.

Installing llama3.java and running a first prompt

Java 21 or newer is required. The README ties that floor to the MemorySegment mmap feature used to map the model file. The repository ships no Maven or Gradle build, so the two supported entry points are jbang and the Makefile. With jbang installed, the README's first command prints the CLI help:

jbang Llama3.java --help

You should see the option list for the CLI, including the --chat and --instruct modes. The file is also executable directly after a chmod, again through jbang:

chmod +x Llama3.java
./Llama3.java --help

Before any of that you need weights. The README points at GGUF repositories for Llama 3.2 1B and 3B Instruct, Meta Llama 3.1 8B Instruct and Meta Llama 3 8B Instruct, and it recommends the pure Q4_0 files, with Q8_0 as an optional second choice. A chat session then looks like the native-image example in the README:

./llama3 --model Llama-3.2-1B-Instruct-Q8_0.gguf --chat

If you would rather stay on the JVM, make jar produces llama3.jar and the README gives the run command, which must carry the preview and vector module flags:

java --enable-preview --add-modules jdk.incubator.vector -jar llama3.jar --help

For a one-shot generation instead of a chat loop, the benchmark section shows the relevant flags together: --model, --max-tokens, --seed, --stream and --prompt. Note that --stream takes a boolean value in that example (--stream false), not a bare switch.

Quantization purity is the failure mode to watch

The README spends real space on a problem that is easy to miss until output quality drops. It states that Q8_0 quantizations in the wild are fine, but Q4_0 ones are rarely pure: the token_embd.weights and output.weights tensors are often quantized with Q6_K instead of Q4_0. A file like that is not broken in any obvious way, and a loader that assumes uniformity may read it wrongly or reject it. The README's answer is to generate a pure file yourself from a high precision source with llama.cpp's llama-quantize utility:

./llama-quantize --pure ./Meta-Llama-3-8B-Instruct-BF16.gguf ./Meta-Llama-3-8B-Instruct-Q4_0.gguf Q4_0

That means the cheapest path, downloading a Q4_0 file and running it, carries a check the README does not automate. There is no validation command listed, so the practical move is to start from the project's own GGUF repositories or from a --pure requantization.

The second boundary is scope. This is a single-stream inference program. There is no server mode, no request batching and no multi-model scheduling in the feature list, and the CLI exposes one model at a time. If you need an HTTP endpoint with concurrent clients, llama3.java is the wrong layer.

Native Image, AOT preloading and the JVM flags that matter

The Makefile encodes the build in a way that is worth reading before you run it. Compilation uses --enable-preview, --add-modules jdk.incubator.vector and -source set to the detected major version. The native target invokes native-image with -H:+UnlockExperimentalVMOptions, -H:+VectorAPISupport, -H:+ForeignAPISupport, -O3 and -march=native. That last flag means the produced binary is tuned for the machine that built it, which is fine locally and a portability trap if you copy the executable elsewhere.

AOT model preloading is the more distinctive feature. Setting PRELOAD_GGUF to a model path before make native bakes that particular model into the binary:

PRELOAD_GGUF=/path/to/model.gguf make native

The README describes the result as a larger binary with no parsing overhead for that model, and says it can still run other models at the usual parsing cost. The trade is binary size against time-to-first-token.

On the JVM side, the preferred vector size is used by default and can be forced with the system property -Dllama.VectorBitSize=0|128|256|512, where 0 disables vectorization. That property is the first thing to try if you are comparing against a JIT that handles the Vector API differently. GraalVM 25 or newer is recommended in the README for best performance, with the caveat that its Vector API support is partial.

How llama3.java differs from llama.cpp and other Java runners

The honest comparison is with llama.cpp, and the README makes it directly. The benchmark section shows a vanilla llama.cpp build (llama-cli version 3862) measured with llama-bench on Llama-3.2-1B-Instruct-Q4_0.gguf, and the equivalent llama3.java invocation measured on the same 2019 AMD Ryzen 3950X, pinned to a single CCD with taskset -c 0-15 because inference is memory-bandwidth bound. The results are presented as a chart, and the README does not state a numeric summary in text, so treat the shape of the comparison as the project's own claim rather than a settled number.

The architectural difference matters more than the ranking. llama.cpp is a C/C++ library with a large backend matrix (CPU SIMD paths, GPU offload, a server, quantization tooling). Llama3.java is one Java file with no dependencies, running on the JVM or as a GraalVM native image, and it borrows llama.cpp's llama-quantize when a pure quantization is needed. You give up GPU offload, batching and the serving stack; you get a program you can read end to end, modify, and use as a compiler benchmark. For Java teams already invested in GraalVM, that is the interesting axis. For throughput-oriented deployments, it is not a substitute.

Maintenance, licence and what upgrading costs

The repository is not archived, and the last push was on 2026-04-24. No releases were retrieved, so there is no tagged version to pin; you are tracking the main branch and the single Llama3.java file it contains. That has a direct consequence for upgrades: there is no changelog to diff between versions, so the practical record of what changed is the file history itself.

The build surface is small but not zero. Java 21 or newer is required, the Vector API is an incubator module, and the JVM run command carries --enable-preview. Both the incubator module and preview flags are, by their nature, things that move between JDK releases, so a JDK upgrade is the realistic trigger for a build break. The Makefile derives the source level from the running java binary, which reduces one class of mismatch but does not remove the flags.

The licence is MIT, stated in the README and present as a LICENSE file that the jar target bundles into llama3.jar. MIT is permissive, but the model weights are a separate matter: the README links to Meta's Llama model cards and to Hugging Face repositories, and those files carry their own terms. Nothing here is legal advice; check the model licence for your use case independently of the code licence.

Editorial conclusion

Adopt llama3.java if you want to read or modify an entire Llama 3 inference stack in one file, or if you are tuning JVM and Graal compilation for vector workloads. Do not adopt it as a general serving layer: it has one CLI, no batching and no server mode, and its performance claims come from a single 16-core Ryzen 3950X benchmark. Before committing, confirm that your chosen GGUF file is a pure Q4_0 quantization or a Q8_0 one, since the README notes that Q4_0 files in the wild often carry Q6_K tensors for token_embd.weights and output.weights.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. mukel/llama3.java on GitHub
4. README