# chatllm.cpp: a pure C++ chat runtime with continuous conversation

> chatllm.cpp is a C++ implementation of local LLM inference built on a forked ggml, covering models from under 1B to over 300B with streaming output, RAG, tool calling and an OpenAI compatible API. It suits people who want to run models on their own machine and who need conversation that does not stop at the context window.

**foldl/chatllm.cpp** — Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

- Repository: https://github.com/foldl/chatllm.cpp
- Stars: 935 · Forks: 76
- Language: C++
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/foldl-chatllm-cpp

## A C++ runtime on a forked ggml rather than a submodule

chatllm.cpp is described as a pure C++ implementation of several models for real-time chatting on your own computer, on CPU and GPU, and it is built on ggml. The repository has a ggml directory at the top level, and the change log records that on 2024-05-29 ggml was forked instead of used as a submodule. That detail decides how you clone the project. Because the tree is vendored rather than referenced, the recursive clone flag matters, and a plain clone leaves the dependency tree incomplete. The scope the README describes is wide: models from less than 1B parameters to more than 300B, multimodal chat with RAG, running locally. The feature list adds accelerated memory-efficient CPU and GPU inference with int4 and int8 quantization, an optimized KV cache and parallel computing, and an object oriented approach to the similarities between Transformer based models. The README claims accuracy at or above other implementations and links to a series of issue threads as evidence, which is a claim worth checking against your own evaluation rather than taking as settled.

## Continuous chatting through Restart and Shift

The feature that distinguishes this runtime from a plain prompt tool is continuous chatting, described as having virtually unlimited content length. The README says two methods are available, Restart and Shift, and points at the --extending options. The same idea appears in the change log under continued generation, generation steering and layer shuffling, all of which live in the fun documentation, along with a reversed role mode from December 2024. The motivation is practical. A standard chat template fills a context window and then either drops the oldest turns or errors out, and for assistants that need to recall something said twenty exchanges ago, losing the thread is worse than truncating a document. Layer shuffling and generation steering are the more experimental corners of the same feature set, since they intervene in how the model produces tokens rather than in how history is fed back. The README documents these as optional behaviours, which is the right framing: they change output in ways that need evaluating before they meet a user.

## The model format is not GGUF, and that is the main friction

chatllm.cpp expects models converted into its own quantized GGML format, and the README states plainly that the generated .bin files have a different format from the GGUF used by llama.cpp. Only the HF format is supported, with a few exceptions, so the starting point is a Hugging Face model directory that you convert yourself. There is a second format to know about: GGMM, added in March 2025 as a file format of its own. Since a quantized model also changes on disk, the README documents re-quantization at load time, for example with --re-quantize q4_k, so you can move between quantization levels without re-running the conversion step. The practical consequence for anyone arriving from the llama.cpp ecosystem is that the model files are not interchangeable. You keep a separate set of converted artifacts, and a change in quantization scheme means a new set of binaries rather than a flag on an existing file. That is a real maintenance cost, offset by the fact that chatllm.cpp can also serve models at sizes where GGUF-based workflows become awkward.

## Converting and quantizing with convert.py

The conversion path is a Python script that runs outside the C++ runtime. Its dependencies are installed from a requirements file:

```bash
pip install -r requirements.txt
```

That file lists safetensors, torch, transformers at 4.34.0 or later, and sentencepiece. So the toolchain you avoid at inference time still appears in the preparation step, which matters if your goal was a dependency-free runtime. The basic conversion takes an fp16 base model to a q8_0 quantized int8 GGML model:

```bash
python convert.py -i path/to/model -t q8_0 -o quantized.bin --name ModelName
```

The README notes that models such as ChatLLM2-6B, InternLM, LLaMA, LLaMA-2 and Baichuan-2 need no further argument, while some models such as CodeLLaMA require the model type to be given with -a, and the per model value is listed in docs/models.md. The name flag expects an English model name, with an optional --native_name for the name in another language. A LoRA adapter can be merged during conversion, which saves a separate merge step:

```bash
python convert.py -i path/to/model -l path/to/lora/model -o quantized.bin --name ModelName
```

## Per tensor quantization with the -tt option

Quantization is not limited to one type for the whole model. The -tt option takes a regex pattern matched against tensor names and a type, and it can be repeated. The README gives the example where all tensors use q4_k except the embeddings and the output head, which use q8_0:

```bash
... -t q4_k -tt lm_head q8_0 -tt model.embed_tokens.weight q8_0
```

The reasoning behind that pattern is worth understanding before copying it. The embedding table and the output head are the tensors whose degradation shows up first, because they are large and because errors there affect every token rather than one layer. Leaving them at a higher precision while the bulk of the network is compressed at 4 bit is the compromise this pattern encodes. The same mechanism explains why the runtime lists int4 and int8 quantization as supported and why re-quantization at load time exists: once a model is converted, the balance between size and fidelity can still be adjusted without a full reconversion.

## Bindings, a web demo and an OpenAI compatible API

chatllm.cpp is usable as a library rather than only as a program. The README lists Python, JavaScript, C and Nim bindings, a web demo, and more possibilities, and the change log records an OpenAI compatible API under binding documentation from May 2024, alongside support for CodeGemma Base and Instruct. Distributed inference has its own documentation page from February 2025, and GPU acceleration another from the same month. Those two pages together describe a runtime designed for a machine with more than one device or more than one node. The quick start is deliberately minimal, with the README describing it as as simple as:

```bash
main_nim -i -m :model_id
```

The -i flag starts the interactive session and -m selects the model by identifier, and the full walkthrough sits in docs/quick_start.md, which also covers downloading quantized models on demand. The repository root carries the usual pieces for a project like this: CMakeLists.txt, cmake, src, models, bindings, scripts, docs and the ggml fork, plus the build.yml workflow that produces the CI badge in the README header.

## What the README does not settle

The documentation is extensive but uneven, and a few gaps matter for an adoption decision. The README does not publish benchmark numbers of its own, so any throughput or latency figure attributed to this project elsewhere is not traceable here. It also does not explain the licensing status of the individual model architectures it supports, only its own MIT licence, and the model list in docs/models.md is a long one, so per model caveats are a documentation task rather than something the README settles for you. The change log shows an unusually high model turnover, with entries dated through October 2026 including Intern-Decision, Spark-2.5, TeleOCR, MiniCPM5-2B and Instella-MoE, which is a sign of broad compatibility work and also of a moving target: a project that tracks dozens of model families at once will change behaviour as those models change. Multimodal support is split across a dedicated page with image, audio, speech and drawing entries, and the RAG, tool calling, lens and GGMM topics each have their own document, so the README is an index rather than a manual.

## Conclusion

chatllm.cpp fits an engineer who wants to ship or script a local model runtime without a Python toolchain in the serving path, who needs the conversation to continue past a single context window, and who is willing to accept a model format that differs from the GGUF ecosystem. It does not fit a team that wants to reuse existing GGUF files directly, since chatllm.cpp expects its own converted output, nor a team that needs a maintained upstream where a fork matters. Verify first that your model converts with convert.py and that the resulting.bin loads, then check the --extending behaviour of Restart and Shift against your expected turn semantics. The licence field on the repository says MIT, release v25 was published on 2026-10-18, and the last push was on 2026-10-21.

## FAQ

### What models does chatllm.cpp support?

The README describes inference for models from less than 1B to more than 300B parameters, with a supported models document at docs/models.md listing the individual architectures and the arguments convert.py needs for each. Multimodal chat, RAG and LoRA are also supported.

### Can chatllm.cpp load GGUF model files from llama.cpp?

No. The README states that the .bin files generated by convert.py have a different format from the GGUF format used by llama.cpp, and that only the HF format is supported for input, with a few exceptions. Models therefore have to be converted for chatllm.cpp separately.

### How do I build chatllm.cpp from source?

Clone with the recursive flag, because the ggml dependency is forked into the tree rather than referenced as a submodule. If you forgot the flag, run git submodule update --init --recursive inside the checkout. The repository root carries a CMakeLists.txt and a cmake directory for the build.

### What does the --extending option do in chatllm.cpp?

It selects how the runtime continues a conversation past the context limit. The README describes two methods, Restart and Shift, under the continuous chatting feature, where content length is described as virtually unlimited.

### Is there an OpenAI compatible API in chatllm.cpp?

Yes. The change log records an OpenAI compatible API under the binding documentation, and the feature list also mentions Python, JavaScript, C and Nim bindings along with a web demo.

## Sources

- [foldl/chatllm.cpp on GitHub](https://github.com/foldl/chatllm.cpp)
- [Issues](https://github.com/foldl/chatllm.cpp/issues)
- [License: MIT](https://github.com/foldl/chatllm.cpp/blob/master/LICENSE)
- [README](https://github.com/foldl/chatllm.cpp/blob/master/README.md)
- [Releases](https://github.com/foldl/chatllm.cpp/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/foldl-chatllm-cpp