Project Maya: serving GLM-5.3-Flash and DeepSeek-V4.1-Flash from a local GPU

What Project Maya runs, and on which hardware

Project Maya is a local inference stack for two very large mixture-of-experts models. The GLM-5.3-Flash model from Z.ai has 321 B parameters with about 18 B active per token and a context of up to 1 M tokens. The DeepSeek-V4.1-Flash-1017 model from DeepSeek has 748 B parameters. Both upstream models are MIT licensed, and the project itself is MIT licensed with a separate file for the DeepSeek terms.

The hardware claim is the headline: one NVIDIA GPU or up to sixteen, AMD cards as well, Linux and Windows. The repository description puts it as running the models on your own GPUs, from one card to sixteen, and the README adds that AMD support and Windows are experimental. The topic list on the repository gives the technical range, from cuda and rocm to mixture-of-experts, local-llm, quantization and self-hosted.

The project is led by Peasant Smith, with @mw00 as the account behind it and @needmorevram as co-maintainer, and the model artifacts are published on Hugging Face under the peasantsmith account rather than in the repository itself. That split matters for anyone planning an air-gapped install: the code is in GitHub and the weights are a separate download of tens to hundreds of gigabytes.

Releases move quickly. The three most recent tags are v1.1.17, v1.1.18 and v1.1.19, all on 2026-10-20, and the last push was on the same day, so the tree is being changed and rebuilt several times a day at the moment.

The expert offloading engine: GPU, then RAM, then NVMe

The mechanism is the interesting part of the project. Rather than requiring a machine whose VRAM can hold the whole model, Maya keeps the experts a conversation uses most on the GPU, the next ones in RAM, and the remainder on the NVMe SSD, moving them as the chat continues. For the DeepSeek weights the README puts it more directly: the most used experts go to the GPUs and the others are read in place from the model files, from RAM where they fit and from the SSD otherwise.

Two consequences follow. More VRAM and RAM makes the system faster, and the README asks for the model files to sit on an NVMe SSD. The second is the reason the project can claim to run a 748 B parameter model on a workstation: the weights are not loaded in full, they are paged in as the routing decisions of the mixture-of-experts layer select them.

The engine also measures the machine. The README says it measures GPUs, CPU, RAM and SSD and tunes itself to them, and the setup asks which quant to download and recommends one for the PC it is running on. A separate line in the setup helper checks the page file, which is the Windows equivalent of the memory overcommit setting that such a system depends on.

The privacy angle follows from the design. The README states that nothing leaves the machine, which is the point for anyone running a large model next to proprietary code.

The GLM quants and what the accuracy figures claim

Maya publishes its own quantisations of GLM-5.3-Flash, made from the Z.ai FP8 release with statistics taken from the FP8 model, and each keeps the model's MTP block, which drafts tokens ahead of the main pass. The variants differ in size and in the memory pool they are aimed at:

Maya-S at 96.5 GB is the default and targets PCs with a smaller pool of RAM and VRAM together. Maya-S24 at 94.7 GB is for cards of 24 GB or less, where 4-bit attention leaves roughly 1.5 GB more of the card for experts and the README reports a 14 percent decode improvement. Maya-M at 116 GB uses more bits where they count and is calibrated toward tool calls and front-end code. Maya-M-Derisked at 116 GB applies a weight edit by Blackfrost_AI that reduces blanket refusals, and the README marks it experimental and not measured. Maya-L at 156.3 GB targets the biggest memory pools.

The accuracy figures come with each variant, and they are the project's own measurements: Maya-S keeps 97.9 percent of the FP8 model's zero-shot accuracy and picks the same next token 83 percent of the time, Maya-S24 keeps 97.7 percent, Maya-M keeps 97.9 percent with 86 percent token agreement, and Maya-L keeps 99.2 percent with 90 percent agreement. The overall range quoted for the GLM quants is 97.7 to 99.2 percent of zero-shot accuracy. The method and the numbers behind each variant live in the bench/results files, one per quant, and the weights are on Hugging Face.

For the speed claim, the README quotes up to 124 tokens per second on two H100s, 118 on four RTX 4090s, 81 on a single RTX PRO 6000 and 30 or more on one RTX 5090, and it attributes these to real users rather than to a lab run. Treat them as reported figures for those specific machines.

DeepSeek on its own engine, with a drafter and a vision tower

DeepSeek-V4.1-Flash-1017 runs on a separate engine, maya-ds4, and the setup reaches it in one of two ways. You can run the setup and pick it, or you can name the model directly:

./setup.sh --setup
./setup.sh --model DeepSeek-V4.1-Flash-1017

The server for that engine uses the pinned official prompt encoder, with numeric reasoning budgets and ID-matched parallel tool results, and the details are written up in the prompt format notes under docs.

The file table is where the storage requirement becomes clear. Maya's own quant of DeepSeek is 340 GB including the Engram tables at Q6_K, and it runs on NVIDIA cards from Volta, a V100, onwards as well as on AMD, on Linux and Windows. The unsloth UD-Q8_K_XL and UD-Q4_K_XL GGUFs are larger, at 508 GB and 501 GB, the latter keeping dense weights at Q8_0 to use less VRAM. The EXL3 builds from LibertAIDAI come in at 414 GB for 2.9 bits per weight and 461 GB for 3.5, and the 3.5 build is selected with a flag and needs roughly 280 GB of VRAM and RAM combined, on an RTX 30 or an A100 and newer.

--ds4-quant exl3-3.5

Two optional pieces are downloaded alongside the weights. The DSpark drafter is the model's own drafting model, 8 GB from the project repository, and the setup measures whether drafting actually helps on the machine and keeps it only where it does. The vision tower is 1 GB of the DeepSeek checkpoint, and it runs on the first GPU in about 320 MB; a flag leaves it out:

--no-vision

Setup: a pinned llama.cpp, a PC check and a text interface

The installer is a Python program with a text interface. The setup helper carries the pieces: the PC checks for GPUs, CPU, RAM and the page file, the Python packages that have to be present, the source of llama.cpp at a pinned commit, resumable downloads, the engine source hash, the shard check and the console messages. The comment in the source credits the helpers to Strata's setup under the MIT licence.

The pinned llama.cpp commit and the archive URL it derives are hardcoded, which is what makes a build reproducible:

LLAMA_CPP_COMMIT = "3cf03257f219afbe7334045ff7c6a06ac68c627d"
LLAMA_CPP_ZIP = f"https://github.com/ggml-org/llama.cpp/archive/{LLAMA_CPP_COMMIT}.zip"

The Python packages the installer expects are listed in the same file:

PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja", "pillow", "psutil", "jsonschema>=4.18,<5"]

That list explains the shape of the build: cmake and ninja are needed because the engine is C++ built with CMake, and the rest are the serving and chat layer. The repository root holds the entry points, with setup.sh and maya.py for the installer, maya.sh as the shell wrapper, START-MAYA.bat for Windows, and chat.py for the terminal client. The engine itself is in src/ and include/ with its own CMakeLists.txt, and the serving layer is under serve/.

Windows is handled as a first-class path in the code rather than an afterthought, with a check on the operating system name deciding which behaviour applies.

The API surface and the in-browser interface

Maya exposes an OpenAI-compatible and an Anthropic-compatible API, with tool calls, so existing coding agents can point at it without a custom client. The README states the purpose plainly: to plug into your tools, ready for coding agents. The chat interface and a live monitor run in the browser, alongside picture input, thinking levels and what the README calls a one-command setup.

Concurrency is a stated limit rather than an open-ended promise: up to eight requests are answered together, aimed at agents that run several sessions or several people on one PC. The DeepSeek engine adds ID-matched parallel tool results, so tool calls that return together are matched back to the right call, and numeric reasoning budgets let the caller bound the thinking effort.

That API compatibility is the part to be careful about. A compatible endpoint is not the same as the reference implementation, and the quantisation itself changes the answers, since the DeepSeek quant picks the same next token as the original 92.6 percent of the time and the GLM quants sit between 83 and 90 percent on the same measure. An agent that depends on exact output will see differences, and the DeepSeek path documents its own prompt format constraints in a separate file, including the limits of the standalone Jinja templates.

For scale, the model files are the constraint rather than the server. A 340 GB DeepSeek quant or a 156 GB GLM quant on a single machine is a storage and I/O problem, and the README's guidance to keep the files on an NVMe SSD is the practical statement of that.

Where Maya is the wrong tool, and what the licence allows

The honest limitations are visible in the numbers rather than hidden in a section. The smallest GLM quant is 94.7 GB and the smallest DeepSeek quant listed is 340 GB, so this is a workstation-scale project. A laptop with 32 GB of RAM and a 1 TB drive is out of scope, and a machine with a slow SATA disk rather than NVMe will be dominated by paging the experts in.

The platform support is uneven. AMD support and Windows are described as experimental in the README, the DeepSeek path needs an NVIDIA card from Volta upwards or an AMD card, and the EXL3 builds are limited to an RTX 30 or an A100 and newer. Single-tenant use on one PC is the design target, with eight concurrent requests as the ceiling, which is well below what a shared inference server or a cluster would give you.

There is also a question of trust in the measurements. Every accuracy and speed figure in this project comes from the project itself, with the method written up per quant in the bench/results files, and the Derisked variant is explicitly not measured yet. Nobody outside the project has audited these numbers, so treat them as a starting point for your own evaluation rather than as settled fact.

On licensing, the repository is MIT, the upstream GLM and DeepSeek models are MIT as well, and a separate LICENSE.deepseek-v41 file covers the DeepSeek terms, which is the file to read before you redistribute a build that ships the DeepSeek quant. The commercial question the project raises is a licensing one: an MIT-licensed tool and MIT-licensed weights leave room for use that a GPL engine would not.

Editorial conclusion

Adopt Project Maya if you need a 321B or 748B mixture-of-experts model served locally for coding agents, you have an NVIDIA card from Volta upwards or an AMD card, and you can store 95 to 340 GB of weights on a fast NVMe drive. Do not adopt it for a laptop without that disk space, for a multi-tenant server, or if you need a model smaller than roughly 95 GB. Before starting, run the setup on the target machine so its own hardware check reports the recommended quant, and read the bench results under bench/results for the variant you pick, because the accuracy figures for each quant come from those files rather than from any independent review.

Frequently asked questions

Official sources

1. License: MIT
2. mw00/project-maya on GitHub
3. Project website
4. README
5. Releases