evilsocket/cake: a Rust inference server that shards models across phones and desktops
Distributed inference for mobile, desktop and server.
At a glance
- What is it?
- Cake is a multimodal inference server written in Rust that can run a model on one machine or split its transformer blocks across a cluster of iOS, Android, macOS, Linux and Windows devices. The clustering is the interesting part, and the licence is the part to read before you ship anything.
- Who is it for?
- Adopt cake if you have spare heterogeneous hardware, a non-commercial or internal research use, and you want to see how far layer sharding across a phone and a laptop actually gets. Do not adopt it for a product: the FAIR licence requires a signed commercial agreement for any use by or on behalf of a business entity, and the README itself calls the code experimental and changing quickly.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Activity is slowing. The repository last received commits 6 months ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- evilsocket/cake3,141 ★ · +15
| Repository | 2026-09-25 | 2026-11-15 |
|---|---|---|
| evilsocket/cake | 3.1k | 3.1k |
15 Stars in the last 51 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The problem cake targets: models that do not fit on one GPU
Most local inference stacks assume one machine with one accelerator. If the model does not fit, you quantise it, or you buy a bigger card, or you rent one. Cake takes a third route: it splits the model itself across whatever devices you already own. The README frames this as using planned obsolescence to make AI more accessible, which is a fair description of the intent. The target user is someone with a drawer of old phones, a laptop with a discrete GPU and a desktop with another one, who wants to run a workload that none of those machines could hold alone.
The scope is wider than text. The project describes itself as multimodal, covering text generation, image generation with Stable Diffusion and FLUX, and voice synthesis with VibeVoice TTS including voice cloning. Fifteen text model families, six image model variants and two TTS models are listed. That breadth is unusual for a project at this stage, and it also means the surface area to keep working is large.
The honest framing matters here. The README carries a warning in bold that this is experimental code being actively developed and changed very quickly. The last push to the repository was on 2026-04-24, so the project has been quiet for roughly five months. Treat the feature list as a description of intent, not as a support contract.
How the cluster actually works: mDNS discovery, layer assignment, zstd weight streaming
The mechanism is pipeline-style layer sharding. A master node holds the model. Workers start with no model data at all. When a worker joins, the master discovers it over mDNS with zero configuration, or reads a manual topology file if you prefer to be explicit. The master then assigns transformer blocks to each device, proportionally to that device's VRAM and compute, and pushes only the weight shards that device needs.
The transfer is compressed with zstd and verified with CRC32 checksums. Workers cache what they receive locally, so a second run against the same model does not repeat the download. For a cluster where one node is a phone on wifi, that caching is the difference between usable and not.
The manual topology path is where the real detail lives. The docker-compose file documents a topology format that maps layer ranges to named workers, with entries like model.layers.0-15 for one worker and model.layers.16-31 for another, each with a host and port and a description. That means you can decide the split yourself rather than trusting the automatic assignment, which is what you want when the devices are genuinely different.
The workspace is split into cake-core, cake-cli and cake-mobile, with a separate cake-mobile-app directory holding the Kotlin Multiplatform app. The Makefile shows how the mobile side is built: a static library for iOS with the metal feature, and a shared library for Android arm64 built through cargo ndk with UniFFI generating the Kotlin bindings. The mobile worker is a first-class part of the design, not an afterthought.
Building cake and running your first local model
Cake is built from source with Cargo, and the feature flag selects the backend. The README gives one line per platform. Choose the one that matches your hardware, because building with the wrong backend gives you a binary that cannot use your GPU.
cargo build --release --features cuda # Linux (NVIDIA)
cargo build --release --features metal # macOS (Apple Silicon GPU)
cargo build --release --features vulkan # Linux (AMD/Intel/Steam Deck)
cargo build --release # CPU only (portable)Models come from HuggingFace through the cake pull command and land in the standard HuggingFace cache at ~/.cache/huggingface/hub/. That detail is worth noticing: the cache is shared with transformers and huggingface-cli, so a model you already downloaded for another tool is already there. The README notes that models are also fetched automatically on first use if they are not cached.
cake pull evilsocket/Qwen3-0.6B
cake list
cake run evilsocket/Qwen3-0.6B "Explain quantum computing in simple terms"After the pull, cake list should show the cached model, and cake run should print generated text. The architecture is auto-detected from the model's config.json, so you do not pass an architecture flag for text models.
To get an API and a web UI instead of a one-shot answer, use cake serve. The README describes an OpenAI-compatible REST API with streaming, plus a built-in web UI and a TUI chat client. For interactive terminal use, cake chat takes the model name directly.
cake serve evilsocket/Qwen3-0.6B
cake chat Qwen/Qwen3-0.6BImage and audio models need explicit flags, because the auto-detection path is for text. Image generation uses --model-type image-model with --image-model-arch flux1. Voice synthesis uses --model-type audio-model with --voice-prompt pointing at a wav file for cloning.
Turning several machines into one cluster with --cluster-key
The distributed path starts with workers that have no model. Each worker runs cake run with a shared secret and a name, and nothing else.
cake run --cluster-key mysecret --name gpu-server-1 # machine A
cake run --cluster-key mysecret --name macbook # machine BThen the master, which does have the model, runs the same command with a prompt and the same key, or starts a server in the same role.
cake run evilsocket/Qwen3-0.6B "Hello" --cluster-key mysecret
cake serve evilsocket/Qwen3-0.6B --cluster-key mysecretThe cluster key is the only thing joining the two sides, so it is also the only access control in the default path. The README does not describe transport encryption or worker authentication beyond the shared key, and it does not document what happens when a worker drops mid-generation. If you plan to run this across machines you do not fully control, that gap is the first thing to investigate in docs/clustering.md.
The Docker route exists for Linux with NVIDIA. The Dockerfile builds on nvidia/cuda:12.6.0-devel-ubuntu24.04 and uses cargo-chef to cache dependency compilation, and it takes a CUDA compute capability argument you must set to match your target GPU. The docker-compose example defines a master with an API on port 8080 plus two workers, and expects model data under ./cake-data/<model-name>/ and a topology file at ./topology-docker.yml. One constraint is stated plainly in the compose file: Docker on macOS cannot access Metal GPUs, so Apple Silicon clusters must be built and run natively with the metal feature.
Where cake breaks down
The first limitation is the one the project states about itself. The README calls the code experimental and says it is changing very quickly. A repository whose last push was on 2026-04-24, with no published releases, is not something to put behind a production endpoint. The version in the workspace manifest is 0.1.0.
Layer sharding has a structural cost that no implementation avoids. Splitting transformer blocks across devices turns generation into a pipeline, and the slowest link sets the pace. A phone on wifi participating in a cluster with a desktop on ethernet will make every token wait on the phone. Cake's proportional assignment to VRAM and compute addresses balance, not latency. If your goal is throughput on a single prompt, a single machine with a smaller quantised model will usually beat a heterogeneous cluster, and the README does not claim otherwise.
The weight streaming design also assumes the master can hold the full model and push shards over the network. For a genuinely large model, the master needs enough disk and the initial transfer is a real cost, mitigated only by the worker-side cache. There is no documented resume for an interrupted shard transfer, and the README does not document rollback or recovery behaviour for a partial cluster.
Finally, the mobile worker path is a build pipeline, not a download. The Makefile shows iOS and Android targets built through cargo with a specific target triple and a UniFFI binding generation step. Anyone expecting to install a worker app and join a cluster should read the Makefile before assuming that.
How cake differs from Candle-based serving stacks
The closest comparisons people search for are Candle-vllm and dllama, both of which come up alongside cake in search results. The difference is architectural rather than a matter of features.
Candle-vllm is a serving layer built on the Candle Rust framework, aimed at running models efficiently on a single machine or a single node with a GPU. Its job is to be a fast, conventional inference server. Cake's job is to be a distributed one: the clustering, the mDNS discovery, the topology files and the weight streaming are the product, not a wrapper around it. If you have one good GPU and want maximum tokens per second, cake's cluster machinery is overhead you do not need.
Dllama takes the distributed approach too, and the overlap with cake is real. The meaningful difference visible here is scope. Cake describes text, image and voice models in one server, with the architecture detected from the checkpoint, and ships a mobile worker built as a Kotlin Multiplatform app. That is a broader target than a text-only distributed runner, and it is also more code to keep working. The README's own warning about rapid change applies to that breadth.
If your requirement is a stable OpenAI-compatible endpoint on one machine, none of these three is the interesting choice; a mainstream serving stack will be better documented. Cake is worth a look specifically when the hardware you have is heterogeneous and you want to use all of it.
Licence and the cost of keeping up
The licence is the FAIR License (Free for Attribution and Individual Rights) v1.0.0, and the Cargo.toml points at a license-file rather than a standard SPDX identifier, which is why the repository is reported as NOASSERTION. The README summarises three tiers. Non-commercial use, meaning personal, educational, research and non-profit, is freely permitted. Commercial use, meaning SaaS, paid apps or any monetisation, requires visible attribution to the project and its author. Business use, meaning any use by or on behalf of a business entity, requires a signed commercial agreement with the author, with an email address given for inquiries.
That third tier is broader than most teams expect. A company running cake internally for research is business use under this wording, not non-commercial research. Read the LICENSE file itself rather than the README summary, and treat this as a procurement question rather than a technical one.
Upgrade cost is the other ongoing expense. The workspace has three crates plus a separate mobile app directory, and the release profile in Cargo.toml is tuned for iteration rather than distribution: thin LTO, eight codegen units, symbols kept, panic set to abort. Those settings make builds faster and binaries larger, which is a reasonable choice for a project moving quickly and a poor one for something you ship. Dependency licence auditing is handled through cargo license, which the README tells you to install with cargo install cargo-license. Pinning a commit is advisable, given that the project describes itself as changing very quickly.
Editorial conclusion
Adopt cake if you have spare heterogeneous hardware, a non-commercial or internal research use, and you want to see how far layer sharding across a phone and a laptop actually gets. Do not adopt it for a product: the FAIR licence requires a signed commercial agreement for any use by or on behalf of a business entity, and the README itself calls the code experimental and changing quickly. Before committing, verify three things on your own hardware: that the model you need is in the supported families in docs/models.md, that the backend feature you build with matches your GPU, and that the topology file you write maps layer names that exist in your checkpoint.
Frequently asked questions
What is cake ai?
Cake is a multimodal AI inference server written in Rust that runs text, image and voice models either on a single machine or sharded across a cluster of devices including iOS, Android, macOS, Linux and Windows. It exposes an OpenAI-compatible REST API with streaming, plus a built-in web UI and a TUI chat client.
Is cake ai free?
It is free for non-commercial use, which the README defines as personal, educational, research and non-profit. Commercial use requires visible attribution, and any use by or on behalf of a business entity requires a signed commercial agreement with the author under the FAIR License v1.0.0.
How to make cake ai?
You build it from source with Cargo, selecting a backend feature such as cuda, metal, vulkan or the CPU-only default, then pull a model from HuggingFace with cake pull and run it with cake run or cake serve. To distribute, start workers with cake run --cluster-key and run the master with the same key.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/evilsocket-cake)Star history chart for your README
[](https://hysenlabs.com/projects/evilsocket-cake)Was this useful?