Soup: Fine-Tuning an 8B LLM on a 4 GB Laptop GPU with One YAML File
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
At a glance
- What is it?
- Soup is a Python CLI that turns LLM fine-tuning into a single YAML-driven command. Its layer-streaming mode claims to train an 8B model on a 4 GB GPU, but a recent fix exposes the cost of that promise.
- Who is it for?
- Adopt Soup if you are an individual developer or small team with a consumer GPU (4 to 8 GB) who wants to run QLoRA or DPO fine-tunes without managing SSH clusters or complex training scripts. Do not adopt it if you need production-grade multi-node training, if you cannot tolerate beta features, or if your model is not in the supported Transformers or MLX paths.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Soup Actually Solves
Fine-tuning a large language model usually means assembling a stack: a training script, a quantization library, a GPU detection routine, and a way to move data to a remote box. Soup collapses that into one YAML file and one command. The README states that teams spend 30 to 50 percent of their time fighting infrastructure, and Soup targets that waste directly. The intended user is someone who owns a laptop with a modest GPU, like an RTX 3050 with 4 GB of VRAM, and wants to fine-tune an 8B parameter model locally. It also supports cloud training through a Lambda Labs integration, but the core pitch is local, consumer-grade hardware. The project is not aimed at researchers who need custom training loops or who manage large fleets; it is for practitioners who want a repeatable fine-tune without becoming infrastructure engineers.
Layer Streaming: The Mechanism Behind the 4 GB Claim
The central technical innovation is layer streaming. Normally, a frozen base model stays resident in VRAM during LoRA training, which is why an 8B model exceeds a 4 GB card. Soup instead keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. The README describes a pre-flight that shows a 3.60 GB base store pinned in RAM across 32 layers, with two 113 MB VRAM buffers. That allows training with a peak of 3.32 GB on an RTX 3050 Laptop 4 GB, at 119.6 tokens per second. The claim is that this is bit-exact against a normal resident run. However, the release notes for v0.74.0 reveal a serious caveat: the frozen base was being loaded in fp32 the whole time, which wasted VRAM on all three load paths. Fixing that cut peak VRAM by 2.59x on an H100, from 48,241 MiB to 18,658 MiB. That fix landed after the 119.6 tok/s measurement, which was taken on v0.72.2. The README admits the number has not been re-run on a 4 GB card since the v0.73.0 correctness repair that cost 4.8 percent at 32B. So the headline figure is stale, and the mechanism is still marked BETA.
Getting Running: Commands and Configuration
Installation is a standard pip command. The README shows: pip install "soup-cli[train]". The bare soup-cli package is a light CLI, and the [train] extra pulls in the dependencies needed for fine-tuning. After installing, you run soup init --template chat to generate a starter YAML config, then soup train to start training. The configuration is a single YAML file, which the project claims eliminates what it calls config hell. The layer-streaming feature is opt-in, controlled by a stream_layers: true key in that YAML. There is also a web UI mentioned in the README, though the details are not elaborated in the supplied material. The CLI includes commands beyond train, such as soup serve, which was changed in v0.74.0 to exit with code 2 if bound to a non-loopback host without a --tool-auth-token. That change is a breaking one, so users upgrading need to account for it. The project also lists Python 3.10 to 3.12 as supported.
The FP32 Bug and What It Means for Your VRAM
The v0.74.0 release note is the most informative piece of material for evaluating Soup. It states that every SFT load silently upcast the frozen base to fp32. A base that never receives an optimizer step was materialised at twice its checkpoint precision. This affected all three load paths. On an H100 with Llama-3.1-8B and LoRA, the fix reduced peak memory from 48,241 MiB to 18,658 MiB, a 2.59x reduction. The release note says a trainable base still loads fp32 deliberately, so the bug was specific to frozen bases. This is a significant finding because it means earlier versions of Soup were wasting a large fraction of VRAM on every fine-tune. Users who relied on the old 4 GB claims may have been unable to run models that the fixed version can handle. The release title, "the base was loaded in fp32 the whole time," is blunt. For anyone considering Soup, this raises a question: what other silent inefficiencies exist? The project has a history of correctness repairs, including the v0.73.0 fix that cost 4.8 percent throughput at 32B. That suggests the performance numbers are moving targets.
Known Limitations and Failure Modes
The README and release notes expose several concrete limitations. First, the declared torch>=2.5.0 floor does not work with trl>=0.29. At torch 2.5.1, trl cannot import, so a fresh install must resolve a newer torch. This is a dependency conflict that can break an existing environment. Second, the free Colab and Kaggle tiers could not stream at all on T4, P100, V100, or GTX 16xx cards before v0.74.0. The cause was that peft creates LoRA adapters in the checkpoint's dtype while the fp16 GradScaler needs fp32 gradients. That crash was fixed, but it indicates that layer streaming is sensitive to GPU architecture and dtype handling. Third, there were four SSRF bypasses of the same shape, where abbreviated IPv4 spellings reached telemetry and webhook guards, and also the OTLP tracing validator. Security fixes are good, but the repeated pattern suggests the guard logic needed multiple passes. Finally, the README calls layer streaming BETA. A beta feature that claims bit-exactness but has already required a correctness repair should be treated with caution.
Alternatives and How They Differ
The most direct alternative is using Hugging Face's Transformers with PEFT and QLoRA directly, without a wrapper. That approach gives you full control over the training loop, but requires you to write and maintain your own script, handle GPU detection, and manage quantization. Soup's layer-streaming mechanism is its differentiator; a standard PEFT setup loads the entire base model into VRAM, which is why an 8B model does not fit on a 4 GB card without such tricks. Another alternative is Axolotl, which also uses YAML configs for fine-tuning, but it is designed for larger multi-GPU setups and does not focus on layer streaming for consumer GPUs. Axolotl assumes you have a capable GPU or a cloud instance, whereas Soup is explicitly built for the low-VRAM case. There is also the MLX extra for Apple Silicon, which the README mentions as a supported path, but that is a different hardware direction. For someone on a 4 GB laptop GPU, Soup's layer streaming is the only option among these that claims to make an 8B model trainable, but the trade-off is beta-level maturity and a narrower feature set.
Maintenance, Licensing, and Upgrade Cost
Soup is licensed under Apache-2.0, which is permissive for commercial use, though this is not legal advice. The project is under active development, with releases v0.73.2, v0.73.3, and v0.74.0 all within a month of each other in mid-2026. The release notes show a pattern of frequent fixes, including security patches and dependency updates. The upgrade cost is not trivial: v0.74.0 introduced a breaking change to soup serve, and the release notes mention that the two pip extras, train and mlx, previously declared ranges that could not be satisfied together, so users who wanted both had to wait for a fix. The v0.73.0 correctness repair cost throughput at 32B, meaning users who upgraded may have seen slower training. The project also notes that 116 of 120 merged pull requests in v0.74.0 came from outside the maintainer, which suggests a community-driven maintenance model, but also that the core maintainer may be a bottleneck for review. There is a DOI for a paper, which implies some academic rigor, but the benchmarks directory is the place to check for reproducible measurements. The README explicitly invites users to verify on a free Colab T4 notebook, which is a positive sign for transparency.
Editorial conclusion
Adopt Soup if you are an individual developer or small team with a consumer GPU (4 to 8 GB) who wants to run QLoRA or DPO fine-tunes without managing SSH clusters or complex training scripts. Do not adopt it if you need production-grade multi-node training, if you cannot tolerate beta features, or if your model is not in the supported Transformers or MLX paths. Before committing, verify three things: check the v0.74.0 release notes for the fp32 fix and its impact on your exact model size, confirm your torch version resolves above 2.5.1 to avoid the trl import failure, and run the Colab T4 notebook that asserts bit-identical output between streamed and normal training. Soup is not a magic bullet; it is a focused tool that trades some flexibility for a single-command workflow, and the recent correctness repairs show that its headline performance figures need re-validation on your hardware.
Community notes