# Splash Local Inference on Apple Silicon

> Splash is a C++ inference server for coding agents and OpenAI or Anthropic compatible applications on recent Apple silicon Macs. Its main appeal is model-specific decoding with DFlash2 speculation, Metal kernels, cached prefixes, automatic batching, and local API compatibility, but its system requirements are strict.

**incoai/splash** — A local inference engine for Apple silicon, built around the model.

- Repository: https://github.com/incoai/splash
- Website: https://inco.ai/blog/splash/
- Stars: 1,769 · Forks: 169
- Language: C++
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/incoai-splash

## The local Mac inference gap Splash targets

Splash addresses a narrow but concrete problem: running coding agents and applications that speak familiar model APIs entirely on one Apple silicon Mac. The target audience includes developers who want a local OpenAI Chat Completions, Responses, Completions, or Anthropic Messages endpoint without sending prompts and code to a hosted service. Splash also provides a built-in chat page and launcher commands for several coding agents, so the project is intended to occupy both the model server and local assistant workflow.

The platform boundary is unusually strict. The README requires an Apple M3 or newer Mac, macOS 26.4 or later, and Homebrew. A 4-bit Qwen example needs at least 36 GB of unified memory, with 48 GB recommended. Smaller GGUF variants can run on 24 GB Macs, but that path is an explicit exception rather than the default installation profile. This is not a general-purpose cross-platform inference package. Its Metal kernels, memory planning, startup flow, and published performance comparisons are all centered on recent Macs.

Splash also distinguishes model compatibility from mere file format support. It supports listed Qwen families through GGUF and MLX targets, and the README additionally identifies Prism ML Ternary Bonsai 2 in PQ2_0 as supported. Whole-BF16 and F16 models are refused when their projections and experts contain only those higher-precision tensor types. The installer therefore acts as a compatibility gate rather than accepting every local model file.

## DFlash2 speculation, Metal execution, and request batching

Each supported model is paired with a trained DFlash2 draft, which Splash selects automatically. This is the speculative decoding component: a smaller draft proposes possible future tokens while the target model and model-specific Metal kernels execute the accepted work. The repository description and README identify DFlash2, specialized Metal kernels, automatic memory planning, and speculative decoding as central parts of the design, rather than optional extras.

Weights are converted into the layouts expected by those kernels as they load. The README says this conversion does not create another on-disk copy. The runtime, scheduler, cache, and API are shared across supported model shapes, while the paired kernels and draft are specific to each model. That arrangement reduces duplicated server infrastructure but means compatibility is tied to the project defining the necessary kernels, draft pairing, tokenizer behavior, and model loading support.

Splash reuses cached prefixes and automatically batches concurrent requests. Its cache can operate in memory and on SSD, while context and memory limits are planned automatically up to the model's native context window. The cache explains the README's focus on cached time to first token, but the cached figures belong to a documented Splash 1.0 benchmark rather than a guarantee for every Mac, model, prompt, or application. Concurrent batching also means that performance under an agent workload cannot be inferred solely from a single-request token rate.

The default execution path uses the GPU and, for the 27B model, the Apple Neural Engine. Passing `--disable-ane` keeps prefill on the GPU alone. Vision is supplied by the target model, and `--language-only` removes that path for text-only use. This lets one server shape support both image and document inputs without requiring every deployment to pay their loading and execution costs.

## Installing Splash and launching the first model

The documented Homebrew installation requires macOS 26.4 or later on an Apple M3 or newer Mac. Check the unified-memory requirement before choosing the example model. The command below installs Splash, then starts a local server with the 27B GGUF model from the quick start:

```bash
brew install incoai/tap/splash
splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
```

The first run downloads the target model and its matching draft. It then serves on `127.0.0.1:8000`. Later launches reuse those downloads, although disk space is still required for storage. The README points to `DEVELOPMENT.md#model-storage` for storage requirements.

Wait until the server prints `Ready`, and leave that terminal open. The built-in chat interface is then available through the local URL:

```text
http://127.0.0.1:8000
```

To use an installed coding agent, open a second terminal and select its launcher:

```bash
splash opencode
```

The README also documents `splash claude`, `splash codex`, `splash hermes`, and `splash pi`. If Splash was started on another `--port`, pass that same port to the agent command. Stop the server with Ctrl+C in the server terminal.

This startup path downloads substantial model assets. It is not a lightweight demonstration that immediately uses a tiny built-in model. Adoption therefore begins with a storage and memory decision, not merely a package installation.

## Local OpenAI and Anthropic requests through Splash

Once `Ready` appears, the first API request can target OpenAI Chat Completions. This request asks the selected model to explain speculative decoding:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M",
    "messages": [{"role": "user", "content": "Explain speculative decoding in one sentence."}]
  }'
```

The documented API surface includes streaming, tool calls, JSON Schema output, and base64 images and PDFs. Besides Chat Completions, Splash exposes OpenAI Responses and Completions plus Anthropic Messages. This makes it relevant to applications written against either API family without requiring the application to invoke Splash's built-in browser interface.

Reasoning behavior follows the model default in Chat Completions. Setting `"reasoning_effort": "none"` disables it for that interface. Anthropic Messages requests reason only when they include `thinking`. These are different request controls, so application code migrating between API styles must not assume identical reasoning settings.

The server listens on localhost without authentication by default. The README does not treat that as a remote deployment configuration. LAN access, authentication, and browser clients on other origins require separate server configuration covered in `DEVELOPMENT.md` and `splash serve --help`. Binding a local inference server to a broader network without reading that configuration would expose an unauthenticated interface unless additional controls are configured.

## Memory, context, cache, and precision controls

Splash plans memory and context automatically, but it exposes controls for machines shared with editors, browsers, and coding agents. The first decision is how much unified memory can be assigned to Metal without preventing other applications from operating:

```bash
splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M --max-memory 28G
```

The README uses `28G` as the example for `--max-memory`. It also documents `--max-context 100K` for an explicit context limit and `--language-only` for text-only serving. These options trade available resources against workload behavior. A context limit that fits the Mac does not guarantee that simultaneous agent sessions, long prompts, vision inputs, and batched requests will fit.

Cache storage has separate policies. By default, the SSD cache is automatic and may reach 32 GiB. `--max-cache-disk 16G` caps it, while `0` disables it. `--persistent-cache` preserves the SSD cache across restarts; it is off by default. Model data is also retained on disk between launches, so disabling the disk cache does not mean the installation needs no local storage.

Numeric state can be adjusted through `--kv-format bf16` and `--state-format bf16`. The defaults documented by the README are INT8 KV cache and FP32 recurrent states, with BF16 recurrent states described as half the size. These flags expose implementation-level choices rather than generic quality presets. The README does not provide a universal recommendation based on output accuracy, and its llama.cpp prediction comparison specifically states that it used BF16 KV with `--disable-ane`.

## Limits of the Apple silicon and model-specific design

The largest limitation is hardware eligibility. Splash requires an M3 or newer Mac and macOS 26.4 or later, so Intel Macs and older supported Macs are outside the documented setup. The standard 4-bit examples also exceed the unified-memory capacity of many 24 GB configurations. The README identifies smaller GGUF variants as the route for those machines, but it does not say that every listed model has a suitable smaller variant.

Model coverage is another boundary. The table explicitly names Qwen3.8-27B and Qwen3.6-35B-A3B, while the text adds Ternary Bonsai 2 and several MLX quantization formats. It also states that whole-BF16 and F16 targets are unsupported when all projections and experts use BF16, F16, or F32. A GGUF extension alone is therefore not a reliable compatibility test. Users need the exact target and its supported format.

The server defaults to localhost without authentication, which is appropriate for a local workflow but unsuitable as an unchanged network service. The README points to dedicated configuration for LAN access and authentication, yet it does not promise a complete production deployment guide in the quick start.

Splash is also a poor fit when the requirement is portability across Linux, Windows, x86, or NVIDIA GPUs. Its specialized Metal path is the core of the project. Likewise, an application that only needs an embedded library and no HTTP server would need a different kind of inference project because the documented interface centers on a Splash server, compatible APIs, and agent launchers.

## Splash compared with llama.cpp, maintenance, and licensing

The README compares Splash with llama.cpp using the same Unsloth UD-Q4_K_M weights on Metal. Its published tables report higher decode speeds for Splash on M5 Pro and M3 Max systems, including 74 versus 16 tok/s for Qwen3.8-27B on the M5 Pro and 175 versus 69 tok/s for Qwen3.6-35B-A3B. These are attributed project benchmark figures, not a result reproduced for this review. The same section reports high next-token agreement between engines and says Splash closely matches llama.cpp's predictions under its comparison settings.

The practical difference is not simply that one project uses Metal. llama.cpp offers the baseline execution path, with llama.cpp MTP included as another comparison point. Splash adds a model-matched DFlash2 draft, specialized kernels, automatic memory and context planning, prefix reuse, request batching, and a local server that exposes OpenAI and Anthropic APIs. A developer prioritizing a broad engine ecosystem may have different priorities from one optimizing this particular Mac workflow, but the repository does not document a general feature matrix beyond these comparisons.

The repository is not archived, and its last push was on 2026-10-14. Releases 1.3.0, 1.3.1, and 1.3.2 were published on 2026-10-06, 2026-10-09, and 2026-10-14 respectively. That sequence indicates recent project activity, but model-specific kernels and draft pairings also create an ongoing upgrade obligation: users should confirm that their selected target remains supported after updating Splash.

The repository identifies Apache-2.0 as its license and includes `LICENSE` and `THIRD_PARTY_NOTICES`. Organizations considering redistribution should inspect those files and the licensing of separately downloaded model assets. The README does not state the license terms of those assets, so the engine's Apache-2.0 designation should not be treated as proof that every downloaded model has the same terms.

## Conclusion

Developers running coding agents or compatible applications on a qualifying Mac should first confirm hardware, macOS, model memory, and workload requirements. Operators needing x86 Linux, NVIDIA deployment, or a general-purpose model runner should choose another engine. For eligible users, the clearest first check is whether the intended model and matching DFlash2 draft are supported, followed by whether the machine has enough unified memory for the selected 4-bit variant.

## FAQ

### What Apple hardware and macOS version does Splash require?

Splash requires an Apple M3 or newer Mac running macOS 26.4 or later. The documented 4-bit examples also require at least 36 GB of unified memory, with 48 GB recommended.

### How do I install and start Splash with its example model?

Install the Homebrew package with `brew install incoai/tap/splash`, then run `splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M`. The first launch downloads the model and matching draft, then serves on `127.0.0.1:8000` after printing `Ready`.

### Does Splash expose OpenAI and Anthropic compatible APIs?

Yes. The documented server exposes OpenAI Chat Completions, Responses, and Completions, plus Anthropic Messages. It supports streaming, tool calls, JSON Schema output, and base64 images and PDFs.

## Sources

- [incoai/splash on GitHub](https://github.com/incoai/splash)
- [License: Apache-2.0](https://github.com/incoai/splash/blob/main/LICENSE)
- [Project website](https://inco.ai/blog/splash/)
- [README](https://github.com/incoai/splash/blob/main/README.md)
- [Releases](https://github.com/incoai/splash/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/incoai-splash