Speech Swift: on-device ASR, TTS and diarization for Apple Silicon

What Speech Swift actually solves

Most speech stacks assume a server. You send audio to an endpoint, pay per minute, and accept that the recording leaves the machine. That is fine for batch transcription and wrong for a dictation tool, a meeting recorder, or anything shipped to a user who expects the microphone to stay private. Speech Swift targets that second category. The README states the pitch directly: on-device recognition, synthesis and understanding for Mac and iOS, with no cloud, no API keys, and no data leaving the device.

The audience is Swift developers building for Apple Silicon who want a package rather than a pile of converted model files. The repository lists 16 public projects with verifiable package references, including AnythingLLM, Anarlog and several dictation apps, so the integration path is not theoretical. The capability list is unusually wide for one package: ASR across many model families, alignment, TTS, speech-to-speech, enhancement, source separation, wake word, VAD, diarization and speaker identity. That breadth is the reason to look at it, and also the reason to check the specific guide you need before you commit.

How the MLX and CoreML split works

The architecture is two runtimes behind one Swift API. MLX Swift handles the models that need more flexible execution, and CoreML targets the Neural Engine. The README describes Qwen3-ASR as running on both, Whisper Large-v3 Turbo as a native CoreML runtime on the ANE, and Kokoro TTS as CoreML on the Neural Engine. Model choice therefore decides which chip does the work, and the package exposes both paths rather than forcing one.

Quantisation is a first-class part of the design, not an afterthought. Several entries list FP16, INT5 and INT8 variants: Cohere Transcribe 2B and Voxtral Mini 3B both ship FP16/INT5/INT8, MOSS Transcribe Diarize lists INT5/INT8, and Magpie TTS lists MLX INT8 at 411 MB or CoreML INT8 at 342 MB. The trade-off is explicit: a smaller quantised bundle costs accuracy, and the README does not publish an accuracy table per quantisation level, so that decision is yours to measure.

The build step reflects the MLX dependency. The Makefile's release target runs swift build in release mode and then calls ./scripts/build_mlx_metallib.sh release, which compiles the Metal shader library MLX needs. Skip that script and the MLX-backed models have no kernels to run on.

Installing Speech Swift and running a first transcription

The README points to Homebrew for the command-line tool and to the Swift package for library use. The formula is published as speech, so the install is a single command. The README does not document what the binary prints on a first run, so treat the output as something to observe rather than something described here.

brew install speech

For library work, clone the repository and build with the Makefile, which wraps the Swift build and the MLX Metal library compilation.

make build

The Makefile defaults CONFIG to release, so make build produces a release build without extra flags. For a debug build with the same Metal step, use make debug. The test target depends on debug and runs a filtered set of unit tests, explicitly skipping E2E tests.

make test

If you want to see the models running rather than wired into your own app, the Examples directory contains SpeechDemo, DictateDemo, VoiceChatMCP, iOSBenchmark, iOSEchoDemo and PersonaPlexDemo, plus a websocket-client.html. Those are the reference integrations the repository ships. The per-model guides live at soniqo.audio, and the README links each model name to its own guide page, so the first real use is: pick a model, open its guide, and follow it. For a dictation-style flow the README points at the Streaming Dictation guide, which describes partials and end-of-utterance detection through Parakeet-EOU-120M.

The 0.0.x version number is the real constraint

The most recent release is v0.0.27, dated 2026-09-02, with v0.0.26 and v0.0.25 landing in the two weeks before that. Three releases in roughly two and a half weeks is a fast cadence, and the version prefix is honest about what it means: the public API has not been declared stable. If you vendor this package, budget for reading release notes between upgrades, because a minor version bump here is not a compatibility promise.

The second limitation is platform. Apple Silicon is the whole target. There is no Android build, no Linux build, and no CUDA path, so a cross-platform product needs a second speech stack for every non-Apple client. The README's own framing, on-device for Mac and iOS, is a scope statement rather than a marketing line.

The third is documentation depth. Model guides are linked per model, and the README is the index rather than the manual. The Makefile's test target skips E2E tests, so the automated suite that runs by default is unit-level. Nothing in the repository states a rollback procedure for a model or package upgrade, and the README does not document one.

Where Speech Swift is the wrong choice

If your audio already lives in a cloud pipeline and latency budgets are measured in hundreds of milliseconds across a fleet of Linux workers, this package solves a problem you do not have. Whisper via a hosted API or a CUDA deployment will be cheaper to operate and simpler to scale, because you are not shipping model weights to every client.

If you need one model and nothing else, the breadth here works against you. A team that only wants English transcription could take Whisper Large-v3 Turbo through CoreML and ignore the rest, but they would still be pulling a package whose release cadence is set by a much larger model catalogue. A narrower wrapper around a single CoreML model is less to maintain.

If your target device is not Apple Silicon, stop here. The two runtimes are MLX Swift and CoreML, and neither has a meaningful non-Apple path. There is no fallback runtime documented in the repository.

How it differs from Apple's own Speech framework

Apple's Speech framework is the obvious alternative and the difference is in what you control. Apple's recogniser is a system service: you get the languages and accuracy Apple ships, you do not choose the model, and you cannot swap in a 1,672-language recogniser or a specific quantisation. Speech Swift inverts that. You pick the model, you pick MLX or CoreML, you pick FP16 against INT5, and the weights run in your process. The cost is that you now own model downloads, bundle sizes and the accuracy consequences of your quantisation choice.

The second alternative is a server-side stack such as Whisper behind an HTTP API. That gives you horizontal scaling and a single place to upgrade, at the price of sending audio off-device. Speech Swift's entire reason to exist is refusing that trade. If privacy is not a requirement, the server option is usually less work.

The third is rolling your own CoreML conversion. Speech Swift's value there is the catalogue and the packaging: Kokoro at 82M with 54 voices and iOS readiness, Parakeet TDT on the Neural Engine, Nemotron streaming with punctuation and capitalisation built in. Converting and validating that set yourself is weeks of work the package has already done.

Licensing and the cost of keeping up

The package itself is Apache-2.0, which is permissive and business-friendly for the Swift code. That licence does not cover the model weights it downloads. The README flags exceptions in the model list: F5-TTS is described as non-commercial, and Higgs TTS 3 as research/non-commercial. Other entries name upstream publishers such as NVIDIA, Meta and Microsoft, and those weights carry their own terms. A permissive wrapper around a non-commercial checkpoint does not make the checkpoint commercial. Check the licence of each model you actually ship, not just the package licence. This is not legal advice.

Maintenance cost is dominated by the release cadence. The last push to the default branch was on 2026-09-12, and the most recent tagged release was v0.0.27 on 2026-09-02, so the project is moving. The practical implication is that pinning to an exact version and reading the diff before moving is the cheaper habit than tracking main. Upstream model publishers also revise weights independently of this repository, which means a model swap can change your output without any change to the Swift code.

Editorial conclusion

Adopt Speech Swift if your product runs on Apple Silicon and the audio cannot leave the device: dictation, transcription, voice agents, diarization. Do not adopt it if you need Android, CUDA, or a stable API surface, because the version is 0.0.27 and the public API can still move between releases. Before committing, verify three things on your own hardware: that the specific model guide you need exists at soniqo.audio, that the model's licence matches your use (F5-TTS, Higgs TTS 3 and other upstream weights carry non-commercial or research terms), and that the RTF for your target chip is acceptable when you run the Example app rather than a single model in isolation.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. soniqo/speech-swift on GitHub