MLX Audio Swift: a modular Swift SDK for on-device audio on Apple Silicon

What MLX Audio Swift is for, and who it leaves out

This is a Swift SDK for audio work on Apple Silicon, built on MLX. The README describes it as "a modular Swift SDK for audio processing with MLX on Apple Silicon" and lists the platform badge as macOS 14+ and iOS 17+, with Swift 5.9+. The intended reader is an app developer who wants text-to-speech, speech-to-text, voice activity detection or speaker diarization inside a Swift codebase, rather than shelling out to a Python process or calling a hosted endpoint.

The modularity is the actual pitch. Instead of one framework that pulls in every model family, the package exposes MLXAudioCore, MLXAudioCodecs, MLXAudioTTS, MLXAudioSTT, MLXAudioVAD, MLXAudioSTS and MLXAudioUI as separate products, and the README's installation snippet shows importing only MLXAudioTTS and MLXAudioCore. That matters on Apple platforms where binary size and link time are visible costs: a dictation feature should not carry a dozen text-to-speech architectures.

Who it leaves out is anyone not on Apple hardware. MLX is Apple's array framework, and nothing in the repository suggests a non-Apple backend. If your inference runs on Linux or in a container, this is the wrong layer entirely. The same applies to anyone who needs a hosted API: this is a library you compile into your own binary.

How the modular targets divide the audio pipeline

The split follows the stages of an audio pipeline rather than the model vendors. MLXAudioCore holds base types, protocols and utilities. MLXAudioCodecs holds the tokenizers and vocoders that turn between waveforms and discrete codes: the README names SNAC, Encodec, Vocos, Mimi, DACVAE, Descript DAC, Fish S1 DAC, S3TokenizerV2, MOSS Audio Tokenizer, Higgs Audio Tokenizer and Step-Audio-2 token-to-wav. MLXAudioTTS and MLXAudioSTT sit on top of those with concrete model implementations, MLXAudioVAD covers voice activity detection and diarization through Sortformer, SmartTurn, FSMN VAD and Silero VAD, and MLXAudioSTS covers speech-to-speech with LFM2.5-Audio, SAM-Audio, MossFormer2-SE and DeepFilterNet. MLXAudioUI is the SwiftUI layer.

The data flow visible in the README is consistent across tasks. You load a model with an async fromPretrained call that takes a HuggingFace repository identifier, then call generate with audio or text, then either read a text field or save an audio array. The diarization example is the clearest: load the audio into an array, call generate with a threshold of 0.5, and iterate over output.segments printing speaker, start and end. The TTS example is the mirror image: generate returns an audio array plus a model sample rate, and saveAudioArray writes it out.

One design detail worth noting is that the model list is long but shallow. Many families have a per-model README under Sources/MLXAudioTTS/Models/ or Sources/MLXAudioSTT/Models/, which is where the real per-model parameters live. The top-level README is an index, not a reference.

Installing MLX Audio Swift with Swift Package Manager

The README's installation section adds the package as a SwiftPM dependency pointing at the main branch, and then lists the products you want. Copy the dependency line and the product lines into your Package.swift dependencies and target dependencies respectively. Because the README pins to branch: main rather than a version tag, you are tracking the default branch; the releases listed in the repository are v0.1.1, v0.1.2 and v0.1.3, with v0.1.3 dated 2026-07-09, so a versioned requirement is available if you prefer it.

dependencies: [
    .package(url: "https://github.com/Blaizzy/mlx-audio-swift.git", branch: "main")
]

// Import only what you need
.product(name: "MLXAudioTTS", package: "mlx-audio-swift"),
.product(name: "MLXAudioCore", package: "mlx-audio-swift")

For a first real use, the README's text-to-speech example is the shortest path. It loads Soprano from a HuggingFace identifier, generates with maxTokens, temperature and topP, and writes the result to a URL. Expect the first run to spend its time fetching weights, since the model identifier points at a HuggingFace repository rather than a bundled asset.

import MLXAudioTTS
import MLXAudioCore

let model = try await SopranoModel.fromPretrained("mlx-community/Soprano-80M-bf16")

let audio = try await model.generate(
    text: "Hello from MLX Audio Swift!",
    parameters: GenerateParameters(
        maxTokens: 200,
        temperature: 0.7,
        topP: 0.95
    )
)

try saveAudioArray(audio, sampleRate: Double(model.sampleRate), to: outputURL)

If you would rather stream than wait for the full array, the README shows generateStream returning an async sequence of events: .token during generation, .audio with the final audio shape, and .info carrying an info.summary string.

The model weights are the deployment problem, not the Swift code

Every example in the README resolves its model from a HuggingFace identifier at runtime. fromPretrained takes a string like "mlx-community/Soprano-80M-bf16" or "mlx-community/diar_streaming_sortformer_4spk-v2.1-fp16". The README does not document a download cache location, a way to pre-seed weights into an app bundle, an offline mode, or what fromPretrained does when the network is unavailable. For a desktop tool that is a mild annoyance; for a shipped iOS app it is a distribution question the documentation currently does not answer.

There is a second, quieter constraint. The README uses the word "Beta" in at least one model identifier, mlx-community/VyvoTTS-EN-Beta-4bit, and several quantisation suffixes appear across the tables (4bit, 8bit, bf16, fp16). Those suffixes are part of the identifier string you must pass, so a typo in a quantisation suffix fails at load time rather than compile time. Nothing in the top-level README validates model names ahead of the call.

Finally, the install snippet tracks branch: main while releases exist. That combination means a fresh clone can pick up changes that are not in any tagged release, and the README does not describe a deprecation policy for the Swift APIs it shows.

How it compares with calling a hosted speech API

The obvious alternative is a hosted transcription or synthesis service, where you send audio or text over HTTP and get a result back. The difference in approach is total: a hosted API moves the model, the compute and the versioning to someone else's infrastructure, and you get a stable endpoint instead of a Swift type. You also get a per-request cost and a network dependency on every call, which is exactly what this SDK avoids.

MLX Audio Swift inverts those properties. Inference happens on the device through MLX, so there is no per-request billing and no audio leaving the machine, but you own the model weights, the memory footprint and the Apple Silicon requirement. On a Mac, that trade is usually easy to justify for dictation or a reader feature. On a phone, the memory ceiling and the weight-download problem are the constraints you would be arguing about.

A second alternative is running a Python MLX audio stack and calling it from Swift over a local socket. That keeps you on MLX but adds a process boundary and a Python runtime to ship. The SDK's reason to exist is removing that boundary, and the module layout is what makes it credible: you can take MLXAudioVAD alone for diarization without dragging in the TTS families.

Maintenance, licensing and what a version bump costs you

The repository is not archived, and the last push was on 2026-09-13, three days before the date used here. The release cadence visible in the release list is modest: v0.1.1 on 2026-03-07, v0.1.2 on 2026-03-14, and v0.1.3 on 2026-07-09. The gap between the last release and the last push means the default branch carries work that is not in v0.1.3, which is the practical cost of the README's branch: main instruction.

Upgrade cost concentrates in the model layer rather than the core types. Because each model family has its own directory under Sources/MLXAudioTTS/Models/ and Sources/MLXAudioSTT/Models/, a change to one family's implementation should not force a change in another, but the shared GenerateParameters type in MLXAudioCore is used across the examples, so a change there touches every call site. The repository also carries ADDING_A_MODEL.md, CONTRIBUTING.md and Agents.md at the top level, which suggests the maintainers expect model additions to be a routine contribution.

The licence is MIT, stated in the README badge and present as a LICENSE file at the repository root. MIT is permissive, but it covers this SDK's code, not the weights. Each model you load comes from a separate HuggingFace repository with its own terms, and the top-level README does not summarise those terms. Check the licence on the specific model repository you intend to ship before you ship it; this is a factual gap in the documentation, not a legal opinion.

Editorial conclusion

Adopt it if you are building a macOS 14+ or iOS 17+ app and want speech models running through MLX without leaving Swift. Do not adopt it if you need server-side inference, a documented offline weight distribution story, or a stable API surface, since the README installs from branch: main and the newest release, v0.1.3, is dated 2026-07-09. Verify first that the specific model you want has a README under Sources/ and that its fromPretrained identifier resolves on HuggingFace, because the top-level README does not document what happens when it does not.

Frequently asked questions

Official sources

1. Blaizzy/mlx-audio-swift on GitHub
2. Issues
3. License: MIT
4. README
5. Releases