FluidAudio: A Swift SDK for On-Device Audio AI on Apple's Neural Engine

What FluidAudio Solves and Who It Is For

FluidAudio addresses a specific problem: running state-of-the-art audio AI models locally on Apple devices without the memory and power overhead of GPU-based inference. The README states that inference is offloaded to the Apple Neural Engine (ANE), which results in less memory usage and generally faster inference compared to CPU or GPU approaches. This makes it suitable for developers building iOS and macOS apps that need real-time speech recognition, text-to-speech, speaker diarization, or voice activity detection, especially for 'background processing, ambient computing, and always-on workloads.' The target audience is Swift developers who want to integrate these capabilities with a few lines of code, without managing model conversion or dealing with cloud latency. The SDK is also positioned for apps that prioritize privacy, since everything runs on-device.

The Mechanism: CoreML and ANE Offloading

The core mechanism is the use of CoreML models optimized for the Apple Neural Engine. The README explicitly states that the SDK avoids GPU/MPS entirely, which is a deliberate design choice. Instead, models are converted and optimized by the FluidInference team and published on HuggingFace. The SDK then loads these models and runs inference on the ANE, which is designed for low-power, high-throughput neural network operations. This approach allows for background processing, meaning audio AI can run while the app is not in the foreground, which is critical for always-on voice assistants or meeting recorders. The documentation mentions that models are optimized for background processing, minimizing CPU usage. The data flow is straightforward: audio input goes to the model, and the model output is returned to the app. For streaming tasks like speech-to-text, the SDK supports end-of-utterance detection, as seen in the Parakeet EOU model, which enables true real-time transcription.

Supported Models and Capabilities

FluidAudio bundles a catalog of open-source models. For automatic speech recognition (ASR), it offers Parakeet TDT v3 (0.6b) for batch transcription supporting 25 European languages and Japanese, plus SenseVoice and Paraformer for Mandarin Chinese. For streaming, Parakeet EOU (120m) provides English-only recognition with end-of-utterance detection. The SDK also includes inverse text normalization (ITN) to convert spoken forms like 'two hundred' into written '200'. For text-to-speech, Kokoro (82m) supports parallel synthesis with SSML and pronunciation control across nine languages (EN, ES, FR, HI, IT, JA, PT, ZH), and PocketTTS supports streaming TTS with voice cloning in six languages. Speaker diarization is available in both online (streaming) and offline (batch) modes, and the SDK can extract speaker embeddings for voice comparison. Voice activity detection uses Silero models. All models are publicly available on HuggingFace, and the README points to a full model catalog in Documentation/Models.md.

Getting Started: Commands and Configuration

The README does not provide explicit installation commands, but it links to the documentation at docs.fluidinference.com/introduction and to the model catalog. Based on the repository layout, integration likely involves adding FluidAudio as a Swift package dependency, either via Xcode's built-in package manager or by specifying a URL in your Package.swift file. The README states that models can be integrated with 'just a few lines of code,' but the exact API calls are not shown in the provided material. For model conversion, the README mentions 'möbius', a separate tool from FluidInference, which you can use to convert your own models to CoreML format. The documentation is the primary source for exact setup steps, but the key configuration points are the model selection and the ANE usage. Since the SDK is Swift, you would need an Apple development environment with Xcode and a target device or simulator that supports ANE, which includes modern iPhones, iPads, and Macs with Apple silicon.

Limitations and Failure Modes

The most obvious limitation is platform lock-in: FluidAudio is Swift-only and targets Apple devices. There is no mention of Android, Windows, or Linux support. This means it is the wrong tool for cross-platform apps or server-side processing. Another limitation is language coverage. While ASR supports many European languages and Japanese, streaming ASR is English-only. TTS languages are limited to nine, and voice cloning is only available in six languages. If your use case requires streaming transcription in, say, German, you would need to use batch transcription or wait for updates. The README also notes that models are optimized for background processing, but this depends on ANE availability; older devices may not have the same performance. Additionally, the SDK relies on the ANE, and if a model is not fully optimized for ANE, inference might fall back to CPU, which could increase latency and memory usage. The documentation does not specify how to handle such fallback scenarios. Finally, the README mentions 'low RAM' as a benefit, but actual memory usage depends on the model size; Parakeet TDT v3 (0.6b) is not small, and loading it may still use significant memory.

Alternatives and How They Differ

A common alternative for on-device speech recognition is Apple's own Speech framework, which provides built-in speech-to-text on iOS and macOS. The key difference is that Apple's Speech framework is limited to recognition and does not offer speaker diarization, TTS, or VAD in the same integrated package. It also runs on Apple's servers for some languages, which conflicts with the fully local approach. Another alternative is to use Whisper via a CoreML conversion, such as the whisper.cpp project or Apple's CoreML Whisper examples. Whisper offers broader language support and is open-source, but it is not optimized for the ANE in the same way and often runs on CPU or GPU, which can be slower and more power-hungry. FluidAudio's advantage is its pre-converted models and a unified API for multiple audio AI tasks, whereas alternatives require more manual integration and model management. For speaker diarization specifically, pyannote is a popular Python library, but it is not Swift-native and would require a bridge or separate process. FluidAudio's approach is to provide a single Swift API that handles all these tasks, which is a significant convenience for Apple developers.

Maintenance, Upgrade Cost, and License

The repository is actively maintained, with the latest release v0.15.6 pushed on 2026-08-19, and previous releases in July and June. This suggests a regular release cadence, which is good for bug fixes and new models. However, each release may introduce breaking changes, and you will need to update your code accordingly. The SDK depends on models hosted on HuggingFace, so model updates are separate from the SDK releases. You will need to track model versions to ensure compatibility. The project is licensed under Apache-2.0, which is permissive for commercial use, but you must include the license notice and note any changes. The models themselves have MIT or Apache 2.0 licenses, as stated in the README, but you should verify each model's license on HuggingFace, as some may have restrictions. The README also mentions a Discord for support, which is a good resource for troubleshooting, but it is not a formal support channel. Overall, the maintenance cost is moderate: you need to keep the SDK updated and monitor model changes.

Editorial conclusion

Adopt FluidAudio if you are building a Swift app for Apple platforms that needs fully local, low-latency audio AI, especially speech recognition, diarization, or TTS, and you want to avoid GPU/MPS usage in favor of the ANE. Do not adopt it if you need cross-platform support, non-Apple hardware, or models outside the provided catalog. Before committing, verify that your target models (e.g., Parakeet, Kokoro, Silero) are available for your languages and that the SDK's background processing works on your specific iOS/macOS versions. Check the Apache-2.0 license and the model licenses on HuggingFace, and test the SDK's latency on your actual devices, as ANE performance varies by chip.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes