jarvis-mlx: an offline voice assistant stack for Apple Silicon

What the project is, and the warning that comes with it

The repository description calls it an all-in-one solution to stark-level productivity running offline on your MacBook using SOTA technology, and the README expands that to MLX, Apple's new machine learning framework optimized for Apple Silicon.

But the first thing in the README is a warning, and it should be read before anything else. The project is very out of date, and while it offers a great place to start, it does not reflect current progress. The README then points at an Instagram account for updates.

That framing changes what the repository is. It is a reference implementation, the wiring of a familiar stack into a working shape, rather than a product with a release cadence. The repository publishes no GitHub releases, the last push to the repository was on 2026-05-11, and the top level of the tree is small enough to read in one sitting: a README, a .gitignore, main.py, a requirements.txt, a beep.mp3 audio file, and three directories, melo for the text to speech work, stt for the speech to text work, and test.

There is also no licence file in the tree, and the platform records the licence as unknown. That is worth pausing over, because the code is short enough that reading it is the obvious alternative to using it, and reading it is not restricted by a licence in the way that copying it would be.

So the audience is a developer who wants to see a local voice assistant assembled end to end, and who is prepared to rebuild rather than install. The three component projects it stands on are the durable parts, and each of them is maintained by somebody else.

The technology claim worth noting is the offline part. Nothing in the requirements list is an API client, and the language model runs through MLX on the machine's own accelerator, so the assistant works with the network off.

Whisper, Phi 3 and MeloTTS: three models in a loop

The stack has three parts, and each is a well known open project rather than something invented here.

Speech to text uses Whisper, described as SOTA speech recognition opensourced by OpenAI and trained on 1.5k hours of audio. The README's point about it is the size, calling it a tiny model that runs on your MacBook, and notes that it is upgradable to larger models in the series for better performance. The linked reference is the Whisper implementation in the MLX examples repository, so the model runs on the Apple accelerator rather than through PyTorch.

The language model is Phi 3 out of the box, and the README claims you can achieve 60 tokens per second on an M1 Max with it. That is the author's figure on his own hardware, not a benchmark you can rely on, but it is the right order of magnitude for a small model on that silicon. The library is mlx-lm, the language model binding for MLX. The README also says you can finetune your own models and load them in, and that it would be highly recommended to use Mistral or Llama 3, and then adds that finetuning Phi 3 will also achieve much better behaviour.

Text to speech uses MeloTTS, with the reasoning given in the README: it is not as hyped up as some other offerings, but it is fast, it runs on a Mac, and it is finetunable on custom data using the original repo. A stripped down version of the English inference code is vendored into the melo directory, which is what makes the whole thing self contained rather than a wrapper around a pip package.

That vendor point matters more than it looks. Vendoring the inference code means the audio path can be read and changed without tracking an upstream release cycle, which for a project that is explicitly out of date is a deliberate choice to be immune to that problem.

The beep.mp3 file at the root is the smallest possible sample of the system's output, and it is the kind of artifact that tells you a project has actually been run.

The install path, and the arm64 constraint that comes first

The installation section is short and the order matters, because everything depends on the first step.

Native Python is required to run the MLX libraries, and the README gives the check:

python -c "import platform; print(platform.processor())"

It should say arm. If it says x86_64, you are running Python under Rosetta or on an Intel Mac, and the MLX libraries will not work as intended.

The next step creates a conda environment with numpy for arm64, named native:

CONDA_SUBDIR=osx-arm64 conda create -n native numpy -c conda-forge

The CONDA_SUBDIR override is the interesting part of that command. A conda environment defaults to the platform of the machine that creates it, and a mixed architecture setup needs to be told explicitly which subdirectory to resolve packages from. The README's comment says it will create a conda env with numpy for arm64 called native.

Then the requirements are installed:

pip install -r requirements.txt

The three steps together are a careful sequence. The architecture check is free and catches the most common failure, the environment creation fixes the package resolution problem before it starts, and only then do you install the dependencies.

The README does not say which environment to activate for the pip step, which is a small gap. The natural reading is that the native environment is the one to use, since the MLX wheels are arm64 builds, and the whole point of creating it with numpy in it is to have a working arm64 environment to install into.

What the dependency list says about the design

The requirements.txt file is the most informative document in the repository, because it is the actual composition of the system rather than a description of it.

The MLX and model group pulls in mlx at 0.11 or later, mlx-lm, numba, numpy, torch, tqdm, more-itertools, a pinned tiktoken at 0.3.3, huggingface_hub and scipy.

Two of those deserve comment. The presence of torch alongside MLX is unusual, and it comes from the model ecosystem rather than from this project's own design: the upstream tooling for finetuning and for some of the pipeline pieces expects PyTorch, so it is there. And the pinned tiktoken at exactly 0.3.3 is a hard pin on a package that has since moved on, which is a sign of a dependency set frozen at a point in time rather than maintained.

The MeloTTS group is g2p_en, which is the grapheme to phoneme conversion needed to turn English text into phonemes, and cached_path for fetching model assets. The speech group is pyaudio, playsound, webrtcvad, sounddevice, librosa and torchaudio, and the vad library is the one doing real work: voice activity detection is what lets the system know you have stopped speaking, and it is the difference between a push to talk system and one that feels like a conversation. librosa and torchaudio handle the audio decoding and resampling, which is where a microphone at 44.1 kHz and a model expecting 16 kHz meet.

The miscellaneous group is pytest, for the test directory, and pydantic, which is the only hint about the shape of the code itself. A pydantic model is likely how configuration and the data flowing between the three stages is validated, which is a sensible choice for a pipeline where each stage is separately replaceable and a malformed intermediate should fail loudly rather than turn into noise in the synthesiser.

It is a heavy dependency set for a small program, most of it inherited from the model projects rather than chosen here.

The voice is a starting point, and the README says so

There is a note in the text to speech section that is easy to skim and important to read. Out of the box, jarvis-mlx has a female voice, and you will need to train your own MeloTTS model to change this. Finetuning Phi 3 will also achieve much better behaviour.

Two separate limitations are stated in one paragraph. The first is the voice, and it is a hard one for anyone deploying this: the default speaker is fixed, and the only documented way to change it is to train a model. That is a research task with a GPU and a dataset, not a configuration change, which makes the default voice a decision point rather than a detail.

The second is the model's behaviour, and the recommendation is to fine-tune Phi 3 rather than use it as shipped. The README has already said elsewhere that it would be highly recommend Mistral or Llama 3, and that you can finetune your own models and load them in. So the intended path is: take one of the open base models, fine-tune it, point the system at your checkpoint, and accept that the out of the box configuration is a demonstration.

That is a coherent position for a reference implementation, and it is worth saying plainly: the value of this repository is the plumbing between the three models, not the models themselves or the voice.

The test directory and the pytest dependency suggest the assembly logic is at least partly covered, and the repository ends with an invitation for pull requests, which is a reasonable thing to offer for a project whose author has flagged it as out of date.

What a fork inherits, and what it does not

Anyone planning to build on this repository is effectively forking it, so it is worth being clear about what that means.

What you inherit is a working assembly of three specific upstream projects at specific versions, with the English MeloTTS inference code vendored into the melo directory. That vendored copy is a liability as well as an asset. It does not get the upstream fixes, and it does not get the upstream model improvements, and since the README itself says the project does not reflect current progress, the copy is already behind.

What you do not inherit is any kind of service, interface or deployment story. There is no web front end, no microphone permission handling in the interface layer, no session or conversation state, and no way to trigger the assistant other than by running main.py. The beep.mp3 file in the root is the extent of the user-facing output.

The licence position is the other thing to settle before you copy anything. There is no licence file in the tree and the platform classifies it as unknown, so the default position under copyright applies, which means no permission to redistribute the code in any form. For a repository whose own README suggests using it as a starting point, that is a gap rather than a permission, and it is a gap the upstream projects themselves do not have, since Whisper, the MLX examples, Phi 3 and MeloTTS each carry their own terms.

The realistic adoption path is therefore to read it, then rebuild against the current upstream releases rather than copying the vendored inference code, and to write the microphone loop, the conversation state and the interface yourself. On that reading the repository is worth exactly what it says it is, a great place to start.

Editorial conclusion

Adopt jarvis-mlx if you want to see how a fully offline voice assistant is assembled on Apple Silicon, particularly if you are building a pipeline that mixes speech recognition, a local language model and speech synthesis without any cloud call. Do not adopt it as a product, because the README states plainly that the project is very out of date and does not reflect current progress, and the pinned tiktoken version and the several unmaintained-sounding dependencies are the visible symptoms. Verify first that you are on Apple Silicon and can create an arm64 environment, since the whole stack depends on MLX, that the MeloTTS voice is acceptable to you, because the default is female and changing it means training your own model, and that the model licences and the dependency versions in requirements.txt still fit your plans, since the file pins an old tiktoken and pulls in torch alongside MLX.

Frequently asked questions

Official sources

1. huwprosser/jarvis-mlx on GitHub
2. Issues
3. README