NeuralNote: a local audio-to-MIDI plugin built on MuScriptor

What NeuralNote solves, and for whom

Turning a performance into editable notes has traditionally meant either hand-transcribing or accepting monophonic pitch detection that collapses under chords. NeuralNote targets the first problem by running a learned transcription model inside the DAW as an AudioFX plugin, so the audio never has to be exported to a separate application. The README describes it as a VST3/AU/Standalone plugin that you apply to the track you want to transcribe, with the result appearing as MIDI in a piano roll inside the plugin window.

The audience is narrow but specific. You need a DAW that hosts VST3 or AU, a machine running macOS or Windows, and enough patience to build from source. The v2 line replaces Spotify's Basic Pitch with MuScriptor from Kyutai and Mirelo, a transformer model the README sizes at 103M to 1.4B parameters against fewer than 17K for Basic Pitch. That jump is the reason v2 advertises multi-instrument transcription: the model can be pointed at a full mix rather than one instrument at a time.

How transcription runs inside the plugin

The architecture is split in two. NeuralNote itself is the JUCE plugin and UI. The transcription engine lives in a separate repository, muscriptor.cpp, included as a git submodule at ThirdParty/muscriptor.cpp and linked as a static library. That engine takes 16 kHz mono float32 audio and returns note events, and its public API is documented in its own docs/API.md. If you want to embed transcription in another program, that submodule is the interface to look at, not the plugin.

Weights are not shipped with the source. NeuralNote downloads GGUF files from the DamRsn/muscriptor-gguf repository on Hugging Face, in three sizes called small, medium and large. The Model button in the top bar switches between installed models, starts downloads for the others, and opens the folder where they live. You can also place a file there by hand, but the README is explicit that it must come from the v1/ directory of that HF repo, unchanged and with its original name, for example muscriptor-medium-f16.gguf, and that NeuralNote only picks up files whose name and size match what it downloads. Renaming a quantisation or trimming a file will make it invisible.

Inference is local. The README states that NeuralNote only goes online to download models and to check for updates. GPU acceleration is used when available, through Metal on macOS and Vulkan on Windows, and the README strongly recommends a GPU for the medium and large models. Decoding is streamed: the MIDI fills into the piano roll as it is produced, with a progress indicator and a cancel button, and you can press play before transcription finishes to hear only the part decoded so far. That streaming design is the most interesting choice here, because it makes the tool usable on long takes where a full pass would otherwise be a waiting game.

Installing NeuralNote and making a first transcription

There are no installers or prebuilt binaries for v2. The README states this directly and points at the build-from-source path. You need git, CMake, a C++23 compiler, Python 3 at configure time to fetch the synth's soundfont, and internet access on the first configure, because muscriptor.cpp fetches ggml and roughly 38 MB of soundfont. Only Clang has been tested so far, on macOS and Windows.

Clone with submodules so the engine comes along:

git clone --recurse-submodules https://github.com/DamRsn/NeuralNote
cd NeuralNote

If you already cloned without them, the README gives the repair command:

git submodule update --init --recursive

Then configure and build in Release. Ninja is optional but the README notes it makes builds faster:

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release

The Standalone app and the plugin artefacts land in build/NeuralNote_artefacts/Release/. On Windows, GPU support requires the Vulkan SDK at build time. Without it the project still builds, but transcribes on the CPU only. With it, configure from a Visual Studio developer prompt or set CC, CXX and RC in the environment, because ggml builds its shader generator as a separate project that does not see CMake's compiler settings.

For a first real use, open the Standalone app or load the plugin on a track. Either click record while the track plays, or drop an audio file on the plugin. The README lists .wav, .aiff, .flac, .mp3 and .ogg (vorbis) as supported. Pick the instruments in the left panel, or leave it on Automatic to let the model detect them. The README notes that transcriptions tend to be better when the model is given the correct set of instruments, which is a hint to prefer manual selection when you already know what is in the mix. Press Transcribe, wait for the piano roll to fill, then drag the MIDI onto a MIDI track or save it as a file.

Model sizes, hardware, and where the speed goes

The three model sizes trade speed for quality, and the README publishes approximate real-time factors measured on an Apple M1 Pro, where above 1x means faster than real time. The small model reaches roughly 3.5x on Metal and 2x on CPU. Medium sits at about 1.5x on Metal and 0.7x on CPU. Large is about 0.5x on Metal, which is slower than real time, and the README marks the CPU column as not recommended for it.

Read that table as a constraint rather than a spec sheet. Medium on CPU at 0.7x means a three-minute take takes over four minutes to decode, and large on CPU is not a path the project endorses. The README also says testing so far covers only a few machines and GPUs, and asks users to report failures, wrong results or unexpected slowness with their OS, GPU and model size, either in the NeuralNote issue tracker or in muscriptor.cpp's if the problem is in the engine.

Model storage is per-user rather than per-project. On macOS the folder is ~/Library/NeuralNote/models; on Windows it is %APPDATA%\NeuralNote\models. Deleting a file from that folder removes the model, and the Model button can fetch it again. This is convenient for switching sizes and awkward for teams that want a pinned model checked into a build pipeline, because nothing in the README describes a shared or project-local model path.

Where NeuralNote is the wrong tool

The clearest limitation is packaging. With no installers or prebuilt binaries for v2, anyone who cannot build C++23 with CMake is excluded. That rules out most casual users who found the project through a plugin directory, and it makes the v1.1.0 release from 2025-01-11 the practical option for people who want a binary rather than a build. Linux is not supported in v2; the README lists it as coming soon and as the first roadmap item.

A second limitation is model management. Because NeuralNote only recognises files whose name and size match its downloads, hand-placed weights must be the exact files from the v1/ directory of the HF repo. If you have your own fine-tune, or a quantisation the project does not publish, the plugin will not load it. There is no documented path for custom checkpoints.

A third is the absence of MIDI out. Per the roadmap, MIDI out with per-instrument channel selection is still planned, so the current workflow is drag-and-drop or save-to-file rather than routing notes straight to a synth track. If your session depends on live MIDI routing, this is not the plugin for it yet. Finally, the README's own warning that testing covers only a few machines and GPUs means you should treat unusual hardware as untested rather than supported, and budget time for an issue report if the GPU path misbehaves.

NeuralNote versus cloud transcription services

The obvious alternative is a hosted audio-to-MIDI service, where you upload a file and download MIDI. The difference is not only privacy. A hosted service can run a much larger model on server GPUs and can update it without you rebuilding anything, which is exactly the trade NeuralNote makes in the other direction. NeuralNote keeps audio on the machine and lets you choose between small, medium and large locally, but the ceiling is your own hardware: large runs at roughly 0.5x on an M1 Pro, so a hosted service will often finish sooner on long material.

The second real difference is integration. A hosted service lives in a browser tab; NeuralNote lives on the track as a VST3 or AU plugin, records from the DAW transport, and streams MIDI into its own piano roll while decoding. If your workflow is already inside a DAW and you dislike round-tripping files, the plugin form is the point. If you want the best possible transcription quality regardless of local hardware, a server-side model is the more sensible choice.

A third comparison is the engine itself. Because muscriptor.cpp is a separate repository with a documented API, you can bypass the plugin entirely and call the engine from your own C++ program with 16 kHz mono float32 input. That is a genuinely different adoption path from using the plugin, and it is the one to take if you are building a tool rather than editing a track.

Licence and what upgrading costs you

NeuralNote's code is published under Apache-2.0, and the repository carries a LICENSE file at the top level. The README's licence section also points at the GGUF weights on Hugging Face, which are a separate artefact with their own terms. If you plan to redistribute the plugin or ship it inside a product, read the licence section of the README in full and check the terms attached to the specific model files you intend to bundle, because the code licence and the weights licence are not the same document. This is not legal advice; it is a pointer to where the two licences diverge.

Upgrade cost is currently low in the versioned sense and high in the operational sense. The last push to the repository was on 2026-09-16, and the most recent tagged release is v1.1.0 from 2025-01-11. The v2 work described in the README is not tagged, so tracking v2 means tracking the master branch and rebuilding when it moves. There is no documented rollback path for models either: you can delete a model from the models folder and re-download it, but the README does not describe pinning a specific revision of a GGUF file. For a studio machine that must keep working, that is the cost to weigh against the newer model.

Editorial conclusion

Adopt NeuralNote if you work on macOS or Windows, are comfortable building C++23 with CMake, and want transcription to stay on your machine rather than in a cloud service. Do not adopt it if you need a signed installer, a Linux build, a CLI, or a stable v1-style release: v2 has no prebuilt binaries, and the README states that installers and a CLI are still roadmap items. Before committing, verify three things: that your GPU path works (Metal on macOS, Vulkan on Windows, with the Vulkan SDK present at build time), that the model size you plan to use runs faster than real time on your hardware, and that the model files you place in ~/Library/NeuralNote/models or %APPDATA%\NeuralNote\models match the names and sizes NeuralNote expects.

Frequently asked questions

Official sources

1. DamRsn/NeuralNote on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases