UltraSinger builds an UltraStar file from a song with transcription and pitch models

From a song file to an UltraStar chart

UltraSinger is a tool to automatically create UltraStar.txt, midi and notes from music. It automatically pitches UltraStar files, adding text and tapping to UltraStar files, and creates separate UltraStar karaoke files. It can also re-pitch current UltraStar files and calculates the possible in-game score.

The pipeline is the interesting part, because the output format is a specialised one. UltraStar is a karaoke game format, and an UltraStar.txt file is essentially a word-level lyrics track with timing information, and making one by hand is the tedious part of making a song playable. UltraSinger automates exactly that: the text comes out of a transcription model, the pitch comes out of a pitch detection model, and the tapping is the timing alignment between them.

The project states that multiple AI models are used to extract text from the voice and to determine the pitch. That is the honest summary. The text is a speech recognition result and the pitch is an estimate, and the chart quality is bounded by both.

The project also says something about attribution that is worth repeating verbatim in spirit: please mention UltraSinger in your UltraStar.txt file if you use it, because it helps others find the tool and helps the tool get improved and maintained.

The command line, and the options that are still in progress

The app is driven from the command line, and the documentation prints the full usage shape. The core is UltraSinger.py with an options section, a mode section, a transcription section, a pitcher section and an extra section.

The options are small. -h prints the help text, -i takes the input, described as Ultrastar.txt, audio or video like .mp3, .mp4, .wav, or a youtube link, and -o is the output folder.

The modes are where the incompleteness shows. The default is full automatic mode, which creates everything depending on the command line options, and --interactive asks all the options at runtime for easier configuration. Then there are three single-file selection flags, -u to create the ultrastar txt file, -m to create a midi file and -s to create a sheet file, and each of those three is marked as In Progress in the documentation. The note above them says single file creation selection is in progress and that you currently get all of them.

So the tool as it stands is a one-shot generator. You either take the full automatic output or you answer the questions interactively, and you cannot yet ask for only the txt file and skip the rest.

Transcription with Whisper, and the size knobs

The transcriber is Whisper, and the options are extensive.

The --whisper option selects the model. The multilingual models are tiny, base, small, medium, large-v1, large-v2 and large-v3, with large-v2 as the default. There is a separate set of English-only models, tiny.en, base.en, small.en and medium.en. The --whisper_align_model option uses a model for another language for the Whisper alignment step, provided from huggingface.co.

The --language option overrides the language detected by Whisper. The documentation is precise that it does not affect transcription but steps after transcription, which is the alignment phase, so setting it changes where the words are placed in time rather than what the words are.

Then there are two resource options, and they are the ones that decide whether this runs on your machine. --whisper_batch_size should be reduced if you are low on GPU memory, with a default of 16. --whisper_compute_type should be set to int8 if you are low on GPU memory, which may reduce accuracy, with a default of float16 for cuda devices and int8 for cpu.

And --keep_numbers transcribes numbers as numerics instead of as words, which matters for songs that count.

Separation, hyphenation and the pitch model

Two preprocessing steps sit between the audio and the models.

The first is separation, which splits the vocals from the rest of the track. The --demucs option takes a model name, and the list is htdemucs, htdemucs_ft, htdemucs_6s, hdemucs_mmi, mdx, mdx_extra, mdx_q and mdx_extra_q, with htdemucs as the default. Separation is enabled by default and --disable_separation turns it off. The point of separation is that the transcription and pitch models work better on a voice track than on a mix.

The second is hyphenation, which splits long words into the syllables an UltraStar line needs. It is enabled by default and --disable_hyphenation turns it off. The dependency list confirms the mechanism, with pyhyphen and python-Levenshtein present, so the word splitting uses a hyphenation dictionary and Levenshtein distance for matching.

The pitcher itself is a separate section in the documentation, and the dependency list shows the tools behind it. swift-f0 is the pitch detection library, librosa and scipy do the audio work, pydub handles the audio format handling and ffmpeg-python wraps ffmpeg, and pretty-midi and music21 are there for the midi and sheet outputs.

The dependency list also has yt-dlp, which is how a YouTube link becomes an audio file, and musicbrainzngs for metadata.

Installation, and the version that actually works

The installation instructions are short and the constraints are explicit.

Install Python 3.12, with the warning in bold that older or newer versions may have compatibility issues. Install ffmpeg and make sure it is on the PATH. Then go to the install folder and run the install script for your operating system, choosing GPU if you have an NVIDIA CUDA GPU and CPU if you do not or if you want CPU-only processing.

The manifest matches that instruction exactly, with requires-python set to >=3.12,<3.14. So the supported window is two Python versions, and the project is candid that this is where it works.

Running it is a shell script per platform, run_on_windows.bat, run_on_linux.sh or run_on_mac.command in the root folder, and then the Python entry point with the options described above.

There is also a container route. The Dockerfile is based on the NVIDIA CUDA 12.8.1 runtime on Ubuntu 22.04, installs Python 3.12 from the deadsnakes PPA along with ffmpeg, then installs uv and uses it to install the project as an editable install with the system Python, and then overrides the PyTorch packages with the CUDA wheels from the PyTorch index. The comment explains the ordering, with the pyproject.toml copied first so the image build cache can be reused, the sources copied late so the dependency layers stay cached, and the container not running as root.

Prebuilt packages, and a development warning

The dependency list is heavy, and it is heavy for a reason.

torch, torchaudio and torchvision are pinned to exact versions, torch 2.8.0, torchaudio 2.8.0 and torchvision 0.23.0, which is what keeps a CUDA and a CPU install consistent. On top of that sit whisperx for the transcription and alignment, demucs for separation, librosa and scipy for signal processing, pretty-midi and music21 for the midi and sheet outputs, matplotlib for whatever the score calculation draws, num2words and language_data and langcodes for text handling, unidecode, chardet, tqdm for progress, inputimeout for interactive timeouts, deno, and dataclasses with dataclasses-json for the data structures.

The build system is hatchling with setuptools and setuptools_scm, and the version in the manifest is 0.0.13.dev16. The published releases are v0.0.12 from 2024-12-19, v0.0.11 from 2024-07-06 and v0.0.10 from 2024-04-30, and the last push was on 2026-06-30.

The project status badge says development, and the readme opens with a warning that the project is still under development. So the honest summary is a tool with a working pipeline and a version number that says it is not finished, and the readme also asks you to mention UltraSinger in the files you create and to use it only on Creative Commons licensed songs.

Editorial conclusion

Adopt UltraSinger if you make UltraStar karaoke content and want the transcription, pitch detection and note timing done for you, and you are working on material you are allowed to use. Do not adopt it if you need commercial accuracy on a chart hit, since every stage is a machine estimate and the project itself asks you to use it only on Creative Commons licensed songs. Before you run it, check three things: that the song is licensed for the use you intend, since the documentation is explicit about this, that you have Python 3.12 exactly, because the project states older and newer versions may have compatibility issues, and whether you have a GPU, since the default compute type is float16 for CUDA devices and int8 on CPU, and the model choices are sized for a card.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. rakuri255/UltraSinger on GitHub
4. README
5. Releases