YuE2: Symbolic Planning as an Editable Middle Layer in Music Generation
YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.
At a glance
- What is it?
- YuE2 splits song generation into a melody-and-chord plan and an audio render, which makes the composition inspectable before it becomes sound. The trade-off is a Linux, Python 3.12, 24 GB VRAM environment and a two-stage workflow that assumes you want to read ABC notation.
- Who is it for?
- Adopt YuE2 if you have an NVIDIA GPU with BF16 support and at least 24 GB of VRAM, you are comfortable in a Python 3.12 virtualenv, and your workflow benefits from a score you can read and edit before rendering. Do not adopt it if you want a single end-to-end call from a text prompt to audio, if you cannot run the model locally, or if your team has no way to review ABC notation.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem YuE2 targets: generation you cannot inspect before it becomes audio
Most text-to-music systems hand back a waveform. If the harmony drifts in the second verse, the only remedy is to change the prompt and regenerate, which discards the parts that were already right. YuE2 inserts a representation between the prompt and the render. The README describes the pipeline as writing a melody-and-chord plan first, then realizing that plan as a complete song with vocals and accompaniment. That plan is the product as much as the audio is.
The intended user is not someone who wants a background loop for a video. It is someone who wants to revise a specific bar, or hand a song to a language model agent and ask it to change the key of the bridge. The README's agentic demo follows one track, The Last Train, through 9 steps and 14 versions, moving from Mandarin pop to English jazz with new harmony and a saxophone solo. That workflow only makes sense if intermediate state is addressable, which is what the symbolic plan provides.
The secondary audience is cover work. A transcribed recording becomes a melody score, new lyrics or a new style prompt are applied, and the accompaniment is regenerated around the retained melody. The README recommends cot="melody" with a score that has no chord symbols for this case, so the accompaniment is free to adapt.
One backbone, two prediction modes, and a VAE at the end
The architecture is described as an AR-NAR Mixture-of-Transformers backbone. The autoregressive half predicts the score and semantic music tokens. The non-autoregressive half, using flow matching, generates acoustic latents. A VAE decodes those latents into stereo audio at 48 kHz without quantization, per the quick start notes.
What differs between creating, covering, and editing is not the model but the origin of the score. In creation, YuE2 writes it. In covering, it comes from a transcribed recording via SheetSage2. In editing, it comes from a composition a person or agent has already modified. That is a clean separation: one checkpoint serves all three, and the README states this explicitly.
The Python surface mirrors the stages: plan(), generate_semantic(), synthesize(), decode(). The generation guide is cited for exact-plan reuse and decoder selection, which implies you can re-render from a stored plan without re-planning. The output directory retains the score, semantic tokens, acoustic latents, generation settings, and model identities, so a run is reproducible from its own artifacts rather than from a prompt string alone.
Installing and running the first song
The stated environment is Linux, Python 3.12, and an NVIDIA GPU with BF16 support and 24 GB of VRAM. Model files download from Hugging Face on first use. The README's commands are:
git clone https://github.com/multimodal-art-projection/YuE.git cd YuE python3.12 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install . python examples/generate.py --output outputs/first-song
The result lands at outputs/first-song/audio.flac. The programmatic path is short as well: construct YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda"), load a request from examples/song.json, call the pipeline, and call song.save_artifacts("outputs/my-song"). The example prints song.truncated, which is the flag to check if you are feeding long lyrics.
The cot parameter is the main control. cot="full" generates an editable melody-and-chord plan and is the default for new songs. cot="melody" uses a melody plan with free accompaniment and is recommended for covers. cot="off" generates directly from lyrics and style. The abc parameter accepts your own score in full or melody mode. There is also a prebuilt wheel for yue2-v0.1.6 published under the release tag, which avoids a source install.
Where the approach gets in the way
The 24 GB VRAM floor is a hard constraint, not a suggestion. Anyone on a consumer laptop, an Apple Silicon machine, or a CPU-only server is outside the supported path as documented. The README does not describe a quantized or CPU fallback.
The symbolic layer is also a commitment. If you supply abc, you are expected to produce or review ABC notation. The cover workflow makes this explicit: you transcribe with SheetSage2, then review its melody ABC before generating. That review step is not optional in spirit, since transcription errors propagate into the render. A team without anyone who reads a lead sheet will find cot="off" easier, and at that point the distinguishing feature of the project is no longer doing any work for them.
There is a second environment to manage. SheetSage2 runs separately and loads its MERT2 encoder automatically, according to the README. So a cover pipeline is two installations, not one, and the cover guide is cited for the complete transcription and generation commands rather than reproduced in the README.
Finally, the headline quality number needs reading carefully. The 6.9632 SongBench Avg is described as YuE2 best-of-8, the highest observed mean among all evaluated settings on 192 WildSongBench prompts. Best-of-8 means eight candidates were generated and the best selected. That is a legitimate way to report a ceiling, but it is not the score you get from a single call, and the README does not present a single-sample figure alongside it.
How this differs from end-to-end audio generation
The obvious comparison is a text-to-audio model that maps a prompt straight to a waveform with no intermediate score. The difference is not quality, it is where correction happens. In an end-to-end system, correction means editing the prompt and accepting a wholly new render. In YuE2, correction can mean editing the plan and re-rendering, which is why the README frames melody and chords as explicit controls that a person or an agent can inspect and edit.
That control has a cost. You carry an extra representation, you need tooling to read it, and you inherit whatever errors the planner makes. An end-to-end model has no planner to be wrong. For a one-off jingle where nobody will revise the harmony, the plan is overhead.
The interesting middle ground is the editing loop. Because the score, style, and lyrics are separate inputs, an agent can revise one without regenerating the others. The README's 14-version demo is the case study for this. Whether that loop is faster than prompt iteration depends on how often your revisions are musical rather than atmospheric, and the repository does not offer a measurement of that.
Licence, releases, and what upkeep looks like
The repository is Apache-2.0 and is not archived. That is a permissive licence, but it covers the code, not the weights or the training data. The model files are hosted separately on Hugging Face under m-a-p/YuE2-3B, and the README does not state terms for the checkpoint, the MERT2 encoder, or the SheetSage2 transcription model. If you plan to ship generated audio commercially, the checkpoint terms are the thing to check, and this article cannot tell you what they say. This is not legal advice.
The project also carries a version boundary worth noting. The README opens by pointing to a YuE-v1 branch, where the original code, documentation, and license are preserved. The current main branch is a different system with a different API, so any tutorial or script written against YuE v1 will not transfer. The most recent release is yue2-v0.1.6, dated 2026-09-09, with a prebuilt wheel published under that tag.
Upgrade cost is mostly environmental. The install is a plain pip install . into a Python 3.12 virtualenv, and model files are fetched on first use rather than vendored, so a version bump means re-downloading weights and re-checking that your GPU still meets the BF16 and 24 GB requirement. The staged API means a breaking change in plan() or synthesize() would affect any stored plans, which is why the retained artifacts and the model identities recorded alongside them matter.
Editorial conclusion
Adopt YuE2 if you have an NVIDIA GPU with BF16 support and at least 24 GB of VRAM, you are comfortable in a Python 3.12 virtualenv, and your workflow benefits from a score you can read and edit before rendering. Do not adopt it if you want a single end-to-end call from a text prompt to audio, if you cannot run the model locally, or if your team has no way to review ABC notation. Before committing, verify three things against the repository: that the 3B checkpoint downloads and fits your card, that the wheel archive for yue2-v0.1.6 matches your Python version, and that the benchmark protocol in docs/benchmarks.md is one you accept, since the headline 6.9632 SongBench Avg is a best-of-8 figure rather than a single-sample result.
Community notes