BS-RoFormer: a PyTorch implementation of Band Split RoPE Transformers for music source separation
Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs
At a glance
- What is it?
- lucidrains/BS-RoFormer packages the ByteDance Band Split Roformer architecture, plus the Mel-Band and Flow-Matching variants, as importable PyTorch modules. It is a model definition, not a trained separator, and the README is explicit that weights come from elsewhere.
- Who is it for?
- Adopt BS-RoFormer if you already have a separation training pipeline and need the architecture as a PyTorch module you can modify; the three model classes, the multiscale STFT loss and the stereo and multi-stem support are all in the package. Do not adopt it if you want to separate a song today, because the README ships no checkpoints and the usage example ends at a loss.backward() call with the comment 'after much training'.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 93 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What BS-RoFormer is, and the gap it leaves open
Music source separation means taking a mixed recording and recovering the individual parts: vocals, drums, bass, and whatever else was summed into the master. The Band Split Roformer paper from ByteDance AI Labs attacks this with axial attention applied separately across frequency bands and across time, and the repository description states it beat the previous first place by a large margin. lucidrains/BS-RoFormer is the PyTorch reimplementation of that architecture.
The audience is narrow and specific. This is a package for people who train separation models, not for people who want to remove vocals from a track. The README's own usage example builds a model, draws random tensors, computes a loss, calls backward, and then writes 'after much training' before showing inference. There is no pretrained checkpoint in the install path, no CLI, and no audio file handling. If you want a working separator, the README points you to Roman's replication and open-sourced weights in the Music-Source-Separation-Training repository, and to Kimberley Jensen's Mel-Band Roformer vocal model. Those are separate projects. This one gives you the network.
Band split, axial attention, and why rotary positions matter here
The core mechanism is named in the title. A spectrogram is split into frequency bands rather than fed to the transformer as a flat sequence of bins, and attention is then applied in two passes: one across the frequency axis within each band, and one across time. The README calls this 'axial attention across frequency (hence multi-band) and time'. The practical consequence is that attention cost scales with band count and frame count separately instead of with the full time-frequency grid, which is what makes long audio windows tractable.
Positional encoding is the other design decision the README highlights. The paper authors ran experiments showing rotary positional encoding gave a large improvement over learned absolute positions, and the implementation exposes a flag called use_pope, described in a code comment as 'a successor to rotary embeddings', defaulting to False. That default is worth noticing: the rotary path is what the paper credits, and the alternative is opt-in and unproven by anything in the repository.
The class signature also reveals the shape of the compute budget. BSRoformer takes dim, depth, time_transformer_depth and freq_transformer_depth as separate arguments, so you can allocate transformer layers unevenly between the time and frequency axes. The README's example sets dim to 512 and depth to 12 while leaving both axial depths at 1, which suggests the outer depth carries most of the capacity in the default configuration.
Three model classes and what separates them
The package exposes more than one architecture, and the differences are not cosmetic.
BSRoformer is the original band split model. MelBandRoformer, imported from the same package, implements the follow-up paper and changes how the frequency axis is divided, using mel-scaled bands. The README's Mel-Band example uses a dramatically smaller configuration, dim of 32 and depth of 1, which is a signal that the two classes do not share a sensible default scale. FlowBSRoformer replaces masking with flow matching: instead of predicting a mask over the input spectrogram, it predicts 'the flow between pure noise and the target audio'. The README notes the sampling entry point differs accordingly, calling model.sample(x) rather than model(x).
Stereo training and multiple stems are supported according to the README, and the appreciation section credits contributors for fixing a stereo training bug in Mel-Band Roformer and an issue with multiple stems in the same model. Those fixes are recorded as acknowledgements, not as changelog entries, so if you are tracking a specific defect you will need to read the issue history rather than the release notes.
Getting it running: install, instantiate, train
Installation is a single command from the README:
pip install BS-RoFormer
Note the mixed case and hyphen in the distribution name; the import name is different, using an underscore: from bs_roformer import BSRoformer. The example constructs the model with dim=512, depth=12, time_transformer_depth=1, freq_transformer_depth=1 and use_pope=False, feeds a tensor of shape (2, 352800), passes a target of the same shape, and calls loss.backward(). The 352800 figure is the audio length the example uses, which at 44.1 kHz corresponds to eight seconds.
For the Mel-Band variant the import changes to from bs_roformer import MelBandRoformer, and for flow matching to from bs_roformer import FlowBSRoformer. The flow variant's constructor in the README omits use_pope. Inference after training is model(x) for the masking models and model.sample(x) for the flow model.
The README's Todo list shows the multiscale STFT loss is checked off as done, along with deciding n_fft and reviewing the band split and mask estimation modules. That list is the closest thing to a statement about which parts of the architecture are considered settled. The n_fft item is worth reading twice, because the default STFT window function was separately reported as incorrect by a contributor and credited in the appreciation section, meaning STFT configuration has been a moving target across the project's history.
Where this package stops and your own work begins
The most important limitation is structural, not a bug. This repository is a model definition plus a loss. Everything around it is missing: dataset loading, audio decoding, chunking of long tracks, checkpoint management, evaluation with standard separation metrics, and inference on real files. The README does not describe any of it. You are expected to supply all of it, or to borrow it from the replication the README links to.
A second limitation is that the defaults are not obviously tuned for anything. The band splitting hyperparameter is credited in the appreciation section to two contributors who 'worked out' it, and the Mel-Band example's dim of 32 sits two orders of magnitude away from the original's 512. Treating either configuration as a recommended starting point for your own data would be a guess.
Third, the package is a moving target. Three releases shipped between January and February 2026, and the last push to the default branch is dated June 2026. That cadence is healthy for a research implementation and awkward for anyone pinning behaviour, since a defect fix in band splitting or STFT handling can change what a trained checkpoint expects.
Against a general-purpose separation toolkit
The obvious alternative is a full separation framework, and the README itself names one: ZFTurbo's Music-Source-Separation-Training repository, which the README credits with a successful training run, open-sourced training code and open-sourced weights. The difference in approach is the boundary of the deliverable. That project is oriented around producing and shipping trained models, with the training harness included. BS-RoFormer is oriented around the network itself, with the harness left to you.
If your goal is to separate audio, the framework is the shorter path, because the weights exist there and not here. If your goal is to change the architecture, for example to swap the band splitting strategy, alter the axial depth allocation, or experiment with use_pope, then a framework's training loop is mostly scaffolding you will fight. This package gives you a clean module and a loss function and gets out of the way.
Mel-Band Roformer is a third position worth naming, since the README links Kimberley Jensen's separately trained vocal model. That is a single-stem model for one source, which is a different scope from the multi-stem support described here.
Licence, maintenance, and what an upgrade costs you
The repository is MIT licensed. In practical terms that is permissive: you can use, modify and redistribute the code, including in commercial products, provided the licence notice is preserved. It says nothing about the trained weights, because this repository does not ship any, and the weights hosted in the linked replication repositories carry their own terms. Check those separately before shipping a product built on them. This is a description of the licence text, not legal advice.
Maintenance cost is low if you consume the package as a dependency and high if you track the branch. The release history shows a steady stream of point releases, and the appreciation section records a series of correctness fixes: the default STFT window function, stereo training in Mel-Band Roformer, multiple stems in the same model, and suggested fixes to MelBand Roformer filed as issue 46. Each of those was a behavioural change. If you train a checkpoint against one version and later upgrade, the safest assumption is that the checkpoint may not load or may not perform identically, and you should pin the version in your requirements until you have retrained or revalidated.
The dependency surface is PyTorch plus whatever the package pulls in, and the README credits FlashAttention, XCiT and a quantizable-transformers paper among the influences, so expect a reasonably heavy install. There is no stated minimum Python or torch version in the material available, so verify that yourself before pinning.
Editorial conclusion
Adopt BS-RoFormer if you already have a separation training pipeline and need the architecture as a PyTorch module you can modify; the three model classes, the multiscale STFT loss and the stereo and multi-stem support are all in the package. Do not adopt it if you want to separate a song today, because the README ships no checkpoints and the usage example ends at a loss.backward() call with the comment 'after much training'. Before committing, verify that the band splitting defaults match your sample rate and n_fft, confirm which of BSRoformer, MelBandRoformer or FlowBSRoformer matches the checkpoint you intend to train or fine-tune, and check the ZFTurbo replication for released weights if training from scratch is not the plan.
Community notes