QuarkAudio, Alibaba's unified audio model, and the tasks it does and does not cover

One model family for speech restoration, separation and voice conversion

QuarkAudio is the name of the project published from the unified-audio repository, and the stated goal is to explore a unified framework that handles different audio processing and generation tasks. The tasks in the README table are not side projects around the model. They are the model. The same prompt-free interface covers speech restoration, target speaker extraction, speech separation, voice conversion, language-queried audio source separation, audio tokenization, and audio editing.

That design choice is the interesting part for an engineer. A classic audio pipeline is a folder of unrelated models: one for denoising, one for separation, one for voice conversion, each with its own preprocessing, its own checkpoint and its own way of expressing what it wants. The QuarkAudio approach collapses that into a single decoder-only autoregressive language model that predicts speech tokens, so a task is expressed as an audio prompt and a reference rather than as a switch between systems.

For the target audience this is a research framework for reproducible work in audio, covering speech, music and general audio events. If you are shipping a product, the absence of a stable release and the presence of a research dependency list are the first two things to weigh, and the README says plainly that this collection of works exists to help reproducible research in the field of audio.

The WavLM, H-Codec and language model pipeline in one chain

The architecture description in the README is short but load-bearing. QuarkAudio uses a decoder-only autoregressive language model backbone, which brings LLM style autoregressive generation to speech token prediction, and the pipeline integrates a feature extractor, a discrete codec and the language model end to end. The feature extractor is WavLM or Hubert, the discrete codec is H-Codec, and both sit in front of the language model in a single chain.

The codec is the module the project treats as fundamental, and the README says so directly: the project also provides works involving neural audio codec, which it calls the fundamental module to combine the audio modality with language models. That framing explains the tree layout. The QuarkAudio-HCodec directory holds the codec work, with pretrained models and inference examples released alongside it, and a separate QuarkAudio-HCodec/HCodec-spatial subdirectory holds the spatial audio codec announced in the May 2026 news entry, which supports stereo encoding and mono-to-stereo spatialization.

So the data flow is worth naming precisely. Continuous audio enters through the WavLM or Hubert feature extractor, that signal becomes discrete tokens in the H-Codec representation, and the language model then generates those tokens autoregressively, with decoding returning waveform. Tasks such as separation, editing and voice conversion differ in the conditioning and in the token stream, not in the presence of separate models. A codec-based approach also explains the interest in combining audio with language models, since discrete tokens are something an autoregressive model can be trained on the same way text is.

The supported task table and the two entries that are not ready

The README marks each task with a status, and the distinction is the part worth reading twice. Speech restoration is supported, and it recovers clean speech from corrupted inputs such as noise, reverb and packet loss. Target speaker extraction is supported, using reference enrollment audio to pick one speaker out. Speech separation is supported, for mixed speakers or sound sources. Voice conversion is supported, changing speaker identity while preserving linguistic content. Language-queried audio source separation is supported, so you can ask in plain language to remove the man's voice. Audio tokenization is supported, encoding speech into compact discrete tokens and reconstructing high fidelity audio by decoding. Audio editing is supported, inserting, deleting or substituting words and phrases in the audio domain.

Text to audio and acoustic echo cancellation carry a different marker: both are listed as developing, with the note that they are upcoming in the next release. Echo cancellation is described as removing echo artifacts in teleconferencing scenarios. This matters more than a status column usually would, because text to audio is the task most people imagine first when they hear about a unified audio model, and it is the one the project says it does not have yet.

A third supported row, CODEC, and the AE row for audio editing are worth separating from the rest when you plan a rollout, since tokenization and editing are infrastructure tasks for a pipeline rather than endpoints a user would call directly.

What the repository actually contains and where the models live

The top level of the repository is short, and that shape tells you how to navigate it. It holds LICENSE, README.md, index.html, an environment.yml, a requirements.txt, an image, and three work directories: QuarkAudio-HCodec, QuarkAudio-UniSE and UniTok-audio. The news entries in the README give each of those a history. QuarkAudio-HCodec was released in December 2025, with pretrained models and inference examples. QuarkAudio-UniSE appeared in September 2025 as a foundation model for unified speech generation supporting target speaker extraction and universal speech enhancement, and the July 2026 entry says the UniSE V2.0 code will be updated there, following acceptance of two papers at INTERSPEECH 2026. UniTok-audio was announced in October 2025 for tokenization, restoration, voice conversion, language-queried separation and speech restoration, with the note that code will come soon.

Model weights are not hosted in the repository. The README links to Hugging Face Spaces under the QuarkAudio organisation and to a ModelScope organisation, and a demo page is linked as well, so the intended flow is to pull code from GitHub and weights from those model hubs. The January 2026 entry also notes a hybrid discriminative and generative system for universal speech enhancement built on the same architecture that took third place in the URGENT 2026 Challenge, Track 1, with the paper linked on arXiv.

One documentation gap to note: the license section in the README still carries a comment that reads as an unfinished check before the Apache 2.0 statement.

The pinned Python dependency set is the real installation cost

There is no install command in the README, and no package to import from an index. What the repository does provide is a conda style environment.yml and a requirements.txt with a long list of pinned versions. A representative slice of that file shows the shape of the environment: it mixes audio codecs and analysis packages such as descript-audio-codec, encodec, vector-quantize-pytorch, funasr, espnet-model-zoo, s3prl and speech measurement tools such as jiwer, pysndfx, torch-pesq and visqol, with training utilities like peft, torchtune, torchdiffeq and x-transformers, and infrastructure helpers like hf-transfer and hf-xet.

A short excerpt, copied from that file, shows the pinning style you inherit:

asteroid==0.7.0
audioldm-eval==0.0.1
descript-audio-codec==1.0.0
encodec==0.1.1
espnet-model-zoo==0.1.7
funasr==1.2.9

Those exact pins are the closest thing the project gives to setup instructions, and they tell you the environment expects a recent PyTorch stack with GPU oriented extras such as onnxruntime-gpu and gpustat in the list. Because the versions are pinned rather than ranged, merging this file into an existing environment will usually conflict, and the file also includes packages that pull in their own native dependencies, such as sox and gmpy2. The README does not document a supported Python version or a CUDA version, and it does not describe a container image, so those are questions for the maintainers rather than something to read off the file.

The practical order of work is therefore: create an environment, install the pinned file, fetch the model weights from the hub, then run the inference examples in the subdirectory for the model you need. The last step is where the per-subproject README becomes essential, because the top level README does not describe entry points.

Where the unified approach is the wrong tool

The first wrong case is a requirement that includes text to audio or echo cancellation. Both are listed as developing and due in the next release, and a project with no GitHub releases has no version you can pin to guarantee they have arrived. A product plan that depends on them needs a different plan.

The second wrong case is a strict latency or footprint budget. A single checkpoint that loads an audio encoder, a codec and an autoregressive language model is not the shape of a fast streaming denoiser, and the unified design trades specialisation for coverage. If your workload is high volume cleanup of fixed-type corruption, a task specific restoration model is the more natural shape, and the README itself points at such a system existing in the challenge entry, where a hybrid discriminative and generative setup was used for universal speech enhancement and placed third in the URGENT 2026 Challenge.

The third wrong case is a team that needs a versioned, packaged dependency. There are no GitHub releases, the repository is a collection of subdirectories with their own histories rather than a single installable library, and the code for UniTok-audio was still flagged as coming soon when that project was announced in October 2025.

The fourth wrong case is a general audio library need. This is a model project, not an audio processing toolkit, and treating it as a drop-in library for decoding or resampling will not work.

A unified model against a collection of task specific models

The alternative is not a product name but an architecture of solution: keep a separate model per task, each a specialised network with its own checkpoint, and switch between them in a pipeline. The difference in approach is real and it is not only about elegance.

A task specific setup is easier to reason about per problem. A restoration model tuned for noisy recordings can be small, fast and evaluated against a clean reference for that corruption type, and it can be swapped out without touching the rest of the pipeline. It also degrades gracefully, since a failure in one stage does not affect the others.

The unified setup wins on a different axis. Because the same model handles enhancement, separation, voice conversion and editing, adding a capability does not mean adding another deployment, and a natural language request such as removing one voice becomes an input to the same interface as everything else. The cost is that every task runs through the same pipeline, so the model is only as fast as its heaviest configuration and the per-task accuracy story is less visible in the interface.

QuarkAudio also positions the codec as the shared foundation, and that is a reusable idea beyond the unified model itself. The H-Codec and HCodec-Spatial work is useful on its own if you need discrete audio tokens for any other system, and the neural audio codec is described as the module that joins audio to language models in the first place.

Editorial conclusion

QuarkAudio suits researchers who want one checkpoint across restoration, separation, voice conversion, tokenization and audio editing, and who can accept a pinned research dependency tree and the fact that this is a collection of subprojects rather than a packaged library. It does not suit production work that needs text to audio or echo cancellation today, since the README marks both as developing for a future release, nor teams that want a stable versioned API, because the repository has no GitHub releases. Before adopting it, confirm that the task you need appears in the supported table, check that the QuarkAudio-HCodec or QuarkAudio-UniSE subdirectory holds the model you actually want, and read the pinned entries in requirements.txt to see whether your CUDA and Python setup can satisfy them.

Frequently asked questions

Official sources

1. alibaba/unified-audio on GitHub
2. Issues
3. License: Apache-2.0
4. README