LookOnceToHear is the research code behind looking at a speaker to hear them

Look at a speaker for a few seconds, hear only them

The repository provides code for the paper Look Once to Hear: Target Speech Hearing with Noisy Examples, and the README describes the system in one sentence. Look Once to Hear is an intelligent hearable system where users choose to hear a target speaker by just looking at them for a few seconds.

The paper won best paper honorable mention at CHI 2024, and the arXiv entry is 2405.06289, with a Gradio demo linked from the same page. The primary language of the repository is Python.

Two design choices in that one-sentence description explain the rest of the repository. It is hearable, which means the output is what arrives at your ears, so the pipeline is binaural rather than a single channel of separated audio, and the target is chosen by gaze rather than by a button, which is where the interaction research sits. The code is published so the method can be reproduced and built on, and the repository is organised as a training pipeline rather than a package.

Setup is a conda environment on Python 3.9

The setup section is three commands:

    conda create -n ts-hear python=3.9
    conda activate ts-hear
    pip install -r requirements.txt

The Python 3.9 pin is worth noticing, because the dependency list is not a light one. `requirements.txt` names numpy, matplotlib, soundfile, ipykernel, pandas, python-sofa, torch, torchaudio, scipy, tqdm, scaper, speechbrain, lightning, wandb, asteroid, pyyaml, GitPython, resemblyzer, espnet, cdpam, auraloss, seaborn and ptflops.

Read that list as a map of what the project does. Torch and torchaudio are the base, Lightning is the training loop, speechbrain and espnet supply speech models and recipes, asteroid and auraloss are separation models and audio losses, resemblyzer is speaker verification, which is how a target speaker is identified, python-sofa handles the spatial audio file formats, wandb tracks experiments, and ptflops counts the operations of a model.

The practical consequence is an environment with several deep learning frameworks in it at once, which is where version friction tends to appear. The name of the environment, `ts-hear`, is for target speech hearing.

Training audio is synthesised on the fly

The data preparation section explains what the model actually learns from, and the answer is that the mixtures are generated rather than collected.

Training data includes clean speech, background sounds, head-related transfer functions and binaural room impulse responses. The project uses the Scaper toolkit to synthetically generate the audio mixtures, and each mixture is generated on the fly during training or evaluation, using Scaper's `generate_from_jams` function on a `.jams` specification file.

Generating on the fly rather than shipping a fixed dataset has a clear benefit for a paper: the training signal is effectively unlimited and the mixture conditions are controlled by the specification files, so an ablation changes a `.jams` file rather than a data download. The cost is that the specifications are now the dataset, and they are the part of the project you must read carefully.

Self-contained datasets are provided through a Google Drive link, together with the source `.jams` specifications used for training. For a training run, the README says it is sufficient to download `MixLibriSpeech.tar` from there, extract the contents into the `data/` directory, and then run the training command. To rebuild all datasets from scratch instead, the repository points to `data/README.md`.

One command runs a training job, and the same one resumes it

The training entry point is a module, and the configuration is a file in `configs/`:

    python -m src.trainer --config <configs/tsh.json> --run_dir <runs/tsh> [--frac <0.05 (% train/val batches)>]

The `run_dir` argument is where the run's outputs go, and the optional `--frac` limits the fraction of the training and validation batches that are used, with the README noting 0.05 as the example.

Resuming a partial run uses the same command, without the fraction flag:

    python -m src.trainer --config <configs/tsh.json> --run_dir <runs/tsh>

That symmetry is the useful part. Because a run is identified by its config and its run directory, restarting after an interruption is a matter of repeating the command, and the repository also ships `slurm.py`, which points at the kind of cluster environment this kind of job is expected to run in.

Evaluation uses the same mixture format as training

The evaluation section is brief and states the assumption it makes: evaluation is done on speech mixture in similar format as training samples.

    python -m src.ts_hear_test

One module, one command. The reason it can be that short is that the mixture generation is shared with training, so the evaluation inputs are produced the same way and there is no separate data preparation step to remember.

The repository layout is small and follows from the three phases. `src/` holds the trainer and the test module, `configs/` holds the configuration file the trainer takes, `data/` holds the extracted dataset and the README for rebuilding it, and there is a `requirements.txt` at the root alongside the `slurm.py` script. There is no packaged API and no inference entry point described in the README, which is the clearest signal about what kind of artefact this is.

What the project is and what it is not

It helps to be clear about the shape of this repository, because the README's brevity can suggest a finished tool and it is not one.

This is a paper's code, published so the method can be reproduced and extended. That means the pipeline assumes you have a machine with a GPU and the patience for a training run, a dependency set that pulls in several deep learning frameworks, and a data path that either downloads a prepared archive or rebuilds the datasets from the `.jams` specifications.

It also means the evaluation numbers in the paper can be reproduced, and the method can be modified, because the configuration is a JSON file and the data conditions are specification files rather than baked in tensors. That is the value of publishing it.

The alternative for a team that needs speech separation in a product is to use an existing pretrained separation model rather than training one, which trades away the specific interaction the paper contributes, the choice of a target by looking, for a system you can ship. The paper's contribution is precisely the part a general separation model does not have.

The honest caveat for anyone planning to build on this is that the README documents the pipeline, not the interface. There is no described API for taking live audio and a gaze signal and returning a binaural stream, so the engineering work of turning the method into a running system is still ahead of you, and the CHI 2024 honorable mention is evidence about the research, not about production readiness.

Editorial conclusion

Adopt this repository when you are reproducing the Look Once to Hear paper, extending the method, or learning how a hearable target speech system is trained and evaluated, and when you have GPUs and time to run the training. Do not adopt it as a speech separation library for a product, since it is a research pipeline with no inference API described in the README. Verify first that the data preparation path works, since rebuilding all datasets from scratch follows a separate document, that the released `MixLibriSpeech.tar` matches the `.jams` specifications you use, that your environment can hold the dependency set listed in requirements.txt, and that the licence in the repository's LICENSE file covers the use you intend.

Frequently asked questions

Official sources

1. Issues
2. README
3. vb000/LookOnceToHear on GitHub