# Realtime_PyAudio_FFT: a localhost server that turns live sound into OSC features

> Realtime_PyAudio_FFT captures a local audio device, computes band energies, an optional FFT spectrum, onset triggers and a BPM estimate, and publishes them over OSC and WebSocket. The judgement here is that the value is in the server side signal path and its tuning controls, not in the feature list itself.

**aiXander/Realtime_PyAudio_FFT** — Realtime audio analysis in Python to extract audio features from streaming audio and send them over OSC to any client app.

- Repository: https://github.com/aiXander/Realtime_PyAudio_FFT
- Stars: 1,087 · Forks: 205
- Language: Python
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/aixander-realtime-pyaudio-fft

## A localhost server that turns live sound into OSC features

Realtime_PyAudio_FFT is a Python program that sits between a capture device and whatever you want to react to sound. It reads live audio from a microphone, a line-in, a soundcard or a loopback device, and it publishes a small set of derived values instead of the raw samples. The target audience is visual and interactive work: VJ tools, game engines, creative coding sketches and custom scripts.

The interesting decision is where the analysis lives. The project describes itself as a feature provider for other projects, and it offers two transports that can be used together. OSC over UDP is one way out to clients such as TouchDesigner, Max/MSP, Unity and p5.js. WebSocket is the way back in, carrying runtime control messages and richer payloads, and it also feeds the bundled browser UI.

That split means the browser tab is not doing the work. The project documentation states that all signal processing, including smoothing, peak normalization, gating, tanh compression, strength blending and spatial peak smearing, happens server side in the same code path that feeds OSC, and that what appears in the FFT graph is byte-identical to the `/audio/fft` OSC payload. Tuning in the UI therefore changes what downstream applications actually receive.

## The PortAudio callback stays empty while worker threads do the DSP

The realtime safety argument is structural. The project documentation states that the PortAudio callback is allocation-free and runs zero DSP, and that every filter, FFT and smoother runs in worker threads using vectorized NumPy and SciPy paths that drop into optimized C. The stated end-to-end input to OSC latency target is 8 to 15 ms, and the project claims the server runs on a Raspberry Pi without trouble. Those are the project's own numbers, not measurements, and no benchmark procedure is described.

Audio blocks are small and frequent. The documentation gives a rate of roughly 187 Hz at 48 kHz with 256 samples, and that figure is repeated for every per-block message. The dependencies listed in `pyproject.toml` are `numpy>=2.0`, `scipy>=1.11`, `sounddevice>=0.4.6`, `python-osc>=1.8`, `websockets>=14,<17` and `pyyaml>=6.0`, with the project requiring Python 3.10 or newer. So the capture path is `sounddevice` on top of PortAudio, the transform path is SciPy and NumPy, and the transport path is `python-osc` plus the `websockets` library.

The design consequence is worth stating plainly. If you are chasing a hard deadline for visual response, the interesting question is not only how fast the DSP runs, but where the buffering sits between the capture thread, the worker threads and the UDP send.

## Band energies, a 128-bin spectrum and rising-edge onsets per band

Three signals carry most of the weight. Low, mid and high band energies come from three independent IIR bandpasses, each cleaned and auto-scaled into a clean `[0, 1]` signal. The scaling is a peak follower plus a soft noise gate plus a tanh step, and `/audio/lmh` sends the three values on every audio block.

An optional 128-bin log-spaced FFT spectrum is windowed with rFFT and routed through the same auto-scaler pipeline. It is off by default and switched on either from the browser UI or by setting `send_fft` in the configuration file.

The onset detectors are the part with the most personality. Three independent detectors run on the post-processed low, mid and high signals rather than on a parallel copy of the raw audio, so any UI knob that shapes a band's visible track also tunes that band's detector. Each detector has its own `sensitivity`, `refractory_s` and `slow_tau_s`, and the documentation gives default refractory values of 250, 180 and 120 ms for low, mid and high, which works out to roughly 240, 333 and 500 onsets per second as ceilings. Events go out as `/audio/onset/low`, `/audio/onset/mid` and `/audio/onset/high` with a single integer argument, and only on the rising edge, so no zeroes are transmitted. A `/audio/bpm` message carries a slowly smoothed estimate derived from the low onset stream, folded into the `[60, 180]` range.

## OSC out and WebSocket in travel on two different paths

The two transports have genuinely different jobs, and the project frames the choice as lowest latency against runtime control. OSC over UDP is described as the lowest latency path, delivering every audio block to every destination listed under `osc.destinations` in `configs/main.yaml`, with the default being `127.0.0.1:9000`. WebSocket is full duplex: the browser UI and any other client can toggle the FFT, change band crossovers, switch the input device, change smoothing and save or load presets at runtime. The server can run with both enabled, which is the default, or in OSC-only headless mode using `--no-ws`.

A startup message, `/audio/meta`, is sent once at startup and again whenever the device, the FFT setting or the cutoffs change. It carries the sample rate, block size, FFT bin count and the three band edges in Hz. The documentation explicitly frames this as a way to size a spectrum buffer and to learn the sample rate the device actually opened at, which is a good sign that the author expects devices to behave differently from what you request.

The two transports also mean two different failure surfaces. A UDP destination that is not listening simply receives nothing, while the WebSocket path is the one that carries state changes such as switching the input device, so a client that only speaks OSC has no way to retune the running server.

## Installing PortAudio and starting the server

The project needs Python 3.10 or newer and PortAudio. PortAudio is a system library rather than a Python package, so it comes from the platform package manager first:

```bash
# macOS
brew install portaudio

# Ubuntu / Debian
sudo apt install libportaudio2 portaudio19-dev
```

Then the server itself is installed from the repository root. The project documents two routes, and the first one is the one it recommends:

```bash
uv sync

pip install -e ".[dev]"
```

The `uv` route creates a virtual environment from the committed `uv.lock` and needs no activation step. The pip route installs the project in editable mode with the `dev` extra, which pulls in `pytest>=8.0` and `pytest-asyncio>=0.23`. The package registers a console script named `audio-server` that points at `server.main:main`.

Running the server and opening the interface in a browser is a single command:

```bash
uv run audio-server --open
```

The alternatives cover the cases you would expect:

```bash
uv run audio-server                    # reads ./configs/main.yaml, opens WS on 8765, UI on 8766
uv run audio-server --open             # also opens the UI in your default browser
uv run audio-server --no-ws            # headless OSC-only mode
uv run audio-server --config /path/to/cfg.yaml --log-level DEBUG
```

The WebSocket server listens on port 8765 and the browser interface is served on port 8766. The setup notes state that `uv run python -m server.main` is equivalent, and that when you use plain pip you should drop the `uv run` prefix and make sure your environment is active first. One gotcha is called out explicitly: open the UI at `http://127.0.0.1:8766` rather than opening `ui/index.html` directly over `file://`, because the ES modules will not load that way.

## Retuning bands through the browser and configs/main.yaml

Settings live in `configs/main.yaml` and persist there, so the server comes back in the state you left it. The OSC destination list and the FFT switch are configured in that file:

```yaml
osc:
  destinations:
    - { host: 127.0.0.1, port: 9000 }
    - { host: 192.168.1.42, port: 7000 }
  send_fft: false              # set true to also stream /audio/fft
```

The point of showing this is the shape of the configuration, not the specific values. Destinations are a list of host and port pairs, and adding a second receiver means adding a second entry and restarting. The FFT is a single boolean rather than a level, so a downstream application that wants the spectrum has to have it enabled before it starts receiving.

The browser side is where the interesting tuning happens, since the project ties the visible controls to the detectors that feed OSC. Changing a band crossover in the UI changes both the shape of the track you see and the behaviour of that band's onset detector, and presets can be saved and loaded from the interface. There is no documented hot-reload for the YAML file itself, and the published configuration does not describe a schema beyond the OSC block shown here, so most of the file's keys are not publicly specified.

## Where the feature set stops and the versioning does not exist

Several limits are visible in the repository state. The project has no GitHub releases, so there is no tagged version history to pin against, and the package version in `pyproject.toml` is `1.0.0` with no accompanying release notes. The last push was on 2026-10-08, which is recent, so the project does not read as abandoned, but the absence of releases means upgrades arrive as raw repository changes.

The feature list is also narrow by design. You get three band energies, an optional spectrum, three onset streams and a BPM estimate derived only from the low band. There is no chord detection, no key estimation, no pitch tracking, no vocal separation and no beat-downbeat labelling, and the documentation does not mention any of these. A client that needs those has to compute them itself, which partly defeats the point of a shared feature server.

The BPM figure deserves particular caution. It is derived from the low onset stream and folded into `[60, 180]`, so material with no low-frequency content has no meaningful value to fold, and the range itself excludes anything slower than 60 BPM and faster than 180 BPM. The documentation does not cover what happens outside that range, and it does not give the smoothing constant used for the estimate.

The repository also carries a design document, `realtime_audio_server_plan.md`, that the documentation points to for the full spec, and separate `docs/` and `ui/` directories. A design spec is useful, but the README is where the concrete behaviour lives, and it is short enough that edge cases such as device errors, reconnect behaviour after a device switch and buffer overruns under load are simply not described.

## A server side analyser against an analysis rig inside the host

The real alternative to this project is not a different package, it is a different arrangement. The usual shape for a VJ or interactive audio tool is to open the soundcard inside the host application, run the analysis there, and let the visual code read the result directly. TouchDesigner, Max/MSP, Unity and p5.js are all named as destinations in the documentation.

The difference Realtime_PyAudio_FFT offers is that the analysis lives in one place, above the host, and is shared. Band edges, scaling and onset sensitivity are configured once and then behave identically whether a TouchDesigner patch, a Max patch and a p5.js sketch are all listening at once. The documentation makes the same point from the other side by stressing that the browser is a thin renderer and that the graph you tune in the UI is byte-identical to the OSC payload.

The cost of that arrangement is an extra process and an extra network hop on localhost, plus the fact that the OSC path is one way. If the OSC-only mode suits your rig and you only need band levels, running the analysis inside the host is simpler. If you want a single tuned feature source shared by several tools, or you want to drive a headless Raspberry Pi installation, this project's shape fits.

On licensing, the repository carries an MIT `LICENSE` file, which is a permissive grant and generally the lowest friction option for embedding in commercial or closed tooling. That is a licensing observation rather than legal advice, and the MIT file's own terms govern.

## Conclusion

This suits people who already have a visual or game side that reacts to sound and need a dependable feature source without writing their own DSP, and the browser UI makes tuning a band quick. It does not suit anyone who needs a production audio pipeline, a stable release series, or a documented set of guarantees: the repository has no GitHub releases, the last push was on 2026-10-08, and the README does not document a versioning or support policy. Before adopting it, check that your client can receive OSC on the addresses listed under `osc.destinations`, confirm PortAudio opens your capture device at the sample rate you expect, and verify by listening to `/audio/lmh` and `/audio/onset/low` that the auto-scaling matches your material.

## FAQ

### How do I install and run Realtime_PyAudio_FFT on my machine?

Install PortAudio from your platform package manager, then run `uv sync` from the repository root and start the server with `uv run audio-server --open`. The project also documents `pip install -e ".[dev]"` as an alternative to uv, and requires Python 3.10 or newer.

### Which ports does Realtime_PyAudio_FFT use, and where does the browser UI live?

The WebSocket server opens on port 8765 and the browser interface is served on port 8766, so you reach the UI at `http://127.0.0.1:8766`. The setup notes warn against opening `ui/index.html` directly with `file://` because the ES modules will not load.

### How do I receive audio features from Realtime_PyAudio_FFT in another application?

Add the destination host and port under `osc.destinations` in `configs/main.yaml`, restart the server, and listen on the documented OSC addresses such as `/audio/lmh`, `/audio/onset/low` and `/audio/bpm`. The default destination is `127.0.0.1:9000`.

### Can I tune the onset detection of each frequency band separately in Realtime_PyAudio_FFT?

Yes. Each of the three onset detectors has its own `sensitivity`, `refractory_s` and `slow_tau_s`, and detection runs on the post-processed band signal, so retuning a band in the browser UI tunes its detector too. The README lists default refractory values of 250, 180 and 120 ms for low, mid and high.

### Is Realtime_PyAudio_FFT suitable for a production audio pipeline?

The repository has no GitHub releases, so there is no tagged release stream to pin against, and the last push was on 2026-10-08. The project presents itself as a localhost feature provider for visual and interactive work rather than a production pipeline component.

## Sources

- [aiXander/Realtime_PyAudio_FFT on GitHub](https://github.com/aiXander/Realtime_PyAudio_FFT)
- [Issues](https://github.com/aiXander/Realtime_PyAudio_FFT/issues)
- [License: MIT](https://github.com/aiXander/Realtime_PyAudio_FFT/blob/master/LICENSE)
- [README](https://github.com/aiXander/Realtime_PyAudio_FFT/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/aixander-realtime-pyaudio-fft