jacbz/Lofi: a VAE picks 100 numbers, Tone.js turns them into a track

The problem is not generating music, it is deciding where the generation happens

Most browser music generators do everything in JavaScript, which keeps deployment trivial and keeps the model small. Lofi takes the opposite route. The README states that a VAE was trained in PyTorch to represent a lo-fi track as a vector of 100 features, and that the web client uses Tone.js to turn those parameters into audio. So the neural network never runs in the browser. It runs behind a Flask instance, and the browser only receives a compact parameter set that it renders locally. That is the design decision the whole project is organised around. The audience is narrow: someone who wants to inspect or replace the model without rewriting an audio engine, or someone who wants a working Tone.js arrangement layer without writing a model. It is not aimed at musicians looking for a finished tool, and it is not a library you import into an existing app.

Three folders, one REST call, and a 100-number handoff

The README describes three components. The client is written in TypeScript, built with Webpack, and uses Tone.js to generate music. The model is implemented in PyTorch, and the README says various datasets were synthesized, naming Hooktheory and Spotify. The server is described as a basic Flask instance that deploys the trained model checkpoint, with the client communicating over a REST API. The data flow implied by that description is short: browser requests parameters, Flask loads the checkpoint and samples the latent space, the response carries the 100-feature vector, and Tone.js maps those features onto chords, melodies and the other musical parameters the README mentions. The repository also lists tonejs as a topic alongside pytorch, flask and vae, which matches the README's account. What the material does not give is the endpoint path, the request or response schema, or the mapping from a feature index to a musical decision. Those live in client/src/api.ts and the server folder, and you will have to read them. Treat the 100-dimensional vector as an interface you inherit rather than one you can reason about from the README alone.

Getting the client up before you touch the model

The README is explicit that the client folder works on its own and defaults to the project's server as the backend. That is the cheap path. Install Node.js LTS, then from the client folder run npm install, followed by npm run serve for development or npm run build for a distributable. If you only want to tinker with the client, the README says you need nothing else. The moment you want your own model, the sequence changes. Train a model following the instructions in the model folder, place the checkpoint in the checkpoints folder, deploy the server (the README points at the server folder and notes a provided Dockerfile), and then change the server address inside client/src/api.ts. The README also offers a shortcut: a pre-trained checkpoint zip hosted as a GitHub file attachment, dated in the URL path, which you can use instead of training. Two things are worth flagging. The checkpoint link is a file attachment rather than a release asset, and the repository shows no retrieved releases, so there is no versioned artefact to pin. And the API address lives in a source file rather than an environment variable, which means pointing the client at a different backend is a code edit and a rebuild, not a config change.

The hosted backend is the part you do not control

The default path routes every generation request through the project's own server, hosted with thanks to ZOOPRA UG per the acknowledgments. That is convenient for evaluation and unsuitable for anything you depend on. The README offers no rate limits, no uptime statement, no API versioning and no deprecation policy, and the repository material contains no releases that would signal a compatibility contract. If the server moves or the endpoint changes, a client built against the default backend breaks with no migration note to follow. This is the clearest case where Lofi is the wrong tool: if you need a generation service with a stability guarantee, the hosted backend is not it, and the self-hosted path means running and paying for a Flask instance with a PyTorch checkpoint loaded. The second limitation is subtler. Because the model output is a latent vector and the musical interpretation happens in the client, a checkpoint and a client version are coupled through an undocumented 100-dimensional contract. Swapping in a checkpoint trained with a different latent size or a different feature ordering will not fail loudly at the API boundary; it will produce plausible-looking audio that is simply wrong. Nothing in the README describes a schema check or a version field to catch that.

How this differs from running a model in the browser

The obvious alternative for a web music generator is to ship the model to the client, typically by exporting it to a browser runtime and doing inference in a worker, with the audio graph built on the Web Audio API or a wrapper around it. That approach removes the server entirely: no Flask process, no checkpoint hosting, no network round trip per generation, and no dependency on someone else's uptime. The cost is a much larger client bundle and a hard ceiling on model size, because the model has to download before it can run. Lofi inverts both sides of that trade. The client stays small and the model can be whatever fits on the server, but every generation needs a reachable backend, and the project's default backend is not yours. There is a third option worth naming: use a symbolic music library or a rule-based generator and skip machine learning altogether. That gives deterministic, inspectable output with no checkpoint to manage, at the cost of the variation a sampled latent space provides. Lofi sits between those poles, and the Flask hop is the price of keeping the model out of the bundle.

Licence, forks and the cost of staying current

Lofi is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant, with the usual conditions around retaining notices and stating changes. That is a permissive licence, and it is more forgiving than a copyleft one would be for anyone embedding the client in a product. It does not resolve the questions the licence cannot answer: the provenance and licensing of the training data. The README says datasets were synthesized from Hooktheory and Spotify, and it says nothing about the terms under which that data was obtained or what constraints might attach to a model trained on it. If you plan to ship generated audio commercially, that is a question for your own counsel, not something the Apache-2.0 file settles. On maintenance, the material available here is thin. There are no retrieved releases, so there is no changelog to read, no semantic version to pin, and no upgrade path documented beyond re-running npm install and rebuilding. The last push date is recent, but a recent push is not the same as a compatibility promise. Budget for reading the source when you update, particularly client/src/api.ts and the server entry point, because those are the two places the client-server contract actually lives.

Editorial conclusion

Adopt Lofi if you want a small, readable example of splitting generative music between a Python model and a browser audio client, or if you want to swap in your own checkpoint behind the existing REST API. Do not adopt it if you need a maintained package with release notes, a stable hosted backend or a documented model card: none of those appear in the repository material. Before committing, verify that the pre-trained checkpoint download link still resolves, that client/src/api.ts contains the server address you intend to call, and that the server Dockerfile builds against the Python and PyTorch versions currently pinned in the server folder.

Official sources

1. Issues
2. jacbz/Lofi on GitHub
3. License: Apache-2.0
4. Project website
5. README