Five demos of the LLM pipeline, ending in a transformer written by hand

The build is identical in all five sections

Five demos, and the structural point is that they are the same program five times.

Running it takes a package manager and a recent runtime:

pnpm install
pnpm dev

If the installer reports that build scripts were ignored, you are told to run an approval command, select one named package with the space bar, confirm, and then install again.

Each demo then has three parts. A server route that takes input and streams events. A client hook that holds the state and translates those events into updates. And a result component that renders the stream as a visualisation.

The consequence is that reading one section teaches you the architecture, and reading the other four is a matter of what the visualisation shows. The generic chat hook is shared, as is the streaming implementation on both sides of the wire, and so is the message type definition.

That is a good design for a teaching application, because it means none of the complexity in the later sections is framework complexity. When the transformer demo takes a while to draw, that is the transformer and not the plumbing.

The stack is two choices. One framework handles the server, the component rendering and the client components together, which is why the readme lists a single entry for it rather than splitting server and client libraries. A bundler and dev server is the other.

So there is no separate template engine, no component framework, and no state library. The server-rendered pages, the interactive visualisations and the streaming endpoint are all the same tool.

That is unusual for an application that also has to run numeric code in a loop, and it is why the last section needed a workaround.

Section one has the plumbing and none of the intelligence

The first demo is the design of the whole project.

It is a pattern-matching chatbot of the kind written in the nineteen sixties: it responds using conditional statements, with no model and no learning. The readme is blunt about the status of this section, describing it as the simplest possible artificial intelligence and noting that it has the same plumbing as a chat product with none of the intelligence.

That framing is doing real work. Every later section reuses the identical streaming path, the identical message types and the identical client hook. So when you move to the second demo and something responds that did not respond before, you know it came from the algorithm rather than from the transport.

The response is streamed word by word rather than delivered at once, which matters more than it sounds. A streaming endpoint that emits chunks is the same interface a real model server exposes, so the client code written for the first demo is the client code you would write against one.

The second demo then trains a small network live in the browser. A single-layer perceptron fails on the XOR problem, which is the historical result that undermined single-layer networks, and a multi-layer network succeeds once backpropagation is applied.

So the sequence goes from no learning at all, to the specific architecture that cannot learn, to the specific algorithm that lets depth work. Each step isolates one variable, which is the only way an interactive demo can actually teach.

The fifth demo implements the optimizer as well as the attention

The final section is the one that determines whether this is a toy or a demonstration.

It trains a decoder-only transformer entirely from scratch, and the constraint stated is that no machine learning libraries are used. Every operation is written by hand: multi-head causal self-attention, layer normalisation, the feed-forward layers, backpropagation, and the optimizer.

That last inclusion is the significant one. Most from-scratch transformer demos use a library's optimizer and optimizer implementations are exactly the part people skip, because they are not the interesting idea. Writing it means the demo can show the update rule rather than call it.

Two implementation details are worth knowing. Initial weights use the scheme described in a paper about the difficulty of training deep feedforward networks, rather than a naive default, and layer normalisation is applied before the attention and feed-forward layers rather than after.

For speed, the training uses multi-threaded data parallelism through a shared array buffer. That is what lets it run in a browser at all, and it is also the reason you may need a cross-origin isolation header for the page, since that mechanism is only exposed to isolated contexts.

So the last demo is doing several things at once: it is the architecture, it is the training loop, and it is a demonstration that both can be written by one person in a weekend.

The tokenizer is written once and used by three sections

Between the transformer demo and it sit the two that surprise people, and one of them is shared.

The third section builds a byte pair encoding tokenizer from scratch, training on whatever text you supply it. The visualisation animates each merge step as the vocabulary grows from characters towards words.

That animation is the teaching device. Byte pair encoding is normally described in prose as repeatedly merging the most frequent adjacent pair, which is comprehensible and conveys nothing. Watching it run on your own sentence shows that the merges are driven by frequency in your specific text, so a corpus of unusual vocabulary produces an unusual vocabulary table.

The fourth section trains word embeddings with a skip-gram model and negative sampling, again from scratch, with a visualisation of vectors that gradually cluster words appearing in similar contexts.

The third, fourth and fifth sections all share a single tokenizer implementation on the server side.

So the tokenizer is written once, and the three demos that need it consume the same code. That is a small architectural decision with a large pedagogical payoff: the same tokenisation runs through the embedding trainer and the transformer trainer, so nothing is normalised or truncated differently between them.

It also means the pipeline the readme describes is genuinely continuous rather than five isolated illustrations.

Ten papers, each tied to one section

The reference list is ordered by where each concept appears across the demos, and every entry is annotated with the section that implements it.

The first is the original pattern-matching chatbot paper from the nineteen sixties, tied to section one. The second is the backpropagation paper that made multi-layer networks trainable, tied to section two.

Then come two papers on word vector estimation, in two separate parts, one introducing the skip-gram architecture and one introducing negative sampling, both tied to section four.

One paper on initialization schemes is tied to section five, and one on the adaptive optimizer is also tied to section five.

Then the subword tokenization paper, which is annotated as being used across sections three, four and five, which matches the shared tokenizer described above.

Two more tied to section five: the layer normalisation paper, annotated as showing that it is applied before the attention and feed-forward layers, and the original transformer paper.

And the generative pre-training paper, which is the last entry.

The ordering is chronological within the pipeline rather than alphabetical, and the annotations are the useful part. Rather than a reading list, this is a table saying: this is where the idea enters the code, so you can read the paper and then find the twenty lines that use it.

For an application whose purpose is explaining a pipeline, that mapping is the difference between citing sources and actually using them.

Modern toolchain, unusual build, and no test script

A few details in the package manifest are worth flagging.

First, the package is named simply for an application rather than scoped to a publisher, and it depends on a small routing library published under the same scope as the repository itself. So the author maintains both the application and a router it uses.

Second, the build script runs the bundler twice, once in a client-only mode and then a full build, rather than once. That is the shape you need when the server output and the client output are separate artifacts rather than one bundle, and the start script then runs a single file out of the distribution directory.

Third, the dependency versions are all recent majors across the board, which is unusual to see pinned together and suggests the project is kept current deliberately rather than upgraded opportunistically. Validation uses a schema library at version four through a framework integration, and the linting configuration comes from an opinionated preset with a plugin that formats as part of linting rather than running a separate formatter.

Fourth, and the one real gap: a test framework is present as a development dependency, but there is no test script in the manifest at all. So tests may exist and may be run by hand, but nothing in the package scripts invokes them.

For a teaching project where the demonstrations are the product, that is a defensible omission, and it is worth knowing before you assume a green install means anything is verified.

Editorial conclusion

This suits someone who can use a transformer but has never watched one train, because the value is the intermediate stages rather than the final one, and the XOR failure in the second section is the argument in miniature. It is the wrong resource if you want to build with these techniques, since nothing here is a library you would depend on. Before running it, expect the training section to be slow in a browser, note that the multi-threaded path depends on a cross-origin isolation header, and set expectations that the visualisations are the point rather than the numbers.

Frequently asked questions

Official sources

1. Issues
2. README
3. w3cj/how-llms-work on GitHub