Recurrent Looped Transformer: a project page for recurrence across prompt and response
Official Project Page for Recurrent Looped Transformer (RLT)
At a glance
- What is it?
- RLT is a research project page and paper, not a library. The repository holds the report, the architecture figures and a depth-eight comparison against an eight-layer Transformer, and the README is the only documentation of the mechanism.
- Who is it for?
- Adopt this as reading material if you work on recurrent depth, looped decoders or state carried across the prompt-response boundary, and if you want the exact configuration behind the depth-eight tables. Do not adopt it if you need a pip package, a training script or a maintained codebase: the top-level entries are LICENSE, README.md, two PDFs, assets/, figure1.png and index.html, so there is nothing to install and nothing to run.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly HTML, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What RLT is for, and the problem it targets
Most decoder-only stacks spend a fixed amount of computation per token. Depth is set at design time and does not change when a token is hard. RLT explores the other direction: after t tokens, the README states that the recurrent path traverses tL_D decoder blocks while the number of blocks evaluated per token stays fixed. Compute per token is constant; compute along the sequence grows with the recurrent path.
The target reader is a researcher working on recurrent depth, looped decoders or state that survives the prompt-response boundary. The README describes the same update running over prompt and response tokens, and figure1 is captioned as recurrence across the last prompt tokens and the first response token. That boundary is the interesting part. A conventional decoder resets its state when generation starts; RLT carries the previous final decoder output into the next token's gated merge, so the state does not restart at the first response token.
This is a paper and a project page. The top-level entries are LICENSE, README.md, Recurrent_Looped_Transformer.pdf, Recurrent_Looped_Transformer_ZH.pdf, assets/, figure1.png and index.html. There is no package to install and no training script in the repository, so the audience is people who read reports and reimplement, not people who want to import a model.
The recurrent state, the global KV memory and the per-layer SWA cache
The README writes the decoder state as H_t = (s_t, C_t^D), starting from H_0 = (s_star, empty). The update takes a gated merge of the token's causal encoder representation e_t and the previous recurrent output s_{t-1}, feeds it through the decoder D_phi together with the encoder-derived global KV memory M_<=t, the previous per-layer cache C_{t-1}^D and the step index t.
Two memory paths run in parallel. Global context: encoder outputs are projected into cached KV and cross-attention reads positions up to the current token. With one memory group (G=1), every decoder layer reads the same projected KV using its own queries. Local memory: each decoder layer projects its own sliding-window attention KV, and a window of W includes the current token and retains up to W-1 past entries for the next update. The experiments use W=8 and G=1.
The README also documents a control it calls RLT w/o feedback, which removes the previous-output feedback path, the learned initial state, state normalization, the gated merge and the feedback projection. The decoder then receives z_t^0 = e_t directly while keeping global cross-attention and layerwise SWA. The README states that known tokens can run in parallel within each decoder layer during training and prefill in that variant, with generation still one token at a time. That parallel path is the practical reason the control exists: feedback serializes the update, and removing it restores parallelism over known tokens.
There is no install: what the repository actually gives you
The README does not document an installation procedure, a package name, a Python module or a CLI. The top-level entries are LICENSE, README.md, Recurrent_Looped_Transformer.pdf, Recurrent_Looped_Transformer_ZH.pdf, assets/, figure1.png and index.html. The project page is published at https://yifanzhang-pro.github.io/recurrent-looped-tranformer, and the README links the report as ./Recurrent_Looped_Transformer.pdf. There is no pip install line to quote, and inventing one would be wrong.
The realistic first use is to read the architecture section of the README alongside the paper, then check whether the configuration matches your own setup. The parameters the README states for the depth-eight experiments are the ones you would need to reproduce anything:
width 512
FFN width 1365
attention heads 4
global batch 512
AdamW, same schedule across models
SWA window W = 8
encoder-memory groups G = 1
feedback scale 0.1
TBPTT 128, covering every training sequence in the snapshotThe README notes that RLT models have 26.10 to 28.73M parameters while Transformer 8 has 25.31M, and that RLT 8+0 has no decoder blocks and still applies the recurrent merge. If you want to reimplement, those are the numbers to match before comparing anything. The repository does not ship the task generators, the training loop or the evaluation harness, so the README is a specification, not a starting point you can clone and run.
The depth-eight results, and where they are thin
The September 15, 2026 snapshot (16:11:47 to 16:12:10 UTC) compares RLT 4+4, 5+3, 6+2, 7+1, 8+0 and Transformer 8 across six tasks. The README is explicit that of 48 runs, 21 had reached the planned 2,000 optimizer steps and 27 were unfinished. All results measure held-out validation accuracy at training lengths, and each table row compares the models at the latest step available for every required run within that task, which means the comparison step differs per task: 1000 for addition, 500 for parity, 800 for both mod-5 variants, 1000 for both S5 variants.
The strongest claim concerns parity. At step 500, RLT 6+2 reaches 99.44 +/- 0.98% against 48.48 +/- 0.53% for Transformer 8, with all three RLT 6+2 seeds at 100% by step 600 while Transformer 8 reaches 94.84 +/- 3.43% at step 2,000. The README also reports that RLT 4+4, 5+3 and 7+1 average about 83% at step 500 with standard deviations of 28 to 30 percentage points. That spread is the part worth pausing on. A mean of 83% with an SD near 29 points across three seeds is not a stable result, and the README says so rather than hiding it.
The completed mod-5 subset, at 2,000 steps and seed 42, is the cleanest comparison: RLT 7+1 at 95.44 and RLT 8+0 at 90.89 on flat mod-5 against 20.57 for Transformer 8, but on bracketed mod-5 the gap closes to 78.26, 75.52 and 79.56. Transformer 8 wins that variant. The README attributes the difference to task generators that differ in operator structure and label distribution, and notes that parentheses also occupy token positions. Standard S5 is a dead column: 1.17, 3.52, 1.95, 1.56, 1.17 and 1.17 percent. No model learns it, and any conclusion drawn from it would be noise.
Where RLT is the wrong tool, and what to compare it against
The feedback path is sequential. The README states that the RLT w/o feedback variant can run known tokens in parallel within each decoder layer during training and prefill, which implies the full RLT cannot: the previous final decoder output enters the next token's gated merge, so token t depends on token t-1 through the recurrent state, not only through attention masking. If your bottleneck is prefill throughput on long prompts, the feedback design works against you, and the no-feedback control is the configuration to look at.
Memory is the second constraint. Each decoder layer projects its own SWA KV, so cache cost scales with decoder depth, and the global memory is cached separately. The README gives W=8 and G=1 for these experiments, which keeps the local window small, but it does not report memory or throughput numbers anywhere in the retrieved text. Anyone planning to train at longer sequence lengths is working without that data.
The natural alternative is the plain eight-layer Transformer, which is exactly what the tables use as the baseline. The difference in approach is not scale but where the computation goes: Transformer 8 runs eight blocks per token with no state carried between tokens, while RLT runs a fixed number of blocks per token and lets the recurrent path traverse tL_D blocks over t tokens. The README notes that compatible encoder and decoder attention and FFN weights can be shared, with a tied 48+48 layout illustrating the option in the report, while the experiments use untied eight-layer layouts. So the parameter counts above are not the tied configuration.
A second comparison is the no-feedback control itself. It keeps global cross-attention and layerwise SWA and drops the feedback machinery, which isolates how much of the parity result comes from recurrence rather than from the memory layout. The README states that the 4+4 control was implemented after this snapshot for future matched ablations and has no results in the tables. That is the ablation the current tables are missing.
Licence and maintenance cost
The repository is licensed Apache-2.0. That is a permissive licence, and it covers the repository contents: the README, the two PDFs, the assets and index.html. It does not tell you anything about the licence of any code that does not exist here, because there is no training code in the repository to license. The paper itself is a separate artefact and its terms are not stated in the retrieved material; if you plan to redistribute figures or tables, check the PDF rather than assuming the repository licence covers them. This is not legal advice.
The maintenance picture is narrow. The last push was on 2026-09-16, and the README carries a report dated September 12, 2026 with an update dated September 15, 2026. The retrieved material lists no releases. The README states that 27 of 48 runs were unfinished at the snapshot and that the 4+4 control has no results yet, so the tables are explicitly a snapshot rather than a finished comparison. If you depend on these numbers, you are depending on a document that says it is incomplete, and the upgrade cost is re-reading the tables when a later snapshot changes them.
Editorial conclusion
Adopt this as reading material if you work on recurrent depth, looped decoders or state carried across the prompt-response boundary, and if you want the exact configuration behind the depth-eight tables. Do not adopt it if you need a pip package, a training script or a maintained codebase: the top-level entries are LICENSE, README.md, two PDFs, assets/, figure1.png and index.html, so there is nothing to install and nothing to run. Before you cite any number, open the September 15, 2026 snapshot note and check which of the 48 runs had reached the planned 2,000 optimizer steps, because 27 were unfinished, and check which task each accuracy belongs to, since the standard S5 column sits near 1% for every model and carries no signal.
Frequently asked questions
What is a looped transformer?
In this project, it is a decoder whose previous final hidden state is fed into the next token together with that token's causal encoder representation, so the same update runs over prompt and response tokens. The README states that after t tokens the recurrent path traverses tL_D decoder blocks while the number of blocks evaluated per token stays fixed.
What is a recurrent transformer?
RLT carries a state H_t = (s_t, C_t^D) from token to token, where s_t is the recurrent output and C_t^D is the layerwise sliding-window attention cache. The previous output enters the next token's gated merge, and the state continues across the prompt-response boundary.
Did transformers replace RNNs?
The README does not make that argument. It presents RLT as a decoder that keeps global cross-attention and layerwise sliding-window attention while adding a recurrent feedback path, and compares it against an eight-layer Transformer rather than against an RNN.
What is a recurrent depth transformer?
The README frames depth as something the recurrent path accumulates: after t tokens the path traverses tL_D decoder blocks, while the blocks evaluated per token stay fixed. The experiments vary how the eight layers split between encoder and decoder, from 4+4 through 8+0.
Community notes