OpenMythos: A Looped Transformer Reconstruction for Research, Not Production

What OpenMythos Actually Reconstructs

The architecture divides the transformer into three blocks. A Prelude of standard transformer layers runs once, then a Recurrent Block loops up to `max_loop_iters` times, and a Coda of standard layers runs once. The recurrent block's update rule is h_{t+1} = A·h_t + B·e + Transformer(h_t, e), where h_t is the hidden state after loop t, e is the encoded input from the Prelude, and A and B are learned injection parameters. The input e is injected at every loop, which the README says prevents the model from drifting and keeps the original input signal alive throughout the recurrence. This is a concrete mechanism, and it is the heart of the project. The design choice to inject the original input at every step is what differentiates this from a simple recurrent network that might forget the input over depth.

Attention and MoE: The Switchable Components

The attention implementation has a practical caveat. GQA uses Flash Attention 2 only if `flash-attn>=2.8.3` is installed, and it requires CUDA and build tools. The README notes a transparent fallback to manual scaled dot-product attention when the package is absent. That fallback is important because it means the code runs on CPU or older GPUs, but it will be slower. For MLA, the README does not mention a flash-attn integration, so you are likely relying on a manual implementation. If you plan to train at scale, you will want to verify the performance of the MLA path on your hardware before committing.

Installation and Configuration: What You Actually Run

The README also provides pre-configured variants from 1B to 1T parameters, named `mythos_1b`, `mythos_3b`, `mythos_10b`, `mythos_50b`, `mythos_100b`, `mythos_500b`, and `mythos_1t`. The table shows that larger variants have more experts, higher `expert_dim`, and longer `max_loop_iters` (up to 64 for the 1T variant). Context length scales from 4k to 1M, and max output length scales from 4k to 128k. These are aspirational configs; the README does not claim any of them have been trained. The 3B variant has a training script, but no checkpoints are provided. You are expected to train from scratch.

Training Script: The 3B FineWeb-Edu Example

The training script is minimal. It does not include any evaluation loop, checkpointing strategy, or logging setup beyond what PyTorch DDP provides. The README does not mention gradient accumulation, mixed-precision edge cases, or how to resume training. For a research project, this is acceptable; for a production training run, you will need to build those pieces yourself. The script is a starting point, not a complete solution.

The Central Hypothesis and Its Trade-Offs

Another limitation is the lack of any pretrained weights. The README provides configs but no checkpoints. You cannot download a model and run inference. You must train from scratch, which requires a significant data budget and compute. The 3B training target of 30B tokens is a concrete number, but it is not validated. The README does not report any loss curves, evaluation metrics, or sample outputs. This is a theoretical reconstruction, and it stays theoretical until someone trains it.

Alternatives: DeepSeek-V2 and Standard Transformers

The choice between GQA and MLA is also a point of comparison. GQA is the more standard approach, with broad support in inference engines. MLA is more memory-efficient for long contexts but is less widely implemented. If you plan to deploy a model, GQA might be safer. If you are researching long-context efficiency, MLA is worth exploring. OpenMythos lets you switch between them, which is a practical feature for comparative experiments.

Maintenance, License, and What to Verify

Before adopting OpenMythos, verify a few things. First, check the actual code in `open_mythos/main.py` to confirm that the MLA and GQA implementations match the README's claims, especially the fallback behavior. Second, test the memory usage of the recurrent block with your intended `max_loop_iters` on your hardware. Third, confirm that the training script works with your dataset and tokenizer; the README only mentions FineWeb-Edu and the gpt-oss tokenizer. Finally, note that the project has no pretrained weights, so you must be prepared to train from scratch. If you are not, this is the wrong tool.

Editorial conclusion

OpenMythos is for researchers who want to experiment with recurrent-depth transformers and compute-adaptive reasoning in a clean PyTorch codebase. It is not for teams needing a production-ready LLM; the project is a theoretical reconstruction with no pretrained weights, no verified training results, and no support for inference beyond basic generation. Before adopting it, verify that your hardware can handle the memory footprint of the looped hidden states and that you have the data budget (30B tokens for the 3B variant) to train from scratch. If you need a working model today, look at existing open-weight transformers instead. If you want to test the looped-depth hypothesis, OpenMythos gives you a concrete starting point, but expect to write your own training loops and debugging.

Official sources

1. Official documentation
2. Official README
3. Project repository