Orca: BAAI's world foundation model that predicts the next state, not the next token

A world model trained on states, not objectives in isolation

Orca, from the Orca Team at the Beijing Academy of Artificial Intelligence, is described by its README as an initial instantiation of a general world foundation model. The bet it makes is about the training objective. Most generative systems optimize next-token, next-frame or next-action prediction separately; Orca centers everything on next-state prediction, one state-transition model that is supposed to serve understanding, forecasting and acting alike.

Learning runs on two channels. What the team calls unconscious learning consumes dense, unlabeled transitions from continuous video. Conscious learning adds sparse, meaningful transitions supervised by language: event descriptions, task intentions, causal explanations and VQA. In this first version the inputs are deliberately narrow, visual signals for how the world evolves and language signals for what it means, which keeps the first paper testable.

Freeze the latent, train the readouts

Architecturally Orca is an encoder-decoder system with an unusual division of labor. The encoder learns the world latent through the two learning channels. After pre-training it is frozen, and only lightweight modality-specific decoders are trained to read the latent back out as text, as images, or as actions.

The README also separates the kinds of dynamics the latent has to carry. Implicit dynamics cover the unobserved: physical laws, object properties, scene behavior, environmental forces. Explicit conditions are the observed inputs: an instruction, an event description, a causal premise. The claimed scaling property is straightforward: stronger world modeling should yield stronger downstream readouts, and the frozen-backbone design is what makes that measurable without retraining the encoder for every new task.

The data budget behind the latent

Pre-training data comes in three kinds, and the numbers are concrete. The mixture includes 125K hours of video spanning egocentric interaction, exocentric manipulation, robot execution and natural dynamics, used for observation-only state transitions. It adds 160M event annotations with fine- and coarse-grained captions, which drive event-conditioned transitions. General VQA data aligns the latent with language understanding and response generation.

That split mirrors the learning channels: video teaches what happens next, events teach when something happened, and VQA teaches answering about it. For a self-hosting researcher the practical consequence is that the interesting artifact is the released checkpoint, not the data; only partial data releases are on the roadmap, and reproducing the pre-training corpus is out of reach for most labs.

Getting the checkpoint today

The README documents no install commands, so the practical path is short. The Orca-4B checkpoint, released on 2026-07-14, sits on Hugging Face under BAAI/Orca-4B, and the technical report is on arXiv. What the repository itself ships in this tree is documentation and evaluation: the README, the assets, and an evaluation directory containing the PRICE toolkit under evaluation/image_gen/PRICE for the image-prediction benchmark.

The todo list marks inference code for text and image as released and downstream fine-tuning code as still pending, with an Orca-0.8B checkpoint promised for lightweight research. Anyone adopting this today should expect to work from the checkpoint plus the report, and to wait for the remaining pieces or take the project at its word about what is coming.

What the reported numbers show, and what they skip

The README's evaluation table reports mean scores across four text benchmarks: MVBench, TemporalBench, 3DSRBench and SWITCH. At 4B parameters, Orca averages 51.8 against 46.7 for Qwen3.5, winning TemporalBench, 3DSRBench and SWITCH while trailing Qwen3.5's 67.1 with 65.3 on MVBench. The 0.8B row averages 40.8, ahead of Emu3 at 8B on the same average.

Two pieces of fine print deserve attention. The table reports a 0.8B Orca even though the model zoo still lists that checkpoint as coming soon, so part of the table cannot be reproduced today. And image prediction is evaluated on PRICE-V0.1 with a toolkit in the repository, while action generation is named as a readout but gets no table in this README. The results section, in other words, is strongest where the artifacts already exist.

Where the project stands: open weights, partial everything else

This is a research release, and the edges show. The repository tree holds no licence file at all, so reuse terms are currently unstated. The file tree itself is minimal, documentation and evaluation; the inference code the todo marks as released is not visible in this tree, and the fine-tuning code is explicitly pending. Questions route through a WeChat contact rather than a public tracker, which matters if you were hoping to read past issues before adopting.

None of that undermines the checkpoint. It does mean the total cost of adoption today includes reading a technical report, tolerating a thin repository, and accepting that some listed deliverables are promises with dates attached rather than downloads.

How it sits against frame predictors and action models

Video generation models learn to continue pixels: give them frames, they extrapolate footage, and they are passive, with no notion of a task. Vision-language-action systems go the other way: they train on demonstrations and emit actions, with little explicit model of how a scene evolves when nobody acts. Orca's position is between the two: one latent meant to support prediction, description and action, with the differences handled by cheap readout decoders instead of separate models.

Whether that unification earns its complexity is exactly what the evaluation section is testing, and the current evidence, one strong average on text benchmarks and a toolkit for image prediction, is early. The 0.8B checkpoint, when it lands, will be the cheapest way for an outside lab to judge the claim.

Editorial conclusion

Orca-4B is worth a download if you research world models or video understanding and want a state-transition latent with readouts to probe; the checkpoint is on Hugging Face and the report is readable in an afternoon. Skip it if you need runnable inference and fine-tuning code today, a licence you can cite, or reproduced 0.8B results. Before building on it, check the repository for the promised inference and fine-tuning releases, because the todo list, not the file tree, is currently the project's contract.

Frequently asked questions

Official sources

1. Issues
2. orca-wm/Orca on GitHub
3. README