Reef: continual learning infrastructure that versions agent updates like git commits

The gap Reef claims: serving and training frameworks both stop short of versioning

Inference engines such as vLLM and SGLang serve live traffic but do not train weights. RL training frameworks such as Slime, veRL and AReaL train weights but do not serve live traffic. Neither group, per the README's comparison table, manages versions or stays live through updates. Reef positions itself in the fourth column of that table, claiming all five rows: serving, training, version management, staying live through updates, and evolving things that are not weights at all, meaning prompts, rules and skills.

That last row is the interesting one. Most agent stacks treat the harness (the prompt template, the tool rules, the skill definitions) as source code edited by a human and deployed like any other change. Reef treats it as a learning surface alongside model weights. The README states plainly that a deployment's recipe determines which surface its scenarios update, so the same serving endpoint can be pointed at weight training or at harness optimization depending on configuration. The intended user is a team that already has one half of the loop working and wants the other half without hand-rolling a feedback store, an eligibility filter and an artifact registry.

The four-step loop and which directory implements each step

Reef describes each learning cycle as four steps, and the README maps each to a module path. Step one, Serve: agent requests arrive and interactions are recorded, handled by reef/service for requests and records and reef/runtime for inference and artifact updates. Step two, Observe: feedback is matched back to recorded interactions, handled by reef/records.py for storage and reef/train/processors for matching and eligibility. Step three, Grow: an update is produced from the eligible records, via reef/recipe for recipe integration and reef/train for batches and update jobs. Step four, Commit: a selection policy is applied, candidates are evaluated in reef/train/evaluation, the version history lives in reef/artifact, and delivery happens through reef/surface.

The mechanism that makes this more than a pipeline diagram is the receipt. When a request hits the inference endpoint, Reef adds an x-reef-agent-record-id response header. That value is what a later report cites to identify which interaction it is judging. The README calls it the receipt. Feedback therefore arrives out of band, possibly seconds or minutes later, and the Observe step's job is to join it back to the right record and decide whether the pair is eligible for training. Eligibility is not automatic: it is computed in train/processors, which means a scored interaction can be recorded, matched, and still never influence an update.

Scenarios are created by header, not by configuration file

A request carries the x-reef-scenario header. According to the README, a new scenario name creates a scenario using the deployment's configured recipe. Requests do not select recipes. This is a deliberate constraint worth noting: you cannot route one request to weight training and the next to harness optimization by changing a header. The recipe is fixed at deployment level, and the header only partitions traffic into scenarios within it.

That design keeps the training path predictable, but it means experimenting with two learning surfaces requires two deployments. For teams evaluating whether harness optimization or weight training pays off more on their traffic, that is a real cost in GPU time and operational surface. The README does not describe any mechanism for running multiple recipes in one deployment.

Getting it running: install, serve, and report

Reef's artifact and checkpoint functionality requires the git-lfs system package, and the README notes that Reef initializes Git LFS locally for its artifact repositories. That is a system-level dependency, not a pip extra, and it belongs in your container image before anything else.

From PyPI, the documented sequence is uv venv && source .venv/bin/activate, then uv pip install reef-infra, then a version check with python3 -c "import reef; print(reef.__version__)". From source, the README adds git lfs install before cloning, then uv pip install -e . in the checkout. The source path is the one the training examples assume.

For the weight-training path, the README gives the SAO example (arXiv:2607.07508). You install the extra and the runtime group: uv pip install -e ".[slime]" followed by uv pip install --no-deps --group runtime. Then you export MODEL_PATH and REEF_TOKEN, and start the server with reef serve -c recipes/sao/examples/sao/serve.yaml --reef.model_path "$MODEL_PATH" --reef.port "8900". Readiness is checked with curl -f http://127.0.0.1:8900/healthz.

The inference endpoint is OpenAI- and Anthropic-compatible: /v1/chat/completions and /v1/messages accept the provider's own request body. In the README's Python example, a client posts to /v1/chat/completions with a model and messages array, reads the receipt from the x-reef-agent-record-id response header, then posts to /reef/report with a JSON body containing score, feedback and a references array holding the receipts being evaluated. The example reports both a numeric score and a short textual explanation. Note that the README text is truncated inside that final report call, so the exact complete shape of the references payload is not fully shown in the supplied material.

The harness path exists to avoid GPUs, and that is the real selling point

The README's goal table separates three paths by what you need. Weight training needs a trainable model, a supported GPU stack, and feedback your recipe can use. Harness optimization needs a model endpoint, representative tasks, and an evaluator, with no local training GPUs. Test-time training needs an execution environment, a correctness checker, and a measurable objective.

For a large share of teams, the harness path is the practical one. Prompts, rules and skills can be revised and evaluated without renting accelerators, and the feedback signal is an evaluator you write rather than a reward model you train. The trade-off is that harness optimization is bounded by the quality of that evaluator. If your evaluator is a string match, as in the README's "Return exactly: reef is ready" example, the loop will optimize toward exact string matching, which is a narrow objective. The README does not describe built-in evaluators, so this is work you supply.

Where Reef is the wrong tool

Reef assumes a live service. If you are doing offline fine-tuning on a static dataset, the serve, observe and commit steps are overhead: you already have your records, you already have your labels, and you do not need receipts or a selection policy. A standard training run plus your own checkpoint directory is simpler.

Reef also assumes you can evaluate candidates before committing them. The Commit step applies a configured selection policy and publishes accepted updates, which implies a way to compare a candidate against the incumbent. If your objective is subjective, expensive to measure, or only observable long after deployment, the selection policy has nothing solid to stand on. The README does not describe a fallback for that case.

The git-lfs dependency is a second constraint. Artifact and checkpoint functionality requires it, and Reef initializes Git LFS locally for artifact repositories. In environments where Git LFS is unavailable, blocked, or where large binary objects in git are prohibited by policy, the artifact and versioning half of Reef does not function. Since version management is one of the five rows Reef claims in its own comparison table, losing it removes a large part of the reason to choose Reef over wiring a serving engine to a training framework yourself.

Compared with an RL framework plus a hand-rolled feedback store

The obvious alternative for the weight-training path is to use Slime, veRL or AReaL directly and write your own glue: a table of interactions, a feedback endpoint, a join query, and a script that exports eligible rows into the training job's expected format. That is not exotic. Many teams already have it.

The difference in approach is where the state lives. In the hand-rolled version, the training framework owns the run and your glue produces inputs for it. In Reef, the deployment owns the loop: the server that answers /v1/chat/completions is the same system that records the interaction, matches the later report, produces the update, evaluates it, and writes it into a version history under reef/artifact. The training framework becomes a component behind reef/recipe rather than the center of the system. Whether that inversion is worth it depends on how much you care about the version history and about staying live through updates, which are the two rows the README says neither inference engines nor training frameworks provide.

Maintenance, maturity and licence

Reef is at v0.0.2, tagged 2026-09-02, with the last push to main on 2026-09-10. That is a pre-1.0 project with a single tagged release. The README links a roadmap issue for direction, and the documentation lives at reefinfra.ai rather than in the repository, so the repository alone does not tell you the full intended surface. Expect the recipe layout and configuration keys to move.

The practical maintenance cost is concentrated in two places. First, the recipe: the SAO example is a directory of YAML under recipes/sao/examples/sao/, and any change to how train/processors computes eligibility or how train/evaluation compares candidates will likely require recipe changes. Second, the artifact store: Git LFS repositories grow, and the README does not describe a retention or pruning mechanism for old versions. Upgrading means re-checking the GPU requirements page before bumping, since the slime extra and the runtime dependency group are installed separately and could drift.

Licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is permissive and standard for infrastructure. This is not legal advice; if you redistribute Reef or bundle it into a product, read the NOTICE and patent-termination clauses with your own counsel.

Editorial conclusion

Adopt Reef if you already run an OpenAI-compatible endpoint or a Slime/SGLang GPU stack and the missing piece is the loop that turns scored interactions into versioned artifacts you can roll back. Do not adopt it if you need a mature, documented release: v0.0.2 is the only tagged version and the README itself is truncated mid-example. Before committing, verify that git-lfs is installed on every host that will hold artifact repositories, confirm your GPU environment matches the Evolve your model requirements page, and check whether your recipe's feedback shape is one the train/processors eligibility logic actually accepts.

Official sources

1. Human-Agent-Society/reef on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases