auto-deep-researcher-24x7: A Leader-Worker Agent That Keeps Running Your Training Jobs

The gap between a training script and a research loop

Most people running deep learning experiments already have the hard part: a script that trains, a dataset, and a GPU. What they do not have is a process that decides what to try next at 3am, launches it, notices it died, and writes down that it died so the same idea is not proposed again on Tuesday. That bookkeeping is the target of this project. The README frames it as an agent that runs experiments 24/7 while you sleep, and the repository is organised around the loop rather than around any particular model architecture. The intended user is someone with an existing workspace of training code, not someone starting from an empty directory. The agent reads and edits that code, submits runs, and records outcomes. It does not supply the dataset, the model, or the hypothesis space you care about.

Leader-Worker: who decides and who executes

The architecture is split into a leader and a worker. The leader plans: it takes the current state, the experiment ledger, the journals, and any stagnation signal, and produces the next action. The worker carries that action out through a multi-turn tool-use loop, with tool results handed back to the model as authoritative rather than paraphrased. The release notes describe a 2026-04-19 change that added this real multi-turn loop along with stricter CLI behaviour and safer tool-call parsing, which tells you the earlier version was shakier about parsing what the model asked for. Execution is abstracted behind a backend selected by execution.mode. In local mode the controller runs commands on the same machine. In ssh mode the controller stays put and edits, training, log reads, PID checks, and GPU queries happen on one remote host. In slurm mode the controller submits with sbatch --parsable over a single transient SSH call that exits immediately, so nothing is left running on the login node, and sacct is treated as the only authority on whether a job is alive. That last detail matters because it is the difference between a monitor that polls a process table and one that asks the scheduler.

Memory that costs no tokens: the ledger and the journals

The v2.0 release added an experiment ledger at workspace/experiments.jsonl. Each cycle appends its hypothesis, metrics, and outcome. The file is described as crash-safe and zero token cost, which is the point: the agent's memory of what it tried lives on disk, not in the context window, so the context does not grow with the number of experiments. Two append-only journals sit alongside it. DEAD_ENDS.md records failed approaches so they are not retried, and INSIGHTS.md holds observations worth keeping. Neither is compacted; when they grow large they are rotated to dated backups, so history is not silently dropped. A separate stagnation signal reads the metric trajectory out of the ledger, keyed on ledger.metric_key, and tells the planner whether results are still improving or have flattened. That is a more useful input than a binary repeat counter, though it only works if the metric you name is actually written by your training script in a form the ledger can parse.

Getting it running: providers, execution mode, and the config keys that matter

The README gives a Quick Start section and a config.yaml with optional sections named ledger, stagnation, journal, safety, gates, and an agent.max_cycles_per_hour key. All of them default to the previous behaviour, so an existing project keeps working after an upgrade. Provider selection is the first decision. You can run against a subscription-backed claude_cli or codex_cli provider, or against an OpenAI-compatible endpoint. The 2026-06-03 update added one-word presets for Chinese vendors, so setting agent.provider to deepseek, qwen, kimi, or glm auto-fills the base_url and the default key environment variable, with base_url and api_key_env left overridable for self-hosted or proxied endpoints. The documented example is short: agent.provider set to "deepseek" and agent.model set to "deepseek-chat". For cluster users the second decision is execution.mode, which takes local, ssh, or slurm. The release notes state that the new gate and rate limit are opt-in, and that the test suite runs without a GPU or network, growing from 60 to 99 tests across the v2.0 changes. I have not installed or run any of this; the commands and keys above are what the documentation states.

The failure that motivated final_status, and what it still cannot see

Before 2026-06-02, a run that failed, timed out, or was cancelled could be recorded as completed. The fix routes a finished job's real terminal state through a final_status() call so the outcome reaches state.json, the ledger, and the REFLECT context. On Slurm the state comes from sacct. On pid-only backends, meaning local and ssh, the documentation says the state is reported as indeterminate and prior behaviour is kept. That is the honest limitation: if you run in local or ssh mode, the agent cannot tell you whether your training job converged or crashed, only that the process ended. You are back to reading logs yourself, which is the work the tool was supposed to absorb. The Slurm path also carries its own constraints. The liveness check has two bounds, a consecutive-unknown grace period and a wall-clock backstop derived from --time, so the monitor loop terminates even if the cluster becomes unreachable. The trade-off is that termination is not the same as correctness: a job the agent stops tracking may still be queued or running, and the notes are explicit that the check never reaps a job sacct still reports as active.

Where it is the wrong tool, and what a plain scheduler does instead

If your experiments are a fixed grid of hyperparameters, this is more machinery than you need. A Slurm array job or a shell loop over a config file launches the same set of runs, costs nothing in API calls, and has no failure mode where a language model misreads a log. The difference in approach is that a scheduler executes a plan you wrote, while this agent writes the plan. That is the entire value proposition and also the entire risk: the agent can propose a hypothesis you would not have, and it can also spend an afternoon on a dead end before the ledger records it. The stagnation signal and the optional agent.max_cycles_per_hour cap exist precisely because the second case is real. A team running a well-understood ablation study should reach for the scheduler. A team whose bottleneck is deciding what to try next, and who already has the execution path working, is the audience here.

Maintenance cost and the licence

The project ships no releases according to the repository metadata, so there is no tagged version to pin against. Installation means tracking the main branch, and the changelog shows why that matters: within roughly eight weeks the repository added an SSH backend, a Slurm backend, a ledger, two journals, a stagnation signal, a safety scanner, and a change to how experiment outcomes are recorded. Each of those is described as additive and backward-compatible, and the config sections default to prior behaviour, which limits the blast radius of an upgrade. The cost is that you are reading release notes to know what your agent will do differently this week. The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant, with the usual requirement to preserve notices and state changes; the repository does not carry a separate NOTICE file in the material reviewed here, so confirm that before redistributing. This is a description of the licence text, not legal advice.

What to check before you point it at a GPU

Three things are worth verifying against your own setup. First, that execution.mode matches reality: slurm mode assumes sacct is the liveness authority and that the login node shares an NFS workspace with the compute nodes, which is stated in the release notes and is not universal. Second, that ledger.metric_key names a metric your training script emits in a parseable form, because the stagnation signal is only as good as that trajectory. Third, that the provider preset you choose resolves to an API key you are willing to spend without a hard ceiling, since agent.max_cycles_per_hour is optional and defaults to the old uncapped behaviour. The 24/7 claim in the README is a description of the loop, not a guarantee about your budget.

Editorial conclusion

Adopt it if you already have a working training script and a machine or Slurm partition to point it at, and you want the cycle-by-cycle record in workspace/experiments.jsonl rather than in your head. Do not adopt it if the research question itself is still open, or if you cannot give it a bounded API budget, since the agent will keep proposing hypotheses until agent.max_cycles_per_hour or your key stops it. Verify first that execution.mode matches your actual infrastructure, that ledger.metric_key names a metric your training script really writes, and that your provider preset resolves to a key you are willing to burn.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. Xiangyue-Zhang/auto-deep-researcher-24x7 on GitHub