DeepResearcher: training a search agent against the live web with GRPO

DeepResearcher puts live web search inside the reinforcement learning loop

DeepResearcher is the official repository for a paper on scaling deep research with reinforcement learning. Its pitch is narrow and specific: instead of training an agent against a static retrieval corpus, the environment it rolls out against is the real web, with genuine search calls made during training. The README frames this as the first end-to-end framework for training LLM deep research agents in real world environments. The reported effect is a set of behaviours that appear without being programmed directly, namely planning, cross checking sources against each other, self reflection that redirects a search when the first trail goes cold, and admitting when the evidence is not there. The README reports gains of up to 28.9 points over prompt engineering baselines and up to 7.2 points over RAG based RL agents. A 7B checkpoint is published on huggingface-hub as GAIR/DeepResearcher-7b.

The handler process is where search keys and model keys live

The training loop does not talk to a search provider directly. It talks to a separately launched server handler, and the configuration for that handler lives in ./scrl/handler/config.yaml. Three keys are set there: serper_api_key for the Serper service, azure_bing_search_subscription_key for Bing, and search_engine to choose between them. The handler process itself, ./scrl/handler/server_handler.py, needs a separate qwen-plus API key, and the README shows it as an OpenAI style client with a base URL, which means you can point it at a compatible endpoint rather than the vendor default. The same config file carries server_url_list, the list of handler addresses that a training host uses to reach the handlers you have started. The consequence for an operator is that there are three pieces to wire up before a single training step runs, and each one fails separately, at a different moment, with a different message.

Ray has to be started by hand, even on one machine

Before any training or inference command, the documentation tells you to start Ray and to set PET_NODE_RANK first. The emphasis is unusual and deliberate: the README states the variable is compulsory even if you only have one node. On the head node the sequence is an export followed by the Ray start command. The rank value distinguishes the head from worker nodes, and setting it incorrectly leaves the job looking for a coordinator that never announces itself. This ordering also has a practical consequence for anyone scripting the environment: the handler processes are launched separately from the training host, so a single node still ends up running several long lived services before the trainer attaches to them. Nothing in the documentation describes a teardown procedure for those services.

Environment setup pins torch 2.4.0 and builds flash-attn from source

Installation is a conda environment on Python 3.10 followed by a set of pinned installs. The clone and environment creation come first, then the GPU stack, then the package itself and its dependency file.

git clone https://github.com/GAIR-NLP/DeepResearcher.git 
conda create -n deepresearcher python=3.10 
conda activate deepresearcher
cd DeepResearcher
pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn --no-build-isolation
pip3 install -e .
pip3 install -r requirements.txt

Two of those lines are where a fresh environment will hurt. The torch wheel comes from the cu124 index, so the GPU driver on the host has to match that CUDA generation, and the flash-attn install uses --no-build-isolation, which means the compiler toolchain and CUDA headers have to be present on the machine before pip starts. The requirements file adds a wide surface beyond training, including serpapi, google_search_results, ddgs, openai, flask, pymongo, smolagents, youtube_transcript_api, Bio, scikit-learn, chess and symbol maths libraries. That breadth suggests the agent was built to handle many tool types rather than only search.

Training is one script, evaluation is a rollout file and a metrics script

Training is launched with a single shell script, train_grpo.sh, which selects the GRPO trainer among the trainers vendored in the repository. Evaluation is split into three steps that you perform by hand. The first runs evaluate.sh to generate a rollout, and the resulting file lands at a path built from the project name and the experiment name. You then rename that file into the evaluate directory under the expected result name, and only then run the metrics script with the experiment name as its argument, which writes a score file you can read.

 bash train_grpo.sh

 bash evaluate.sh

 python ./evaluate/cacluate_metrics.py {experiment_name}

The manual rename between rollout and result is the fragile part. Nothing checks that the file you copied matches the experiment name the metrics script expects, so a mismatch surfaces as a missing score rather than as a clear error. The filename cacluate_metrics.py also carries a typo that you have to type exactly as written, which is the kind of detail that costs a first time user an hour.

DeepResearcher and a RAG pipeline differ in where the corpus comes from

The useful comparison here is not against another research agent framework but against the standard retrieval setup the README itself uses as a baseline. In a RAG pipeline, retrieval reads from a corpus that was indexed once, so the agent can only recall documents that someone decided to collect, and the same corpus is available during training and at inference. DeepResearcher moves the environment outside the system: the search happens against the live web during rollouts, and the reward signal comes from research behaviour that depends on results the framework does not control. The cost of that choice is operational. Every rollout consumes paid search quota, results are less reproducible because the corpus moves, and a training run depends on an external service staying up and staying funded. The README reports up to 7.2 points of gain over RAG based RL agents, which is a narrower and more meaningful comparison than the 28.9 point figure against prompt engineering.

Apache-2.0 code, a vendored verl tree, and no tagged releases

The repository is licensed under Apache-2.0, and the code credits its lineage: inspired by Deepseek-R1, with the implementation based on veRL and search-r1. The vendoring is visible in the tree, which contains a verl directory, a VERL_README.md, and a pyproject.toml whose distribution name is verl with a homepage pointing at the upstream project and authors listed as Bytedance Seed MLSys. In other words this repository embeds another framework rather than reimplementing it, so an upgrade of the training stack means tracking the vendored copy as well as this project. There are no GitHub releases to pin against, and the last push was on 2026-05-10, so the practical unit of adoption is a commit rather than a version. The dependency list pins vllm to 0.6.3 or lower and tensordict below 0.6, and those bounds are where an upgrade will hurt first.

Editorial conclusion

DeepResearcher fits research teams that already have GPUs, a paid search API and a Qwen endpoint, and that want to reproduce or extend reinforcement learning over real web search rather than a fixed corpus. It does not fit a team expecting a pip install and a chat demo, because the pipeline requires a Ray head node, a separately launched search handler and pinned GPU wheels. Before committing, check that the pinned vllm and tensordict versions still install on your CUDA stack, that your search provider quota survives real training rollouts, and that the 7B checkpoint on huggingface-hub is the base you intend to fine-tune.

Frequently asked questions

Official sources

1. GAIR-NLP/DeepResearcher on GitHub
2. Issues
3. License: Apache-2.0
4. README