# II-Researcher: a self-hostable deep search agent with a think-act-reflect loop

> II-Researcher is an Apache-2.0 Python framework that wraps web search, page scraping, context compression and referenced report writing into one agent loop. It is aimed at developers who want to own the whole research pipeline, and it costs you API keys for a search provider and a scraper.

**Intelligent-Internet/ii-researcher** — II-Researcher: a new open-source framework designed to aid building search / research agents

- Repository: https://github.com/Intelligent-Internet/ii-researcher
- Website: https://www.ii.inc/web/blog/post/ii-researcher
- Stars: 504 · Forks: 78
- Language: Python
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/intelligent-internet-ii-researcher

## The gap II-Researcher fills between a chatbot and a research script

Ask a general chat model a question that needs five sources and it answers from memory. Write your own script and you spend a week on retries, PDF parsing and context limits before you get a single cited paragraph. II-Researcher sits in that gap. The README describes it as "a deep search agent that browses the web, reasons over what it finds, and writes comprehensive, fully-referenced answers to hard questions", and the intended reader is a developer who wants that loop as a library rather than as a subscription.

The target user is fairly specific. You are comfortable setting environment variables, you already have or can obtain keys for a search API and a scraping API, and you want the agent's internals visible: which pages it visited, what it kept, what it discarded. If you want a finished product where someone else runs the crawler, this is the wrong layer. The project ships a CLI, a FastAPI backend, a Next.js frontend and an MCP server, but all four are front ends onto the same agent, so choosing II-Researcher means choosing to operate the agent.

## Inside the think-act-reflect loop and the compression stage

The architecture diagram in the README is explicit about the data flow. A question enters the reasoning agent, which runs a multi-step loop described as think, act, reflect, streaming its reasoning tokens as it goes. Each iteration can call two tools: web search through SerpAPI, Tavily or Jina, and page visit through Firecrawl, a headless browser, BeautifulSoup, Tavily Extract or Jina. Page handling includes dedicated paths for PDFs and YouTube, which is why pymupdf and yt-dlp appear in the dependency list.

The interesting stage is the one between scraping and the model. Retrieved text passes through context compression before it reaches the prompt: embedding similarity filtering with an optional LLM compressor on top. The .env.example exposes the knobs directly, with COMPRESS_EMBEDDING_MODEL defaulting to text-embedding-3-large, COMPRESS_SIMILARITY_THRESHOLD at 0.3, COMPRESS_MAX_INPUT_WORDS at 32_000 and COMPRESS_MAX_OUTPUT_WORDS at 6500. That is the design bet: rather than trusting a long-context model to ignore irrelevant page furniture, the project filters first and compresses second.

When the agent decides it has enough evidence, the report builder writes the final answer with references, in either basic or advanced form. Every model call is routed through a LiteLLM proxy, so the same pipeline can run on OpenAI, DeepSeek, Gemini, OpenRouter or a self-hosted OpenAI-compatible endpoint. Structured outputs come from BAML, which is pinned at baml-py==0.77.0 in pyproject.toml. That pin is worth noticing: BAML generates typed clients from schema files, so a version bump is not a no-op upgrade.

## Installing II-Researcher from PyPI and running a first question

The README gives two install paths, PyPI and source. Python 3.10 or newer is required. The PyPI route is a single command.

```bash
pip install ii-researcher
```

From source you clone the repository and install it in editable mode, which is what you want if you intend to modify the agent.

```bash
git clone https://github.com/Intelligent-Internet/ii-researcher.git
cd ii-researcher
pip install -e .
```

Configuration is entirely environment variables, and the repository ships a .env.example you can copy. At minimum you need a model key plus a search key and a scraper key matching whichever providers you select.

```bash
export OPENAI_API_KEY="your-openai-api-key"
export TAVILY_API_KEY="your-tavily-api-key"
export FIRECRAWL_API_KEY="your-firecrawl-api-key"
export OPENAI_BASE_URL="http://localhost:4000"
export SEARCH_PROVIDER="serpapi"
export SCRAPER_PROVIDER="firecrawl"
```

The README's comment on TAVILY_API_KEY is worth repeating: it is required when SEARCH_PROVIDER=tavily, and SERPAPI_API_KEY when SEARCH_PROVIDER=serpapi. The same conditional logic applies to FIRECRAWL_API_KEY and SCRAPER_PROVIDER=firecrawl. Setting a provider without its key is a configuration error you will hit at the first search call, not at startup.

If you prefer containers, docker-compose.yml defines four services: frontend on port 3000, litellm on 4000, api on 8000, and the api service depends on litellm. The api service runs python api.py and receives OPENAI_BASE_URL=http://litellm:4000, so inside Compose the model traffic goes through the bundled proxy rather than out to a vendor directly. Bring it up and the web interface is at localhost:3000, with the SSE backend at localhost:8000.

The README also documents an MCP server for Claude Desktop and other MCP clients. That is the shortest path if you want to test the agent's behaviour before writing any integration code, because the client handles the conversation surface and you only supply the keys.

## Where II-Researcher breaks or is simply the wrong tool

The dependency on external providers is the first real constraint. There is no bundled search index and no bundled crawler in the repository layout. Every question the agent answers is billed by at least two third parties, and if a provider changes its response shape or rate limits you, the failure surfaces inside the agent loop rather than at a clean boundary. The README does not document a fallback chain that switches providers automatically, so a dead provider means editing configuration.

The second constraint is cost and latency shape. The loop is iterative by design: search, scrape, compress, reflect, repeat. Each reflection is a model call, each scrape may pass through an embedding model, and USE_LLM_COMPRESSOR defaults to TRUE in .env.example, which adds another LLM call per compression. The timeout variables in the example file (SEARCH_PROCESS_TIMEOUT=300, SEARCH_QUERY_TIMEOUT=20, SCRAPE_URL_TIMEOUT=30) tell you the authors expect individual steps to be slow. A question that needs one lookup is a bad fit; you are paying for a loop you do not need.

The third is that this is beta software by its own classification. pyproject.toml carries the classifier "Development Status :: 4 - Beta". The README does not document rollback behaviour for a failed report, nor what happens to a partially written report if the process dies mid-stream. Version 0.1.6 in pyproject.toml sits above the 0.1.5 release listed on the releases page, so the published version and the repository state are not the same thing. If you need a stable interface to build on, pin a version and read the diff before moving.

## How II-Researcher differs from GPT Researcher and from a plain search API

GPT Researcher is the obvious comparison: both are open-source Python research agents that take a question, search the web and return a cited report. The difference in approach shows up in two places. II-Researcher routes every model call through a LiteLLM proxy and exposes named model slots (STRATEGIC_LLM, SMART_LLM, FAST_LLM, R_MODEL, R_REPORT_MODEL in .env.example), so the pipeline is designed to be split across different models per stage rather than pointed at one. It also treats context compression as a first-class stage with its own embedding model, similarity threshold and word budgets, which is a different answer to the long-page problem than simply choosing a bigger context window.

A plain search API is the other alternative, and the comparison is honest: Tavily and SerpAPI both return ranked results with snippets, and if your question can be answered from snippets, II-Researcher is overhead. What it adds is the visit-and-read step, the reflection that decides whether more evidence is needed, and the report builder that attaches references. You should reach for the raw API when the answer fits in a snippet, and for II-Researcher when you need the agent to open the page and defend a conclusion.

## Licence, maintenance and what an upgrade actually costs

II-Researcher is Apache-2.0, and pyproject.toml declares the license as a file reference to LICENSE. Apache-2.0 permits commercial use and modification and includes a patent grant, but the usual obligations apply: keep the licence and notice files, and state significant changes if you redistribute. One dependency deserves a flag rather than legal advice: clean-text is pulled in with the [gpl] extra in pyproject.toml, which selects GPL-licensed components. If you plan to redistribute a modified II-Researcher, read what that extra actually installs before you ship.

Maintenance signals are mixed. The repository is not archived, and the last push was on 2026-07-02. The most recent listed release is v0.1.5 from 2025-05-09, while pyproject.toml declares version 0.1.6, so the project has moved between releases without tagging. The README's news section stops at 2025-08. Treat the release cadence as slow and verify the state of the main branch yourself.

Upgrade cost is dominated by two pinned surfaces. baml-py is pinned to an exact version, so a BAML change is a deliberate migration rather than a routine bump. The environment variable set is wide: the Compose file alone forwards more than twenty variables, and .env.example supplies defaults for most of them. Any upgrade that renames a key will fail silently at the point where the default takes over, which is the kind of failure that shows up as worse answers rather than as an error. Diff .env.example against your own file on every upgrade.

## Conclusion

Adopt II-Researcher if you want the research loop itself, not a hosted answer box: you get the reasoning agent, the compressor and the report builder as Python you can edit, plus a CLI, an SSE API, a Next.js UI and an MCP server. Do not adopt it if you expect one install to work with no third-party accounts, because SEARCH_PROVIDER and SCRAPER_PROVIDER each need their own key, and the README does not document a provider path that avoids them. Before committing, verify two things on your own machine: that your chosen search and scrape providers still answer, and that the LiteLLM route at OPENAI_BASE_URL resolves the model names you put in R_MODEL and R_REPORT_MODEL. The repository's last push was on 2026-07-02, so check the issue tracker for anything filed since then.

## FAQ

### What is II-Researcher?

It is an open-source deep search agent that searches the web, reads pages, reflects on what it finds and writes a referenced answer. The repository ships it as a Python library and CLI, a FastAPI backend, a Next.js web UI and an MCP server.

### How do I install II-Researcher?

Install from PyPI with pip install ii-researcher, or clone the repository and run pip install -e . for an editable install. Python 3.10 or newer is required.

### Which search and scraping providers does II-Researcher support?

Search runs through SerpAPI, Tavily or Jina, selected with SEARCH_PROVIDER. Scraping runs through Firecrawl, a headless browser, BeautifulSoup, Tavily Extract or Jina, selected with SCRAPER_PROVIDER, with separate handling for PDFs and YouTube.

### Can II-Researcher run with a self-hosted or non-OpenAI model?

Yes. All model calls route through a LiteLLM proxy, and the README states that OpenAI, DeepSeek, Gemini, OpenRouter or any OpenAI-compatible endpoint, including self-hosted models, can power every stage. You point OPENAI_BASE_URL at your proxy.

### What is the licence for II-Researcher?

The project is Apache-2.0, and pyproject.toml references the LICENSE file. Note that the dependency list pulls in clean-text with the [gpl] extra, so check what that extra installs if you plan to redistribute.

## Sources

- [Intelligent-Internet/ii-researcher on GitHub](https://github.com/Intelligent-Internet/ii-researcher)
- [License: Apache-2.0](https://github.com/Intelligent-Internet/ii-researcher/blob/main/LICENSE)
- [Project website](https://www.ii.inc/web/blog/post/ii-researcher)
- [README](https://github.com/Intelligent-Internet/ii-researcher/blob/main/README.md)
- [Releases](https://github.com/Intelligent-Internet/ii-researcher/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/intelligent-internet-ii-researcher