SurfSense: a typed REST and MCP surface for agent web research

The gap SurfSense targets: agents have no trustworthy place to look

A capable agent can reason over a static index, but it cannot answer a question about what Reddit said this week or what ten store locations are actually complaining about. The README frames the problem in operational terms: official platform APIs are rate-limited, priced for enterprises, or absent, scraping plumbing breaks, and driving a browser with a model costs minutes and tokens per page. SurfSense exists to remove that plumbing from the agent loop. The intended user is not a person browsing a notebook. It is a developer or an agent framework that needs structured records from social platforms, marketplaces, search results and job boards, delivered as JSON. The README's own note to existing users makes the shift explicit: reasoning over a static index is becoming something every capable agent does out of the box, so the project is pointing its energy at live data and the workflows around it. That is a candid admission that the original NotebookLM framing no longer describes where the effort goes.

One typed surface: connectors as REST endpoints, tools over MCP

The mechanism is uniform across sources. Every connector is a REST endpoint that returns structured JSON: posts, comments, transcripts, reviews, SERPs or cleaned page content. There is no HTML parsing step for the caller and no browser session to manage. The same catalog is exposed through an MCP server, where each connector becomes a named tool such as surfsense_reddit_scrape or surfsense_google_search, usable from Claude, Cursor or another agent framework. On top of the raw endpoints sits what the README calls an agent harness: retries, structured output and credit metering are handled by the platform, so a caller goes from a question to a cited brief without writing its own retry logic. The connector list covers Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, Walmart and a general web crawl, plus external MCP connectors for bringing in servers such as Notion, Slack or Jira with one-click OAuth. The README states the catalog is growing and that new sources land on the same API and MCP server, which matters more than any single connector: the interface stays fixed while the source list changes.

Calling a connector: the curl shape and the MCP config block

The README's quick start gives one concrete example. A POST to /workspaces/$WORKSPACE_ID/scrapers/reddit/scrape carries a bearer token and a JSON body with search_queries, community, sort and time_filter. The four body keys shown are the whole visible surface for that endpoint: an array of queries, a community name, a sort mode and a time window. Connector pages are said to carry copy-paste examples in Python, JavaScript, Go, PHP, Ruby, Java and C#, so the REST path is the portable route. For agent use, the MCP configuration is a small JSON object with a url of https://mcp.surfsense.com/mcp and an Authorization header holding the API key. Two environment variables appear in the example: SURFSENSE_API_URL and SURFSENSE_API_KEY, plus WORKSPACE_ID in the path. The README does not show a self-hosted deployment command, a docker compose file or an environment file, so anyone planning to run it on their own infrastructure has to get that from the repository rather than from the README. That is a real gap in the documentation as supplied.

Billing is metered per item, and self-hosted installs run with it off

The pricing model is stated plainly. Connectors bill per item actually returned, crawls bill per page successfully fetched, and failed calls are never billed. Self-hosted installs run with billing off. That last sentence is the most consequential line in the README for anyone evaluating cost, because it means the metered path and the free path are the same software with different accounting. It also means the hosted service is where the operational burden sits: rate limits, retries and upstream breakage are someone else's problem when you call the hosted endpoint. The trade-off is that per-item billing scales with research volume rather than with seats, which suits bursty agent workloads and penalises broad crawls. The README does not publish rates, so the only honest statement is that the unit of billing is defined and the price is not.

Where it is the wrong tool, and where the documentation is thin

SurfSense is the wrong choice if your research corpus is documents you already have. The project's own note says the general research agent chapter is behind it and that agents now handle reasoning over a static index themselves. Buying connector capacity for a question that never leaves your own files adds a dependency for nothing. A second limitation is legal and operational rather than technical: several connectors are described as working without an official API, without Graph API access or without Research API approval. The README presents that as the point, and it is, but it means the reliability of those sources depends on the upstream platform not changing its defences, and the README does not describe how often connectors break or how quickly they are repaired. Third, the licence is reported as NOASSERTION, so the terms under which you may use, modify or redistribute the code cannot be determined from the repository metadata alone. Fourth, the README truncates mid-sentence in the MCP section, and no self-hosting instructions, resource requirements or database dependencies appear in the supplied text. Treat the deployment story as unverified until you read the repository.

How this differs from Firecrawl and from a general-purpose agent

The closest comparison is a general scraping and crawling service such as Firecrawl. The difference is in the shape of the output. A crawler takes a URL and returns page content, leaving you to decide which URLs matter and how to extract fields from each one. SurfSense takes a query and a source, and returns domain-shaped records: posts with comments, videos with transcripts, products with reviews and sellers, job postings with salaries. You do not write extraction logic per site, and you do not enumerate URLs. The cost of that convenience is scope. If your target is a site with no connector, the web crawl endpoint is the fallback, and at that point the advantage over a plain crawler narrows to the harness around it. The second comparison is a coding agent with a browser. That approach handles arbitrary pages but pays per page in time and tokens, and the README's own framing of browser loops as expensive is the argument for a typed API. Neither comparison is settled by the README, but the design intent is clear: move extraction out of the agent and into the endpoint.

Release cadence and the cost of staying current

The repository shows three releases in the weeks before this writing: v0.0.37, v0.0.38 and v0.0.39, the last dated 2026-08-29, with the most recent push to main on 2026-09-10. A 0.0.x version series moving that quickly is a signal about upgrade cost. Connector code is coupled to the current shape of external platforms, so a fast cadence is appropriate here, but it also means the API surface can move between releases. Anyone pinning a version should read the release notes for each step rather than jumping, and anyone self-hosting should expect to redeploy on a similar rhythm to stay ahead of upstream changes. The README points to a changelog on the project site for the repositioning announcement, which is where breaking changes would be described. Maintenance effort is therefore not a one-off integration cost. It is a recurring one, and it is the price of depending on platforms that do not offer stable public interfaces.

Editorial conclusion

Adopt SurfSense if your agents need current posts, transcripts, reviews or job listings and you would rather call a typed endpoint than maintain scraping plumbing and a browser loop. Do not adopt it if your research is confined to documents you already own, because the project has repositioned itself away from general knowledge work. Before committing, verify the licence terms, since the repository reports NOASSERTION, and check that the connectors you need are present in the release you deploy.

Official sources

1. Issues
2. MODSetter/SurfSense on GitHub
3. Project website
4. README
5. Releases