BigSet: self-hosted live datasets built by TinyFish web agents

What BigSet is for, and who ends up using it

The README opens with a sentence you could type into it: "YC companies that are currently hiring engineers, with their funding stage, location, and number of open roles." That is the whole pitch. You describe a dataset in plain English, and BigSet infers a schema, sends agents to research the live web, verifies what they find, deduplicates, and returns a structured table you can download as CSV or XLSX.

The project positions itself against a stack you would otherwise assemble by hand. Scraping frameworks extract from URLs you already know. Search APIs return ranked links. Pre-built actors cover specific sites. BigSet's claim is that the moment your dataset cuts across those categories, you are back to stitching search, extraction, schema design, deduplication, verification and a cron job together yourself, once per dataset. That stitching is the problem it targets.

Who is this for? Engineers and analysts who need a small, recurring table from public pages and do not want to own a scraping pipeline. It is also aimed at agent builders who want live data to reason over. The README is explicit that it is experimental: "It works, sometimes surprisingly well, but expect rough edges." Treat that as the operating assumption, not marketing hedging.

The agent fan-out behind each dataset

The mechanism has five named stages. First, schema inference: column names, types, primary keys, and where on the web to look. Second, an orchestrator agent discovers entities through web search. Third, sub-agents fan out in parallel, one per entity; each investigates a single entity, fetches real data, and inserts a verified row. Fourth, you get a structured table you can browse in the UI and export. Fifth, you set a cadence and the agents re-run on schedule.

The parallel sub-agent step is where the design gets interesting and where the cost sits. One dataset of N entities is N independent research tasks, each doing searches and page fetches. The README warns that dataset generation takes 2 to 5 minutes because "the agents are doing real web research: searching, fetching pages, verifying data." That is a per-dataset latency floor, not a per-row one, and it is the honest number to plan around.

The two external services map cleanly onto the two kinds of work: TinyFish handles web search and page fetching, OpenRouter handles LLM calls for schema inference and the agents themselves. That split matters for failure analysis. A bad schema is an LLM problem; an empty dataset is more likely a fetching or search problem.

Installing BigSet and creating your first dataset

The README lists one prerequisite: Node.js 22+ with npm. No Docker. The global install pulls the current local release, starts Convex, the backend, the frontend and a local credential bridge, then prints the app URL.

npm install --global @adamexu/bigset
bigset

Open 127.0.0.1:3500 in a browser. The first run caches release files under ~/.bigset; later starts are described as taking only a few seconds. If you would rather not install globally, the README gives a one-off form:

npx @adamexu/bigset

On first launch you are sent to setup and asked to connect two services: TinyFish for web search and page fetching, and OpenRouter for LLM calls. Keys are stored in your OS keychain. Once running, the CLI can create a dataset, wait for population, and write CSV in one command:

bigset create "fintech startups in the bay area" --rows 10 --wait --csv fintech.csv

Other CLI verbs the README documents are `bigset list`, `bigset status <datasetId>`, `bigset rows <datasetId> --json`, `bigset export <datasetId> --csv out.csv`, `bigset populate <datasetId>`, and `bigset stop <datasetId>`. Useful local options include `bigset --force` to redownload the cached release, `bigset --app-port 4500 --backend-port 4501` for alternate ports, and `bigset --home ~/.bigset-dev` for a separate cache directory. Create flags include `--rows <n>`, `--wait`, `--csv <path>`, `--skip-populate`, and `--cadence`.

Where BigSet breaks: schema drift, paywalls, and the missing query layer

The README names its own limitations, and they are the ones that will bite. Schema inference "isn't always perfect, and some topics work better than others." That is a soft warning for a hard problem: if the inferred columns are wrong, every row is wrong in the same way, and you find out after the 2 to 5 minute generation run.

Data behind logins or paywalls is out of reach. So a dataset about pricing that lives behind a signup wall, or inventory behind an account, is the wrong fit regardless of how well the agents work.

The sharpest constraint is that "datasets are downloadable, not queryable." You browse in the UI and export CSV or XLSX; SQL query support is on the roadmap. If your workflow assumes a database you can join against, BigSet is not that. It is a table generator with a refresh timer.

There is also a refresh cost question the README does not answer. A dataset on a 30-minute cadence re-runs its agents every 30 minutes, and each run consumes TinyFish and OpenRouter calls. The documentation does not publish a cost model per refresh, so you should measure your own before setting anything tighter than daily.

BigSet against a scraping framework plus a cron job

The obvious alternative is a scraping framework pointed at URLs you choose, plus a scheduler. The difference is who decides what to fetch. A framework executes the plan you wrote: these pages, these selectors, this schedule. BigSet does the opposite. You supply a sentence, and the agents decide which pages to search, which to fetch, and which rows to keep.

That inversion is the whole trade. You gain coverage of sources you would never have enumerated, and you lose determinism. A selector-based scraper returns the same shape every run until the site changes. BigSet's schema is inferred per dataset, and the README's own caveat about imperfect inference means the shape can vary. For a one-off exploratory table, the flexibility wins. For a pipeline feeding a downstream system with a fixed contract, the framework wins.

The second difference is verification. BigSet's sub-agents insert rows only after verifying against real sources and deduplicating. A hand-rolled scraper gives you whatever the page said. Whether that verification is worth the extra latency and API calls depends on how much you trust the source pages.

Licence, maintenance, and what an upgrade costs you

BigSet is licensed AGPL-3.0. If you modify it and offer it as a network service, the licence's source-availability obligations apply to your modified version. That is a real consideration for anyone planning to wrap this in a hosted product, and it is a reason some teams will rule it out early. This is not legal advice; read the licence text in the repository.

The repository is not archived, and the last push was on 2026-09-15. Releases are early: v0.1.1 on 2026-06-09, v0.1.2 the same day, v0.1.3 on 2026-06-10. Three patch releases inside about 24 hours, then a gap before the September push, is the shape of a project still finding its footing. The README's own warning about shipping fast and things breaking is consistent with that.

Upgrade cost is low on the surface: `bigset --force` redownloads the cached release, and the cache lives under ~/.bigset. The risk is schema and data compatibility across versions, which the README does not address. It does not document rollback, migration of existing datasets, or what happens to a dataset created under an older release. Back up exports before upgrading if the data matters.

Editorial conclusion

Adopt BigSet if you need a small, refreshable table assembled from public web pages and you are comfortable running an experimental AGPL-3.0 stack on your own machine with a TinyFish key and an OpenRouter key. Do not adopt it if you need SQL queries, data behind logins or paywalls, or a stable schema you can depend on for months. Before committing, verify three things: that your target topic has public web pages the agents can fetch, that the inferred schema matches the columns you actually need, and that the refresh cadence you pick (30m to weekly) is one the release you installed supports, since the README's flag table is truncated at the cadence row.

Frequently asked questions

Official sources

1. Issues
2. License: AGPL-3.0
3. README
4. Releases
5. tinyfish-io/bigset-oss on GitHub