Open Scouts: Firecrawl's Self-Hosted AI Web Monitoring Stack

What Open Scouts actually automates

The problem is recurring web search with memory. You want to know when a new restaurant opens near you, when a topic shows up in AI news, or when a page changes, and you do not want to run the same query by hand every morning. Open Scouts packages that loop into a named object called a scout. The README describes scouts as "automated tasks that run on a schedule to continuously search for and track information", and the product promise is an email when something matches.

The audience is narrower than the README implies. This is a self-hosted Next.js application with a Supabase backend, so the person deploying it is a developer or a technically comfortable operator. There is no hosted signup flow described in the README, only a clone, a Supabase project, and a list of API keys. If you want a monitoring product you can buy and forget, this repository is not aimed at you.

The dispatcher, the agent and the vector store

The architecture is visible in the setup script's own description of what it does. Supabase is not just the database here. The README says setup configures a "scalable dispatcher architecture (pg_cron + pg_net + vault)", which means the schedule lives inside Postgres rather than in a long-running worker process. pg_cron fires the job, pg_net makes the HTTP call out, and supabase_vault holds the credentials so the database can authenticate that call without them sitting in plain text.

That call reaches an Edge Function named scout-cron, which is the agent runtime. The function uses the OpenAI API for the agent and for embeddings, and the Firecrawl SDK for search and scraping. Results land in tables the setup script creates: scouts, scout_executions, scout_execution_steps and others. The pgvector extension is enabled so execution summaries can be embedded and searched semantically, which is how the app surfaces past runs rather than making you read every step.

Two details are worth flagging. First, the repository contains a proxy.ts at the top level, so some request path is intercepted before it reaches the app, though the README does not explain what it does. Second, PostHog appears in the topics list and as .posthog-events.json and POSTHOG.md in the repository root, so analytics instrumentation ships with the code. For a self-hosted tool that stores your search history, that is a configuration decision you should make deliberately rather than inherit.

Installing Open Scouts and running a first scout

Start by cloning and installing dependencies. The README lists bun as the default package manager, with npm and pnpm as alternatives, and requires Node.js 18 or newer.

git clone https://github.com/firecrawl/open-scouts
cd open-scouts
bun install

Next, copy the environment template. The README states the .env.example file contains all required variables with instructions and links for obtaining each key, so open it rather than guessing at names.

cp .env.example .env

The variables you must fill in include NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, DATABASE_URL and OPENAI_API_KEY. The .env.example warns that SUPABASE_SERVICE_ROLE_KEY bypasses Row Level Security and must never be exposed in the browser.

Before running setup, enable the extensions in the Supabase Dashboard under Database, then Extensions. The README names four: pg_cron, pg_net, vector and supabase_vault. The setup script checks for the first three and stops with instructions if they are missing.

Link the project and run the setup script. You can find the project ref in the Supabase Dashboard URL.

bunx supabase login
bunx supabase link --project-ref <your-project-ref>
bun run setup:db

Finally deploy the two Edge Functions the README names. The scout-cron function is the agent runtime; send-test-email exists so you can verify mail delivery from the app's Settings page.

bunx supabase functions deploy scout-cron
bunx supabase functions deploy send-test-email

For a first real use, the README's recommended path is to get a Firecrawl API key from the dashboard and paste it into the Firecrawl Integration section of Settings, then click Save. It explicitly states that no Firecrawl environment variable is required for this path, because each user's usage is tracked to their own Firecrawl account. Add the Resend key the same way or via bunx supabase secrets set RESEND_API_KEY=re_..., then press Send Test Email in Settings and check the inbox. Only after that test arrives should you create a scout and trust the alert path.

Where the setup will fight you

The largest constraint is that Open Scouts is not one deployment, it is four. You need a Supabase project with specific extensions, a Next.js host, two deployed Edge Functions, and three third-party accounts. The README's setup script automates a lot of the database work, but it cannot enable extensions for you; it detects them and asks you to go click in a dashboard. That is a one-time cost, but it is a real one.

Email delivery has a hard boundary that the README states plainly. Without a verified domain in Resend, the service only sends to your Resend account email address. If you skip domain verification, your scouts will run and find things and the alerts will go nowhere useful. The free tier is documented as 3,000 emails per month with a 100 per day limit, which is generous for personal use and thin for a team.

The AI layer is also a cost surface rather than a fixed expense. Every scout execution calls OpenAI and Firecrawl, so a scout that runs frequently over broad queries burns tokens and scrape credits continuously. The README does not document a per-scout budget cap or a dry-run mode, so the only lever described is how often the scout runs and how narrow its prompt is. Design your scouts accordingly.

One more gap: the README does not document rollback, migration downgrades, or what happens to in-flight executions when you redeploy scout-cron. For a system whose whole job is notifying you, an execution that fails silently during a deploy is the failure mode to watch.

Open Scouts compared with a hosted change monitor

The obvious alternative is a hosted page-change monitor, where you paste a URL and get an email when a selector changes. The difference in approach is fundamental. A selector monitor watches a known page for a known change; it is deterministic, cheap, and useless when you do not yet know which page matters. Open Scouts inverts that. You describe what you are looking for, the agent searches, scrapes with Firecrawl, and reasons over what it finds, so it can surface pages you never named. That is the entire value and also the entire risk: an LLM deciding relevance is not the same as a diff, and the README offers no precision or recall figures to tell you how often it is right.

Against a general automation platform, the trade is different again. Open Scouts ships the scheduling, the storage, the vector search and the email path as one opinionated stack, which is less assembly than wiring a scraper into a workflow tool yourself. You pay for that convenience by accepting Supabase, OpenAI, Firecrawl and Resend as dependencies, and by accepting that the dispatcher lives in Postgres cron rather than in a queue you control.

Maintenance, licence and upgrade cost

The last push to the default branch was on 2026-05-22. That is roughly four months before today, so the repository is not archived but it is also not something to describe as under active development. Treat the dependency set as the maintenance surface: Next.js 15, React 19, Tailwind CSS v4 and the Supabase client libraries all move, and a self-hosted deployment means you own those upgrades. The package.json pins ranges rather than exact versions, so a fresh bun install months from now will not resolve to the same tree you tested.

Licensing is muddier than it should be. The package.json declares "license": "MIT", but no licence file appears in the top-level repository entries listed for this project, and the Topics list does not include a licence identifier. The README is silent on licensing entirely. Treat the MIT declaration in package.json as the project's stated intent and confirm it against whatever the repository actually ships before you depend on it commercially. That is a factual gap in the repository, not a legal opinion.

Upgrade cost is mostly Supabase-shaped. Because the dispatcher, the tables and the secrets live in your Supabase project, a schema change from upstream means re-running bun run setup:db against a database that already holds your execution history. The README describes setup as creating tables and configuring cron jobs, not as a migration tool, and the separate migrate script (tsx scripts/migrate.ts) is listed in package.json without documentation in the README. Back up before you upgrade.

Who should run this and who should walk away

Open Scouts fits a developer or small technical team that already operates Supabase, wants recurring AI-assisted web search, and is comfortable owning the API keys and the failure modes. It fits particularly well if you are already a Firecrawl user, since the per-user key model in Settings means each person's scraping is billed to their own account.

It does not fit anyone who needs a managed service, anyone who cannot verify a sending domain, or anyone who needs documented rollback and migration procedures before putting it in front of a team. The README's coverage of deployment is thorough on the happy path and thin on operations, and that asymmetry is the thing to weigh.

Editorial conclusion

Adopt Open Scouts if you already run Supabase and are willing to enable pg_cron, pg_net and vector, deploy two Edge Functions, and manage your own API keys for OpenAI, Firecrawl and Resend. Do not adopt it if you want a managed monitoring service or cannot verify a sending domain in Resend, because the free tier only delivers to your own Resend account address. Verify first that your Supabase plan allows the extensions and cron jobs the setup script expects, and confirm the MIT licence in package.json against any licence file you actually find in the repository.

Frequently asked questions

Official sources

1. firecrawl/open-scouts on GitHub
2. Issues
3. Project website
4. README