# DeepCrawl and the crawl-shaped API an agent can actually call

> DeepCrawl is an open-source Firecrawl alternative that returns cleaned Markdown and a hierarchical links tree from a page, aimed at agent workloads rather than at scraping defences. The interesting parts are the monorepo structure with separate API, auth and database workers, and the very explicit warning at the top of the README that it should not be used in production yet.

**lumpinif/deepcrawl** — 100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

- Repository: https://github.com/lumpinif/deepcrawl
- Website: https://deepcrawl.dev
- Stars: 690 · Forks: 82
- Language: TypeScript
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/lumpinif-deepcrawl

## What it returns, and for whom

DeepCrawl describes itself as an agent-oriented website data context extraction platform. It extracts cleaned markdown of page content, an agent-favoured hierarchical links tree, and metadata that LLMs can digest with minimal token cost to reduce context switching and hallucination.

That third capability is the differentiator. A crawler that returns a links tree, not a flat list of URLs, is giving the calling model the shape of a site rather than a bag of pages. The reasoning in the README is about token cost and about the model's ability to stay oriented, and it is a claim about what an agent can do with the result rather than about scraping throughput.

The positioning against Firecrawl is stated plainly: 100% free and open-source Firecrawl alternative with better performance and flexibility, and the project description adds that you can deploy it to Cloudflare or Vercel yourself.

The scope is deliberately narrow. The README notes that DeepCrawl does not target anti-scraping or anti-bot purposes, and that it is optimised for high-frequency agent workloads that scrape public pages. So this is a tool for reading documentation, not for getting past a site's defences. If you need to defeat a challenge or a block, this is the wrong project, and the author says so.

## The warning is the first line of the README

The readme opens with a blockquoted warning that reads, in full, DO NOT USE DEEPCRAWL IN PRODUCTION RIGHT NOW AS IT IS SUBJECT TO CHANGE AND STILL UNDER RAPID DEVELOPMENT. USE AT YOUR OWN RISK.

That is unusual, and it is worth taking at face value rather than skimming past. A project that says its interface may change and that it is under rapid development is telling you that a version pin is not a stability guarantee. You can deploy it, and you will be the one handling the breakage.

The version numbers support the reading rather than contradict it. The published package versions are deepcrawl@0.5.5 from 2026-02-08 and deepcrawl@0.5.4 from 2025-10-23, while the root package.json in the repository says version 0.0.1 and is marked private. So the workspace root is not itself the released artifact, and the released artifact is on a 0.x line that carries no compatibility promise.

The licence is MIT, and the project is presented as open source with the full platform described as open and transparent, including the Next.js dashboard, the API workers, the auth workers and the database. The author is Felix Lu.

## Four workers and a database, in one pnpm workspace

The repository layout is where the architecture becomes clear. The root package.json declares the workspaces, and they are specific:

```json
  "workspaces": [
    "apps/*",
    "apps/workers/*",
    "packages/*",
    "packages/db/db-auth",
    "packages/db/db-d1",
    "packages/sdks/js-ts"
  ]
```

That is a Next.js app or apps under apps/, the Workers under apps/workers/*, shared packages under packages/, two separate database packages split by backend, and a TypeScript SDK.

The two database packages are the interesting detail. db-d1 and db-auth exist because the two deployment targets need different storage, and Cloudflare D1 is a SQLite-compatible database that runs at the edge. The repository topics list cloudflare-workers, nextjs, nextjs16, hono, orpc, better-auth, ai-sdk, html-to-markdown, html-cleaner and links-tree, which is essentially the stack in one line: a Next.js dashboard, Hono for the edge workers, oRPC for the RPC layer, better-auth for authentication, the AI SDK for the agent-facing interface, and the HTML to Markdown and cleaner libraries for the extraction itself.

The database abstraction is why the same codebase deploys to Cloudflare and Vercel. A project that had hard-wired Postgres would not run on D1, and the split into two packages suggests the storage layer is behind an interface with two implementations.

## The environment scripts are the real documentation

The root package.json has an unusually large number of environment-related scripts, and they tell you more about how this project is operated than the README does.

There is env:generate, which runs scripts/env/generate-env-examples.ts, and env:bootstrap, which runs scripts/env/bootstrap-env-vars.ts and then triggers Cloudflare type generation for the auth worker and the v0 worker. There are four separate sync scripts: env:sync:local, env:sync:vars, env:sync:cloudflare:secrets and env:sync:vercel, each pointing at a distinct file under scripts/env/, one per target platform.

Then there is jwt:mint, which runs scripts/jwt-mint.ts, and a test:env script that runs the node test runner over scripts/env/__tests__/*.test.ts. The environment layer has its own test suite, which says something about how much of the platform's surface is configuration.

There is also AUTH_MODES_AND_WORKERS.md at the repository root, a dedicated document about the authentication modes and the workers, and the README points to deepcrawl.dev/docs for the documentation and to CONTRIBUTING.md for the contribution guide.

So the operational model is: configure locally, generate example env files, bootstrap, then sync the same variables to whichever platform you are deploying to. That is a lot of machinery, and it is a fair proxy for how much of the difficulty of running this project is configuration rather than crawling.

## A monorepo with a full quality gate

The build and check scripts are a good indicator of the project's engineering standards.

Turbo drives the tasks: build is turbo build, dev is turbo dev, typecheck is turbo typecheck. The aggregate check script is the interesting one, since it chains the whole gate:

```json
    "check": "pnpm install && pnpm typecheck && pnpm sherif && biome check . && turbo check"
```

That runs a fresh install, a typecheck across every workspace, a dependency consistency check via sherif, a Biome lint and format check, and then the Turbo check task. The CI variant swaps in biome ci and adds the SDK build and the test run.

sherif is a pnpm-dlx invocation of the sherif tool, which checks that dependency versions are consistent across a monorepo, and it has a fix mode that is wired into the fix script alongside biome check --write and turbo fix. Biome handles both linting and formatting, with its own config file at the root in biome.jsonc and a prettier-style config in .prettierrc-style files, and lint-staged is configured in lint-staged.config.cjs.

Releases go through changesets. The changeset script creates a changeset, cv versions with changeset version, and release publishes with changeset publish, and there is a .changeset directory at the root. Husky handles the git hooks through the prepare script.

The SDK is versioned separately from the platform, with a dedicated build script at packages/sdks/js-ts, and the repository is a pnpm workspace with a workspace file and a Turbo configuration at the root.

## Where this does not fit

Three limits are clear from the repository itself.

The first is the production warning, already covered. The second is the deployment target, since the platform is built for Cloudflare Workers and Vercel with D1 as the edge database, and the repository topics are cloudflare-workers and nextjs16. A team that needs to run a crawler inside its own network, or against an internal service with no public DNS, is outside what the environment scripts are built for.

The third is the scope of the extraction. The README describes the output as cleaned markdown, a links tree and metadata for LLM consumption. There is no mention of structured extraction, of schema-driven selectors, of JavaScript rendering, of PDFs, of authentication handling for the target site, or of any output format other than markdown and the tree. A crawler that only needs a paragraph of text is exactly what this is, and a crawler that needs a table parsed into rows is not.

On the flip side, the project is explicit about what it does not do. It does not target anti-scraping or anti-bot purposes, it is not trying to be a general scraping framework, and it does not need a hosted service with a per-request price.

For an agent workflow that reads public documentation, that narrowness is a feature. The cost is that the extraction surface is small, and the token-cost argument only holds if your pages are documentation-shaped.

## Conclusion

DeepCrawl is a good fit for a developer prototyping an agent that reads documentation sites and wants a self-hosted crawl endpoint instead of a hosted dependency, since the whole platform, dashboard, workers, auth and database, is in the repository and the licence is MIT. It is a poor fit for anything you need to run reliably, because the README opens by telling you not to use it in production while it is under rapid development. Before adopting, read the warning at the top of the readme, work through the environment bootstrap scripts rather than hand-writing configuration, and check the documentation site for the current API surface, since the repository publishes the source but the API reference lives outside it.

## FAQ

### What does DeepCrawl return for a crawled page?

Cleaned markdown of the page content, a hierarchical links tree, and metadata that LLMs can digest with minimal token cost. The README describes the platform as agent-oriented website data context extraction.

### Can I deploy DeepCrawl myself?

Yes. The project description states it can be deployed to Cloudflare or Vercel by yourself, and the full platform, meaning the Next.js dashboard, API workers, auth workers and database, is described as open and transparent in the repository.

### Is DeepCrawl ready for production use?

The README opens with a warning not to use DeepCrawl in production right now, since it is subject to change and still under rapid development, and advises using it at your own risk. The published versions are 0.5.5 and 0.5.4, both on a 0.x line.

### What licence is DeepCrawl released under?

MIT. The root package.json declares the license as MIT, the README carries the MIT licence badge, and the project is presented as fully open source.

## Sources

- [License: MIT](https://github.com/lumpinif/deepcrawl/blob/main/LICENSE)
- [lumpinif/deepcrawl on GitHub](https://github.com/lumpinif/deepcrawl)
- [Project website](https://deepcrawl.dev)
- [README](https://github.com/lumpinif/deepcrawl/blob/main/README.md)
- [Releases](https://github.com/lumpinif/deepcrawl/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lumpinif-deepcrawl