LibreCrawl: an open source SEO crawler with a plugin tab API

What the application does

LibreCrawl crawls websites and gives you detailed information about pages, links, SEO elements and performance, and it is built as a web application using Python Flask with a modern web interface supporting multiple concurrent users. The project description calls it a free desktop SEO crawler and an open source alternative to Screaming Frog and similar tools, and the README is explicit that it will always be free and open source, with a paypal link for a coffee framed as the cost of replacing a $279 a year Screaming Frog license or the deepcrawl and sitebulb equivalents. The feature list gives the scope. Multi-tenancy with isolated sessions, so multiple users can crawl simultaneously. SEO analysis extracting titles, meta descriptions and headings. Link analysis tracking internal and external links with detailed relationship mapping. JavaScript rendering for dynamic content, naming React, Vue and Angular. PageSpeed Insights integration for web performance metrics. Multiple export formats, CSV, Excel XLSX, JSON and XML. Automated issue detection. And real-time crawling progress with live statistics. The primary language is Python, the licence is MIT, and the last push was on 2026-09-29.

The startup script, and what it does for you

The quickest way in is a startup script, and on Linux and Mac it is:

chmod +x start-librecrawl.sh
./start-librecrawl.sh

with a Windows batch equivalent, `start-librecrawl.bat`. The script automates six things. It checks for Docker and, if found, runs LibreCrawl in a container, which is the recommended path. If there is no Docker it checks for Python, and if Python is missing it downloads and installs it, though that step is currently temporarily disabled on Windows because it causes some batch file issues. It installs all the dependencies with `pip install -r requirements.txt`. It installs the Playwright browsers for JavaScript rendering. It starts LibreCrawl in local mode with no authentication. And it opens your browser to `http://localhost:5000`. The existence of a script that does all of this is a signal about the audience, which is someone who wants to point it at a site and get a report, not someone building a service.

Docker, and the environment file that matters

The manual Docker route is four commands:

# Clone the repository
git clone https://github.com/PhialsBasement/LibreCrawl.git
cd LibreCrawl

# Copy environment file
cp .env.example .env

# Start LibreCrawl
docker compose up -d

The compose file publishes the port with a bind controlled by the environment, `${HOST_BINDING:-127.0.0.1}:${EXTERNAL_PORT:-5000}:5000`, with the comment explaining that local mode binds only localhost and production mode binds all interfaces for LAN or WAN access. The service runs with `FLASK_APP=main.py` and `PYTHONUNBUFFERED=1`, and a `./data` volume persists the user database and settings. The environment file is where the deployment decisions live, with `LOCAL_MODE`, `HOST_BINDING` and `REGISTRATION_DISABLED` as the three that the README shows in its local mode example, and the production example adding a `SECRET_KEY` with the comment to generate it with `python -c "import secrets; print(secrets.token_hex(32))"`.

The security switches, and what local mode really means

The environment file is unusually explicit about its dangerous settings, and that is worth reading carefully before any deployment. `LOCAL_MODE=false` is the default and the warning is that setting it to true disables all authentication and grants every visitor admin access, and it should only be enabled for local development. `DANGEROUSLY_SKIP_AUTH=false` is the other one, and setting it to true lets anyone log in as any username with no password, with the username used only to keep per-user sessions separate. `REGISTRATION_DISABLED` controls whether new users can sign up at all, which is the one to set on a networked deployment where you want a closed set of accounts. There is also a `DEMO_MODE` with a 1.5GB per-user memory limit, which tells you the crawler holds a lot of memory per session, and the compose file carries SMTP settings for sending mail, commented out. The `SECRET_KEY` note is also specific, since an ephemeral random key is generated at startup if you leave it unset, which is secure but means sessions do not survive a restart.

The plugin API, and how a tab gets added

The extension mechanism is unusually simple, because it does not require a rebuild or a plugin manifest. You drop your custom plugin files into `/web/static/plugins/` and each `.js` file automatically creates a new tab in LibreCrawl. The example registers through a global API object:

LibreCrawlPlugin.register({
  id: 'my-plugin',
  name: 'My Plugin',
  tab: {
    label: 'My Tab',
    icon: '🔥',
  },
  onTabActivate(container, data) {
    container.innerHTML = `
      <div class="plugin-content">
        <h2>My Custom Analysis</h2>
        <p>Found ${data.urls.length} URLs!</p>
      </div>
    `;
  },
  onDataUpdate(data) {
    if (this.isActive) {
    }
  }
});

The lifecycle hooks are `onLoad`, `onTabActivate`, `onTabDeactivate`, `onDataUpdate` and `onCrawlComplete`, and the data a plugin receives is the same as the built-in tabs get, namely the `urls` array with full metadata, the `links` of internal and external discovery, the detected `issues`, and the `stats` covering discovered, crawled, depth and speed. Utilities are reached through `this.utils`, with a notification helper among them. There is also a Plugin Workshop site for browsing what others have built or sharing your own.

Running it without Docker

The Python route needs Python 3.8 or later and a modern web browser such as Chrome, Firefox, Safari or Edge. You clone or download the repository, install the dependencies with `pip install -r requirements.txt`, and for JavaScript rendering support, which is optional, `playwright install chromium`. Then you run the application, either in standard mode with `python main.py`, which gives you authentication and the tier system, or in local mode with `python main.py --local` or `python main.py -l`, where all users get the admin tier and there are no rate limits. You then open `http://localhost:5000` in local mode or `http://<your-ip>:5000` on the network. The dependency list is short and pinned, with requests, beautifulsoup4, urllib3, flask, flask-compress, waitress, playwright, nest-asyncio, psutil, bcrypt, markdown, python-dotenv and openpyxl. Waitress is the WSGI server rather than the Flask development server, and openpyxl is what produces the Excel export.

The container image, and the scale of the dependency

The Dockerfile is worth reading because it shows what JavaScript rendering costs. The base is `python:3.11-slim`, and the system packages installed are a long list of Chromium runtime libraries, including libgtk-3-0, libnss3, libatk-bridge2.0-0, libgbm1, libcups2, libdrm2, libasound2 and xdg-utils, plus fonts-liberation and the wget, gnupg and ca-certificates trio. Playwright's own dependency installer is then run with `playwright install-deps`, and the browsers are installed as a non-root user so they land in that user's cache rather than the system one. The image creates a `librecrawl` group and a user with uid 1000, chowns the application directory and the data directory to it, and switches to it before running `playwright install`. The container exposes port 5000 and the command is handled by the compose file. The cost of this design is image size and build time, and the benefit is that a container with a browser in it behaves the same on your laptop and on a server.

Editorial conclusion

Adopt LibreCrawl if you need SEO auditing and link analysis of a site without a subscription, and you are willing to run it yourself with Playwright installed for JavaScript rendering. Skip it if you need the large-scale crawling, scheduling and log file analysis of a commercial tool, since this is a focused crawler with a web interface. Before exposing it to a network, set a `SECRET_KEY` and keep `LOCAL_MODE=false` and `DANGEROUSLY_SKIP_AUTH=false`, because local mode grants every visitor admin access and disables all authentication.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. PhialsBasement/LibreCrawl on GitHub
4. README