# scrapyrt: an HTTP API for one-off Scrapy crawls

> ScrapyRT is a small BSD 3-Clause tool from Scrapinghub that puts a Scrapy spider behind an HTTP endpoint, so you can send a URL and a spider name and receive the scraped items as JSON. The judgement is that it is a well scoped bridge for on-demand extraction, and the README is unusually clear that it is not a crawler runtime.

**scrapinghub/scrapyrt** — HTTP API for Scrapy spiders 

- Repository: https://github.com/scrapinghub/scrapyrt
- Stars: 884 · Forks: 161
- Language: Python
- License: BSD-3-Clause
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/scrapinghub-scrapyrt

## A spider behind an endpoint

ScrapyRT is described as adding an HTTP API to your Scrapy project in minutes, and the contract is simple. You send a request naming a spider and a URL, and in response you get back the items collected by a spider visiting that URL. The important detail is what does not change. The README states that all Scrapy project components are supported, including middleware, pipelines and extensions, so the extraction logic, the item cleaning and the retry and throttling behaviour are the ones you already wrote. ScrapyRT does not reimplement any of that, it only puts a request and a response in front of it. You run ScrapyRT in the Scrapy project directory, and it starts an HTTP server that lets you schedule spiders and get spider output as JSON. The intended user is a developer who already has a Scrapy project and wants to call it from another system, rather than someone learning to crawl.

## The quickstart is four commands

The README walks through the whole thing in four steps, and the first one is the install:

```shell
> pip install scrapyrt
```

The second step is to move into the Scrapy project directory, in the README's phrasing switching to the Scrapy project such as the quotesbot project:

```shell
> cd my/project_path/is/quotesbot
```

The third launches the server:

```shell
> scrapyrt
```

And the fourth runs a spider through it, naming the spider and the URL to visit:

```shell
> curl "localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"
```

The port 9080 is the default, and the crawl.json endpoint is where the spider name and the target URL go. What you should see in return is the JSON of the items the spider produced, which means the whole round trip, from HTTP request to scraped item, happens inside one request cycle.

## Callbacks and crawl arguments travel in the request body

The simple form is not the whole API, because a real spider usually needs more than a URL. The README gives a second, more complex query where you specify a callback for the Scrapy request and a zipcode argument for the spider, and the body is JSON with a request object, a spider_name and a crawl_args object:

```shell
>  curl --data '{"request": {"url": "http://quotes.toscrape.com/page/2/", "callback":"some_callback"}, "spider_name": "toscrape-css", "crawl_args": {"zipcode":"14000"}}' http://localhost:9080/crawl.json -v
```

The three parts map to the three things a spider needs. The request object carries the URL to fetch and the name of the callback that should handle the response, which is how you drive a spider that has several entry points. The spider_name selects which spider runs. The crawl_args object passes arguments down to the spider, and the zipcode value is a name and a value pair that the spider code is expected to read. The verbose flag on the curl call is what you want while you are working out whether your spider received what you sent.

## It insists on a real Scrapy project

ScrapyRT is not a standalone crawler and the README is direct about the requirement. ScrapyRT will look for the scrapy.cfg file to determine your project settings, and it will raise an error if it cannot find one. You also need all of your project requirements installed, since the server runs inside your project rather than bundling a copy of it. That constraint is what makes the earlier steps necessary, since the install command is followed by a change into the project directory before the server is started, and the project is found through the configuration file rather than through a path you pass in. The packaging reflects the same design. The manifest declares a single console script, scrapyrt, mapped to scrapyrt.cmdline:execute, and the runtime dependencies are only Scrapy itself, at scrapy>=2.7, and packaging, so the tool adds an HTTP layer without dragging in a framework of its own.

## It is not a crawler runtime, and the README says so

The Note section in the README is the most useful part of the document for anyone deciding whether to install this. The project is not a replacement for Scrapyd, or for Scrapy Cloud, or for other infrastructure to run long running crawls. It is also not suitable for long running spiders. What it is good for is spiders that will fetch one response from some website and return items quickly. That distinction is architectural rather than a matter of configuration. A long running crawl wants a scheduler, a queue, persistence of state across restarts and a way to see what is in flight, and ScrapyRT provides none of those, because it is a request handler in front of a spider rather than a place spiders live. The documentation is on readthedocs, with an API page and an index, and the project keeps a CHANGES.rst that the manifest links as the release notes.

## Packaging, versions and a container image

The project metadata is current and unremarkable, which is a good sign. The version is 0.18.1, the build backend is hatchling with hatchling>=1.27.0 required, the licence is BSD-3-Clause, and the supported interpreters are Python 3.9 through 3.13 with requires-python set to >=3.9. The classifiers describe it as Production/Stable, as targeting the console and also as a daemon with no input or output, and as OS Independent. The repository carries a Dockerfile that is worth a look, because it shows the intended container topology. It builds on python:3.10-slim-buster, creates a source directory and a project directory, installs the package from source, sets the working directory to the project mount, and sets the entrypoint to scrapyrt listening on 0.0.0.0, exposing port 9080. The comments at the top of that file show the two commands to run, building the image and starting it as a daemon with the host project directory mounted at /scrapyrt/project, so the container finds the same scrapy.cfg and the same project requirements the local run would.

## Who supports it and where it stands

ScrapyRT is maintained by Scrapinghub, and the manifest lists the authors and maintainers as Scrapinghub with the opensource at zyte.com address, while the topic tags place it firmly in the Scrapy and Twisted ecosystem, with hacktoberfest among the tags from its history. The support section says that open source support is provided on GitHub and asks people to create a question issue, using the question label, and that commercial support is available from Zyte. The licence is BSD 3-Clause. On the release history, the tags are v0.15.0 from 2023-09-20, v0.16.0 from 2024-02-14 and v0.17.0 from 2025-07-28, while the manifest sits at 0.18.1 and the last push to the master branch was on 2026-10-13 with the repository not archived. In other words the release line moves once or twice a year, the branch is still active, and the changelog file rather than the tag list is where you should read about what changed between the version you run and the version you want.

## Conclusion

scrapyrt fits a service that needs to fetch a handful of pages on demand and hand back structured items, such as an extraction API sitting in front of an existing Scrapy codebase, and it inherits the full Scrapy stack so middleware, pipelines and extensions keep working. It is the wrong tool for a scheduled or long running crawl, because the README says plainly that it is not a replacement for Scrapyd or Scrapy Cloud, and it needs a full Scrapy project on disk with its scrapy.cfg and its installed requirements. Before you wire it up, run the curl call from the quickstart inside a project you already trust, check the CHANGES.rst before upgrading across the 0.15 to 0.17 releases, and decide whether the daemon image with its 9080 port exposed suits your network.

## FAQ

### How do I run a Scrapy spider through an HTTP API with ScrapyRT?

Install it with pip install scrapyrt, change into your Scrapy project directory so that the scrapy.cfg file is found, run scrapyrt to start the server, then call the crawl.json endpoint on port 9080 with a spider_name and a url parameter. The response contains the items the spider collected as JSON.

### Can I pass a callback and extra arguments to a ScrapyRT request?

Yes. Send a JSON body to the crawl.json endpoint with a request object holding the url and the callback, a spider_name, and a crawl_args object carrying arguments for the spider, which is how the README shows a zipcode value being passed.

### Is ScrapyRT suitable for long running crawls?

No. The README states that the project is not a replacement for Scrapyd, Scrapy Cloud or other infrastructure for long running crawls, and that it is not suitable for long running spiders. It is aimed at spiders that fetch one response and return items quickly.

## Sources

- [Issues](https://github.com/scrapinghub/scrapyrt/issues)
- [License: BSD-3-Clause](https://github.com/scrapinghub/scrapyrt/blob/master/LICENSE)
- [README](https://github.com/scrapinghub/scrapyrt/blob/master/README.md)
- [Releases](https://github.com/scrapinghub/scrapyrt/releases)
- [scrapinghub/scrapyrt on GitHub](https://github.com/scrapinghub/scrapyrt)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/scrapinghub-scrapyrt