# scrapy-selenium: a middleware for pages that need a browser

> Scrapy is fast because it does not run a browser, and that is also why it cannot see content that only appears after JavaScript runs. scrapy-selenium is the small piece that bridges the two: a downloader middleware that hands a request to a real browser, waits for the page to settle, and hands the rendered HTML back to your spider.

**clemfromspace/scrapy-selenium** — Scrapy middleware to handle javascript pages using selenium

- Repository: https://github.com/clemfromspace/scrapy-selenium
- Stars: 953 · Forks: 344
- Language: Python
- License: WTFPL
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/clemfromspace-scrapy-selenium

## A downloader middleware, and nothing more

The project is small and its description says exactly what it is: a Scrapy middleware to handle JavaScript pages using Selenium.

That framing matters, because there are several ways to solve this problem and they have very different costs. You can render the pages in a headless browser yourself and feed the HTML to your parser. You can rewrite the requests to hit the data endpoint the page itself calls, which is the fastest correct answer when it exists. Or you can install a middleware that does the first thing for you, scoped to the requests you nominate.

This is the third option, and the scoping is the point. The middleware sits in the downloader middleware chain, which is the part of Scrapy that performs the actual request. When it is enabled, a request that asks for a browser is handled by Selenium instead of by Scrapy's own HTTP client, and the request comes back as an ordinary response.

What makes that useful is what happens to the response afterwards. The readme says the selector attribute of the response works as usual, but contains the HTML processed by the Selenium driver. So the page your callback receives is the rendered page, and your existing extraction code does not need to know that a browser was involved.

The repository is correspondingly small. There is the package directory, the tests, a requirements directory, a coverage configuration, a manifest for the source distribution, the packaging script, and a configuration file for the test runner.

## The browser is your dependency now

The installation instructions are short, and they come with a condition that changes the nature of the tool.

```bash
$ pip install scrapy-selenium
```

You should use Python 3.6 or newer, and you will also need one of the browsers that Selenium supports.

That second sentence is the real operational change. A Scrapy project normally has no browser dependency at all, and adding this middleware means every machine that runs the crawler now needs a browser and a matching driver installed, kept up to date, and compatible with the automation library version. That is a meaningful new failure mode for a deployment that previously only needed an HTTP client.

The configuration section shows how that dependency is wired in. You name the browser, you give the path to the driver executable, and you give the arguments to pass to it.

```python
SELENIUM_DRIVER_NAME = 'firefox'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('geckodriver')
SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox
```

The path is found with the standard library helper that looks up an executable on the path, which is the right way to do it, since the driver location differs on every machine and in every container image.

The headless argument is what makes this usable in a server environment, and the comment notes the difference between the two browsers' spellings of the flag, which is the kind of detail that otherwise costs an afternoon.

## Local browsers, remote grids, and where the middleware sits

There are two more configuration knobs, and the second one changes how you would run this in anger.

You can optionally set the path to the browser executable itself, which is useful when the browser is installed somewhere unusual, such as a snap or a flatpak, or in a container image where the path is not on the environment path.

```python
SELENIUM_BROWSER_EXECUTABLE_PATH = which('firefox')
```

And to use a remote Selenium driver you set the command executor instead of the driver executable path.

```python
SELENIUM_COMMAND_EXECUTOR = 'http://localhost:4444/wd/hub'
```

That second route is the one that matters for scale. A browser is a heavy process, and running one per request inside a crawler is a poor use of a machine. Pointing the middleware at a Selenium grid means the browser work happens elsewhere, on a fleet you can size independently, and the crawler stays a crawler.

The last piece of configuration is the registration itself, and the number after the middleware path is the priority, which sets where in the downloader chain it runs.

```python
DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800
}
```

And there is one piece of advice that saves a lot of confusion. The readme says to add the middleware to the downloader middlewares, and to use the request class from this package instead of the built-in one. If you register the middleware but keep sending ordinary requests, the middleware has nothing to act on, and the most common symptom is that it appears to do nothing.

## Usage: one request class and one extra key

The usage section is a few lines of code, and the design is clean enough to explain in one paragraph.

You use the request class from this package instead of the built-in one.

```python
from scrapy_selenium import SeleniumRequest
```

The request is handled by Selenium, and the request comes back with an additional meta key, named driver, containing the Selenium driver with the request processed.

```python
def parse_result(self, response):
    print(response.request.meta['driver'].title)
```

So the response is a normal Scrapy response for everything you care about, and the live browser object is available alongside it for the things a scraper cannot do, such as reading the document title, clicking a button or taking a snapshot of the current state.

The readme points at the Selenium documentation for the available driver methods and attributes, and the effect is that the browser becomes an escape hatch rather than the main path. Your extraction code keeps using selectors, and only the step that needs a real browser touches the driver.

That is the right balance, and it is also why the extra arguments below matter.

## Three arguments that turn a race into a procedure

The request class accepts four additional arguments, and they are the difference between a middleware that works and one that intermittently does not.

The first pair is about waiting. When used, Selenium performs an explicit wait before returning the response to the spider, and the wait can be a fixed number of seconds or a condition to be satisfied.

```python
yield SeleniumRequest(
    url=url,
    callback=self.parse_result,
    wait_time=10,
    wait_until=EC.element_to_be_clickable((By.ID, 'someid'))
)
```

The condition form is the one to reach for. A fixed wait is a guess about how long the page takes, and the guess is wrong on a slow day, and a wrong guess in the fast direction means you scrape a half-rendered page and get an empty result that looks like a data problem. Waiting for an element to become clickable asks the question you actually care about.

The second argument is a screenshot. When used, Selenium takes a screenshot of the page, and the binary data of the captured image is added to the response meta.

```python
yield SeleniumRequest(
    url=url,
    callback=self.parse_result,
    screenshot=True
)
```

That is a debugging tool of the first order. When a scrape returns nothing, the screenshot tells you what the browser actually saw, and the readme shows the two lines that write it to a file.

The third argument is custom script execution. When used, Selenium runs your own JavaScript in the page, which is how you scroll to the bottom to trigger lazy loading, or dismiss a cookie banner, or expand a section that a click would not reach.

```python
yield SeleniumRequest(
    url=url,
    callback=self.parse_result,
    script='window.scrollTo(0, document.body.scrollHeight);',
)
```

The scroll example is the canonical one, because infinite scroll pages are the most common reason a scraper appears to work and then silently returns a fraction of the data.

## The packaging, the licence and the state of the project

The repository details are worth a short note, because they say what kind of project this is.

The packaging script reads the requirements from a file in the requirements directory and passes them to the installer, and it does so by reaching into the package manager's own internals to parse that file. The script includes a comment acknowledging that this is a quick hack to support a particular version of the installer whose internal structure changed, with a fallback to the newer internal path. That is a small piece of technical debt sitting in the packaging of an otherwise tidy project, and it is the kind of thing that breaks when the installer changes again.

The licence file is present, and the repository listing records the licence as the public domain variant, which is an unusual and generous choice for a package that others depend on.

The continuous integration is Travis, and the badges include test coverage and maintainability from a code quality service, which suggests the project has been measured rather than merely run. There is a coverage configuration and a tests directory, and the default branch is the development one.

There are no tagged releases, so the version to install comes from the package index rather than from a release list, and the last push was on 2026-04-13 with the repository not archived.

The practical summary is that this is a small, focused, well documented integration that does one thing, and that its age shows mostly in the packaging script rather than in the middleware itself. If you are scraping pages that need JavaScript, the four extra arguments are the part to invest time in, because they are what make the results reproducible rather than occasional.

## Conclusion

scrapy-selenium suits a crawler that has to reach a handful of pages that only render with JavaScript, and the four extra arguments are what make it usable rather than merely possible, since a screenshot and a wait turn a race into a procedure. It is a poor fit as the default for a whole crawler, because driving a browser for every request is enormously slower than an HTTP fetch, and a poor fit if your pages need no JavaScript at all. Before you reach for it, check whether the site exposes a data endpoint that Scrapy can read directly, and if you do use the middleware, drive it through a remote grid so the browser overhead does not sit on your crawler.

## FAQ

### How do I use scrapy-selenium with Scrapy?

Install the package, add the middleware to the downloader middlewares with a priority of 800, and yield the request class from the package instead of the built-in one. The response is then the HTML rendered by the browser, and the Selenium driver is available on the request meta under the key driver.

### How do I wait for a page to finish loading in scrapy-selenium?

The request accepts a wait time in seconds and a wait until condition, and the condition is a Selenium expected condition such as an element becoming clickable. The readme links the Selenium documentation for explicit waits, and the condition form is the one that avoids guessing how long a page takes.

### Can scrapy-selenium use a remote browser?

Yes. Instead of pointing at a driver executable, you set the command executor to the address of a Selenium hub, and the browser work happens on that remote machine rather than inside the crawler.

### What extra features does the SeleniumRequest class offer?

Four additional arguments. A wait time and a wait until condition, a screenshot flag that adds the captured image data to the response meta, and a script option that runs custom JavaScript in the page, for example to scroll to the bottom to trigger lazy loading.

## Sources

- [clemfromspace/scrapy-selenium on GitHub](https://github.com/clemfromspace/scrapy-selenium)
- [Issues](https://github.com/clemfromspace/scrapy-selenium/issues)
- [License: WTFPL](https://github.com/clemfromspace/scrapy-selenium/blob/develop/LICENSE)
- [README](https://github.com/clemfromspace/scrapy-selenium/blob/develop/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/clemfromspace-scrapy-selenium