# Rotating proxies inside Scrapy: how the middleware decides what is dead

> scrapy-rotating-proxies is a Scrapy downloader middleware that hands each request to a proxy from a list, judges whether a proxy has been banned, and backs off on dead ones. It ships no proxies and no ban rules, so almost all of the work stays with the person running the spider.

**TeamHG-Memex/scrapy-rotating-proxies** — use multiple proxies with Scrapy

- Repository: https://github.com/TeamHG-Memex/scrapy-rotating-proxies
- Stars: 771 · Forks: 159
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/teamhg-memex-scrapy-rotating-proxies

## A downloader middleware, with no proxies attached

The first thing to understand about scrapy-rotating-proxies is what it is not. It is not a proxy provider, and it is not a scraping framework. It is a Scrapy middleware, which in practice means a piece of code that sits in the downloader path and decides which proxy each outgoing request uses.

The description in the repository summary is blunt about the scope: the package provides a Scrapy middleware to use rotating proxies, check that they are alive and adjust crawling speed. Three jobs, all of them about the health of the proxy pool rather than about the page being scraped.

The proxy list is entirely your responsibility. The project ships with a note saying it is up to you to find proxies and maintain proper ban rules for web sites, and that the package does not have anything built in for this. So the middleware is the scheduling and bookkeeping half of a rotating proxy setup, and you supply the other half. The project listing shows MIT licensing, with the licence text sitting in LICENSE.txt at the repository root and the same licence named in setup.py.

Who is it for then? People running long crawls against sites that rate limit or block by IP, who already have access to a proxy pool and want Scrapy to spread load across it, retry around blocks, and stop wasting retries on proxies that have already been banned.

## Two middleware entries and two settings to start

Installation is a single pip command:

```
    pip install scrapy-rotating-proxies
```

After that you add a proxy list to settings.py, either inline:

```
    ROTATING_PROXY_LIST = [
        'proxy1.com:8000',
        'https://proxy2.com:8000',
        'login:password@proxy3.com:8031',
        'https://login:password@proxy4.com:8032',
        # ...
    ]
```

or as a path to a file with one proxy per line, which is the more maintainable option when the pool changes often:

```
   ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt'
```

The precedence rule is stated plainly: ROTATING_PROXY_LIST_PATH takes precedence over ROTATING_PROXY_LIST if both options are present. That ordering is worth knowing, because a leftover inline list in a settings file will be silently ignored as long as the path setting exists, and the symptom is simply that your edits to the inline list appear to do nothing.

Then the two middlewares go into DOWNLOADER_MIDDLEWARES:

```
    DOWNLOADER_MIDDLEWARES = {
        # ...
        'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
        'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
        # ...
    }
```

The order numbers 610 and 620 are not decoration, they set the position in the middleware chain. RotatingProxyMiddleware at 610 picks and assigns a proxy, BanDetectionMiddleware at 620 observes the response to decide whether that proxy is now banned. Once both are in place, requests are proxied using one of the proxies from the configured list or file.

## Every concurrency setting turns per-proxy

This is the behaviour most likely to surprise you, and it is a deliberate design choice rather than a bug. By default, all the default Scrapy concurrency options, including DOWNLOAD_DELAY, the AUTHTHROTTLE settings and CONCURRENT_REQUESTS_PER_DOMAIN, become per-proxy for proxied requests when RotatingProxyMiddleware is enabled.

The worked example in the write-up is short: if you set CONCURRENT_REQUESTS_PER_DOMAIN=2, the spider makes at most two concurrent connections to each proxy, regardless of the request URL domain. Note the consequence. A limit you thought of as a politeness cap toward one site is now a cap toward every proxy you own, and the number of simultaneous connections your crawler can open grows with the size of your pool. A spider with fifty proxies and a per-domain limit of two can have far more requests in flight than it did before, aimed at many different hosts. If you have an internal rate limit with a single site, that setting no longer protects you.

The flip side is that per-proxy throttling is what makes a proxy pool useful at all. One slow proxy no longer holds up the whole crawl, and the aggregate throughput of the pool becomes the limit. Teams migrating from a single-proxy setup should recheck their delay and concurrency numbers rather than carry them over unchanged.

## Ban detection is a policy you plug in, and it is site-specific

The middleware keeps track of working and non-working proxies and re-checks the non-working ones from time to time. The judgement of whether a proxy is banned is where most of the real configuration work sits, and the write-up states plainly that detection of a non-working proxy is site-specific.

The default heuristic is simple. If a response status code is not 200, the response body is empty, or there was an exception, the proxy is considered dead. That is a coarse rule. Any non-200 response, including a 301 redirect or a 404 from a badly formed URL, marks a perfectly healthy proxy as dead, and the pool drains faster than it should.

You can replace the rule through the ROTATING_PROXY_BAN_POLICY option, which takes a path to a custom policy class:

```
    # settings.py
    ROTATING_PROXY_BAN_POLICY = 'myproject.policy.MyBanPolicy'
```

The policy class needs response_is_ban and exception_is_ban methods, and each of them can return True for ban detected, False for not a ban, or None for unknown. Subclassing is the recommended route the default BanDetectionPolicy rather than starting from scratch, which keeps the standard rules and lets you add one condition:

```
    # myproject/policy.py
    from rotating_proxies.policy import BanDetectionPolicy

    class MyPolicy(BanDetectionPolicy):
        def response_is_ban(self, request, response):
            # use default rules, but also consider HTTP 200 responses
            # a ban if there is 'captcha' word in response body.
            ban = super(MyPolicy, self).response_is_ban(request, response)
            ban = ban or b'captcha' in response.body
            return ban

        def exception_is_ban(self, request, exception):
            # override method completely: don't take exceptions in account
            return None
```

If you do not want a separate module, the same two methods can live on the spider itself:

```
    class MySpider(scrapy.Spider):
        # ...

        def response_is_ban(self, request, response):
            return b'banned' in response.body

        def exception_is_ban(self, request, exception):
            return None
```

The three-state return value is the part that rewards thought. Returning None for an ambiguous case means unknown rather than a verdict, and the write-up is emphatic that the rules need to be correct because the action for a failed request and the action for a bad proxy should differ. If a proxy is at fault it makes sense to retry the request with a different proxy. A policy that returns False too eagerly turns every transient failure into a wasted proxy.

## Bans have a price, and the price is counted in proxies

There is a setting that makes the cost of a wrong ban explicit. ROTATING_PROXY_PAGE_RETRY_TIMES is the number of times a page download is retried using a different proxy, and after that many retries the failure is counted as a page failure rather than a proxy failure. The practical framing is worth quoting: every improperly detected ban costs you ROTATING_PROXY_PAGE_RETRY_TIMES alive proxies. The default is 5.

So a ban rule that is too eager does not just fail one request. It burns five proxies from a pool you paid for, and the site is not even blocking you yet. With a small pool this is the difference between a crawl that lasts and a crawl that stalls with a dead proxy list.

The value can be changed per request through the max_proxies_to_try request.meta key, which the write-up suggests for pages you are confident should work, where spending more proxies is acceptable.

There is also a switch on the whole pool: ROTATING_PROXY_CLOSE_SPIDER stops the spider when there are no alive proxies, while the default behaviour of False re-checks all dead proxies instead. Stopping the spider is a reasonable default for a scheduled job that should fail loudly rather than spin against a dead pool, and the re-check mode suits a long-running crawler that can afford to wait.

## Dead proxies come back on a randomized exponential backoff

Because a proxy that failed now may work later, the middleware does not discard dead entries permanently. It re-checks them, on a randomized exponential backoff: the first check happens soon, and if the proxy still fails, the next check is delayed further.

Two settings control the shape of that curve. ROTATING_PROXY_BACKOFF_BASE is the base backoff time in seconds and defaults to 300, which is described as five minutes, although the initial delay is random from 0 to 5 minutes. ROTATING_PROXY_BACKOFF_CAP is the cap on backoff time in seconds and defaults to 3600, an hour.

The practical effect is that a temporarily blocked proxy returns to service within minutes, while a permanently dead one is checked roughly hourly and costs a small amount of traffic each time. Since the backoff is randomized rather than fixed, a pool that recovered in a burst does not have all its proxies retried at the same instant.

One more setting belongs here. ROTATING_PROXY_LOGSTATS_INTERVAL is the stats logging interval in seconds and defaults to 30, so the middleware reports on the state of the pool at a predictable interval. A run that ends with no requests and no obvious error is often a pool that ran dry, and this is where you look.

## Per-request opt out when you need a direct connection

There is one escape hatch, and it matters if you mix proxied and direct traffic in the same spider. Requests with proxy set in their meta are not handled by scrapy-rotating-proxies at all. To disable proxying for a request you set request.meta['proxy'] to None, and to set a proxy explicitly you assign the address to request.meta['proxy'].

That gives you a way to send a login flow or a health check straight from the machine running the spider, or to pin a request to a known-good proxy, without spinning up a second spider. It also means the middleware cannot correct you: whatever address you set is used, so a typo there is your problem and not the pool's.

The other project worth naming is a commercial proxy service. The project's own FAQ points to Zyte Smart Proxy Manager, which integrates with Scrapy through the scrapy-crawlera plugin and takes care of proxy sourcing and ban handling. The difference in approach is straightforward. This middleware is a scheduling layer over a list you maintain, with detection rules you write in Python, and it never touches the network itself. A managed service moves both of those jobs off your machine, at the cost of a commercial dependency and a per-request billing model. If you have no proxy source and no time to write ban rules, that is the alternative the project itself points you to.

## Alpha status, no releases, and a last push on 2026-04-08

The maintenance picture deserves a straight answer. The repository is not archived, but the last push was on 2026-04-08 and there are no GitHub releases, so you cannot pin a version from the release history. The version number itself is not hardcoded in setup.py: it is read out of the __init__.py file of the rotating_proxies package with a regular expression, so the package version and the repository state are the same fact.

setup.py describes the maturity plainly, with a Development Status :: 3 - Alpha classifier, and the supported interpreter list in that same file stops at Python 2.7 and Python 3.4 to 3.6, with a typing dependency added only for interpreters at or below Python 3.6. The install requirements are attrs above version 16.0.0 and six. Those classifiers describe the packaging as it was written, so on a current interpreter the dependency set and the compatibility claims deserve a check rather than trust.

Development happens in the open in other ways. The repository carries a CHANGES.rst file that setup.py folds into the long description, a tox.ini and a tests/ directory, and contributors are asked to install tox and run tox from the source checkout to run the tests. There is also an AGENTS.md file at the root.

For licensing, setup.py declares MIT license and the classifier set includes the MIT OSI approval, with the full text in LICENSE.txt. That is permissive enough for internal tooling and commercial use without the copyleft obligations of the GPL, though the choice of dependency and how you deploy a spider against a given site are separate questions from the licence.

## Conclusion

Adopt this middleware when you already have a proxy source and want Scrapy to spread requests across it while handling bans and dead proxies for you, especially when a single IP is already blocked. Do not adopt it expecting a turnkey solution: it contains no proxy list, no ban detection rules and no handling for sites that challenge with a captcha. Verify first that your ban policy returns None rather than False for ambiguous responses, since a false ban costs you one alive proxy per mistake, and check the alpha status in setup.py before depending on it in a long-running crawler.

## FAQ

### How do I install scrapy-rotating-proxies and enable it in Scrapy?

Install it with pip install scrapy-rotating-proxies, then set ROTATING_PROXY_LIST or ROTATING_PROXY_LIST_PATH in settings.py and add both rotating_proxies.middlewares.RotatingProxyMiddleware at 610 and rotating_proxies.middlewares.BanDetectionMiddleware at 620 to DOWNLOADER_MIDDLEWARES. ROTATING_PROXY_LIST_PATH takes precedence over ROTATING_PROXY_LIST when both are present.

### How does scrapy-rotating-proxies decide that a proxy has been banned?

The default heuristic treats a proxy as dead when the response status code is not 200, the body is empty, or an exception occurred. You can override it with the ROTATING_PROXY_BAN_POLICY option or by defining response_is_ban and exception_is_ban on your spider, and each method may return True, False or None for ban, not ban, or unknown.

### Where do I get a proxy list for scrapy-rotating-proxies?

The project ships no proxies of its own, and its own FAQ says finding proxies and maintaining ban rules is up to you. It points to commercial proxy services such as Zyte Smart Proxy Manager, which integrate with Scrapy through the scrapy-crawlera plugin and handle proxy sourcing and ban detection.

### Is scrapy-rotating-proxies still maintained and what licence does it use?

The repository is not archived and the last push was on 2026-04-08, with no GitHub releases published. The licence is MIT, declared in setup.py with the text in LICENSE.txt, and setup.py classifies the package at Development Status :: 3 - Alpha.

## Sources

- [Issues](https://github.com/TeamHG-Memex/scrapy-rotating-proxies/issues)
- [License: MIT](https://github.com/TeamHG-Memex/scrapy-rotating-proxies/blob/master/LICENSE)
- [README](https://github.com/TeamHG-Memex/scrapy-rotating-proxies/blob/master/README.md)
- [TeamHG-Memex/scrapy-rotating-proxies on GitHub](https://github.com/TeamHG-Memex/scrapy-rotating-proxies)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/teamhg-memex-scrapy-rotating-proxies