# gain: a three-object crawling framework built on asyncio and aiohttp > gain is a Python 3.10+ web crawling framework where you declare Spider, Parser and Item classes and the framework runs concurrency, retries and persistence. Version 1.0.1 is on PyPI under the name gain, with an optional uvloop extension for Linux. **elliotgao2/gain** — Web crawling framework based on asyncio. - Repository: https://github.com/elliotgao2/gain - Stars: 2,019 · Forks: 203 - Language: Python - License: MIT - Published: 2026-10-11 · Updated: 2026-10-11 · Language: en - Canonical page: https://hysenlabs.com/projects/elliotgao2-gain ## Spider, Parser and Item: three objects with separate jobs gain separates a web crawl into three distinct roles. A Spider holds the starting URL, a concurrency budget, and HTTP headers. After the Spider issues the first request, it dispatches response URLs to a parsers list. Each Parser decides whether a matching URL should produce more URLs to crawl or a populated Item object to save. A Parser with one argument only follows: every URL matching its regex goes into the crawl queue. A Parser with two arguments matches URLs and instantiates the Item class passed as its second argument. An Item class maps HTML fields to Python attributes using selectors, and its async save() method receives the populated object and persists it however you choose. Writing a file, pushing to a queue, and inserting into a database are all options the README demonstrates in its examples. The README's architecture diagram shows this as a feedback loop. start_url feeds into parsers, parsers feed into items, and save() feeds newly discovered URLs back into parsers. That cycle runs until no more URLs match any pattern in the parsers list. gain schedules all requests up to the concurrency limit and handles retries internally. Your code provides the class declarations; the framework provides the scheduling. ## Installing gain and running the quickstart example gain is on PyPI under the name gain. Four dependencies install with it automatically: aiohttp 3.9 or newer, aiofiles 23.0 or newer, pyquery 2.0 or newer, and lxml 5.0 or newer. Python 3.10 or newer is required. ```bash pip install gain ``` Linux users can add the uvloop extension for a faster event loop: ```bash pip install "gain[uvloop]" ``` The quickstart in the README targets the Scrapinghub blog. A Post class defines two Css selectors: title pointing to .entry-title, and content pointing to .entry-content. A save() method appends each title to scrapinghub.txt using aiofiles. The Spider sets concurrency to 5, a User-Agent header, and a two-entry parsers list: one Parser follows pagination links, and a second Parser routes individual article pages to the Post class. Running the file starts the crawl: ```bash python spider.py ``` aiofiles installs as a direct dependency of gain, so it is available without a separate step, but the example imports it with import aiofiles rather than through any re-export from gain's namespace. ## Parser follows URL strings; XPathParser follows HTML trees Parser takes a regex as its first argument and matches it against URL strings. XPathParser takes an XPath expression and traverses the parsed HTML structure of each fetched page instead. Both types sit together in a Spider's parsers list, and a single Spider can hold a mix of both, which means different sections of a site can use whichever discovery strategy fits their structure. The README's second example uses three XPathParser objects on one Spider. The first selects category links using //span[@class="category-name"]/a/@href. The second selects pagination anchors inside an element with class pagination. The third routes drama detail pages to a Post item class. Those three parsers share the Spider's concurrency budget and fetch pages from the same aiohttp session. That example also shows the only proxy setting the README documents: ```python proxy = "https://localhost:1234" ``` No credential syntax, rotation mechanism or per-request override is described. The README does not explain whether this setting applies to all requests from that Spider or only to the initial request. Authentication at the proxy level is not covered, and no documentation addresses what happens when the proxy itself requires credentials. ## The README lists three Item selector types but demonstrates only Css The 'How it works' section of the README names three selector types for Item field extraction: Css, Xpath and Regex. Across both code examples in the README, only Css appears inside Item class definitions. The first example uses Css(".entry-title") and Css(".entry-content"). The second uses Css(".breadcrumb_last"). XPathParser appears in parsers lists for URL discovery, not inside Item classes for field extraction. The import line in the XPathParser example is from gain import Css, Item, Parser, Spider, XPathParser, which does not include Xpath or Regex as names. Neither name appears inside any Item class in either code block. This matters when a target page's structure is easier to reach with XPath or a regular expression than with a CSS selector. The README does not show what using Xpath inside an Item definition looks like. The examples directory holds scripts for Scrapinghub, V2EX and Sciencenet, and all three use Css in their Item classes. A reader who needs XPath or regex field extraction on Item objects will need to read the source under gain/ directly rather than adapting the examples. ## No built-in rate limiting, authentication or JavaScript rendering The concurrency attribute on Spider sets how many requests run in parallel, and that is the only throttle the README documents. There is no per-domain request delay, no minimum interval between requests, no Retry-After header handling and no robots.txt compliance mentioned in the README. A crawl configured with concurrency = 10 sends ten simultaneous requests to the target domain with no built-in pause. Authentication is not covered. The headers dict on Spider can hold any HTTP header value, including a static cookie or a bearer token, but the README does not demonstrate this. There is no session object, no login form handling and no documentation for sites that require more than a static header value. JavaScript-rendered content falls outside what gain can reach. Requests go through aiohttp, which fetches the initial HTML response. Pages that populate fields after the initial load, through JavaScript execution in a browser, return only the raw HTML to gain's parsers. For that class of site, Item selectors will find empty or missing fields. The README makes no claim about JavaScript support, and no integration with headless browsers is documented. ## uvloop runs only on Linux and uses an unbounded version floor The uvloop extension gives gain a faster event loop. The optional dependency in pyproject.toml pins it at uvloop>=0.19 with no upper bound. A fresh install on Linux may pull any version of uvloop released after gain 1.0.1, with no guarantee of compatibility beyond the floor. On macOS or Windows, uvloop cannot be installed because the package depends on libuv, which only supports Linux. A developer who runs gain locally on macOS and deploys to Linux will get different event-loop behavior in each environment, since the macOS run uses the default asyncio loop and the Linux run uses uvloop when the optional extra is present. The README does not explain how to detect which loop is active or whether the behavioral difference matters for a particular crawl. The pyproject.toml classifies the project as 'Development Status :: 5 - Production/Stable' in its PyPI classifiers. gain is version 1.0.1, MIT licensed, with the last push on 2026-08-09 and no GitHub releases to pin against. ## Scrapy is the larger alternative; the scope difference is deliberate Scrapy is the most commonly compared alternative to gain. Both handle async Python crawling with item classes and selector-based field extraction. The practical difference is what each framework includes beyond the core loop. Scrapy ships with a middleware layer, an item pipeline, built-in download delays, robots.txt compliance through its ROBOTSTXT_OBEY setting, and an extensible spider class hierarchy. A larger ecosystem adds libraries for JavaScript rendering and proxy rotation. gain has none of those in its documented interface. Where Scrapy's configuration overhead grows with project complexity, gain's three-object model stays flat. Writing a Spider in gain means one file with two or three class definitions and a single run() call. Scrapy's directory structure, settings file, and command-line tooling add setup time before a first crawl runs. That overhead is worthwhile when the project needs middleware customization or a full pipeline, but unnecessary when it does not. The examples in gain's repository cover Scrapinghub, V2EX and Sciencenet: public, paginated, non-authenticated sites. That is the use case the framework demonstrates, and staying within it is where gain's compact design is most useful. For a long-running production crawl that must handle anti-scraping measures, proxy rotation or authentication flows, Scrapy's middleware configurability addresses those requirements where gain's documentation does not. ## Conclusion gain fits engineers who need to crawl simple, paginated, publicly accessible sites in Python without JavaScript rendering or login. The three-object model keeps a spider script short for that case. Where it breaks down is any site that needs session cookies, authentication or dynamic rendering, because the documented API covers none of those. Before adopting it, fetch one target page using aiohttp directly and check whether your fields appear in the initial HTML response: if they do, the Parser and Item model handles the rest; if they do not, gain provides no documented path forward. ## FAQ ### How do I install gain for Python? gain is on PyPI under the name gain and installs with pip install gain. Python 3.10 or newer is required. Linux users can add uvloop support with pip install "gain[uvloop]". ### What is the concurrency setting in gain? The concurrency attribute on the Spider class sets how many requests run in parallel. The quickstart example in the README sets it to 5. There is no documented maximum, and the README describes no per-domain delay or rate-limiting beyond this attribute. ### Does gain support JavaScript-rendered pages? No. gain fetches HTML over aiohttp and passes the initial response to lxml and pyquery for parsing. Pages that populate their content through JavaScript after the initial load return only the raw HTML, and the README makes no claim about JavaScript rendering support. ### Can I use XPath selectors inside a gain Item class? The README names Css, Xpath and Regex as Item field selectors, but both code examples only demonstrate Css inside Item classes. XPathParser appears in the parsers list for URL discovery, not for Item field extraction. No example in the README shows Xpath or Regex used as Item-level field selectors. ### Does gain work on Windows or macOS? gain installs on Windows and macOS using the base pip install gain command. The optional uvloop extension is not available on those platforms because uvloop depends on libuv, which only supports Linux. ## Sources - [elliotgao2/gain on GitHub](https://github.com/elliotgao2/gain) - [Issues](https://github.com/elliotgao2/gain/issues) - [License: MIT](https://github.com/elliotgao2/gain/blob/master/LICENSE) - [README](https://github.com/elliotgao2/gain/blob/master/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/elliotgao2-gain