AutoScraper: Learning Scraping Rules from a Sample Instead of Writing Selectors

The problem AutoScraper solves: writing selectors before you know the page

Most Python scraping starts with a manual step. You open the page, inspect the DOM, guess a CSS selector or XPath, run it, find it matches twelve extra elements, and iterate. AutoScraper inverts that. You supply one or more sample values that already exist on the page, and the library derives the matching rule for you. The README frames the input as "a list of sample data which we want to scrape from that page" and states that this data "can be text, url or any html tag value of that page." The intended user is someone who knows what the output should look like but does not want to reverse-engineer the markup to get there. That is a reasonable position for one-off extraction jobs, for internal tools, and for pages whose class names are hashed or meaningless. It is a weaker position when the page is a core dependency and you need a selector you can read, version-control and explain to a colleague.

How build() turns a sample into a rule

The flow has two phases. First, build() fetches the page (or accepts raw HTML through the html parameter, as the README notes) and searches for the sample values you passed in wanted_list. From the positions where those values were found, it infers a set of rules that describe the surrounding structure. Second, get_result_similar() or get_result_exact() applies the learned rules to a new URL. The README's Stack Overflow example is the clearest illustration: you pass a single title, "What are metaclasses in Python?", and build() returns a list of nine related question titles from the same page, including the one you supplied. The library did not just find your sample; it generalised from it to sibling elements. get_result_exact() behaves differently. In the Yahoo Finance example, the wanted_list contains a price like "124.81" and get_result_exact() is used to pull the same field from a different ticker page. The README also states that if you append more values to wanted_list, get_result_exact() "will retrieve the data as the same exact order in the wanted list," which makes it usable for building a fixed-width row of fields rather than a bag of matches.

Getting it installed and running the two-argument example

Installation is a single pip command: pip install autoscraper. The README also documents installing from the git repository with pip install git+https://github.com/alirezamika/autoscraper.git and from source with python setup.py install. The minimal script is short. Import AutoScraper, set url and wanted_list, instantiate the scraper, and call scraper.build(url, wanted_list). The README's Stack Overflow example uses exactly that shape and prints a list of question titles. To reuse the learned rules on another page, call scraper.get_result_similar('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string') for sibling-style matching, or scraper.get_result_exact(...) when you need the same field position. One detail worth noting: the README warns that in the Yahoo Finance example you "should update the wanted_list if you want to copy this code, as the content of the page dynamically changes." That is not a throwaway remark. The sample value is the training signal, so a stale sample produces a rule built against content that no longer exists.

Passing proxies and headers through request_args

AutoScraper delegates fetching to the requests library and exposes its parameters. The README shows a proxies dictionary with http and https keys passed as request_args=dict(proxies=proxies) into build(). The same mechanism is how you would add custom headers. This matters more than it first appears. Many sites return different HTML to a default Python user agent, and if build() learns rules from a stripped-down or blocked response, the rules will not match the real page. Because request_args is a plain dict forwarded to requests, anything requests accepts should be passable here. The README only demonstrates proxies, so treat headers as an inference from the mechanism rather than a documented example.

Saving and loading the learned model

Once a scraper is built, scraper.save('yahoo-finance') writes the learned rules to a path you choose, and scraper.load('yahoo-finance') restores them. The README gives no format details, no schema, and no versioning guidance for the saved file. That is a real gap if you plan to check the model into a repository or ship it inside a package: you have no documented way to know whether a model saved by one version of the library will load correctly in another. The practical consequence is that the saved model is a build artifact tied to the library version that produced it, and any upgrade of autoscraper should be followed by re-running build() against a live page to confirm the rules still resolve. The release history is thin on this point: the most recent release listed is v1.1.14 from July 2022, with v1.1.12 and v1.1.11 before it.

Where AutoScraper breaks: dynamic pages, scale, and unreadable rules

The library operates on HTML it can fetch with requests. It has no browser engine, so anything rendered client-side by JavaScript will not be in the response and cannot be learned from. The README's own Yahoo Finance example is a hint here: the note that page content "dynamically changes" and that wanted_list must be updated is exactly the kind of instability that makes sample-based learning fragile. A second limitation is scale. There is no documented queue, no retry policy, no rate limiting, no concurrency model in the material provided. If you need to crawl thousands of pages politely, you are building that layer yourself around AutoScraper calls. A third issue is debuggability. Because the rules are inferred rather than written, a wrong result gives you a list of wrong strings and no selector to inspect. When get_result_similar() returns neighbours you did not want, the fix is usually to add more sample values to wanted_list, not to edit a rule. That is a different debugging loop from the one most engineers are used to.

Compared with BeautifulSoup and Scrapy

The nearest alternative for the extraction step is BeautifulSoup. With BeautifulSoup you write soup.select('.some-class') and you get exactly what the selector matches. With AutoScraper you pass a sample and accept whatever rule the library infers. The trade is explicit: BeautifulSoup gives you a selector you can read, test and reason about, at the cost of inspecting the markup first. AutoScraper skips the inspection but hands you a rule you cannot read. For a page you will scrape once, that is a good trade. For a page you will scrape daily for a year, a hand-written selector that breaks visibly is often easier to maintain than an inferred rule that breaks quietly by returning a slightly different list. Scrapy sits at a different level entirely: it is a crawling framework with scheduling, middleware and pipelines, and AutoScraper could in principle be used inside a Scrapy spider for the extraction step. Choosing between them is choosing between a framework and a parsing helper, not between two parsers.

Licence, maintenance and what to verify before adopting

AutoScraper is MIT-licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence and imposes no copyleft obligation on your own code. This is a description of the licence text, not legal advice; if the licence terms matter to your organisation, have counsel review them. On maintenance: the repository is not archived, and the most recent release in the supplied material is v1.1.14 from July 2022, so the release cadence is slow. That is not disqualifying for a small library with a narrow job, but it does mean you should not expect the saved-model format or the inference behaviour to be actively stabilised. The concrete thing to verify before adopting is behaviour on your own target page, not the README's examples. Run build() with a single sample value and check whether the returned list contains the sample plus the neighbours you actually want. Then run get_result_exact() with a multi-value wanted_list and confirm the output order matches the input order, since the README states that ordering is guaranteed for that method. Finally, save() and load() the model and confirm the loaded object produces the same results without re-fetching the original page.

Editorial conclusion

Adopt AutoScraper when the target page is server-rendered, the fields you want can be named by pasting one real example of each, and you are willing to re-run build() whenever the site changes. Do not adopt it for JavaScript-rendered pages, for sites you must scrape at scale with retries and throttling, or where you need a selector you can read and debug by hand. Before committing, verify three things on your own target: that build() returns the sample value itself plus the neighbours you expected, that get_result_exact() returns fields in the same order as your wanted_list, and that save()/load() round-trips without the library re-fetching the page.

Official sources

1. alirezamika/autoscraper on GitHub
2. Issues
3. License: MIT
4. README
5. Releases