ScrapFly's scraper examples carry their own test badges

Examples for forty plus domains, all built the same way

The repository contains educational example scrapers for popular web scraping targets, built with the ScrapFly web scraping API and Python. The word educational is doing real work in that sentence, because these are not a library and not a service you subscribe to here. They are worked examples, one directory per site, each showing how to extract a particular kind of page from a particular domain.

The directory listing shows the range: Airbnb, Air France, Aliexpress, Allegro, Amazon, the App Store, AutoScout24, BestBuy, Bing, Booking.com, Capterra, ChatGPT, Crunchbase, DHL, DigiKey, eBay, Emirates, Etsy, Expedia, Facebook, G2, Glassdoor, GoodReads, Google Flights, Google Jobs, Google Play, Google itself, Idealista, Idealo, IMDb, Indeed, Instagram, Kayak, LeBonCoin, LinkedIn, Loopnet, Lowe's, Maersk, Marriott, Mouser, Naver, Nordstrom, Perplexity, Pinterest, and the real estate and retail sites. Each one is a folder with a scraper and a results/ directory of sample output.

The stack is identical across the examples, which is what makes them comparable. Python 3.10 or newer, the ScrapFly Python SDK for sending the HTTP request, bypassing blocking and parsing the HTML with the built-in parsel selector library, asyncio for concurrency with the async and await syntax, JMESPath and nested-lookup for JSON parsing when the data needs it, and loguru for logging.

The audience is a developer learning to scrape a specific site, and a developer writing their own scraper who wants a template. The tutorials the repository links to cover these targets and many others, and the topics of the repository include antibot and captcha bypass, which tells you that the hard part of scraping a large site in 2026 is not the parsing but getting the request through in the first place.

The status table is generated, and the nightly run orders it

The most interesting part of the repository is a detail in a comment above the table of scrapers.

The table is generated by a script in the .github/scripts/ directory, from the scraper folders and a scrapers.json file, and the comment says to change those and run the script instead of editing the table by hand. That is a small decision with a large effect: the list of forty plus scrapers, each with a guide link, sample dataset links and a status badge, cannot drift out of sync with the folders, because nobody types a row.

The status column is the part with the most information. It comes from the nightly test run, and that run also orders the table, with scrapers whose tests pass first and the failing and untested ones last. So the first rows of the table are the ones that worked last night, and a reader who scrolls to the bottom finds the ones that are broken or that have never been tested. That is a much better default than a flat alphabetical list, because the failure mode of a scraper is not that it never worked, it is that it worked until the site changed.

Each status cell is a badge pointing at a markdown file in a ci-status/status directory, and those files are the per scraper reports from the test run. A reader can therefore find out not just whether a scraper passed but what its last run produced, which is the difference between a badge and a log.

The design point generalises past scraping. A repository of examples that talk to live systems is a repository whose examples rot, and the way this one fights that is to actually run them every night and to publish the result in the index.

Sample output is committed for every scraper

Each scraper directory carries a results/ directory with sample output, and the table links to them per data type rather than per scraper.

Looking at the examples in the table, the granularity is finer than one file per site. The Airbnb scraper publishes a property file and a search file, so the two page types are separate outputs. Aliexpress publishes product pages, search pages, product reviews and category products, four kinds of page. Amazon publishes product pages, search pages and product reviews. The App Store scraper publishes app metadata and reviews, with no guide link, which suggests it is maintained but not written up. BestBuy publishes sitemap pages, product pages, review pages and search pages, and AutoScout24 publishes car details and listings.

Committing those files does two things at once. It documents the shape of the data a scraper produces, so a user can see what they will get before running it, and it gives the nightly test something to compare against, which is what makes a test pass or fail meaningful rather than a check that the request did not time out.

The third thing the sample files do is separate the two concerns that scraper tutorials usually blur. The guide link teaches the approach for a domain, and the sample file shows the contract, the fields, the types. A reader can copy the approach from one and the shape from the other.

And the table is a catalogue of data sources as much as of code. A list of domains, each with the page types it can extract, is a specification for what a data team could collect, which is arguably the most reusable thing in the repository.

llms.txt, and how to write a new scraper

The repository includes an llms.txt file, and the readme explains its purpose precisely: for AI agents and LLM tools, it lists every scraper with its functions and sample output, and explains how to run a scraper and how to write a new one.

That is a notable choice for a repository of examples, and it says something about where the audience is. A machine reading llms.txt can find the scraper for the domain it needs, see the functions it exposes and the shape of the output, and get instructions for running it and for contributing a new one, without parsing the generated HTML table of forty plus rows.

It also means the repository has two audiences with two different entry points, and the human one is the table while the machine one is the text file. For a project that wants to be used as a reference by tools rather than only read by people, that is a sensible piece of plumbing, and it is the kind of thing that tends to be added late; having it in the structure from the start suggests the maintainers think about consumption rather than only about publishing.

The rest of the onboarding material is linked from the readme: the full tutorials on the ScrapFly site, under a scrapeguide tag, covering these targets and many others. So the repository is the code and the blog is the teaching, and the two are meant to be read together.

The dependency on a hosted service is the real constraint

Every example here calls the ScrapFly API rather than sending a request itself, and that single fact shapes what the repository is good for and what it is not.

What it is good for is reading. The parsing logic, the concurrency structure, the data model, the sample output and the tests are all visible, and they are the parts of scraping that a target's redesign does not invalidate. A developer learning to extract structured data from a page learns that from these files.

What it is not is a dependency you can install and forget. Blocking is the reason the service exists, which means the code needs an API key, the behaviour depends on a third party's infrastructure, and the cost model is the service's rather than yours. The repository does not hide this: the SDK is described as being for sending the HTTP request, bypassing blocking and parsing the HTML, and the topics list includes antibot, proxies and captcha bypass.

For a user, that means the honest framing is that these are recipes for a particular way of scraping, not a portable toolkit. If you already run your own proxy stack with your own retry and rendering logic, the value here is the parsing and the sample data, and you will rewrite the request layer.

There is also the question that any scraping project has to answer for itself, and that this one does not settle for you. The targets include social networks, review sites, marketplaces and job boards, and each has terms of service and, in some cases, personal data regimes that apply to what you collect. The repository is educational and the authors are explicit that these are examples, but the example code still scrapes a real site, and the legal question belongs to whoever runs it. The same caution applies to the licences of the data itself, which is separate from the licence of the code.

Maintenance, licences and what the repository is for

The repository is not archived, and the last push was on 2026-10-15, with no tagged releases. The work is continuous and released as commits, which fits a repository whose contents are examples that are tested nightly rather than a versioned library that consumers pin.

The licence is where a user has to look rather than assume. The GitHub licence field is listed as a custom licence that GitHub cannot classify, and there is a LICENSE file in the repository root. For a repository of example code that people copy into their own projects, the terms in that file are the part that matters, and they should be read rather than inferred from the fact that the code is on GitHub.

The structure is worth noting as well. Each scraper is a top level directory, with the results and the sample data inside it, and the .github directory holds the generation script, the scrapers.json manifest and the nightly test configuration. The manifest is the file you edit to add a scraper, and the script turns it into the documentation table, so a new scraper arrives with its own entry, its own badge and its own sample output, and the table stays in step.

That is the whole project in miniature: a large set of worked examples, kept honest by a nightly test run, published with a generated index, and pointed at a hosted service that does the part of the job that a laptop cannot.

For a reader deciding whether to use it, the order is simple. Look at the status badge for the domain you are interested in, read the guide and the sample output for it, and use the example as a pattern rather than as a dependency.

Editorial conclusion

This repository suits a developer who wants to see how a scraper for a specific site is put together, including the parsing, the sample output and the shape of the code, and it suits a team that wants a reference for writing their own. It is the wrong choice if you need a dependency you can rely on without a hosted service in the loop, because every example calls the ScrapFly API, and a poor match if you need the code to run against the target directly with your own proxy and retry logic. Before using anything here, read the licence file, since the licence field is listed as unclassified, and check the status badge for the domain you care about, because a scraper whose nightly test fails is a scraper whose selectors have drifted.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. scrapfly/scrapfly-scrapers on GitHub