stealth-requests: a requests-like scraper with the parsing built in

What it is, and the single dependency

Stealth-Requests presents itself as the easiest way to scrape the web, and the package description is blunter: undetected web scraping and easy HTML parsing in Python.

The design has two halves that happen to be joined in one package. The first is an HTTP client whose API is deliberately a copy of the one everybody already uses, so that the mental cost of adopting it is close to zero. The second is a set of parsing conveniences attached to the response object, so that the data you usually want comes out without you writing the extraction yourself.

What is striking is the dependency situation. The manifest lists exactly one required dependency, the library that does the browser impersonation, and everything else is optional. The parsing extras, which are the tree parser, the HTML to text converter and the soup library, are in an optional group that has to be asked for at install time.

That is a very lean core, and it is worth understanding what it means. The HTTP layer is the part the package cannot work without, so it is the part it depends on directly. The parsing layer is the part a user might already have, might want a different implementation of, or might not need at all, so it is kept out of the way until it is asked for.

The package requires Python 3.10 or newer, is MIT licensed, and is published to the package index with a badge showing the install count. The version in the project manifest is ahead of the newest tagged release, which tells you the development tree is further along than the published artefact, and that is worth knowing if you are pinning versions.

The repository is small and well organised: the package directory, a tests directory, the project manifest, a lock file for the fast package manager, and a readme. There is no documentation directory, which means the readme is the documentation, and it is long and structured with a table of contents rather than a short summary. For a library whose whole surface is a small number of objects and methods, that is a reasonable place to put the reference.

The request layer

The request layer is where the package's name comes from, and its feature list is explicit about what it does.

It mimics a Chrome browser for undetected scraping, using the impersonation library as the transport. It automatically rotates the user agent between requests. It tracks and updates the referer header so that the chain of requests looks like a chain rather than a series of unrelated hits. And it has built-in retry logic for failed requests, with the examples given being the too-many-requests, service-unavailable and connection-timeout status codes.

Those four are a coherent set, and each one addresses a different way a naive client gets noticed. The transport is the deep one: a client that presents an ordinary TLS handshake does not look like the browser it claims to be, and there are now checks that look at exactly that. The user agent is the shallow one, and rotating it is the oldest trick in the book, though rotating it without keeping the rest of the request consistent is worse than not bothering, because a request whose user agent claims one browser and whose other headers claim another is more suspicious than either. The referer is the one that models a browsing session rather than a single hit, since a page fetched with a referer of another page on the same site is a request a browser makes, and a request with no referer is a request a script makes.

And the retries are the feature that makes the other three bearable in practice. A rate-limited response is not a failure, it is an instruction, and a client that waits and retries is behaving the way a well-mannered client should. The readme says the built-in retry waits two seconds and retries the request, and the number of attempts is passed as an argument.

The interface itself is the familiar one, and there are two forms of it. There is the one-off call, where the package is imported under the name of the library it mimics so that the rest of the code does not change, and there is a session object that keeps certain headers between requests, the referer being the one named.

import stealth_requests as requests

resp = requests.get('https://link-here.com')

The session form is the interesting one, because it is where the referer tracking actually lives. A one-off call has no previous request to derive a referer from, so the chain-building only works within a session, and the session is the object that has to be reused for the effect to be real.

The response object and its metadata

The response is a custom class, and the readme explains the relationship to the standard one in a single sentence: it has all of the same methods and attributes as a standard response object, with a few added features.

That is the right way to build this kind of thing. A wrapper that is a superset can be dropped in where the original was used, and the code that only needs the original behaviour keeps working. A wrapper that replaces the type is a rewrite.

The added feature that gets its own property is metadata extraction, and the property is called meta. It is parsed automatically from responses that are HTML, and the fields are listed with their types: a title, an author, a description, a thumbnail, a canonical link, a Twitter handle, keywords, and robots.

That list is a social and search preview in one: what a link preview service would show for a page, which is a genuinely useful set of fields for anybody building a link shortener, a reader, a feed, or an archive. The fact that they come out of one attribute rather than a dozen separate lookups is the convenience the package is selling.

The robots field is the one worth pausing on, because it is unusual for a scraping library to surface it. Most such libraries either ignore the directives or warn about them in the documentation. This one parses them out of the page and hands them to you, which does not make the tool compliant, but it does mean the information is right there in the object when you decide what to do with a page.

The other convenience properties are for the four things people most often want out of a page. There are emails, telephone numbers, images, and links, each returned as a tuple of strings, and the readme shows example output for each. The telephone numbers in particular are worth noting: the example output shows both an international format with punctuation and a local one, which means the extraction recognises the conventions rather than just stripping the tags.

And the metadata access is used the way you would expect, through a property rather than a method.

The design principle across all of it is that the response knows how to read itself, so the calling code is short. Whether that is a good idea depends on your taste: a response object that parses its own body is doing work that arguably belongs elsewhere, and for a large document the parsing cost lands on the first attribute you touch rather than on an explicit call. For a script, the convenience is worth it, and for a long-running service, it is a design decision to make deliberately.

Tables, and the parsing extras

HTML tables are the case that most often breaks a scraper, because they are the one structure that looks like data and is not quite. A table without headers is not a table with empty headers, it is a layout device, and a parser that treats it as data produces nonsense.

Stealth-Requests handles that by skipping tables that have no recognisable headers, and the readme says so explicitly. The tables it does recognise come back as dictionaries, where each key is a column header and each value is the list of that column's cell values.

That representation is a sensible middle ground between a list of rows and a fully typed table. The column-oriented form is what you want for a spreadsheet, for a database load, or for a comparison across rows, and it is also what makes the missing-header case detectable in the first place: if the first row cannot be read as headers, there is nothing to key the columns on.

The parsing extras install two well known libraries alongside the package, so the response can be handed to either of them when the convenience accessors are not enough. There is a method that gives you the tree from the fast parser and a method that gives you the soup object, and both are one call.

And then there are two convenience methods taken from the tree library and put directly on the response. One gets all the text content of a response, which is what you want when you want the words and not the structure. The other evaluates an expression against the document, which is the shortcut that lets you skip building a tree at all when you know the one element you want.

The second of those is a bigger deal than it looks, because it is the difference between a two-line script and a five-line one. The usual pattern for this library is: make the request, build the tree, find the element, read the text. With the expression method, three of those four steps collapse into the call, and the tree is only built if you actually need it.

The extras are installed as a named group, so the cost of the convenience layer is opt-in. That is a good default for a package whose core is an HTTP client, because most of its users are making requests rather than parsing documents, and the ones who do parse can install the parsers they want.

Markdown conversion, and the two knobs

Sometimes the useful form of a web page is not its HTML and not a parsed tree but Markdown, because that is what you want when you are feeding the page to something that reads text: a model, a search index, a document pipeline, or a note-taking system.

The response has a method for that, and it takes two optional arguments, and both of them are worth understanding because they are the difference between a readable document and a wall of navigation.

The first is an expression, in string form, that narrows down what part of the document is converted. The readme's example of what it is for is a good one: excluding the header and the footer of the page, which are the two parts that a naive conversion turns into forty lines of navigation links and site chrome. Pointing the expression at the article body gives you the article.

The second is a boolean that tells the underlying text converter whether to include links in the output. That is a smaller decision than it looks, because it changes what the output is for. With links, the Markdown is a document you can click through, which is what you want for an archive or a reading list. Without them, the Markdown is a body of text, which is what you want when the text is going into a model or an index and the link targets are noise.

The conversion is delegated to a separate text conversion library, which is named in the readme, and that is worth noting for the same reason the tree parsing is delegated: the package is not trying to be the best HTML to text converter, it is trying to give you a good default that you can point elsewhere when you need to.

What is missing from the readme is any mention of JavaScript execution, and that absence is informative. Every method described here works on the HTML the server returned. If a page needs a script to render its content, this library will not see it, and the result will be the shell of the page rather than the page. That is a limit rather than a bug, and it is the limit that separates this kind of library from a browser automation tool, which is a different cost and a different set of obligations.

Asyncio, proxies, and the tooling

The asynchronous support mirrors the synchronous one, with a session class of its own that is used as an asynchronous context manager, so the shape of the code is the same and the only difference is the await.

Proxy support is there too, and the readme's section on it is the last one in the document, so the details are cut off in the version available here. What can be said is that it exists, which matters because proxy support in a scraping library is rarely about one proxy: it is about rotation, about routing, and about separating the address that makes the request from the address the site sees.

The project manifest is where the tooling conventions become visible. The linter is configured with a line length of a hundred and twenty, and the selected rules are the import-sorting and the tidy-imports ones, with single quotes for formatting. That is a small, sharp configuration: a formatter and a linter that between them enforce a consistent style and a consistent import order, with nothing else.

The test configuration points at a tests directory, and the development dependency group includes the tree parser, the test runner, the asynchronous plugin for it, and a notebook library. The presence of the asynchronous plugin tells you the asynchronous paths are tested rather than assumed, which matters for a package whose asynchronous surface is the one people will hit when they are making many requests.

The notebook dependency in a development group is a small curiosity. For a scraping library whose selling point is that you do not have to write a parsing script, a notebook is a natural place to demonstrate it, and it is a sign that the examples are meant to be run and poked at rather than copied.

And the lock file for the fast package manager is checked in, which means the development environment is reproducible, and the build system is the standard one with the setuptools backend and a minimum version declared. Nothing about that is remarkable, and that is the point: the packaging is conventional so that nothing about it is the reason the library is hard to use.

Editorial conclusion

Stealth-Requests fits a Python developer who wants to fetch a page and have the obvious things already extracted, and who would rather not assemble a parsing stack for every script. It does not fit a job that needs a maintained, dependency-light client for production traffic, since the whole point of the library is to look like something other than what it is. Verify first whether the site you are fetching permits automated access, because the library also parses the robots metadata out of the page and makes it available to you, that the parsing helpers you rely on are in the optional extra and not in the core, since only the HTTP layer is a required dependency, and that the personal data rules apply to what you collect, because extracting addresses and telephone numbers from pages is the part of scraping that most often runs into the law rather than a technical obstacle.

Frequently asked questions

Official sources

1. jpjacobpadilla/Stealth-Requests on GitHub
2. License: MIT
3. Project website
4. README
5. Releases