MechanicalSoup: a Requests and BeautifulSoup browser for forms, cookies and redirects

The problem MechanicalSoup solves for Python scrapers

Most Python scraping starts with Requests and ends with BeautifulSoup. That combination fetches a page and parses it, but it does not remember a session cookie between calls, does not follow a redirect chain for you, and gives you no object that represents a filled-in form. You end up hand-building dictionaries of field names, tracking a cookie jar, and re-parsing the response to find the next URL. MechanicalSoup exists to remove that glue work. The README describes it as a library that "automatically stores and sends cookies, follows redirects, and can follow links and submit forms." The audience is anyone automating an HTML site that a browser could handle with scripting turned off: a search box, a login form, a multi-step checkout, a paginated listing. The project's own history explains the design: its author was a user of Mechanize, which was "incompatible with Python 3 until 2019" and whose development had stalled, so MechanicalSoup reimplements a similar API on top of Requests for HTTP sessions and BeautifulSoup for document navigation. If you already know those two libraries, the learning curve is mostly about the browser and form objects layered on top.

How StatefulBrowser and the form API actually work

The central object is mechanicalsoup.StatefulBrowser. It owns a Requests session, so cookies set by one response are sent with the next request without any manual jar handling. When you call open on a URL, the browser fetches it and stores the parsed result as the current page, which you reach through browser.page. That page is a BeautifulSoup document, so every selector method BeautifulSoup offers works on it unchanged. The form layer is where the abstraction earns its keep. select_form takes a CSS selector and returns a form object; indexing the browser with a field name sets that field's value. submit_selected then serializes the fields according to the form's method and encoding and sends the request through the same session. Redirects are followed by the underlying Requests session, and the browser updates its current page to the final response. The README is explicit about the boundary of this model: "It doesn't do JavaScript." There is no headless browser, no DOM mutation, no event loop. If a form's action attribute is rewritten by client-side code, or the submit button only exists after a script runs, MechanicalSoup sees the HTML the server sent and nothing more. That single sentence in the README is the most important constraint in the whole library, and it should drive your decision before you write any code.

Install MechanicalSoup and submit a real search form

Installation is a single pip command from PyPI. The README also documents installing the development version straight from GitHub with pip install git+https://github.com/MechanicalSoup/MechanicalSoup, and installing from a local checkout with pip install ., adding --user in any case to install into your home directory. The package requires Python 3.10 or newer according to setup.py, and its declared dependencies are requests >= 2.22.0 and beautifulsoup4 >= 4.7, with lxml, certifi and urllib3 listed in requirements.txt. PyPy3 is supported and tested against, per the README.

pip install MechanicalSoup

After installation, import the package and create a browser. The README example uses a user_agent string so the request is identifiable. The code below follows the shape of examples/expl_qwant.py: open a page, select the search form by CSS selector, assign a value to the q field, submit, then read the results from the new page.

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup')
browser.open("https://lite.qwant.com/")
browser.select_form('#search-form')
browser["q"] = "MechanicalSoup"
browser.submit_selected()

for link in browser.page.select('.result a'):
    print(link.text, '->', link.attrs['href'])

What you should see is one line per result, with the anchor text and the href attribute of each link. If select_form raises because the selector does not match, the form is either not in the server HTML or your selector is wrong; inspect browser.page to check. The example file notes that Qwant returns redirect links rather than final URLs, which is a general lesson: the href you read from the page is not always the destination, and resolving it may need your own parsing. The examples directory also contains expl_duck_duck_go.py, expl_google.py and expl_httpbin.py, and the README points to tests/test_browser.py and tests/test_form.py for checkboxes, radio buttons and textareas.

Where MechanicalSoup stops: JavaScript, logins and dynamic pages

The hard limitation is JavaScript, stated plainly in the README. Any page that renders its content in the browser after load is invisible to this library: you get the initial HTML, not the DOM the user sees. That rules out a large share of modern sites, including most single-page applications and anything behind a JavaScript challenge on the login path. A second, quieter constraint is maintenance cost. The last push to the repository was on 2025-05-30, which is also the date of the v1.4.0 release; the release before that, v1.3.0, was on 2023-07-04. So the project is not archived, but the gap between those two versions is roughly two years, and a user should plan on the library changing slowly rather than tracking browser behaviour closely. That matters because the web does not stand still: sites add client-side rendering, change form markup, and introduce anti-automation measures. MechanicalSoup will not adapt to those for you. Third, form filling is only as good as the HTML. If a form is built from a framework that generates field names at runtime, or if submission requires a token injected by a script, the library has no mechanism to produce it. In those cases the correct move is to reach for a browser automation tool, not to fight MechanicalSoup's model.

MechanicalSoup against Selenium, Requests and BeautifulSoup alone

The most common comparison is with Selenium, and the difference is architectural rather than a matter of features. Selenium drives a real browser: it executes JavaScript, so it can handle client-rendered content, but it needs a browser binary and a driver, and each interaction goes through the browser process. MechanicalSoup makes plain HTTP requests and parses the response, so it is lighter to install and run, and it needs no display or driver. The trade is capability: Selenium can do things MechanicalSoup structurally cannot. If your target needs JavaScript, Selenium or a similar driver-based tool is the answer, and no amount of configuration makes MechanicalSoup execute scripts. The second comparison is with Requests plus BeautifulSoup used directly, which is what MechanicalSoup is built on. The README frames the library as providing "a similar API" to Mechanize on top of those two projects. Using them raw gives you full control and no extra abstraction, but you rebuild session cookies, redirect handling and form serialization yourself. MechanicalSoup is the right choice when that glue is the bulk of your work and the site is server-rendered. It is the wrong choice when you need fine-grained control over the HTTP layer, because the browser object hides some of it.

Licence, dependencies and the cost of upgrading

MechanicalSoup is released under the MIT licence, and setup.py declares the matching classifier, "License :: OSI Approved :: MIT License". MIT is permissive: it allows commercial and closed-source use, and it requires preserving the copyright notice and licence text. That is a description of the licence, not legal advice; check your own obligations with counsel if the distribution matters. The dependency chain is short and permissive in the same way: requests, beautifulsoup4 and lxml are the declared runtime requirements, and requirements.txt pins certifi>=2022.12.7 and urllib3>=2.2.2 with comments explaining that they are indirect dependencies pinned to avoid a vulnerability. Those pins are the practical upgrade concern. Because MechanicalSoup sits on Requests and urllib3, security updates to those packages reach you through your own dependency resolution, not through a new MechanicalSoup release. The version history suggests releases are infrequent, so do not expect a MechanicalSoup bump to be the mechanism that patches an underlying HTTP library. Pin your own dependencies and update them on your schedule.

Editorial conclusion

Adopt MechanicalSoup when the target pages are server-rendered HTML and you need cookies, redirects and form submission in a few lines of Python. Do not adopt it for single-page apps, login flows that run JavaScript challenges, or pages whose content only appears after client-side rendering. Before committing, open the target form in a browser with JavaScript disabled and confirm the fields and submit endpoint are present in the raw HTML; if they are not, MechanicalSoup cannot reach them.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes