social-media-profile-scrapers fetches public profiles through one shared engine

One script per platform, all calling the same engine

The repository holds a separate script for each supported site, which is facebook.py, github.py, instagram.py, medium.py, pinterest.py, quora.py, reddit.py, tiktok.py and twitter.py. What makes the collection coherent is that they all share one fetching mechanism, scraper_engine.py, and the README states plainly that all scraper scripts share it. The per-site scripts are therefore thin: they describe what to ask for and how to present the result, while the transport, the retry behaviour and the browser handling live in one place. That is a sensible structure for a project that will be patched frequently, because when a platform changes something you usually fix one engine method rather than nine scripts. The repository also carries tests.py, a requirement.txt, a GraphQL query file for the Medium scraper in medium_graphql_query.graphql, an assets/ directory, and a screenshot/ directory with the captured output.

A curl_cffi request first, a real browser only as a fallback

The engine's design is the part with the engineering in it. Each script first tries a fast curl_cffi request, described as browser impersonated TLS with no browser started, and automatically falls back to a real browser through SeleniumBase UC mode, which is undetected-chromedriver, when the HTTP attempt does not return usable data. The order matters. The HTTP path is fast and cheap, and in the majority of cases it succeeds, so the browser is reserved for the cases where a plain request gets blocked or returns a page the parser cannot use. The engine is also extensible: the README notes that new backends such as a paid API service or FlareSolverr can be plugged in without touching the scripts, with the module docstring of scraper_engine.py as the place to look. The extra flags expose the choice, with the default being the HTTP attempt followed by the UC browser, a headed flag to run the fallback browser non-headless, and a JSON flag to print the result as JSON.

The browser is optional for three platforms and a fallback for GitHub

The usage section is more nuanced than a single command, and the exceptions are the interesting part. The general invocation takes a script name, a username and a browser name, with chrome as the default when the browser argument is not passed, and only firefox and chrome are currently supported. Three scripts do not need a browser at all, since Pinterest, Medium and Twitter can be used as plain invocations with a username. GitHub is the most interesting case, because the script reads the GitHub REST API directly and a browser is only started as a fallback for the yearly contribution count, and that fallback can be skipped with the --no-contributions flag. The environment therefore has three tiers: no browser needed, an API call with an optional browser fallback, and a full browser driven path. A user running these on a server without a display needs to know which tier each script falls into.

The command line is the whole interface

There is no package to install and no library to import, which makes the learning curve small. The install step is one command in the project directory:

    pip install -r requirement.txt

And the run step takes a script, a username and optionally a browser:

    python SCRIPT_NAME USERNAME --browser BROWSER_NAME

Per script help is available with the standard flag, for example:

    python SCRIPT_NAME -h

The engine flags then layer on top of the per-script ones:

python facebook.py USERNAME --engine auto     # curl_cffi first, then UC browser (default)
python facebook.py USERNAME --headed          # run the fallback browser non-headless
python facebook.py USERNAME --json            # print the result as JSON

The stated technology underneath is chromedriver or geckodriver, the selenium Python bindings, a fake-headers package, and webdriver-manager for the driver binaries. The combination of webdriver-manager and SeleniumBase means the driver version is fetched on demand, which removes a common installation failure but adds a network dependency at first run.

The legal ground is not addressed in the documentation

Nothing in the README discusses terms of service, rate limits, robots directives or the legality of automating against these platforms, and the design of the engine makes the omission worth naming. The whole approach is built around not looking like a bot, with browser impersonated TLS in the HTTP path and an undetected browser driver in the fallback. That is a deliberate evasions posture, and it means the scripts are most likely to work against platforms whose terms prohibit automated collection. The licence situation is Apache-2.0 for the code, which is straightforward, and the README invites suggestions for more social media. There is also a sponsor section promoting a commercial scraping platform with a free trial offer, which tells you the maintainer is aligned with the scraping tooling market rather than with a research agenda. None of that makes the code unusable for legitimate research on accounts you own or on data you have permission to collect, but it does mean the person running it should be clear about which side of that line they are on.

One script per site means breakage is a per-site problem

The maintenance reality of this design is worth stating plainly, because it determines whether the project is a tool or a set of examples. There are no GitHub releases, so there is nothing to pin, and the default branch is master. The last push was on 2026-09-15, so the code is being worked on. But a scraper for a third-party site is a snapshot of that site's markup on a particular day, and when the markup changes the corresponding script stops working until someone rewrites the selector. The engine reduces the blast radius of that change, since a request that returns HTML is the same problem regardless of site, but it cannot remove it. There is a further factor, that the sites with the strictest defences are the ones where the HTTP path fails and the browser fallback runs, and the browser fallback is exactly the part that is most sensitive to changes in the client environment. The realistic expectation is that some scripts work today, some need small repairs, and the ones for the most protected platforms need work more often.

Where this collection is the wrong tool

Three situations argue against it. The first is production data collection, where a scraper that breaks silently when a site changes is worse than no scraper, because the pipeline looks healthy while the data goes stale. For that you want a maintained library per platform with a commercial backstop, or an official API. The second is a task that needs more than the public profile, because these scripts fetch what a visitor to the profile page sees, so anything behind a login, in a private post or in a closed group is out of scope by construction. The third is a team that needs an audit trail, since the JSON output is a snapshot with no record of when it was taken, how it was obtained, or what was returned when the request failed, and a collection tool without provenance is a liability in a compliance context. The realistic alternative for anything beyond a one-off lookup is the platform's own API where one exists, which for GitHub is exactly what the script already does, and for the other platforms means a data provider such as the one the repository sponsors. The scripts are most valuable as a learning reference for how the curl_cffi plus browser fallback pattern is put together, and that is a real use even when you do not run them yourself.

Editorial conclusion

Adopt these scripts when you need a one-off public profile lookup across several platforms and are willing to repair them when markup changes, and write your own scraper or use a paid extraction service when the collection has to keep running. Verify first which platforms the current code still handles, since the repository ships one script per site with no version pinning, and read the terms of service of each platform before automating against it, because the technical approach is built around not being detected.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. shaikhsajid1111/social-media-profile-scrapers on GitHub