socid_extractor turns a profile page into a record with the platform's internal ID

The output is a flat dictionary with a shared field ontology

socid_extractor turns any public profile page into a structured account record, with usernames, display names, bios, avatars, locations, joined-at dates, follower counts, external links, and the stable internal identifiers that pin an account across renames, redesigns and deletions. The project description calls it the extraction engine behind Maigret, and the topic list names OSINT as the field.

The mechanism is a single function over response text. It parses HTML pages and API responses from 250+ schemes and returns a flat, machine-readable dictionary of account fields, with no API keys required and no headless browser, just a single function call on response text.

The design decision that makes it useful is the field ontology. Normalized field names are used across platforms, for example username, fullname, created_at and is_verified, so downstream pipelines do not need 130 mappings. The repository holds FIELDS.md as the specification of that ontology, METHODS.md as the full list of supported sites, and socid_extractor/schemes.py as the parsing logic.

A flat dictionary is a deliberate simplification. It means the result drops straight into a JSON document, a database row or a search index, and it means there is no object model to learn. The cost is that a field present on one platform and absent on another is simply absent, and the caller has to know which fields to expect.

The library is MIT licensed and free for commercial use without restriction, and it is published on PyPI as socid-extractor. The last push was on 2026-10-04.

The stable internal ID is the whole reason to use this

The first listed use of the library is the feature that distinguishes it from a generic scraper, and the README is explicit about why it matters.

The useful part of a profile page is not the display name or the bio, both of which a user can change at will. It is the platform's internal identifier: the GAIA ID from Google, the Facebook UID, the Yandex Public ID, the Instagram pk, and dozens more. Those values survive username changes, and they are the join key for correlating an account across leaks, archives and search engine indices.

That is what the use cases are built on. The first is pivoting from a profile to everything you can see, where one call returns the visible information plus the hidden internal IDs the platform uses behind the scenes. The second is tracking accounts across renames, redesigns and deletions, so that you can re-identify the same person even when every visible field has changed. The third is searching by cross-service UID, which means pivoting into SQL or leaked databases such as forum dumps and breach data where the UID is the join key, and into Google, Yandex or archive.org indices that captured URLs containing the UID.

The third use case is the one to think about ethically, because it is the one that turns a scraping library into an investigation tool. The library itself is neutral, a parser over public pages, and the README frames the pivot in terms of correlating public data with public archives. But the tool is only as neutral as the use of its output, and a team deploying it should be clear about that boundary before they start.

The README also maps the project onto a classification system called SOWEL, listing the techniques Analyze Internal Identifiers and Check Outdated And Unused Functionality, which is a way of naming what the library does and what breaks it.

Installation is pip or pipx, and the CLI gives you the record directly

The installation section is short. Python 3.10 and later is required, and the package installs from PyPI:

pip install socid-extractor

For a clean CLI install on a workstation, the recommendation is pipx, which isolates the tool rather than installing it into the current environment:

pipx install socid-extractor

The development version comes straight from the repository:

pip install -U git+https://github.com/soxoj/socid-extractor.git

The CLI usage shows the shape of the output, and the field names are the same normalized names the ontology defines:

$ socid_extractor --url https://www.deviantart.com/muse1908
country: France
created_at: 2005-06-16 18:17:41
gender: female
username: Muse1908
website: www.patreon.com/musemercier
links: ['https://www.facebook.com/musemercier', 'https://www.instagram.com/muse.mercier/', 'https://www.patreon.com/musemercier']
tagline: Nothing worth having is easy...

As a library, the call is a single function on the response text:

import requests
import socid_extractor

r = requests.get('https://www.patreon.com/annetlovart')
print(socid_extractor.extract(r.text))
# {'patreon_id': '33913189', 'patreon_username': 'annetlovart',
#  'fullname': 'Annet Lovart',
#  'links': "['https://www.facebook.com/322598031832479', ...]"}

Note that the library does not fetch. The caller fetches, with requests or anything else, and passes the text in. That separation is what makes the library testable and what lets you use a session with your own headers, proxies or rate limiting.

There is also a performance flag, --skip-fetch-if-no-url-hint, which skips the HTTP request when the URL does not match any known site hint. The README notes it is faster, but that it may skip generic engines such as forum templates, which is the trade.

250 schemes, and a parser that will rot

The supported site list is the most honest indicator of what the project is, and the README is careful about its framing. There are 250+ schemes, covering roughly 2,000 of the sites Maigret checks, with the full list in METHODS.md.

The sample covers the major networks, Facebook user and group pages, Instagram, VK.com, OK.ru, Reddit, TikTok, Bluesky, Tumblr and Flickr. The Google ecosystem is there, with docs and maps contributions needing cookies, plus Google Play and YouTube. Mail.ru has user mainpage, photo and video pages. The dev and writing platforms include GitHub, Stack Overflow with both HTML and API paths, LeetCode, Hashnode, Medium, Substack, Paragraph, WordPress.org and Virgool. The forum detectors are the interesting engineering, because Discourse, MediaWiki and Fandom wikis, and Mastodon, are universal templates, so one detector covers thousands of sites at once.

The niche and vertical list shows how far the coverage goes: Chess.com, Roblox, MyAnimeList, Scratch, Wikipedia, DailyMotion, SlideShare, Weebly, Calendly, Amazon Author, Boosty, Warpcast for Farcaster, Fragment for TON and Telegram, Rarible, CSSBattle, lnk.bio, Spatial, TwitchTracker and Max.

The decay problem is stated by the project itself. The Commercial Use section says the open-source socid_extractor is MIT licensed and free for commercial use without restriction, but that page parsers break over time as platforms change their HTML and APIs, and they need active maintenance. For serious commercial use, such as a hosted extraction API, or maintenance of the parsers your product depends on, the project points to a contact address, and the Commercial Use list names an extraction API offering.

The release history supports that reading. The versions are v0.1.1 from 2026-07-13, v0.1.0 from 2026-05-26 and v0.0.28 from 2026-04-09, so the project only reached a 0.1 line in the middle of 2026 even though the parsing work behind Maigret is older.

Maigret is the consumer, and the plugin API is the extension point

The tools section lists Maigret, described as a powerful namechecker that generates a report with all available info from accounts found across 3000+ sites, and TheScrapper, described as scrape email.

The relationship between the two is worth stating clearly, because it explains the split between the engine and the tool. Maigret does the searching across thousands of sites and produces the report, and socid_extractor does the parsing of the pages that Maigret finds. That is why the library covers 250+ schemes and roughly 2,000 of the sites Maigret checks: the rest of Maigret's coverage is handled by Maigret's own detectors, and the two overlap where the generic engine is better than a site-specific one.

The extension mechanism is a plugin, and the repository carries an example at examples/example_plugin.py. The documented extension points are the same pattern used by similar tools: a hook that recognises a URL or a page and returns fields.

There is also a SOWEL classification section, mapping the project onto techniques including Analyze Internal Identifiers and Check Outdated And Unused Functionality, and linking to a site at sowel.soxoj.com. It is the author's own taxonomy, and it is a way of naming the two properties of the tool that a user should think about: what it gives you, and what decays.

The package metadata is worth a note for anyone integrating it. The version is dynamic, read from socid_extractor/__init__.py, the build backend is setuptools with wheel, and the dependency versions use greater-than-or-equal and less-than ranges in a maigret style so that downstream installs are not pinned to a single point release but also cannot silently jump a major version that might break the parsing.

Where socid_extractor is the wrong tool

The first wrong-tool case is a team that needs a maintained service. The project says so directly: parsers break over time and need active maintenance, and the commercial path is a hosted extraction API or a maintenance contract. Adopting the open source library means adopting the maintenance of the parsers your product depends on, and that work is periodic and unfun.

The second is a workflow that starts from a username rather than a page. This library parses a URL you already have. Discovering which profiles exist for a given name across thousands of sites is Maigret's job, not this one's, and the difference matters for how you build a pipeline around it.

The third is anything that needs authentication or headless rendering. The library works on response text, with no headless browser, which is its main practical advantage and also its limit. A site that renders its profile in JavaScript, or one behind a login, gives you a response that this library cannot extract fields from, and the README notes that the Google docs and maps contributions need cookies.

The fourth is scale and politeness. The tool is a parser, not a crawler, so the rate limiting, the retry policy and the respect for a site's terms are all yours. A pipeline that runs thousands of profiles through this needs the kind of control and pacing logic that belongs in the calling code, and the --skip-fetch-if-no-url-hint flag is a small hint of how much of that is expected upstream of the library.

Editorial conclusion

socid_extractor fits an OSINT engineer who already has a profile page and wants normalised fields plus the platform's internal identifier from it, particularly anyone building on Maigret. It does not fit a team that wants a maintained hosted service or who cannot accept that page parsers decay. Before adopting it, read METHODS.md to confirm the sites you care about are covered, and note that the project itself states parsers break over time as platforms change their HTML and APIs, which is why it offers a commercial extraction API and parser maintenance.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. soxoj/socid-extractor on GitHub