Facepager: a desktop pipeline for collecting public web data into SQLite

What Facepager collects and where the data ends up

Facepager is described in one line as a tool for fetching publicly available data from YouTube, Twitter and other websites on the basis of APIs and web scraping. All of that data is stored in a SQLite database and may be exported to csv.

The framing matters, because the product is not a library. It is a desktop application with a graphical interface, an installer for each version, a help button, and a wiki, aimed at a person who runs a collection rather than a developer who writes a pipeline in code.

The problems it takes on are enumerated in the README: multi-threaded data collection, rate limits, pagination, data extraction, and many other aspects which otherwise require extensive development work. That list is the honest description of what a scraper needs and what most one-off scripts get wrong. Rate limiting in particular is the thing that turns a weekend script into a two-week project once you respect the API terms.

The output side is deliberately plain. SQLite is a single file you can copy, and csv export is one click away. For a researcher who wants to hand a table to a colleague or load it into a statistical tool, that avoids writing an export path and avoids putting a database server in the way.

The citation in the README gives the academic context, crediting Jakob Jünger and Till Keyling for a 2019 paper describing Facepager as an application for automated data retrieval on the web. The paper it refers to is linked from the README, with links to the two authors' institutional pages.

The presets have moved from social platforms to knowledge infrastructures

The README carries a prominent note about support for the official Facebook, Twitter and YouTube APIs being limited, and it explains what happened. In the last years the project provided presets with pre-registered access to support the community, but due to changes in the APIs and the increasing complexity of the review processes, that service could not be maintained anymore. The consequence for users is concrete: to use Facebook, Twitter or YouTube with Facepager, you need to register for your own API access and provide the client ID and client secret in the respective settings.

So the presets that remain are aimed elsewhere, and the README says the promoted use cases now focus on APIs in the domain of knowledge infrastructures. The list is long and specific: Open Library, OpenAlex, Crossref, arxiv.org, Nominatim geo coding, the Google Maps API, Geonames, Wikidata, Wikipedia, DBpedia, SemOpenAlex, Culture Knowledge Graph, Getty Thesaurus, Mastodon and Reddit, plus web scraping from news sites.

The shift is a technical reality rather than a preference. Pre-registered API access that a project can hand to any user was viable when the platform review processes were simpler, and it stops being viable when they are not. A tool that depends on a courtesy credential has a shelf life tied to someone else's policy.

There is also a second use case the README mentions: interacting with cloud services such as OpenAI APIs or Amazon Web Services. The project is candid that this involves a decent understanding of the respective APIs, and that you use Facepager to set up pipelines with your specifically targeted pipelines. In other words, for those services Facepager is a shell for your own configuration rather than a ready-made integration.

Installing: an installer for Windows and macOS, source only for Linux

Installation packages for each version are on the releases page, and the README warns in the same line that database files may be incompatible between versions. That warning is the first thing to internalise, because it means an upgrade is not always a safe operation on an existing collection.

For Windows, you download and execute the latest exe-installer from the releases page. The README notes that if Windows complains about an unknown publisher and refuses to launch the app, you click More info and start anyway.

For macOS, you download and install the latest package from the releases page. The system will complain that it cannot install the package, and the README links to Apple's guidance on opening an app from an unidentified developer. The path it describes is System Settings, then Privacy and Security in the sidebar, then scrolling to the Security section and clicking Open and Open Anyway.

Older versions of Facepager were distributed as zip files and were not code signed, and the README gives a different path for those: unzip the file, drag the app into the Applications folder, then remove the download flag from the terminal with `xattr -cr Facepager.app` and open the app using ctrl-click.

For Linux there is no binary release at all, and the README points to `src/readme.md` for steps to run under Linux. The same file covers running from source.

The practical shape of this is worth stating plainly. The supported, tested path is a graphical application installed on Windows or macOS. Linux users are running from source, and the source instructions live in a separate file rather than in the main documentation.

The dependency list tells you what a collection run touches

The `requirements.txt` file in the repository lists the Python packages Facepager needs, and the list explains a lot about how it works.

The HTTP layer is `requests`, with the `requests[security]` and `requests[socks]` extras, plus `requests_oauthlib` and `rauth` for OAuth flows and `requests-toolbelt` for multipart and streaming uploads. The `rauth` dependency in particular indicates that the OAuth services are handled in the application rather than left to a wrapper script.

For parsing, there is `lxml` for structured extraction, `xmljson` to move between XML and JSON, `cssselect` for CSS selector support, `pyjsparser` for JavaScript parsing, and `markdown` for rendering. `tldextract` handles the public suffix problem, which is what you need to split a URL into domain and subdomain correctly, and `charset-normalizer` deals with encoding detection, a recurring source of garbled scraped text.

The storage layer is `SQLAlchemy`, and the graph layer is `rdflib`, which fits the knowledge infrastructure presets since Wikidata and DBpedia are RDF sources. The interface is `PySide6` with the `PySide6-Addons` extra, so this is a Qt based desktop application. `pillow==11.3.*` is pinned to a specific patch line, and `python-dateutil` and `datetime` cover the date handling that any collection tool needs.

The full list, as recorded in `requirements.txt`, is:

SQLAlchemy
python-dateutil
requests
requests[security]
requests[socks]
requests_oauthlib
requests-toolbelt
rauth
lxml
xmljson
charset-normalizer
pyjsparser
tldextract
cssselect
markdown
PySide6
PySide6-Addons
rdflib
pillow==11.3.*

One dependency is worth flagging as a maintenance point. `pillow` is pinned to the `11.3.*` line, so an environment that upgrades Pillow beyond that will not match what the project was built against.

Architecture as the repository layout reveals it

The top level of the repository is a reasonable map of the application. There is an `apis/` directory, which is where the API-specific request and response handling lives, and a `presets/` directory holding the ready-made configurations for the services listed in the README. A `build/` directory suggests the packaging scripts, `docs/` the documentation, `icons/` the images including the logo the README embeds, and `src/` the application source. There is also a `tests/` directory, so the project has automated tests, and a `requirements.txt` at the top level alongside a `README.md` and a `.gitignore`.

That split between `apis/` and `presets/` is the extension point. The APIs directory holds the generic machinery for talking to a service, and the presets hold the configuration that binds that machinery to a particular endpoint, its parameters and its pagination scheme. This is why the README can claim that you can use Facepager with any API or website, once you are familiar with the basic concepts.

The `tests/` directory also tells you something about how the project expects to be worked on. A scraper whose tests depend on live endpoints would be flaky, so a meaningful test suite here most likely covers the parts that can be tested offline, which is the request construction, the parsing and the database writing.

The build directory and the fact that the project ships platform installers point to a compiled or frozen distribution rather than a plain source checkout that you run with a Python interpreter and a virtual environment. That is what makes the Windows exe and the macOS package possible, and it is also why the Linux route is explicitly a source build.

Where Facepager is the wrong tool

The clearest wrong-tool case is a production collection that needs commercial platform API access. If your project depends on Facebook, Twitter or YouTube data at scale, Facepager will not carry it, because the shared access it used to ship is gone and you are left managing your own application registration, review process and rate limits. The tool is still there, but the operational problem moves onto you and stays there.

The second case is headless or scheduled operation. Facepager is a desktop application with an installer, a help button, a wiki and video tutorials. Running it on a server, on a schedule, in a container or as part of a reproducible pipeline is not what the packaging supports, and the Linux instructions point to a source run rather than a service.

The third case is a very large collection. The data lands in SQLite, which is a fine choice for a research dataset and a poor one for a dataset that will outgrow a single file with concurrent writers. The README's own note that database files may be incompatible between versions is a smaller version of the same problem: the format is tied to the application version, so a long-lived collection database is a maintenance commitment.

The fourth is legal and ethical scope. The tool is for public data, and the presets it ships do not include access that bypasses a platform's terms. If your research question requires data that is not public, the right answer is an access agreement, not a scraping tool.

The alternative worth naming is a script you write yourself against the API client of the service you care about, using `requests` or the service's own SDK, storing results in whatever database your pipeline already uses. The difference in approach is that Facepager gives you a configured pipeline with rate limiting, pagination and multi-threading already handled, plus a database you can explore without writing code, while a script gives you full control over the data model and no dependency on a desktop application whose presets track platform policy changes.

Maintenance, versions and the MIT licence

The licence is stated in the README as MIT, with copyright 2019 Jakob Jünger and Till Keyling, and the full permission and warranty text is reproduced there. The project listing does not record a licence identifier, so the README text is the authoritative statement for anyone who needs it. MIT is permissive, with no obligation to publish derived work, and this is a description of the terms as written rather than legal advice.

The release history is uneven, and the unevenness is informative. Version 4.7.0 is dated 2025-10-14, version 4.6.3 is dated 2025-01-22, and version 4.5.3 is dated 2022-08-17. The last push to the default branch was on 2026-05-17, so work continues on master, but the tagged releases are irregular and there is a gap of more than three years between the 4.5.3 and 4.6.3 tags.

That pattern fits the project's own explanation. When a preset depends on a third-party service that changes its API, the work required to keep it working is not optional, and it arrives on someone else's schedule. The README's note about the pre-registered access service being unsustainable is the same story told from the access side.

For an adopter, the practical consequence is to treat the version you install as the version of the data format you own, since the README warns that database files may be incompatible between versions, and to expect that a preset for a service that has changed may need your own fix. Help is available through the built-in help button, the wiki, the video tutorials, and a user group, and the project also points to a page for staying informed about updates.

Editorial conclusion

Adopt Facepager when your work is repeated collection from open knowledge services such as OpenAlex, Crossref, arxiv.org or Wikidata, and when you want the results in a local SQLite database you can query and export yourself, because the presets and the built-in handling of rate limits and pagination are the value, not the scraping engine. Do not adopt it expecting turnkey access to Facebook, Twitter or YouTube, since the project states that the shared pre-registered access has ended and you must supply your own client ID and client secret. Before you install, check the releases page for the version you want, since database files may be incompatible between versions, and read the MIT licence text in the README, since the licence is not recorded in the project metadata.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. strohne/Facepager on GitHub