xscrape collects public X data through cookie sessions, and the auth pool is the part that carries the risk

A library for public X content, with an explicit disclaimer

xscrape is described as an asynchronous Python client for collecting public data from X, covering posts, profiles, threads, replies and media. The feature list names post search by keywords, hashtags and operators such as from:, since: and until:, profile metadata with follower and activity counters, conversation chain reconstruction, and media link extraction.

Around that core sit three pieces of infrastructure: a session pool with automatic failover, a built-in rate limiter, and storage backends for JSON, CSV and SQLite. There is also a plugin system for custom handlers and exporters, a command line interface, Docker support, and structured logging with JSON output and request tracing.

The positioning is explicit about scope. The project states it is intended for educational purposes and work with public data only. That framing is not decoration, because the authentication model depends on it: the library talks to public GraphQL endpoints using session cookies, which means it operates with a logged in identity rather than with an anonymous reader.

The package is on version 0.4.2 in pyproject.toml, requires Python 3.10 or newer, and depends on aiohttp, pydantic, tenacity, orjson, python-dotenv, click and rich.

The pipeline: client, auth pool, fetcher, parser, rate limiter, storage

The architecture section describes the data flow as a chain, and the diagram in the documentation makes the dependency direction clear.

The Client is the public interface, exposing search, user, thread and replies. The AuthPool is the session pool, which picks a free account and handles 429 and 401 responses. The Fetcher makes the low-level HTTP requests, with retries and exponential backoff. The Parser normalises raw responses into typed models, Tweet and User. The RateLimiter applies a per-account token bucket plus a global cap. The Storage layer serialises results into the chosen format.

Fetcher, rate limiter and parser are the parts that determine whether the library behaves well under load. Retry with exponential backoff handles transient failures, and the typed models mean downstream code deals with a stable shape rather than with raw JSON.

The AuthPool is where the design has a cost. Because it reacts to 429 and 401 by moving to another account, the library treats account count as the lever for throughput. The rate limiter's per-account token bucket with a global cap keeps a single account from being hammered, but the global cap, not the pool, is what ultimately bounds request volume. A pool of ten accounts does not make a scrape ten times more polite to the platform.

Details beyond the summary live in docs/ARCHITECTURE.md.

Configuration is entirely environment variables

There is no configuration file for options. Every setting is read from environment variables, and the reference table in the documentation lists nine of them with their defaults.

The two that matter most are XSCRAPE_COOKIES, a cookie string containing auth_token and ct0, and XSCRAPE_POOL, the path to a JSON file holding the account pool. The rest are operating parameters: XSCRAPE_CONCURRENCY defaults to 4 parallel requests, XSCRAPE_TIMEOUT to 20 seconds, XSCRAPE_RETRIES to 3, XSCRAPE_PROXY for an http://user:pass@host:port proxy, XSCRAPE_USER_AGENT which falls back to a built-in agent, and XSCRAPE_LOG_LEVEL with XSCRAPE_LOG_FORMAT set to text or json.

The shipped template shows the same set with comments:

XSCRAPE_COOKIES="auth_token=YOUR_TOKEN; ct0=YOUR_CT0"
XSCRAPE_POOL="accounts.json"
XSCRAPE_CONCURRENCY=4
XSCRAPE_TIMEOUT=20
XSCRAPE_RETRIES=3
XSCRAPE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"

Note that the credentials live in a plain environment file that the Docker compose file reads directly, and that the account pool is a JSON file mounted into the container. Any deployment inherits the standard risk of shipping session cookies alongside application code.

Full reference is in docs/CONFIGURATION.md.

The CLI covers search, profiles, timelines and threads

The command line interface is where most of the practical value sits, because the same operations are reachable both from the CLI and from the Python client.

The quick start in the documentation shows the shape of the commands:

xscrape search "python asyncio" --limit 50 --out tweets.json
xscrape user elonmusk
xscrape timeline elonmusk --limit 200 --out timeline.csv
xscrape thread 1234567890123456789 --out thread.json
xscrape hashtag "#opensource" --limit 1000 --db hashtag.db
XSCRAPE_POOL=accounts.json xscrape search "data engineering" --limit 2000

Two output modes are visible here. A file path with --out writes JSON or CSV depending on the extension, while --db takes a SQLite file for a hashtag sweep of a thousand items. The last line sets the pool through the environment for a single run, which is the same mechanism the configuration table documents.

The examples/ directory holds ready-to-run scripts that cover the same ground with more control: search_tweets.py for pagination and filters, user_timeline.py, export_to_csv.py, export_to_sqlite.py, thread_dump.py for a full thread, hashtag_monitor.py as a long running watcher, and multi_account_pool.py.

One detail worth copying carefully is that the hashtag monitor is invoked with its own flags:

python examples/search_tweets.py --query "openai" --limit 200

The script names and the compose file both show that the monitor takes a hashtag, an interval and an output path, which is the shape you would want for a continuous feed.

Docker deployment mounts the account pool read only

The container setup is defined in two compose files, one for the normal stack and a development variant at docker-compose.dev.yml, with the Dockerfiles under docker/.

The build step in the documentation is a single command:

docker build -f docker/Dockerfile -t xscrape:latest .

The compose file for the main stack defines two services on a bridge network named xscrape_net. The xscrape service runs the command xscrape --help with a restart policy of unless-stopped, mounts ./data to /data and mounts accounts.json into the container read only at /app/accounts.json.

A second service, monitor, runs the hashtag monitor as a separate container with the same image and the same two mounts, and depends on the xscrape service. Its command runs examples/hashtag_monitor.py with a hashtag of #ai, an interval of 60 and an output path of /data/stream.jsonl.

The read-only mount of accounts.json is a good detail to copy. It means the container can read the pool but cannot rewrite it, so a compromised or misbehaving process inside the container cannot silently replace the credentials it uses.

The development stack is started separately with docker compose -f docker-compose.dev.yml up --build, and the local run path expects you to copy .env.example to .env first so that the env_file reference resolves.

Version 0.4.2, no releases, and unfinished plugin work

The package version is 0.4.2 and the repository has no GitHub releases, so there is no tagged artifact to depend on. The last push was on 2026-10-20 and the repository is not archived. In practice, a consumer tracks the main branch and reads CHANGELOG.md for what changed.

The roadmap in the documentation marks search and profiles, the account pool and rate limiter, export to JSON, CSV and SQLite, the CLI and Docker support as done. Still open are the plugin exporter system, media download support, webhook notifications, a web monitoring dashboard, Prometheus metrics and a GraphQL query cache.

That last item is interesting for anyone running at volume. Without a query cache, repeated work hits the network again, and the per-account token bucket plus the global cap remain the only brakes on request frequency.

The licence is MIT, which is permissive and carries no copyleft obligation, and the repository also carries SECURITY.md, CODE_OF_CONDUCT.md, CONTRIBUTING.md, a Makefile, setup.cfg, a pre-commit configuration, requirements.txt and requirements-dev.txt. The project metadata in pyproject.toml still lists placeholder URLs under yourname, and the clone command in the install instructions points at a different account, so treat the packaging metadata as unfinished rather than authoritative.

Editorial conclusion

Use xscrape if you are doing research on publicly visible X content, you already have session cookies you are willing to store, and your volume stays inside what a handful of accounts can carry. Do not use it as a production data source, since it depends on unofficial GraphQL endpoints and cookie sessions that X can invalidate at any time, and it ships at version 0.4.2 with no GitHub releases to pin against. Before adopting, check the clone URL in the install instructions against the repository you actually intend to use, confirm the project metadata URLs in pyproject.toml, and read the disclaimer in README.md against the use you have in mind.

Frequently asked questions

Official sources

1. gunkonchi/xscrape on GitHub
2. Issues
3. License: MIT
4. README