# reddit-universal-scraper collects what Reddit actually serves you

> This suite scrapes Reddit through two routes, an approved OAuth application or a browser extension that captures what you load while signed in, and stores the results in SQLite behind a Streamlit dashboard, a FastAPI service and a Parquet export. It is explicit that you get a window of posts, not an archive.

**ksanjeev284/reddit-universal-scraper** — Universal Reddit Scraper - Works on any Subreddit or User

- Repository: https://github.com/ksanjeev284/reddit-universal-scraper
- Stars: 609 · Forks: 94
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/ksanjeev284-reddit-universal-scraper

## Two collection modes, and they are not the same thing

The suite advertises itself as a full-featured Reddit scraper with an analytics dashboard, a REST API, scheduled scraping and plugins, and the first thing it tells you is that there are two collection modes: free browser-page capture, or approved OAuth API access.

That distinction runs through the whole project and it is the most important thing to understand before choosing it.

The API mode is the direct Python scraper. It talks to oauth.reddit.com with a descriptive User-Agent, using credentials for an application Reddit has approved. The readme says Reddit blocks unauthenticated API traffic, so this path is only available to an app that has been granted access, and the environment file carries a comment saying to use only credentials for an app or use case Reddit has approved.

The browser mode is the Browser Collector Companion, a Chrome and Edge extension in the extension/ folder. It works while you browse Reddit signed in, captures the posts, comments and media links that load, supports filters and session limits, and exports JSON or CSV without paid services or API keys. The readme is clear about its boundaries: it observes loaded content as you browse, it does not retrieve an archive, and it does not automatically expand unloaded comments. The browser's Reddit cookies stay in the browser.

So the choice is between an approved app with a quota, and a human-driven capture with no approval needed but no automation.

## The API path depends on approval, and a proxy does not change that

The section on Reddit access and reliability is the most useful part of the readme, because it is where the project is honest about its constraints.

The direct scraper uses oauth.reddit.com with a descriptive User-Agent, cached OAuth tokens, token renewal, bounded retries and shared request pacing. Both the sync and the async scrapers respect Retry-After and the Reddit rate-limit headers. API requests are serialized within each client while media downloads stay concurrent, which is the shape you would expect from a client that has to stay inside a quota. Independent processes share your app quota, so the readme says to schedule them accordingly.

The environment file makes the pacing explicit, with REDDIT_REQUESTS_PER_MINUTE set to a conservative 60 and a comment noting the cap is 100 and that processes share the same app quota.

The failure modes are named too. Missing credentials, denied access, HTML responses and exhausted retries produce explicit failures, command line failures exit with status 1, and a job preserves the posts it already collected while reporting a failed status when a later request fails, so a partial run is not silently presented as a complete one.

The proxy section is the one to read carefully. An optional stable network proxy can be configured with PROXY_URL or --proxy, with ScrapingAnt country and session targeting through PROXY_COUNTRY and PROXY_SESSION_ID, and --proxy none disables it. The readme also lists what the project no longer does: mirrors, browser impersonation, warm-up requests and automatic proxy rotation. And then the sentence that decides whether any of it works for you, which is that a proxy does not grant API approval.

## What you collect is a window, not an archive

The readme states the collection limits in a short paragraph, and it is worth reading as a specification of what your data will not contain.

Listings expose a limited window of posts, not a complete historical archive. Comment collection includes the initially loaded tree up to the configured depth, and more placeholders are not expanded. Deleted or restricted content cannot be recovered.

That last point has a knock-on effect for the extension mode as well, since it captures the same loaded content from the browser, and the same limitation applies to anything the extension sees.

In practice this means the tool is a sampling instrument. If your analysis depends on the full history of a community, a keyword trending over years, or a complete count of comments, this suite cannot give you that, and no configuration change fixes it. The three scrape modes reflect the same point from different angles: full collects available posts, media and loaded comments, history is a faster pass with no media or comments, and monitor polls on a schedule.

The dry run mode is the honest way to find out what a given configuration would return before you store anything, and the readme frames it exactly that way, as a way to test scrape rules without saving data.

## One CLI, several modes, and a dry run that saves nothing

The command line is the primary interface, and the readme documents it with a subreddit name as the first argument.

```bash
# Collect available posts + media + loaded comments
python main.py delhi --mode full --limit 100

# Fast history-only (no media/comments)
python main.py delhi --mode history --limit 500

# Live monitor (checks every 5 min)
python main.py delhi --mode monitor
```

There is a user mode for scraping a specific account's posts, and flags to skip media or to skip comments when you do not need them. Dry run mode adds --dry-run to any of these, and the readme shows what it prints, a count of the posts and comments that would be scraped, with the header stating that no data will be saved.

Plugins are enabled with --plugins and listed with --list-plugins. Three are built in: sentiment_tagger adds sentiment scores to posts, deduplicator removes duplicate posts, and keyword_extractor extracts top keywords. Custom plugins go in the plugins/ folder, so the extension point is a directory convention rather than an entry point you have to register.

The rest of the command surface is for the surrounding system: --dashboard, --api, --export-parquet, --job-h for job history, --check-access to verify credentials, --import-browser to bring in a JSON export from the extension, --bridge-token to generate a local pairing token, and --schedule with an --every interval for the cron-style scheduler.

## SQLite at the centre, with a dashboard, an API and Parquet around it

The value of the suite is less the scraping than the storage and the surfaces around it.

Data lands in a SQLite database with automatic backup, and the job tracker keeps a full history with status, duration and errors, which is what makes a scheduler usable in practice. Export to Parquet is a first-class command for loading into DuckDB or a warehouse, and the dependency list includes duckdb, pyarrow and openpyxl for exactly that path.

The dashboard is a Streamlit application with seven tabs: overview with stats and charts, analytics for sentiment and keywords, search over the scraped data, comments, a scraper tab to start new runs, job history, and an integrations tab. It runs on port 8501.

The REST API is FastAPI with uvicorn on port 8000, with interactive docs at /docs and endpoints for posts, comments, subreddits, jobs, a raw SQL query route, and a Grafana time-series route. The API route for raw SQL is the kind of thing to think about before exposing the service beyond your own machine, and the compose file in the repository has an optional nginx service for HTTPS termination and basic authentication on top of it.

The whole stack ships as a Docker Compose file with three services built from the same image: a scraper service that only runs when explicitly asked through a profile, an api service with a health check against /health, and a dashboard service that depends on the api.

## Configuration is a file of credentials you should be careful with

The environment file is short and it is the whole configuration surface for the access side of the project.

```
REDDIT_USER_AGENT="python:your_app_id:2.0 (by /u/your_username)"
REDDIT_CLIENT_ID=""
REDDIT_CLIENT_SECRET=""
REDDIT_REFRESH_TOKEN=""
REDDIT_ACCESS_TOKEN=""
REDDIT_REQUESTS_PER_MINUTE="60"
```

The User-Agent format is prescribed, platform:app-id:version by /u/your_username, and the comment above it says to use only credentials for an app or use case Reddit has approved. You can supply a user-authorized refresh token with read scope, or an existing approved bearer token, which the comment notes is not automatically renewable.

The rest of the file is for the surrounding system: the optional proxy settings, BROWSER_BRIDGE_TOKEN for pairing the extension with the application, which the comment says should be 24 characters or more and can be left blank to generate a local token, and the Discord and Telegram notification webhook values.

The container side follows the same pattern. The Dockerfile starts from python:3.11-slim, installs ffmpeg and curl, copies requirements first for caching, copies each source directory individually, creates data/backups and data/parquet, exposes 8501 and 8000, and defines a health check against the API health endpoint. The Python requirement for local use is 3.10 or newer, and ffmpeg is optional and only needed for video with audio, with install commands given for Windows through chocolatey, macOS through brew and Ubuntu through apt.

## Licence, maintenance, and the cases this suite cannot serve

The licence is MIT, the repository is not archived, and the last push was on 2026-10-02. There are no GitHub releases, so the version you install is the version on the main branch, and the Docker workflow badge in the readme is the visible sign of how the images get built.

The first case where this is the wrong tool is historical. Everything the suite produces is bounded by what the API or your browser session returns, so a question about what a subreddit said three years ago cannot be answered with it, and the readme says so directly.

The second is approval. If you cannot get Reddit to approve an app for your use case, the API path is closed, and the extension path is a manual capture with no unattended scheduling. That constraint is not a setting you can work around, and the readme's sentence about proxies not granting approval exists precisely to close off that line of thinking.

The third is sensitive data. The suite stores posts and comments, tracks them in a job history, can send Discord and Telegram notifications, and exposes a raw SQL query endpoint. Collecting and keeping other people's posts and comments is a decision with obligations attached, whatever the licence of the tool says, and the readme's own instruction is to collect only data you are authorized to use.

There is also a documentation pointer worth following before you commit: docs/reddit-access.md covers credentials, restrictions, troubleshooting and the Developer Platform migration timeline, which is where the real constraints on your app will be spelled out.

## Conclusion

reddit-universal-scraper fits a researcher or analyst who needs a working set of Reddit conversations in a queryable store, and who is prepared to go through Reddit's approval process for the API path or to browse and capture manually for the extension path. It is the wrong tool if you need a complete historical archive, if you cannot satisfy Reddit's approval conditions, or if the data is sensitive enough that collecting it raises questions you have not answered. Check first which collection mode your use case actually needs, that your REDDIT_REQUESTS_PER_MINUTE setting matches the quota you were granted, and that you understand the collection limits before you design a pipeline on top of the output.

## FAQ

### Does the Reddit scraper need an approved API application?

Yes for the direct Python path, which uses oauth.reddit.com with a descriptive User-Agent and credentials for an app Reddit has approved. The readme states plainly that a proxy does not grant API approval, so the browser extension is the route that needs no approval.

### How much Reddit data can this scraper actually collect?

Listings expose a limited window of posts rather than a complete historical archive, comment collection covers the initially loaded tree up to the configured depth, more placeholders are not expanded, and deleted or restricted content cannot be recovered.

### How do I run the dashboard and the REST API?

Run python main.py --dashboard for the Streamlit dashboard at http://localhost:8501, and python main.py --api for the FastAPI service at http://localhost:8000 with interactive docs at /docs.

### Does the browser extension need an API key or a paid service?

No. The Browser Collector Companion runs in Chrome or Edge while you are signed in to Reddit, captures loaded posts, comments and media links, and exports JSON or CSV without paid services or API keys. It can also send collections to the application for storage and dashboard analytics.

## Sources

- [Issues](https://github.com/ksanjeev284/reddit-universal-scraper/issues)
- [ksanjeev284/reddit-universal-scraper on GitHub](https://github.com/ksanjeev284/reddit-universal-scraper)
- [License: MIT](https://github.com/ksanjeev284/reddit-universal-scraper/blob/main/LICENSE)
- [README](https://github.com/ksanjeev284/reddit-universal-scraper/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ksanjeev284-reddit-universal-scraper