socialreaper and the case for a uniform interface over six social APIs

Six platforms, one calling convention

socialreaper is a Python 3.6+ library that scrapes Facebook, Twitter, Reddit, YouTube, Pinterest and Tumblr, and the design is a class per platform with a consistent shape.

Installing it is one command:

pip3 install socialreaper

The platform classes are imported from the top level of the package, and each takes its credentials in the constructor. Facebook and YouTube take a single api_key. Reddit takes two values. Twitter takes the full OAuth set of app_key, app_secret, oauth_token and oauth_token_secret.

That consistency is the library's value. The first example pulls the comments from the most recent posts on a page:

from socialreaper import Facebook

fbk = Facebook("api_key")

comments = fbk.page_posts_comments("mcdonalds", post_count=1000, 
    comment_count=100000)

for comment in comments:
    print(comment['message'])

The method names are the interesting part, because they encode the shape of each platform's data model. Facebook has page_posts and page_posts_comments, YouTube has channel_video_comments, Twitter has user, Reddit has subreddit_thread_comments. You learn the vocabulary of one platform once and it reads like the vocabulary of the others.

The primary language is Python, the licence is MIT, the default branch is master, and the documentation is hosted on readthedocs with a Gitter room linked.

Each platform's example shows what its API makes awkward

The four worked examples in the documentation are worth reading side by side, because each one shows a different platform quirk rather than a different feature.

The Twitter example saves the most recent tweets to a CSV file, using a tools helper:

from socialreaper import Twitter
from socialreaper.tools import to_csv

twt = Twitter(app_key="xxx", app_secret="xxx", oauth_token="xxx", 
    oauth_token_secret="xxx")
    
tweets = twt.user("realDonaldTrump", count=500, exclude_replies=True, 
    include_retweets=False)
    
to_csv(list(tweets), filename='trump.csv')

The Reddit example is the one that shows the library's real limitation. Getting the top comments from the top threads of a subreddit returns a nested structure, and the example has to flatten it, then sort by score, then slice. The flatten call comes from the tools module and converts a nested dictionary into a flat one:

from socialreaper import Reddit
from socialreaper.tools import flatten

rdt = Reddit("xxx", "xxx")
 
comments = rdt.subreddit_thread_comments("all", thread_count=50, 
    comment_count=500, thread_order="top", comment_order="top", 
    search_time_period="all")
    
# Convert nested dictionary into flat dictionary
comments = [flatten(comment) for comment in comments]

So the platforms are not uniform underneath, and the library smooths the differences in method signatures rather than in the data model. The Reddit example then reaches into a nested data key to sort and to print, which is a reminder that the returned dictionaries mirror whatever the platform's JSON looked like.

The YouTube example adds a lookup step before the fetch, guessing a channel id from a handle, and can filter comments by text, which is a filtering feature the other platforms do not expose in the same shape.

The tools module is the part that outlives any one platform

A shared helpers module is where the durable value sits, and the documentation shows two entry points.

One is a CSV export class, used to write a list of dictionaries to a file:

from socialreaper import Facebook
from socialreaper.tools import CSV

fbk = Facebook("api_key")
posts = list(fbk.page_posts("mcdonalds"))
CSV(posts, file_name='mcdonalds.csv')

The other is the flatten helper, which converts a nested dictionary into a flat one, and the to_csv function used in the Twitter example.

The reason this matters is that export is the part of a scraping job that is the same regardless of source. Getting the data is platform specific, and it changes whenever the platform changes. Turning a list of dictionaries into a CSV, or flattening a nested structure so it can be sorted, is not platform specific at all, and a user who has two of these jobs will write that code twice if the library does not provide it.

The setup metadata lists the runtime dependencies, and they are minimal: requests at 2.11.1 or newer, requests-oauthlib at 0.7.0 or newer, and oauthlib at 2.0.1 or newer. The requirements file pins the same three, with requests at 2.20.0. So the library is requests plus OAuth, with no parsing framework, no async runtime and no data model layer of its own.

Version 0.3.0 examples in a 1.0.3 package, and what that tells you

There is a small inconsistency in the documentation that a reader should notice, and it says something about the maintenance history.

The examples section is headed with a note that it applies to version 0.3.0 only, and the pinned install command is given alongside it:

pip3 install socialreaper==0.3.0

Meanwhile the package metadata in setup.py declares the version as 1.0.3. So the documented examples were written against a release two major versions behind what the package now claims to be, and the version line was bumped without the examples being refreshed.

The repository structure is small, which is consistent with a library that has not been restructured in a long time. There is a socialreaper/ package directory, a tests/ directory, a docs/ directory, setup.py, requirements.txt, a .travis.yml and a LICENSE.txt. The CI configuration is Travis, which is a reasonable signal about when the project was last actively set up.

There is also a separate GUI project linked, described as an option for people who are not programmers, at a different repository. That split is worth understanding before installing anything, because the library and the GUI are separate codebases and only the library is what this repository contains.

The YouTube example shows the filter that only one platform has

The YouTube example is the one that departs from the pattern the other three follow, and it is worth reading because it shows a capability the other platforms do not expose in the same shape.

The first step is a lookup, because YouTube identifies a channel by id rather than by name:

from socialreaper import Youtube

ytb = Youtube("api_key")

channel_id = ytb.api.guess_channel_id("mkbhd")[0]['id']

comments = ytb.channel_video_comments(channel_id, video_count=500, 
    comment_count=100000, comment_text=["prize", "giveaway"], 
    comment_format="plainText")
    
for comment in comments:
    print(comment)

Two things stand out. The comment_text parameter takes a list of strings, so you can ask only for comments containing particular words, which on a platform where a channel's comment volume runs into the thousands is the difference between a usable sample and a firehose. And the fact that channel lookup is a separate api call with its own result shape shows how much of the awkwardness of these libraries is just the platform's own model of identity.

The comment_format option controls how the text comes back, with plainText as one choice, which is a small thing that only matters when you are about to do something with the text.

The Pinterest and Tumblr classes are not shown in the documentation at all, which is itself a useful signal about how completely the examples cover the library.

Where socialreaper is the wrong choice

The first and largest case is a production data pipeline. The library wraps the platform APIs as they were, and those APIs change without notice and without a deprecation cycle a client library can respond to. A research script that breaks is an inconvenience; a pipeline that silently stops returning data is worse than one that fails loudly.

The second is completeness. These APIs paginate, and they rate limit, and the library's own examples pass large counts such as a hundred thousand comments or five hundred videos. Whether those counts are honoured in one call, or paged through, or silently truncated, determines whether you have the data you think you have. The documentation does not say, and that question has to be answered before you rely on the results.

The third is anything touching accounts you do not own. Scraping public pages and reading a user timeline you can see are different activities from enumerating a private account, and a library that offers both will be used for both.

The fourth is a project that needs a maintained dependency. The licence is MIT and the code is readable, so forking and fixing a broken endpoint is a realistic fallback, and that is arguably the right way to use a library at this level of fragility. But it does mean you own it, and the 0.3.0 examples against a 1.0.3 version number is a reasonable signal of how much attention the code has had recently.

Editorial conclusion

socialreaper suits a researcher who needs comments or posts from several platforms with one calling style, because the per-platform classes share a shape and the tools module handles the export that every one of these jobs needs. It does not suit a production pipeline, since the library depends on platform APIs that change without notice and the last push was on 2026-09-29, and it does not suit anyone who needs a guarantee of completeness from a platform that paginates or rate limits. Before you build on it, check that the specific endpoint you need still exists, and note that the package version in setup.py is 1.0.3 while the documentation still carries examples labelled for version 0.3.0.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. ScriptSmith/socialreaper on GitHub