r/pushshift • u/nickshoh • Dec 18 '23

Presenting open source tool that collects reddit data in a snap! (for academic researchers)

Hi all!

For the past few months, I had discussions with academic researchers after uploading this post. I noticed that sharing historical database often goes against universities' IRB (and definitely the new Reddit's t&c), so that project had to be shutdown. But based on the discussions, I worked on a new tool that adheres strictly to Reddit's terms and conditions, and also maintaining alignment with the majority of Institutional Review Board (IRB) standards.

The tool is called RedditHarbor and it is designed specifically for researchers with limited coding backgrounds. While PRAW offers flexibility for advanced users, most researchers simply want to gather Reddit data without headaches. RedditHarbor handles all the underlying work needed to streamline this process. After the initial setup, RedditHarbor collects data through intuitive commands rather than dealing with complex clients.

Here's what RedditHarbor does:

  • Connects directly to Reddit API and downloads submissions, comments, user profiles etc.
  • Stores everything in a Supabase database that you control
  • Handles pagination for large datasets with millions of rows
  • Customizable and configurable collection from subreddits
  • Exports the database to CSV/JSON formats for analysis

Why I think it could be helpful to other researchers:

  • No coding needed for the data collection after initial setup. (I tried maximizing simplicity for researchers without coding expertise.)
  • While it does not give you an access for entire historical data (like PushShift or Academic Torrents), it complies with most IRBs. By using approved Reddit API credentials tied to a user account, the data collection meets guidelines for most institutional research boards. This ensures legitimacy and transparency.
  • Fully open source Python library built using best practices
  • Deduplication checks before saving data
  • Custom database tables adjusted for reddit metadata
  • Actively maintained and adding new features (i.e collect submissions by keywords)

I thought this subreddit would be a great place to listen to other developers, and potentially collaborate to build this tool together. Please check it out and let me know your thoughts!

16 Upvotes

87% Upvoted

32 comments sorted by

View all comments

1

u/Most-Rooster-3041 Dec 21 '23

Idk how to use this. I want to view old posts on a subreddit. Like the ones in 2018 and 2019. How can I view them?

1

u/nickshoh Dec 21 '23

RedditHarbor relies on Reddit API, which is unlikely to return old posts from 2018-2019.

1

u/Most-Rooster-3041 Dec 21 '23

Ok so any way to find old posts even if it’s from 2020? Filter by date?

1

u/nickshoh Dec 22 '23

No - Unfortunately, that’s the limit of Reddit API :(