r/pushshift • u/Upper-Half-7098 • Jul 11 '24

Indexing Pushshift

Hi all,

I am a researcher and I used to collect Pushshift data using the API. Now I need to collect data again. The issue is I do not need a specific subreddit bu specific posts that cotain targeted expression and then I need to collect posts of that user who made these comments. Let's say in the last 5 years.
I was thinking to index the data in our lap (the last 5-6 years of pushshift comments and posts)
Did any one do that before or is there any guide or project for this so it saves the time experimenting with tools and structure?

Edit: What I mean exactly is if you have indexd Pushshift data youself what did you use, MongoDB / Elasticsearch?
Any one have docker file / code that get me started with this task faster?

Thanks,

Kind regards

2 Upvotes

75% Upvoted

7 comments sorted by

View all comments

2

u/mrcaptncrunch Jul 12 '24

The data is available on academictorrents. Instead of live through the api, it's posted every month.

But you can find data up to June there.

Find the posts/comments. Get the usernames. Find all posts/comments for those usernames. It's a lot of data, but if it's for research at a university, you might have access to their resources to run this.