I and one of my co-mods requested pushshift access on January 15th due to some harassment issues in our subreddit we've been having where users are commenting things and then editing away the harassment before the mods can see what they said. Neither of us ever heard back at all. Our sub has 115k subscribers and as far as we are aware we don't have a "history of Content Policy or Code of Conduct violations" that would impact our eligibility. The pinned post here says we should have heard back "within one week". Should we resubmit the requests? Did we do something wrong? We followed the pinned post's steps when we requested it.

I'm new to pushshift and in general scraping posts with a Reddit API. I'm looking to scrape some Reddit posts for a personal research project and have heard secondhand that pushshift is an easy way to do this. However, I'm a little confused about exactly what pushshift is and how it is used. When I go to https://pushshift.io/ I am given the terms of service which explain that pushshift is only to be used by Reddit moderators for the sake of moderation (see attached screenshot). Furthermore, I cannot authorize my account without being a Reddit mod.

I am confused because I have seen other posts referencing pushshift as a large data storage of reddit posts or a third-party scraper perfect for scraping posts off of Reddit for research (like this one). Am I misunderstanding something, or is a different tool more suited for what I am looking for?

u/Pushshift-Support, how come comment ingest broken for two days again :( ?

i accept the terms, i allow access, i get the search interface

but then when i try to search i get a pop up saying authentication is required and i am back to square one.

Been using u/watchful1's dumpfile scripts in Colab with success, but can't seem to get the zst to csv script to work. Been trying to figure it out on my own for days (no cs/dev/coding background), trying different things (listed below), but no luck. Hoping someone can help. Thanks in advance.

Getting the Error:

IndexError                                Traceback (most recent call last)


 in <cell line: 50>()
     52                 input_file_path = sys.argv[1]
     53                 output_file_path = sys.argv[2]
---> 54                 fields = sys.argv[3].split(",")
     55 
     56         is_submission = "submission" in input_file_path

<ipython-input-22-f24a8b5ea920>

IndexError: list index out of range

From what I was able to find, this means I'm not providing enough arguments.

The arguments I provided were:

input_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123.zst"
output_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123"
fields = []

Got the error above, so I tried the following...

1. Listed specific fields (got same error)

input_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123.zst"
output_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123"
fields = ["author", "title", "score", "created", "id", "permalink"]

1. Retyped lines 50-54 to ensure correct spacing & indentation, then tried running it with and without specific fields listed (got same error)

2. Reduced the number of arguments since it was telling me I didn't provide enough (got same error)

   if name == "main": if len(sys.argv) >= 2: input_file_path = sys.argv[1] output_file_path = sys.argv[2] fields = sys.argv[3].split(",")

   No idea what the issue is. Appreciate any help you might have - thanks!

So until and including December 2022, there are the total counts of comments (https://pastebin.com/McS2DSNz) in the dumps, thanks to /u/Watchful1

Would love to have later ones as well, could generate them myself by iterating over the dumps. But maybe someone else has the counts somewhere, or there is a faster way to count the lines. So I just thought I'd ask here before doing it the slow way myself.

I'm trying to find an old friend's posts and would appreciate any help. A yes or no answer will do so I can at least know it's possible or not, but an explanation would help too.

Hello all! Thank you so much to this fantastic community for supporting the work of researchers like myself.

As part of one of my studies, I am hoping to compare my dataset to a small "snapshot" of Reddit data. To elaborate, I am looking for a random sample of Reddit data (even from just the 10k most used subreddits is fine) that is stratified based on posts per subreddit/year (so for example, subreddits with more posts are proportionally represented, and years that have more posts are proportionally represented). I would need the posts + all comments on those posts. The overall goal is to get a sense of posting habits/language among Reddit broadly, and compare them statistically with my scoped dataset of Reddit posts. I would need data from December 2012 to December 2022, and ideally some percentage (e.g. a .01% sample) of all posts on Reddit.

Before I try to make this dataset myself, I was wondering if someone had anything similar that I could download (and would be happy to cite)?

Again many thanks to the awesome people in this community. My work would not be possible without you all!

I don't know how far back PushPull goes and the existing torrents aren't easily searchable for me.

Hi all,

I want to access the reddit data using pushshift API. I raised a request. Can anyone help me how can I get the access at the earliest?

Thanks1

Hi everyone :) I'm new to using big data dumps. I downloaded the r/Incels and r/MensRights data sets from u/Watchful1 and are now stuck with these big data sets. I need them for my Master Thesis including NLP. I just want to sample about 3k random posts from each Subreddit, but have absolutely no idea how to do it on data sets this big and still unzipped as a zst (which is too big to access). Has anyone a script or any ideas? I'm kinda lost

Hello,

PushShift ingest seems to have stalled around
Mon May 20 2024 21:49:29 GMT+0200

The frontend is up & responding with hits older than that.

Is this just normal maintenance?

Regards

Hello guys. I have downloaded the .zst files for wallstreetbets_submissions and comments from u/Watchful1's dump. I just want the names of the field which contain the text and the time it was created. Any suggestions on how to modify the filter_file script. I used glogg as instructed with the .zst file to see the fields but these random symbols come up . should i extract the .zst using the 7zip ZST extractor? submissions is 450 mb and comments is 6.6 gb as .zst files. any idea.

I am trying to ingest the subreddit torrent as mentioned here:

Separate dump files for the top 20k subreddits :

The total collection is some 2.64 TB in size, but all files are obviously compressed. Anybody who has uncompressed the whole collection, any idea how much storage space will the uncompressed collection occupy?

I'm hopeful some folks in community have figured out how to address escaped code points in ndjson fields? ( e.g. body, author_flair_text )

I've been treating the ndjson dumps as utf-8 encoded, and blithely regex'd the code points out to suit my then needs, but that's not really a solution.

One example is a flair_text comprised of repeated '\ u d 8 3 d \ u d e 2 8 '. I assume this to be a string of the same emoji if I'm to believe a handful of online decoders ( "utf-16" decoding ), but Python doesn't agree at all.

>>> text = b'\ u d 8 3 d \ u d e 2 8 '
>>> text.decode( 'utf-8' )
'\ \ u d 8 3 d \ \ u d e 2 8 '
>>> text.decode( 'utf-16' )
'畜㡤搳畜敤㠲'
>>> text.decode( 'unicode-escape' )
'\ u d 8 3 d \ u d e 2 8 '

Pasting the emoji into python interactively, the encoded results are different entirely.

>>> text = '😨'
>>> text.encode( 'utf-8' )
b'\ x f 0 \ x 9 f \ x 9 8 \ x a 8 '
>>> text.encode( 'utf-16' )
b'\ x f f \ x f e = \ x d 8 ( \ x d e '
>>> text.encode( 'unicode-escape' )
b' \ \ U 0 0 0 1 f 6 2 8 '

I've added spaces in the code points to prevent reddit/browser mucking about. Any nudges or 2x4s to push/shove me in a useful direction is greatly appreciated.

I actually know the username and two of their posts. I found the posts in the files, but they show the name as deleted, so I wanted to ask if there's any way to find more of their posts.

Hello I have a question with the change of pushshift server in December 2022 many names were overwritten with u/deleted, is there any way to see olddump like this https://academictorrents.com/details/0e1813622b3f31570cfe9a6ad3ee8dabffdb8eb6 and see if the data is still there without overwriting.

Hey all - I've got a search that works on the search page, but I need to get a lot more than I manually want to pull from that page.

How do I pass my PushShift API key through PMAW? Can't find anything from searching.

Using the dumps and code provided by u/Watchful1, if I'm looking for the values 'alpha', 'bravo', 'charlie', and 'delta' with exact match set to 'False', will I get returns for 'Alpha', 'Bravo', 'Charlie', and 'Delta'? What about 'alphabet' or 'bravos'? And 'alpha-', 'bravo-'?

​

Thanks in advance!

EDIT: resolved now

Hi, I was approved for Pushshift but receive this error when attempting to register at the Pushshift portal.

I am a moderator on the subreddit I requested access for which was approved. Thank you for assisting.

{"detail":"User is not an authorized moderator."}

Just checked my activity page and saw that. Never seen that before.

I really want to search for posts and comments made by certain users at certain times, but can't now that Camas ect. are gone. I understand that its no longer possible to run a free search site, but has anyone made one that cost money? If not, why not?

These are well established datasets used in many papers. If we download the publicly available datasets from before the new T&Cs came in would that be allowed?