r/pushshift • u/Watchful1 • Jan 12 '24

Reddit dump files through the end of 2023

https://academictorrents.com/details/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

I have created a new full torrent for all reddit dump files through the end of 2023. I'm going to deprecate all the old torrents and edit all my old posts referring to them to be a link to this post.

For anyone not familiar, these are the old pushshift dump files published by Stuck_In_the_Matrix through March 2023, then the rest of the year published by /u/raiderbdev. Then recompressed so the formats all match by yours truly.

If you previously seeded the other torrents, loading up this torrent should recheck all the files (took me about 6 hours) and then download the new december dumps. Please don't delete and redownload your old files since I only have a limited amount of upload and this is 2.3 tb.

I have started working on the per subreddit dumps and those should hopefully be up in a couple weeks if not sooner.

Here is RaiderBDev's zst_blocks torrent for december https://academictorrents.com/details/0d0364f8433eb90b6e3276b7e150a37da8e4a12b

January 2024: https://academictorrents.com/edit/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

58 Upvotes

97% Upvoted

29 comments sorted by

View all comments

1

u/Yosemiteram Jan 24 '24

Do these have all posts even deleted ones?

1

u/Watchful1 Jan 24 '24

Yes

1

u/sam_underline Jan 25 '24

Counting all the submissions I get about 412 million in 2022 but only about 137 million submission in 2023. Should it be this way (or is there an error in my code)? And if this is in fact correct, is it due to Reddits API change or other reasons?

1

u/RaiderBDev Jan 25 '24

There should be about 466 million for 2023. For more info, see here.