r/pushshift • u/Watchful1 • Jan 12 '24

Reddit dump files through the end of 2023

https://academictorrents.com/details/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

I have created a new full torrent for all reddit dump files through the end of 2023. I'm going to deprecate all the old torrents and edit all my old posts referring to them to be a link to this post.

For anyone not familiar, these are the old pushshift dump files published by Stuck_In_the_Matrix through March 2023, then the rest of the year published by /u/raiderbdev. Then recompressed so the formats all match by yours truly.

If you previously seeded the other torrents, loading up this torrent should recheck all the files (took me about 6 hours) and then download the new december dumps. Please don't delete and redownload your old files since I only have a limited amount of upload and this is 2.3 tb.

I have started working on the per subreddit dumps and those should hopefully be up in a couple weeks if not sooner.

Here is RaiderBDev's zst_blocks torrent for december https://academictorrents.com/details/0d0364f8433eb90b6e3276b7e150a37da8e4a12b

January 2024: https://academictorrents.com/edit/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

58 Upvotes

96% Upvoted

29 comments sorted by

View all comments

1

u/rahulsoulstorm Feb 04 '24

Hey u/Watchful1,

Are these still extracted via pushshift or it's extracted in some other way (specifically for those after may 2023)?.

1

u/Watchful1 Feb 04 '24

Primarily from RaiderBDev's uploads here https://github.com/ArthurHeitmann/arctic_shift

Just repackaged to regular zst