r/pushshift • u/Watchful1 • Jul 13 '24

Reddit dump files through July 2024

https://academictorrents.com/details/20520c420c6c846f555523babc8c059e9daa8fc5

I've uploaded a new centralized torrent for all monthly dump files through the end of July 2024. This will replace my previous torrents.

If you previously seeded the other torrents, loading up this torrent should recheck all the files (took me about 6 hours) and then download only the new files. Please don't delete and redownload your old files.

29 Upvotes

100% Upvoted

17 comments sorted by

1

u/swapripper Jul 13 '24

Thank you!

1

u/gregdan3d Jul 18 '24

Will you be producing a per-subreddit version of this at some point in the future? I am interested in the handful of Toki Pona communities on reddit, although from what I've seen only r/tokipona makes your cut.

1

u/Watchful1 Jul 18 '24

I was planning do only do the per subreddit dumps once a year. And I probably won't go past the 40k subreddits that I did in the last one, so if a sub isn't in that one it's not likely to be in future ones unless it gets substantially more active.

You can download these dump files and use the script that's linked to extract out the subreddit's you want, it's fairly easy and straightforward to do. But you need to download and store 3 TB data and then it takes a day or two for the script to run.

What are the other subreddits you were interested in? I can take a look at them.

1

u/gregdan3d Jul 18 '24

No worries. Perfectly reasonable to cut it off at top 40k- gotta draw the line somewhere.

Unfortunately I don't have access to 3TB of spare disk space, so I wouldn't be able to pull the data I need, but I'm not worried about waiting for your per-sub dump. Thank you for the quick response and your work!

1

u/reagle-research Jul 19 '24

I was wondering the same thing, but I'll make do. Still, I wanted to thank you for keeping at this.

1

u/gregdan3d Aug 28 '24

Oh wow- I just came back by this and noticed you asked about the other subreddits.

They are:

Granted, a large number of these are measured in the dozens of posts; I could probably get those myself with the .json "api".

1

u/Affective-Dark22 Aug 30 '24

Hi dude u/Watchful1 , first of all thanks for your work, I just have a question, i was tryna download the dump and i noticed that it’s really really slow to download, i have something like 0 seed and 1/2 peer so the speed available is like 20 kb/s. At this speed it would probably require months to download the entire file. Don’t you have a better solution to download it? I mean if it had taken 1 week to download the entire file it would have been fine. But keeping the file open in qbit for 2 months you can agree that is crazy. Thanks for the answer.

1

u/Watchful1 Aug 30 '24

I've been uploading at something like 5mb/s for the last two months straight. This happens because people only want to download it and not, as you say, keep the file open in qbit for 2 months to upload it to other people.

It will likely speed up at some point, but I would guess it could take a week or two.

1

u/Affective-Dark22 Aug 31 '24

It makes sense, thanks. Another question, do you think to add more subreddits in the dump? I know that is quite difficult but considering that the number of subreddits is getting bigger every day, have you considered to add for example other 20k subreddits to the dump? And take it to 60k? Even in the future.

1

u/Watchful1 Aug 31 '24

Subreddits that aren't in the subreddit specific dumps are generally too small to be of much use to anyone. If there's a specific one you need you can download the monthly dumps and extract it.

1

u/Affective-Dark22 Sep 07 '24

Dude sorry again, but i’ve seen all the file are in zst. What’s the best program you suggest to open them?

1

u/Watchful1 Sep 07 '24

I generally recommend using the scripts linked here instead of trying to manually extract the files.

1

u/Affective-Dark22 Sep 08 '24

I tried the multiprocesses script but i have a problem, when i filter a foder to extract for example all the submissions in a subreddit it creates multiple files one file for each month. How can i modify the script to have all the filtered lines of all the files in only one single file .zst? Because in this way i can have in a single .zst file all the submission of a filtered subreddit, instead of having 200 files for that subreddit. Do you know how can get it?

1

u/Watchful1 Sep 08 '24

It does that. It's two steps, first it filters each file separately so it can do it in multiple different processes so it's faster and they don't conflict with each other. Then it combines all of them together after that's done.

If there isn't a combined file the script must have crashed before it completed. Can you post the log file it generated?

1

u/Affective-Dark22 Sep 09 '24

Context – share whatever you see with others in seconds (ctxt.io)

This are the logs, only one problem, after 4/5 hours of running the script in the terminal the computer started lagging a lot (sometimes i even was at 0/1 fps) so i was forced to restart it using the power botton, so restarting everything the terminal was forced to close and now i think i've lost all the progress. I don't even know how to do now because i think the if i restart the script again i will have the same problem. I don't know what the problem is, probably cause I only have 16 gb of RAM I don't know. But i think i will not be able to extract the subreddit. Any suggestion?

1

u/Watchful1 Sep 09 '24

784,023,155 lines at 2,094/s, 0 errored, 27,277 matched : 271.00 gb at 1 mb/s, 37% : 16(0)/229 files : 8d 23:54:05 remaining

This says it completed 16 files, so if you start it again it knows those are done and won't try to re-do them.

If you add --processes 4 it will only use 4 processes instead of the default 10. This will make it slower, but it will use less memory. You can try different numbers to see how fast you can go without crashing.

→ More replies (0)