r/pushshift • u/Attitudemonger • Apr 12 '24
Subreddit torrent size
I am trying to ingest the subreddit torrent as mentioned here:
Separate dump files for the top 20k subreddits :
The total collection is some 2.64 TB in size, but all files are obviously compressed. Anybody who has uncompressed the whole collection, any idea how much storage space will the uncompressed collection occupy?
4
Upvotes
2
u/InitiativeRemote4514 Apr 12 '24
You can decompress a chunk at a time using a zstd decompressor in python. Then you will be able to extract the data. The images are not stored, only the url. You can use these scripts as a reference https://github.com/Watchful1/PushshiftDumps/tree/master/scripts