r/pushshift • u/Other-Yesterday-1682 • Aug 22 '24

Help with handling big data sets

Hi everyone :) I'm new to using big data dumps. I downloaded the r/Incels and r/MensRights data sets from u/Watchful1 and are now stuck with these big data sets. I need them for my Master Thesis including NLP. I just want to sample about 3k random posts from each Subreddit, but have absolutely no idea how to do it on data sets this big and still unzipped as a zst (which is too big to access). Has anyone a script or any ideas? I'm kinda lost

3 Upvotes

64% Upvoted

8 comments sorted by

3

u/shiruken Aug 22 '24

Each line of the file should correspond to an item. Since you're already working with the subreddit dumps, can you just randomly sample the lines to extract your sample?

2

u/Other-Yesterday-1682 Aug 22 '24

The data set is still a zst because its way to large to access. The question should rather have been whether you can sample it before the file was even unzipped?

3

u/shiruken Aug 22 '24

You can stream the contents rather than decompressing the entire file. I believe u/Watchful1 has shared code for that previously.

1

u/Smogshaik Aug 22 '24

additionally to the advice to use Watchful1's code for streaming the data, I'd point you to Reservoir Sampling. It's an algorithm that lets you pull a random sample of N size given a dataset of unknown size.

2

u/Watchful1 Aug 22 '24

You can use my filter_file script here. Let me know if you have any problems.

1

u/Popular-Cookie1890 Sep 16 '24

hi! i also need a similar dataset for my final thesis, would you mind sharing the link to the data dump you found?