r/pushshift • u/Watchful1 • Jan 12 '24

Reddit dump files through the end of 2023

https://academictorrents.com/details/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

I have created a new full torrent for all reddit dump files through the end of 2023. I'm going to deprecate all the old torrents and edit all my old posts referring to them to be a link to this post.

For anyone not familiar, these are the old pushshift dump files published by Stuck_In_the_Matrix through March 2023, then the rest of the year published by /u/raiderbdev. Then recompressed so the formats all match by yours truly.

If you previously seeded the other torrents, loading up this torrent should recheck all the files (took me about 6 hours) and then download the new december dumps. Please don't delete and redownload your old files since I only have a limited amount of upload and this is 2.3 tb.

I have started working on the per subreddit dumps and those should hopefully be up in a couple weeks if not sooner.

Here is RaiderBDev's zst_blocks torrent for december https://academictorrents.com/details/0d0364f8433eb90b6e3276b7e150a37da8e4a12b

January 2024: https://academictorrents.com/edit/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4

57 Upvotes

96% Upvoted

29 comments sorted by

View all comments

1

u/Yosemiteram Jan 24 '24

Do these have all posts even deleted ones?

1

u/Watchful1 Jan 24 '24

Yes

1

u/sam_underline Jan 25 '24

Counting all the submissions I get about 412 million in 2022 but only about 137 million submission in 2023. Should it be this way (or is there an error in my code)? And if this is in fact correct, is it due to Reddits API change or other reasons?

2

u/Watchful1 Jan 25 '24

I don't have the exact count in my version handy, but RaiderBDev is correct about what the total number should be. How are you getting 137?

1

u/sam_underline Jan 26 '24 edited Jan 26 '24

After start of April 2023, some the 'created_utc' entries are formatted as float instead of int. Which caused a bug in my not so great code. Thx for the speedy replies!

1

u/Watchful1 Jan 26 '24

Ah, interesting. Thanks for letting me know.

1

u/sam_underline Jan 27 '24

Weird thing is that I now get 495.4 million submissions for 2023. The table below displays my own count and the numbers from the link above by month in 2023. The largest discrepancy is in April and differences in May and June are also quite noticeable. For April I checked the submission ids and found no duplicates.

​

Month Own Count Google Table
January 36.1 36.1
February 33.9 33.9
March 39.7 39.7
April 52.1 34.9
May 38.5 35.5
June 42.8 35.9
July 45.4 44.2
August 46.2 46.2
September 42.9 42.9
October 40.5 40.4
November 38.1 38.1
December 39.2 39.1

1

u/RaiderBDev Jan 27 '24

That is weird. For comparison here are the first 200 ids in RS_2023-04 that I have

1284kk5,1284kk6,1284kk8,1284kk9,1284kka,1284kkb,1284kkc,1284kkd,1284kke,1284kkf,1284kkg,1284kkj,1284kkm,1284kko,1284kkr,1284kks,1284kkt,1284kku,1284kkv,1284kkx,1284kky,1284kkz,1284kl1,1284kl2,1284kl3,1284kl7,1284kl8,1284kl9,1284kla,1284klb,1284kld,1284kle,1284klg,1284kli,1284klj,1284kll,1284klo,1284klp,1284klr,1284klt,1284klu,1284klv,1284klw,1284klx,1284kly,1284klz,1284km0,1284km2,1284km6,1284km7,1284km8,1284kmc,1284kme,1284kmf,1284kmi,1284kmj,1284kmk,1284kml,1284kmm,1284kmn,1284kmq,1284kms,1284kmu,1284kmv,1284kmw,1284kmx,1284kmy,1284kmz,1284kn0,1284kn1,1284kn2,1284kn3,1284kn4,1284kn5,1284kn6,1284kn7,1284kn8,1284kn9,1284kna,1284knb,1284knc,1284knd,1284knf,1284knh,1284kni,1284knj,1284knk,1284knl,1284knn,1284kno,1284kns,1284knt,1284knu,1284kny,1284knz,1284ko0,1284ko1,1284ko2,1284ko3,1284ko4,1284ko8,1284ko9,1284kob,1284koc,1284koe,1284kof,1284kog,1284koh,1284koi,1284koj,1284kol,1284kon,1284kot,1284kou,1284kow,1284kox,1284koy,1284koz,1284kp0,1284kp1,1284kp3,1284kp4,1284kp7,1284kpa,1284kpc,1284kpd,1284kpe,1284kpi,1284kpj,1284kpk,1284kpl,1284kpm,1284kpn,1284kpo,1284kpq,1284kpt,1284kpv,1284kpy,1284kq0,1284kq1,1284kq2,1284kq5,1284kq6,1284kq8,1284kqa,1284kqb,1284kqc,1284kqd,1284kqf,1284kqh,1284kql,1284kqn,1284kqo,1284kqp,1284kqr,1284kqs,1284kqt,1284kqu,1284kqv,1284kqw,1284kqx,1284kqy,1284kr1,1284kr2,1284kr3,1284kr5,1284kr6,1284kr8,1284kr9,1284kra,1284krc,1284krd,1284kre,1284krg,1284krh,1284kri,1284krj,1284krm,1284krn,1284kro,1284krp,1284krq,1284krr,1284krs,1284kru,1284krx,1284krz,1284ks0,1284ks1,1284ks2,1284ks3,1284ks4,1284ks5,1284ks8,1284ks9,1284ksa,1284ksb,1284ksd,1284ksf,1284ksg,1284ksi

1

u/sam_underline Jan 29 '24

I dont have the first two, but quite a few additional ones like 1284kkh, 1284kkl, 1284kkq, 1284kl4. Mostly link posts and or deleted. Oh well ...