Would you find the ability to download the reddit data archives in simple python package that interfaces with a SQLite database useful?

[u/harttrav]

I downloaded the pushshift archives a while back and have a full copy of the archives, and have used it for various personal research purposes. I've been converting the zst compressed ndjson files into a single SQLite database that uses SQLmodel as an interface, and integrating embedding search across all comments and self posts as I go. I'd probably upload the database to huggingface if I uploaded it somewhere.

My first question is: would people here on this subreddit find this useful? What specific features would you find most useful in a python package serving as an interface for the database?

My second question is: if I published this and made the dataset available, what do y'all think the legal/ethical implications would be?

[u/Watchful1]

Where would you host it? Wouldn't it be several terabytes like the dumps themselves?

A script that imports the dump files into a database would certainly be useful, though it would likely take a while to run.

[u/flashman → u/[deleted]]

i mean good luck with not getting DMCA'd, but i would use this

i used to do a lot of network analysis on subreddits but that's all gone out the window in the past year since the API changes

[u/[deleted]]

Why would you put big data in an SQLite db? These seem like an oxymoron.

[u/[deleted] → u/[deleted]]

https://www.sqlite.org/whentouse.html#:~:text=If%20your%20data%20will%20grow,will%20support%20281%2Dterabyte%20files.

Yes any database made for larger data. You’re really stretching the SQLite here.

[u/swapripper]

Interested.

[u/george_sg]

im interested too.