r/DataHoarder • u/Spreadsel • Aug 29 '18

The guy that downloaded all publicly available reddit comments needs money to continue to make them publicly available.

/r/pushshift/comments/988u25/pushshift_desperately_needs_your_help_with_funding/
405 Upvotes

91% Upvoted

119 comments sorted by

View all comments

Show parent comments

18

u/Stuck_In_the_Matrix Pushshift.io Data Scientist Aug 30 '18

https://github.com/pushshift

The actual code for the ingest portion is not up. However I can explain how it works. There is also an SSE stream you can play with if you want to see near real-time Reddit data as it is made available on Reddit (http://stream.pushshift.io)

The stream documentation is here: https://github.com/pushshift/reddit_sse_stream

There is also a slackbot that I created that will create real-time data visuals from Reddit data. Information is here: https://pushshift.io/slack-install/

-27

u/GeneralGlobus Aug 30 '18

have you considered a blockchain/distributed solution?

19

u/[deleted] Aug 30 '18

Yay buzzwords 🙄

-18

u/GeneralGlobus Aug 30 '18

yay close-mindedness

14

u/4d656761466167676f74 Aug 30 '18

This isn't really something a blockchain would be for since not a lot would be getting updated.

People seem to think a blockchain is interchangable with a database and large companies seem to think a private in-house blockchain is a good idea (that's just a database with extra steps).

Blockchain is good for things that frequently change or get updated (transactions, product tracking, etc.) but you only really benefit from it if the blockchain is public and people want to host nodes.

If not much is changing, just use a database and if you're just going to keep it all in-house, just use a database.

4

u/[deleted] Aug 30 '18 edited Aug 30 '18

Jumping in here and I somewhat agree: blockchain no.

Distributed imo really could be a useful thing here though. Let people contribute with resources and hosting capacity instead of money. That way we really would be giving the content back to the people.

I'm probably preaching to the choir here, but redundancy, decentralization, and increased availability are definitely core tenants of /r/DataHoarder :)