I'm Scraping all of Reddit
by Michal MazurekThis was the last backend I wanted to add. That's a lot of traffic! I'm monitoring logs closely, expecting the unexpected.
There was no open source Go library for the Reddit API, so I had to write my own. I'll open source it shortly, after I iron out the details.
Now it's time to get out there and find more users.
-
1
Anyone know how to get all the subreddit using API? Like Autocomplete search subreddits
-
1
This is great! I've already signed up to see if anybody mentions me :P
By the way, I've seen Indie Hackers is not on your monitoring list
-
1
Are you using something like proxycrawl to get around the reddit limits? Or how do you do it?
-
1
If you connect authenticated with oauth then the limits are much more generous - and sufficient for my needs.
-
-
1
This is a great idea... You could also probably get away with just selling your own API to access the data for these platforms too.
-
1
Cool idea! I added an API just now, check it out :)
-
-
1
I have always been curious on how much traffic hacker news gets. They seem very secretive.
-
1
About 4M visits per day. The data is available here: https://twitter.com/yazijys/status/1135976760162553856
-
1
Rough guess is 17M pageviews/month: https://www.similarweb.com/website/news.ycombinator.com
-
-
1
How do you store the data? Elasticsearch?
-
1
I don't store the data, luckily I have no need. I just match the keywords as new posts arrive.
-
