Where do people that make use of web scraping meet?
Hi!
For various side projects I had started in the past I always found web scraping to be a very important step in the beginning when I have either no data about the subject or just need to gather some emails/information. Naturally, that's not something just for the beginning of a project - huge companies have been making use of the technique.
Something I found weird though is that I couldn't easily find what's the best place to hang around and discuss topics related to web scraping.
I'm about to search a product that might be interesting to the field (a one-click-away dynamic pool of IPs that you can use in your scripts) but I'd like to perform some validation first.
Are you aware any forums or communities that focus on such area? Do you have any groups that I could refer to?
Would appreciate your input!
Thanks!
Nope, don't know any "places" but I'd start with talking to the companies who already operate in this niche. Like where are THEY hanging out, where are they finding their customers, etc.
Just step into the visible parts of the "iceberg" and digg from there....
Hey patrics,
that sounds like a pretty good suggestion.
Thanks!
How do you get dynamic pool of IP? Just curious since I can only think of buying loads of VPS to get IPs.
There's some hacking/malware forum in the dark net (.onion etc) which discuss about web scrapping/ selling captcha bypass tool/ selling pool of IP proxies.
I think there's very less public forum on web scraping stuff because it is kinda gray area? Many company doesn't like people scrapping their stuff especially for profit. (eg: Craiglist, Amazon, etc)
Hey! Thanks for the reply!
I have the pool dynamically created by just constantly creating and destroying servers which run the proxy agent on a periodic basis.
For the end user (developer) it's just a plain request to a single address (it hits our public load-balancer which then forwards the request to a server in the pool) so that the product acts as the middleware (you don't have to query our API to get the list of IPs) such that the destination (the site you're scraping) receives requests from many different regions and various IPs.
The point you raised was something that I'm a bit concerned - it's true that it's kinda gray but at the same time, it's an area that has been generating considerable amounts of money.
For now, I only found some google groups (email groups); your point makes a lot of sense.
Thanks!