Can you suggest me a good web scraper?
I need to collect data from a bunch of pages. Data like sales count, dates, titles etc.
Can you suggest me a web scraper? If it has an API, even better.
Thanks.
I need to collect data from a bunch of pages. Data like sales count, dates, titles etc.
Can you suggest me a web scraper? If it has an API, even better.
Thanks.
@kinder has a sweet looking one called automatio.co. one of the best I've seen. he's in closed beta still (I'm on the list) I think but likely worth looking.
Oh, indeed: it looks really promising. Would be interesting to know something about the pricing.
That is something upon release. However, we will have an Early adopter Program, which will contain a yearly package of $900/year.
Thanks for recommending Automatio @asyncscrum.
@AndrasSzabo Automatio is indeed a really good no-code web scraper, but it's more than that.
Yes, it has API or gives you the ability to export data in Google Sheet in real-time.
@kinder Would love to try it as I am trying to find one that can solve my use case but have not been able to. I just signed up for your early access but need access:). Thanks.
@kinder Why did you start to write your own tool? You also struggled to find a suitable one?
I was always in need of web automation. Another bot or scraper which will save me from boring repetitive work. Automatio started as our internal tool more than 3 years ago, and then we figured out it should be a SaaS, since it brings tremendous value.
I am surprised to not see webscraper.io in the comments yet, are all the others mentionned really much better than it ? I'm curious, and if yes, i'll give a try to them ^^
Did you use it before? Looks really great.
I am using it quite often, works great for creating e-commerce scraping scenarios fast, I've never tried the paid version, but definitely would if I had a real need for repetitive scraping!
I use texau.com to build prospecting lists on Linkedin and to scrape Yelp reviews. It integrates with Google Sheets. Maybe it's what you're looking for?
very interesting. Thx, i take a look on it.
If you are familiar with python, you can use https://scrapy.org/ self hosted. If not, then something like https://lmgtfy.app/?q=web+scraper+with+api
Hey, I would recommend you PhantomBuster! If you don't confirm your mail you can get the free trial again and again.
Looks pretty promising. Thanks. I wonder if it is possible to create my own scraper scripts or I can combine the available ones.
prowebscraper.com is a good no-code platform that also has an API,
I did an article showing how to use their service in case you'd be interested
https://lewiskori.com/blog/introduction-to-no-code-web-scraping-with-prowebscraper/
another great tool is scraperapi though not no-code, it helps beat some challenges like web captcha, IP address blocking and provides a headless browser.
hope this article will help :)
https://lewiskori.com/blog/web-scraping-managing-proxies-and-captcha-with-scrapy-and-the-scraper-api/
Thanks, I take a look on it!
https://tryspider.com/
Thanks. Do you know one which runs in the cloud?
i do not.
If that's a simple-one-time kind of a deal, I usually use this:
https://chrome.google.com/webstore/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn
Its a simple scraper that uses css selectors/xpath and then gives you ability to export data via csv. If its something big then I use python+selenium, it lets me bypass almost any antiscraping stuff. Or you can use paid boutique solutions as suggested by other commenters :)
Well, I think it will be more of an ongoing thing. For example I would like to know how sales goes day by day for certain WP themes and plugins.
Then I would probably suggest some custom solution that would run itself once a day and neatly puts all the data in viewable format, try pinging to fellows who specialize in that. If you want to develop it yourself then just start by poking around and googling everything about scraping and picking the right tools for the job.
scrape.do is also amazing product with boutique support approach. I can also recommend them too.
Thanks, I'll read the documentation.
Hi Andras
DataOx ( https://data-ox.com/ )mainly focuses on complex large-scale data scraping projects and web data delivery feeds for business needs.
Hey @AndrasSzabo, try Simplescraper (https://simplescraper.io/) - API , export to Google sheets and as much support as you need to get setup.
I can provide an IndieHacker discount if you like.
webscraper
Data Miner