Webscraping: how to get started?
The challenge:
I want to monitor my competitors' pricing pages. Since I am talking about 50+ different sites I thought it would make sense to build a webscraper for that purpose. Its sole purpose would be to scrape pricing data from my competitors' pricing pages 2-4 times a day and save that information to a Google Sheet.
My question:
What would you recommend how to achieve that goal? Does it make sense to build this from scratch and learn how to build a webscraper (e.g., using python)? Or are there any (free / low budget) low code/no code tools out there that you would recommend?
Thanks a lot for your ideas!
Here's a no-code tool from another indiehacker:
https://automatio.co/
Seems really easy to use. I myself use Python for my current project to scrape data because I'm familiar with the language, but if you have zero knowledge this seems like a useful tool.
Cheers
Thanks Hugo :)
Here's a contrary opinion: you don't need to monitor 50 of your competitors' pricing pages. Manually checking them every once in a while (e.g. 3-6 months) is generally good enough.
This made me think of Amazon’s mechanical turk. Perhaps that’s the way to go - outsource the manual work instead of automating it.
When building a business, your time is scarce. The question you should always ask is, "Is this the best use of my time?" which usually should involve how to best deliver value to your customers or how to find/market to customers.
It's easy to get obsessed with competitors, which is almost always not the best use of your time. I say this as a person who made this mistake. At one point I actually made a web scraper (using selenium-webdriver) ... not to gather pricing data, but other competitive data. It was something that felt necessary at the time, and in retrospect it was completely unnecessary and a huge waste of time.
Just one person's perspective. Good luck!
Interesting. So, in retrospect, what would have been better invest of time?
Building product, working on marketing & distribution, talking to customers, getting user feedback. In essence, working towards product/market fit.
If you spend an hour researching your competitor's pricing, that's fine.
If you spend 10+ hours building a web scraper, that's not a good use of time.
Another indie hacker here. I'm building https://datagrab.io, a no-code tool.
It has a Chrome extension that allows you to set up scrapers visually, then you can either choose to run them in your browser (free and unlimited for now, but fairly slow), or upload them to the cloud service and run them there at scale.
Give it a shot! I'm open to talking to you to find out more about the sites you're trying to scrape. ;)
Hi Robert. This looks promising, and I will give it a shot tonight! Will share my feedback here in this thread. Thank you 🤓🤓🤓
Awesome! Thanks very much! :)
If you already know python, it can be done in less than a week. Most of the sites will have the same/similar structure.
https://github.com/baptisteArno/tinking is quite easy to use no code tool. I'm preparing a blog post about how to deploy those scripts to Phantombuster, to have scheduling.
If you want to learn I would advice you to go for puppeteer/JavaScript framework. It is usually more efficient than Python scripts.
If you want to outsource the development tetriz.io can do it for you.
I also did quite a few scraper throughout my career. The task is not complicated at all, but repetitive. I’d advise you to build it on your own, using puppeteer or playwright.
Hi @davidgyori, puppeteer has been recommended to me over and over again - so I think I will give a try :-) Or would you rather recommend using playwright?
Can you think of a well-done tutorial that would help to get started with either puppeteer or playwright?
Thanks!
I've build many scrapers, some for very complicated and dynamic form values, etc.. I can tell you that in the end nothing works better than writing your own custom script. And the best tool for it, hands down, is Puppeteer. It's basically like telling a website opened in Chrome what to do with JavaScript, which is already the language of the website. It's also super easy to connect with Google Spreadsheets via the Node API to dump the data. Don't use Python.
Again, puppeteer. It's been recommended to me quite a few times now. @bober can you recommend any good tutorial / site / youtube video that would help me getting started?
I only used the official Puppeteer docs to learn it. They offer a few examples on how to get started. And the API is fairly simple because all the actual scraping is done with regular JavaScript just like you would on your own website. All you need is a few functions to open the url and wait for things to load. The best thing is you can write everything asynchronously using async/await so you don't get stuck in JS callback hell. Good JavaScript and NodeJS knowledge is necessary.
Searching "competitor pricing tools" gives a lot of results. I'd say much better to use an existing tool and continue to focus on your core biz value to your customers.
Unfortunately they don’t cover the ones that I need. But I generally agree that if in doubt, focusing on your core value prop is always a good idea. However I cannot outsource / rely on external services in every single case as costs add up…
Got it, in that case there's some .py code here I open sourced that may be helpful: https://github.com/Goldiserv/screen-capture-from-url-py
Namely the looping of sub-links on a site + headless chrome.
It doesn't add any delay between each page browse so likely to get blocked if used on a site with request limits.
Awesome. I would love to learn and I am looking for a mentor. Of course I am willing to pay / give back in return. Please let me know.
Although happy to help, could you email me (see my profile) your expectations in a mentor?
In a similar spirit to customer consulting work, I'll only accept if I think I can add value.
There are many web scraping no-code tools, like SimpleScraper, OctoParse, etc. None is perfect and will never be since their job is to be ready for the whole internet. Which is hundreds of millions of websites, built using hundreds or even thousands of different tech stacks, languages, frameworks.
Depending on your needs, how much time you wanna invest, how complex those websites are. Some might be really simple and straightforward, and some really tricky.
Like @pras2018 said, web scraping is fragile and you need to take care of a lot of stuff, especially if that is not one time scrape, but you wanna monitor it every X hours/day.
Anyway, I am the founder and visionary of Automatio.co, which is no-code web automation & data extraction tool, where you could build not just web scrapers but complex bots in minutes, instead of days or weeks.
We are rolling out monthly packages which start at $99/month, so if you are interested to get in, feel free to reach me at stefan@vanila.io
Thanks @kinder - I watched the youtube video on your site and must say I would immediately give it a try if there wasn't the price tag. $99/month is simply out of scope for "just" automating monitoring of my competitors' pricing pages. Don't get me wrong, from what I saw in your video I think the price is just fair but I guess I am not your target audience. If I was a corporate then this would be just fine, but for my side project that's not feasible. It's like 5x of my monthly hosting / web builder costs, and that's just too much for a minor side feature of my project.
If you want to convert people like me there would have to be a more flexible pricing model (value-/traffic-based instead of a fixed price tag). To give you an idea, I would be open to pay around $2.5 for a use case like that. And I would be open to pay more as the volume/value I get increases. Maybe that's not worth the effort from your POV. But I can actually not think of any other no/low code tool that targets people like me so maybe this is really an interesting niche for you? The long tail of indie hackers? Happy to discuss / serve as a pilot customer :-)
Hey I know man, and I feel you. $99/m is the solid price tag, but it depends on your needs. If you have a problem to solve to save tons of time and make you a good amount of money, then it's worth it. We have other packages, which start at ~$30/m but with fewer features/tools.
We gonna experiment with the pricing, since we are just rolling it out, we are going from the top to the bottom. It's easier to decrease than to increase to existing customers.
Btw, our early adopter program was $2,000/year, so this is a good start with $99/m ^^.
Interesting app you are building!
I have not built a scraper in a while but might need one sometime soon. I was wondering what has your experience been with dynamic/javascript sites? The few times I tried tools that allowed you to select visually, they had a problem with dynamic sites. But this was years ago. I am wondering how easy/difficult it is now a day.
Javascript websites are mostly no problem. They are types of websites which make us trouble, and those are with randomly generated selectors. But we gonna tackle that down soon.
It depends on what you are trying to scrape. But if you are just getting started, try and get someone or a service to do it for you. Scraping can be pretty fragile, depending on how you do it. Plus, you have to deal with the possibility that your server might get blocked, so you have to rotate proxies. Scraping sounds easy in theory, but there are many nitty-gritty details you have to work out and constantly keep an eye on. Better to leave it to the professionals if you can afford it.
But it might turn out expensive based on your requirements. I am curious, what kind of site are you trying to scrape? I cannot think of a site where prices change 2-4 times a day. And how many pages are you talking about within a site? If you need to monitor 2-4 times a day and its multiple pages. It can get pretty expensive fast.
@pras2018 thanks for your reply. Have you done scrapers before / can you recommend to me somebody for mentoring / to help me getting started?
Based on your comment elsewhere, your best route would be to write a program to scrape.
Every language has at least one great scraper. I was on Ruby, so I used Nokogiri. Depending on the language you pick, you can find the most popular option.
Another alternative, assuming you are only trying to get pricing data off 1 page per site and you have a total of 50 sites. And assuming you are ok reducing the frequency i.e., you do not need it 2-4 times a day, instead, you are ok with, say once in a week. I would suggest going to a place like https://old.reddit.com/r/slavelabour/ and finding someone to do it for you. It will work out cheaper than going to a site like Fiverr and much easier than spending time building a robust scraping script.
Very cool. Thanks for your advice!
Just created an infographic related to web scraping 101.
Hope it can be helpful:
https://miro.medium.com/max/1400/1*hURiKBiolo2cqWO4NGICmg.jpeg
the big problem in scraping is avoid blocking , and this is done by doing your request via proxy's . and this is voddo .
what i mean when you try to get good proxy there are many pricing models .
i trying to find how to build small proxy farm and if you search on the web your will find nothing .
Parsehub is an app for visually building a scraping script . I've played with it a little bit. It seems intuitive enough to use. But I haven't actually pulled the trigger on any scraping with it, so I can't attest to how good it is.