Unlimited enterprise web scraping for a flat fee.
Hello fellow makers,
I have extensive experience with web scraping. I was Team Lead at one of Europe largest retailers in charge of price monitoring. We scraped billions of web pages annually.
Right now, considering to launch an unlimited monthly web scraping service for a flat fee.
If you did web scraping what data were you looking for?
Did you do it yourself? If so it was time intensive? What did you use?
At what price range would you be comfortable to pay for such a service?
Also, if anyone needs help or advice on web scraping I can happily help give back the community.
Any thoughts and feedback would be welcome.
I built a pretty resiliant web scraper myself and CloudFlare was alwas a pain in the ass, but not too difficult to get around, though it was always a manual task. How do you plan on getting around this via automated means?
As a product, I'd have paid for it if it were very simple and reasonable. I spent a lot of time learning how to build one using Chrome Headless then FF/Gecko Headless which could have been used elsewhere.
For anti-scraping measures we would do our best to mask Headless signatures. And what really helped was warming up user data profiles. Chrome Headless instances browsing websites creating cookies and activity for a couple of weeks. Then we would swap live the user profile data directory. Combined with residential proxies and proper timeouts would pass most anti - bot tests.
"Unlimited" is very difficult to explain, you should figure our 2-3 tiers.
Have a look at these type of pricing models: https://manypixels.co/pricing
Makes sense. I will get right to it
I was thinking 100$, 300$ and custom. An interesting use case would be as @CameronBrown mentioned one off data scraping projects. These are difficult to scale time wise. I think the ideal target customer (if it exist) would be people that need data scraped regularly. Similar to monitoring where you do the scraper once and then you just run it to get fresh data.
I send you a message trough the Drift chat on your site. Looking at the proposition and website I think you should experiment with a 100% focus on E-commerce right now.
With your experience you can offer tailor made projects to mid-size webshops, and after the setup process charge a retainer fee.
Your current offering is nice, be requires some advanced tech work on the customers' side.
I would like to pay it forward so if at any point you need some tech work or advice let me know.
thanks for the advice @LucasH I am going to start gathering some leads, and do some customer interviews.
I've done quite a few small-scale web scraping jobs using Google Cloud/App Engine. With a bit of work I was able to my scrappy.. scrapper to work, and deployed it and loaded a few thousand URLs into a Task Queue while my bot slowly did the work over the course of a few days (to not overwhelm the website).
The problem with web scraping is that it's hard to monetise with a flat fee unless you're operating at super high scale, because many people who need to can write just write a small program small scale stuff easily.
Have you thought about also selling the bespoke development of a scraper?
Thanks for taking the time to respond. Insightful.
I developed a product that extract e-commerce product data from web pages automatically. No rules, no selectors and works for any e-commerce site.
Demo: https://quantati.com/#demo-page
Might be a niche market and need to put more marketing effort into it.
worked amazing for an example URL I tried!
but currency is missing, I'm sure you can figure it out quickly.
thanks for testing @imeralpb. Can you provide me with an URl to add it to our testing pipeline?
also tell me more about spouse detective I am curious of how you achieve that. Thanks!
Can you tell me more about FeedSub?
@kaizen Sure. It's just a simple service that lets you subscribe to RSS feeds and have updates/digests delivered to your inbox. I'm planning on launching soon.