4 Hours into my scraper market place - what do you think?
Hello IH,
So on Monday morning I had a hair brained idea about running web scrapers as a managed service.. I was surprised at the interest and figured there was something there.
One comment from @seansquared (5 upvotes) resonated with me:
*"Better: a site that aggregates everyone else's scrapers and allows you to call any of the scrapers with a 2 or 3-line API call. People can submit their scrapers and others can vote up/down.
Yes, GitHub, but all scrapers all day. To add value, you could add articles and videos about creating scrapers, using them for fun and profit, etc."*
I'm not sure I interpreted it correctly but I read it as building a marketplace for scrapers, which I did.
I have been playing around and come up with Scraper Palace www.scraperpalace.com where people can trade web scrapers.
The MVP will be a simple classified style site where people can list their scraper on the site at a cost of $5 for 12 months. Buyers will simply contact the seller direct and do a deal. Later on it will be a full on market that will handle payments etc.
One killer feature (imho) will be that the scraper developers will submit tests to the market place and I will keep them running and update the status of the scraper in real time (working / broken). There can also be a graph showing this and developer response time to fix. This can help devs get a great reputation and sell more.
Any and all feedback very much appreciated.
Cheers
Conor
Original thread: https://www.indiehackers.com/forum/just-a-thought-website-screen-scrape-rest-api-as-a-service-8340ecc7b9
Seeing you were asking feedback on your pricing.
In the product developer view, I would rather recommend you to earn a small rate per sell instead of a 12-month plan. Eventually having sponsored scraper listings could be a good deal too.
As a full-stack dev, if you have all those scraper tests running, $4.99 (always better than $5) is REALLY cheap for the value you provide.
Hey, thanks for your input.
I agree, $5 is cheap overall.. but that is just a gimme to get people listing in a "classifieds" type of model.
I do plan on developing a full on marketplace where I handle payment and take a commission of each transaction. That function will probably be built before the testing pipeline feature. The testing might be a feature for "premium listings".
I have an old well specc'ed computer that used to be a home server in my parents. I can use that to run through all the tests. If the machine bricks no issues as it's not totally critical (can bring back up cloud compute quick).
Might give you a perfect fit for your tests features: Serverless.
I seem it as the best way to run that specific use case and the best: it costs pennies (and a 1M free requests per month).
But in an initial MVP sure, a home computer would work :)
Yea it could be but I don't think it would be economical at scale. Probably better off to get cheap VPS somewhere.
At scale the jobs would probably be running 24/7. But I see where you coming from..
I do agree that for a solution running 24/7 - might - be cheaper to have a VPS for that. It would depend on the pricing people pay to use that specific service (or if that is included in the bundle).
Don't forget however the headaches (and the costs too) you would avoid by going with a solution like Serverless regarding scalability and devops.
Agreed. It's a question of scale.
I'd probably start off on serverless I would think.
Don't forget to sign up to the mailing list i you want to stay updated.
http://eepurl.com/dPkPU1
How do you deal with the legalities of web scraping?
Well scraping is a very grey legal area.
I think the only issue I will have is if a website doesn't like the fact that a scraper is advertised to work with their site. They might try and send me a cease and desist or something to remove that listing.
I probably wouldn't remove it to be honest because I am just a market place.
The worst case scenario is that a judge rules in favour of a website owner and I as the marketplace curator must remove it. That could result in owners for sites of all the requesting I remove the listing.. then poof, no listings.
I'll cross that bridge if (and most probably when) I encounter it.s
My thoughts on this is that the web is an open place and sites shouldn't try and counter scraping. The should encourage interactivity etc. If there is something of value on their site they should open up an API and monetise it.
Selfish me shouldn't say that because then there would be no need for scrapers :)
Looks good. I would keep going with the idea.
Thanks man :)
Really cool project and awesome design, maybe you can think of a contact form rather than a copy-paste of the user email address :D
Thanks. I think for the MVP the email will be enough. Later on I will just have a buy it now button with PayPal / Stripe.
Yeah, that's true. Good luck with your project and keep us up to date :->
Thanks man.
would defiantly use this when I am looking for a scraper?
regarding your supply side:
Is your plan to only source scrapers from the public or also build and support them internally
You can probably find the most looked for scrappers using search engine data
Hope the supply side picks up fast - as a buyer im all in.
Thanks for building this!
Menan
Super feedback thanks, I am looking forward to matching buyers and sellers.
While I am launching this product I probably will not focus on building scrapers myself.
That said, I might do research and employ some people on elance to build scrapers in high demand and sell them myself on the platform.
No problem Conor - you can email me message and email me at anytime for feedback too.
Thanks.
I just whipped up a mailing list for interested people. Feel free to subscribe for update. Cheers.
http://eepurl.com/dPkPU1
Conor
Done!
Hmm, nothing yet. You might want to check your inbox for the verification email. (I have now removed that step..)
Thanks
Just did it again! Cheers
Interesting. Looking forward to following this.
I started scraping with Import.io.
Then moved to Scrapy for Python (on ScrapingHub).
And am currently using Apify.
Will this support Apify? Or maybe we'd just say that's JQuery?
How would you source supply/demand for this?
Good luck.
Supply/demand is a chicken and egg type problem.
I would hope to develop a lead pipeline to get developers interested in building and submitting scrapers.
Then I would just to regular marketing to rank and get customer traffic. All the usual suspect:
.. few others I won't share here.
Thanks.
What do you mean by this line?
Will this support Apify? Or maybe we'd just say that's JQuery?Well, I mean Apify code takes JQuery to identify selectors. Though uses its own crawling engine.
In a bit over my head on the technicals.
Ah ok. I can't comment on how they would tie together at the moment, sorry.
another place to keep an eye on is https://phantombuster.com/
They are doing something very similar, except they provide the platform to run the scrapes as well. I've never used them myself so can't say how good they are.
Interesting, very! Thanks!
They seem to be focussed on the solving the data extraction problem as are many companies. I think the benefit of a compiled library is you can integrate it into your solution, exactly the way you need.
Perhaps all the others are focussed on extraction because that is where the money is, we shall see how much of a niche I end up in :/
this is a great concept, very interested to see where this goes
You should fix your meta tags, I just linked it to a friend on discord and the meta tag still says "MartPlace - Complete Online Multipurpose Marketplace HTML Template"
Lol yea I know that.. saw it myself. This draft is verrry early. I will fix all soon.
THANKS FOR LINKING -- Woo!
Awesome. Kudos to you for executing on this. Looking forward to seeing where you take it!
Cheers dude.
Yea, it's actually relatively quick to pump something out these days. I did it in the time that I would be spending time in front of Netflix and reading IH and Hackernews lol.
I hope to be in a position to accept listings by the weekend.
What do you think of $5 per one year?
Damn, couple of days to an MVP! Wish someone told me 20 years ago to stick with software development instead of shifting to IT infrastructure.
For pricing, I don't know enough about the dev market. Would you pay $5 a year to have access to a curated and hopefully low-spam (the $5 acts as a gateway to ward off at least -some- spammers) scraper site? It would be smart to ask folks here and on Hackernews, too.
Yea it can be pretty quick these days. My stack is:
I'm still tricking with the pricing but I think some kind of cash will help prevent people uploading any old crap. Quality is going to be key.
Thinking more about this model.. I think the code libraries is a good niche. There are a lot of services out there for data extraction etc but code libraries are useful for when people want to incorporate interactions to a site in their own workflows..
This is great. Thanks so much for making this project. How many libraries do you have already?
No worries - thanks for the feedback!
I haven't received any listing requests from third party developers. I have some scrappy code in various places to scrape some sites.
I don't think I will concentrate on listing my code. I will focus on the site and let others sell.
Do you know about Apify.com?
Good luck otherwise for your project!
Cool - I didn't know about that particular site but I know of a few others.
I guess I lost the emphasis somewhere but I think a lot of existing stuff is for data extraction.
There is also a need to be able to interact with sites also. For example, auto order on Amazon, booking flights etc
As opposed to straight up extraction of data.
Perhaps I have messed up naming... perhaps they are referred to more as "bots", but not really I think.
This comment was deleted 6 years ago
How to we flag comments..?