There are may be only a few people who want to scrape with headless Chrome, but there are many people who want answers to questions that can best be answered by scraping with headless Chrome.
As a developer, here's something that would be interesting to me: build a serverless marketplace on top of browserless. Developers post scripts that answer particular questions (like what is the price of a particular item on Amazon vs. Walmart). The services those scripts provide are then sold at some rate. Revenue is split between browserless and the developer.
2
Huh, that's incredibly fascinating. I haven't given much thought to a market-place can see the value since most folks are just rinsing/repeating failures of the past.
It'd be cool to subscribe to a serverless workflow via a webook and updates can happen seamlessly when scraping begins to fail (as it does). That way we're isolating consumers of the function from the implementation details.
Man this is sounding better the more I think about it.
2
You could take this even further - a marketplace where developers write the "scripts" and maintain them, business analysts or just non technical users pay for the scripts that are pretty much gauranteed to work since there's a dev behind it. Dev gets a cut you get a cut for the platform, win win :)
2
If you're looking for a partner in building something like this out, I've got some spare cycles and think it could be pretty fun.
5
This is much needed product. As a developer, I understand it is not easy to run browserless scraping on services like Heroku.
There are few ways you can grow.
provide free trials option
For indie developer/freelancer from Asia $30 is high price. so may not adopt it. You can provide small price package.
Provide more details how to integrate with different programming languages like Java, python etc.
2
There was also an article on IH about geography based pricing.. If you have better search skills than me, it might be something to read as well :)
2
Thanks so much for the feedback Keyul. I'm definitely pursuing a pay-as-you-go model which should land in the coming weeks. I'm hoping that this will engage some of the smaller use-cases vs large enterprise clients.
I'll definitely keep you posted!
2
If it works, is secure, and generates revenue, then that's all that really matters.
Having seen how things can go wrong for businesses that don't at least think about security, I think it's great that you called it out here.
1
Yeah, security is something that keeps me up at night. Even when you think you've got it you're probably overlooking something trivial. More needs to be said about securing web-apps
2
I remember seeing this posted on HN and probably other places, glad to hear it's going well!
1
Same here: saw Joel on a Github issue. It may have been PhantomJS :)
2
Hey Joel,
This was an interesting write up. Answering questions on StackOverflow and Quora is a great way to produce real value and introduce your product at the same time.
My question for you is: what issues did you run into while building this business? Payments (Stripe), Hosting (Docker/Heroku etc.), VCS (Github/BitBucket) all already have great products filling those needs. While building your product, was there any particular issue that took significantly longer than others?
2
Awesome awesome question: I think that there definitely needs to be more done around account management and authorization. Those problems are extremely hard to get right, and tend to be insecure when dealing with authorization. Also knowing where your audience is can be tricky, and the answer is generally to build it yourself. That takes years and isn't something that's easy for everyone to do, so having a way to find users and know how to approach them would be an interesting problem to tackle. Unfortunately I'm still a beginner in that regard so it's hard for me to know what the solution there looks like.
On the technical side managing your app's topology is pretty challenging. I have a good mental model of how browserless works, but to try and describe it to someone else would have me looking like a conspiracy theorist! Kubernetes and docker are steps in the right direction, but it'd be nice to have some sort of static artifact that describes your architectures dependencies would be interesting to tackle.
1
Regarding user auth, did you look at providers like Auth0 or Firebase?
When Meteor.js was introduced, one of my favorite features was the ability to drop in a simple Spacebars tag ({{> loginButtons}}) and instantly have a preconfigured user system. I always thought it might make a good standalone product.
If you did check out solutions like Firebase or Auth0 (or any others), what did you find lacking?
1
I very briefly took a look at Auth0 and Passport (since the infra is written in node). For reasons that probably aren't accurate I felt that I would move quicker if I just did it myself. If Auth0 had a "quickstart" guide that wasn't more than 5 steps I likely would have gone with them. Stripe really knocks it out of the park with this: their quick guides are amazing.
Firebase is something that's still in the back on my mind begging to be looked at. I'm not sure exactly what problems it solves and need to investigate further.
1
For reasons that probably aren't accurate I felt that I would move quicker if I just did it myself. If Auth0 had a "quickstart" guide that wasn't more than 5 steps I likely would have gone with them.
I've avoided their service for this reason as well.
Firebase is an entire mBaaS and as such has a wide array of products. Though, like Auth0, setting up accounts is not as easy as it was with Meteor.
1
you should consider combining your service with proxies
1
I used Phantombuster before after I had endless hassles getting Chromeless to work on AWS. There is definitely a market for this sort of product. Good luck with this. I have a project coming up which needs to do scraping, so I'll give it a try some time in the future
1
Hey. I’m sorry to be the one who asks this, but I don’t understand why people would need this. I appreciate they do but I dont understand why. Can you give me an example use case? I read your site but there was none in there.
2
Great question -- It's definitely a challenge with this project since most people won't need it until they run into the problem. A great use-case is taking screenshots of your website. Most service just let you specify a URL, which works for the majority of cases, but what happens if you need to click-through to get to the site you need to capture? It's cases like this where you'll need to run headless Chrome yourself and tell it what to do and when/how to screenshot.
Another case is screenshot resolution -- again something that most services have limits around. If you need high-DPI screenshots then you're left with running Chrome on your own infrastructure and managing it.
This is where browserless steps in to help. It's quite tedious to manage Chrome, get it setup properly for linux systems (which most servers use), and ensure it operates like one would expect. The value browserless adds is that you can do any of the cases above without losing an incredible amount of time getting it setup.
Hope that clears it up. I'm working on making this more clear in my site and in documentation, so great question.
1
Congratulations Joel, this looks like an awesome and much needed product. Have used Navalia and was as happy as one can be with a headless browser library so I'll definitely check this out next time I need to run large numbers of headless scripts.
Do you think it's possible that developers could write headless scripts so extensive that they could essentially create unofficial APIs for some web apps? As an example of what I mean, I tried to write a headless script to order online groceries a little while ago, with the ultimate goal of me being able to hand a script some product names and have it order all those products using the supermarket user interface. I ran into a lot of issues and gave up since I didn't really need it but I couldn't see why it wasn't technically possible. What do you think?
2
It's definitely possible -- but challenging. Site continuously change and update, so things like selectors and what not will break frequently. I think there's a lot that can be done in this regard to capture issues, like taking snapshots of where failures happen. Definitely a hard problem
1
I think diffbot.com deals with this problem using machine learning. None the less, indeed a hard problem. But user generated content (scripts) can easily be incentivized for the developers to continuously maintain them as they break.
1
It's not quite the same problem I was referring to, I want to create APIs from sites that can be used to replicate user actions rather than just extract data (although diffbot and its ilk are very impressive). This would in a lot of cases require the bot to log in as well so I'm not sure how that would work regarding security? User generated scripts is a good idea for sure.
1
Ah I see, so an API to control any website... One example I can think of is an API to control say an ecomm site like Amazon and place orders like a real user?
1
Exactly like that yes, seems like a difficult but doable thing.
1
Delighted to see you here Joel. Just wanted to say thanks for your help when I was trying browserless, my app has since moved away from needing screenshots (for now), but will probably be back in the future. For anyone reading this, browserless absolutely took the pain out of running headless chrome on heroku, I was battling it for weeks before Joel's plug and play app made it easy. This is starting to sound like an over the top plug, but just wanted to say congrats Joel, I haven't been back in the slack chat of September sprint since I got a new phone, is it still going strong?
1
Chip! Thanks so much for the kind words! Our Slack channel is still going well, there's a good few core folks there that are always discussing stuff and offering advice. It's been really refreshing to be a part of.
1
That's great. I must pop back in! All the best going forward anyways. 👍
There are may be only a few people who want to scrape with headless Chrome, but there are many people who want answers to questions that can best be answered by scraping with headless Chrome.
As a developer, here's something that would be interesting to me: build a serverless marketplace on top of browserless. Developers post scripts that answer particular questions (like what is the price of a particular item on Amazon vs. Walmart). The services those scripts provide are then sold at some rate. Revenue is split between browserless and the developer.
Huh, that's incredibly fascinating. I haven't given much thought to a market-place can see the value since most folks are just rinsing/repeating failures of the past.
It'd be cool to subscribe to a serverless workflow via a webook and updates can happen seamlessly when scraping begins to fail (as it does). That way we're isolating consumers of the function from the implementation details.
Man this is sounding better the more I think about it.
You could take this even further - a marketplace where developers write the "scripts" and maintain them, business analysts or just non technical users pay for the scripts that are pretty much gauranteed to work since there's a dev behind it. Dev gets a cut you get a cut for the platform, win win :)
If you're looking for a partner in building something like this out, I've got some spare cycles and think it could be pretty fun.
This is much needed product. As a developer, I understand it is not easy to run browserless scraping on services like Heroku.
There are few ways you can grow.
provide free trials option
For indie developer/freelancer from Asia $30 is high price. so may not adopt it. You can provide small price package.
Provide more details how to integrate with different programming languages like Java, python etc.
There was also an article on IH about geography based pricing.. If you have better search skills than me, it might be something to read as well :)
Thanks so much for the feedback Keyul. I'm definitely pursuing a pay-as-you-go model which should land in the coming weeks. I'm hoping that this will engage some of the smaller use-cases vs large enterprise clients.
I'll definitely keep you posted!
Having seen how things can go wrong for businesses that don't at least think about security, I think it's great that you called it out here.
Yeah, security is something that keeps me up at night. Even when you think you've got it you're probably overlooking something trivial. More needs to be said about securing web-apps
I remember seeing this posted on HN and probably other places, glad to hear it's going well!
Same here: saw Joel on a Github issue. It may have been PhantomJS :)
Hey Joel,
This was an interesting write up. Answering questions on StackOverflow and Quora is a great way to produce real value and introduce your product at the same time.
My question for you is: what issues did you run into while building this business? Payments (Stripe), Hosting (Docker/Heroku etc.), VCS (Github/BitBucket) all already have great products filling those needs. While building your product, was there any particular issue that took significantly longer than others?
Awesome awesome question: I think that there definitely needs to be more done around account management and authorization. Those problems are extremely hard to get right, and tend to be insecure when dealing with authorization. Also knowing where your audience is can be tricky, and the answer is generally to build it yourself. That takes years and isn't something that's easy for everyone to do, so having a way to find users and know how to approach them would be an interesting problem to tackle. Unfortunately I'm still a beginner in that regard so it's hard for me to know what the solution there looks like.
On the technical side managing your app's topology is pretty challenging. I have a good mental model of how browserless works, but to try and describe it to someone else would have me looking like a conspiracy theorist! Kubernetes and docker are steps in the right direction, but it'd be nice to have some sort of static artifact that describes your architectures dependencies would be interesting to tackle.
Regarding user auth, did you look at providers like Auth0 or Firebase?
When Meteor.js was introduced, one of my favorite features was the ability to drop in a simple Spacebars tag (
{{> loginButtons}}) and instantly have a preconfigured user system. I always thought it might make a good standalone product.If you did check out solutions like Firebase or Auth0 (or any others), what did you find lacking?
I very briefly took a look at Auth0 and Passport (since the infra is written in node). For reasons that probably aren't accurate I felt that I would move quicker if I just did it myself. If Auth0 had a "quickstart" guide that wasn't more than 5 steps I likely would have gone with them. Stripe really knocks it out of the park with this: their quick guides are amazing.
Firebase is something that's still in the back on my mind begging to be looked at. I'm not sure exactly what problems it solves and need to investigate further.
I've avoided their service for this reason as well.
Firebase is an entire mBaaS and as such has a wide array of products. Though, like Auth0, setting up accounts is not as easy as it was with Meteor.
you should consider combining your service with proxies
I used Phantombuster before after I had endless hassles getting Chromeless to work on AWS. There is definitely a market for this sort of product. Good luck with this. I have a project coming up which needs to do scraping, so I'll give it a try some time in the future
Hey. I’m sorry to be the one who asks this, but I don’t understand why people would need this. I appreciate they do but I dont understand why. Can you give me an example use case? I read your site but there was none in there.
Great question -- It's definitely a challenge with this project since most people won't need it until they run into the problem. A great use-case is taking screenshots of your website. Most service just let you specify a URL, which works for the majority of cases, but what happens if you need to click-through to get to the site you need to capture? It's cases like this where you'll need to run headless Chrome yourself and tell it what to do and when/how to screenshot.
Another case is screenshot resolution -- again something that most services have limits around. If you need high-DPI screenshots then you're left with running Chrome on your own infrastructure and managing it.
This is where browserless steps in to help. It's quite tedious to manage Chrome, get it setup properly for linux systems (which most servers use), and ensure it operates like one would expect. The value browserless adds is that you can do any of the cases above without losing an incredible amount of time getting it setup.
Hope that clears it up. I'm working on making this more clear in my site and in documentation, so great question.
Congratulations Joel, this looks like an awesome and much needed product. Have used Navalia and was as happy as one can be with a headless browser library so I'll definitely check this out next time I need to run large numbers of headless scripts.
Do you think it's possible that developers could write headless scripts so extensive that they could essentially create unofficial APIs for some web apps? As an example of what I mean, I tried to write a headless script to order online groceries a little while ago, with the ultimate goal of me being able to hand a script some product names and have it order all those products using the supermarket user interface. I ran into a lot of issues and gave up since I didn't really need it but I couldn't see why it wasn't technically possible. What do you think?
It's definitely possible -- but challenging. Site continuously change and update, so things like selectors and what not will break frequently. I think there's a lot that can be done in this regard to capture issues, like taking snapshots of where failures happen. Definitely a hard problem
I think diffbot.com deals with this problem using machine learning. None the less, indeed a hard problem. But user generated content (scripts) can easily be incentivized for the developers to continuously maintain them as they break.
It's not quite the same problem I was referring to, I want to create APIs from sites that can be used to replicate user actions rather than just extract data (although diffbot and its ilk are very impressive). This would in a lot of cases require the bot to log in as well so I'm not sure how that would work regarding security? User generated scripts is a good idea for sure.
Ah I see, so an API to control any website... One example I can think of is an API to control say an ecomm site like Amazon and place orders like a real user?
Exactly like that yes, seems like a difficult but doable thing.
Delighted to see you here Joel. Just wanted to say thanks for your help when I was trying browserless, my app has since moved away from needing screenshots (for now), but will probably be back in the future. For anyone reading this, browserless absolutely took the pain out of running headless chrome on heroku, I was battling it for weeks before Joel's plug and play app made it easy. This is starting to sound like an over the top plug, but just wanted to say congrats Joel, I haven't been back in the slack chat of September sprint since I got a new phone, is it still going strong?
Chip! Thanks so much for the kind words! Our Slack channel is still going well, there's a good few core folks there that are always discussing stuff and offering advice. It's been really refreshing to be a part of.
That's great. I must pop back in! All the best going forward anyways. 👍