Wibbly.co
Browser Automation API
I started working on Wibbly two months ago and I finally have an MVP version which onboards some cool features which will hopefully be of interest!
Since this is an MVP any feedback would be greatly appreaciated. I am at a point where I need some external perspective to drive the next steps for this project! Hope to get some free users in soon and start gathering feedback!
I offer a free plan which includes 3,000 requests a month, so there is plenty of space there to test and play with the API!
I also came up with a pretty detailed documentation page just in case anybody is curious: https://www.wibbly.co/docs
I am building a service which extracts metadata and content from any website via a simple and fast API.
The web is a wild place and most of the websites have ways to identify headless browser sessions. I needed to make sure that the API could see the actual state of a website once all JavaScript is loaded so I am designing the system to internally spawn real browser sessions to get the information from.
Maintaining a browser session is no trivial task, where installations, update, sudden crashes, upgrades, are a major source of frustration for web developers.
Wibbly tries to solve all those issues allowing developers to focus on what they are building, making the browser interaction completely transparent.
I have an overall idea of the features that API can offer and these are the main ones:
- take screenshots of a website
- take pdfs of a website
- extract normalized set of metadata from html, opengraph, jsonld.
An additional feature is a way to query a website, providing css expressions, to grab custom content, like returning the latest 10 stories from Hacker News. That's what I am working on right now!
I can see that the API response time highly depends on the response time of the website which gets inspected, so I don't expect the api to come back in milliseconds, but I am working hard keep the response time below the 2 seconds mark.
I pushed up a first version of the homepage here: www.wibbly.co
-
1
Your website is down.
-
1
Can you explain how this is done without exposing the vulnerable data on a website to a third party service?
-
1
I have worked on/with such a product (for scraping). Real-life issues we had to overcome: One of the big issues was cookies (need to save and load them as some info of content selection/display sometimes is stored there), other was elements (HTML blocks) that pop up and cover the content (a problem when you take a screenshot) or that you are required to click, so the main content would fully load.
Hope these real-life cases helps you.-
1
Thanks a lot for the advices! Wibbly can already manage popups and dialogs and hide/dismiss them.
The cookie feature is something I have on my roadmap.
-
-
1
Hi!
It seems like you offer some similar services to my onesimpleapi.com
I also provide a metadata service (https://onesimpleapi.com/docs/web-page-meta-tags-and-open-graph), and I know of at least a dozen more services that do, so there's definitely demand, I would simply figure out a use case and specialize.
Feel free to reach out to me on Twitter to chat!
-
2
Thanks a lot for the heads-up. I like the style of your landing page. well done there.
On top of extracting meta from any webpage, Wibbly can also extract any content from a webpage in a simple and declarative fashion.
-
-
1
Will try
-
1
thank you so much! I just released today so any feedback would be hyper appreciated!
-
-
1
cool but it does sound like something that is probably already out there.
-
1
sounds cool
Comment