visual web scraper
Hey, so after thinking about building this for some time, I decided to start with a POC. Any ideas/comments/feature requests appreciated: https://www.youtube.com/watch?v=i5tTcsFb2Jg
Hey, so after thinking about building this for some time, I decided to start with a POC. Any ideas/comments/feature requests appreciated: https://www.youtube.com/watch?v=i5tTcsFb2Jg
Web scraping comes in different forms. Some people just want everything on the website, some people want everything from page 1 to page 10, and some people want only specific information from a page or a list of pages.
I think your tool would come in handy for the last example. Websites usually follow a template. For example, you can expect a user's profile page to always have the same structure (no matter which profile you're on).
Today, (unless it already exists) you need some form of web competences to be able to build a web scraper. They would have to analyse the website's HTML to find the right element target it and then write a script that loops through all available pages to get that specific information.
This is quite limiting for non-devs. With your tool, you could potentially solve this problem. Users could simply click on elements they're interested in, which creates a 'scrape template' (where they can add or delete what they clicked on, kind of like what you showed with username and user_mentions). With this template, the user could apply the scrape template from page A to page Z and run your plugin.
Thanks for taking your time and the long explanation! Your answer not only shows me that there might be a market, but that is has more possibilities.
Thank you!
BTW my current chrome extension scrapes content from Instagram accounts for the purpose of building a Progressive Web App from the posts (only if the user owns the account).
It would be nice if I could scrape a users IG account directly without the use of an extension and send the data to my back end. Can't get around CORS and IG on that one. So if you or anyone has ideas please contact me.
How is the PWA built? In a new tab of the browser?
You could probably go around CORS by server side rendering of IG with puppeteer.
To help I would need a bit more info on which part talks to what in your app.
The PWA is built when the user downloads my chrome extension AND then logs into their Instagram account. The extension detects this and then the extension icon turns green. When the green icon is lit the user can click it and the extension will scrape the first 12 images/comments and the users profile. It also opens up several options in a dropdown that allows the user to select PWA features and add many other Social Media links etc. The PWA is actually built when the user clicks a Build/Save button in the extension dropdown. The new PWA app is accessed by the URL ceri.us/<IG acct name> once it is built.
Without the IG login the extension icon stays red.
I have heard of puppeteer but I really don't know how to do this from my server. My users can do all of the above WITHOUT registration. What I would like to do is potentially allow the user to skip the extension download (at least on the initial build) and the act of just typing in ceri.us/<IG acct name> will build the most basic version of the PWA app (even if the user does not own the account which shouldn't matter as long as they can't add or otherwise change the PWA content) until they are logged into their IG account and have the extension).
It just makes it that much easier to introduce someone to the service. Especially those who may not even know what a browser extension is or how to get it.
In essence all I really need to have happen is to scrape the IG account for the profile text and first 12 posts image short codes (and maybe initial comment/hashtags although comments are tougher to get to). From that I already have back end scripts that can build the PWA app given that input.
With the extension, the user can subscribe to premium features and up to 1500 posts that can go into the PWA.
Hi Maciej,
I am not sure this helps because it is for a different purpose. But there is some crossover you might find useful. (and I did this many years ago but I think it still works)
This is something I have thought quite a bit about and actually have chrome/safari extensions that search (and optionally highlight or mark on mouseover). But I do not collect the data in classified arrays. Instead, I search for Images, URL's and phone numbers. Optionally I highlight the URL's and phone numbers as the user surfs or upon mouseover. And in all cases upon mouseover I pop up a temporary clickable button.
When the user clicks the button I digitally modulate an audio signal that can be picked up by a nearby computer (also running the extension) or by a native mobile app listening for these pulses.
I am not sure if the google play native mobile app still works because I ported the receiver side to Javascript (which still does not work in IOS). But you can still get the native mobile receiver app from the apple store. Called AirBridge Communicator (it is free but I think there was something with a similar name that had a cost)
And in the chrome extension store it should have the same or similar name.
Works across five audio band where the highest band is above MOST human hearing.
But the main part for you may be to simply get the chrome extension to see how I inject and highlight data and that may be useful or give you ideas.
Best of luck
Kerry