Deep learning based web scraper as a platform and an API(For developers)
We are working on developing deep-learning based web scraper (which will scrape information from website visually rather than by looking at their source code)
We think we can make it as generic as possible and release it as an API and a platform that anyone can use/integrate into their application.
Currently, if you look at web scraping it is mostly done by experts and even they need to change scripts repeatedly as websites keep changing their source codes.
The deep learning framework which we are creating does not need to rely on the source code of any website and hence it doesn't matter how it is structured. Even a novice person can enter URL of any new (unseen by our algorithm) website and fetch data of his/her interest. In the end, we will provide them with a way to integrate that data into some channels like download as a file (.xlsx, .csv, .txt, etc), push it to drive, AWS S3, etc.
I would love to get feedback from the IndieHackers community on this.
Hey,
I've spent some time thinking about this. While a well trained machine learning model can perform what feels like magic 80% of the time, if the valuable data is in the remaining 20%, that 80% counts for zero. So you have to be very good.
Diffbot are currently leaders in this space and they've got some big clients and seem to be doing well. Their capital expenditure is also likely in the millions as they chase down the promise of an intelligent scraper.
If you can compete with that, go for it. As a bootstrapped biz, the cost of labor to build and maintain traditional web scraping methods was less than the cost of capital (and time) required to build an accurate machine learning based scraper, so we went with a selector/xpath approach. Once you have a foothold in the market, you can revisit the ML approach.
Good luck.
Fair points @humanismusic_. In order to be building a good deep learning model, we are focusing on one domain at a time. Also, we are not exactly having "one model fits all" philosophy and hence we will be building multiple models and stack them accordingly to bring out the best results.
I went through the Diffbot website and related material and they have done excellent progress. Though they are trying to build a very generic ML model (Which is very hard to achieve). We are trying to be focused and provide lots of add on services which will make our product plug and play kind of a thing to use.
Once we start it commercially I would love to have a collaboration with you guys (Or maybe while I am training and building the models) to ensure we build sufficiently accurate enough Deep learning models and they are commercially viable and scalable.
Would love to hear more from you.
Thanks
The idea is novel.
The success will depend on the generic ML model that you use.
All data captured as a screen shot, will have to be converted into structured data.
Additional processing power is required for converting image to text.
I would like to see when the POC is done.
Thanks @o2 for encouraging response. Currently working on building a dataset. Once done with the model, we will share POC here on indie hackers.
We have built few models (By tuning tesseract) to extract text from web page images and converting it into a structured one (This along with page identification model are consuming the most processing power so far).
Doing this as well, I'll follow you too track your progress
Thanks @Deus_Ex. Once we are at good progress we can explore collaboration opportunities on this. Also, would love to know your progress.
Certainly, once we develop proof of concept I'll share it and we can proceed from there.
Sounds great. Looking fowrard to it.