Library that can extract relevant text (readable text) from a web page.
Developers - I am working on a project where it can extract the readable content from a web page. I have been searching around for a while (I have tried: Apache Tika and few other Java libraries). No success yet. Noticed that few chrome extensions are doing this job nicely (Hewizo, Natural Reader, Elocance etc). Loved the precision of their reading. I was wondering if the indie-hacker community can shed some lights. Thank you!
If you're after something java specific I can't help you. However I have used this javascript package in the past: https://github.com/ageitgey/node-unfluff
Unfortunately it relies on coffeescript and cheerio, but if you can make that work for you use it. It works well.
There are some links to java projects that it's based on in the readme though!
Looks promising , I will be exploring more about it. Thank you for sharing!
There is neat little trick to do that using text based browser called Lynx. You can install it and then just call from commandline
$ lynx https://docula.app -dump > content.txt
(just change the url and you are done)
Thank you! Looks like it extracts all content (including menu) .
Depending on what you are looking to do Scrapy might be what you are looking for (https://docs.scrapy.org/en/latest/index.html) I have used this with Splash (https://splash.readthedocs.io/en/stable/). FYI always read the robots.txt of the site you are "extracting" data from. If you are not interested in learning Python/Scrapy/Splash it wouldn't cost much to hire this out. I might be able to help although I'm certainly not an expert.
Best of luck!
Thank you! I am looking for something in Java or javascript space.
Hi Philip! I suggest having a look at Safari's 'read mode'.
What do you want to do with the readable text?
I would like to programmatically extract the text and process and process them to produce audio. I have been working on a text to audio app (Chrome plugin & Web application). Its calle "XpressCue" https://xpresscue.com
Google Chrome Plugin "XpressCue" is available in the chrome store as well.
compromisejs ?
Have you had a look at Jericho yet? There's also a few APIs out there, but I'm guessing you want to do it yourself rather than pay for a third party.
http://jericho.htmlparser.net/docs/javadoc/net/htmlparser/jericho/TextExtractor.html
I haven't tried Jericho. I will take a look. yeah that is correct I am trying to do it myself rather than paying. Though I would like to take a look at paying options. Thank you!
This comment was deleted 6 years ago
I will give a try. Thanks for sharing!