Finally launched my Web-Scraping tool Scrapify!
Check it out: https://scrapify.io/
It's a tool which makes extracting Data from Websites easy. Just select elements on a website and the tool will do the rest. Scrapify also supports following Pagination and detail pages.
Always looking for feedback!
Very nice, worked well for me.
Some feedback/questions from first impression:
Feature request:
Thanks for the Feedback!
This is an awesome idea! Nice job!
I like the idea + UI. It would be cool if you offer a widget in addition to the API to get the data directly displayed in a nice way as well.
Do you have a particular use case in mind? Would you want the widget to look like the DataGrid we already have or more like e.g. Air Table?
Use case would be embedding the results on a webpage / in a webtool without the need to put a lot of coding into connecting the API data. I usually prefer widgets to get a first impression if this is something I am going to use long-term or not before using API. Look of DataGrid would be fine for me.
How did you make sure it works with most websites?
Hired an virtual assistant who tests random e-commerce websites. We have a huge Spreadsheet and try to find common bugs to fix first. Doing this is making sure that we are not wasting time on a bug which only happens on one website.
Cool tool! Just tried it and bookmarked it =)
Side note. How did you go about validating the need for this tool? I'm still learning about validation and it would be a big help to hear how you did yours. =)
Super cool. The interface is great. Nice work.
Testing it out right now :)
One note: The chatbot "Tawk.to" has it's header message in German
Thanks, fixed!
Was very easy to setup a scraper for IndieHackers, one question why do I need to contact you for API access? Still working on the part to automate that part?
I feel like a tool like this could be really helpful to me
Yeah it's not automated yet :)
If you need API access or have any problems, just send me an email: contact@scrapify.io
Ah cool :)
I will send a request later on then.
Is there anyway to handle web authentication on the scraper? I'd love to be able to collect Private messages from various reddit accounts
Authenthication is a feature we will implement soon
Cool!
Just one note, i setup a scraper on the 12th, (when we last spoke) and set it to run daily, and today i log on and there was only a report for the 12th when i created it and none for yesterday or today (until i manually triggered it)
Appears that daily is not working (at least for mine)
my scraper job id is 15
Sorry, fixed it.
Thanks for the feedback!
No problem! I'm happy to help :)
If they do not have that feature, you could use nightmare js/casper.js to login and get the messages automatically.
Or..you can use their API to fetch message it seems: https://www.reddit.com/dev/api/#GET_prefs_messaging
https://www.pythonforbeginners.com/api/how-to-use-reddit-api-in-python
Sounds like any other scrapping tool. How do differentiate from other solutions on a market?
The focus is really on WYSIWYG and using smart algorithms to make it really simple to use. Columns are grouped, so the tool knows about the page structure and can keep the format in the Excel and Json export. A lot of other tools fail on sites where Scrapify would still work
From my experience (I write web scrapers for a living), having this WYSIWYG approach limits your scraping very much, you are limited to scraping homogeneous pages, and a lot of websites out there are more complex.
Also, can it scrape dynamic JS rendered pages? Or only html pages? I can't figure this out just by surfing on the website.
Dynamic websites with JavaScript are supported. The JavaScript is executed in a headless browser. The Goal of the tool is not to handle all websites, but to make it possible to scrape most websites without coding. Thanks for the feedback, I will make it clear that JavaScript is supported.
Dynamic websites with JavaScript are supported. The JavaScript is executed in a headless browser. The Goal of the tool is not to handle all websites, but to make it possible to scrape most websites without coding. Thanks for the feedback, I will make it clearer that JavaScript is supported.
This comment was deleted 8 years ago
Really nice idea, I was also thinking about somethingg similar when trying to scrape some site information.
From viewing the video, maybe show the outline of clickable elements at the start? It wasn't obvious to me that "Dean guitars" only would be the clickable part, I assumed the whole line of text would be.
Wow! Ok, this will help me a lot. When and where can i throw money at you? ;)
Seriously, you are miles ahead of what i am trying to code for myself at the moment. Keep up the good work!
Thanks, do you have an email where I can reach you? I would love to get your feedback to make the tool fit your needs
I send you an email at hello@scrapify.io with subject Indiehackers thread you can hit me up if you want ;)
Quite excited about this, actually. But connection errors are limiting my test.
Sorry, fixed it
Signed up :) FYI – Got stuck in the account creation process, kept loading. Went back to try again and said account was already created.
Sorry that you had issues. Did you use Safari on MacOS Mojave?
I did. It worked fine for me.
test