What is Web Scraping?
Hi All,
This is Gürkan. I’m founder of Scrapeak.com
I started bloging about web scraping, The first subject, of course, is “ What is Web Scraping?”
It is simply called cleanly extracting the irregular data on the website. You can do that with most programming languages. But we use and recommend the Python.
Because Python is super strong at web scraping. Python has web scraping modules and libraries. For example;
- ‘Requests’ for HTTP requests
- Beautifulsoup(a.k.a bs4) for Parsing HTML
- Lxml for best encoder
- Selenium for rendering javascript
- Re for regular expression
This is what the html code of the imdb page looks like. You can see with devtools.
We can build, integrate and manage web scraping projects!
www.scrapeak.com | info@scrapeak.com
Hello there!
I'm also Web Scraping enthusiast.
Python is my go to, but it's important that for example Selenium is written for a lot of other languages like JS, Java, C# or Ruby.
For static sites, I used Scrapy, what's your opinion on it?
More important question for me, as I'm starting diving into the topic of avoiding getting blocked by anti-bots. Do you have resources that could help me get familiar with it?
Good luck with growing Scrapeak!
Hi!
Yes, selenium is written for a lot of other languages. But Selenium is a test automation. Its main target is not web scraping.
Selenium is very slow compared to bs4 and requests. If I can do it with only requests and bs4 I definitely wouldn't use it
I use selenium for cookie extracting, login, javascript rendering and dynamic sites.
Scrapy is a product so inflexible and gives you a fixed frame. I like to build my own special structures and be flexible. That's why I don't use scrapy.
Anti bots are all about number of requests, headers, cookies, and ip.
If you have a question you can ask without hesitation. I can help you
I'm aware of Selenium being created for test automation, being slow and clunky. As far as I know, we can't replace it with anything better when it comes to scraping dynamic websites, is it correct?
Is it possible to combine Selenium with let's say bs4, use Selenium to login, load page via clicks and then extract given HTML with bs4, or it doesn't work that way?
So far I only used one or the other, never thought of combining them.
Really appreciate your time put to answer me! :)
Yes you can combine!
It's possible to combine Selenium with let's say bs4, use Selenium to login, load page via clicks and then extract given HTML with bs4.
No problem, even a good way.
That's great to hear!
You made my projects better with this! 😊
How does it compare to JavaScript. I must admit I’m a JS nerd and don’t want to pick up python for this if there are equivalent tools available in Javacsript to do it. What do you think?
I haven't many experience at web scraping with javascript.
Puppeteer looks nice for selenium alternative. After a short research, I found a module called "cheerio" in the bs4 alternative.
My advice to you. Is if you really want to make good and issueless projects on web scraping. You should learn Python. Since you know Javascript, it will be very easy for you to learn.
I usually go with Scrapy as I often work with large scale scrapers so it is easy to manage. Your website looks kinda empty, what scraping projects are you working on right now?
What would you recommend for my website?
Q : What scraping projects are you working on right now? ;
A : To a company that is currently working on location analysis. We have a scraper network that tracks (day by day) stores of over 1000+ retail brands.
We also have a scraper network that tracks local events from 10+ websites. We provide this as an API. Our client shows this on his site in an up-to-date as a list
We also built scrapers for the website of the agents of a famous real estate portal. It routinely enters the websites of the agencies and automatically add the new advertisements to the famous real estate portal. In this way, real estate agents save time and gain information accuracy.
and more ...
Do you think I can use Scrapeak to scrape for trending Instagram location hashtags in any given area and put them on my static website? Can it be automated to scrape in time intervals of every two weeks?
Yes, you can use!
You can automate it or even use also use api web service that is special to you.
Reach it via e-mail address. info [at] scrapeak.com
This comment was deleted 3 years ago
Yeah, it's possible.
If you need help you can contact me!
This comment was deleted 3 years ago