4
9 Comments

Web Scraping

Anyone have recommendations on a platform or api to use for web scraping?

I’m looking to regularly grab product data from a set of pages (in stock, price, other metadata).

I stumbled upon simplescraper but it’s a bit out of budget to start off.

on February 2, 2022
  1. 3

    Scraping is pretty easy with some Python libraries like Beautiful Soup and Selenium. If you have any coding experience with Python, it won't take more than a weekend to write your own scraper and save that data directly in your database. To get the data regularly, you can use cron jobs or something similar, which will run your code at the specified intervals automatically.

    1. 1

      Thanks for the insight. I’d never built one before so wasn’t too sure about the level of effort involved.

  2. 1

    Consider using Promptcloud for tailored regular data extraction needs, or 42signals, which specializes in ecommerce data.

  3. 1

    Hey! You should try a new API based scraping tool ScrapeIN" - https://scrapein.app/. It is completely free now for the first beta users.

  4. 1

    It depends on your use case. If you want to scrape a website that doesn't have anti-scraping mechanisms, you should be good developing a custom python code to get the data.
    If you're thinking about more demanding websites, you indeed need a web scraping API unless you want to invest into good quality proxy pool and apply multiple bot detection solutions.
    We have recently released a web scraping API: https://scrapingfish.com which is powered by our own mobile proxy pool infrastructure.
    You can buy 1.000 API requests for $2.

  5. 1

    Hi Zane!

    Web scraping API and no-code tools are very basic. I'm working on web scraping agency and managment tool. It's Scrapeak.com. We'll do the heavy lifting for you.

    We can build your scraper quickly and manage (regularly update or grab and api service).

    You can contact to me info[at]scrapeak.com

  6. 1

    Another option for scraping Scraping Bee

  7. 1

    Proxycrawl.com is the one that offers the best API service to scrape any publicly available data, including stock price data. It is a complete web data crawling and scraping solution. Proxycrawl offers a trustworthy API for programmatically requesting sites, as well as machine learning-based data filtering and extraction.