6
10 Comments

A productized web scraping service

I have seen a lot of posts regarding web scraping and data collection, be it leads, reviews or other data from web here in Indiehackers.

A lot of products exist for web scraping like Phantombuster, Simplescraper etc. But at the end these are products not services. Sometimes you need a customized scraper that does exactly what you need which may not be possible using these tools.

Would you prefer a productized web scraping service that is configured by professionals and does exactly what you want?

  1. 1

    I would totally welcome it. I am looking to scrape to build my data sets and few things I found are close but not exactly.

  2. 1

    I think this is a good idea! One way to control variability of effort required is by limiting it to popular sites like Amazon (for reviews), craigslist (for listings) etc. Since you will already have a script ready for these sites you can offer a good rate and make a healthy profit without spending too much time. :)

  3. 1

    Well.. it can be interesting for some people but if I were in your shoes I wouldn't do it because it can't be Scalable.

    1. 1

      I agree about the scaling thing but I do think people would be willing to pay more for a dedicated service.

  4. 1

    You can combine both approach.
    It is what apify is doing; They have productized API listing plus they have a some network of developer you can query for custom dev. So they are using the productized API as lead magned.
    I'm doing something similar with Phantombuster. I'm managing a community of users and offering custom development for use case not supported by the official API store.

    1. 1

      Yes I am familiar with Apify, they offer both. Nice to hear about a community of users looking for scraping and automation. How did you manage to build one?

      1. 1

        It is quite simple: create a linkedin group with "Phantombuster" in the name. As it is the only group mentioning "Phantombuser" in the name I'm getting 5 to 10 request to join per week.
        If you know other webscraping tools without groups it is a nice hack to implement ;)

  5. 1

    There are a ton of companies that do this, and they are pretty successful.

    Some examples on the first SERP:

    Search for "data as a service +scraping" to find more.

  6. 1

    Hi Sagun,
    Can you give an example? Would this be specific to one industry?

    1. 1

      No it won't be industry specific. Some of the use cases might be

      1. Reviews from playstore about competitors app
      2. Twitter account that have used a certain hashtags
      3. Aggregated news about certain technology (blockchain, AI)
      4. Monitoring price from Amazon, Bestbuy, Target

      And the list goes on. Basically a customized scraper as per your need that will run every day/week/month and get you the data. In return you pay a certain recurring fee just as you do for a product. Additionally you can request future minor changes, a human would be on the other side listening your request.

Trending on Indie Hackers
How I grew a side project to 100k Unique Visitors in 7 days with 0 audience 47 comments Competing with Product Hunt: a month later 33 comments Why do you hate marketing? 27 comments $15k revenues in <4 months as a solopreneur 14 comments Use Your Product 13 comments How I Launched FrontendEase 13 comments