Are there opportunities with web scraping?
I don't see many people talking about it. It seems like it gets a bad reputation or something. I remember I made a small web scraper using Puppeteer and it was pretty cool.
One Idea I had was scraping various sites and creating a gift aggregator for your girlfriend/wife. It would have a good UI, filters, categories, etc. Just a thought I had. Another possibility is selling data you scrape which would be of interest for creators, e-commerce or marketers, maybe make a dashboard.
Anyway, do any of you have experience with web scraping or believe a business can be created with it?
I'm a frontend dev trying to monetize my skills.
Trending on Indie Hackers
The IG angle mentioned in this thread aged well. We are running a data API for Instagram influencer search — Social Intel (https://api.socialintel.dev). The new unlock in 2026: x402 payments mean AI agents can discover, pay, and use the API with no human onboarding. The API key management problem disappears entirely. Good margins too — $0.003/call and DataLikers does the actual scraping for us.
The IG angle mentioned here aged well. The new unlock in 2026 is agents - not humans - consuming the data. An AI agent running an influencer campaign can discover, query, and pay for data programmatically with no human in the loop. That changes both the TAM (agents run 24/7 at scale) and the business model (pay-per-call instead of subscriptions). Margins are solid because the actual scraping is outsourced to a data provider, you are just the API layer on top.
I would switch from thinking about scraping as a potential technology for gathering data, to think of what data exists but are not easily available and would provide more value if aggregated.
Within social media/influencer marketing space it is a war. I started building a data aggregation/analysis solution focusing on Instagram and YouTube about five years ago. Then Cambridge Analytica happened and Instagram gradually did shutdown APIs - creating a market for scraping the data instead. But it is a pure hell to maintain when things work one day and break the next (not to mention the grey zone with regard to T&C, etc). I'm happy I'm not in a business that relies on that anymore.
There are probably opportunities - for example, could you scrape some sort of information and sell access to a nice uniform API that organises this information for you? The problem is probably maintaing your scrapers - websites will change interface in strange and unpredictable ways and not notify you, necessarily. You'll likely sign up for a lifetime of trying to keep up with them.
Exactly this. And not only will the websites constantly change shape, but every websites will also have its own way of describing / categorising things, and those can change too. So producing filters/categories etc can be really hard.
There are definitely companies out there that do this already, but doing it at scale is expensive - not because the tech is hard, but because there's so much change to manage.
If you can find a way to intelligently keep up with the changes then you might be onto something!
Google Search is backed a web scraper, and they keep finding new ways to present and index the data they find. I don't see why others couldn't.
Having left a job where scraping was a huge part of it, we had to accept that we'd be re-writing constantly as things changed. I had ideas for variables and changed them time and again.
If your categories are bulletproof and having data en-masse has value, then there's some value. But working out what the problem you want to solve is the big thing, in my opinion. Then whether scraping is going to be able to tackle that problem for users profitably