I track competitor prices for agencies at $200/mo — built the whole thing in python

started building web scrapers for my own projects about a year ago. needed to monitor competitor pricing, track review scores, pull lead lists from directories. standard stuff.

then an agency friend asked if i could do the same thing for one of their clients — a retailer who wanted to know every time a competitor changed their price on a specific product. took me about 3 hours to set up. charged them $200/mo to keep it running.

that was 4 months ago. still running. still getting paid. total maintenance time per month: about 15 minutes to check the logs and make sure nothing broke.

what the scraper actually does:

checks 8 competitor websites every morning at 6am. pulls current prices on ~120 products. compares them against yesterday. if anything changed by more than 5%, it fires off an email to the client with a spreadsheet showing exactly what moved and by how much.

the client uses this to adjust their own pricing within hours instead of finding out weeks later that they've been undercut.

the tech is embarrassingly simple:

python + requests + beautifulsoup. runs on a $5/mo server. cron job kicks it off daily. csv output gets emailed via smtp. the whole script is maybe 200 lines.

the hard part was never the code. it was understanding what the client actually needed. they didn't want "all the data" — they wanted to know "which prices changed and should i care?" that filtering logic took more thought than the scraping itself.

other scraping jobs ive done:

• lead list generation: pull business names, emails, phone numbers from industry directories. agencies use these for outbound campaigns. $150-500 per list depending on size
• review monitoring: aggregate google, yelp, and facebook reviews into one report. flag negative reviews for immediate response. $100/mo per client
• job board aggregation: scrape 6 job boards daily for a recruiter who wanted all new postings in one spreadsheet. $300/mo

what i learned:

the money in scraping isn't in the scraping. it's in turning raw data into something someone can act on in 30 seconds. a csv with 10,000 rows is worthless. a summary that says "competitor X dropped widget price by 15% yesterday" is worth $200/mo forever.

if you're a developer sitting on scraping skills, you're sitting on a service business. agencies and businesses will pay monthly for data they'd otherwise spend hours gathering manually.

happy to answer questions about the setup, pricing, or how to find clients for this kind of work.