How i scraped 1,000 marketing agencies across 54 countries in two weekends (the technical breakdown)

the whole scraping pipeline is about 400 lines of python across 3 scripts. no selenium, no puppeteer, no headless browsers. just requests and regex.

heres how it works.

step 1: find agency websites

i use web search to find agencies in a target city. the query is usually something like "digital marketing SEO agency {city} {country} contact email". the search results give me 8-15 agency names and URLs per city.

i started with australian cities (melbourne, sydney, brisbane) then expanded to NZ, UK, US, canada. then i got ambitious and started hitting every country i could think of — india, philippines, dubai, south africa, kenya, colombia, japan, south korea, 54 countries total.

step 2: extract emails from their websites

for each agency URL, i hit three pages: the homepage, /contact, and /contact-us. most agencies put their email on one of these.

the email extraction is just a regex. then i filter out junk — sentry DSNs, image filenames with @ in the path, schema.org references, WordPress defaults. my filter list has 30+ patterns.

step 3: dedup and batch

each scraping run saves results to a JSON batch file. i ended up with 257 batch files across all sessions. a compilation script reads all of them, deduplicates by email, and outputs a clean CSV.

final numbers: 798 unique agencies with verified emails from 257 source files.

what went wrong

1. some agencies appeared in multiple searches. i scraped the same email from different batch files and didnt dedup until later. result: some agencies got 2-3 identical pitch emails.

2. the regex catches garbage. image filenames, CSS class names that look like emails, visitor tracking pixels. the filter list grew from 5 patterns to 30+ over two weekends.

3. non-english markets have much lower email-on-website rates. in AU/UK/US, about 60-70% of agency websites have a visible email. in some asian and african markets its closer to 20%.

4. some websites block automated requests. adding a browser-like User-Agent header fixed most of them.

the output

798 agencies, 54 countries, clean CSV with columns: agency name, email, website URL, region. i packaged this as a product and its available for $19: https://vemtrac-outreach.pages.dev/leads

the whole pipeline runs on a linux server i already had. total cost: $0.

if youre building a cold outreach pipeline: start with web search, not databases. the hit rate from searching "{industry} {city} contact email" is 60-80%. paid databases give you more data but you pay per record. scraping is free and the data is just as fresh.

landing page: https://vemtrac-outreach.pages.dev