I run a niche job board. Learning to scrape Google Jobs kept it alive

I run Semiconductor Design Jobs, a job board for chip design engineers. The website was the easy part. The thing that nearly killed it was keeping the listings fresh enough that an engineer had a reason to come back on a Tuesday.

A job board is a promise of freshness

Nobody bookmarks a board with three-week-old postings. Every listing needs a title, a company, a location, the source it came from, and an apply link that still works when someone clicks it. In the beginning I gathered all that by hand: career pages, searches, copy, paste, check the link, repeat. It works for a demo. It doesn't work for a board that claims to be current, and falling behind is invisible until a visitor hits a dead apply link and never comes back.

Why I scrape Google Jobs instead of twenty separate sites

The realization that changed the project: Google already aggregates most job postings from the individual boards and career sites. So instead of maintaining a scraper for every source, I learned to scrape Google Jobs once and let Google do the aggregating.

I packaged the tool as an Apify Actor, originally just for my own pipeline. You give it a job title, a location, and optionally a country and language. It returns structured JSON: title, company_name, location, via (which platform the posting came from), posted_at, schedule_type, and apply_options with direct apply links per platform. It covers 11 countries, 103 languages, and 184 Google domains.

Two honest gaps, so you don't find out the hard way: there's no numeric salary field and no experience-level field. Google doesn't expose those consistently, and I'd rather tell you here than have you discover it in your data.

The same job search API now feeds other people's projects

Here's the part I didn't plan. Strangers started running it. As of this week, 2,763 accounts have used the Actor, 513 of them in the past month, with a 99.8 percent success rate on runs.

And they use it for things that have nothing to do with semiconductors. One published example tracks nursing jobs in Texas with apply links for healthcare recruiting. Another feeds remote software engineer listings to AI agents over MCP, so a Claude or ChatGPT agent can pull live jobs mid-conversation. A job search API is a building block, it turns out, and other people had very different buildings in mind.

What I'd tell another founder

The moat of a niche job board was never the scraping. It's curation and audience: knowing which fifty companies matter in your niche and being the place their engineers actually check. The data plumbing underneath is undifferentiated work, and that's exactly why it was worth building properly once and selling to everyone else who needs it. Look at the internal tool you've already rebuilt more than once. It might be your second product, and its customers may look nothing like you.

How does job scraping work?

A job scraper sends a search query to a jobs source, parses each listing into fields like title, company, location, posting date, and apply link, and returns structured rows (JSON or CSV) instead of web pages. Good ones also record which platform each posting originally came from.

Is job scraping legal?

Job postings are public data that employers actively want distributed; job boards have syndicated each other's listings for decades. That said, terms of service and local rules vary, and I'm not a lawyer, so check the rules that apply to your use.

Which job search sites have an API?

Very few official ones. LinkedIn and Indeed restrict their job APIs to approved partners, and Google Jobs has no official public API at all. That gap is the reason third-party job search APIs exist.

---

The tool from this story is Google Jobs Scraper on Apify; you can run it free. If you want the developer walkthrough with code, that's on Medium, and there's a shorter JSON-focused guide on Peerlist.