Scraping job posts

For a project I've started that is very dependant on scraping pages, I use diffbot. It's expensive but the results it achieves from such little input are quite astonishing.

For a side project I scraped job postings for keyword analysis, and used a golang package called colly

I have a site that does this (www.remotelyawesomejobs.com), and it's a mix of RSS feeds, hitting the page with an HTTP client (I'm using ruby), or in extreme cases where Javascript is required, I use the same headless chrome tools (e.g. capybara) which are more typically used to do automated feature testing.

It's very hit-or-miss. For example, I often have problems with Stack Overflow because they use several different layouts, and they change them frequently, making scraping a difficult cat-and-mouse game (they have RSS, but the feed doesn't provide the full job description). Angellist works very aggressively to block scraping using a bunch of techniques, so I stopped trying.

I've been experimenting with using fetchrss.com to auto-generate RSS feeds from sites that don't have RSS, but haven't implemented any yet.

Just a post I needed to see. Looking for this exact info.

You don't ? At least I didn't for findamaker.io