I'd really appreciate some technical advice
The goal
I'd like to build a job board filled with certain kinds of jobs from specific companies (I have a list of about 200 companies).
My current workflow
Right now, while I'm in the validation phase, I'm doing this manually. Here are the steps I currently take:
- I separately go to each company's job board
- I manual scroll for the type of job(s) that fit my criteria
- Finally, I copy/paste the link(s) when I find an appropriate job post
Request for advice
Is there any way to automate this process for the 200 companies I have shortlisted?
For example, how do managers of this job board aggregate and manage so many job listings? I'm pretty sure they've automated the collection of job listings to create a critical mass, but correct me if I'm wrong!
For now, I'd like to know if there's a reasonable/reliable solution for my question above, then I'll assess whether I should invest the time or money it might cost.
Thanks so much!
You need to use a web scraper on which you can define bots/behaviours for specific websites.
I did something like this previously using https://apify.com/
Essentially you point a bot at a webpage, tell it where the new jobs appear (e.g. in a certain <div> with a specific ID) and then tell it to grab the job data, item by item.
It's quite a bit of work to define these rules for multiple sites, let alone 200 sites - but it is possible!
btw I have zero affiliation but I was answering a similar question before and someone else suggested using https://www.parsehub.com/ - I have not used this tool but it looks good and if I was doing web scraping today I would give that tool a try.
Thanks a lot for the recommendations! I've set up a meeting with Parsehub and will likely do the same with Apify.
Absolutely. Use web scraping. Will have to tailor to each site to extract data. Then store in your db. Check T&C's and you'll prob find you're abusing them.
Thanks!
If your coding experience is shy of building custom web scrapping code then you can try Katalon studio. https://www.katalon.com/ You can perform routine tasks/automation using this tool. It may take a little to ramp up on how to use it but you may find it helpful. As with any automation/scraping solution, if anything on the source site changes then it is likely to break existing methods. Good Luck
Cool, I'll definitely check it out--thanks!
Hi Jehan, I'm building this product at https://rolecat.com. It's currently getting 230k roles from company job boards and will get more in the future (I'm currently focused on the quality of job location data rather than the number or breadth of roles captured). I'm happy to talk with you about using my platform to build your site / business if that's of interest. Feel free to contact me via email at (my first name) @rolecat.com.
I would expect job boards to web scrap content from sites and stores them in a db. They would then use particular key words to associate a particular job into a category (e.g. Analytics, Biz management, Design etc).
You would build an Admin page, to (quickly) approve jobs that were automatically assigned or move jobs that were mis-assigned for the short term.
Medium term (with move confidence with your algorithm keyword filtering), everything is approved and you just fix the mis-assigned ones.
Long term, sky-net is born, everything is automatic and it takes over the planet! :)
NOTE: if you're going to web scrap, make sure it's not obnoxious and kills the content server (becz, you'll get IP banned). Put delays and be considerate.
Appreciate the tips!