Squashed a few bugs over the year-end break and added one key new feature - checking the robots.txt file to make sure that it isn't blocking access to a site. This compliments the check for the x-robots-tag header and robots meta-tag introduced late last year.