Some of the challenges of building an internet search

Based on a review of recent failures in my indexing log, roughly 5% of all sites were not being indexed due to indexing errors, and of these indexing errors, 62% were sites that had disappeared, 19% were sites where robots.txt blocked indexing, and 5.5% were sites where Cloudflare blocks indexing. Further information at https://blog.searchmysite.net/posts/some-of-the-challenges-of-building-an-internet-search/ .