- Custom robots.txt is a way for you to instruct the search engine that you don't want it to crawl certain pages of your blog.“Crawl” means that crawlers, like Googlebot, go through your content, and...