Web crawlers are spiders or bots that crawls your websites in detail to rank them on search engine. Every search engine has it's own crawler.