A crawler is a program used by search engines to collect data from the internet. When a crawler visits a website, it picks over the entire website's content.