Web crawlers or spiders are software programs which are designed to crawl, index, rank and return webpages to users.