Web crawler is a search engine automated program that is responsible to read through webpage source and provide information to search engines. They are known as spiders, crawlers and robots.