Robot.txt also known as robots exclusion protocol (REP) is a text file created to guide the search engine robots (web crawler) which information on the website should or should not be processed or...