Results 1 to 9 of 9

Thread: What is robots.txt

  1. #1

  2. #2
    Registered
    Join Date
    Oct 2017
    Posts
    41
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  3. #3
    Registered
    Join Date
    Oct 2017
    Posts
    54
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
    Appliance repairs Brisbane | Fridge repairs Brisbane | Oven repairs Brisbane | Clothes dryer repairs Brisbane | Washing machine repairs Brisbane | Dishwasher repairs Brisbane | Electrical services Brisbane | Workshop repairs Brisbane

  4. #4
    Registered
    Join Date
    Nov 2016
    Posts
    649
    It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page , you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty.

  5. #5
    Registered
    Join Date
    May 2016
    Posts
    221
    A "robots.txt" file is a text file placed on your server which contains a list of robots and "disallows" for those robots.
    Using a robots.txt file is easy, but does require access to your server's root location. For instance, if your site is located at:
    http://abc.com/mysite/index.html
    you will need to be able to create a file located here:
    http://abc.com/robots.txt
    If you cannot access your server's root location you will not be able to use a robots.txt file to exclude pages from your index.

  6. #6
    Registered
    Join Date
    Jul 2017
    Posts
    120
    Robots.txt is a text file webmasters create to instruct web robots typically search engine robot to crawl pages on their website.

  7. #7
    Registered
    Join Date
    Nov 2017
    Posts
    28
    The robots exclusion protocol (REP), or robots.txt is a text file webmasters make to instruct robots (usually search engine robots) how to crawl and index pages on their website

  8. #8
    Registered
    Join Date
    Sep 2017
    Location
    Tamilnadu
    Posts
    137
    robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard determines how to educate the web robot about which regions of the site ought not be prepared or filtered.

  9. #9
    Registered
    Join Date
    Sep 2017
    Posts
    164
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Similar Threads

  1. Robots.txt
    By AndyH in forum Website Programming & Databases
    Replies: 4
    Last Post: 01-29-2005, 01:55 PM
  2. Robots.txt
    By incka in forum Search Engine Optimization
    Replies: 2
    Last Post: 04-10-2004, 06:59 PM
  3. robots.txt
    By delpino in forum Search Engine Optimization
    Replies: 2
    Last Post: 01-23-2004, 02:13 AM
  4. robots.txt?
    By Mike in forum Search Engine Optimization
    Replies: 11
    Last Post: 12-24-2003, 09:27 AM
  5. JS or Robots
    By lemon in forum Search Engine Optimization
    Replies: 1
    Last Post: 09-26-2003, 08:19 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •