Results 1 to 5 of 5

Thread: What is robots.txt?

  1. #1

  2. #2
    Registered
    Join Date
    Jul 2019
    Posts
    486
    A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site

  3. #3
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  4. #4
    Registered RH-Calvin's Avatar
    Join Date
    Mar 2017
    Location
    Forum
    Posts
    1,667
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | VPS Starting from $12 PER Year
    Cheap Dedicated Server | Unmetered Bandwidth | Free Setup and IPMI

  5. #5
    Registered
    Join Date
    May 2020
    Posts
    64
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as �follow� or �nofollow�).

Similar Threads

  1. What is the X-Robots tag?
    By sandymary in forum Search Engine Optimization
    Replies: 5
    Last Post: 09-21-2018, 11:52 PM
  2. Is it necessary to use Robots.txt?
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 2
    Last Post: 05-27-2016, 05:53 AM
  3. Robots.txt help please
    By Blue Cat Buxton in forum Search Engine Optimization
    Replies: 2
    Last Post: 07-20-2005, 06:57 AM
  4. Robots.txt
    By AndyH in forum Website Programming & Databases
    Replies: 4
    Last Post: 01-29-2005, 01:55 PM
  5. robots.txt?
    By Mike in forum Search Engine Optimization
    Replies: 11
    Last Post: 12-24-2003, 09:27 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •