Results 1 to 5 of 5

Thread: Why Is A Robots.txt File Used?

  1. #1

  2. #2
    Registered
    Join Date
    Feb 2018
    Location
    Dallas
    Posts
    181
    Robot.txt tells the crawler that how it should crawl and index a particular web page of a website.

  3. #3
    Registered
    Join Date
    Jul 2018
    Posts
    142
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  4. #4
    Registered RH-Calvin's Avatar
    Join Date
    Mar 2017
    Location
    Forum
    Posts
    1,667
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | VPS Starting from $12 PER Year
    Cheap Dedicated Server | Unmetered Bandwidth | Free Setup and IPMI

  5. #5
    Website owners use the robots.txt file to give instructions about their site to web robots.

    The "User-agent: *" means this section applies to all robots.
    The "Disallow: /" tells the robot that it should not visit any pages on the site.

Similar Threads

  1. Is it necessary to use Robots.txt file?
    By newsheadlines in forum Search Engine Optimization
    Replies: 8
    Last Post: 02-21-2018, 01:57 AM
  2. What is Robots Text File
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 11
    Last Post: 08-01-2017, 05:14 AM
  3. How to Create the Robots txt file.
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 2
    Last Post: 03-16-2016, 03:55 AM
  4. How to add the robots txt file in weebly blog
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 0
    Last Post: 01-18-2016, 07:34 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •