Results 1 to 12 of 12

Thread: What is Robots Text File

  1. #1
    Web Design Company
    Join Date
    Aug 2015
    Location
    Bangalore
    Posts
    732

    What is Robots Text File

    What is Robots Text File

    Robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

  2. #2
    Thoughtgrid Thoughtgrid's Avatar
    Join Date
    Jul 2016
    Location
    Bangalore
    Posts
    219
    The main purpose of using robots.txt file used to keep sensitive information in private it is used to avoid canonical problem it is mainly used for crawling it instruct search engine which page has to follow and which page has to no follow.

  3. #3
    Junior Registered
    Join Date
    Aug 2016
    Location
    Hyderabad
    Posts
    3
    The purpose of robots.txt file is it will give the instructions to google crawlers which page can crawl, which page cannot crawl for google indexing purpose

  4. #4
    This text file which sits in the root of your website's folder communicates a certain number of guidelines to search engine crawlers. For instance, if your robots.txt file has this line in it; User-agent: * Disallow: / it's basically telling every crawler on the web to take a hike and not index ANY of your site's content.

  5. #5
    Registered RH-Calvin's Avatar
    Join Date
    Mar 2017
    Location
    Forum
    Posts
    1,667
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | VPS Starting from $12 PER Year
    Cheap Dedicated Server | Unmetered Bandwidth | Free Setup and IPMI

  6. #6
    Registered
    Join Date
    May 2016
    Posts
    221
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door.

  7. #7

  8. #8
    Registered
    Join Date
    Oct 2016
    Posts
    270
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Best Android Training | Devops Training | SQL Server Training and many more IT courses.

  9. #9
    Quote Originally Posted by Nexevo Technologi View Post
    What is Robots Text File

    Robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  10. #10
    Gemstones
    Join Date
    Jul 2017
    Location
    Bangalore
    Posts
    36
    Hi,
    Mainly used for Robots.txt file
    what are the pages want to allow the search engine and restrict to some other main pages.

  11. #11
    The robots exclusion protocol or simply robots.txt.is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  12. #12
    Registered
    Join Date
    Jul 2017
    Location
    Surat
    Posts
    187
    Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.

    Robots.txt files are useful:

    If you want search engines to ignore any duplicate pages on your website
    If you don’t want search engines to index your internal search results pages
    If you don’t want search engines to index certain areas of your website or a whole website
    If you don’t want search engines to index certain files on your website (images, PDFs, etc.)
    If you want to tell search engines where your sitemap is located

Similar Threads

  1. How can i create Robots Text file for my website
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 0
    Last Post: 07-11-2016, 02:17 AM
  2. How to create the Robots Text file.
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 0
    Last Post: 07-08-2016, 04:07 AM
  3. How to Create the Robots txt file.
    By Nexevo Technologi in forum Search Engine Optimization
    Replies: 2
    Last Post: 03-16-2016, 03:55 AM
  4. Is Google ignoring my robots.txt file?
    By flyingpylon in forum Search Engine Optimization
    Replies: 9
    Last Post: 01-13-2004, 09:47 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •