Results 1 to 3 of 3

Thread: How to Block folder and file through Roots.txt File?

  1. #1

    How to Block folder and file through Roots.txt File?

    How to Block folder and file through Roots.txt File?

  2. #2
    Registered
    Join Date
    Jan 2019
    Posts
    259
    The way this is done is through a file called Robots.txt. Robots.txt is a simple text file that sites in the root directory of your site. It tells “robots” (such as search engine spiders) which pages to crawl on your site, which pages to ignore.

  3. #3
    A robots text file or the robots.txt file is a must have file for every website. This file is often added at the root directory of the website and adding this file is a very simple process.

    Also having this file in the website is considered as a sign of good quality by the search engines. Here are the simple steps to add the robots.txt files to your website:

    Identify which directories and files on your web server you want to block from being crawled by the crawler.
    Identify weather or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives.
    Use a text editor to create the robots.txt file and directives to block content
    Optional: Add a reference to your sitemap file (if you have one)
    Check for errors by validating your robots.txt file
    Upload the robots.txt file to the root directory of your site.

Similar Threads

  1. Why Is A Robots.txt File Used?
    By chetna in forum Search Engine Optimization
    Replies: 9
    Last Post: 06-03-2021, 12:24 AM
  2. Why We Use Robot.txt File In Seo..?
    By pihu147741 in forum General Chat
    Replies: 0
    Last Post: 03-24-2019, 10:25 PM
  3. Why Is A Robots.txt File Used?
    By worldescape in forum Search Engine Optimization
    Replies: 4
    Last Post: 12-06-2018, 10:15 PM
  4. Is it necessary to use Robots.txt file?
    By newsheadlines in forum Search Engine Optimization
    Replies: 8
    Last Post: 02-21-2018, 01:57 AM
  5. Password Protect a file/folder
    By computerikon in forum Website Programming & Databases
    Replies: 2
    Last Post: 04-18-2011, 12:39 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •