Results 1 to 5 of 5

Thread: Which is better..?

  1. #1

  2. #2
    Registered
    Join Date
    Jun 2017
    Posts
    314
    It's a pretty common misconception that the robots.txt will prevent indexing. It's only purpose is actually to prevent crawling, anything disallowed in there is still up for indexing if it's linked to elsewhere. If you want something deindexed, your best bet is the robots meta tag, but make sure you allow crawling of the URLs to give search engine bots an opportunity to see the tag.

  3. #3

  4. #4
    Registered RH-Calvin's Avatar
    Join Date
    Mar 2017
    Location
    Forum
    Posts
    1,667
    Robots.txt is better compared to meta robots tag. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | VPS Starting from $12 PER Year
    Cheap Dedicated Server | Unmetered Bandwidth | Free Setup and IPMI

  5. #5
    Registered
    Join Date
    Jan 2018
    Posts
    66
    The Meta tag option tells bots not to index individual files, whereas Robots.txt can be used to restrict access to entire directories.
    Use a Meta tag if you have the odd page in indexed folders that you want skipping, but generally, I'd recommend you most of your non-indexed content in one or more folders and use robots.txt to skip the lot.
    Last edited by traveloweb; 02-01-2018 at 03:30 AM.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •