Results 1 to 7 of 7

Thread: robots.txt question

  1. #1
    Senior Member Kyle's Avatar
    Join Date
    Jun 2003
    Location
    Chicago
    Posts
    840

    robots.txt question

    I'm sure I could figure this out on my own, I just wanted to be 5000% sure I do this right.

    How would my robots.txt file look if I wanted to block .php extensions from being crawled by all search engines? Only .php, everything else is allowed.

    Thanks!!
    Kyle

  2. #2
    Resident smart a$$
    Join Date
    Sep 2003
    Location
    UK
    Posts
    152
    I dont think robots.txt files where designed to be that sophisticated.

    I take it there's to many different php files to list them individually in robots.txt?
    Last edited by Best.Flash; 02-27-2004 at 12:11 AM.

  3. #3
    Registered GCT13's Avatar
    Join Date
    Aug 2003
    Location
    NYC
    Posts
    480
    You could put all your .php files in a separate directory and then disallow that directory in robots.txt.
    ....

  4. #4
    Senior Member Kyle's Avatar
    Join Date
    Jun 2003
    Location
    Chicago
    Posts
    840
    Is there no way to just block *.php?
    Kyle

  5. #5
    Registered GCT13's Avatar
    Join Date
    Aug 2003
    Location
    NYC
    Posts
    480
    Just found this (scroll to bottom):
    http://www.searchengineworld.com/mis..._txt_crawl.htm

    It indicates Google will recognize disallowing of certain file extensions, but other bots may not.
    ....

  6. #6
    Senior Member Kyle's Avatar
    Join Date
    Jun 2003
    Location
    Chicago
    Posts
    840
    Thank you.. that's extremely helpful!
    Kyle

  7. #7
    Administrator Chris's Avatar
    Join Date
    Feb 2003
    Location
    East Lansing, MI USA
    Posts
    7,055
    You could put the robots meta tag on each php page.
    Chris Beasley - My Guide to Building a Successful Website[size=1]
    Content Sites: ABCDFGHIJKLMNOP|Forums: ABCD EF|Ecommerce: Swords Knives

Similar Threads

  1. Question...
    By incka in forum Domain Names
    Replies: 4
    Last Post: 04-01-2004, 07:17 PM
  2. DADA question
    By incka in forum General Management Issues
    Replies: 2
    Last Post: 01-29-2004, 09:39 AM
  3. robots.txt
    By delpino in forum Search Engine Optimization
    Replies: 2
    Last Post: 01-23-2004, 02:13 AM
  4. Is Google ignoring my robots.txt file?
    By flyingpylon in forum Search Engine Optimization
    Replies: 9
    Last Post: 01-13-2004, 09:47 AM
  5. robots.txt?
    By Mike in forum Search Engine Optimization
    Replies: 11
    Last Post: 12-24-2003, 09:27 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •