PDA

View Full Version : robots.txt question



Kyle
11-04-2003, 02:15 AM
I'm sure I could figure this out on my own, I just wanted to be 5000% sure I do this right.

How would my robots.txt file look if I wanted to block .php extensions from being crawled by all search engines? Only .php, everything else is allowed.

Thanks!!

Best.Flash
11-04-2003, 04:08 AM
I dont think robots.txt files where designed to be that sophisticated.

I take it there's to many different php files to list them individually in robots.txt?

GCT13
11-04-2003, 08:44 AM
You could put all your .php files in a separate directory and then disallow that directory in robots.txt.

Kyle
11-04-2003, 12:45 PM
Is there no way to just block *.php?

GCT13
11-04-2003, 01:05 PM
Just found this (scroll to bottom):
http://www.searchengineworld.com/misc/robots_txt_crawl.htm

It indicates Google will recognize disallowing of certain file extensions, but other bots may not.

Kyle
11-04-2003, 01:07 PM
Thank you.. that's extremely helpful!

Chris
11-04-2003, 03:13 PM
You could put the robots meta tag on each php page.