-
robots.txt question
I'm sure I could figure this out on my own, I just wanted to be 5000% sure I do this right.
How would my robots.txt file look if I wanted to block .php extensions from being crawled by all search engines? Only .php, everything else is allowed.
Thanks!!
-
I dont think robots.txt files where designed to be that sophisticated.
I take it there's to many different php files to list them individually in robots.txt?
-
You could put all your .php files in a separate directory and then disallow that directory in robots.txt.
-
Is there no way to just block *.php?
-
Just found this (scroll to bottom):
http://www.searchengineworld.com/mis..._txt_crawl.htm
It indicates Google will recognize disallowing of certain file extensions, but other bots may not.
-
Thank you.. that's extremely helpful!
-
You could put the robots meta tag on each php page.