PDA

View Full Version : robots.txt?



Mike
06-02-2003, 08:44 AM
Hi there,

Would someone please tell me what robots.txt is? I have heard it loads of times but never actually come across it on my server...

Thanks,
Mike

Chris
06-02-2003, 09:18 AM
check out http://www.robotstxt.org

Mike
06-02-2003, 10:01 AM
Ok, thanks Chris.

Just a couple of questions I have...

Would you use a robots.txt file or the Robots meta tag? and, If I am doing a redirect thing to link to other sites should I use this robots technique so that I make sure it count as a external link?

Thanks for information so far,
Mike

Chris
06-02-2003, 10:53 AM
robots.txt is better than the robots meta tag

Mike
06-02-2003, 11:01 AM
Thanks.

incka
08-15-2003, 12:16 PM
Looks interesting... Im going to use it to block my stats... I seem to be looking pagerank because google is following all the links in it.

s2kinteg916
12-10-2003, 04:53 PM
bringing back the old post ... if you make a robots.txt file do u need to linclude it in a webpage via metatags ?

lo0ol
12-10-2003, 05:42 PM
Just place robots.txt on your domain and you're set to go. The search engines and most other crawlers/spiders will check for it there. If you would like to view examples of other sites' robots.txt, just check them out yourself at theirdomain.com/robots.txt.

incka
12-22-2003, 12:25 PM
Google has a page indexed even though my robots.txt tells it not to, but it had indexed it before I told it not to, but it should delete its index of it shouldn't it...

s2kinteg916
12-23-2003, 11:15 PM
i believe you will have to wait till google updates...i seem to be in the same boat...

incka
12-24-2003, 04:44 AM
The problem is I've had that robots.txt file there since September and it hasn't found it...

r2d2
12-24-2003, 09:27 AM
I doubt it would delete the index of the file, robots.txt will just stop the next googlebot from looking at the file I would think.