robots.txt?

**Mike** · 06-02-2003, 08:44 AM

Hi there,

Would someone please tell me what robots.txt is? I have heard it loads of times but never actually come across it on my server...

Thanks,
Mike

**Chris** · 06-02-2003, 09:18 AM

check out http://www.robotstxt.org

**Mike** · 06-02-2003, 10:01 AM

Ok, thanks Chris.

Just a couple of questions I have...

Would you use a robots.txt file or the Robots meta tag? and, If I am doing a redirect thing to link to other sites should I use this robots technique so that I make sure it count as a external link?

Thanks for information so far,
Mike

**Chris** · 06-02-2003, 10:53 AM

robots.txt is better than the robots meta tag

**Mike** · 06-02-2003, 11:01 AM

Thanks.

**incka** · 08-15-2003, 12:16 PM

Looks interesting... Im going to use it to block my stats... I seem to be looking pagerank because google is following all the links in it.

**s2kinteg916** · 12-10-2003, 04:53 PM

bringing back the old post ... if you make a robots.txt file do u need to linclude it in a webpage via metatags ?

**lo0ol** · 12-10-2003, 05:42 PM

Just place robots.txt on your domain and you're set to go. The search engines and most other crawlers/spiders will check for it there. If you would like to view examples of other sites' robots.txt, just check them out yourself at theirdomain.com/robots.txt.

**incka** · 12-22-2003, 12:25 PM

Google has a page indexed even though my robots.txt tells it not to, but it had indexed it before I told it not to, but it should delete its index of it shouldn't it...