From indexing certain pages of your site?
Printable View
From indexing certain pages of your site?
Yes, use robots.txt
http://searchengineworld.com/robots/robots_tutorial.htm
Thanks for the link!
Question...what does this imply? I think I more or less understand a lot of it except this.
Quote:
The robots.txt file should be created in Unix line ender mode! Most good text editors will have a Unix mode or your FTP client *should* do the conversion for you
Unix and windows have different ways of determining a new line. For example in PHP unix/linux will understand \n for newline but windows needs \r\n (if i remember correctly!).
You don't need to worry about this when ur dealing with robots.txt, it's a programming issue. Ur ftp client will sort it.
:cool:
Could you also use this or is it not a very reliable way to block pages from being crawled?
<META NAME="ROBOTS" CONTENT="NOINDEX">