PDA

View Full Version : robots.txt and pagerank



s2kinteg916
06-02-2004, 09:29 PM
how come on pages where i dissallowed robots still has pagerank. Shouldnt it be PR 0 ?

mobilebadboy
06-02-2004, 09:38 PM
It's probably because Google hasn't requested the robots.txt file since you changed it. Once a day or on a "big" crawl of your site is when it usually references it. And even then it might be a while before the updates are made once it reads the file.



In order to save bandwidth Googlebot only downloads the robots.txt file once a day or whenever we have fetched many pages from the server. So, it may take a while for Googlebot to learn of any changes that might have been made to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Finally, you may want to check that your syntax is correct against the standard at: http://www.robotstxt.org/wc/norobots.html. If there still seems to be a problem, please let us know, and we will correct it.

s2kinteg916
06-02-2004, 09:45 PM
ive had the robots file for around 3 or 4 months after numerous pr updates... i might be doing something wrong

incka
06-03-2004, 02:20 AM
robots.txt muss disallow a folder, not just a file.

s2kinteg916
06-04-2004, 10:29 AM
yeah i have one example with it dissallowing a folder and i still see PR

chrispian
06-04-2004, 11:04 AM
Does anyone link to this site or page? If so, then you have PR because of the links. robots.txt just keeps spiders from viewing your page, not from my site transferring PR to it. That would be my guess.