Too many Googlebots

[HiddenVorlon]

I have many sites on my server with hundreds of thousands of pages of content that are all dynamically driven. I’ve optimized my sites pretty well and clients generate new content on a daily basis. Day to day operations are fine, but when an army of Googlebots hits several of the sites at a time that have hundreds of pages of content, it brings my database server to a crawl.

Is there anything that you can do to throttle the Googlebots? Something in robots.txt?

[otherground]

Could you possibly cache your dynamic content to help reduce the amount of work that the database has to do ?

[Peter T Davis]

Are you using the same script on all of your sites? Maybe there are some problems in the script that need fixing.

[HiddenVorlon]

I'm looking at the caching now... that might help. The sites are very dynamic: navigation, content, footer, etc. are all database driven. Under normal operation this runs just fine. It's just the sheer volume of googlebots hitting the thousands of pages just brings the system to a crawl.

[otherground]

alternatively you could just block the Googlebot ... but your site won't be indexed ..

[paul]

I saw this posting on Sitepoint that might apply to your situation.

http://www.sitepoint.com/forums/showpost.php?p=1145730&postcount=16

[HiddenVorlon]

That post looks like it could help. I'm running IIS, but it should have similar settings...

[Percept]

alternatively you could just block the Googlebot ... but your site won't be indexed ..

That's not something I would recommend ;)