So far today, Googlebot has hit my AWS sites about 45,000 times. It just extensively crawled some of these sites a few days ago. I've checked the logs, and Googlebot is often requesting pages one or more times per second, which is out of line. When Googlebot is crawling (which always seems to be during peak hours), it definitely loads the CPU. I'd contact Google, but am somewhat afraid I might incur the "thin affiliate" penalty. I'm looking into using mod_bwshare or bw_mod. Anybody else getting hit hard by Googlebot with their AWS sites?
Bookmarks