Printable View
Hi all,
I have a quick question on SEO. Does Google crawl your site all at once, or in little bits? A new site of mine has thousands of pages, Googlebot comes everyday but only seems to visit a few hundred pages. Should this be the case? Or should it visit them all in one swoop?
Thanks,
Mike
Both.
It typically does a deep crawl once a month or so, and then otherwise pecks and nibbles at the site throughout the month.
If you're worried that Google isn't crawling all of your site check out the tools at http://www.google.com/webmasters/ . You can use this to check for any pages on your site that Google is having trouble crawling.
Looks like things are ok then. I added a sitemap to google last week, and it's coming more regularly than normal. I just looked on the crawl error page and nothing was listed, so it seems like it'll just take a little bit of time.
Cheers anyway guys.
Be careful if you set crawl rate to high!!
Here's some stats from one of my sites where I enable dit:
(proxy site - they spyder proxified pages people link to, you may remember my thread in general area where ppl are complaining my site is outranking theirs)
- 350,000 pages and 10gb in Jan
- 380,000 pagies and 13gb in Dec
Heh, I better bear that in mind!
I shouldn't have a problem quite as big though, even though there are a lot of pages on my site, it doesn't have hundred of thousands just yet :)
For constantly changing content, google automatically sets is crawlers to crawl those pages once or even twice a day.
It is impossible for Google to index my site all at once. Instead it is more like a continuous crawl with Googlebot indexing a few thousand pages a day. Some sections seem to be crawled more often than others, but typically there is at least on complete deep crawl completed each month with core pages being indexed several times during the month.
Well yesterday Googlebot apparently visited over a thousand pages, so hopefully it'll find the rest soon as well.