Results 1 to 9 of 9

Thread: How many times does Googlebot visit?

  1. #1
    Registered Mike's Avatar
    Join Date
    May 2003
    Location
    UK
    Posts
    2,755

    How many times does Googlebot visit?

    Hi all,

    I have a quick question on SEO. Does Google crawl your site all at once, or in little bits? A new site of mine has thousands of pages, Googlebot comes everyday but only seems to visit a few hundred pages. Should this be the case? Or should it visit them all in one swoop?

    Thanks,
    Mike
    Don't you just love free internet games ?

  2. #2
    Administrator Chris's Avatar
    Join Date
    Feb 2003
    Location
    East Lansing, MI USA
    Posts
    7,049
    Both.

    It typically does a deep crawl once a month or so, and then otherwise pecks and nibbles at the site throughout the month.
    Chris Beasley - My Guide to Building a Successful Website[size=1]
    Content Sites: ABCDFGHIJKLMNOP|Forums: ABCD EF|Ecommerce: Swords Knives

  3. #3
    Chronic Entrepreneur
    Join Date
    Nov 2003
    Location
    Tulsa, Oklahoma, USA
    Posts
    1,112
    If you're worried that Google isn't crawling all of your site check out the tools at http://www.google.com/webmasters/ . You can use this to check for any pages on your site that Google is having trouble crawling.

  4. #4
    Registered Mike's Avatar
    Join Date
    May 2003
    Location
    UK
    Posts
    2,755
    Looks like things are ok then. I added a sitemap to google last week, and it's coming more regularly than normal. I just looked on the crawl error page and nothing was listed, so it seems like it'll just take a little bit of time.

    Cheers anyway guys.
    Don't you just love free internet games ?

  5. #5
    4x4
    Join Date
    Oct 2004
    Posts
    1,043
    Be careful if you set crawl rate to high!!

    Here's some stats from one of my sites where I enable dit:

    (proxy site - they spyder proxified pages people link to, you may remember my thread in general area where ppl are complaining my site is outranking theirs)
    - 350,000 pages and 10gb in Jan
    - 380,000 pagies and 13gb in Dec

  6. #6
    Registered Mike's Avatar
    Join Date
    May 2003
    Location
    UK
    Posts
    2,755
    Heh, I better bear that in mind!

    I shouldn't have a problem quite as big though, even though there are a lot of pages on my site, it doesn't have hundred of thousands just yet
    Don't you just love free internet games ?

  7. #7
    Web Software Creators
    Join Date
    Jun 2006
    Location
    Chennai India
    Posts
    23
    For constantly changing content, google automatically sets is crawlers to crawl those pages once or even twice a day.
    Chris, Director, Chrisranjana.com,
    php asp programmers, Our portfolio

  8. #8
    Site Contributor KLB's Avatar
    Join Date
    Feb 2006
    Location
    Saco Maine
    Posts
    1,181
    It is impossible for Google to index my site all at once. Instead it is more like a continuous crawl with Googlebot indexing a few thousand pages a day. Some sections seem to be crawled more often than others, but typically there is at least on complete deep crawl completed each month with core pages being indexed several times during the month.
    Ken Barbalace - EnvironmentalChemistry.com (Environmental Careers, Blog)
    InternetSAR.org: Volunteers Assisting Search and Rescue via the Internet
    My Firefox Theme Classic Compact: Based onFirefox's classic theme but uses much less window space

  9. #9
    Registered Mike's Avatar
    Join Date
    May 2003
    Location
    UK
    Posts
    2,755
    Well yesterday Googlebot apparently visited over a thousand pages, so hopefully it'll find the rest soon as well.
    Don't you just love free internet games ?

Similar Threads

  1. Does Googlebot hit your AWS Sites Hard?
    By pas in forum General Management Issues
    Replies: 10
    Last Post: 08-27-2005, 11:42 PM
  2. Googlebot crawling behavior
    By ASP-Hosting.ca in forum Search Engine Optimization
    Replies: 5
    Last Post: 03-29-2005, 07:11 PM
  3. Google refusing to visit a site
    By mobilebadboy in forum Search Engine Optimization
    Replies: 11
    Last Post: 10-01-2004, 01:48 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •