PDA

View Full Version : Getting deep crawled



chromate
11-12-2003, 05:15 AM
I'm being visited once a day or so by googlebot. Thing is, it only crawls the entrance page and has never been beyond that.

The site's kind of new, but it does have a temporary listing. I would have thought it would have been deep crawled by now considering the number of times googlebot's visited the site.

I'm not using any robots.txt file and my links are standard like:

http://www.mydomain.com/linkdir/

And I'm not using any meta tags.

I really want it to get deep crawled before the next update. Is there nothing I can do about it? Or are there any reasons beyond the robots.txt file that would prevent a deep crawl?

Mayet
11-12-2003, 05:41 AM
Personally I would metatag the description at least..I know keyword metas are not as useful as they once were but i still use them just in case. I do't find they detract from the pages crawl value.
As to the first bit I read about this the other day. Apparently google does crawl the deep links, and you can find them if you google them but overall it takes your site as a whole and assumes the deeplink is exactly that..a deep link and wont return them highly in the search results.
From what I read and I will try and find a link to the article it is better to create subdomains and not to link them back to the main site. Then google sees them as separate entities. This way you can then create recipricol links around the board and boost each individual subject rankings. This is also easier for dmoz submission as they do not like listing deep links and have come down rather hard on it lately pruning many deeplinks from the directory.

I also submit every page singular. I sit there and just use the back button..submit..back button submit till i have every page submitted. I know google says you don't have to but I still found that google was only returning the url's I actually submitted highly on the results

Its also well worth to read on how google works. How they actually update through the datacentres. "The google dance"
It doesn't take place monthly as it did it's supposed to be live updating.

Once again this is my humble opinion only and only from what I have found myself asking the same questions.

Chris
11-12-2003, 07:57 AM
Submitting subpages is definitely a good idea, one or two atleast, to get the ball rolling.

If your site hasn't been up very long though (two weeks) it wouldn't suprise me that Google has yet to deep crawl it.

michael_gersitz
11-12-2003, 08:37 AM
Give it at least a month.

Also be sure to build a sitemap, so once it does "deep crawl" it will list your whole site.

chromate
11-12-2003, 09:51 AM
I have submitted all my subpages once. I only have about 20 that I really want it to crawl. But still, googlebot never visits the other pages. Just the root page.

It's not really worth me building a site map because my structure is so simple really. The front page is like a complete site map in itself :)

I'll try submitting the "deeper" pages agian and see what happens.

chromate
11-13-2003, 03:03 AM
I'm pleased to say, my site finally got deep crawled last night :)

MarkB
11-13-2003, 03:56 AM
How can you tell? By stats, or by pages indexed on Google?

chromate
11-13-2003, 04:07 AM
By looking at the stats and seeing what pages googlebot has crawled.

chromate
11-13-2003, 02:22 PM
I just checked my stats again. GoogleBot's being going crazy crawling every possible page of my site. It's crawled 890 pages so far. Most of which I wasn't expecting would be crawled. More to the point, I didn't want them crawled because some of them have very minimal content. I just hope I don't get penalized for it! :eek:

michael_gersitz
11-13-2003, 04:05 PM
How much is minumal content?

chromate
11-13-2003, 04:33 PM
Couple of lines sometimes. I should have used a robots.txt file to exclude those ones. I didn't think google would follow their links due to the structure of the URL.

Oh well. I think it should be alright. Even if it's not, a quick email to them should sort it out.

michael_gersitz
11-13-2003, 05:03 PM
I don't know how my new font site will do, because all the content is similar on most of the font pages, except for like 4 lines that decribe the fonts...

Mayet
11-13-2003, 06:28 PM
Googles been acting weird I will say that.

Thanks for the font site. Thats great btw just what I was looking for.:D :D

Mike
11-14-2003, 09:31 AM
I submitted to google about a week ago, and googlebot hasen't been seen crawling around yet:(

chromate
11-14-2003, 10:52 AM
Do you mean you haven't been deep crawled or that you haven't been crawled at all? If the latter, try submitting again on a daily basis until you do get crawled. Wont do any harm.