Results 1 to 9 of 9

Thread: Spiders and websites....

  1. #1
    (future) SEOmaster
    Join Date
    Apr 2006
    Posts
    3

    Spiders and websites....

    Hey all,

    How do spiders recognize that they are still on the same website, for example, for big websites have alot of servers/ip's who host articles on many of them.

    Do the crawlers only look at the URL to see if they are still on the same webpage ?

    Regards

  2. #2
    Registered
    Join Date
    Feb 2005
    Posts
    45
    I would think so yes, cause one ip can have many different domains aswell, so the domain name is what matters.

  3. #3
    Administrator Chris's Avatar
    Join Date
    Feb 2003
    Location
    East Lansing, MI USA
    Posts
    7,055
    For the most part though a crawler doesn't care if it is on the same site or not. It is judging pages, not sites.
    Chris Beasley - My Guide to Building a Successful Website[size=1]
    Content Sites: ABCDFGHIJKLMNOP|Forums: ABCD EF|Ecommerce: Swords Knives

  4. #4
    (future) SEOmaster
    Join Date
    Apr 2006
    Posts
    3
    Would it be possible to fake a renoun website (ie cnn.com) write an article about a website I want to promote (with links of course) and when the crawler sees that page, a custom httpd displays a fake URL confirming that the article is on CNN.com for example.

    So we would be able to have links from any website as long as you spoof a page URL.

    Regards

  5. #5
    Working. Masetek's Avatar
    Join Date
    Aug 2005
    Location
    Aust
    Posts
    543
    That sort of behavior will eventually lead in your site getting banned

  6. #6
    Administrator Chris's Avatar
    Join Date
    Feb 2003
    Location
    East Lansing, MI USA
    Posts
    7,055
    That doesn't work, you cannot change where domains resolve to.
    Chris Beasley - My Guide to Building a Successful Website[size=1]
    Content Sites: ABCDFGHIJKLMNOP|Forums: ABCD EF|Ecommerce: Swords Knives

  7. #7
    (future) SEOmaster
    Join Date
    Apr 2006
    Posts
    3
    Would a spider keep crawling a website after a page that would have reloaded automaticly ?

  8. #8
    Site Contributor KLB's Avatar
    Join Date
    Feb 2006
    Location
    Saco Maine
    Posts
    1,181
    Automatic page reloading normally relies on client side scripting like JavaScript since bots don't run these types of server side scripting the pages would not reload for them.
    Ken Barbalace - EnvironmentalChemistry.com (Environmental Careers, Blog)
    InternetSAR.org: Volunteers Assisting Search and Rescue via the Internet
    My Firefox Theme Classic Compact: Based onFirefox's classic theme but uses much less window space

  9. #9
    Registered
    Join Date
    Nov 2003
    Posts
    215
    That is something that I wouldn't consider. It is likely to get your site banned and may even damage the site that you are faking and you are then likely to get into trouble.

    Quote Originally Posted by mrVJ
    Would it be possible to fake a renoun website (ie cnn.com) write an article about a website I want to promote (with links of course) and when the crawler sees that page, a custom httpd displays a fake URL confirming that the article is on CNN.com for example.

    So we would be able to have links from any website as long as you spoof a page URL.

    Regards
    Affiliate Programs Directory - Over 2,000 Programs - Contextual Ads, Datafeeds, 2-tier, plus more
    -----> 140+ CPA Affiliate Networks | Earn upto $1.20 CPM on Banners | Play Online Games

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •