Hey all,
How do spiders recognize that they are still on the same website, for example, for big websites have alot of servers/ip's who host articles on many of them.
Do the crawlers only look at the URL to see if they are still on the same webpage ?
Regards
Bookmarks