I use IP blocking to deny access to a page after a certain # of views.
Works good, but it's also blocking google and only indexing a small portion of my pages.
Does anyone have a list of google IPs that the spyder uses or a way to prevent this?
I use IP blocking to deny access to a page after a certain # of views.
Works good, but it's also blocking google and only indexing a small portion of my pages.
Does anyone have a list of google IPs that the spyder uses or a way to prevent this?
The easiest (and safest) way would be to create a whitelist for search engine spider user-agents.
If you can only do it by IP you can gets a list from here: http://www.iplists.com/
________
Vaporizer review
Last edited by rpanella; 03-17-2011 at 11:07 AM.
I could do it by user agent I guess, and check for search engines.
Bookmarks