Major Search Engine Robots
The follow list is of major search engine robots. You will find these listed in the web browser section of your site stats program if one of these search engines has visited your site.
| Search Engine | Crawler Name | Notes |
| Googlebot | There are 2 versions of Googlebot, one does a deep crawl monthly, the other is constantly crawling new content. | |
| AltaVista | Mercator, Scooter | AltaVista was recently bought by Overture |
| AllTheWeb (FAST) | FAST-WebCrawler | AllTheWeb was recently bought by Overture |
| Alexa | ia_archiver | Alexa isn't a search engine, they run an archive of the Internet (among other things) |
| Inktomi | Slurp | Inktomi's web search has been bought by Yahoo |
| Grub | grub | Grub is a distributed search engine crawler that was recently bought by LookSmart, you'll probably start seeing it more often. |
| Teoma | teoma_agent | Teoma is a search engine owned by Ask Jeeves. |
Site Categories
Recent Blog Posts
