Its usually only unscrupulous robots/spiders that don't follow robots.txt.