webvista
06-11-2006, 10:43 AM
Chris said:
My robots.txt file is below, you may need to edit this either to add additional pages you want to be blocked, or to allow some of the pages I block.
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/images/
Disallow: /forums/mod/
Disallow: /forums/attachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printarticle.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/search.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disalow the individual php file makes sense when you are trying to ban robot from following the link, however, for individual folder, if you put the index.php file there with blank page, woud this aslso serve the purpose? The robot wouldn't be able to peek inside the folder anyway (there is no way it can find any file inside the folder like the ls command does. Or did I missing something?
My robots.txt file is below, you may need to edit this either to add additional pages you want to be blocked, or to allow some of the pages I block.
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/images/
Disallow: /forums/mod/
Disallow: /forums/attachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printarticle.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/search.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disalow the individual php file makes sense when you are trying to ban robot from following the link, however, for individual folder, if you put the index.php file there with blank page, woud this aslso serve the purpose? The robot wouldn't be able to peek inside the folder anyway (there is no way it can find any file inside the folder like the ls command does. Or did I missing something?