The Ultimate vBulletin Optimization Guide
Make Use of Robots.txtAnother thing you need to do specifically for search engines is make judicious use of a robots.txt file. Forums are full of pages that, while extremely important for functionality, are of no importance to search engines. Pages such as member lists, profiles, posting pages, search pages, private messaging pages. These things a search engine does not need to see, and also by blocking access to pages such as profiles and member lists you'll take a reward away from forum registration spammers…. You know those Russian guys who register with a name like "!! Barney !!" and don't post but put a porn link as their homepage? Every forum has these losers and while by blocking search engine access to these pages you'll be making sure they don't get an incoming link out of their spam.
My robots.txt file is below, you may need to edit this either to add additional pages you want to be blocked, or to allow some of the pages I block.
User-agent: * Disallow: /forums/admin/ Disallow: /forums/images/ Disallow: /forums/mod/ Disallow: /forums/attachment.php Disallow: /forums/avatar.php Disallow: /forums/editpost.php Disallow: /forums/member.php Disallow: /forums/member2.php Disallow: /forums/memberlist.php Disallow: /forums/misc.php Disallow: /forums/moderator.php Disallow: /forums/newreply.php Disallow: /forums/newthread.php Disallow: /forums/online.php Disallow: /forums/poll.php Disallow: /forums/postings.php Disallow: /forums/printarticle.php Disallow: /forums/printthread.php Disallow: /forums/private.php Disallow: /forums/private2.php Disallow: /forums/register.php Disallow: /forums/report.php Disallow: /forums/search.php Disallow: /forums/sendtofriend.php Disallow: /forums/threadrate.php Disallow: /forums/usercp.php
You could also accomplish this same thing by editing your templates and inserting the new rel="nofollow" attribute to all links you do not want crawled. This would be time consuming though (but perhaps one day the forum software makers will allow you to turn it on as an option).
Making URLs Search Engine FriendlyI'm sure a few of you are confused and wondering why I have not yet mentioned installing a friendly URL hack for the forum. Well I wrote what is probably the most popular article on the Internet for search engine friendly URLs, so believe me, I'm generally a fan of them, and if forums used them by default I'd be glad, but they don't and I'm against installing a hack to change that.
The reason is that the URL structure is such a fundamental component of forum software that when you change it you're changing a lot. Likely every time you need to install a security update that hack will need to be removed and reapplied. Also in the future the hack may stop being supported, or you may decide to stop using it after an upgrade. At this point you'll run into URL integrity issues. All your incoming links would now be broken since they used the old search engine friendly URLs, and losing all your income links is about as calamitous as having a hard drive failure. Plus, as I said above, search engines generally don't have a problem with query strings anymore. So, in my mind, its simply not worth the risk and potential problems to heavily hack a forum so much, when you really are gaining only a minor benefit by doing so.
Show it All to the SpidersThe final thing you need to do is make sure all your forums and subforums are set to show all threads by default, rather than just threads for the past month, week, or whatever. The reason is that search engines will use whatever the default is so if you do not show them older threads you're going to be missing out on the traffic those older threads provide. This is also helpful when your forum is first growing and you may not be getting many new threads on a daily basis. Without it set to "show all" visitors will see less threads, read less threads, and respond to less threads, thus hampering forum growth. Registered users can always set this to their own preference, and that's fine, you just need to make sure that the default is "show all."
Site Categories
Recent Blog Posts
