Go Back   Website Publisher Forums > Website Promotion > Search Engine Optimization
Reload this Page Robots.txt Verification
Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Notices

Reply
Page 1 of 2 1 2
 
Thread Tools Rating: Thread Rating: 53 votes, 5.00 average.
Old 06-07-2007, 11:21 AM   #1
Todd W
4x4
 
Join Date: Oct 2004
Posts: 1,019
Robots.txt Verification
I want to prevent search engines from accessing mysite.com/rss/whatever
and
mysite.com/rss.php?blah=whatever

I believe this si the correct stuff for robots.txt but wanted to verify with everyone here to be 100% sure.

User-Agent: *
Disallow: /rss/
Disallow: /rss.php
Todd W is offline   Reply With Quote

Old 06-07-2007, 11:34 AM   #2
KLB
Site Contributor
 
KLB's Avatar
 
Join Date: Feb 2006
Location: Portland Maine
Posts: 1,185
Looks good to me.
__________________
Ken Barbalace - EnvironmentalChemistry.com (Environmental Careers, Blog)
InternetSAR.org: Volunteers Assisting Search and Rescue via the Internet
My Firefox Theme Classic Compact: Based onFirefox's classic theme but uses much less window space
KLB is offline   Reply With Quote
Old 06-07-2007, 12:28 PM   #3
MaxS
Registered
 
Join Date: Mar 2006
Posts: 354
Google's Sitemaps has a tool in the control panel that tests your robots.txt file for you.
__________________
Max
MaxS is offline   Reply With Quote
Old 06-07-2007, 04:15 PM   #4
agua
Senior Member
 
agua's Avatar
 
Join Date: Sep 2005
Location: Pottsville, NSW
Posts: 529
Do rss feeds count as duplicate content?
__________________
I Do Website Design - but I am here to learn all about publishing
agua is offline   Reply With Quote
Old 06-07-2007, 06:11 PM   #5
Todd W
4x4
 
Join Date: Oct 2004
Posts: 1,019
Quote:
Originally Posted by agua View Post
Do rss feeds count as duplicate content?
YES! Watch out.
Todd W is offline   Reply With Quote
Old 06-07-2007, 06:41 PM   #6
KLB
Site Contributor
 
KLB's Avatar
 
Join Date: Feb 2006
Location: Portland Maine
Posts: 1,185
Quote:
Originally Posted by ToddW View Post
YES! Watch out.
Especially if others use your RSS feeds on their sites. My advice is to only provide short descriptions in RSS feeds, not entire articles.
__________________
Ken Barbalace - EnvironmentalChemistry.com (Environmental Careers, Blog)
InternetSAR.org: Volunteers Assisting Search and Rescue via the Internet
My Firefox Theme Classic Compact: Based onFirefox's classic theme but uses much less window space
KLB is offline   Reply With Quote
Old 06-07-2007, 09:20 PM   #7
agua
Senior Member
 
agua's Avatar
 
Join Date: Sep 2005
Location: Pottsville, NSW
Posts: 529
Thanks - time to edit the robots.txt on a few sites
__________________
I Do Website Design - but I am here to learn all about publishing
agua is offline   Reply With Quote
Old 06-08-2007, 10:59 AM   #8
Xander
Registered
 
Xander's Avatar
 
Join Date: Oct 2004
Location: UK
Posts: 264
Quote:
Originally Posted by ToddW View Post
YES! Watch out.
Do you have a link where Google confirm that? I'm surprised if their bots can't tell the difference.
Xander is offline   Reply With Quote
Old 06-08-2007, 02:05 PM   #9
Todd W
4x4
 
Join Date: Oct 2004
Posts: 1,019
Quote:
Originally Posted by Xander View Post
Do you have a link where Google confirm that? I'm surprised if their bots can't tell the difference.
Content is content... google doesn't care if it's RSS, XML, CSV if it's the SAME content and is in duplicate places it hurts you.

More information about duplicate content and supplemental results can be found on google blog.
Todd W is offline   Reply With Quote
Old 06-09-2007, 05:53 PM   #10
agua
Senior Member
 
agua's Avatar
 
Join Date: Sep 2005
Location: Pottsville, NSW
Posts: 529
On this subject again, do you have any concrete proof that feeds are classed as duplicate content?

The reason I ask is because I checked the robots.txt on a few authority sites (seobook.com/robots.txt) and they didn't have them blocked. And also watched this wordpress SEO video from Michael Gray and didn't catch a mention of blocking feeds

Also if you blocked them - how would that work with Google Blog Search?
__________________
I Do Website Design - but I am here to learn all about publishing
agua is offline   Reply With Quote
Old 06-10-2007, 02:22 PM   #11
Xander
Registered
 
Xander's Avatar
 
Join Date: Oct 2004
Location: UK
Posts: 264
Quote:
Originally Posted by ToddW View Post
Content is content... google doesn't care if it's RSS, XML, CSV if it's the SAME content and is in duplicate places it hurts you.

More information about duplicate content and supplemental results can be found on google blog.

It depends on how you set it up, if done correctly it isn't counted, you just have to look at Google's blog itself.
Xander is offline   Reply With Quote
Old 06-10-2007, 06:51 PM   #12
ozgression
Trench Warfare
 
Join Date: May 2003
Location: Australia
Posts: 813
Quote:
Originally Posted by agua View Post
On this subject again, do you have any concrete proof that feeds are classed as duplicate content?
I do. Google was indexing my rss feed and my page with the same content on it was in the supplement index. Once I stopped the search engines from being able to index the feed (wordpress allows search engines to spider the feeds easily btw), the rss feed dissapeared from the search engines and my page came out of the supplement index.
__________________
ozgression is offline   Reply With Quote
Old 06-10-2007, 09:01 PM   #13
agua
Senior Member
 
agua's Avatar
 
Join Date: Sep 2005
Location: Pottsville, NSW
Posts: 529
Thanks ozgression
__________________
I Do Website Design - but I am here to learn all about publishing
agua is offline   Reply With Quote
Old 06-10-2007, 10:46 PM   #14
Todd W
4x4
 
Join Date: Oct 2004
Posts: 1,019
Quote:
Originally Posted by ozgression View Post
I do. Google was indexing my rss feed and my page with the same content on it was in the supplement index. Once I stopped the search engines from being able to index the feed (wordpress allows search engines to spider the feeds easily btw), the rss feed dissapeared from the search engines and my page came out of the supplement index.
Exactly.
Todd W is offline   Reply With Quote
Old 06-11-2007, 12:14 AM   #15
Xander
Registered
 
Xander's Avatar
 
Join Date: Oct 2004
Location: UK
Posts: 264
Quote:
Originally Posted by ozgression View Post
I do. Google was indexing my rss feed and my page with the same content on it was in the supplement index. Once I stopped the search engines from being able to index the feed (wordpress allows search engines to spider the feeds easily btw), the rss feed dissapeared from the search engines and my page came out of the supplement index.
Thanks for the info, I'm just surprised Googlebot is not smart enough to tell the difference when there is a clear enough difference.
Xander is offline   Reply With Quote
Reply
Page 1 of 2 1 2

Bookmarks

« Previous Thread | Next Thread »
Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Insecurity of robots.txt ? Nico Website Programming & Databases 8 04-24-2007 04:26 AM
robots.txt question webvista Website Programming & Databases 2 06-12-2006 01:30 AM
Robots.txt help please Blue Cat Buxton Search Engine Optimization 2 07-20-2005 06:57 AM
robots.txt delpino Search Engine Optimization 2 01-23-2004 01:13 AM
Is Google ignoring my robots.txt file? flyingpylon Search Engine Optimization 9 01-13-2004 08:47 AM



All times are GMT -7. The time now is 02:34 PM.

Contact Us - Website Publisher - Archive - Privacy Statement - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
Site Copyright © 2003-2006 Jalic Inc. All rights reserved.