I was just fixing up our Robots.txt tutorial today, and figured that I should blog this as well. From Eric Enge's interview of Matt Cutts I created the following chart. Please note that Matt did not say they are more likely to ban you for using rel=nofollow, but they have on multiple occasions stated that they treat issues differently if they think it was an accident done by an ignorant person or a malicious attempt to spam their search engine by a known SEO (in language that is more rosy than what I just wrote).
Crawled by Googlebot?
Appears in Index?
|robots.txt||no||If document is linked to, it may appear URL only, or with data from links or trusted third party data sources like the ODP||yes||
People can look at your robots.txt file to see what content you do not want indexed. Many new launches are discovered by people watching for changes in a robots.txt file.
Using wildcards incorrectly can be expensive!
|robots meta noindex tag||yes||no||yes, but can pass on much of its PageRank by linking to other pages||
Links on a noindex page are still crawled by search spiders even if the page does not appear in the search results (unless they are used in conjunction with nofollow on that page).
Page using robots meta nofollow (1 row below) in conjunction with noindex do accumulate PageRank, but do not pass it on to other pages.
|robots meta nofollow tag||destination page only crawled if linked to from other documents||destination page only appears if linked to from other documents||no, PageRank not passed to destination||If you are pushing significant PageRank into a page and do not allow PageRank to flow out from that page you may waste significant link equity.|
|link rel=nofollow||destination page only crawled if linked to from other documents||destination page only appears if linked to from other documents||no, PageRank not passed to destination||If you are doing something borderline spammy and are using nofollow on internal links to sculpt PageRank then you look more like an SEO and are more likely to be penalized by a Google engineer for "search spam"|
If you want to download the chart as an image here you go http://www.seobook.com/images/robotstxtgrid.png
Gain a Competitive Advantage Today
Your top competitors have been investing into their marketing strategy for years.
Now you can know exactly where they rank, pick off their best keywords, and track new opportunities as they emerge.
Explore the ranking profile of your competitors in Google and Bing today using SEMrush.
Enter a competing URL below to quickly gain access to their organic & paid search performance history - for free.
See where they rank & beat them!
- Comprehensive competitive data: research performance across organic search, AdWords, Bing ads, video, display ads, and more.
- Compare Across Channels: use someone's AdWords strategy to drive your SEO growth, or use their SEO strategy to invest in paid search.
- Global footprint: Tracks Google results for 120+ million keywords in many languages across 28 markets
- Historical data: since 2009, before Panda and Penguin existed, so you can look for historical penalties and other potential ranking issues.
- Risk-free: Free trial & low price.