Robots.txt vs Rel=Nofollow vs Meta Robots Nofollow

Aug 6th

I was just fixing up our Robots.txt tutorial today, and figured that I should blog this as well. From Eric Enge's interview of Matt Cutts I created the following chart. Please note that Matt did not say they are more likely to ban you for using rel=nofollow, but they have on multiple occasions stated that they treat issues differently if they think it was an accident done by an ignorant person or a malicious attempt to spam their search engine by a known SEO (in language that is more rosy than what I just wrote).

Crawled by Googlebot?
Appears in Index?
Consumes PageRank
Risks? Waste?
robots.txt no If document is linked to, it may appear URL only, or with data from links or trusted third party data sources like the ODP yes

People can look at your robots.txt file to see what content you do not want indexed. Many new launches are discovered by people watching for changes in a robots.txt file.

Using wildcards incorrectly can be expensive!

robots meta noindex tag yes no yes, but can pass on much of its PageRank by linking to other pages

Links on a noindex page are still crawled by search spiders even if the page does not appear in the search results (unless they are used in conjunction with nofollow on that page).

Page using robots meta nofollow (1 row below) in conjunction with noindex do accumulate PageRank, but do not pass it on to other pages.

robots meta nofollow tag destination page only crawled if linked to from other documents destination page only appears if linked to from other documents no, PageRank not passed to destination If you are pushing significant PageRank into a page and do not allow PageRank to flow out from that page you may waste significant link equity.
link rel=nofollow destination page only crawled if linked to from other documents destination page only appears if linked to from other documents no, PageRank not passed to destination If you are doing something borderline spammy and are using nofollow on internal links to sculpt PageRank then you look more like an SEO and are more likely to be penalized by a Google engineer for "search spam"

If you want to download the chart as an image here you go http://www.seobook.com/images/robotstxtgrid.png

And The Winner Is...

Jul 31st

I decided to pick David Lubertazzi and Elisabeth Sowerbutts as the winners for their SEO Knol improvement comments.

I added a few pictures and fixed up some writing errors and incorporated a bunch of the feedback (like making the introduction better - thanks Andrew). There are many things (like domain names, duplicate content, blogging, social media, conversion, history and background of SEO) that I could have discussed, but I was unsure of how long I should let the Knol get, while still claiming that it was a basic introduction. Thanks for the feedback everyone!

What Are Your Favorite SEO Analogies?

Jul 30th

I try to teach my mom SEO stuff from time to time, and often do so through the use of analogies. Some analogies perhaps oversimplify the SEO process, but are good for helping get the basic concepts across.

On Page Content

  • fish and a fishing pole - when explaining how text heavy sites often outrank thin ecommerce sites, I like to call searchers fish and each word on the page an additional fishing pole in the water. This is really powerful when used in combination with analytics data, showing her the hundreds of phrases that people searched for to find a given page on her site...helping her see the long tail as schools of fish. :)
  • Don't Make Me Think - people scan more than they read. Large blocks of text are imposing. People are more likely to read well formatted content that uses many headings, subheadings, and inline links. Expect people to ignore your global navigation, and do whatever you ask them to do (via inline links).

Site Structure

  • Broadway Street in Manhattan - used to describe the value of descriptive .com domain names, and when describing what top search engine rankings are worth.
  • a pyramid - when explaining how some phrases are more competitive than others, and how to structure a site.
  • chapters of a book - used to describe the importance of focused page titles, and how to structure a website.

Link Reputation

  • search engines follow people - helps explain why new sites tend to not rank well, and how links are seen as votes.
  • roads and highways - used to describe PageRank and why some votes count more than others.
  • multiple audiences - used to describe why many types of content are needed to address different audiences, and the importance of creating content that is loved by buyers, linkers, and search engines.
  • rising tide lifts all boats - used to describe how links to one part of your website help other pages on your website rank better
  • pet rocks & overpriced dolls - describing how perception becomes reality when describing cumulative advantage, and how some poor quality sites are popular while better content remains hidden

Pay Per Click

  • instant market feedback - describing how it can be cheaper to test and learn than it is to theorize
  • taxing irrelevancy - explaining how irrelevant ads are priced out of the marketplace.
  • users vote with clicks - if your ad does not get clicked on it costs way more or is not shown

Tracking Results

  • flying blind without autopilot - when explaining the importance of analytics, and how most businesses that do not track results stand a good chance of failing.
  • people are lazy - describing the power of defaults and how a #1 ranking gets way more traffic than a number 5 or number 10 ranking.

Google Relevancy

  • Trust is Important - cares deeply for user experience, unless they are paid enough to think otherwise.
  • The House Advantage - when explaining why YouTube and Knol pages rank better than they deserve to.
  • Link Authority is Important - explaining why garbage general made for AdSense sites like eHow clog up the search results when higher quality information is hidden.
  • Informational Bias - when explaining that Google's business model relies on people clicking paid ads for commercial sites, and why Wikipedia ranks for everything.

How do you describe SEO to people who are not deep into the field?

Update: A few months ago Jaan Kanellis posted many analogies.

What Are Your Favorite Foreign SEO Information Sources

Jun 7th

I have worked with some large multi-national brands who had multi-lingual sites, but they typically hired us for English optimization, and never really asked for much more than general advice and strategies (internal link flow, subdomains vs unique domains, etc.) when it came to other languages and cultures. I noticed a few differences between Google.com & International Google results while traveling, but I still only analyzed stuff that was published in English.

What are the best informational sources for SEO in Japanese? SEO in Chinese? SEO in Spanish? SEO in your language or region? How do you feel SEO in your area differs from the SEO advice you read from those of us who operate in the English US marketplace? I also would love to publish a guest article for each language.

PBS MediaShift Covers SEO

May 1st

Mark Glaser recently queried me about improving the SEO of PBS's MediaShift. The tips and advice I gave him apply to most blogging and media websites. The piece was well balanced, with information from Poynter, and he mentioned Joost's great article on Newspaper SEO.

The Changing Face of Link Buying

Mar 27th

As Google clearly states (with their actions), bartering for links is fine as long as money is not part of the exchange (or if there is editorial discrimination and relevancy when it is). What is the guiding principal for bartering links? Massa said:

When do other people WANT to accept your link request, publish your article, run your press release or accept your submission?

When it does something for them.

Either it makes them money, saves them time, provides added value to their visitors or they believe it makes them look good or smart or benevolent to their visitors, their peers, their friends, their relatives, to the search engines, award sites or just about anyone that can make them a buck or stroke their ego.

So, the absolute best chance you have of getting that link is to cover as many of those bases as possible at the same time. When you can satisfy some need, want or desire of the webmaster, the visitor to the hosting site and it makes the search engine look smart, BINGO. You just hit the SERP buster hat trick!

If you are going to sell links, you can cloak them as being AdSense ads to lower your risk profile, because if it's Google its got to be Good! Or you can require the link purchase be wrapped in a guest article or some other format that does not look like a link purchase. :)

Update: Gab Goldenberg wrote this detailed post on real sneaky text link ad disguises.

Excessive Creativity

Mar 4th

The line between being clever and giving out too much information, can be seen here.

Makeshift Anchor Text... Link Title Attribute vs Image Alt Tags: Which is Better?

Feb 6th

One of the reasons I was so motivated to change the tagline of this site recently was because the new site design contained the site's logo as a background image. The logo link was a regular static link, but it had no anchor text, only a link title to describe the link. If you do not look at the source code, the link title attribute can seem like an image alt tag when you scroll over it, but to a search engine they do not look that same. A link title is not weighted anywhere near as aggressively as an image alt tag is.

The old link title on the header link for this site was search engine optimization book. While this site ranks #6 and #8 for that query in Google, neither of the ranking pages are the homepage (the tools page and sales letter rank). That shows that Google currently places negligible, if any, weight on link titles.

I have ranked other sites for more competitive queries based exclusively on internal links (without thousands of links from other sites, like those pointing at the SEO Book home page).

Does the image alt text carry more weight? In a word, yes. Here is now I proved that to myself through yet another site error. :)

One of my hobby sites has a fairly flat file structure, and some of the internal pages are somewhat linkworthy. The site was not marketed aggressively and the only sitewide link to the homepage was the logo, which I forgot to put an image alt tag on. Google ranked 2 pages on the site well for the core keyword, but neither of those pages were the homepage. I noticed the lacking image alt tag, fixed it, and within a week my homepage was outranking the other pages.

If the only link to your homepage is a logo check the source code to verify you are using descriptive image alt text.

Have a High PageRank Site? Link to Us for a Free Membership

Jan 28th

Would this be considered a paid link? Where is the line drawn?

Nice business model too. Are those arrows pointing at the ads legit? How about the request to support our sponsors?

Get a few hundred million in VC funding, buy some old domains. Fill the domain with user generated content. Place a thin layer of link schemes and please click ads on the top. Web 3.0...Wow. Exciting times.

Also, learn how to rent millions of links in a Google friendly way.

SearchGuild.com Domain Auction at Sedo

Jan 27th

The SearchGuild domain auction at Sedo just ended. $8,655 for a known brand, an old trusted site with references from sites like Wired.com, and 28,000+ inbound links. Even without getting the content that is still cheap. Some of my leading affiliates make that per year just from recommending my book, let alone all their other affiliate income from recommending other products and services.

If you had to try to build those same links Search Guild has in today's market they could cost $100,000 and/or a couple years to build. Even after the site changes ownership, most of the inbound links will stick too. In fact, if you move a domain to a new URL and ask people to redirect the links, over 90% of them will not.

Just like a strong domain name offers defensible traffic, so do those old links pointing at an old trusted site. If I had not already bought, ran, and closed Threadwatch, I might have considered buying SearchGuild, but I already have too many SEO sites to spend another $8,000 on a domain name that I would then have to develop. I hope they do something cool with it.

Pages






    Email Address
    Pick a Username
    Yes, please send me "7 Days to SEO Success" mini-course (a $57 value) for free.

    Learn More

    We value your privacy. We will not rent or sell your email address.