I was just fixing up our Robots.txt tutorial today, and figured that I should blog this as well. From Eric Enge's interview of Matt Cutts I created the following chart. Please note that Matt did not say they are more likely to ban you for using rel=nofollow, but they have on multiple occasions stated that they treat issues differently if they think it was an accident done by an ignorant person or a malicious attempt to spam their search engine by a known SEO (in language that is more rosy than what I just wrote).

	Crawled by Googlebot?	Appears in Index?	Consumes PageRank	Risks? Waste?
robots.txt	no	If document is linked to, it may appear URL only, or with data from links or trusted third party data sources like the ODP	yes	People can look at your robots.txt file to see what content you do not want indexed. Many new launches are discovered by people watching for changes in a robots.txt file. Using wildcards incorrectly can be expensive!
robots meta noindex tag	yes	no	yes, but can pass on much of its PageRank by linking to other pages	Links on a noindex page are still crawled by search spiders even if the page does not appear in the search results (unless they are used in conjunction with nofollow on that page). Page using robots meta nofollow (1 row below) in conjunction with noindex do accumulate PageRank, but do not pass it on to other pages.
robots meta nofollow tag	destination page only crawled if linked to from other documents	destination page only appears if linked to from other documents	no, PageRank not passed to destination	If you are pushing significant PageRank into a page and do not allow PageRank to flow out from that page you may waste significant link equity.
link rel=nofollow	destination page only crawled if linked to from other documents	destination page only appears if linked to from other documents	no, PageRank not passed to destination	If you are doing something borderline spammy and are using nofollow on internal links to sculpt PageRank then you look more like an SEO and are more likely to be penalized by a Google engineer for "search spam"

If you want to download the chart as an image here you go http://www.seobook.com/images/robotstxtgrid.png

Question: How long does it take to rank a website? How many hour of work do I have to do each day to compete and rank my website at the top of the search results?

Answer: I get this question almost every day, and it is one of my least favorite questions to answer. So I figured I would answer it as best I can once here, then point people to this page when I get asked again. To compete in competitive marketplaces you have to out-think the competition or invest more than they do. When you start from how little or how quickly you have the wrong mindset. Ask not what your search results can do for you, but what you can do for your search results. :)

Keep launching quality original content, keep working at brand building, keep making social connections, and watch the traction build. When you are new, can you predict what one idea is going to make the difference? For most people I don't think so. Some types of success are deliberate, but for most independent webmasters, I think they accidentally step into success by working hard, being ignored, and then watching something blossom that they did not realize the importance of when they first launched it.

After you have some success then you can engineer further success, but for many it starts out as an accident or a byproduct of constant motion.

How long does it take to rank?

If the search results are uncompetitive (use SEO for Firefox to survey the competitive landscape) you might be able to rank in a month even if your site is brand new.

Is your site brand new? If so, how old are the top ranked competing sites. If they are a number of years old then it is probably going to take at least a year to catch up unless they are bad at marketing and link building or you have a great marketing idea that will help you build many organic links. If the companies that are ranking are multi-billion dollar corporations then you can't outrank them with a one man website unless your site is integrated into the conversation of that marketplace and/or your site offers valuable tools and/or original linkworthy content.

If your site is brand new, you probably want to develop links over time in a fairly consistent manner. If you grow x links this month then you want to create x or more the next month. And that number sets the baseline for the following month. Months where you have no viral marketing ideas try to list your site in a few quality directories, join trade organizations, and get other clean links.

If you are sitting on an older site you may be able to grow links a bit more aggressively, and you may be able to get away with being a bit more aggressive with the anchor text you use in the inbound links.

In many regional search markets outside of the US the competition is much less fierce than it is in the US, and it is easy to rank for some fairly competitive keywords.

An exact match domain name may also provide ranking benefits in some search relevancy algorithms, which allows you to rank quicker without needing to build up as much link authority.

How many hours of work will it take?
I have ranked sites on 5 hours of work, and I have put hundreds of hours of work into sites that do not rank as well as I want them to.

• How hard are your competitors working? If you are unsure track their current link count and their link growth. Also look for signs of public relation and the quality level of their inbound links.
• How big is their head start?
• How much are they investing?
• Are there ideas they forgot to focus on?
• Is your brand more focused than their brand?
• How much risk are you willing to take?
• Are competitors weighed down by bureaucracy?
• Are you more passionate about your topic than the leading websites? Eventually people will discover that, especially if you are not afraid to market yourself.

Ask not what your search results can do for you, but what you can do for your search results. :)

Let me know if you have any questions about SEO or internet marketing stuff. I will try to reply to your comment right below it in less than a day, often within minutes, for as long as this thread is open.

Please ask do not ask for in depth site reviews or questions that would be applicable to just one website.

Update: thread closed... I have to start working on a big project. Thanks for the questions everyone.

Question: I am new to the market and have a product I am going to launch soon at a similar price-point to SEO Book. I am wondering if I should launch with an affiliate program, and if it will provide a substantial return prior to

Answer: I would say off the start don't enable an affiliate program until you have a better feel for the amount of effort required per purchaser and have brand awareness. I like launching without an affiliate program because it makes citations appear more natural. If you are launching a MLM hype program then a multi tiered affiliate program is a must, but if not the best option is to wait until you better learn your market and are well integrated into your market.

Why it is Best to Launch Without an Affiliate Program

• If you test your marketing first you are going to be able to achieve a higher customer value and attract better affiliates.
• If you are uncertain as to how to best sell your product and you have a new brand then it is going to be pretty tough for affiliates to sell it for you.
• The best marketing looks organic. With a new product it is best to get reviewed by top editorial channels and earn a few organic citations to help others see your product as more than just hype. If a product is primarily or only hyped by affiliates off the start that can undermine your brand.
• Some of my customers who did not sign up as affiliates have read my book and misquoted me to make me sound dumber than I am, and have misquoted to make me sound much smarter than I am. If others have profit incentive behind describing who you are or what you offer many will take the low road and use empty hype. Repositioning a brand after others have positioned it incorrectly is tougher than positioning it correctly off the start.
• Another advantage of waiting on establishing an affiliate program is that by the time you launch it you have built enough trust that people will be willing to sign up with you directly.

The Dangers of a Loose Affiliate Program

If you start off with an affiliate program I think your best best is to make the affiliate program invite only off the start to ensure your brand is not dragged through the mud before you were able to build brand equity. Some of my affiliates have done things like

• stealing content from my friends and wrapping it in banners for my site
• mass email spamming (while signing my name to it)
• a list of other dirty things I will not mention ;)

Because affiliates are hard to control, it is best to have a direct affiliate program rather than going through a third party website. This takes more time to manage, but enables you to

• immediately warn affiliates and/or remove the profit incentive if they are doing things that are illegal and/or undermine your brand value
• altert affiliates in advance of price changes, special promotions, or other changes to your offerings
• keep in contact with your affiliates, protecting you from drastic cost increases seen at third party affiliate management firms like MyAffiliateProgram
• more easily track historical trends
• create your own dirty affiliate sites without risking your main brand

Some affiliate software also tracks referrals, which enables you to learn the difference between what sorts of traffic streams and marketing sell and which do not.

Corporations have sued other corporations based on ad targeting. Even if the lawsuits are bogus they still eat up time and valuable resources. Affiliate control is useful for mitigating some such conflicts.

A few years back a well known SEO said that I was deplorable for bidding on a broad basket of keywords including their name. One of their affiliates was bidding on my name, amongst others. Once questioned about the hypocrisy, this person eventually admitted that they were too lazy to police their own affiliates, which showed their complaint had no validity. If their complaint had validity they would have better been able to claim it by hosting their own affiliate program and holding their own affiliates to the same expectations they try to hold others to.

Affiliate Software Options

• Some of the larger affiliate networks (like CJ) have a wide array of affiliates, but typically the best affiliates will hunt out the best affiliate programs wherever they are (instead of staying inside third party networks), so it might be just as easy to attract the best affiliates by ranking well for your keywords + affiliate program.
• Google offers cost per action based referral ads, but they don't have much inventory, and most of the best affiliates do not like using Javascript or sharing their business model with Google.
• Some of the smaller affiliate networks (like shareasale.com, as an example) have pitched me as a merchant for rates far below their published rates, but after seeing what happened with MyAffiliateProgram it does not make sense for me to pay a third party to place a roadblock between me and my affiliates.
• ClickBank does not cost much to set up, but I have found that I get an extraordinary return rate there when compared with Paypal. Paypal does not have affiliate features baked into it, but does work with many affiliate software programs.
• I use Idevaffiliate on Seo Book. Fully featured it costs $300.
• As ShoeMoney recently highlighted, BOTW has a custom affiliate program which even tracks direct links to any page on their site. Creating such a program would probably only cost about $1,000 at most. And that link for BOTW may have been an affiliate link, and Google has no way of knowing.

Question: I have recently launched a new site and got it a few authoritative links. For my search queries related to my brand Google is ranking an internal page instead of the home page. Why?

Answer: When a site is new and has few inbound links the PageRank computations tend to be rough estimations. It is common for a page linked to sitewide, like an about us page, to outrank the homepage until they better understand the internal site structure and PageRank flow.

What if My Site is Older?

Filtered Page

Some pages get filtered out of the search results for being too focused on a keyword or keyword phrase. Make sure you mix up your inbound anchor text, page title, meta description, page headings, and page content. If you see a women's volleyball page outranking your core volleyball page and your core volleyball page used to rank really well but is now nowhere to be found then likely it got filtered out.

Poor Internal Site Link Structure

If the wrong page style ranks and gets linked to then that advantage sorta builds on itself and the wrong pages continue to rank. If you have printer friendly pages or other duplicate content pages ranking you may want to look into demoting their role in your internal link structure.

Information & Sales Pages

Informational pages tend to outrank most sales / conversion oriented pages because typically they have more relevant on page content and deliver greater value to common web users, which makes people more likely to link to them. If you have an informational page ranking where you would like your sales page to rank you have numerous options.

Advertise your product or service aggressively on the information page. This allows you to maintain rankings, maintain forward link momentum, and increase conversion rates. This is a low risk approach. If the idea of advertising too overtly scares you then you may want to consider using smaller ads and working the advertisement into the content with a text link.

Switch the contents of the two URLs. This carries some risks, as some people who linked to the information page may remove their link if they see the advertisement page, though most old links stick. The other big risk is that competitors may eventually catch up because your ranking page may not gain links as quickly as their informational pages do, so the day you switch to an ad page you help them catch up.

Change internal link weights. If you are placing a lot of link weight on the informational page but little on the sales page you can try to increase your internal link weight to the sales page while lowering the link weight to the informational page.

Market both pages aggressively. It is typically better to have two listings in the search result than one. If one of your documents is a self reinforcing authority and your other document is within striking range of the top results you can market both of them. Ensure you make the descriptions and titles for each of them appealing to the right searchers. For example, the listing for the sales page can use commercial words like buy and best, whereas the informational page can use words like free, tips, reviews, etc.