When creating a robots.txt file, if you specify something for all bots (using a *) and then later specify something for a specific bot (like Googlebot) then search engines tend to ignore the broad rules and follow the ones you defined specifically for them. From Matt Cutts:
If there's a weak specification and a specific specification for Googlebot, we'll go with the one for Googlebot. If you include specific directions for Googlebot and also want Googlebot to obey the "generic" directives, you'd need to include allows/disallows from the generic section in the Googlebot section.
I believe most/all search engines interpret robots.txt this way--a more specific directive takes precedence over a weaker one.
You can learn a lot about how the search results will change based on recent changes that have been made and by seeing what tests the engines are running. As SEOs we track the algorithms quite intensively, but the search result display is just as important. Google allows webmasters to see what search tests they are currently performing via Google Experimental Search.
Google bought YouTube, but is struggling with ironing out ad revenue shares and advertising. What is the easiest way for Google to fix these issues? Integrate YouTube and Google Video directly into Google's search results.
Using what legal loopholes they may and something they call universal search, you can now listen to music videos directly from Google.com search results. This creates a marketplace that many businesses will need to be in to stay relevant, destroys a whole vertical of web spam, AND allows Google to monetize the organic search results (via YouTube). If you think video is a passing fad you bet wrong, but if you are doing it you are best off branding your videos to be associated with your domain name and uploading them to YouTube. Eli offered tips on how to make $1,000 a month re-purposing video, but now the number is more like $20,000.
Sure Google has done many YouTube users wrong, but if you need exposure, Google turned back the clock on SEO. Top ranking have never been easier. All you need to succeed is to format your content in video and upload it to YouTube.
If Google or Amazon were your bank or credit card, they'd let you know which merchants had the best prices for the same products, so you'd be a smarter shopper next time. They'd let merchants know what products were popular with people who also bought related products. They'd help merchants stock the right products by zip code. They'd let you know when you were spending more on dining out than you have set in your family budget. They'd let you know when you were approaching your credit limit, with a real-time fuel gauge, not just a "Sorry, your card has been declined."
By making search richer you have less reason to leave Google. Google started with targeted text ads, but it is even better if they can combine their targeting with trusted brands and offers while making their ads look like a useful piece of content in any format.
A friend recently launched a new site and promptly crafted a great linkbait award idea that got so many links that over 95% of the website's inbound links were reciprocal links. The award program worked so well that traditional PR firms used our list of award winners to seed their list of people they wanted to contact to talk about a client.
The site did not rank anywhere near as well as it should have because there were too many reciprocal links gained far too quickly when you consider the rest of the site's link profile.
One of the reasons that it is so important to mix link types is such that if any of your marketing really takes off you want some semblance of balance to your link profile.
DaveN offers his recovery plan for a recent Google algorithm which has affected the rankings of many sites engaged in buying and selling links.
Key points from Dave's post and comments:
DaveN focuses on the importance of building topical and trusted links before reaching into the outlier parts of the web. Older and more trusted sites can then loop back to buying lower quality links to shore up their rankings for important keywords.
Links from high quality trusted blogs are a more effective way to buy / build links than links that have obvious footprints associated with being bought in a group of other links.
Dave also noted that in the past people who bought links may have got hit, and link sellers might have got their outbound link passing ability blocked, but this is the first time Google actively lowered the rankings of link sellers.
If you are passionate, a site can have value without ranking, as rankings are a lagging indication of site quality, market timing, and/or marketing savvy. If you are offering something that is substantially similar to competing sites, it has virtually no value until it ranks at the top of the results. In the quest to build value, mindshare, and rankings it is easy to focus on unimportant things that eat time and provide little return. For example, you could write a 3000 page website that is the encyclopedia for your topic or you could try to create the ultimate branded property, but if nobody sees it then the content or brand it doesn't flourish. You need the site to look good enough to compete, but there is little value is trying to make it perfect right out of the gate.
Brand Developement and Market Leverage
While one is writing page after page or tweaking away building a perfect new site, the competitors are leveraging Indian copywriters who write thin informational pieces wrapped in AdSense. Those same low quality sites garner self reinforcing links because they are already ranking, and most people are lazy, just linking to whatever they can easily find.
The results of any tests to monetize a low traffic site are going to provide inadequate and inconclusive results, which also likely feed into your biases and expected outcomes. If you build authority first and then come back and test later you will receive a greater ROI for the amount of effort required to perform the tests.
To put into perspective the testing errors that small samples can create, a friend of mine has a site which makes virtually the same amount from AdSense every day. The same site sells leads. Some days it generates 6 conversions and other days it does 21, all while the traffic flow and AdSense earnings are fairly constant. If you compared one revenue stream to the other, the obvious winner would look different based on what day you chose.
Everything on the Web is Broken
If you try to look really polished that might not be remarkable. You are not cutting edge if you have to be perfect before you are willing to be seen. If I wasn't willing to release my first ebook prior to when I should have you probably would not be reading these words right now.
Everything on the Web is Biased
I believe people have more of a tendency to talk about and share things that are unpolished. Google gets talked about by getting sued, Digg gets talked about by getting gamed, Fox news gets talked about by entertainment sold as news, etc etc etc.
When you try to come out of the gate perfect, it is hard to relate to your end audience without spending thousands of dollars on marketing. It is far more remarkable to come out of the gate slightly broken and biased and appeal to the overt biases of those who can give you authority. I am not suggesting to be racist or sexist or anything like that, but people are generally more receptive to (and thus likely to share) things that reinforce their worldview. Appeal to a known bias, market that story, then create another story that works another group. Do it over and over until you have enough authority to clean up the site and become the market leader.
Rough edges appealing to deep niches is a far better approach to marketing than broad and polished to a fine dull.
Get authority by appealing to smaller groups of your audience, grab marketshare, THEN try to look authoritative. Most people don't know HOW you acquired your authority...it is not something most think to question, and if they do you can always change your look and feel as needed to accommodate the market.
You don't have to do anything deceptive to gain authority, but if you think perfect content is the answer you are only deceiving yourself.
Many publishers hide additional information sections that they want people to be able to select viewing if they show interest in the topic. For example, each of Think Progress's navigational sections are expandable, and some publishers have more information or other informational cues to make additional page content visible. These can be used deceptively, but if you have a strong brand and are trying to use them with the end user in mind, I doubt search engines will think the intent is bad.
AdSense Section Targeting:
As search has taken a larger and larger piece of the web search engines have given us ways to mark up our pages to suit their needs. AdSense section targeting made it easier for Google to target content ads to your site. That sounds like a good idea, but they also offer tags that offer publishers no value.
Nofollow was originally recommended to stop blog comment spam, but it has morphed into a tag that Matt Cutts wants you to use on any paid or unnatural link. What makes a link unnatural? In one form or another almost everything is paid for, by giving away value, exchanging currency, or nepotism.
Do You Trust Yourself?
If a page has many nofollow tags on it isn't that another way of saying that the publisher does not trust their own content? If a publisher says that they don't trust their own content or their own advertisers then why would search engines (or savvy webmasters) want to trust them?
Mr. Cutts, speaking on behalf of Google presumably, made the comment, "if you want to buy links just for traffic, totally fine just donâ€™t do it so they affect search engines".
This concept is completely flawed. This self serving philosophy is also at the very core of the problem. When the machine attempts to modify the behavior of people to satisfy itâ€™s own ends, the machine is broken. What people do should not be seen as affecting the search engine. What people do should be the very reason for the engine to exist in the first place. If the search engine is being affected by the actions of people, is any logical person going to honestly assume that it is the people that are broken? That is exactly what is happening here.
Yahoo!'s Robots-Nocontent Attribute:
Search engines have got better at identifying duplicate content. Some search engines may boilerplate strip obvious navigational elements from pages. Some may place pages with too much duplicate content in supplemental results. Some may sites with too much duplicate content in reduced crawling status.
There are all of these ways to fight off content duplication and Yahoo! offers a robots-nocontent tag. One of the first people to comment on the news was Google's Matt Cutts, who said:
Danny, can you ask how Yahoo intends to treat links in the "robots-nocontent" section?
Don't Use the Robots-Nocontent Attribute:
It might be easy to add class="robots-nocontent" to some of your divs, but should you? I think it has little value. Sure you could use it in a sneaky way, as suggested by Jay Westerdal, but the problems with that are:
it looks sneaky
you are removing content from your pages (and will thus rank for fewer phrases)
there are easier and more effective ways of changing the meaning of a page without looking so sneaky...like just rewriting an article, adding a spammy comment that looks like it came from a third party, or adding a few additional words here or there.
Yahoo! is the top network of sites on the web. Internally they have publishing teams and an SEO team. If their search engineers can't figure out how to use their own internal traffic stats and other relevancy measurements to refine their duplicate detection algorithms they deserve to bleed marketshare until they no longer have relevancy in the marketplace.
How to Change the Focus of a Page Without Using Robots-Nocontent:
If you want to change the focus of your pages here are some of the best ways to do it
Ensure your page title and meta description are unique. Do not place the same words at the start of every page title on all the pages of a new website.
Make your h1 headings and subheadings target a slightly different word set than your page title.
If your page is thin on content, add more additional relevant unique content to the page. The solution to not getting killed by duplicate content filters is adding more unique content, not stripping out obvious required duplication (such as navigation and advertisements) that search engines should be able to figure out.
If your site has comments or consumer feedback you can post or encourage feedback that targets other keywords. Comments offer free text. A 500 word page with an additional 1,000 words in the comment section may rank for 2 or 3 times as many search queries. Don't throw away the free content.
For those who are really aggressive and have crusty links that will never be removed, consider placing your commercial messages on one of your highly trusted high ranking pages. People buy and sell websites, who is to say that the contents of a URL can't change?
Forbes recently wrote an article about Google's supplemental results, painting it as webpage hell. The article states that pages in Google's Supplemental index is trusted less than pages in the regular index:
Google's programmers appear to have created the supplemental index with the best intentions. It's designed to lighten the workload of Google's "spider," the algorithm that constantly combs and categorizes the Web's pages. Google uses the index as a holding pen for pages it deems to be of low quality or designed to appear artificially high in search results.
I have worked on some of the largest sites and network of sites on the web (hundreds of millions+ pages). When looking for duplicate content or information architecture related issues, the search engines do not allow you to view deep enough to see all indexing problems, so one of the first things I do is use this search to find low quality pages (ie: things that suck PageRank and do not add much unique content to their site). After you find some of the major issues you can dig deeper by filtering out some of the core issues that showed up on your first supplemental searches. For example, here are threadwatch.org supplemental results that do not contain the word node in the URL.
If you have duplicate content issues, at best you are splitting your PageRank, but you might also affect your crawl priorities. If Google thinks 90% of a site is garbage (or not worth trusting much) I am willing to bet that they also trust anything else on that domain a bit less than they otherwise would, and are more restrictive with their willingness to crawl the rest of the site. As noted in Wasting Link Authority on Ineffective Internal Link Structure, ShoeMoney increased his search traffic 1400% after blocking some of his supplemental pages.
I once saw a college professor cite a page about caffiene on a low quality site about pornography, gambling, and drugs on his official profile page. Many people never look beyond the page when linking to a story.
This is not to say that one should put a story on a bad website, but that one should make the story page they are currently marketing as clean as possible so it is easy to link at. And you are probably better off placing your marketing stories on your key site if you think they will still spread.
Over time people will become more aware of using content bait on a crappy site, but for now most people don't look beyond the page when referencing a story.