Google and Bing admitted publicly to having ‘exception lists’ for sites that were hit by algorithms that should not have been hit. Matt Cutts explained that there is no global whitelist but for some algorithms that have a negative impact on a site in Google’s search results, Google may make an exception for individual sites.
The idea that "sites rank where they deserve, with the exception of spammers" has long been pushed to help indemnify Google from potential anti-competitive behavior. Google's marketing has further leveraged the phrase "unique democratic nature of the web" to highlight how PageRank originally worked.
But why don't we conduct a thought experiment for the purpose of thinking through the differences between how Google behaves and how Google doesn't want to be perceived as behaving.
Let's cover the negative view first. The negative view is that either Google has a competing product or a Google engineer dislikes you and goes out of his way to torch your stuff simply because you are you and he dislikes you & is holding onto a grudge. Given Google's current monopoly-level marketshare in most countries, such would be seen as unacceptable if Google was just picking winners and losers based on their business interests.
The positive view is that "the algorithm handles almost everything, except some edge cases of spam." Let's break down that positive view a bit.
Off the start, consider that Google engineers write the algorithms with set goals and objectives in mind.
Google only launched universal search after Google bought Youtube. Coincidence? Not likely. If Google had rolled out universal search before buying Youtube then they likely would have increased the price of Youtube by 30% to 50%.
Likewise, Google trains some of their algorithms with human raters. Google seeds certain questions & desired goals in the minds of raters & then uses their input to help craft an algorithm that matches their goals. (This is like me telling you I can't say the number 3, but I can ask you to add 1 and 2 then repeat whatever you say :D)
At some point Google rolls out a brand-filter (or other arbitrary algorithm) which allows certain favored sites to rank based on criteria that other sites simply can not match. It allows some sites to rank with junk doorway pages while demoting other websites.
To try to compete with that, some sites are forced to either live in obscurity & consistently shed marketshare in their market, or be aggressive and operate outside the guidelines (at least in spirit, if not in a technical basis).
If the site operates outside the guidelines there is potential that they can go unpenalized, get a short-term slap on the wrist, or get a long-term hand issued penalty that can literally last for up to 3 years!
Now here is where it gets interesting...
Google can roll out an automated algorithm that is overly punitive and has a significant number of false positives.
Then Google can follow up by allowing nepotistic businesses & those that fit certain criteria to quickly rank again via whitelisting.
Sites which might be doing the same things as the whitelisted sites might be crushed for doing the exact same thing & upon review get a cold shoulder.
You can see that even though it is claimed "TheAlgorithm" handles almost everything, they can easily interject their personal biases to decide who ranks and who does not. "TheAlgorithm" is first and foremost a legal shield. Beyond that it is a marketing tool. Relevancy is likely third in line in terms of importance (how else could one explain the content farm issue getting so out of hand for so many years before Google did something about it).
When Google did the Panda update they highlighted that not only did some "low quality" sites get hammered, but that some "high quality" sites got a boost. Matt Cutts said: "we actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side."
Here is the problem with that sort of classification system: doorway pages.
The following Ikea page was ranking page 1 in the search results for a fairly competitive keyword.
Once you strip away the site's navigation there are literally only 20 words on that page. And the main body area "content" for that page is a link to a bizarre, confusing, and poor-functioning flash tour which takes a while to load.
If you were trying to design the worst possible user experience & wanted to push the "minimum viable product" page into the search results then you really couldn't possibly do much worse that that Ikea page is (at least not without delivering malware and such).
I am not accusing Ikea of doing anything spammy. They just have terrible usability on that page. Their backlinks to that page are few in number & look just about as organic as they could possibly come. But not that long ago companies like JC Penny and Overstock were demoted by Google for building targeted deep links (that they needed in order to rank, but were allegedly harming search relevancy & Google user experience). Less than a month later Google arbitrarily changed their algorithm to where other branded sites simply didn't need many (or in some cases any) deep links to get in the game, even if their pages were pure crap.
We are told the recent "content farm" update was to demote low quality content. If that is the case, then how does a skeleton of a page like that rank so high? How did that Ikea page go from ranking on the third page of Google's results to the first one? I think Google's classifier is flashing a new set of exploits for those who know what to look for.
A basic tip? If you see Google ranking an information-less page like that on a site you own, that might be a green light to see how far you can run with it. Give GoogleBot the "quality content" it seeks. Opportunity abound!
You can't learn great SEO from an e-book. Or buying software tools.
Great SEO is built on an understanding.
Reducing SEO To Prescription
One of the problems with reductive, prescribed SEO approaches - i.e. step one: research keywords, step two: put keyword in title etc can be seen in the recent "Content Farm" update.
When Google decide sites are affecting their search quality, they look for a definable, repeated footprint made by the sites they deem to be undesirable. They then design algorithms that flag and punish the sites that use such a footprint.
This is why a lot of legitimate sites get taken out in updates. A collection of sites may not look, to a human, like problem sites, but the algo sees them as being the same thing, because their technical footprint is the same. For instance, a website with a high number of 250-word pages is an example of a footprint. Not necessarily an undesirable one, but a footprint nevertheless. Similar footprints exist amongst ecommerce sites heavy in sitewide templating but light on content unique to the page.
Copying successful sites is a great way to learn, but can also be a trap. If you share a similar footprint, having followed the same SEO prescription, you may go down with them if Google decides their approach is no longer flavor of the month.
The Myth Of White Hat
A lot of sites that get taken out are white hat i.e. sites that follow Google's webmaster guidelines.
It's a reasonably safe approach, but if you understand SEO, you'll soon realize that following a white hat prescription offers no guarantees of ranking, nor does it offer any guarantees you won't be taken out.
The primary reason there aren't any guarantees comes down to numbers. Google knows that when it makes a change, many sites will lose. They also know that many sites will win i.e. replace the sites that lost. If your site drops out, Google aren't bothered. There will be plenty of other sites to take your place. Google are only concerned that their users perceive the search results to be of sufficient quality.
The exception is if your site really is a one-of-a-kind. The kind of site that would embarrass Google if users couldn't find it. BMW, for example, in response to the query "BMW".
It's not fair, but we understand that's just how life is.
For those readers new to SEO, in order to really grasp SEO, you need to see things from the search engines point of view.
Firstly, understand the search engines business case. The search engine can only make money if advertisers pay for search traffic. If it were too easy for those sites who are likely to use PPC to rank highly in the natural results, then the search engines business model is undermined. Therefore, it is in the search engines interest to "encourage" purely commercial entities to use PPC, not SEO. One way they do this is to make the natural results volatile and unpredictable. There are exceptions, covered in my second point.
Secondly, search engines must provide sufficient information quality to their users. This is an SEO opportunity, because without webmasters producing free-to-crawl, quality content, there can be no search engine business model. The search engines must nurture this ecosystem.
If you provide genuine utility to end users, the search engines have a vested interest in your survival, perhaps not as an individual, but certainly as a group i.e. "quality web publishers". Traffic is the lifeblood of the web, and if quality web publishers aren't fed traffic, they die. The problem, for webmasters, is that the search engines don't care about any one "quality publisher", as there are plenty of quality publishers. The exception is if you're the type of quality publisher who has a well recognized brand, and would therefore give the impression to users that Google was useless if you didn't appear.
Thirdly, for all their cryptic black box genius, search engines aren't all that sophisticated. Yes, the people who run them are brilliant. The problems they solve are very difficult. They have built what, only decades ago, would have been considered magic. But, at the end of the day, it's just a bit of maths trying to figure out a set of signals. If you can work out what that set of signals are, the maths will - unblinkingly - reward you. It is often said that in the search engine wars, the black hats will be the last SEOs standing.
Fourthly, the search engines don't really like you. They identified you as a business risk in their statement to investors. You can, potentially, make them look bad. You can undermine their business case. You may compete with their own channels for traffic. They tolerate you because they need publishers making their stuff easy to crawl, and not locking their content away behind paywalls. Just don't expect a Christmas card.
SEO Strategy Built On Understanding
Develop strategies based on how a search engine sees the world.
For example, if you're a known brand, your approach will be different to a little known, generic publisher. There isn't really much risk you won't appear, as you could embarrass Google if users can't find you. This is the reason BMW were reinstated so quickly after falling foul of Google's guidelines, but the same doesn't necessarily apply to lesser known publishers.
If you like puzzles, then testing the algorithms can give you an unfair advantage. It's a lot harder than it used to be, but where there is difficulty, there is a barrier to entry to those who come later. Avoid listening to SEO echo chambers where advice may be well-meaning, but isn't based on rigorous testing.
If you're a publisher, not much into SEO wizardry, and you create content that is very similar to content created by others, you should focus on differentiation. If there are 100's of publishers just like you, then Google doesn't care if you disappear. Google do need to find a way to reward quality, especially in niches that aren't well covered. Be better than the rest, but if you're not, slice your niche finer and finer, until you're the top dog in your niche. You should focus on building brand, so you can own a search stream. For example, this site owns the search stream "SEO Book", a stream Aaron created and built up.
Remember, search engines don't care about you, unless there's something in it for them.
Google tries to wrestle back index update naming from the pundits, naming the update "Panda". Named after one of their engineers, apparently.
The official Google line - and I'm paraphrasing here - is this:
Trust us. We're putting the bad guys on one side, and the good guys on the other
I like how Wired didn't let them off the hook.
Wired.com: Some people say you should be transparent, to prove that you aren’t making those algorithms to help your advertisers, something I know that you will deny.
Singhal: I can say categorically that money does not impact our decisions.
Wired.com: But people want the proof.
This answer, from Matt Cutts, was interesting:
Cutts: If someone has a specific question about, for example, why a site dropped, I think it’s fair and justifiable and defensible to tell them why that site dropped. But for example, our most recent algorithm does contain signals that can be gamed. If that one were 100 percent transparent, the bad guys would know how to optimize their way back into the rankings
Why Not Just Tell Us What You Want, Already!
Blekko makes a big deal about being transparent and open, but Google have always been secretive. After all, if Google want us to produce quality documents their users like and trust, then why not just tell us exactly what a quality document their users like and trust looks like?
Trouble is, Google's algorithmns clearly aren't that bulletproof, as Google admit they can still be gamed, hence the secrecy. Matt says he would like to think there would be a time they could open source the algorithms, but it's clear that time isn't now.
Do We Know Anything New?
So, what are we to conclude?
Google can be gamed. We kinda knew that....
Google still aren't telling us much. No change there....
Google have filed a patent that sounds very similar to what Demand Media does i.e looks for serp areas that are under-served by content, and prompts writers to write for it.
The patent basically covers a system for identifying search queries which have low quality content and then asking either publishers or the people searching for that topic to create some better content themselves. The system takes into account the volume of searches when looking at the quality of the content so for bigger keywords the content would need to be better in order for Google to not need to suggest somebody else writes something
If Google do implement technology based on this patent, then it would appear they aren't down on the "Content Farm" model. They may even integrate it themselves.
How To Avoid Getting Labelled A Content Farmer
The question remains: how do you prevent being labelled as a low-quality publisher, especially when sites like eHow remain untouched, yet Cult Of Mac gets taken out? Note: Cult Of Mac appears to have been reinstated, but one wonders if that was the result of the media attention, or an algo tweak.
Google want content their users find useful. As always, they're cagey about what "useful" means, so those who want to publish content, and want to rank well, but do not want be confused with a content farm, are left to guess. And do a little reverse-engineering.
BeatThatQuote.com today was sold to Google for GBP37.7 million. We think this deal is a tremendous opportunity for our company to develop new and innovative options for personal finance in the UK. Our team is excited about becoming a part of Google. We look forward to working with their engineers to create new tools making it easier for consumers to choose the right financial products. We think
Remember how Overstock.com was recently penalized for offering discounts in exchange for links? BeatThatQuote partnered with Oxfam to create CompareForGood.com. The homepage consists of a bunch of links into BeatThatQuote.com. If you look at those links using our server header checker you will see some 301 redirects. Of course doorway pages are considered spam & we know that Google has torched some other affiliate programs for using 301 redirects.
With that in mind, can anyone explain why Google's newest purchase buying links like
Not so much a categorized listing with an editorial review...just a paid link for the sake of buying links to flow PageRank.
That one is only totally flagrant.
A bit off color. Like comment spamming.
Sorta like the link exchanges in German.
But some are even more outrageous. Consider that BeatThatQuote is buying links from pages with ad sections like
Paid Blog Reviews
Remember those "evil" paid reviews Matt Cutts wrote of? Plenty of those to go around ;)
In fact, some of the paid blog links were in place so long that BeatThatQuote got thank you's for advertising for over a year straight.
I don't have a decade of spam fighting experience like Google does. But is it too much to suggest that before Google buys *any* website they should do a basic compliance audit to verify that the site is operating within Google's TOS. I am an independent SEO and it took me all of 2 minutes to find numerous FLAGRANT violations.
What sort of message does Google send the market with the above behavior?
How Can Google Police Anyone?
Google has on multiple occasions penalized other UK based finance sites for SEO issues & link buying. But now that Google owns one of the horses in the race, and that horse has been found to be using anabolic steroids, can they legitimately penalize any of their new competitors?
If I had a UK finance site I would go on a link buying binge right now. Google can't penalize you for it because they are doing the same damn thing. And if they do penalize it for DOING THE EXACT SAME THING GOOGLE IS DOING then you know you have a legitimate gripe for the media, and I have no doubt Microsoft would be willing to help pick up the legal tab.
Google Eats Microsoft's Lunch Again!
Ultimately this is a body blow to Microsoft. Microsoft started to gain momentum in search through verticalization, but has since backed off. Meanwhile Google took Microsoft's ball and ran it in for a touchdown (acquiring MetaWebs, trying to buy ITA Software, and buying BeatThatQuote). And now one of MSN's portal ad partners is owned by Google:
Head of partnerships at MSN, Phil Coxon said, “At Microsoft Advertising, we’re passionate about collaborating with brands to create compelling advertising campaigns. By providing new and exclusive content that appeals to consumers, this partnership both enhances the overall MSN user experience as well as providing a great platform for BeatThatQuote to engage with their target audience on a meaningful level. This deal builds on our previous partnership with BeatThatQuote, which led to a 400% increase in revenue generated from insurance products. We’re delighted to continue to build on this relationship with this new campaign.”
How often do you ever hear the phrase worst practices? Probably never.
Everything is a best practice approach, right up until things change.
Consider AdSense websites.
Hey Look, a Case Study!
When you look at some of the biggest losers in the Google content farm update, many of them happened to be premium AdSense publishers which were even used by Google as case studies! For instance, Hub Pages or EzineArticles.
We are confident that over time the proven quality of our writers' content will be attractive to users. We have faith in Google's ability to tune results post major updates and are optimistic that the cream will rise back to the top in the coming weeks, which has been our experience with past updates - Paul Edmondson
The problem is that for many businesses there will be no bounce back. Some are simply over. The web has evolved & the algorithm has moved beyond them.
Where is the Much Needed Disclaimer?
What makes this worse is that when Google gives a site their premium AdSense feed & sets something up as a case study others will see that as an explicit endorsement.
THIS IS HOW YOU SHOULD DO IT.
Even after Google torches the companies that follow Google suggested best practices those case studies live on, offering what now amounts to maps to Google hell.
Adding Insult to Injury
What makes such filters/penalties even more infuriating is that in some cases when your site is slapped with a negative karma penalty, others who steal your content & wrap it in AdSense will outrank you, since their site does not yet have a negative karma penalty against it. :)
Individually the splog sites may not live long, but collectively they can keep outranking you to ensure you are invisible for your own words, even if you poured years of your life into creating something beautiful & important.
As we noted yesterday, Cult of Mac was collateral damage in Google’s war on crappy content farms. For some inexplicable reason, we got downgraded when Google tweaked its algorithms last Thursday.
But today we’re back in. We’re on Google News (a very important source of daily traffic) as well as Google’s general search results. However, we still get outranked by some of the scraper sites that steal our content, so not everything’s perfect.
That whooshing sound you just heard was MFA sploggers making a mad dash to steal content from the list of currently penalized sites.
Cult of Mac is lucky they had enough pull with the press to get reconsidered. Most webmasters who got hit did not & anyone who has contracts based on set traffic levels or tight margins which just turned negative are in a pretty crappy situation. Yet another example of the importance of not fueling growth with debt & the importance of profit margins and a cash-on-hand safety net.
Who Are the Opportunistic Maximizers?
The problem with such an approach of maximizing everything you do to suck peak revenue out of the pageview is that things can change on a whim. I have seen some of Google's 1 on 1 AdSense optimization advice they sent to a friend of mine. I told my friend that the optimization advise was at best short-term opportunism that would end up crushing them in the long run if they actually implemented it.
Google doesn't care if following their advice torches your site if it makes them a bit more money, because ultimately there is another person standing in line waiting to follow.
My friend is lucky that they realized my advice was more trustworthy than the advice they were getting direct from Google. If they listened to Google back then their business might be destroyed today.
Google likes to position SEOs as exploiters out for the quick buck, but what honest analysis shows is that it is Google which is pushing the boundaries in terms of:
One of the worst hit sites in the AdSense farm update was WiseGeek. Sure WiseGeek must have had something like a 20% ad clickthrough rate. But with traffic falling 75%, maybe they would have been better off building a cleaner experience with a 5% CTR.
What was the most profitable best practices based approach suddenly falls short. And the results are not always predictable. When Google decided to attack content farms who honestly knew that:
somehow eHow.com would survive
yet somehow Google's "algorithmic" approach would punt 10,000's of smaller websites that have far higher content quality
In advance of the solution I was fairly certain eHow would survive, but what I underestimated was the Google engineers. Or rather the ignorance of same. I simply couldn't imagine such a content farm algorithm going live that missed eHow and decimated the lives of so many independent webmasters.
I guess we can simply view this as an extension of Google's you can have any web you want so long as it is corporate TM policy. I think Brett Tabke said it best in a recent AdSense thread:
When the rules and the enforcements are made up by monopolies in a make believe world - there is no cheating.
The only "cheatings" is when it gets outside the lines of the law. - Brett Tabke
AskTheBuilder is yet another Google AdSense case study. In spite of being a niche player well regarded in his community, Sistrix data shows the site off 87% after the most recent Google update!
Who Caused the Content Farm Problem?
Everyone likes to vilify the content farms and scrapers (and they deserve it) but the real villain behind all of this is CPC/CPM based advertising.
Can you imagine a world where your attention was sold off based on how long you stayed on a page rather then how often you switched pages? If google wants to fix their search results, they should focus on fixing adsense. The technology to more accurately measure a viewer's exposure to an ad are there, it just needs a trustworthy player to bring it to market. Someone trusted by both users and advertisers.
Google made click/impression-based advertising appealing to both groups and it made them what they are now. It's time to get away from it. - po
roll in an algorithm that aggressively penalizes tons of borderline edge cases
see who complains to the media & has connections with the media
fix the rankings of those who you like & those with sway, while ignoring the rest
Can You Trust Google?
All of this leads to the obvious question: can you trust Google?
The short answer is yes.
The long answer is you can *always* expect Google to do what is in the best interest of Google. As they plow into field after field (payments, local, mobile, ecommerce, mortgage, credit cards, travel, weddings, fashion, etc.) & use their search dominance to manipulate other markets one would have to be blind to view Google as anything other than a competitor.
Maybe not today. Maybe not tomorrow. But some day they will come. And it is never fun when it happens to you. :(
Until that day may come, if you always follow their best practices, just remember ... ;)
Put it this way. Any algorithm that takes out Demand Media content is going to take out a lot of SEO content, too. SEO copy-writing? What is that? That's what Demand Media do. As I outlined in the first paragraph, a lot of SEO content in not that different, and any algorithm that targets Demand Media's content isn't going to see any difference. Keyword traffic stream identical to title tag? Yep. A couple of hundred words? Yep. SEO format? Yep. Repeats keywords and keyword phrases a few times? Yep. Contributes to the betterment of mankind? Nope. SEO's need to be careful what they wish for....
There were a lot sites following the SEO model of "writing for the keyword term" taken out, not just sites pejoratively labelled as "Content Farms". Ironicly, the pinup example I used, Demand Media, got off lightly.
Some people have suggested there has been much collateral damage. Google have taken out legitimate pages, too.
What happened is that the pages that were taken out shared enough similarity to pages on Content Farms and the algorithm simply did what it was designed to do, although Google have admitted - kinda - that the change still needs work. The ultimate judgement of whether this is a good or a bad thing comes down to what Google's users think. Does Google deliver higher quality results, or doesn't it?
I'm pissed because we've worked our asses off over the last two years to make this a successful site. Cult of Mac is an independently owned small business. We're a startup. We have a small but talented team, and I'm the only full timer. We're busting our chops to produce high-quality, original content on a shoestring budget.We were just starting to see the light at the end of the tunnel. After two years of uncertainty, the site finally looks like it will be able to stand on its two feet. But this is a major setback. Anyone got Larry's cell number?
Scroll down, as there's also some very interesting comments in reply to that post.
This is nothing new, of course, It's been going on since search began. The search engines shrug, and send businesses that depend on them flying, whilst elevating others.
What can be done?
Spread The Risk
"Be less reliant on Google!", people say.
It's an easy thing to say, right, but what do you do when Google is the only search game in town? We know any business strategy that relies on an entity over which we have no control is high risk, but what choice is there? Wait for Bing to get their act together? Hope Blekko becomes the next big thing?
None of us can wait.
Sometimes, no matter how closely we stick to Google's Guidelines, Google are going to change the game. Whether it is fair or not is beside the point, it's going to happen.
So, we need to adopt web marketing strategies that help lessen this risk.
The best way to lessen this risk, of course, is to not rely on Google at all. Design your site strategy in such a way as that it wouldn't grind to a halt if you blocked all spiders with a robots.txt. Treat any traffic from Google as a bonus. Such a strategy might involve PPC, brand building, offline advertising, social media, email marketing and the wealth of other channels open to you.
Try the above as an academic exercise. If you had to operate without natural traffic, does your business still stand up? Are you filling a niche with high demand, a demand you can see in other channels? Is there sufficient margin to advertise, or does your entire model rely on free search traffic? Are there viral elements which could be better exploited? Are there social elements which could be better exploited?
Academic exercises aside, we can also look to mitigate risk. Think about not putting all your eggs in one basket. Instead of running one site, run multiple sites using different SEO strategies on each. Aaron talks about running auxiliary sites in the forum.
Try to get pages (articles, advertising) on other sites in your niche. If your site is taken out, at least you still have a presence in your niche, albeit on someone else's site. A kindly webmaster may even agree to repoint links to any new site you devise.
Do you have other ideas that help mitigate the risk? Add them to the comments.
It's An Advantage Being An SEO
Finally, be pleased you're an SEO.
SEO just got that much harder, and the harder it gets, the more your services are required, and the higher the barrier to entry for new publishers. Every day search is getting more complex. At the end of the day, it's an algorithm change. It can be reverse engineered, and new strategies will be adopted to maximize the opportunity it presents.
Until such a time as Google tells us exactly what they want to see, and rewards such content, SEO's will just keep doing what they do. And thank goodness Google isn't entirely transparent. If they were the value of your SEO knowledge as a competitive advantage would plunge. For many of us, wages would quickly follow.
Sure a short-term hit is painful, but the best SEOs will recover.
As they do, other content producers will be left scratching their heads.
Media is all about profit margins. eHow was originally founded in 1998 & had $36 million in venture capital behind it. But the original cost structure was flawed due to high content costs. The site failed so badly that it was sold in 2004 for $100,000. The original site owners had GoogleBot blocked. Simply by unblocking GoogleBot and doing basic SEO to existing content the site had a revenue run rate of $4 million Dollars within 2 years, which allowed the site to be flipped for a 400-fold profit.
People are arguing if Demand Media is overvalued at its current valuation, and at one point the NYT was debating buying Demand Media by rolling About.com into Demand & own 49% of the combined company. But the salient point to me is that we are talking about something that was bought for only $100,000 7 years ago. Sure the opportunities today may be smaller scale and different, but if you see value where others do not then recycling something that has been discarded can be quite valuable.
The key, he said, was to keep costs low. If possible, don't pay for the intellectual content. Look for material, he urged, on which the copyright has expired. Any book published in the U.S. before 1923 was available.
He said he was in the process of turning vanloads of old books into websites. With a few hours of labor, for example, you could take a turn-of-the-century Creole cookbook and transform it into the definitive site for vintage Creole recipes. Google's AdSense program would then load the thing up with ads for shrimp and cooking pots and spices and direct people looking for shrimp recipes to your website.
How did ArticlesBase grow to that size? It and Ezinearticles were a couple of the "selected few" which lasted through the last major burn down of article directories about 3 or 4 years ago. But it seems their model has peaked after this last Google update.
Search is Political
Content farms are proving to be a political issue in search. They are beginning to replace "the evil SEO" in the eye of the press as "what creates spam." Rich Skrenta created a spam clock which stated that a million spam pages are created every hour. He then followed up by banning 20 content farms from the Blekko search results & burning spam man. ;)
Microsoft also got in on the bashing, with Harry Shum highlighting that Google was funding web pollution. When Blekko's model is based on claiming Google polluted the web with crap, Microsoft says the same thing, and there is a rash of end user complaints, there are few independent experts the media can call upon to talk about Google - unless they decide to talk to SEOs, who tend to be quite happy to highlight Google's embarrassing hypocrisy. Freelance writers may claim that marketing is what screwed up the web, but ultimately Google has nobody credible and well known to defend them at this point. The only people who can defend Google's approach are those who have a taste in the revenue stream. Hence why Google had to act against content farms.
Google stepping up their public relations smear campaigns against Bing and others is leaving Google looking either hypocritical or ignorant in many instances, like when a Google engineer lambasted an ad network without realizing Google was serving the scam ad.
At first glance StackExchange's growth looks exciting, but it has basically gone nowhere outside of the programming niche. In my opinion they are going to need to find subject matter experts to lead some of their niche sites & either pay those experts or give them equity in the sites if they want to lead in other markets. Worse yet, few people are as well educated about online schemes as programmers, so the other sites will not only lack leadership, but will also be much harder to police. Just look at the junk on Yahoo! Answers! There are Wordpress themes and open source CMS tools for QnA sites, but I would pick a tight niche early if I was going to build one as the broader sites seem to be full of spam and the niche sites struggle to cross the chasm. As of writing this, fewer than 50 Mahalo answers pages currently indexed in Google have over 100 views. It flat out failed, even with financial bribes and a PR spin man behind it.
At the organizational level, Google is essentially chaos. In search quality in particular, once you've demonstrated that you can do useful stuff on your own, you're pretty much free to work on whatever you think is important. I don't think there's even a mechanism for shifting priorities like that.
We've been working on this issue for a long time, and made some progress. These efforts started long before the recent spat of news articles. I've personally been working on it for over a year. The central issue is that it's very difficult to make changes that sacrifice "on-topic-ness" for "good-ness" that don't make the results in general worse. You can expect some big changes here very shortly though.
A good example of the importance of padding out results with junk on-topic content to aid perceived relevancy can be seen by looking at the last screenshot of a search result here. Blekko banned the farms, but without them there is not much relevant content that is precisely on-topic. (In other words, content farms may be junk, but it is hard to have the same level of perceived relevancy without them).
Google created a Chrome plugin to solicit end user feedback on content mills, but that will likely only reach tech savvy folks & the feedback is private. Google can claim to use any justification for removing sites they do not like though, just like they do with select link buying engagements. Look the other way where it is beneficial, and remove those which you personally dislike.
In a recent WSJ article Amit Singhal was quoted as saying new signals have been added to Google's relevancy algorithms:
Singhal did say that the company added numerous “signals,” or factors it would incorporate into its algorithm for ranking sites. Among those signals are “how users interact with” a site. Google has said previously that, among other things, it often measures whether users click the “back” button quickly after visiting a search result, which might indicate a lack of satisfaction with the site.
In addition, Google got feedback from the hundreds of people outside the company that it hires to regularly evaluate changes. These “human raters” are asked to look at results for certain search queries and questions such as, “Would you give your credit card number to this site?” and “Would you take medical advice for your children from those sites,” Singhal said.
Evolving the Model
One interesting way to evolve the content farm model is through the use of tight editorial focus, a core handful of strong editors, and wiki software. WikiHow was launched by a former eHow owner, and when you consider how limited their relative resources are, their traffic level and perceived editorial quality are quite high. Jack Herrick has struck how-to gold twice!
Notice any content farms missing from the above list? Maybe the biggest one? Here is a list of some of eHow's closest competing sites (based on keywords, from SEM Rush). The ones in red got pummeled, the ones in yellow dropped as well & were fellow Demand Media sites, and the ones in green gained traffic.
Jason "will do anything for ink" Calacanis recently gave an about face speech claiming people need to step away from the content farm business model, and in doing so admitted that roughly everything he said about Mahalo over the past couple years was a complete lie. Surprise, surprise. The interesting bit is that the start up community - which used to fawn over his huckster PR driven ploys - no longer buys them. Jason claimed to have "pivoted" his business model again, but once again we see more garbage content. His credibility has been spent. And so have his rankings! Sistrix shows that not only is he ranking for fewer keywords, but that the graph has skewed downward to worse average positions.
After the Crash, What is Next?
The biggest content farms like Ask & eHow will still do well in the short run. Over the long run I see Google bringing the results of content farms to the attention of book publishers & then working to slowly rotate out from farmed content to published book content. Most readers do not know that most book writers are lucky if they earn $10,000 writing a 300 or 400 page tome. Publishers tell book authors that with the additional exposure they can often sell lots more other things, but unless the content is highly targeted that might not back out well for the author. But that cheap content is far better structured and far more vetted than the mill stuff is.
Over the past week I have been seeing more ebooks in the search results, though I am not entirely sure if that is just because I am searching for more rare technical stuff that simply might not be online.
But what they didn't mention is that eHow's rankings are actually up! In fact, their new distribution chart looks just like their old one, only skewed a bit to the left with higher rankings. eHow's profile is 15% better than it was before the update & the only site which gained more traffic from this update than eHow did was Youtube.
If you listen to Richard's interviews, you would never know him to be the type to redirect expired domains:
We really want to let Google speak for themselves. Whatever Matt Cutts and Google want to (say) about quality we totally support that because again that’s their corporate interest. What we said and would have said is we applaud Google removing duplicate content ... removing shallow, low quality content because it clogs the search results. Both we and Google are 100 percent focused on making the consumer happy. It’s the right thing to do and it’s good for our business.
If you syndicate Google's spin you can get away with things that a normal person can't. Which is why eHow renounced the content farm label even faster than they created it.
Article directories & topical hub sites have been online since before eHow was created. But eHow's marketing campaign was so caustic & so toxic that it literally destroyed the game for most of their competitors.
And now that Google has "fought content farms" (while managing to somehow miss eHow with TheAlgorithm) most of Demand Media's competitors are dead & Richard Rosenblatt gets to ride off into the sunset with another hundred million Dollars, as eHow is the chosen one. :D
Many of the changes we make are so subtle that very few people notice them. But in the last day or so we launched a pretty big algorithmic improvement to our ranking—a change that noticeably impacts 11.8% of our queries—and we wanted to let people know what’s going on. This update is designed to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful. At the same time, it will provide better rankings for high-quality sites—sites with original content and information such as research, in-depth reports, thoughtful analysis and so on.
Currently this update is US only, so if you are outside of the United States you may need to get a US VPN or add &gl=us to your search string's results on Google (likeso). Recent updates have had a variety of impacts and implications outside of content mills.
A couple days ago there was a blog post on TheDomains about "how stupid SEOs are" and "the amazing power of domaining" where they highlighted how awesome domaining was because a guy registered a domain name he saw in a comic and it sent a bunch of traffic.
What that article failed to mention was:
That traffic wasn’t from the power of the domain name…that was the power of free advertising & the distribution of the comic strip.
The same domain name likely received ~ 0 traffic until it was featured in the comic strip. If it had an organic traffic stream for years before being highlighted it most likely would have already been purchased.
As that comic strip falls into the archives & into obscurity the organic traffic it was driving will drop back to roughly where it started at: 0.
The flood of new found traffic was hardly a goldmine anyhow. It was entirely irrelevant to his main business, and thus entirely worthless. The only exception there would be unless the person was offering information about comics, installing malware, pushing reverse billing scams, etc.
Being Ignorant Doesn't Create Profit
The laughable (and ignorant) thing about the comments on that post were that some of the people who were commenting were equating SEOs to misdirection & scams that sell traffic off to the highest bidder. Sorry, but that is what PPC domain parking is all about...the ad networks optimize yield & the publishers agnostically push whatever generates the most yield: often scams!
Stating that all SEOs are dumb spammers is precisely the same as stating that all domainers are cybersquatters. Neither are true, and neither serves much purpose, other than aiding the spread of ignorance.
Why Domainers View SEOs Dimly
Many domainers who try to hire SEOs fail badly because they are too cheap & buy from lousy service providers. They feel that since they bought domain x (and sat on it while literally doing nothing for a decade) that they somehow deserve to be the top ranked result. To be fair, it is pretty easy to become lazy and not want to change things when you register domain names & then literally watch them spit money at you. ;)
Against that approach, the smart SEOs (the ones actually worth hiring) realize that it is more profitable to buy their own domain names and keep all the cashflow for your efforts rather than doing 95% of the job for 5% of the revenues. Yes a good domain name is helpful, but with the right attitude you can still do quite well even on a hyphenated nasty looking .info domain name. ;)
Why SEOs View Domainers Dimly
A combination of squandered opportunity & arrogance.
I frequently tell myself that in 3 years or 5 years that the web will be so competitive that it will no longer be as profitable as it is today. And every year I have pushed that mindset back another year while we grew. But who knows how long that will last? Sure as long as there are signals there will be ways to influence them, but if you are not one of the favored parties then at some point it will be challenging to compete.
The Real Challenge: the Search Duopoly vs Publishers
When Google got into the web browser game, one of the big "innovations" was the Omnibox. They integrated search right into the address bar to help drive incremental search volume.
As they were a new browser it was not a big risk or big concern to domainers (as most people who use direct navigation are either people revisiting a website they already visited or people new to the web who are likely to use the most common default web browser - Internet Explorer). Nonetheless, address bar as search box highlighted things to come & a way the web would change.
When Google announced their Chrome OS they decided to do away with the CAPS LOCK BUTTON AND REPLACE IT WITH A SEARCH BUTTON. OOPS SORRY ABOUT THAT. Again, it is not a big deal today, but if that ever became standard the future would grow more challenging.
The big problem with Google doing such innovations is that whatever they do, they also give Microsoft permission to do. Google can't complain about what Microsoft is doing if Microsoft is only following Google's lead.
Just like the Omnibox, Internet Explorer 9 integrates search into the address bar.
As soon as IE9 rolls out, domainers can count on losing traffic month after month. This trend is non reversible in well established markets like the United States & Europe, and in 3rd world markets the ad markets pay crumbs.
If that happens, it won't impact domainers much, but if Microsoft copies it, then look out below on domain prices. You wouldn't be able to get to a domain name without first being intercepted by a search engine toll booth. In that environment, a PPC park page produces ~ $0. And even established sites that are generic might not be a great strategy for creating *sustainable* profits if/when the organic results are below the fold. People who invest in brand have some protection against pricing pressures & irrelevant search results, but those who are generic don't typically have much brand to protect their placement nor profits.
Google despised how Microsoft bundled services & believes all other competitors should win market by market based on the merit of the product. Google does not believe this line of thinking should be applied to TheGoogle though, as you need to be a seriously dominant market player to stay in the lead position while opting out of appearing in the search results of the default search engine.
Even on the regular web staying competitive is growing increasingly challenging due to these moves to lock up and redirect normal user behaviors to shift it through an increasingly ad dominated search channel.