How to Fix the Broken Link Graph

When Teoma launched in 2002 GoogleGuy made the following comment:

"I just have one question. Are the Overture results on top an April Fool's Day joke, or is that for real? ;) "

Since then Google has put ads above their organic search results, done selective self-serving within their "organic" results, and built a business that is pulling in over $20 billion a year. It turns out aggressively carpet-bombing the web with ads is no joke. :D

But the net effect of that success is that many people know the value of links. SEO is a zero sum game, and so if you don't engage in heavy self promotion you will remain obscure on Google's linkless internet:

  • no one links honestly any more.
  • all links are suspect.
  • no one links freely any more.
  • those that do link freely are considered naive.
  • page rank is specifically worth money.
  • links are currency.
  • articles that once contained great links - no longer link to story targets.

On the web ideas (and business models) spread quickly. So as companies learn that Google encourages and pays you to pollute the web with garbage that is what many people will do. Google goes so far as to explicitly state what not to do, and many people view that as a checklist of opportunities, as it wouldn't be on that list unless it worked. ;)

The lack of community and camaraderie within the SEO industry both remarkable & unsurprising give that the SEO industry is a bit of a canary in the coal mine in terms of adopting new best practices (or worst practices, in some cases).

  • At the lower end of the market people are operating like robots, mechanically spamming in a way that sure feels like crapping on the virtual living room floors in established public forums.
  • Just yesterday I read a blog post listing me amongst a list of resources where everything recommended had a link - except for our site. The lack of link was so bizarrely out of place that they literally had to explain why they didn't link to our site. Crazy stuff, especially from an SEO "professional" who claims to like and value your work!
  • As attention becomes more scarce, many people are willing to do anything to get a bit of it.

Meanwhile Google has no issues funding that very "linkless" web pollution by paying the likes of Demand Media to syndicate their ads, and encourage the use of rel=nofollow on links while trying to build out the model for digital sharecropper overlords. Google has pushed so far on nofollow that content spam is the single most profitable SEO business model going today.

The end result of this business decision? Google becomes its own nemesis by funding spam:

This triple occurrence of Optimization by Proxy creates a self-reinforcing cycle where the made-for-adsense website owners are rewarded with cold hard cash for their efforts. What's worse, this cash flow has been effectively subtracted from the potential gains of legitimate content producers. One can say that the existence of Google search/adsense/adwords makes all this commerce possible in the first place, but this does not make the downward spiral of inefficiency disappear.

What is even more strange about the above quoted article is that a Google search-quality engineer submitted it to Hacker News using "Sufficiently advanced spam is indistinguishable from content" as the title & wrote the following:

All of the fascinating things about signals are confidential for all of the reasons listed in the article, and Google has been sued so many times by sites that think they should rank better than they do that I can't really give examples.

I think it's safe to say though that there are a lot of people worried about and thinking hard about what the web is turning into and how to rank it appropriately.

Most of the content is no longer written by devoted hobbyists, people no longer link as often to things they like, and much of the content on the front pages of reddit, digg (and sometimes even hackernews) was put there by people trying to make your search results worse.

Given that Google pays for the creation of content and is the most profitable distribution channel for many webmasters, few businesses have anywhere near Google's influence on "what the web is turning into."

The smaller the corpus of voters there are the fewer people you need to influence to manipulate the search results. And so stuff like this becomes popular:

But if you have a live and flourishing link graph then efforts of spammy delight won't be able to compete anywhere near as well against the best sites. The problem is the best sites often remain in obscurity & even when they spread through social networks most of those links use nofollow.

The powerful element of links is that they give search an informational bias. Most other forms of user feedback (even awareness) are to some degree driven by ad budget, which would give the results a commercial bias that would cut Google's AdWords revenues.

Years ago on porn search Matt Cutts stated "One thing to consider is that our rankings (because of PageRank and the link structure of the web) often lean more toward information sites. You usually have to look a little more deliberately to find porn on Google." And last year a Google search engineer on Reddit stated: "Incredibly, we take an active role in ignoring porn search. As in, we neither care nor not care about whether our changes affect the search for porn. I'm quite impressed myself at how good it is given this, and sometimes I wonder how much better it could be..."

The point being that Google can choose to be a passive reflection of the web, but they choose *not* to and have impacted the web by perverting the link structure. Even Google engineers admit "people no longer link as often to things they like" - this is not due to "scalable algorithms" - but due to a long history of calls for spam reports, calls for paid links reports, threatening to remove PageRank, pushing rel=nofollow out for a niche purpose and then claiming it to be a panacea, associating (non-Google!) link selling with bogus cancer cures, creating the AdSense API, excessive emphasis on domain authority, claims like "we are packrats at Google. We never seem to throw out information about the link structure of the web. :)", etc.

Yes any new webmaster can quickly rank any trashy content in Google - not on their own site - but on one owned and controlled by a Google arbitrage partner like Demand Media.

Google engineers focused on making their own jobs easier (by making links harder to get) rather than encouraging and promoting the advancement of the corpus of content & links/votes they rely on. The fundamental question for search is if the ecosystem is healthier with many micro-parasites or fewer macro-parasites. I always promote diversity as a good thing, but capitalism typically promotes homogenization to increase yields. That will work for Google right up until...

  • some of the large arbitrage players form their own ad network, or sign an exclusive deal with Bing
  • quality publishers pull out of search and people prefer to ask their own communities what is good rather than searching to find bland 3rd grade answers wrapped in AdSense

...at which point Google will start promoting other sites & building a new model. Or they will become irrelevant.

But Google is not required to go through that pain.

They could undo the years of FUD that destroyed the link graph by stating the importance of outbound links, and then putting a bit of weight on it. It is something that was hinted at in the past, but the focus on making links harder to get has undermined the utility of using links as *the* measuring stick for quality because "people no longer link as often to things they like."

Published: May 17, 2010 by Aaron Wall in business

Comments

MyContent
May 17, 2010 - 12:59pm

Google has become the go-to place for search, and it's going to take a serious screw-up on their part to lose their gigantic market share. As such, like most gigantic enterprises, Google is now in the business of protecting the status quo.

Google knows that they had better show the expected entities for particular searches, and it would not surprise me at all if they were tweaking the results to make sure those entities show up.

For competitive content-driven informational searches, it gets a little trickier. But as long as a handful of the big boys fill the first page of the results, no one is going to notice. They don't all have to rank all the time. There are enough of them out there that Google can mix it up, and no one will be the wiser.

This is bad for small independent sites. They are the ones that are harmed by Google's arbitrary penalties and filters (particularly that nasty dup content filter that never seems to remove the 3 billion copies of shared stories by major newspapers but always trashes the original source of an article copied verbatim from an independent site).

No one is going to notice if Joe's News suddenly falls off the face of the earth. But if the Guradian and the Washington Post and the LA Times don't show up with some regularity, it's apparent.

This is why big sites can get away with just about anything. But if a small site sneezes, it gets wiped out.

JamesDNash
May 17, 2010 - 8:28pm

They could undo the years of FUD that destroyed the link graph by stating the importance of outbound links, and then putting a bit of weight on it.

Matt has already stated that:

In the same way that Google trusts sites less when they link to spammy sites or bad neighborhoods, parts of our system encourage links to good sites.

and im gona say i believe with almost 100% certainty that linking out does have benefit if done correctly.. im not going to say what correctly is but :).. i know this personally with a site i started 9 months ago.. it has rapid growth in its niche even when competing against multi million dollar companies with millions of dollars in ad spend and seo..

I have several #1 spots for very lucrative terms jumping right over sites that have been around for years and years... soooo.. imo they are already putting quite a bit of weight behind it, if done right :)

May 18, 2010 - 3:54am

Great quote :D

But they don't push that quote anywhere near as much as they push the FUD. I also agree that it can help, but I think they would be well off to put even more weight on it.

JamesDNash
May 18, 2010 - 4:25am

"They" don't need to push it, they have people like "You" to make people aware of it...

shit Aaron if you could see my site and the stats you would think that they are putting a great deal of weight behind it.. i mean, it only makes sense why they would put a great deal of algo weight behind it :) but shhh.. don't let the cat out of the bag, people may starting linking cuz it does make sense for users

May 18, 2010 - 8:43am

Well the thing is, if Google says "xyz is certain to work" (vs hinting that something might be possible) the public reaction will be far different.

Look at how Google scammed webmasters with rel=nofollow by trying to claim it had some legitimate uses for buy in. And then when too many people were using it in ways they didn't like they changed their stance on it. But they didn't just change the relevancy algorithms, as when they did that most webmasters were unaware and SEOs were still pushing its (then) misuse as a best practice. But then they had to come out and once more threaten that there was a fixed cost with it & it didn't work the way they originally claimed.

That sorta proves how hard it is to isolate some variables, especially given a limited data set.

Patrick
May 20, 2010 - 4:26pm

@Aaron:

"That sorta proves how hard it is to isolate some variables, especially given a limited data set"

Does this mean you kind of doubt that JamesDNash can really tell whether his great rankings with that one site are caused by the way he uses outbound linking? (or did I completely misunderstand this phrase)?

@JamesDNash:

What you said sounded extremely interesting, but with a sample size of 1, I'm usually skeptical...just saying so you don't misunderstand this and take offense or anything :-)

May 21, 2010 - 6:34pm

I didn't say anything bad about James. He tends to not be the bullshitter type & tends to write based on research and experience (rather than what a lot of folks write on).

But I was just trying to mention that isolating variables generally can be tough :D

Add new comment

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.