Outside of Google, Amazon and eBay own perhaps the 2 largest streams of ecommerce traffic. In fact, both are amongst the top 10 sites in Alexa (even though ~ 100% of their traffic is commercial while ~ 0.001% of Twitter's traffic is). In spite of Google's prominent promotion of Wikipedia, Amazon still gets more traffic.
Recently TechCrunch posted an article outing [link nofollowed] the Sequoia-backed start up Milanoo for ranking using paid links:
Here’s what Digital Due Diligence found: For a number of very valuable keywords in Google search,, Here’s how Milanoo ranks for “cheap dresses” (position 2), “evening gown” (1), “cheap wedding dresses” (1), and “summer dresses” (2). Digital Due Diligence partner Doug Pierce (who also served as an expert in the New York Times J.C. Penney expose), writes that those four keywords alone have an equivalent cost of nearly $200,000 per month in Google AdWords.
It’s a red flag, explains Pierce’s fellow partner Byrne Hobart
The article left a fairly foul stench in the air which is hard to get over.
Risk Analysis vs Risk Creation
These folks claim to do due diligence for investors, which essentially means "risk analysis" and "risk management." But then to market themselves, they throw active investments under the bus for self promotion. And now they have done so repeatedly. It is not an isolated incident, but rather a pattern of conduct.
To be frank, that form of marketing from an outfit claiming to do SEO risk analysis can be described using no other word than this: sleazy.
Just Another Form of Competitive Sabotage
An outing like the above is typically driven (at some level) by a competitor looking to take down a competitor. And such a high profile outing literally can destroy lives. It is a high stakes game of public relations. The media outlet gets a story, the expert gets quoted, and the competitor gets torched.
If I was an investment firm I would never spend a single Dollar with Digital Due Diligence. Why? Well they may do a valuable service on some project you are funding them for, BUT if you fund them at all you are encouraging more outing in the future and more risk for your own future investments.
Outing is Anti-Innovation
Eric Schmidt has stated that lobbyists write the laws. Markets are rigged to favorestablishedinterests. If you are an investor you are betting that you can take smart calculable risks & disrupt markets. But SEO outing is yet another layer of unknown risk which harms all start ups while rewarding existing market leaders: the exact opposite of innovation.
Even if you encourage your own investments to be ultra-conservative you still have no protection from this sort of activity.
A competitor could easily buy a bunch of links for one of your sites (they could even pay cash for a gift card while traveling & use that to buy links from a clean browser with cookies cleared on a public wifi connection, making themselves untraceable). After throwing a few hundred or few thousand Dollars at setting up the site, they can then leak a tip to Digital Due Diligence, who will then leak it to TechCrunch or the NYT.
Why This Sort of Outing is Horrible for SEO Professionals
The core issue here is professionalism. Should we let people who screw other people over get ahead while trying to paint themselves as the good guy? I don't see how there is any hope left for the industry if that becomes the new normal. Every time there is a high profile outing SEO investments become perceived as being more risky and the whole of the industry looks less professional.
An Example of Two Interpretations of Google's Guidelines
The last time I had any significant experience on this front the SEO police asked who Google should "come down on" for our affiliate program passing link juice. That made our affiliate links no longer pass link juice. Shortly after a Google search engineer publicly stated that affiliate links should count and the same SEO firm that threw us under the bus mentioned they were thinking about bringing their affiliate program in-house so they could do the same thing they outed us for. A few years later it was highlighted that Google has an active investment in a start up that builds a scaled paid link network.
In other words, Google claims their guidelines to be black and white, but a particular technique can be fine for some, spam for others, and worth funding on an industrial scale if Google gets a piece of the action. Is it any surprise that the FTC is looking into Google's business practices?
Just Say No!
The above sort of activity is just like Google and Microsoft leaking each other's security flaws publicly to try to screw each other over. Out of such exchanges nobody wins, but everyone looks a bit more like a used car salesman, as we further create a market for lemons.
With the rise of such SEO diligence projects, how long until the primary business model & main form of diligence being done is funded "research" to take down competitors? Is this market even worth participating in if we let it devolve to that point?
These sort of folks who throw the whole of the SEO industry under a bus for self-promotion should be shunned by the industry. If our industry is to have any sense of fair, just, and reasonable meritocracy to it then this behavior can not be condoned.
Like a good neighbor, State Farm is there and there and there and there and there.
The Struggle Real Businesses Face
The big problem with this IMHO is all but the spammer (who is now busy working on "local" signals) loses. Legit online-only pure plays are simply wiped off the result set. The searcher gains nothing by seeing State Farm agents 5 times in the search results. Even the local business which has a new windfall of business is simply overwhelmed with leads, meaning they likely have (at least relatively) poor customer service until they hire up.
To a small business, a sharp rise in demand can be every bit as damaging as a sharp fall in demand.
But should small local businesses hire aggressively, they could be only 1 algorithmic update away from needing to prune staff. Maybe some day Google decides to limit the results to show 1 agent per parent company, and then the agents end up fighting out each other (much like affiliates had to fight each other on bids in AdWords to be the 1 that shows up).
Given that some of the agents ranking page 1 have less than a dozen inbound links & links from only a few unique domains, it won't take long for some new "local" players to come online.
What Makes a Search Result Good?
A lot can be said for getting users where they need to be quickly. When it works it has great value. But when it doesn't work, it makes the market less efficient. Value chains exist for a reason. Sometimes a brand (or an individual agent of brand x) is not in the best position to act as an unbiased advisor.
As a consumer buying car insurance, I don't care that my agent is local. In fact, if I live in an expensive area I may want my insurance provided from someone who lives in an area with a lower cost of living so they can provide the services (while making a comfortable living) for less. For the last decade I have been insured from a company in another state (USAA in Texas). Location had precisely 0 impact on my decision making.
What mattered to me was that they had great rates. Which is precisely what almost all insurance commercials promote.
Geico spends nearly a billion Dollars a year pounding that message into the minds of consumers.
The problem is that almost all the big brands promote the exact same message. They are the cheapest. Save with them. Etc. Online pure plays that provide quote comparisons provide a valuable & value-add function in this marketplace, but they have simply disappeared from Google. They aren't local enough to hit the local signal, they aren't brand enough to hit the brand signal, and since they are not the end brands they can't justify buying $30 AdWords clicks thinking that what they don't get back in direct ROI can be written off to "brand."
Ultimately the end user loses (or at least until Google creates their insurance flavor of "comparison ads.")
This Stuff is Everywhere
This stuff is even happening on search queries where there is absolutely no implied local intent & no need for a local provider. General discovery & topical queries like "web designer" or even informational background searches like "SEO" now bring up service based sites with a local presence.
Leaving Off On a Positive Note
1 day doesn't make a trend, but if this stuff sticks ranking local sites for big keywords just got really easy.
If you know SEO and live near a big city, a second office location might soon be a profitable decision.
If you are a local business who thought SEO was too complex or expensive, that excuse may have just been removed from the marketplace.
If you run a bespoke consulting styled business & ran into a windfall of demand don't forget to increase your rates & be more selective with who you work with. Working all the time leads to burn out. Trust me I know that all too well. ;)
This is another example why it can be a great idea to mix and match your businesses...such that if one jumps out of nowhere or another one tanks you are still fine. Having multiple projects is one of the few ways you can really protect yourself from the likes of Panda & updates like this one. Running multiple businesses allows you to lean into your side gigs when your main one drops off, and push harder on your main gig when it is really humming along.
The term "competitive research" conjures up all sorts of imagery like expensive tools, shiny buttons, cute charts, and fancy (sometimes foolish) language about precise insight into a particular site or marketplace.
In reality we know that such claims are usually best taken with a large grain of salt. Most competitive research data is scraped from search engines and then has custom filters applied to it. Such filters can actually be a detriment to the data because, in desperate attempts at differentiation, tool-sets routinely use metrics which get overly convoluted with custom values and such that the final product because overhyped and underwhelming.
Some tools make up for their sampling errors by allowing you to upload your keywords & data directly into their database. The problem with this is that you are putting keywords which were "below the radar" into a database that your competitors may be using. Why just give away your data to the competition like that? Talk about working against yourself!
Let's remember that these custom metrics and estimates are typically extrapolated off of scraped data, or data purchased from IP's, or data from custom toolbars, all of which are data samples. So it is kind of like; scraped data +/- data extrapolations + in-house data + custom metrics = final product.
It is reasonable to assume that the more custom or guesstimated layers you build off of occasionally unreliable data (waves at Google's keyword tool and SERPS) the less and less targeted that data is. Moral of the story is, "choose wisely young jedi".
Getting Useful Data for Free
Now that we've set the expectation stage (don't expect tools to be a push button, slot machine win) you might feel like paying hundreds or thousands a month for these kinds of tools is a bit much. Sometimes yes, but multiple data points certainly have their advantages and it's not that the data is junk by any means, it's just that the data shouldn't be relied upon as if it were scientific. The data can most certainly be helpful but it comes down to ROI for you and your specific project(s).
There are many tools you can use to get lots and lots of decent keyword competition data for free. We aren't going to be covering free trials, just tools that give you what the have for free or tools that give enough useful data inside of a free version of their product.
If you are in the competitive research stage, you've probably already got a topic in mind. So we'll assume that you are doing competitive research on the keyword "camping equipment".
Accessing Free Tools
You could do a specific bookmark folder which encompasses links to your free tools for easy access. The first things you will probably look at are:
the SERP for your keyword
age of ranking sites
links (total links, links to domain, links to page, edu/gov links)
SEO for Firefox is a free firefox extension which will give you important SEO metrics quickly, from a variety of reputable data sources. Typically, you might want to aim for in at least the top 3 given all the stuff that could be included in a SERP like:
Google shopping results
and so on...
You can get a pretty good initial glimpse of the competition metrics within a few seconds. This is clearly a brand-heavy SERP and it is reflected in the SEO metrics. Here's a screen shot of what you'd see for a particular domain:
All things considered this is a pretty strong domain. It's a brand, has lots of links, .edu links, also ranks highly in Bing-powered Yahoo!, and has a PR 6 to boot.
Another cool thing about SEO For Firefox is that you can export the results into a .csv file for further research, processing and comparison. If you don't have a copy of Office for Mac or Windows already then, in keeping with the "free nature" of this post, you can use Open Office.
Maybe you know the sites you want to research already or maybe you want a graphical, side by side comparison of up to 5 sites in your market. You can use our SEO Toolbar to accomplish this quickly and efficiently. If you click on the green arrows on the right side of the tool bar, you are presented with a GUI for the processing of up to 5 sites at ones (screenshot below)
The comparison feature gives you access to key, relevant SEO metrics side by side for up to 5 sites.
So by now you should have a spreadsheet or three containing relevant data for the top sites on a particular keyword
Now that you've gotten some of the higher-level metrics out of the way, you can dive into examining the link profile of a competing site.
You can use free tools (or free versions of paid tools) to look at the links from a competing site, tools like:
Yahoo's Site Explorer
Blekko's SEO Tools
Open Site Explorer
While it's a good idea to get data from a variety of sources, and run them through a tool like Advanced Link Manager to get a full(er) picture of things, you can get some juicy data for free.
When doing competitive research for a keyword I want to know what the anchor text profile looks like. When I am doing competitive research on a domain there are other relevant data points like top pages, most linked to pages, and total number of unique domains linking at the domain or page (whichever is ranking).
Blekko and Open Site Explorer are the ones I use for targeted and quick anchor text distribution views. Yahoo! generally ranks the best links first and allows for a CSV export, Majestic's free account gives limited data on referring domains, top back-links, and top pages. So for the purposes of looking at anchor text, I prefer Blekko and Open Site Explorer.
Blekko has a link to SEO data and Links data, as shown below:
The Links selection will bring up a Yahoo! Explorer-like list of links, the SEO link option brings up a bunch of SEO data like:
links to the domain
links to the page
anchor text information
links broken down by geography
pie-chart, graphical representation of link data points
and other non-link related, but helpful, data (crawl data, site pages, etc)
The data is free, you get the data they offer without registration requirements.
Open Site Explorer
Open Site Explorer is a quick and easy way to get the type of data we are looking for in this example (anchor text profile).
They currently have a 30 day trial and offer 3 plans:
Free, No Registration - limited to 3 reports per day, shows up to 200 links and top 5 link metrics for a given criteria
Free, Registration Required - no limit on reports, 1,000 links returned, top 20 link metrics for a given criteria (anchor text, top pages, etc)
PRO - part of subscription to SeoMoz, up to 10k links, no limit on metrics
CSV export available for all plans
If you know the sites you want to look at, and the keyword(s), you can likely get away with just using it as a guest. However, the free but registered plan does give you a bunch more data. What I like in this example is that you basically type the domain name in, hit enter, then click on the anchor text distribution tab and the anchor text data is right there:
You'll see the actual anchor text, the number of domains linking with that anchor text, and the total links with that anchor text in them (good way to spot site-wides from one domain). In this example, our target keyword is not in the top 5 (or 20) with respect to anchor text occurrences. This domain is a large brand though, so you'd likely want to make sure you could build an authoritative and useful site about the topic in order to overcome Google's love affair with brands.
Checking the On-Page Optimization
Though I believe the link data and domain data to be mostly paramount, the on-page criteria follows closely in the importance department.
This is pretty self-explanatory and you don't really need a full blown tool for this. Basically you'll want to look at things like the title tag, meta description tag, and the on-page copy itself.
You can do that pretty easily with just your eyeballs, but the SEO Toolbar also has a feature where you type the keyword into the box in the upper-right, and click the highlighter:
In this case I used 2 words, and they are highlighted in different colors:
This can give you an idea of how the site is using the copy to say, scream, or shout what the page is about. Sometimes you'll find that sites might just be ranking for a keyword or phrase based on the authority of their domain. If they are ignoring the on-page and off-page (links) for a keyword, it could signal to you that this might be a keyword worth pursuing and a keyword you can reasonably expect to rank for.
Making it a Process
Competitive research is just one piece of the puzzle, as you know. I find that breaking the entire process down into manageable chunks can help each process be more productive and efficient. This would be my process when researching the competitiveness of a keyword. While there are other pieces to your SEO research you should note that you do not need to spend hundreds or thousands of dollars on fancy competitive research tools off the start.
Save the money you might spend on tools on link development, content development, and content promotion.
In the last post I mentioned how the US government tried to change the cost benefit analysis for some sleazy executives at pharmaceutical corporations which continue to operate as criminal enterprises that simply view repeated fines as a calculable cost of doing business.
If you think about what Google's Panda update did, it largely changed the cost-benefit analysis of many online publishing business models. Some will be frozen with fear, others will desperately throw money at folks who may or may not have solutions, while others who gained will buy additional marketshare for pennies on the Dollar.
"We actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side." - Matt Cutts
Now that Google is picking winners and losers the gap between winners & losers rapidly grows as the winners reinvest.
And that word invest is key to understanding the ecosystem.
Beware of Scrapers
To those who are not yet successful with search, the idea of spending a lot of money building on a strategy becomes a bit more risky when you see companies like Demand Media that have spent $100's of millions growing an empire only to see 40% of the market value evaporate in a couple weeks due to a single Google update. There are literally thousands of webmasters furiously filing DMCA reports to Google after Panda, because Google decided that the content quality was fine if it was on a scraper site, but the exact same content lacked quality when on the original source site.
And even some sites that were not hit by Panda (even some which have thousands of inbound links) are still getting outranked by mirroring scrapers. Geordie spent hours sharing tips on how to boost lifetime customer value. For his efforts, Google decided to rank a couple scrapers as the original source & filter out PPCBlog as duplicate content, in spite of one of the scrapers even linking to the source site.
Outstanding work Google! Killer algo :D
Even if the thinking is misguided or an out of context headline, Reuters articles like Is SEO DOA as a core marketing strategy? do nothing to build confidence to make large investments in the search channel. Which only further aids people trying to do it on the cheap. Which gets harder to do as SEO grows more complex. Which only further aids the market for lemons effect.
At the opposite end of the spectrum, there are currently some search results which look like this
All of the colored boxes are the same company. You need a quite large monitor to get any level of result diversity above the fold. The company that was on the right side of the classifier can keep investing to build a nearly impenetrable moat, while others who fell back will have a hard time justifying the investment. Who wants to scale up on costs while revenues are down & the odds of success are lower? Few will. But the company with the top 3 (or top 6) results is collecting the data, refining their pitch, and re-investing into locking down the market.
Much like the Gini coefficient shows increasing wealth consolidation in the United States, search results where winners and losers are chose by search engines creates a divide where doing x will be very profitable for company A, while doing the exact same thing will be a sure money loser for company B.
Thin Arbitrary Lines in the Sand
The lines between optimization & spam blur as some trusted sites are able to rank a doorway page or a recycled tweet. Once site owners know they are trusted, you can count on them green lighting endless content production.
Scraping the Scrape of the Scrape
Many mainstream media websites have topics subdomains where they use services like DayLife or Truveo to auto-generate a near endless number of "content pages." To appreciate how circular it all is consider the following
a reporter makes a minimally informing Tweet
Huffington Post scrapes that 3rd party Tweet and ranks it as a page
I write a blog post about how outrageous that Huffington Post "page" was
SFGate.com has an auto-generated "Huffington Post" topics page (topics.sfgate.com/topics/The_Huffington_Post) which highlighted my blog post
some of the newspaper scraper pages rank in the search results for keywords
sites like Mahalo scrape the scrape of the scrape
At some point in some such loops I am pretty certain the loops start feeding back into themselves & create a near-infinite cycle :D
An Endless Sea of "Trustworthy" Content
The OPA mentioned a billion dollar shift in revenues which favors large newspapers. But those "pure" old-school media sites now use services like DayLife or Truveo to auto-generate content pages. And it is fine when they do it.
The newspapers call others scammy agents of piracy and copyright violators for doing far less at lower scale, all while wanting to still be ranked highly (even while putting their own original content behind a paywall), and then go out and do the exact same scraping that they complain about others doing. It is the tragedy of the commons played out on an infinite web where the cost of an additional page is under a cent & everyone is farming for attention.
And the piece of pie everyone is farming for is shrinking as:
competition increases faster than the growth of the market
Aware that consumers spend someplace between eight and 10 hours researching cars before they contact a dealer, auto markers and dealers are vectoring ever-greater portions of their marketing budgets into intercepting consumers online.
As but one example, Ford is so keen about capturing online tire-kickers that its website gives side-by-side comparisons between its Fiesta and competing brands. While you are on the Ford site, you can price the car of your dreams, investigate financing options, estimate your payment, view local dealer inventories and request a quote from a dealer.
Search Ads Replacing the Organic Search Results
AdWords is eating up more of the value chain by pushing big brands
comparison ads = same brands that were in AdWords appearing again
bigger adwords ads with more extensions = less diversity above the fold
additional adwords ad formats (like product ads) = less diversity (most of the advertisers who first tried it were big box stores, and since it is priced on a CPA profit share basis the biggest brands that typically have more pricing power with manufacturers win)
Other search services like Ask.com and Yahoo! Search are even more aggressive with nepotistic self promotion.
Small Businesses Walking a Tightrope (or, the Plank)
Not only are big brands being propped up with larger ad units (and algorithmically promoted in the organic search results) but the unstable nature of Google's results further favors big business at the expense of small businesses via the following:
more verticals & more ad formats = show the same sources multiple times over
less stability = more opportunities for spammers (they typically have high margins & lots of test projects in the work...when one site drops another one is ready to pop into the game...really easy for scrapers to do...just grab content & wait for the original source to be penalized, or scrape from a source which is already penalized)
less stability = lowers multiples on site sales, making it easier for folks like WebMD, Quinstreet, BankRate, and Monster.com to buy out secondary & tertiary competing sites
If you are a small business primarily driven by organic search you either need to have big brand, big ego, big balls, or a lack of common sense to stay in the market in the years to come, as the market keeps getting consolidated. ;)
I got an email the other day titled "small business SEO scam"
My name is Ryan, and I head small business outreach for ConsumerAffairs.com.
Recently, we started receiving a rash of complaints from small business owners concerning illegitimate SEO consulting companies they have used.
These small business owners are paying hundreds (in some cases thousands) of dollars to have these SEO consulting companies remove negative comments from ConsumerAffairs.com.
However, these small businesses are being taken advantage of - ConsumerAffairs has no relationship with these SEO firms and there is no way for them to remove comments/reviews about their firms from our site.
ConsumerAffairs is very concerned that small businesses are being mislead by these SEO firms, and we are trying to get the word out through small business resource blogs, such as yours.
I do not know if you have heard about this scam, but we hope you can help us get the word out and possibly even blog about this. We recently published an article about this if you would like to read more about this topic consumeraffairs.com/news04/2011/04/bogus-complaint-removal-sites-prey-on-small-businesses.html
Also, in response to these complaints, we have launched the ConsumerAffairs.com Accredited Business Program. Under this program we alert the small business owner when a consumer submits a review/complaint, and the company is given the ability to respond to the consumer.
ConsumerAffairs realizes the majority of SEO firms do incredible work for small businesses and in no way are we grouping these illegitimate firms with all SEO firms.
If you have any question please feel free to contact me,
The Problem With Complaints Websites
Here is the problem with complaints sites though: so many of them cover all the "evils" of the marketplace without ever covering the positive sides of said industries. The email solicitation to me states that they are aware that the majority of SEO firms do incredible work, but searching their site comes up empty for any such recognition, just complaint after complaint.
And who was promoting the fake news sites? None other than the mainstream media (which even promotes the scams on articles about avoiding scams like SEO)!
In response to one such hate bait SEO article from a sleazy polarizing news organization I posted about it and flamed them, writing "If people talk trash, lie, and misinform consumers about a topic often enough then they destroy some of the perceived value of that field. Maybe you don't work as hard as I do and maybe you don't help out as many people as I do. But I work way too hard to just not care when a bunch of sleazeballs trash my trade by pumping biased misinformation through a megaphone."
In spite of the above outrageous behavior from the "news" organization, some current & former employers from that news organization requested that I update my post to make nuanced clarifications. In other words, they wanted to hold me to a FAR HIGHER journalistic standard than their own employer IN THE FIELD OF JOURNALISM!
This is how our current model of capitalism works, if you are not large and parasitic then you need to be labeled as the scammer so the media can paint themselves as the great protector. This behavior is by no means new. Reading up on the history of Western Union & the Associated Press around the time of the Hayes election would make one queasy.
Who is the Scammer?
Back to the reputation management "SEO scam" mentioned at the top of this article, if a small business thinks they can pay someone a couple hundred Dollars to fix their bad reputation & it doesn't work then were they really scammed? Weren't they really trying to manipulate the market for pennies on the Dollar? Isn't getting scammed the expected outcome when you under-pay for services?
Also oddly enough, the above complaints site which was out to inform consumers about scams embeds inline AdSense so aggressively that many folks likely can't tell where the content ends and the ads begin
Since those are Google ads, they are contextually relevant & the articles about "scam x" often contain ads with pumped up ad copy for the very services that the article allegedly warns against. Not surprising considering that Google AdSense has a "get rich quick" category.
Why is it that such consumer "protection" services can run ads in the content & simply fall back on this "Advertisements on this site are placed and controlled by outside advertising networks. ConsumerAffairs.com does not evaluate or endorse the products and services advertised. See the FAQ for more information." in the footer? If they wanted to protect consumers, wouldn't they also give you links to report bad ads and/or solicit feedback on them and/or claim some responsibility for them and/or not blend them so aggressively in the content area of the page?
That FAQ page states
I see ads for companies that are criticized on your site. What's that all about?
We don't control which ads appear on our site. They are placed by outside agencies. The fact that an ad appears on our site by no means indicates we approve of the product. Same thing's true for an ad in the newspaper, or on television or radio.
This seems wrong. How can you take money to advertise products you don't approve of?
It's a free country. Companies, even the ones we don't much like, have as much right to advertise as we do to publish our site, just as we have the right to publish critical comments about them.
How is it that if you don't disclose an affiliate relationship for a 3rd party some people will view you poorly, while the media can run on a "hear no evil, see no evil" approach to monetization? Eric Janszen recently highlighted how this isn't an accident:
Assume the laws of human nature are in force and you are not getting the truth when a powerful and politically connected industry is in crisis. It took decades for the health risks of tobacco to come to light. The media was no help until the tobacco industry was already on the ropes. Once cigarette advertising was widely banned and the advertising revenue dried up, it was safe for the media to cover the obvious dangers of a product that killed millions. Only then did the media join in on the side of consumers.
Are Standards a Good Idea?
In spite of the bizarro way that the media world operates (screw whoever you can while claiming you are ignorant that you are selling them down the river) some folks who are concerned about the state of the SEO industry think they can fall back on industry standards. Industry standards are no real solution though:
most people who operate such organizations push self-promotion aggressively (anyone remember the SEMPO tiers with the inner circle at $5,000 level, but free to certain folks?)
such self-promotion also aligns with business biases (remember how early "research" out of such organizations aggressively promoted paid search while making SEO seem like an also-ran?)
those who are desperate need to do "whatever it takes" and that would make standards irrelevant to them. Consider this following "dear team" email I got from a non-customer
As sad as that email is (telling me they are ignorant of SEO, yet are taking on SEO clients, yet need me to do it for them) it is actually worse than it appears at first blush. Why? The anchor text they wanted me to get was for keywords about SEO, so some SEO who is claiming to sell "professional" SEO services is paying dirt to some poor third world worker & is having that person optimize the SEO's site! If their services for their own sites are that bad imagine what they must be doing for clients!
we already have a set of standards (in Google's guidelines) but they are already selectively enforced, an additional layer would do nothing but inhibit potential
some projects have different risk and reward potentials
The table is already tilted toward certain types of sites. If you agree to an across-the-board arbitrary standard you cede marketshare to those chosen few. Matt Cutts said: "we actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side." Some search results already look like the following, where the top brand has 3 AdWords ads and the top 3 organic results
Nearly 300 people fell ill in central China after eating meat suspected of containing illegal additives, the latest in a spate of contamination problems to emerge even as the government vows to crack down on food-safety violators.
The state-run China Daily newspaper blamed clenbuterol, a substance that speeds muscle growth in pigs but can cause headache, nausea and an irregular heartbeat when consumed by humans.
People may be a bit more careful with eating some types of meat in China in the near-term, but based on that news story you don't get an immediate "OMG never eat pork" reaction. Yet so many of the scams in the online space (even those funded by Google & those not directly related to SEO) are conveniently labeled as SEO scams.
I was recently emailed by a PR firm working on behalf of Pfizer, which wanted to make a "documentary" about the escalating issue of counterfeit drugs. They are concerned about legality and consumer safety when someone else is making money, but you know what Pfizer has repeatedly had no problem with? Pushing drugs for off-label purposes:
New York-based Pfizer agreed to pay $430 million in criminal fines and civil penalties, and the company’s lawyers assured Loucks and three other prosecutors that Pfizer and its units would stop promoting drugs for unauthorized purposes. What Loucks, who’s now acting U.S. attorney in Boston, didn’t know until years later was that Pfizer managers were breaking that pledge not to practice so-called off-label marketing even before the ink was dry on their plea.
On the morning of Sept. 2, 2009, another Pfizer unit, Pharmacia & Upjohn, agreed to plead guilty to the same crime. This time, Pfizer executives had been instructing more than 100 salespeople to promote Bextra, a drug approved only for the relief of arthritis and menstrual discomfort, for treatment of acute pains of all kinds.
The pharmaceutical industry has paid billions of dollars in civil and criminal penalties over the past decade, but the government believes they no longer have much deterrent effect.
The new use of exclusion is meant to "alter the cost-benefit calculus of the corporate executives," said Lew Morris, chief counsel for the Department of Health and Human Services's inspector general, in congressional testimony last month.
Scammers operate anywhere there is money to be made. They will even claim to follow standards, while doing every dirty thing in the book. But it doesn't mean that everyone in those markets are scammers simply because their business model doesn't have the margins and scale needed to pay off the mainstream media.
When Google began speaking publicly about content farms Demand Media's Richard Rosenblatt stated that it would be silly to call their stuff a content farm & he emphasized the quality of their content & care that went into it. Of course, those who bothered looking at the content often saw something different
Panda II Hits Demand Media
When Google did the global roll out of Panda earlier this month, they also modified their approach to core Panda algorithm to include user block data:
Today we’ve rolled out this improvement globally to all English-language Google users, and we’ve also incorporated new user feedback signals to help people find better search results. In some high-confidence situations, we are beginning to incorporate data about the sites that users block into our algorithms. In addition, this change also goes deeper into the “long tail” of low-quality websites to return higher-quality results where the algorithm might not have been able to make an assessment before. The impact of these new signals is smaller in scope than the original change: about 2% of U.S. queries are affected by a reasonable amount, compared with almost 12% of U.S. queries for the original change.- Amit Singhal
While many of Demand Media's sites got dinged in the first update, the fall of content farms in general meant that any site operating in that space which was not hit ended up seeing a sharp increase in traffic (as so much of the competition fell). As sites like AnswerBag and Livestrong fell, eHow's traffic increased significantly. I believe Google didn't want to rely on end user block data because it would make it easy for people to do competitive sabotage, however I think they needed to use it in order to hit eHow with the update. eHow had a number of signals (some older quality content, nice web design, syndication partnerships, tons of media exposure, etc.) which made it hard to whack it without creating too much collateral damage unless the block data was used.
In the first two weeks of January, 0.57 percent of those who departed Google next visited a site operated by Demand Media ... by mid-April, with the full suite of Panda updates in place, Demand was feeling the pain. As of April 16, it accounted for only 0.34 percent of Google’s downstream, a 40 percent decline from the start of 2011.
Demand Media's Stock Falls 40%
Incidentally, over the past couple weeks Demand Media's stock is off roughly 40%
That article (which claimed eHow to be profitable as hell, a fuzzy claim depending on how one accounts for content depreciation) was aimed at trying to position Demand for an IPO and to try to pull in more media syndication partnerships.
What it did was inflame the web community & encourage others to play the same game & create content farms based on the blueprint Demand gave away. When a piece of marketing either pisses off almost everyone & encourages many of the people who are not pissed off to compete directly against you & cut your margins it is not a successful marketing approach.
A reason it was so easy for journalists to claim bad things about Demand Media was that the wages were so low that they didn't practically allow for any in-depth research to be done (unless a person was willing to work far below minimum wage). Thus when journalists started to dig into eHow's business model they got eHow writers to state things like:
"I was completely aware that I was writing crap," she said. "I was like, 'I hope to God people don't read my advice on how to make gin at home because they'll probably poison themselves.'
"Never trust anything you read on eHow.com," she said, referring to one of Demand Media's high-traffic websites, on which most of her clips appeared
The larger your scale is the easier it is to find something wrong with what you are doing. 1% of a really big number is much greater than 10% of a rather small number. If you are cutting corners & operating at scale & create a lot of enemies then I wish you the best of luck, because you are going to need it!
In spite of letting a few things fall through the cracks, to this day there are some OUTRAGEOUS eHow titles. A friend showed me a couple and after 5 minutes of searching I found:
How to Pick Your Nose The Proper Way ehow.com/how_5722363_pick-nose-proper-way.html
How to Pick Your Nose or Scratch Surreptitiously ehow.com/how_2181862_pick-nose-scratch-surreptitiously.html
How to Effectively Pick Your Nose ehow.com/how_5067366_effectively-pick-nose.html
Exploring Other Orifaces
How to Fart ehow.com/how_2151823_fart.html
How to Stop Farting ehow.com/how_4785860_stop-farting.html
How to Muffle a Fart ehow.com/how_2320127_muffle-fart.html
How to Poop in the Woods ehow.com/how_2179463_poop-woods.html
How to Not Get an Ehow Article Erased ehow.com/how_5570908_not-ehow-article-erased.html
How to Slack at Work (and not get caught) ehow.com/how_4522164_slack-work-not-caught.html
How to Slack Off at Work and Not Get Caught ehow.com/how_4837878_slack-off-work-not-caught.html
How to Do Nothing at Work and Still Get Paid ehow.com/how_4430256_do-nothing-work-still-paid.html
Honing Your Social Graces & Charm School
How to Manipulate People to do Your Bidding ehow.com/how_2167832_manipulate-people-do-bidding.html
How to Get a DUI ehow.com/how_4825159_get-a-dui.html "You might think getting a DUI is as easy as getting behind the wheel of a car after drinking alcohol. But that's only half the battle. You also need to get pulled over by law enforcement and cited for it."
How to Not Be a Husband Caught Cheating ehow.com/how_5528899_not-husband-caught-cheating.html
"Don't leave trails which can and will turn into signs you're cheating. First point to remember is to not use any computer your partner has access to when you communicate via email or IM to the cohort."
AdSense Click Fraud
How to get banned from Google Adsense ehow.com/how_5740892_banned-google-adsense.html
How to Increase Your Click Through Rate with Google AdSense ehow.com/how_5203081_increase-through-rate-google-adsense.html
How to not get Caught With Google Adsense Click Fraud ehow.com/how_5979999_not-caught-google-adsense-fraud.html
Leveraging Expired Domains
Demand Media bought out a leading domain registrar named eNom & leveraged some of the expired domains with links to prop up eHow, by 301 redirecting those domains into eHow's deep pages.
eHow was not only churning out loads of shallow content, but Demand Media was also using the data gleaned from eHow to make sister sites which included auto-generated pages and feeding search engines their own results.
They Made Google Look Stupid
Doing one thing and claiming another can provide cover for some finite period of time, but ultimately when you create such a spectacle out of Google that your exploitative ways become the core marketing message for Google's competitors you know your days are numbered. And given that the Wired piece made the media hate Demand Media, there was nobody left to defend them other than folks who would also seem in some way conflicted.
Ultimately this goes back to the core issue that hurt Demand Media: branding.
Don't make Google look stupid. That is the #1 rule of SEO.
Much noise was made recently about Google taking a whack at so-called content farms -- sites which apply industrial production techniques to the creation of content targeting the long-tail of the query distribution. This is a subject of huge interest to many Internet businesses, either because they advertise on the AdWords Content Network (and, by extension, on content farms), because they compete with content farms on particular searches, or merely because they hate seeing content farms in their search results. As luck has it, I am three for three. It pains me to say it, but content farming is here to stay. It is an economic inevitability.
The Attention Economy
Much of the Internet currently operates in an attention economy, a level or two removed from direct monetization. Facebook is worth in excess of 50 billion not just because they're making money hand over fist -- though they are -- but because they have achieved a dominant position in the attention economy, and they command such huge rivers of attention that they can trade trickles of it to people for actual money.
Google is the dominant player in the attention economy -- they harvest vast amounts of attention via controlling navigation on the Internet (via a commanding lead in search), they sell attention in the form of AdWords ads, and they provide a marketplace for attention with their AdSense product.
Individual publishers -- from the New York Times down to the smallest hobbyist site on the Internet -- are also largely in the attention economy. For a mega-brand like the New York Times, attention can be generated -- they can literally make news. Disney has a repeatable industrial process which takes as input one female teenager and produces as output a cultural phenomenon with hundreds of thousands of rabid fans.
Smaller players -- Google back in the dorm room days or hobbyist sites today -- largely cannot create attention on these scales, they can only harvest attention which already exists. Attention exists in the world for things independent of their own existence. People play golf. People bake cookies. People read Dan Brown novels. People receive massages. For all these things and more, people demand content: they want to improve their golf swing, they want new cookie recipes, they want new Dan Brown novels, they want massage how-to videos. And they are willing to pay with attention, a scarce commodity which can be converted into cash.
The Economics of Content Creation
Consider a hypothetical Internet with no efficient way of converting attention into money. This is not difficult to imagine: it was essentially the Internet of the dot-com bubble, where everyone wanted "eyeballs" but "eyeballs" plus banner advertising resulted in economically non-viable businesses. In this hypothetical Internet, content is mostly produced by people who have intrinsic reasons for creating it: hobbyists who want to share their passion, law professors who want to increase their professional reputation, governments who need to employ somebody and might as well employ a webmaster, and the like. This is widely viewed as a Garden of Eden scenario: the Internet, without the corrupting influence of money.
We had this Internet, and the average user experience was miserable.
Ability to publish content on the Internet was once dominated by presence of arcane technical skills (being a "webmaster", a title which thankfully has fallen out of fashion). Webmasters were, by and large, very geeky people. They largely scratched their own itches, which (predictably) resulted in an Internet chock-full of Dungeons and Dragons character sheets, trivia about Matter-Eater Lad, and fansubbed anime episodes.
Less well-represented on the Garden of Eden Internet was content appealing to demographics which don't intersect with geeks that often. Women, the very young, the elderly, non-English speakers, etc etc, were across a very real digital divide from the D&D players. You could still find advice on how to make an apple pie online, but if you did, it was because you got lucky and had a CS professor with quirky interests (for a CS professor, at any rate).
This started to change with the widespread adoption of content management systems, which took the level of computer skill for content creation down from "close to programming" to "close to using a word processor." The first very popular CMSes were blogs, and there was much triumphantalist backslapping among bloggers that blogging was democratizing the Internet. You could be blogging in your pajamas and still take on the New York Times, or so the argument went.
Ability to use a word processor is more widely spread among the population than webmastering skills, but it is still a far cry from universal. Blogging caught on primarily with professional communicators: professors, journalists, and other folks who had long been using skill with the printed word and perceived authority with pre-existing audiences. Concurrent with this, there was an explosion of content creation aimed at the concerns of well-educated, middle-class American white urban professionals. Politics, financial advice, education, religion, international news: covered, covered, covered, both by established media and publishing interests moving online and by the new media (rather like the old media, except with orders of magnitude lower capital requirements). Content was now a democracy, in the same sense that America after the Revolution was a democracy: white property owners could be reasonably assured of having their interests represented.
There still existed massive demand -- unharvested attention -- for content outside the early adopters of the Internet. Larger scale online publishers began to go after the head of the demand distribution, and hobbyist sites continued to publish things like apple pie recipes, often with a quantum leap in presentational quality from just a few years previously. Google AdWords was one of the primary lubricants for making this happen -- a hobbyist site dominating a niche like e.g. apple pies could suddenly generate non-trivial amounts of money for the site owner, largely by taking transaction costs about negotiating advertising sales out of the equation. This also allowed Google to monetize its own attention surplus better, because sending a searcher to a site with AdSense on it gives them a second chance at getting paid for a click. AdSense has generated roughly a third of Google's revenue for the last several years.
The Industrialization Of Content Production
With technology continuing to bring down barriers to creating content and business model optimization like AdWords improving the opportunity to monetize attention, it was virtually inevitable that eventually the supply and demand curves would cross. They long since had for high-value verticals like e.g. mortgages, where huge transaction volumes, high margins, gigantic advertising spends, and liquid affiliate/lead gen markets have long subsidized huge volumes of content creation. Many quite savvy Internet users were simply unaware this had happened, since one does not search for mortgages or poker every day. The Internet is a virtually uncountable multitude of attention markets, and in many of them it was more expensive to create content than the harvestable attention could justify. Those niches continued to be underserved, in the capitalist sense of the word: people would have consumed more content for them, but that content did not exist.
Then disruptive innovation happened: basically, a number of firms figured out that the combination of algorithmically predicting attention plus outsourcing content creation could let them exploit relatively small amounts of attention, in parallel, at massive scales. This innovation caused the supply and demand curves to cross for a huge number of attention markets which had not crossed before. The result: content farming at massive, massive scale.
Consider bingo cards for elementary schoolteachers, a very niche subject that happens to pay my rent. Attention exists for it: bingo has long been used in American classrooms to review vocabulary across a variety of subjects. As teachers and parents gradually started using the Internet and using Google, their attention about bingo -- a tiny, tiny sliver of the massive river of attention Google controls -- became up for grabs. Some flowed to hobbyist sites like my own, some flowed to larger publishers like the NYT's About.com unit, and some was simply poorly served. Teachers typed queries into Google and got garbage results which were not responsive.
I have advertised on Google's AdWords Content Network for years, and for the last four years I've been essentially willing to buy as much traffic as Google cares to sell me for a range of quality below a given price. This makes my AdWords stats a proxy for who is getting traffic for bingo-related searches. (Google controls navigation on the Internet, so the presence of traffic for near-term desires like bingo cards strongly suggests that it was searched for. Check your Analytics if you don't believe me.)
My market has massive seasonal changes in attention, so let's look at consistent month-long slices of it, compared year-to-year. Here's a tale of four Februaries.
In 2008, my AdWords spend was dominated by legacy Internet publishers like About.com, niche publishers in education, and hobbyist sites. Total spend was about $370, of which About captured almost $70 (~19%).
In 2009, hobbyist sites and niche publishers decline with the ascendancy of a new publisher called Kaboose, an early iteration of a content farm, focused on topics of interest to women (including, e.g., bingo). Total spend was about $560, of which Kaboose captured almost $160 (a whopping 29%), more than quintupling their performance from 2008. Or, to put it another way, more than half of increase in the size of this small attention market can be attributed to one publisher. 2009 also sees a new site in my top 10: a minor player called eHow run by an obscure firm Demand Media.
In 2010, spend again increases (to $640), and the top positions are dominated by content farms and ezinearticles, a legacy crowdsourced content farm. Kaboose loses share to new content farm entrants, and eHow has comparatively modest 50% year over year growth. Content farms now control over a third of this attention market.
In 2011, spend again increases (to $920 -- nearly 50% year over year growth), and content farms dominate the attention market. eHow has improve its execution again, to the point where they singlehandedly capture $150 in ads, quintupling performance from a year before. (Yep, their revenue is now ten times what it was in 2009.)
The Microeconomics Of Content Farming
Why did content farming capture so much of the attention economy so quickly? Basically, once the process for creating content very responsive to a single search term was repeatable, it could be replicated down the long-tail very, very quickly, in response to market signals such as e.g. successful pages in related searches. My business has long had a page about Valentine's Day bingo cards because I know, being a publisher in the niche, that they're very valuable -- there exists a substantial amount of attention which will be paid to Valentine's Day bingo every February. Do you think Valentine's Day bingo cards is a tiny niche, on Internet scales? eHow has over thirty pages targeting variants on this top -- thirty slices of a fraction of a tiny niche which were worth individualized effort to target. Some representative titles:
Church Valentine's Party Games
Make Valentine Bingo Cards
Classroom Valentine's Day Party Games
Valentine's Math Games
Christian Valentine's Games
Christian Adult Valentine's Games
Valentine's Party Games For Older Kids
etc, etc, etc, etc
Zooming in on the performance of just one of these pages, about Valentine's bingo for churches, I paid $9 for ads on it in February 2011. If we make the unreasonably pessimistic assumption that it never makes money except in February, and that the remnant image advertising is basically a wash (not true, given the amount that Groupon and online games throw around at monetizing it), this suggests that the four text ads on the page probably generated on the order of $30 in revenue. Google's 68% revenue share means that Demand Media got about $20 in revenue from this page... in 2011 alone.
Content farms are targeting evergreen content, though: Valentine's Day is going to happen in 2012, and there will still exist churches who want to play bingo on it. Will revenue from this page go to zero? That is highly unlikely, because this page wasn't written in 2011 -- it was written in 2010, when I paid $1 for ads in it (implying about $2 in revenue). Due to changes in the search environment and Demand Media's increasing sophistication with leveraging internal traffic, it got nine times more valuable at no marginal cost in the course of a single year.
Content farms operate on a portfolio strategy: the pieces of content which succeed, like that page, subsidize the pieces of content which don't. As long as the average revenue portfolio-wide exceeds cost of content production, one should expect the content farms to pour capital into content production and scale it to the moon. The portfolio strategy appears to be winning, judging by eHow's meteoric rise in revenue and the demonstrated ability for content farms to choke out non-farming content sources. eHow alone showed my ads on five times as many pages in 2011 as in 2010.
And why wouldn't they? The unit economics of content farming are stunningly attractive. Demand Media pays on the order of $10 to have the 312 words on that page written and edited. If Wall Street could design an equity which cost $10 and paid $2 per share in 2010, $20 per share in 2011, and an unknown but positive amount thereafter, all other investment classes would be virtually obsolete. The only problem is systemic risks.
The only thing that can reverse this is content getting more expensive to create or attention getting scarcer (or harder to monetize) for these markets.
There is more attention to monetize: It is possible that Internet use will decline in the future, but I will offer excellent odds to anyone who wishes to bet that: Kansas schoolmarms in the elementary bingo market have quite a ways to go before they catch up to the average reader of this blog in online consumption, which predicts a large aggregate increase in attention harvestable on the Internet and even larger proportional increases to the attention markets they care about.
Google and advertisers increasing cost of attention: Ignoring huge sources of attention of dubious worth, like ads displayed next to Facebook games, an AdSense ad displayed to someone 2 seconds after they type in a query into Google is, essentially, a search ad.
Read that again, because it is important.
This means that content farms are essentially in the search ad monetization business -- i.e. the most profitable business in the history of the Internet. Search ads monetize extraordinarily well because in addition to capturing user attention they come with user intent. This makes them orders of magnitude more valuable than the old banner display networks (which users quickly become blind to), sidebar ads next to Farmville or pictures of that cute girl from chemistry class, and the like. Content farms preserve search intent because the laser targetting combination of their one-topic pages and AdSense means that the AdSense ads are guaranteed to be responsive content to the search and, give that everything else on the page was written by a content farm, the ads are the best content on the page.
Sure, farms cede a large portion of the reach of search to actual search engines, since they can't rank for head queries, but even 5% of Google's market cap would be nothing to sneeze at. Google has incentives to help them rather than competing with them. Meanwhile, any market with competitors will tend to drive the cost of ads up until they have expended all of their margin on the sale. For a high-margin category like software, if my competitor is willing to pay 51% of his sale price to generate one marginal sale (when you back it out to cost-per-click prices), I'm willing to bid 51%. The equilibrium outcome is that my advertising costs increase over time while my ROI decreases, but it remains profitable and I'd be a fool not to do it. Google and their publishing partners win and win big.
This Is Old News. Google Fixed Content Farming... Right?
Back in late February 2011, Google rolled out the Panda update, which was widely perceived to be aimed at content farms. What actually happened was that it separated Content Farming 1.0 from Content Farming 2.0 -- earlier entrants like ezinearticles and Mahalo (and a raft of sites you've never heard about) lost out to better executing farms, including eHow.
For example, instead of comparing Februaries like we did earlier, let's see the progression of Marches in the bingo niche. Largely due to the absence of Valentine's Day, March consistently has less attention available than February: aside from an anomalous 2008 (long story with short moral: don't bork your AdWords code), spends fell 28% in 2009 and 17% in 2010. The decline was much more pronounced in 2011, possibly attributable to Panda reshaping the attention economy landscape: it jumped to 37%.
However, the performance of individual publishers was mixed:
eHow (Demand Media) declined only 18%. This looks virtually in line with historical seasonal trends (growth in 2010 was so fast they were actually flat over the interval, i.e. growing much faster than market). Their performance in March 2011 (historically a "bad" month for bingo attention) crushed their performance in February 2009 (historically a "great" month for bingo attention). One could be excused for believing eHow was not net-affected by Panda
LoveToKnow got annihilated -- spend decreased 71%. (The comparable decrease in 2010 was only 25%.)
ezinearticles got annihilated -- spend decreased 68%. (The comparable decrease in 2010 was only ~10%.)
About.com was severely affected -- spend decreased about 46%. (The comparable decrease in 2010 was, again, lower -- only 21%.)
Summed over all the content farmers, Panda appears to have picked a winner with regards to this slice of the attention economy: eHow.
I had been wistfully hoping that when the content farms got crushed that my site, which competes with them for many queries, would pick up some of the redistributed attention. If this happened, it has been too minor to notice in my Analytics stats -- my organic searches from Google look roughly in line with where I would expect them to be absent Panda. The big winner and the big losers appear to be concentrated among farmers, with fairly minor spillover to the rest of this sliver of the attention economy. This makes sense to me, in a way -- I simply don't have a page which is more responsive to the need for church Valentine's bingo activities than eHow's does. I believe my pages are far and away better than eHow's -- my pages about making bingo cards will actually let you make bingo cards -- but reasonable people could disagree on whether that is more important than capturing all parts of the user's intent, including the "specifically for churches" bit of it. Outside of my narrow slice of the online experience, a Big Publisher advocacy group estimates that the Panda update redistributed $1 billion in advertising revenue, which is nothing to sneeze at. However, with the Content Network generating over $20 billion in annual ad sales, $1 billion looks less like a fundamental shift and more like repartitioning scraps left to the losers.
Did Panda Kill Farming?
Only the economics can kill farming, and it does not appear that Panda meaningfully changes the microeconomics of content farms. If you can sell $40 of ads against a $10 page prior to Panda, and after Panda you can only sell $20 of ads, well, farm on. The model scales to the moon as long as the portfolio is even marginally profitable. The losing farms will also be incentivized to go back to the drawing board and reevaluate where they place their content bets: perhaps it is no longer lucrative enough for them to go after certain micro-markets, like elementary school bingo cards (or like the bottom half of elementary school bingo cards), but their more valuable markets are probably still stupidly profitable. Those will get more competitive as they redeploy their content creation resources going forward, assuming they're capable at executing on that.
Demand Media, on the other hand, is grinning like the cat that just caught the canary. Not only is their core business proposition virtually unaffected, in spite of the worst nightmare of their business model (Google coming down on it like the fist of an angry god) coming true, their unit economics down the tail just got radically better. Going forward, they can expect less competition in the less lucrative markets, allowing them to capture larger fragments of the attention available in those markets, and proportionally higher revenues.
The Future: Outfarming The Farmers?
A frequent theme of dystopian science fiction is that man-machine hybrids outcompete the human race. Algorithmic/freelancer hybrids, like content farms, are pretty much there, for a large and increasing portion of the content tail. This is going to get exacerbated by changes in content production and consumption, such as the rise of video (which has orders of magnitude higher production costs than text) and decline of hobbyist content creation. In 2006, my business had significant competition for keywords from individual teachers' sites, where Mrs. Smith decided (back in 1996) to put up a web page to try out this new Internet thing on her computer. In 2021, there will be many less sites created by Mrs. Smiths, because Mrs. Smith in 2011 now has an iPad to watch her Khan Academy videos on, and the iPad is virtually useless for creating websites. I'm already seeing anecdotal behavioral changes in my customers ("Say, how do I hook a printer up to an iPad so I can make my cards from there? I hate turning on the computer -- I think it has a virus or something, and it is slow."), and ordinarily they're quite behind the curve.
Additionally, while Mrs. Smith had sufficient dedication to the niche to target the most common activities, she never made more than 5 or so pages about bingo. I have about a thousand bingo activities, created with focused application of custom software by freelancers. The content farms are making me look practically lazy with their scale of publication.
This suggests an obvious route for improvement for me: if it is stupidly profitable for me to pay $200 to Google so that it can pay $130 to eHow so that it can pay $50 to freelance writers to create 5 pages, why don't I just take the $200 and pay freelance writers to write those same 5 pages... and then 15 more? The only thing which has stopped me from doing it so far is concern about polluting the Internet. But the economic attraction of doing it is undeniable. If the choice is a user getting their bingo content from an anonymous freelancer at eHow working through their queue of 400 articles for the week or getting it from someone who is only employed to write bingo articles, shouldn't they get it from me?
This dilemma, repeated a thousand times across a thousand markets, is going to create the Internet of 2020. Break out your straw hats, folks: we are all going to be farming or, at best, a step removed from farming by paying intermediaries (Google and the farms) to do our farming for us. The main distinction is going to be between successful execution of farming strategies (like eHow) and poor execution of farming strategies (like their competitors who recently got whacked).
Demand Media is already shopping out their business model as a service: since newspapers and other legacy publishers are a) dying but b) scare Google (because they can cause Google to have bad PR, which might result in government regulation, which is Google's sole competitive risk), Demand Media would love newspapers to be the front man for their farmed content. That, or parallel arrangements, is going to be almost irresistible to anyone with sufficient signals of trust to rank for arbitrary longtail content in their niche. I mean, "Create a repeatable process to create content of a known level of quality, throw money at the process to scale it, then sell ads against the result" is the entire newspaper business model! Content farming just takes out the sucky bits like "own a multi-billion-dollar distribution network for dead trees" and "write articles which are relevant to a few hundred people at an amortized cost of over $1,000 per article." (It is an open secret that the most lucrative ads in a newspaper aren't around the news, but are in sections like Style and Travel. It does not take Pulitzer Prize-winning journalism to write articles on this season's hottest shade of fuchsia or compelling reasons to go to Cancun. "Real" news has always been a loss leader to sell advertising against their other content. If they can create ten times as much fluff at a tenth of the cost, why not? And if they can... do they need the "real" news in the first place?)
Can Google just tighten the screws with another Son of Panda update? That is unlikely to work unless they repeal the laws of economics: farming happens on every topic for which the supply and demand curve crosses. Slashing content farm's ability to rank across the board by 40% just makes a fraction of the content space monetarily unattractive to them, but the content space is virtually infinite and the ability to monetize attention is increasing all the time. If Groupon will pay for remnant inventory on a page about How To Pick Your Nose, who will compete for that attention except a content farm?
Is that the Internet I want? Probably not. But then again, I'm privileged -- as a geek, my interests in content will always be well represented on the Internet, even without monetary incentives to create it. People will go to StackOverflow to answer my questions before I've even asked them, they'll create Starcraft XII walk-through videos, they'll even write software, all without seeing a penny for it. The experience for less privileged folks, though, demonstrably sucked at the dawn of the Internet, and it is not obvious to me that removing most of the growth in content responsive to their needs is a net win for them. We might see an Internet where the content-rich win and everybody else gets farming.
When your boss/client asks you why your web site can't be found in Google, what are you going to say? You should prepare, because, eventually, that question will be asked.
Should you cross that bridge when you come to it? Employ an SEO specialist to "do some stuff", after launch, in order to get the site ranked? Isn't SEO just another marketing function, like buying advertising?
In this article, I'll outline why it's a bad idea to treat SEO as an add-on. I'll look at how to roll SEO, seamlessly, into your web strategy.
These days, SEO is not a series of easily-repeated, technical steps.
You may have heard that SEO is about adding meta tags. Changing the underlying code. Making a few minor changes to content and submitting a site to a search engine. If you follow this process, your site will be found on the first page of search results.
This was true years ago ago. It isn't true now.
If that is all you do, chances are your site won't appear on the first page of results. It might not appear until page 72. If at all. The search engines have grown more sophisticated. They look at many different factors, and they don't place weight on meta-tags when determining rank.
One factor is the vote. In search, a link is a vote. In order to get people to vote for your site, you need a site that is link-worthy. And the voting box is rigged. A vote from a huge brand, like Microsoft, for example, is worth way more than many votes from sites few people have heard of. The search engines tend to reward popularity, as determined by other sites. If the information you're providing isn't popular enough, you won't be ranked.
In order to get these links, you need to publish pages people will link to. And not just the home page. You need links into internal pages, too. Ask yourself: what sites would you link to? What pages would you link to? Chances are, you're unlikely to link to a competitor. You're unlikely to link to someone else's e-commerce product catalogue. You'll most likely link to pages of note. Pages of reference material, pages of news, and other remarkable content that is noteworthy.
That's what everyone else does, not just in search, but in social media, too.
Another factor is the quality of your information, which we'll look at shortly.
Sound difficult to achieve?
Do You Even Need SEO?
You may not.
If you have a known brand that your existing customers will navigate directly to, you won't need to do much in the way of SEO. So long as you can be found under your brand name, you'll be rewarded. However, if you want to attract new customers, and attract customers away from competing brands, then you need to give SEO serious thought. At very least, you'll need to ensure your site is crawlable.
You can participate in the search channel without using SEO. You can buy clicks, using PPC. The downside is this can get expensive, as the incentive for Google is to force click prices ever higher via bid competition. You need to weigh the ongoing cost vs the cost of implementing an SEO strategy. Many people undertake both SEO & PPC, of course, in order to maximize a sites' visibility.
You need SEO if a long-term, cheap, visitor traffic stream is important to you. You need SEO if you seek to attract search visitors who may not have heard of your company before. You need SEO if your competitors are doing it, as they'll take your market share, given they have a presence in the channel, and you may not.
OK, I Need SEO
What is the optimal way to approach SEO?
If you've yet to launch a site, or you're planning on launching a new site, you're at a distinct advantage to those who must retrofit an SEO strategy. This is because SEO flows from strategy. It is very difficult to retrofit if the web strategy works against SEO, which can easily happen.
For example, Google tends to favor a regularly updated, well linked, reference information publishing model. One example is Wikipedia. Obviously, commercial sites aren't going to look anything like Wikipedia, however there are a few lessons to be learned. The key point is to integrate some form of detailed, text information publishing into your site, which preferably has a reference angle i.e. it's not just a page of sales copy.
Take a look at Amazon. Amazon is a product catalog, with a twist. Amazon lets users write reviews. The review text can be crawled by search engines. The existence of review text helps distinguish Amazon from other product catalogs, which to a search engines, would all look pretty much identical i.e. book name, publisher name, price, product description, etc. Search engines tend to relegate duplicate content.
So, you should include a section on your site that allows for the regular publication of unique, reference material. For example, industry news, a trade dictionary, discussion forums, blogs, feedback loops encouraging user content and comment, tutorials, user education, and so on. You might decide to split your web strategy across multiple sites. One site is the corporate umbrella site, another site is information based. Your SEO will likely have many ideas on this front, so the key is to involve them early.
SEO can be added after a site is launched, but it can be problematic.
Possibly the worst case scenario is a brochure site, consisting of thin product information and mission statements. Links don't tend to flow to such sites, and they don't tend to be information rich. Links will most likely need to be purchased, adding to the cost, and the search engines take a dim view of this practice, so it can increase risk if pushed too hard. If your competitors are attracting links without having to buy them, then they'll always be at a competitive advantage, and be very difficult to catch as each day passes.
There are a couple of ways around this problem. Create a new section of the site devoted to publication of reference material i.e. industry news, a trade dictionary, glossary, discussion forums, blogs, tutorials, and so on.
If the site isn't suited to this approach, consider splitting your web strategy across a number of sites.
Neither are particularly elegant, but the important takeaway point is that SEO isn't something that can just be tweaked under the hood. It needs to be an integral part of your site, and these days, that means adopting some form of information publishing strategy beyond simple sales copy.
The web is about putting the user first, and search is no exception.
Web content is commodity. If your site doesn't have the information the user wants, then there is nothing keeping them on your site, and no reason for them to visit in the first place. There are plenty of other sites. The site that gives the users exactly what they want, wins.
Luckily, in search, the user is already telling you what they want.
SEO's have a great way of mining this information. They can access keyword data, collected by the search engines. This data shows what terms users are looking for. You could create an entire web strategy based on this information.
For example, let's imagine a site owner sells heating systems. It would pay to know that a lot more people search for "solar heating systems" than "boiler heating systems". Obviously, interest in solar energy is increasing, so the owner may want to feature these products more prominently, and provide news on the latest developments by way of a blog. Keyword research shows a lot of people also want to know about installing heating systems, so the owner may want to provide guidance and/or a nationwide list of installers who install his product. That list of installers could be broken down into regions, which will likely attract regional search traffic. The site owner could encourage his national network of installers to link to his site, especially since he has demonstrated he is happy to send traffic their way. The site owner could also include a glossary of heating terminology, in order to cover every conceivable heating related keyword term.
Do the same with your site. Ask "what are the users really looking for?". Ask your SEO to research keyword lists to see what is really on your potential visitors minds. Provide a means to publish this information. Encourage people to link to your site by giving them a good reason to do so. Create genuinely useful content, then have your SEO and marketing teams get out there and hustle that information.
The Message Is Integrated
This is an integrated SEO strategy, based on the idea of putting the user first, giving them what they want, and encouraging them to share it.
Seth Godin wrote a book called "All Marketers Are Liars". In this book, Seth notes that, these days, marketing needs to be integrated into the product from conception. The days of bolting a pretty marketing face on to a generic box, after it comes off the factory line, are long gone.
It's the same thing with search. It should flow through your web strategy, just like usability, your message, your brand, and your language.
There's not a complex mathematical formula that is needed for one to understand the basic math associated with SEO. It boils down to something like:
Traffic + Conversions = $
That's a pretty easy way of looking at it, and it sort of ignores some of the variables that might go into it like:
no so targeted traffic
However, the basis of profiting via SEO mainly involves getting traffic to your website and converting (or monetizing) that traffic by whatever conversion (or monetization) methods you happen to be using on your website.
There are many means you can factor into the end game of an SEO campaign but at its most basic form it is about getting traffic and monetizing that traffic.
A Small-Minded Approach
The school of thought which postulates that ranking reports or ranking data is either essentially dead, useless, or pointless generally is a small-minded approach with respect to the various ways you can use ranking data and over-dramatizes the effect of changing search results from searcher to searcher. Small-minded simply because you can use ranking reports for more than just blindly monitoring keywords.
If the argument is that you should focus more on conversions than ranking in terms of straight revenue then I can buy that, to a degree, but the problem remains that you can't convert if you don't have traffic and you can't have traffic from search engines unless you rank highly for your keywords.
If the argument is that you shouldn't care because of personalized search, or local search, or different data center results then I would say that you are overstating the adoption and the effect. Sure, there could be a map or products or images in your search results (or tweets or news results) but I believe the idea that search results are so radically different from person to person, so different as to render ranking reports irrelevant, is quite overstated and inaccurate (from reports I've been running over time). All search results start from some starting point!
Knowing where you generally rank matters, watching the trend of your rankings in conjunction with your SEO tactics matters, and watching the evolution (up and down) of competing websites matters. To simply watch analytics data leaves so many opportunities on the table if we stipulate that ranking reports are a waste of time or mostly unimportant. When major algorithm updates or penalties happen, one of the quickest ways to help analyze what happened is to track your rankings before and after for a variety of keywords. That will help you determine things like:
is the issue sitewide?
is the issue related to a singular keyword?
is the issue related to a group of closely related keywords?
is the issue primarily impacting your most competitive keywords?
is the issue related to a particular market?
Pattern matching is key to learning how algorithms work. Sure some of this type of data may be available in your web analytics, but rather than having to hunt and probe for it, rank checking allows you to quickly get a baseline idea of where the problem may be.
Trends & Measurable Effects
Suppose you are interested in finding out whether certain SEO tactics are working or not working for a particular site. By watching your ranking trends over a period of time, parallel to your tactic testing, you can gauge whether or not those particular tactics are working.
Perhaps you've targeted a keyword which doesn't really have as much volume as you thought it did or what the keyword tools told you it did. If you ignore ranking reports then you are removing a key step in figuring out whether the word is viable or not, rather than looking at your analytics and guessing that it is viable or not based on traffic. Maybe you are ranking #4 for that term but the order goes:
Chikita reported (based on 8+ million impressions on their network) the following percentages of search traffic distribution by rank (roughly a year or so ago):
Chikita's chart shows that position 2 roughly in the 15-20% traffic range with position 4 around 5% and position 1 around 35%
Here's the leaked AOL chart from a few years ago, discussing the same topic:
AOL's data shows position 1 at 42%, position 2 at 11%, and position 4 at 6%.
So if you were running monthly ranking reports you could reasonably make the assumption that by increasing your rank +3 you might expect north of 25% in terms of increased traffic. If the sites were reversed and you were getting little traffic, it would be easy to see that this keyword is probably not worth continuing to spend resources on since you are ranking #1 and still getting little traffic.
In either the case of potential opportunity or no opportunity ranking reports would work nicely with your traffic reports to help you make reasonable adjustments to your SEO campaign. If you skipped the reports totally, you are kind of flying blind or more blindly than you need to be .
Sales & Marketing Tools
Everything in SEO comes down to balancing risk vs rewards. It is easy to show a short term boost while leveraging up on risk, but showing sustained performance is much harder. Snake oil salesmen *always* have a smooth sales pitch (along with ranking reports for search engines nobody uses, and some go so far as faking traffic to websites using click bots). The more lenses you can provide your clients of value delivered the more you differentiate from those who are playing games of deception.
A client may view an SEO as incompetent simply because Google changes the rules of the game mid-stream. From month to month search can change in ways that seem both uncontrollable and unpredictable. Nothing kills sales like the words "I don't know." The more answers you can deliver the more confidence clients will have in maintaining & growing their investment in search, even if things are a bit unstable in the short run.
Ranking reports are further evidence of proof-of-value delivered. They help take something fuzzy and make it feel more concrete, helping you show the client not only that you are pushing to build relevant traffic, and serving as a baseline to help clients see how they are doing. If the client knows they are at #3 with a $5,000 monthly budget they can easily see the value of increasing the budget to $10,000 to boost their ranking to #1.
Take it One Step Further With Analytics
Let's say you are starting to see all these keyword variations in your analytics for a core term you are targeting. Here's where you can (again) use analytics and ranking reports together:
The ranking tools mentioned also offer ongoing rank reporting as do the tools from Raven, SeoMoz, and Authority Labs (incidentally, Raven will be using Authority Labs's API for ranking data in the near future as mentioned in the Raven link above).
Now you've got a bunch of new keywords you are already getting some traffic from, along with some predictions on what the potential increase in traffic (and conversions if you have that data from your analytics) might be.
Factoring in Universal & Local Search
Advanced Web Ranking has some interesting features which let you change up the location so you can better track those kinds of results. Google continues to take up SERP real estate so sometimes you run in to situations where you might be ranking #2 for a core keyword but given maps, news, images, and products you could be "ranking" as low as 6 given the SERP layout.
This is another situation where you can use your ranking reports and analytics together to get the most out of an SEO campaign. Perhaps you are not getting traffic, or as much as you though given your research, but you are ranking #2 according to your reports. Using ranking reports and traffic numbers together can help you determine whether to continue pursuing that keyword or maybe use some different strategies (PPC, trying to get into the "universal" search results, etc) to win back the traffic you've lost to universal search.
It's the same premise with local. Can you reasonable expect to rank in whatever position(s) are above the map? Can you get into the map? Is PPC viable for your campaign? Rarely is it useful to go off of one data point. This is another example of how to you use multiple data points together, to more appropriately manage your or your client's SEO campaign.
It's Against Google's Guidelines!
This is absurd in my opinion, more so when it's stated by folks who sell SEO services. If you offer SEO services (which ironically promote the idea of increased rankings and visibility) and those services encompass "Link Building" then the "Google Guideline" stance is hypocritical.
In all fairness, I happen to think that the broad way Google encompasses link schemes is equally absurd (links intended to manipulate PageRank and such). Even Google recognizes the value in ranking data and they have incorporated it into Webmaster Tools.
Not a Singular Solution for Success
Ranking reports shouldn't be used as a single source of success, at all. Simply ranking for a term is not something one should be shooting for unless you are just doing some kind of testing run on tactics.
There is value in running ranking reports and using them in conjunction with your analytics, keyword research, and SEO planning. They are also useful to watch growth patterns of competitors and keyword trends over time for a particular market you might be interested in.
In today's SEO game you can never have enough useful data :)
Pretty basic question for an SEO right? It would be nice if the answer were equally as basic or simple. It's an important question, even more so since a sweeping update by Google knocked out unsuspecting webmasters.
Beyond the stuff we know got hit (some RIGHTLY so), we probably will never know the true ramifications with respect to how many "little guys" had their livelihoods or potential livelihoods destroyed by a heartless, unforgiving, and sometimes inaccurate algorithm.
Evaluating the Risks
I know people who worked their rear ends off and had their business fail, in addition to people who were lazy and failed. Sometimes it's timing, sometimes there's some luck involved (though luck is generally brought about by hard work), and sometimes it's just a matter of working harder or spending more then your competition.
The risks are plentiful for the self-employed and they only scale up if you:
have a family to support
have a mortgage
need to buy health insurance
have limited capital to invest
can't afford to lose on a few of your bets
Ways you can combat those issues are to:
live below your means
work a part-time job at night or during the day
have your spouse work part-time
not buy every single device that Apple makes :)
be prepared to work longer hours than you'd like
The problem is that self-employment, especially if you've worked in the corporate world before, has all the allure of the pipe dreams sold in The 4 Hour Work Week (I guess Tim Ferris doesn't count self-promotion as work, even though he does it about 100 hours a week :) ).
You might think self-employment is all about working less, spending more time with your family, going to the beach while everyone else is in a cubicle, and all that jazz. While it certainly can be at some point, it is not how you will start out 99% of the time.
Self-Employment Screw Jobs
Start looking around to find an accountant who will tell you what your tax liabilities will be as a sole-proprietor or a single-member LLC. If you really want to take a kick to the shins, get a quote for health insurance and see how much it doesn't cover.
A friend of mind recently pointed out to me that all the individual health insurance plans do not cover maternity costs. You can get coverage for that under a group policy but unless you have employees you are going to be paying roughly 4x the cost of an individual plan for you and your family.
He lives in the US, is self-employed, and has a family. The American dream right? For he and his wife to have a second child, they would need to get a group health plan at least 60 days prior to the wife becoming pregnant. Say that everything works out to be on-time, they are looking about an increased cost of about 20-25k over the course of a 9 month period to have a child in the United States!
All of that is assuming a perfect pregnancy and a 100% healthy baby. The point is to layout just one large, large risk you might be unaware of ( rubbish health insurance ).
Another thing to keep in mind, beyond the health insurance, is that unless you start one there's no retirement benefit, no paid vacation time, and no paid sick time. It is a really good idea to purchase short-term and long-term disability insurance for yourself as well.
It's Not an Either Or Question
The smartest thing to do would be to starting whittling down your household bills now and start slowly building your business. Keep your day job and work at night, you'd be working 15 hour days if you started from scratch anyway so why not do it now but get paid (with benefits) for your time?
Once you begin to make some cash on the side, see how far you can scale it (reasonably) before you need to make a decision on whether to fully jump in or whether to keep it as a side gig where you can eventually outsource a good chunk of the tedious work.
Some people loathe the idea of a boss and that's fine. Even good people have bad days so it's not always going to be peaches and cream but if you are in a spot where there is mutual respect and fairness then I'd say hang on to it until you can financially manage to go out on your own.
Even when you are "the boss" working for yourself if you have any level of success you will soon be "the boss" with employees of your own. And many successful businesses don't have 1 boss, but rather hundreds or thousands of them - their customers.
Maybe it's Not for You
Self-employment is not for everyone. It takes awhile for it to really pay off professionally and personally. When you first start out you may not have enough capital to outsource things like:
Not being able to outsource all that upfront is probably a good thing. Many attempts at outsourcing fail because the outsourcer is not competent themselves in those particular tasks. Having experience in these areas is helpful because if a freelancer or staff member leaves you hanging, you'll be able to keep things running efficiently while you search for another staff member.
Working for a family business or a steady corporation isn't something to be looked down upon by any means. The key to mitigating the risks on both sides (getting laid off for example) is to be financially prudent, continually invest in yourself (earn a degree, start a side business), and be loyal to whom you work for or with. If you do those three things you will typically be ok in any reasonable situation.
Start Off as an Apprentice
Maybe you know someone in the SEO or PPC industry that might be willing to have you work with them for awhile or perhaps you aren't ready to fully go at it on your own just yet. Most employers realize that the best employees sometimes are the more motivated and ambitious ones and with that comes the risk that those employees will look to move up and on at some point.
Starting off this way has risks too (might not be as stable as a large corporation for example) but you can learn a lot about the overall and day to day processes that make that particular person or group successful. I'm not for the idea of building your personal brand on the back of someone else's brand equity but if your brand develops from the hard, quality work you do for someone else then that's great.
I wouldn't go in with the sole purpose of using your position to quickly build your brand and bolt. I'd go in with the purpose of working your butt off for someone who gave you a great opportunity and see what develops from that. Usually the latter will result in both sides being more than happy.
This part is more relevant if you have a non-working spouse or a spouse who might work part time. Little things that can really add up to time sucks include:
paying the bills
making the monthly budget
dealing with household vendors
scheduling and rescheduling family appointments
doing the household shopping
You should be focused on your business and the associated responsibilities. If you are doing any of the above, try to transition that to your spouse or significant other. It's not just the hour or two it might take for you to do those tasks but it's the stopping of your business activities and the mind-flow disruption associated with starting and stopping tasks frequently.
Evaluating the Decision
In this weak and unstable economy it is really hard to reasonably project out 5 years on a life changing decision. There are so many variables to take into account that it's difficult to give a tailored answer to each situation.
If you are at the point where you need to make a decision or want to make a decision down the road, there are some key points to keep in mind but financial variables are some of the most important variables in this equation.
It's fine to want to do something and have the drive to do it but if it's going to potentially create a financial hardship quickly then it is really the wrong decision. There are a few really good options for someone who is on the fence for financial reasons.
see if you can start by working with an experienced person in the field
take up a part time job and/or see if a part time job is a good option for your spouse or significant other
start living below your means and save some cash for rainy SEO days and for investing in knowledge + small ventures
The other elephant in the room is stress. The stress of being the breadwinner is stressful enough, but if you have no safety net or if your business is brand new (thus more unstable than 9-5 corporate stuff) as well then it is really stressful. This is something to keep in mind and something you might want to try and emotionally reconcile before you start.
You might also find that doing it part-time and maintaining a "day" job is a really good fit emotionally and financially. If Google blindly swings an ax, and you get hit, you can rest assured that the sun will still come up tomorrow and will that direct deposit on Friday :)
The point remains that there are many options available to you to help figure out if working for someone else or yourself is really the best fit. In either case, working and studying harder (and longer) than others is a solid base to start from :)
You will come up against barriers. These barriers are often put up by designers, developers, copywriters and management. Frustrating as it is for the SEO, this is the reality of working on a site alongside other people, all of whom have agendas and requirements that may differ markedly from your own.
So how do you navigate this space? How do you ensure your SEO objectives can be met when other people may be resistant to change, or openly try to block you? In this post, we'll take a high-level, conceptual look at the challenges the SEO faces when working on a client site, and talking-points to help explore and clarify concepts.
1. Why Are We Doing SEO At All?
SEO is a pain.
It's complicated. It gets in the way, particularly when it comes to design. Why do we need headings and a lot of text when a picture tells a story? SEO appears to be an arbitrary, dark art with little in the way of fixed rules, and the client probably doesn't care about it anyway.
The thing is - if SEO is done well, a client may throw a whole lot more money at the site in future. Everyone likes to build on success, and that means more business, and more exposure, for everyone involved. On the internet, traffic = success. Traffic = money. A site that few people see, no matter how well executed, will likely fail, just like a site that fails to engage and convert visitors will fail. The client may not know they want SEO now, but you can be certain they'll be asking questions about it after launch.
If SEO is done poorly, the site may not be seen by as many people as it otherwise would. What use is a beautiful design that is seldom seen? What use is great code that is seldom used?
The value proposition of SEO is that it helps get a site seen. It's a powerful marketing channel, because most people use search engines to navigate the web. Sites that deliver what the search engines want stand to gain a lot more traffic than sites that do not undertake SEO. If your competitors are undertaking SEO, this puts your work at a competitive disadvantage. Their site will be seen more often by search visitors. Their web agencies will likely get more business as clients see greater returns on their investment.
That's why we do SEO. To be seen.
Of course, a site can be seen by other means. Word-of-mouth, social media, links, brand awareness, and offline advertising. A site doesn't need SEO, but given that it is a relatively easy win in terms of cheap traffic acquisition, the extra effort involved is negligible compared to the upside benefits. It's like being given a choice of having a shop located on main street vs a location way out in the desert. Much the same effort involved in building, but significantly different traffic potential.
2. SEO Is A Design Element
Just as copywriters require space to insert paragraphs and headings, SEO's require space to do their thing.
If you're a designer, an SEO will likely provide you with a list of their requirements. These requirements need not be onerous, any more so than leaving space for copy is considered onerous.
There are two key aspects where SEO needs to integrate with design. One aspect is the requirement for machine readable text, provided in a format the search engines are able to read, and derive meaning. Search engines "think" mostly in terms of words, not pictures. Make design allowances for copy that includes lot of headings and sub-headings, a technique which also dovetails nicely with usability.
The other key aspect is crawl-ability. A search engine sends out a spider, a piece of code that grabs the source code of your website, and dumps it back in a database. It skips from page to page, following links. If a page doesn't have a link to it, or no crawlable link to it, it is invisible to the search engines. There are various means of making a site easy to crawl, but one straightforward way is to use a site map, linked to from each page on the site. Similarly, you should ensure your site navigation is crawlable, which means using standard hyperlinks, as opposed to scripted/executable links. If you must use scripted links, try and replicate the navigation elsewhere on the page in non-scripted form, or within the body of the text.
For most sites, that's pretty much it when it comes to design considerations. In summary, the inclusion of machine readable text, and a means for a spider to crawl easily from page to page.
An SEO may also wish to specify a page hierarchy and structural issues, where some pages are given more prominent positions than others. Of course, this needs to be weighed against navigation considerations for visitors who arrive at the site via other means.
3. SEO For Developers
Like design, there are two key areas of integration.
One is tagging. SEO's will want to specify title tags, and some meta tags. These need to be unique for each page on the site, as each page is an entry page as far as a search engine is concerned. A search visitor will not necessarily arrive at the home page first.
The title tag appears in search results as a click able link, so serves a valuable marketing function. When search visitors consider which link to click, they'll use the title tag and snippet to influence their decision.
The second aspect concerns URL's. Ideally, a URL should contain descriptive words, as opposed to numbers and random letters. For example, acme.com/widgets/red-widgets.htm is good, whilst acme.com/w/12345678&tnr.php is less so. The more often the keyword appears, the more likely it will be "bolded" on a search results page, and is therefore more likely to attract a click. It's also easier for the search engine to determine meaning if a URL is descriptive as opposed to cryptic. For an in-depth look at technical considerations, see "SEO For Designers".
One workaround if the database needs unique codes is to translation at the URL level, using URL rewriting.
4. SEO Is A Marketing Strategy
The on-page requirements, as dealt with above, are half the picture.
In order to rank well, a page needs to have links from external sites. The higher quality those sites, the more chances your pages have of ranking well. The SEO will look to identify linking possibilities, and point these links to various internal pages on the site.
It can be difficult, near impossible, to get high quality links to brochure-style advertising pages. Links tend to be directed at pages that have reference value. This is a strategic decision that needs to weighed during site conception. Obviously, few sites strive, or want to be, Wikipedia, however there are various ways to incorporate reference information into commercial sites where the primary purpose of the site is not the publication of reference information.
For example, include a blog, a news feed, publish the e-mail newsletter to the site, and/or incorporate a reference section within the site. It doesn't matter if this section isn't viewed by visitors who navigate directly to the site. It provides a means to get a lot of information-rich content into the site without disrupting design and other commercial imperatives. Think of it as a "mini-site" within a site.
Not every page needs to be for the purposes of SEO. SEO can be sectioned off, although this is often less ideal than more holistic integration throughout the site.
5. Strategic Factors For Managers
Concept, design and development can screw-up SEO.
Poor integration can result in loss of potential traffic. This traffic will go to competitors. The longer a site doesn't use an SEO strategy, the harder it is to ever catch the competition, as a head-start in link building is difficult to counter.
If your aim, or your clients aim, is to attract as much targeted traffic as possible - as most site owners do - then SEO integration must be taken as seriously as design, development, copy and other media. It may influence your choice of CMS. It may influence your strategic approach in terms of how and what type of information you publish.
Whilst SEO can be bolted-on afterwards, this is a costly and less-effective way of doing SEO, much like re-designing a site is costly and less effective than getting it right in the planning stage. If SEO is integrated in the planning stage, it is reasonably straightforward.
The time to incorporate SEO is during site conception. SEO is a text publishing strategy. Design and development will need to make minor changes to the way they approach a site build. Doing this retrospectively, whilst not impossible, is more difficult, and therefore more costly.
Coda: Flash Workarounds For SEO
There are various workarounds to existing search-unfriendly design, but I'd advise to avoid the problem in the first place.
Flash, whilst a useful tool for embedding within sites, should be avoided for the entire site. Flash is a graphics/animation format, whereas search - and the web in general - is primarily a text format. If you build an entire site using Flash, then your competitors will overtake you in terms of search visitors. The formats simply do not gel.
One work around is strategic - split the site in two. Use Flash as a brochure site, and create a hub site that is text based. Consider including a "printable" version of the site, which will give the search engines some text to digest. Whilst there are technical and strategic ways around Flash, they are often clumsy and tedious.
The search engines can make sense of most sites, but if you're expecting to get rewarded by search engines, then it pays to stick as close to their technological strengths and weaknesses as possible.
Google ignored our page title, ignored our on-page header, and then use the 'comments' count as the lead in the clickable link. Then they follow it with the site's homepage page title. The problem here is if the eye is scanning the results for a discriminating factor to re-locate a vital piece of information, there is no discrimination factor, nothing memorable stands out. Luckily we are not using breadcrumbs & that post at least had a somewhat memorable page URL, otherwise I would not have been able to find it.
For what it is worth, the search I was doing didn't have the words comments in it & Google just flat out missed on this one. Given that some huge % of the web's pages has the word "comments" on it (according to the number of search results returned for "comments" it is about 1/6th as popular online as the word "the") one might think that they could have programmed their page title modification feature to never select 'comments' as the lead.
Google has also been using link anchor text sometimes with this new feature, so it may be a brutal way to Google-bomb someone. It is sure be fun when the political bloggers give it a play. ;)
But just like the relevancy algorithms these days, it seems like this is one more feature where Google ships & then leaves it up to the SEOs to tell them what they did wrong. ;)
You can learn a lot about how search has improved over the years by reading Matt Cutts. Recently he highlighted how search was irrelevant in the past due to a lack of diversity:
Seven of the top 10 results all came from one domain, and the urls look a little… well, let’s say fishy. In 1999 and early 2000, search engines would often return 50 results from the same domain in the search results. One nice change that Google introduced in February 2000 was “host crowding,” which only showed two results from each hostname. ... Suddenly, Google’s search results were much cleaner and more diverse! It was a really nice win–we even got email fan letters.
Thanks to those kinds of improvements, in 2011 we never have to look at search results like this.*
* And by never, I mean, unless the results are linking to fraternal Google pages, in which case, game on!
Why should Google result crowding not apply to Google.com? Sure they can say those books are from different authors, but many websites are ran by organizations with multiple authors. Some websites are even built through the partnerships of multiple different business organizations. Who knows, maybe some searchers are uncomfortable with every other listing being an out of context book highlight.
In the past I have been called cynical for highlighting stuff like the following image
I saw it as part of a trend toward home cooking promotions. And I still view it that way. The above books promotion is simply further proof of concept.
other Google owned and operated sites
a branded website ranking for its own brand
Can you show me *any* occurrence of a result where a site is listed 5 times in the search result? Bonus points if you can find it where the 5 times are not grouped into 1 bunch via result crowding.
As a thought experiment, ask yourself if that Google ranking accident would happen if the content archive being served up was promoting media hosted on Microsoft servers.
A friend of mine summed it up nicely with:
well, it's not everyday you see that kind of power and the fact that other sites aren't afforded the same opportunity makes me think that they are being anti-competitive. Google literally wrote the book (ok scraped it) on anti-competitive practices.
Majestic SEO has long had great link data, but their biggest issue has been usability. They sorta built with the approach of "let's give them everything" as a default, and then allowed advanced filtering to be done over the top to generate custom reports.
For advanced users this type of set up is ideal, because you are able to slice and dice it in many ways on your own terms. It allows you to spot nepotistic networks, pinpoint strategies quickly, and generally just give you a good look at what is going on in ways that wouldn't be able to do if you couldn't get all the data in a table. There are so many valuable edge case uses that can't practically be put in a single interface while keeping usability high for the average use.
But for people newer to the SEO game & those looking for a quick source of data the level of options can be a bit overwhelming when compared against something like Open Site Explorer. A parallel analogy would be that when I want to spot check rankings real quick I rely on our rank checker, but if you want to have a variety of in-depth historical views then something like Advanced Web Ranking can be a quite helpful tool.
How much can you use the Majestic Site Explorer?
The system is designed for silver users and above. Silver subscribers can query upto 10 different domains an HOUR. Gold subscribers can query upto 30 different domains an hour and Platinum subscribers can query upto 100 different domains an hour. All levels are subject to fair use terms.
These allow you to view data on a sitewide basis, at the subdomain level, or drill down to individual pages.
Here is an example of a site level report
and if you wanted data down to the URL level, here is an overview of a top few links (note that the report goes on for numerous pages with data)
This update helped Majestic SEO close the gap a bit with Open Site Explorer, but a couple more things they may want to consider doing are
adding result crowding / limit results to x per domain
allowing you to filter out internal link data
Those features are available via their advanced reports, but making it easier to do some of that stuff in the "at a glance" interface would allow Majestic SEO to provide as a best in breed solution for both the "at a glance" function and the "in-depth deep research" options.
Majestic SEO also announced their new fresh index, which allows you to view fresh link data as recently as within the past day. It doesn't require waiting for a monthly update or such, but offers link data right away. To help spread the word & give everyone a chance to see some of the new features they gave us free discount voucher codes to give out to get a 20% discount on your first month at any level.
If you have any questions about how Majestic SEO works you can sign up & register your own site, which allows you to access many of their features free. As a comparison SEOmoz (which offers Open Site Explorer) is also running a free 1-month trial right now.
Many broken belief systems that exist do so because of a misinformed understanding of how the world works through naive idealism, with various special interests paying to syndicate misinformation that coincides with their current business model to foster culturally constructed ignorance - agnotology.
It is not a bubble. This time is different. The internet changes everything
And then of course we had "Real estate always goes up!"
Who was behind that lie? The bankers, the mortgage brokers, the Realtors, bond raters, hedge funds, construction companies, media running real estate ads, local government tax revenues, current home owners who kept seeing their "savings" go up while doing nothing. Some of those people did not intentionally aim to be deceitful, they just believed a convenient lie that fit with their worldview.
"It is difficult to get a man to understand something when his job depends on not understanding it" - Upton Sinclair.
Fraud vs the Stuff Bankers do
And so the bubble grew until one day the fraud was so integrated into society that there was simply nobody left to sell to.
“In mid-2006, I discovered that over 60 percent of these mortgages purchased and sold were defective,” [Citibank's] Bowen testified on April 7 before the Financial Crisis Inquiry Commission created by Congress. “Defective mortgages increased during 2007 to over 80 percent of production.”
Wachovia was a strong brand. A true pioneer and market leader in the drug money game, which funneled over 1/3 TRILLION Dollars onto the hands of drug dealers. For the crime they got a slap on the wrist. there was no bonus clawbacks. There was no jailtime. There was no honest attempt at the rule of law.
"we actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side." - Matt Cutts
Create an Itch
For a marketer to say what is old and steady and boring is effective is not a way to be perceived as relevant (that old coot is still stuck in '97!)
Being grounded is not a way to get positive headlines. Saying that the web is becoming just like the fraud laced offline world would be considered in poor taste. You have to sell something new...to try to push to inspire, achieve, gain hope, etc.
If you manufacture evidence that your LinkedIn votes are directly tied to better Google rankings then outsiders who are unaware of the workings of your industry may syndicate that misinformation. Even if you run a public experiment that fails it still shows you are trying new things (are cutting edge), and is a low cost branding exercise. Just like how MLM folks say you can get rich by using the same system they used to get rich. Everyone wants to sell a life worth living, even if they are not living that life, but rather sentenced to life in prison.
Pushing the Boundaries
Most profitable belief systems sell into an existing worldview but with a new hook on it. Most new marketing approaches are all about pushing the boundaries of what exists, probing to find the edges. Some people do it on the legal front, others probe on the ethical front, and yet others are just more creative & try to win by using technologies in unique ways. If you never fall off a cliff and never have any hate spewed your way then you are likely a bland marketer who hasn't done very much.
Google is in the press almost every week, aiming to stretch the boundaries on trademark, copyright, privacy, and so on.
I was lucky enough to chat with Larry one-to- one about his expectations for Google back in 2002. He laid out far-reaching views that had nothing to do with short-term revenue goals, but raised questions about how Google would anticipate the day sensors and memory became so cheap that individuals would record every moment of their lives. He wondered how Google could become like a better version of the RIAA - not just a mediator of digital music licensing - but a marketplace for fair distribution of all forms of digitized content.
Google is seen as an amazing company that does a limitless amount of good for the world. Yet the are up for anti-trust review and carry ad categories for "get rich quick." Google massages how they are viewed. Anytime something bad happens to their brand you can count on a new invention or an in-depth story of a rouge spammer getting torched by "justice."
For a company that is so good at manipulating outside rules & guidelines, they really lean hard on the arbitrary guidelines they create.
The web is constantly shifting. Mailing lists, email newsletters, blogs, wikis, Facebook, Twitter, Color, etc. Most of the core infrastructural stuff is boring. But it is essential. If you don't understand email marketing or newsletters you can't create Groupon.
It is the new stuff with some sort of twist that earns the ink, which drives the perceived value, which earns the ink, which builds the actual value. But most people can't tell the difference between real innovation and public relations fluff. And so after a series of failures and burning millions of Dollars of capital it is time to pivot again. Anything to be seen as new and/or relevant.
If you manufacture evidence that your new strategy is better than Google then outsiders who are unaware of the workings of your industry may syndicate that misinformation. Even if you run a public experiment that fails it still shows you are trying new things (are cutting edge), and is a low cost branding exercise.
Sounds familiar, right?
History keeps repeating itself.
Algorithmic Fallout vs Spam
The perfect algorithm is something that does not exist.
Every choice has winners and losers. No matter what happens to the network & how the algorithms evolve people will find ways to exploit them. Many of Google's biggest holes were caused by Google patching old holes.
Which is precisely why Google leans so hard on public relations & shaping market behavior.
It is not the fault of the search engineer when something goes astray, but rather an evil exploitative spammer (even when Google's AdSense is the revenue engine driving the project).
Clean Your ____ Up!
Thinking back to the content farm update (which was never called the content farm update, because it impacted a wide array of websites) the main message that came out of it is that "Google can determine content quality" and "you better increase your content quality." Webmasters who heard that message were stuck in a tough situation if they had hundreds of thousands or millions of pages indexed in Google. How exactly do you *profitably* increase the quality of millions of pages, even while your site is torched to the ground, revenues are off over 50%, and the timetable + certainty for the solution are both unknowns? In many cases it would be cheaper to start from scratch than to repair the broken mess & deal with all the known unknowns.
Based on Google's advice many webmasters decided that as part of their strategy they would improve the quality of some of their best pages & then have a look at some of their worst content sections and try to block and/or remove them from Google. That sounds pretty logical! In response to that overly-logical approach to problem solving, Matt Cutts wrote the following:
What I would not recommend is sending tons (as in, thousands or even tens of thousands) of individual url removal requests to the url removal tool. And I would definitely not recommend making lots (as in, dozens or even more) of Webmaster Central accounts just to remove your own urls. If we see that happening to a point that we consider excessive or abusive, we reserve the right to look at those requests and responding by e.g. broadening, cancelling, or narrowing the requests.
So here you are trying to comply with Google's latest algorithmic approach (after they already torched your website once) and they have to give you another "or else."
Why The SEO Consultant Will NEVER Go Away
It would be nice to know what pages Google thinks are of low quality, but they don't say. It would be nice to know what pages are indexed in Google, but even official data given in Google Webmaster Tools varies widely over time, let alone the data which is shared publicly.
Further, some sites, like forums, are hard to edit to please Google without potentially destroying the flow of the community and enraging the community. Should sites have to delete or de-index their water cooler area because of Google?
What about the pages that GoogleBot arbitrarily creates by putting keywords into search boxes and generating pages that the site owner may not even know are indexed?
The reason so many webmasters are forced to rely on external search advice is that Google's desire to not be manipulated is so strong that they frequently appear dishonest & not worthy of trust. They speak vaguely, distort, and change the numbers as needed to fit the vision. Saying "in an ideal world" doesn't make that ideal world appear. And people don't trust folks like Donald Rumsfeld - at least smart people don't.
As a local advertiser, starting an SEO campaign in your local market is typically built on the strength of your keyword research. Say you are an insurance agent, do more people use car or auto when searching for auto insurance? Do people use "city/town keyword", "city/town, state, keyword", "zip code keyword"?
Some of these questions can be answered using a tool like Google Trends. Here you can see the results for "Texas Doctor" versus "TX Doctor":
So here you can see that it's pretty close, and volume is pretty close in Google's keyword tool as well:
However, when you get into phrase match the volumes separate a bit:
Overcoming Keyword Tool Volume Concerns
The other thing you'll want to keep in mind is that sometimes these tools can be off on volume, sometimes a lot and sometimes not so much. How do you solve this? You can do a PPC campaign to test a few things like:
The beauty of starting your campaign with PPC is that you can not only keep it running if it's profitable for you, rather than it just being a proving ground for keywords, but you are able to discover keywords and keyword groups that are profitable and have enough volume to where an investment into SEO is worthwhile.
Local search, by definition (since it is roughly a quarter of the search market), is on the lower end of the volume pole but in comparsion to a local business's resources and reach the volume is typically relative to that of keywords for a national company pursuing non-local keywords country wide.
Thinking About Campaign Structure
In addition to finding juicy keywords and keyword themes to build on, you can eliminate the poorly performing ones or the ones which have close to no volume from your PPC campaign and remove it from your SEO planning.
This not only helps your PPC account grow and mature but also helps you avoid wasting time and resources on chasing irrelevant or unworthy keywords.
As we discussed, sometimes local keywords can use a variety of modifiers like the city or town name, the state name, and the zip code in conjunction with the keyword(s) so making sure you are targeting the right mix from an SEO perspective is really helpful in getting quicker and better results.
There is no point in optimizing your on-page content and targeting your link building plans on your keyword(s) plus a zip code if your market is searching by city/town and state (and vice versa). In the interest of time and better results, it makes sense to nail down the correct keywords upfront.
Starting off with Research
Generally, my initial research process goes something like this (we are assuming you've got a live site already):
look in analytics to find keywords that you are already receiving traffic for
see if there are any trends in that data in terms of language (car vs auto insurance for example)
begin broad keyword research to find terms related to the market (exclude local modifiers for now)
use free mindmap software or free site planning apps to visualize the main content areas of the site with those keywords
use google trends and insights, in addition to the google keyword tool and the free seobook keyword tool to compare data points on core terms (again, like with car/auto insurance or home versus homeowners insurance)
make a list of competitors in my area and check the volume on their brand name
So now I should have a good idea of which keywords I want to look at locally and some notes on any glaring differences in volume between closely related terms.
Now it's time to "localize" the data. I like the local keyword tool over at PPCblog.com because it does a really good job of working in all the different local modifiers that can be associated with your local PPC campaign.
That is a paid tool, as part of the PPC blog community and training membership (along with a lot of other quality PPC tools), and it's quite robust and easy to use.
If you are looking for a free tool along those lines, with less on the functionality front, you can use this free tool from 5minutesite.com.
Then I move into searching on some of the core terms in Google's keyword tool and the SEObook keyword tool (powered by Wordtracker).
Many times you'll find nothing for some of your local searches, in terms of volume, but you should still keep them around for testing in PPC because keyword tools can be off on local searches based on their traditionally lower volume sets. Also, most keyword tools don't or can't allocate resources to capture every single search.
So now I should have a list of locally modified terms where the keyword portions were driven by non-local keyword research and local modifiers were added via a local keyword tool.
In addition, I should have notes and screenshots of data from Google Trends and Insights showing any language differences (of substance) both nationally and locally (locally when available, sometimes no data exists in the tools). I also should have notes as to any language or keyword trends I found in my analytics or tips I received by talking with employees who deal with customers as well as my own knowledge of the industry.
Working with AdWords
There are different ways of attacking your campaign in AdWords. Initially, I am just doing this for testing on an SEO campaign but if you decide to stick with the PPC campaign you can get into removing the local modifiers and bidding on those broader keywords while targeting searchers geographically.
Google has a few different ways of targeting users based on location:
Locations and Languages offer you the ability to target in 4 ways:
Bundles - mostly specific countries (United States, Spain, Canada, etc) and regions (North America, Central America, East Asia, etc)
Browse - essentially goes country - state - metro area - specific city or town
Search - search for and add just about anything (country, state, town, zip code to find towns or cities)
Custom - a nifty point, click, drag interfact where you can isolate a specific area where you want your ads shown
You also have some advanced options like the Targeting Method:
Google has a really helpful chart on this here, and below is a screenshot of the information:
I like to leave both on as it helps with gauging not only the potential of your keywords but also the overall level of activity for your services (via keywords) in your market. Plus, the search term report can help you breakdown keywords that trigger your ads and this kind of PPC can help you show for broad SEO terms that you might not have the resources to compete for.
Another advanced targeting option is the exclusion method:
Google has information on this method here and here's a chart showing the relationship:
I like to use this in some cases where there may be towns that overlap. For example, you could live in Maine and be targeting "Augusta" as a modifier but you'll probably want to exclude Georgia from your targeting as that is another area which can produce searches for that modifier.
You can also get around that by adding a state modifier, Augusta Maine Insurance or some such, but you may find many folks use just the city or town name. That is when exclusion methods can be helpful.
Starting off on the Right Foot
Now I'll start to build the PPC campaign and pay attention to some of the core principles of trying to obtain a good quality score and good overall performance for a new account:
tight ad groups with keywords that are relevant to the ad group and the query
quality landing pages which speak specifically to the intent of the query (don't use a generic insurance template for all the different kinds of insurance you sell)
starting off with a managable amount of keywords to help focus on quality of traffic rather than quantity, and to help promote good keywords and remove or isolate bad ones
As an example, you might be selling life insurance in a few different towns. I would consider using town-specific ad groups -> keywords -> landing pages as my structure.
You can use helpful landing pages for a specific town by talking about things like average family size in the town, average income, and so on to help residents get a more customized experience when shopping for life insurance.
You can also build product-specific ad groups and group your town/city modified keywords in there if that makes more sense for your specific campaign.
Waiting for Results
In about a month or less I should have a pretty good idea of:
search volume for my proposed keywords
new keywords that I didn't find initially
which keywords convert and which don't
will PPC fit into my ongoing marketing efforts?
what type of SEO investment does my search volume call for?
We live in a world and business environment where we want things yesterday and sometimes it can be tough to play the patience game. In my opinion, lack of patience is a leading cause of SEO and PPC failure these days.
If you take the above approach with a new campaign or a new idea, you will thank yourself in the short, mid, and long run. There are few sources of advice better than hard data, whether it tells you what you do or don't want to hear.