We love free stuff, especially when it comes to SEO tools and SEO data. Recently, we published a post on how to do a good bit of competitive research with free tools and now we are going to do that for competitive research on domains.
There are a number of tools we can use here. We are going to focus on using these tools to help evaluate a domain from a competitive research point of view:
AdWords Keyword Tool
Open SIte Explorer
Google Ad Planner
It is worth noting that we reviewed the paid elements of most of the prominent spy tools about a year ago.
Getting Started with a Domain
Researching a competitive domain can have many benefits. Beyond evaluating the strength of a domain with respect to age, links, and engagement statistics you can find things like:
High traffic keywords
Low hanging keyword fruit (keywords they are ranking for mostly off domain/brand authority)
Competing domains and overlapping keywords
Keywords being purchased for PPC
So you can do a few different things with domains. You might want to evaluate the strength of the domain as a whole if you are beyond the keyword research phase or perhaps you want to do that in addition to checking out potential keywords you can add to your campaign.
There are a few different tools you can use for this and I like to start with the SeoBook Toolbar because it's quick, easy, and incorporates the tools I want to use in one spot.
Using the SeoBook Toolbar
The toolbar links through to a ton of external tools and most of the tools listed above. It also provides a way to quickly review a bunch of the most relevant data with a simple click. Turn the toolbar on, visit the domain you want to research, and click the blue "I" icon shown below, next to the SeoBook icon:
Once you click on the blue info ball you get all this nice data immediately:
So in what really amounts to a quick, 3 step process you are able to instantly see helpful information about:
High level site data about age, Pagerank, indexed pages, and recent cache date
Link data from Yahoo! Site Explorer, Open Site Explorer, and Majestic SEO
Rough traffic estimates from sources like Compete.Com, Alexa, and SEM Rush
Important directory links
It will be somewhat clear just by looking at the chart how strong the domain is. In this case, the domain is one of the stronger ones on the web.
You can link through to each tool/statistic from this chart and also from the icons on the toolbar itself.
As you continue down the toolbar you can see the link-thru icons Open Site Explorer, Majestic SEO, and Blekko. The "Dir" dropdown will show you the appearance of the site in the more important directories on the web.
Then you also can link thru to the Archive, Compete.Com, SEM Rush, the free SeoBook Rank Checker (to quickly check rankings of a keyword on a particular domain you might be researching), and the X-Ray Tool.
The "Competition" drop down will show you the following:
So here you can link through to a variety of sites to check out all sorts of data points about a domain including, but no limited to, domain registration, demographic data, and keyword data.
If that weren't enough, the toolbar also offers more tools:
The first link gives you the following drop down, which links through to a bunch of keyword tools based on the keyword you enter in the form field to the left of the book:
The highlighter highlights the typed in keyword on the current page and then you've got a link to SeoBook archives, recommended RSS feeds, no-follow highlighting, and a button which allows you to compare up to 5 domains at once.
Typically, I use the SeoBook toolbar as my research assistant of sorts when researching different aspects of a domain. It links through to the relevant tools I need to properly evaluate and research a particular domain.
An appropriate disclaimer would be that data can be limited on these free accounts but they can help establish a rough baseline to start off of. From the SeoBook toolbar you can easily link through to an SemRush report which gives you limited data on:
Organic keywords a site is ranking for
Keywords a site is buying in AdWords
Domain competition in organic SERPS
Domain competition in AdWords
Actual AdWords ad copy
Potential traffic/ad buyers/sellers based on the AdWords and Organic competitive data
A comprehensive review on SemRush can be found here.
Focusing on the organic keywords, you can get the top ten keywords driving traffic to a site (disclaimer: Spy tools should be taken as rough data points rather than data that is 100% accurate. In order to achieve 100% accuracy you'd need access to a site's analytics :D )
This can be helpful if you are trying to research whether traffic is heavily branded traffic or if it's more keyword centric traffic as well as the overall rankings of a site across a wide spectrum of keywords.
In the above example you can see that many of the top keywords are brands but they also rank highly for really competitive, core keywords. This conicides with our initial findings, via the SeoBook Toolbar, that this site is a very strong site.
If you wanted to dig deeper you can subscribe to one of SemRush's paid accounts. We also offer up to 1,000 results per query (organic data) with our Competitive Research Tool (which pulls data from SemRush) in both our membership options We also have our own custom data calculations inside the Competitive Research Tool which are pretty sweet :)
Compete is a more expensive competitive research tool but they do give you a fair amount of data for free on a domain.
So here is an example of the free data they give on a "Site Profile" report:
Some of the key points missing on a free account are (besides full access to the teaser data) are demographics and some deeper engagement metrics.
We can get some semblance of demographic data from Google Ad Planner and Quantcast for free.
This report can give you some, albeit small, keyword data outside of a Google tool in addition to traffic history (searching for victims of Panda as an example) and some high level signals about how many sites the domain is getting traffic from.
I would use a free Compete site profile to get a really high level overview of traffic size, top keywords outside of a Google tool, and traffic/visitor trends and history.
This report certainly lines up with the site being an extremely competitive one, a large brand with lots of traffic sources, and a site unaffected by the latest Google update.
AdWords Keyword Tool
So once you move away from looking at some keyword and traffic sampling numbers, as well as the solid high level overview provided by the SeoBook Toolbar, you might want to consider site structure and keyword structure.
A neat feature in the AdWords Keyword Tool is you can enter a domain and Google will list the keywords and the page assigned to that particular keyword (in their eyes):
*Other columns were removed to show this feature specifically:
This can be helpful in terms of breaking down the site structure of a competing site, finding profitable keywords they are ranking for but not necessarily targeting, and helping you plan your site structure.
Open Site Explorer
Since this post is on free tools, I would go with Open Site Explorer here (you could also use Yahoo! Site Explorer and Blekko for more data points but OSE offers a really quick, easy to use interface and has tons of link data).
Using this tool you can find things like the anchor text distribution of a site (see if they are targeting keywords that you might be considering or if lots of their anchor text is brand related)
Inside of OSE you can find other key data points like:
Top linked to pages on the site
List of linking domains
External linking pages
% of no-follow to followed links
% of internal versus external links
301 redirected domains/links
I do like using Yahoo and Blekko as well but I find that when looking at the free data options, OSE provides the deepest data out of the three and it's very easy/quick to use. On the paid side it competes with Majestic SEO which is a solid paid option as well.
I think Alexa can be somewhat useful when doing quick and free competitive research, but it's also a tool that gets a bad rap due to internet hype marketers promoting it as the BEST THING EVER!.
We did an in-depth review of Alexa here and a review of their paid tool here. Alexa gives out a few different data points:
Within those sections Alexa offers a lot of data points (based mainly on their toolbar data). Here we have data similar to Compete's:
You can also see things like global traffic ranks (where the site ranks in Alexa's Top Sites in each country)
Helpful information on where folks are navigating on the site (if you are in the same market are there site features you could be missing out on?)
Similar to SemRush stats but based on a smaller sample:
Trailing data on traffic being received from search engines:
Keywords that they are growing and keywords where they are slipping:
Potentially profitable keywords they are ranking for (factoring in advertising competition)
They also offer some demographic data compared to a relative baseline figure for each demo section:
Find out what sites people are coming to the site from (possible ad partners or related domains you can target in the same way you are targeting the current one from a competitive research perspective):
Where people are going when they leave:
Again, Alexa's data (like most spy tools) should be taken as rough figures rather than exact data. It's helpful to compare data from multiple sources as you can start to see patterns emerge or you can prove or disprove theories you may have about the site and your proposed method of attack.
Most sites I run across are not "quantified" so the data is a rough estimate (again).
So with Quantcast you can get more of that same traffic data along with some deeper demographic data:
This is on the overview page, there are separate sections for traffic and demographic data which break the information down a bit further:
You can also see data about what other sites are used/liked by visitors of the site you are doing research on:
This is on the demographics page and can give you an idea of what type of customer you'll be encountering which can help in determining how to present your offer and what to offer:
I like to use Quantcast mostly for demographc research on competiting or similar website (similar to products or services I am offering to help shape those offers and the presentation of my site).
Google Ad Planner
Ad Planner offers similar demographic data to Quantcast and similar traffic data to Alexa and Compete.
The big difference is the data is obtained from various Google products so it's probably somewhat safer to assume that the data might be a bit more relevant or accurate since Google has lots more data than any of the tools mentioned above (at least in terms of traffic data).
Ad Planner will show you "Google-ized" data for traffic patterns:
Unique visitor data in addition to Google Analytics data (for those who like to share)
You can also see top search queries:
As well as demographic data and audience interest data:
When to Go Paid
As you can see, free tools can give you lots of data but at some point you might have to scale up to use some paid tools. Paid tools certainly give you more data to work with but you can accomplish a lot of competitive research and background research on a domain with free tools.
There are many approaches to online navigation & discovery. On the surface many of them look exceptionally different, but when you look a bit deeper many of them are playing the same game with the same game plan. The main differences are brand perception & marketing angle, but they all copy off of each other.
New Ideas, or The Same Ideas?
There is an illusion that social media is significantly different because messages from friends are mixed in with the other stuff. However, when you look at the aggregate trends, ultimately social media pushes the same stuff that the mainstream media pushes (which is the same stuff that post-Panda Google pushes).
Huberman and three fellow researchers demonstrate that "user activity and number of followers do not contribute strongly to trend creation and its propagation."
Instead, says Huberman, "we found that mainstream media play a role in most trending topics and actually act as feeders of these trends. Twitter users then seem to be acting more as filter and amplifier of traditional media in most cases."
it's amazing how you can be the king at one point, quote unquote, and everyone loves you and the minute your time has passed and it's time for you to get torn down. When it's tear down time -- woah, it's harsh. There's no pity. They will hurt you however. Jabs, uppercut, hooks, and they want to see you on the floor knocked out. TKO. That's what they want. I saw it happen to some people, but when it happens to you, it's amazing how devastating it can be. When the negativity is directed at you, it hurts because you're there as an entertainer. You want to please your fans, have a good time and make them smile. You do the best you can on stage for them, and suddenly, you're nothing but a joke. - Fab Morvan, Milli Vanilli
Most of the "social" signals will generally promote that which is already large, or (for smaller businesses) that which is weird, funny, exotic, extremely biased, comforting & (falsely) empowering, fluff, or brutally honest.
If you are small, race toward an edge & stick with it. Own an idea. ;)
As I wrote earlier, many of them are playing the same game with the same game plan.
Other Search Engines
Blekko's slashtag model is all about highlighting content from the most well known sources in a niche. When you think about some of the really awful misinformation shared in the SEO space it is clear that popularity & quality are not the same. But if you have brand, sound authoritative, and speak of innovative changes; those can easily mask any factual errors. ;)
Bing's Stefan Weitz was recently interviewed by Eric Enge. In the interview Stefan points out how he thinks Bing sees the web differently than Google, in that:
they "look at the web as a digital representation of the physical world"
search will move from a web of nouns to a web of verbs, helping you to complete tasks in a more integrated manner: "there are enough services opening up their protocols and their APIs, Bing can then broker out that request to a number of different services across the web and stitch that information back together to help me go from I want to do this to I have done it."
"Humans have this primal behavior around the social experience where we almost always ask our friends and acquaintances for advice."
I highly recommend the interview as a must read to understand where search is headed. The speed of change in technology is increasing at a logarithmic progression & search is going to become much more of a complete end-to-end service.
The big takeaways for me from that read were that Bing will be doing hard brand pushes at some point (a digital representation of the physical world) & a lot more tight vertical integrations (a web of verbs). In summary, that means their game plan is just like Google's.
Sidewiki feels like another swing at something Google seems to desperately desires — a community of experts offering high quality comments. Google says that’s something that its cofounders Larry Page and Sergey Brin wanted more than a system for ranking web pages. They really wanted a system to annotate pages across the web.
Who are the real wine experts in Portland? Maybe the people who...
actually live there (confirmed by the address associated with their credit cards, the IP address of their home computer & the location information Android phones share with Google)
those who are consistently spending hundreds to thousands of Dollars a month buying wines (the more skin you have in the game the more weight Google can put on your feedback & reviews.)
those who review wines (Google can offer perks & bonuses to get people over the hump, & then add social game dynamics like badges)
those who review wines who have positive reviews from other wine drinkers (nothing like a little peer review to an academic mind)
What wine stores in Portland are the best? Probably the ones which the above wine reviewers shop at. Looking up driving directions can be seen as a relevancy signal.
What wine stores in Portland are less scrupulous? Those which are mostly given 5 star reviews by out-of-town (or, out-of-country) people who have never logged into Google from an IP address in the Portland area, never used an Android phone in the Portland area, and have never looked up driving directions around the Portland area.
At one point in time online was a new and (fairly) level playing field where one could win based on meritocracy. Increasingly though search is becoming "a digital representation of the physical world." To win online you will often be required to win offline.
With all the above data being included in Google's algorithms in the coming years (along with other brand signals) some people might decide that SEO is becoming too hard for the head keywords & that they are better off playing in the tail of search. But even that has 3 big problems:
if you have too many low quality pages Google can torch the whole of your site for it (and at this point it has been close to 100 days since Panda, and virtually no recoveries have been reported)
the eHow model will be reformulated at some point & added into some of the larger publishers that Google's algorithms boosted the rankings for, so if you are not one of those guys then they would still have a cost advantage over you through increased rankings & distribution (along with easier influence of social signals & such)
Certain categories with significant guilt or shame (say genital wart removal, criminal defense lawyers, etc.) won't have much end user data shared publicly (or, at least, users won't intentionally share that data with Google - though they may do so accidentally). Many (most) categories will have a sea of data available. And in those categories, at some point jumping through technical loopholes will be so tiresome & expensive that it will be cheaper and easier to create the signals Google wants to see through brand, public relations, and consumer experience than it is to try to fake them.
As search is becoming "a digital representation of the physical world" some of the best SEO tips in the years to come will have nothing to do with sitting at a computer. In due time, in search, there will be no security through obscurity.
The idea of an exact match domain (EMD) is that you are buying a piece of land right next to the highway. You sink in a lot of money upfront, but hope that it backs out over time by lowering your traffic acquisition costs. For many years this model was both logical and profitable.
At the peak of the domain name bubble recently, the domain name Poker.org sold for a million Dollars.
A domain name is an asset just like a stack of cash, a piece of gold, or a CDO is. But rather than having a fixed universal value, it is only a *relative* store of value that can go up or down based on market conditions. (Many of these other "fixed" stores of value also change in value when measured against other value stores over time, but they typically change value somewhat slowly and due to the acts of the entire market. With domain names, Google can use their dominant search position to drive massive changes in value in a short period of time).
Search Engines Influence the Value of Domain Names
Search is the primary mode of online navigation. For years search has been replacing almost all other forms of online navigation as the new default. There are about 7 billion people in the world with about half of them online. Google likely gets about a search per person every day!
Search engines can decide what variables they want to count & how much. In a world where subjective marketing aspects (like branding) are replacing signals of relevancy the value of keyword domain names is greatly diminished.
If your model works out to where it takes 3 or 4 years to break even & recoup the initial investment, then that model may look quite different if Google manages to redirect 50% or 60% of that traffic stream at some point in time ... at some point the price of the domain has to adjust to the new market conditions.
An Example of the New Normal in Search
With the above in mind, I thought it would be worth highlighting how the domain bubble grew & ultimately popped.
First, lets start with a current search result. The below example is for "pool tables."
Note that brands get a number of options to play here: AdWords ads, AdWords product ads, Google Product Search, branded navigation, big brands in the organic search results, niche vertical brands, and any local results for nationwide chain brands with a local outlet. Go back a couple years and this search result would have mostly been dominated by smaller online retailers & niche hobbyist websites.
The below image is from 2008. Notice how small the AdWords ads are & how 7 of the 10 organic listings on page 1 have "pool tables" in the domain name.
Today most of those results are off onto page 2 or 3 or beyond, where few searchers dare to go. Now even the exact match domain is forced to buy AdWords to compete for it's own name. Without the AdWords ad, the exact match domain would require a searcher to skip over 45 other links before finding it somewhere below the fold.
Other keywords (like engagement rings) which once left room for review & comparison sites have been completely dominated by brands. Outside of end consumer reviews (and who but an expert publicly reviews more than 1 engagement ring? and who is not biased in their review of said rings with emotional attachments?) there is no way to get a comparative view of quality. There is no room for such an idea in Google's brand-only search results.
Update: Checking back in a few years later ... as of August of 2013 this post has only grown more true over time. In the following image, notice how the ads keep on scrolling, the size of the local result insert increased & now even the .com EMD "category killer" isn't even on page 1 of the search results any longer.
The algorithm is only going to keep adding more signals that boost brands. PoolTables.com might have better editorial content than a mega-retailer like Amazon.com, but it is hard for them to collect as many reviews as Amazon can.
For commercially viable keywords these have the net effect of pushing the organic search results further down the page. A recent study by Optify highlighted that while low CPC & tail keywords send most clicks (~89%) to the organic search results, for high CPC & head keywords AdWords ads consume most search clicks (~ 60%).
Google Comparison Ads
In certain high money verticals Google offers Google Comparison / Google Advisor ads, which allow them to place a 4th ad slot above the organic search results.
Notice how much larger some of these ads are than typical ad units. When Google targets your keyword with one of these ads they significantly change the dynamics of the market.
Google has offered graphical product ads automatically matched to the search results. Generally for bigger brands Google offers these on a risk-free cost per acquisition pricing, whereas smaller advertisers need to pay by the click to use this ad format.
Googler announced that searchers clicked on this ad format nearly twice as often as regular search ads & in some cases Google has even started testing including these ads in their ad space that appears above the organic search results.
Search clicks are a zero sum game, so the more risk-free clicks the big box brands get from this ad format the lest clicks there is to go around for everyone else.
Product Search Listings
These serve as more eye candy to distract searchers from the organic search results. Once again these typically feature listings from larger brands & Google doesn't mind if these are a bit off because they still push the eye away from the organic results and toward the AdWords ads.
Look how off those "necklaces" are. Evidently if you are not a sport's fan you have no business wearing necklaces ;)
Localization is a boon for small local businesses which can now gain a slice of the local traffic stream that they were priced out of the market on. However, as a domain buyer, the value of AutoInsurance.com drops significantly after the large metro areas have localized results which do not allow the cost of an expensive domain to be amortized by the potential to rank everywhere. What is worse, is that the largest cities are the ones with the most vibrant economic activities (more businesses, more residents, larger loan sizes, and so on). Through localization any generic unbranded nationwide player simply misses out on the most valuable traffic.
Verticalization & Double Dipping Ads
Much like how localization locks generic players out of local markets, Google's increased verticalization (and allowing certain brands to double or triple dip on ad serving) now means that some results have over 80% of the screen's real estate dominated by a single key player.
Search Box > Address Bar
When Google Chrome launched it replaced the address bar with a search box.
That allows Google to...
intercept & redirect type-in traffic demand
re-highlight content you have already seen in the past (likely to be from some larger brands, as they have larger ad budgets & more ways to be found)
recommend popular searched-for keywords (which are often brands, since awareness-based advertising creates search demand
When Internet Explorer 9 was launched Microsoft also adopted these features
Google Instant's search auto-completion directs users away from some keywords and toward others. At first that statement seems like it could be saying that it consolidates search volume to a smaller set of keywords & thus could make domain names more valuable. However, if you have ever looked at a list of the most popular keywords you would know that they are largely filled with branded keywords. The media was aware of this obvious shift & Amit Singhal had to do an interview stating that there was no brand bias to Google Instant.
In the most recent beta Google has tamed this down a bit from the absurdity they were first testing, but Google has shown an interest in using whitespace trickery to drive the organic search results further down the page.
The rise of mobile applications & mobile search devices further pull leverage away from publishers & toward ad networks.
What's worse, is through personalization they have an asymmetrical information advantage over publishers in their ad network. They can tell you that you are getting 68% of the value of an ad click, but how do you know if they don't undervalue the contribution of that click while overvaluing the contribution for clicks where they keep 100% of the income on?
Google Small Business Taxes
Some sites get the benefit of the doubt, whereas other sites just get doubt. I highlighted how Google's approach to link buying, AdWords penalties & other issues vary based on who is getting whacked in our recent post about Google small business taxes.
Too Small to Matter
Smaller sites are more likely to come under attack from "the algorithm" as they are easier to knock over & are generally less stable. That gives them a higher risk factor & makes it even harder to build reliable business processes around it. How do you scale employment (or even inventory) when one month you are up 50% and the next month you are arbitrarily off 60%?
Further, Google has consistently screwed up original source attribution, which makes it even harder to justify for a small business to go the extra mile & spend extra money creating premium content, if the result will be Google paying someone to steal that content & wrap it in AdSense ads.
Where Does this Lead Us?
If you buy a "category killer" it is critical that you rank #1, but in many niches the exact match domains that ranked #1 for nearly a decade are now #3 or #4 in the organic results. Add in 3 AdWords ads above the organic results & things like product ads and it isn't hard to end up below the fold. If your relationship to that 1 keyword is your core competitive strategy but you can't even promote the keyword (because you are below the fold) then the strategy is a failed one.
Further, as Google keeps adding more usage signals into the relevancy mix that will keep favoring brands.
This is not to say domain names are dead across the board. there is still plenty of opportunity in some areas, but equally some names require large investment & as an SEO strategy may get thrown under the bus by any of the above (or similar future moves in other market niches).
I Stopped Buying Domain Names
I believe I was one of the first SEOs to publicly highlight the benefits of exact match domain names. Back when Google engineers were dismissive of it some of the smart money was dismissive of what the engineers stated and made plenty of money from it. But I have prettymuch stopped buying domains at this point...as in most cases the valuations generally don't make sense on a risk adjusted basis in the current market (let alone what the market will look like after the introduction of +1 & other brand signals).
Deep Pocketed SEOs Are Selling Their Domain Names
The person who was likely the single SEO most responsible for running up the price of exact match domain names (he over-paid for some of them based on the presumption that the numbers would back out similarly to some of his earlier investments in a market that was dominated by a government-sponsored bubble) has now become a domain seller.
You don't get much more amoral capitalist opportunistic than this person is (see the following before and after for his payday loans effort)
In March Matt Cutts talked down exact match domain names, but the truth is that Google never really needed to discount them, simply by adding more criteria to the relevancy algorithm which boosts brands they already had the same impact.
Search has moved away from relevancy toward promoting brands. As SEOs we don't control Google. We can only focus on promoting that which they reward.
The smart money is now saying that domain names are generally significantly overpriced, especially as an asset class valued based on SEO potential.
One of the more perverse things about online markets is that since things are not done in person a lot of people are willing to work in the gray area. That allows people like Vitaly Borker to torture his customers as a marketing strategy. ;)
When Stanley Milgram did his famous study on if people were willing to torture others (so long as they thought it was part of a scientific experiment), he found out that the more distance they put between the torturer and torturee the more brutally they would behave. People would dial the voltage up to x if they were holding the hand of the person getting shocked, a bit more if the person was on the other side of a glass wall, and a bit more if they were behind a brick wall.
Many online economic innovations are ultimately scams, where the cost is hidden, with the scammers operating one step ahead of the legal system. Swoopo was basically a semi-legal form of gambling that took advantage of people's ignorance of math. As soon as it started working dozens of companies followed suit. The rage was so popular that people would buy overpriced "credits" to spend their "credits" to bid on the ability to get more "credits" at a slightly discounted rate. :D
OMG I Just Lost 3,000 Pounds!
The fake blog (flog) strategy where a woman writes about how she lost 40 pounds in a month due to Açaí & colon cleanse was an economic "innovation" where affiliates could rope a person into 2 fraudulent reverse billing scams at the same time.
Public relations hacks were quick to adopt the fake blog "innovation" for their corporate clients & such gems like Walmarting Across America & Working Families for Wal-Mart were born.
In Other "News"
Affiliates who needed to differentiate from all the other affiliates needed a new round of "innovation." So they would take the items or categories or topics they were selling something similar to & add a "as seen in" section on their websites highlighting how something tangently related to what they are selling was once mentioned in the media in some way.
From there it was a quick jump to fake celebrity endorsements & fake news sites offering "a special report on how you can get rich overnight" as a millionaire who wants to give back naively shares everything they know for $4 shipping in an anyone-can-do automated wealth generation blueprint, while losing 50 pounds in 1 month. :)
Step by Step
Ultimately sales is not an event, but a process. Which is why the above mentioned reverse billing scams only charged shipping off the start, so the perceived risk was lower than the actual risk (much like bad faith insurance companies that take your money until you need what you paid for, and then manage to disappear).
Everyone Pays for Fraud
The scammers eat off the plates of everyone honest in the marketplace. Multiple times a month I get refund requests from people who bought something totally unrelated to us from Clickbank, demanding I give them their money back for something that is in no way related to us. I highlight that refunds at clickbank.com is where they need to send their emails, but for every person who manages to get their money back there are likely 5 or 10 more people who are disillusioned & less trusting of online markets & online marketers.
Google Polices Scams *Everyday*
Circling back to the SEO & PPC niche, Google has to deal with the scams & scammers every day. In certain cases where it is exceptionally profitable (say illegal prescription drugs, knock off goods, Obama mortgage modification) Google may choose to look the other way / have lax policy enforcement, but at some point looking the other way creates financial risks & a risk of brand damage.
Doing What is Easy = Doing What is Best?
Generally when it comes down to it, Google's revenues are heavily concentrated & no matter how widespread or profitable any individual scam is, they can cut it out of their ad network without being overly concerned. Last September AdAge shared data showing that there are around a couple thousand advertisers spending over $10,000 a month on Google ads. While the longtail concept is widely praised, in the search ecosystem it reflects more on products rather than merchants. Some time ago SEM Rush sent me the following chart highlighting AdWords spend breakdown estimates.
Their numbers are based off of the publicly shared Google keyword estimates (which generally skew toward trying to get advertisers to spend more), and I believe they may count certain AdWords ad management platforms as being 1 advertiser each, but yet again Pareto principal appears in the data & Google could keep roughly 80% of their US search revenues if they only accepted ads from the top 5,000 advertisers (out of more than 300,000). And those estimates were before Google widely launched their CPA product ads, which only further consolidate traffic to a smaller number of websites.
Ultimately Google can try to police an ecosystem of x thousand companies or an ecosystem of x million companies. The latter will have more innovations in it, but also a wider variety of scams & be much more expensive to police. This is a big part of the reason Google has felt the need to become more & more portal-like over time. Their engineering culture assumes that they can do it better internally.
Depending on how your positioned, their anti-innovation "do what is easy" approach to search can either be a boon or a major hindrance. The companies that were worried about maintaining complex search strategies which sorta fell into top rankings on the Google brand +1 obviously benefited. But Panda (and Google's approach to AdWords) keep raising the bar on smaller businesses.
I thought it would be worth highlighting a few examples:
Google Arbitrage Tax
A friend recently had their AdWords account penalized for running an "arbitrage" business model.
the site wasn't advertising for revenues, but to build links & awareness
the ad campaign was inactive
the site had since been sold & the buyer added another ad unit to the page
To appreciate how arbitrary the above editorial judgement is against that webmaster, not only were they 0% responsible for the alleged editorial infractions, but 3 hours after the sincere "notification only" email Google did a follow up which said the site was once again approved to advertise, but the ads need to be re-submitted for review.
We live in a world where flagrantly parasitic sites like Mahalo required "an algorithm" (with endless collateral damage) to fix & yet the above sequence is somehow a reasonable way for Google to treat there paying customers.
Much like there is "too big to fail" there is also "too small to matter" but if you look at Google's numbers you can see how that happens. We are basically expecting them to be better than all other monopolies if we expect a level playing field.
Google Disclosure Tax
Some smaller advertisers may get their AdWords campaigns disabled because they offer a freeware trial, where a person has to buy the full version to unlock all of it's features. Google suggests that these advertisers lead with the disclaimer that a customer may eventually spend a Dollar if they like what they are trying, but sales is a process rather than an event. Sometimes you first have to show proof of value before people are willing to spend money. The problem is that if you are paying for every click & you have to lead with your disclaimer of potential cost for a full upgrade (and so on) then you are not going to be able to compete with a larger brand that does not need to clearly display disclaimers.
You can only have a strong conversion-oriented page with message clarity if you are a brand, otherwise you need to LEAD WITH THE DISCLAIMERS, which never works if you are paying by the click in an efficient market if you are taxed x% of conversions upfront. Not only are brands given leeway, but Google also allows their own products & services to routinely violate the terms of service they push onto others. The footer on Google Advisor states "The information displayed here was provided directly from the issuer and/or collected from the issuer's website. Check the details pages for each offer for the data source and update time. Google is not currently being paid for these listings."
Note the "NOT CURRENTLY." So everyone else is required to place the entire info harvesting policy next to the form, but Google can have a "google protects you from spam, Learn more" link which they specifically prohibit just linking to as an AdWords advertiser. Where do you see EXPLICITYLY how you can opt out of communications? Look at the table at the bottom of this page.
If you are creative enough you can say that disclosure doesn't have to kill conversions, but the problem is that sometimes it does. And when it does, it is not like there are many other avenues for you to compete online. In some markets Google owns over 95% of the search marketshare.
Part of the reason Google & Facebook are willing to invest into smearing each other publicly is they are trying to damage each other's brands so the other is trusted less, so that they may build an asymmetrical information advantage.
The big issue with data harvesting is that one company's sales funnel or cross-marketing campaign is another's data harvesting scheme. Just like the tax on disclaimers, this is another area where larger businesses of scale get an economic advantage by being able to get more benefit of the doubt on their approach
Google's I am Duplicate Content Tax
In the past I highlighted how some webmasters have trouble ranking for their own content due to duplication & how it could even impact their ability to buy ads. Ultimately this leads to some level of fear & indecisiveness amongst many small business owners who may become afraid to have duplicate content anywhere on their sites.
Brands rank because Google puts weight on brand-like signals (and they can get away with link buying which smaller competitors would get torched for). Then Google claims that the link buying is irrelevant because it doesn't influence the search results.
With that sort of circular logic why does Google even care about paid links at all? Why do they arbitrarily police one segment of the web?
On the flip side, there is a suspicion of smaller webmasters which is core to how Google operates (even if the site they are passing judgement on ranks right at the top of the search results for core industry keywords & is linked to organically from top newspapers). Sometimes it is a "shoot first, ask questions later" approach.
Why, in this instance, do they suggest they can identify the problems & that a business owner should first be penalized for it, then need to fix it before reconsideration, all the while some larger businesses do the same exact things but Google looks the other way because they claim to have already detected it?
Why does one form of detection require penalization while the other doesn't?
Unstable Business Tax
During the Panda update many small businesses (which did nothing wrong other than participating on the same Internet that the likes of eHow abuses) got torched. They were given abstract qualities where they need to improve on. Over 3 months later they are still in the dark hoping the penalty clock hits zero before the bankruptcy clock does.
When traffic heads south & becomes unreliable a business not only loses that income, but it also loses negotiating leverage with suppliers.
The uncertainty not only retards investment, but may lead to physical, emotional, social & mental health issues. People make better life decisions when they are driven by love than when they are driven by fear.
The one big counter-trend to the move away from small businesses is Google's increased emphasis on localization, which allows some small local businesses to participate in search markets that they were simply priced out of in years past.
If you go into the fashion market Google has AdWords, Product Ads, AdSense, the Google affiliate network, Boutiques.com & branded Youtube ads. Like.com, which Google purchased in part to build Boutiques.com, also remains a live website & yet another way for Google to taste the fashion market.
In addition to AdWords, AdSense, Product Ads, product search, merchant reviews & their general web index, Google is trying to pull in information from the offline world. They are set to announce a mobile payment system with the ability to include coupons:
For Google, the system could help boost its digital advertising business. The planned payment system would allow Google to offer retailers more data about their customers and help the retailers target ads and discount offers to mobile-device users near their stores, these people said. Google, which hopes to sell ads and discount offers to the local merchants, isn't expected to get a cut of the transaction fees.
In addition to receiving targeted ads or discount offers, users could manage credit-card accounts and track spending, loyalty points and other things through applications on their smartphones.
Google also invests in technologies that blend ads in content, like VigLink:
tools will allow publishers that opt-in to insert new links automatically into their content, rather them finding the links themselves.
Roup said one of the biggest misconceptions that marketers have begins with Google's disapproval of the affiliate marketing model. Google does not have an issue with affiliate marketing, but rather, with marketers trying to buy page rank -- or links that are paid but try to fool the consumer and appear as unpaid, Roup explains. "Our links are financially motivated, so they don't convert page rank, but neither do any other affiliate links," he said.
Google, which claims that you need to disclose any form of paid links in human & machine readable formats now invests in automated paid links, with blurred & inconsistent levels of disclosure. See for yourself.
Other Vertical Projects
Google can make minor design tweaks to their productivity suite and then launch it under any label they like, from project management to wedding planning.
In dirty ad markets where illegal goods & services are pushed Google can (and does) monetize them at the general ad network level until they cause public relations issues, allowing Google to capture a large portion of the reward with minimal risk.
While Google claims ad disclosure is important, they can "accidentally" leave it off when convenient. I just recently saw the following ad served by Google's DoubleClick. I have no idea what it was promoting though, as I was afraid to click on it.
It's Not Just Google
Obviously I mentioned Google because they are the most successful search company. However, Google is not the only search company monetizing their organic search results & pushing results below the fold.
Cisco, the maker of Internet routing gear, customized its technology to help China track members of the Falun Gong spiritual movement, according to a federal lawsuit filed last week by members of the movement.
The lawsuit, which relies on internal sales materials, also said that Cisco had tried to market its equipment to the Chinese government by using inflammatory language that stemmed from the Maoist Cultural Revolution.
And that from a company which promotes itself using the label "the human network."
And how did Cisco react when the above information became public? "When evidence of the company’s activities in China became public in 2008 through a leaked PowerPoint presentation, Cisco disassociated itself from the marketing materials, stating that they were the work of a low-level employee."
That is what big brands do. The PR team steps in and says "Oops it was a rogue marketer/trader/monkey/employee who was smoking crack at work and they have now been fired. We were ignorant of our actions but we really care about people. We promise to not (get caught) doing it again!" TM
As Google pushes to make the web more corporate, it is worth taking a step back and considering what that means for "the human network."
Google likes to pretend that something is good just because it is a big brand, but many big brands have big ad budgets *precisely* because their business model contains hidden costs. For instance: bad faith insurance which takes your money as long as you pay & then disappears the minute something goes wrong.
The legal system granted large corporations more rights than human beings. Not because they are any better, but because they are more corrupt. I bet many Google engineers are disappointed to see Google following suit & taking the easy way out. Spy & personalize. And when in doubt, brand, brand, brand. ;)
With the vast potential of the web should we settle for making it as corrupt (or more corrupt) than the real world?
If Google's comments and actions of late, are anything to go by, the chances of the little guy, armed only with SEO chops, being able to compete with deep-pocketed corporates are becoming less and less likely. Google algorithms tend to reward the big players - the people everyone talks about, and links to.
How can we combat this situation?
Back To Business
Ever notice how a page on FaceBook, or some other behemoth site, which consists entirely of a Wikipedia cut-n-paste, can often rank well on Google? At the same time, many unique, interesting pages are buried deep on SERP #20?
It's happening a lot.
It's hard to fight against a domain that can distribute high link authority down through hundreds of thousands, or millions of sub-pages. SEO chops alone are unlikely to cut it if your niche is full of such sites. The game is rigged, and it doesn't favor you.
One approach is to not fight such competitors at their own game.
Instead, take a new look at your business. How unique is your offering? Are you competing with many other sites that offer pretty much the same thing?
If you offer a similar product and service to all the rest, then it is inevitable that you'll eventually lose to the company with the deepest pockets. Google, and the world in general, tends to reward those who already have the most.
I'm sure you've heard about the Unique Selling Proposition.
For those who haven't, the Unique Selling Proposition, or USP, is the term is used to refer to an aspect of a service or good that differentiates it from similar services or goods.
For example, a USP of Amazon is that it sells the widest range of books online. Your local rare bookstore, on the other hand, has a USP of stocking and selling rare books. Both Amazon and your local bookseller sell books, but their services are clearly differentiated from one another.
The concept of a USP came about as a result of a marketing problem that exists when markets are crowded. If many companies offer similar things, then how can any one company stand out?
A USP isn't critical if there are few players in a market. This was the case in the early days of the internet, when finding a site that met your needs wasn't assured . As the internet became more populated, webmasters used techniques such as SEO in order to rise above the masses, safe in the knowledge that searchers will typically click on the top few results. They still do, of course, but if Google increasingly favors the most popular sites, then the return on SEO for the smaller player decreases.
These days, with plentiful options, the searcher either finds what they want on their first search, or they rephrase, and make their search more specific. It is in the second option where the most opportunity lies for the little guy. The visitor is rephrasing in order to be more specific. "Dell Monitor Cheap" may become "Used Dell Monitor Free Overnight Delivery". Vagaries of Update Panda aside, the guy who has a USP of dealing in used monitors, and offers fast delivery times, can still compete in Google.
The USP isn't just an add-on marketing tactic. It's a fundamental aspect of your business.
USP's are about specific benefits for the customer. Put yourself in the customers shoes and ask "how does this benefit me?". In the example I used above, the benefit is "a low cost, recycled monitor that will be delivered quickly".
The twist is that you need to make your offer unique. Look at your competition and ask yourself "what aren't they doing that they should be doing, and that the customers wants"? If you find it difficult to answer such a question after having evaluating your competitors, it may be a sign the market is too crowded, and you may be better off trying something else.
But What If You Can't Move Niches?
There are various ways to introduce a USP if you're selling a similar product or service to others.
One idea is to make your process unique by making your site more usable.
For example, I buy cases of discount wines online. Whilst there are many other sites offering this service, I use one particular site mainly because the ordering process is so streamlined. The benefit is saved time. The site retains my login and billing details, and it prompts me for re-orders with emails sent out at intervals based on my previous order history, and the previous selections I have made. The site pretty much "knows" what I want before I've even thought about it, and I can order with a couple of clicks. The wine always arrives promptly.
So their USP is in their process. They sell the same wine as the other sites, but the process is "unique", from what I can tell. It's also troublesome for me to switch. It invites a set-up cost (time), risk (they may not deliver), and I lose my history.
What's this got to do with SEO?
Once your visitor finds you, give them a very good reason to bookmark you, join, and keep coming back. Once that happens, you don't need to rely on new leads form Google so much.
A USP Must Be Supported By The Fundamentals Of Your Market
It's not enough to just come up with unique angle.
The unique angle has to be workable. There has to be a niche of people who want the unique aspect you deliver, and are prepared to pay you enough for it to make the effort worthwhile. For example, offering fresh pizza in the middle of a desert may be unique, but it is unlikely to succeed as a business model, because of low demand.
Finding a workable USP is a matter of research, and trial and error. Look thought the search keywords related to your term and look for an angle. What are people asking for? Type that keyword term into Google and see if anyone is servicing that demand. Ask your existing customers what they want, or what you could do better. Buy third-party research to help discover where the market is heading, and how demands are changing.
Imagine the future, as opposed to mimicking the past.
How To Define Your USP
1. List Your Key Benefits
What aspect do you do really well, and that other people really like? If you're at a loss, what could you change to make it so?
2. What Pains Your Customers?
They kinda want something. They might vaguely feel they need it. But if you find something they absolutely must have, so much so that it pains them not to have it, then you're onto something big. What is that thing?
3. Be Specific & Provide Proof
It's one thing to say it. It's another thing to do it. How many sites say "we're the best". Or "Experts in SEO". It's meaningless.
"We get your site thousands of qualified visitors at half the cost of your Adwords spend" is a specific, meaningful benefit.
Then you need to show how you do that. Case studies are great. You can seldom have enough case studies. Say what you were going to do it, do it, then tell them you've done it.
4. Be Concise
You only have a few seconds. You need to state your USP quickly. Short phrases. People read the first line, then the next, but only if the first line was worth reading. They'll scan through to pick something that interests them.
This is where graphic design is important. Pictures really are worth a thousand words if the person is scanning for information. Does you graphic design underscore, or detract from, your USP?
5.Your USP Flows Through Everything You Do
If your USP is, say, to provide individual attentive service, then you need to answer the phone right away. You need to respond to emails quickly. You need to make it easy for people to talk to you.
If you USP is a massive inventory, then the user has to be able to get to that inventory easily.
You can never repeat your USP too often. Do so on many levels. People aren't really paying attention, so take every opportunity to remind them what is special about you :)
Somebody, it seems, hired Burson-Marsteller, a top public-relations firm, to pitch anti-Google stories to newspapers, urging them to investigate claims that Google was invading people’s privacy. Burson even offered to help an influential blogger write a Google-bashing op-ed, which it promised it could place in outlets like The Washington Post, Politico, and The Huffington Post.
And why would Facebook run such a campaign?
Confronted with evidence, a Facebook spokesman last night confirmed that Facebook hired Burson, citing two reasons: First, because it believes Google is doing some things in social networking that raise privacy concerns; second, and perhaps more important, because Facebook resents Google’s attempts to use Facebook data in its own social-networking service.
So now Facebook is trying to position itself as an advocate of consumer privacy rights?
Google Inc. is close to settling a U.S. criminal investigation into allegations it made hundreds of millions of dollars by accepting ads from online pharmacies that break U.S. laws, according to people familiar with the matter.
The pharma corporations are powerful & are in bed with the government. In spite of repeated felonious behavior in marketing their drugs for illegal off label use (which has literally murdered millions of people) these companies can have the government step in and protect their property rights, by having the government enforce unto others the same laws that these same pharma corps regularly break (literally killing millions of people).
Maybe Google is Philosophically Opposed to Property Rights?
While Google tramples on the property rights of everyone else, the first sniff of someone operating anything like they do drives Google into black-ops mode & they conduct a smear campaign. Google launched Buzz without warning, but when their feared Facebook was collecting more personal information than they could Google went into black ops PR mode warning against security issues in Facebook.
Remember that bogus "Bing is copying our results" stuff Google engineers did earlier this year? Google later rolled out their content farm update & many of the sites which were torched by Google are now getting more traffic from Bing. What does that tell us? If Bing was putting *any* significant weight at all on Google rankings & traffic then why didn't that carry any weight when Google torched a bunch of websites?
Here is the Google traffic profile for a site that was torched by Panda
And that same website's Bing traffic
Google traffic fell through the floor, while Bing traffic kept climbing. Some sites that were hit by Panda are getting more visits from Bing or Yahoo! Search than from Google.
Conclusion? Once again Google distorts media to promote itself & its business interests, while bogusly smearing competitors with fabricated trash.
Part of why Microsoft's search marketshare is less than Google's is that Microsoft is willing to block sleazy traffic partners, unlike Google. But Google's treatment of their partners is inconsistent. Using "inside voices" Googlers openly explain in plain English how they treat their partners: "we are using compatibility as a club to make them do things we want" - Google's Dan Morrill.
Big Companies Hate Honest Market Innovation
Large companies are largely counter to honest innovation in the marketplace. They are comfortable atop the perch and want to lock down innovation to maintain their current dominance.
Sure the big banks welcomed CDOs, MERS, etc. ... but those were welcomed precisely because they were part of an elaborate scheme of dishonesty and fraud. But the same society which brings us CDOs built on fraud (that ultimately cost you your job, your house, your retirement savings, the value of the currency, etc.) is also a society where dirty corporate whores push to force smaller market competitors to be entirely transparent.
I have for years been telling you even if you have no interest in the new gTLD’s you had to pay close attention to the process as whatever rules come out of that process will be attempted to be applied to all existing TLD’s including .com, .net and .org.
This is especially troubling because as you know the new gTLD process has not even been approved yet since the .Net contract is up for renewal, trademark groups are going to push for this new system to take away domains, be imposed on .net
The very domain name of the front organization that is pushing to remove domain privacy is registered using a private registration. ipconstituency.org uses Domain Discreet!
When a consumer or small business owner gets caught (acting like a big business, and) doing something illegal they go to jail. When a big business repeatedly commits serious crimes the wost thing that could possibly happen is a shake up of management. A company has no soul. A corporation can't go to jail.
Since the Panda update happened, some scraper websites (monetized by Google AdSense) have started outranking Patrick for his own content.
Panda = No AdWords Soup for You
Distraught with the decline in traffic, Patrick turned to AdWords to try to bridge the gap and drive some revenues.
Unfortunately, Google wouldn't let him do that either:
I asked on what grounds he had decided that my site does not produce original content. His answer was that he had typed a sentence into Google and found it contained at many sites around the web. Seriously, I made a lengthy strong case for my site's record of having 100% original content and he typed one sentence into a Google search.
I emailed back and asked him to be specific about his search. This was his reply:
"An example of a specific sentence that appears in multiple websites is "a superb app for iPad and iPhone that lets you quickly and easily transfer photos and videos between iOS devices and computers – has been updated this week, to Version 2.3."
Google Rolls Out the Red Rug (for AdSense Scrapers)
Think about how perverse this is:
Google algorithmically penalizes your site
Google won't say why it is penalized, other than some abstract notion of "quality"
Google offers no timetable on when things can improve, but suggests you keep spending increasing sums of capital to increase "quality"
Google pays scraper sites to steal your content & wrap it in AdSense ads
Google ranks the stolen content above your site (so the content has plenty of "quality" but it is just not "quality" on your website)
Google tells you that you can't even buy AdWords ads, because you are now duplicate content for your own content!
Contributory Copyright Infringement
So now we have Google telling advertisers "I won't even take your money" precisely because Google is paying people to steal their content. Small publishers likely don't have the capital needed to sue Google, but clearly what Google is doing here *is* flagrant, systematic, abusive, and illegal (contributory copyright infringement).
One of Google's larger enemies may want to fund some sort of class-action lawsuit. Google deserves far more of a black eye than they have got in the press from the embarrassment that is the Panda update.
Um, Could You Please Help Me Out a Bit Here Google?
Yet Another Webmaster Loses Faith (& Trust) in Google
Since Google has ignored him (for months), Patrick felt he had to rebrand & redirect his old website to a new iPad website. Google made (a rather long and egregious series of) mistakes. And he had to pay the price for it, because Google is a monopoly that doesn't give a crap about how destructive their business is on the ecosystem, so long as it increases their profits.
Now that Google is aware that the panda fallout is costing THEM money, it will likely get cleared up quickly. I suspect to see an update within the next couple weeks at most. And it would happen even quicker if the press actually did its job. ;)
Update: Matt Cutts stated that the site wasn't hit by Panda here, so that wasn't what caused this. However that still means that Google has to work on better highlighting original content sources over the scrapers, stop funding the scrapers via AdSense, and improve the internal policies which state that you can't buy ads if a scraper outranks you for your own content!
Here is a fun webmaster help video from March 10th of 2010, answering the following question:
"If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?"
The answer kinda jumps around a bit, but here is a quote:
I believe if you were to talk to our crawl and index team, they would normally say "look, let us crawl all the content, we'll figure out what parts of the site are dupe (so which sub-tree are dupes) and we'll combine that together.
Whereas if you block something with robots.txt we can't ever crawl it, so we can't ever see that its a dupe. And then you can have the full page coming up, and then sometimes you'll see these uncrawled URLs where we saw the URL but we weren't able to crawl them and see that its a dupe.
I would really try to let Google crawl the pages & see if we can figure out the dupes on our own.
Trust in GoogleBot
The key point here is that before you consider discarding any of your waste you should give GoogleBot a chance to see if they can just figure it out on their end. Then, without updating said advice, Google rolled out the Panda update & torched 10,000's of webmasters for following what was up to then a Google best practice. Only after months of significant pain did Google formally suggest on their blog that you should now block them from indexing such low value pages.
Matt's video also suggested some of the other work around options webmasters could do (like re-architecting their site or using parameter handling in Webmaster Tools), but made it sound like Google getting it right by default was anything but an anomaly. What such advice didn't take into account was the future.
What Does a Search Engineer Do?
The problem with Google is that no matter what they trust, it gets abused. Which is why they keep trying to fold more signals into search & why they are willing to make drastic changes that often seem both arbitrary & unjust.
Search engineers are well skilled at public relations. A big part of what search engineers do is managing the market through FUD. If you can get someone else to do your work for you for free then that is way more profitable than trying to sort everything out on your end.
Search engineers are great at writing code. A lot of what the search engineers do is reactionary. Some things get out of control and are so obvious that FUD won't work, so they need to stomp on them with new algorithms. Most search engine signals are created through tracking people, so they usually follow people. Even when it seems like they are trying to change the game drastically, a lot of that data still comes from following people.
What to Do as an SEO?
The ignorant SEO waits until they are told by Google to do something & starts following "best practices" after most of the potential profits have been commoditized, both by algorithmic changes & a market that has become less receptive to a marketing approach which has since lost its novelty.
The *really* ignorant SEO only listens to official Google advice & trusts some of the older advice even after it has become both stale & inaccurate. As recently as 2 years ago I saw a published author in the SEO space handing out a tip on Twitter to use the Google toolbar as your primary backlink checking tool. Sad!
The search guidelines are very much a living breathing document. If search engines are to remain relevant they must change with the web. Those blazing new paths & changing the landscape of internet marketing often operate in ways that are not yet commonplace & thus not yet covered by guidelines that are based on last year's ecosystem. Individual campaigns fail often, because they are trying something new or different. Off of each individual marketing campaign the expected outcome is failure. However they generally win the war. Those who follow behind remain in their footprints (unless they operate in less competitive markets).
The savvy SEO is a trail blazer who is pushing & probing to test some of the boundaries. They are equally a person who watches the evolution of the web through the lens of history, attempting to predict where search may lead. If you can predict where search is going you are not as likely to get caught with your pants down as the person who waits around for Google telling them what to do next. It may still happen in some cases, but it is less common & you are more likely to be able to adjust quickly if you are looking at the web through Google's perspective (rather than through the perspective they suggest you use).
Google's Noble Respect for Copyright
Google has a history of challenging the law & building a business through wildcatting in a gray hat/black hat manner.
They repeatedly broke the law with their ebook scanning project. Their ebook store is already open in spite of a judge requiring them to rework their agreements.
They bought Youtube, a den of video piracy & then spent $100 million on legal bills after the fact. When they were competing with Youtube they suggested that they could force copyright holders to pay Google for lost ad revenues if they didn't give Google access to the premium content. :D
They sold ads against trademarks where it was generally viewed as illegal and awaited the court's decisions after the fact.
They tried doing an illegal search tie-up with Yahoo & only withdrew after they were warned that it would be challenged. They later slid through a similar deal with Yahoo Japan that was approved.
They "accidentally" collected personally identifiable information while getting router information & scanning streets (and we later learn via internal emails in court documents how important some of this "accidental" data collection was to them).
They pushed Buzz onto Gmail users and paid the fine.
Google torched UK finance comparison sites for buying links. Then Google bought one of the few they didn't torch (in spite of its spammy links). After getting flamed on an SEO blog they penalized that site, but then it was ranking again 2 weeks later *without* cleaning up any of the spammy links.
When the Panda update torched one of your sites Google AdSense was probably already paying someone else to steal it & outrank you. Google itself scrapes user reviews & then replaces the original source with Google Places pages. The only way to opt out of that Google scrape is to opt out of Google search traffic.
For years Google recommended warez and keygens and serials to searchers, all while building up a stable of over 50,000 advertisers pedaling counterfeit goods. That only stopped when the US government applied pressure, and then Google painted themselves as the good guys for fighting piracy.
Google is reportedly about to launch their music service, once again without permission of the copyright holders they are abusing.
Those were examples of how Google interpreted "the guidelines" in modern societies.
Google doesn't wait for permission.
What are you doing right now?
Are you sitting around hoping that GoogleBot sorts everything out?
If so, grab a newspaper & pull out the "help wanted" section. You're going to need it!
If you want to win in Google's ecosystem you must behave like Google does, rather than behaving how they claim to & tell you to.
In October of 2008 Eric Schmidt announced that SEO was about to get really ugly for anyone who doesn't own a brand. He didn't word it that way though. Rather, he stated
"Brands are how you sort out the cesspool. Brand affinity is clearly hard wired. It is so fundamental to human existence that it's not going away. It must have a genetic component." - Eric Schmidt
In response to that comment (& some of Google's pro-brand algorithmic updates) I created the following video.
Google's Brand Promotion History
Ultimately Google promotes brands for the same reason they promote Wikipedia: it is (generally) safe & easy.
Here is a history of how brand promotion became part of "the algorithm"
In 2003 Google did the infamous Florida update & ever since then they have generally trended toward placing more weight on domain authority (about the only big counter point to this would be Google's recent localization push)
Google's sandbox took it one step further by making it harder for new smaller sites to break through & giving them what amounts to a purgatory period
in 2006 BMW was caught spamming (after years of increased search traffic from spamming) they got a couple day slap on the wrist from Google. Smaller webmasters who were caught doing similar were penalized for far longer periods of time.
in 2005, shortly after announcing rel=nofollow, Google stepped up a campaign promoting FUD against link buying & promoting snitching (when combined with preferential treatment toward brands, this further favored big business at the expense of smaller webmasters)
over the years Google built increasingly sophisticated algorithmic filters to detect & demote aggressive link strategies (which, when coupled with brand promotion algorithms, further made it harder for small businesses to compete online)
in April of 2007 Google bought DoubleClick, highlighting Google's aspirations to move from demand fulfillment direct marketing ads into the lucrative brand advertising market
when the Google Panda update happened Matt Cutts stated "we actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side." That algorithm allowed doorway pages & scraper sites to rank while killing off lots of smaller legitimate websites.
Sleazy Outing for Self-Promotion
In the later half of 2010 & the first few months of 2011 Google was getting beat up in the press about content farm spam (created by a combination of loose AdSense standards & Google putting too much weight on domain authority). To help deflect some of the bad press & show "who is boss" Google penalized both J.C Penny & Overstock.com for using manipulative links.
This past week the folks from Digital Due Diligence tipped of a NYT reporter for another hit piece. A lot of the top flower sites increase their ad budget around their busiest times of year, so coinciding with Mother's Day the New York Times highlighted how sites like ProFlowers, 1800Flowers, Teleflora & FTD were buying seedy links. I won't link at the NYT article because doing so would only promote more sleazy pageview journalism.
A Googler named Jake Hubert was quoted in the above mentioned article as saying the following:
"None of the links shared by The New York Times had a significant impact on our rankings, due to automated systems we have in place to assess the relevance of links. As always, we investigate spam reports and take corrective action where appropriate."
(Even Big Brands) Can't Rank Higher than #1
What is hilarious about that official Google comment is that sometimes Google has whacked websites based on perceived intent rather than results, & when I searched Google those 4 sites owned 6 first page results for that search query (along with the NYT article being listed as a 7th result (and 8th if you count the Google News result).
Google hard coded the algorithm to favor big brands (not once, but twice), promoted the big brands to the top of the search results, watches those brands violate their guidelines (in spite of said promotion) and then claimed that there is no corrective action needed for the violation since they already rank #1.
Well of course the paid links can't further improve a #1 ranking. You can't get any better than first place.
The good news for brands is that Googlers feel the sleazy outing angle is getting tired after J.C. Penny & Overstock.com & Google changed webmaster perception of their results with Panda (by making the common smaller webmaster pay for eHow's sins). Soon reporters won't justify wasting ink or bits on another sleazy SEO outing article because the pageviews won't be there.
At this point it is safe to say that Googlers don't really need to think of brands. All they have to do is search for *any* commercial keyword and click on the first result. The brand takes care of itself. :D
Would you trust the information presented in this article?
Sounds like this is all about promoting perceived authority.
Did you (or anyone you know) trust this outlet to give you a complete worldview at any point in the last couple decades & end up bankrupt, utterly decimated, or destitute because you followed its advice (on say internet stocks or the housing bubble or using excessive credit because "this time may actually be different")?
A lot of news sites are given additional distribution through services like Google news, which start them from a position of authority (because if you go to search to find something & Google promotes their news vertical right away, then sites in that news vertical will rank highly instantly & accrue backlinks from that early exposure). The education system itself is partly a propaganda tool to teach you to trust an obey authority. If the banking crisis taught us nothing else it should have taught us that many authorities are not worthy of our trust as they act in self interested ways at the expense of the whole.
Is this article written by an expert or enthusiast who knows the topic well, or is it more shallow in nature?
Mainstream media sites saw a $1 billion Dollar lift in annual ad revenue from the Panda update. Most mainstream media articles are *not* written by true subject matter experts, but rather by devout generalists who grab a couple quotes to fill out the shallow piece & make it feel more informed.
A lot of the "official" quotes are from officials who represent industry trade organizations. That means those folks support the interests of folks in that trade, even if/when that trade is working against the interest of the common man.
The problem is, you don't get to see who is a whore until *after* they already ____ed you. See for example David Lereah: "Ahhh, so he admits to being nothing more than a paid shill whose mouth was available for a price. How does that job description vary from the Trannies who hang out by the West Side Highway? In my book, not by very much. A whore is a whore is a whore."
Does the site have duplicate, overlapping, or redundant articles on the same or similar topics with slightly different keyword variations?
*Cough* Google Video vs Youtube vs Vevo.
Aren't most AP articles by their definition redundant duplication?
How are some of Google's late-to-the-party services like their ebook store or their places pages justified if we seek to minimize redundancy?
Would you be comfortable giving your credit card information to this site?
Some sites aim to sell, while others aim to tell.
If a passionate hobbyist desires to share but isn't selling something (and thus uses a quirky site design or a more personal formatting structure) should they be dinged for putting their passion ahead of getting an unneeded SSL certification & paying firms like TRUSTe, McAfee & VeriSign?
Sites which don't go out of their way to sell you something are more likely to be built on passion.
Does this article have spelling, stylistic, or factual errors?
"Correct spelling, indeed, is one of the arts that are far more esteemed by school ma'ams than by practical men, neck-deep in the heat and agony of the world." - Henry Louis Mencken
Further, the error of omission is one that is constantly made in the mainstream media, which is precisely why you have to read fringe rags like the Rolling Stone to get an honest look at how bankers are robbing the country blind. Of course you will read the same article in the mainstream media in 6 or 7 years, after the statue of limitations runs out. And they will sell it as "new" news, even though the story at that point is nearly a decade old.
Are the topics driven by genuine interests of readers of the site, or does the site generate content by attempting to guess what might rank well in search engines?
I can tell you sure as hell that the auto-generated spam stub pages on the mainstream media sites (driven by services like DayLife or Truveo) which scrape blogs like mine are not driven by passion. You can't program a bot to have "passion."
Does the article provide original content or information, original reporting, original research, or original analysis?
Further, I have had a client featured in a well read trade magazine where they wrote an entire article on the client. They were unwilling to link to the client's site (even though the client was the only source & entire purpose for the article) because they said they felt it would be too promotional. How warped is it that they will do a photo shoot at your house & make you the feature of an article, yet they are afraid to link because that might be seen as being too promotional!
Does the page provide substantial value when compared to other pages in search results?
This is actually a bit of a bait and switch styled topic. Let me explain. In an ideal world every single page would be great.
Most articles individually are failures that do not pay for themselves. It is the rare success that helps carry the failures. You do not know which is which in advance, but you hope that with some level of effort and scale you are marginally profitable out the other end.
This is how literally all forms of publishing work: online, music, movies, books, etc.
In terms of a money loser, take for instance this article. I am already rather well known, have a wide following, spent hours writing that article, and ultimately it garnered 1 comment & 0 inbound links (once you back out scraper sites, automated links, and links with nofollow on them).
Making things worse, you not only compete against others who will copy anything of yours that is successful, but if Google does decide to whack your site with a penalty then a scraper site (which Google paid with AdSense money) that steals your content will outrank you for your own work. How exactly do you provide a unique substantial value add when Google is paying others to steal & republish your work wholesale?
Things like source attribution issues, brand bias, and Google competing against publishers with scraper pages have a very real and significant impact on profit margins. A good sustainable company is generally lucky to have 20% profit margins. When Google introduced their places pages that scraped TripAdvisor Google instantly redirected 10% of TripAdvisor's search traffic.
Ultimately the above issue with content is not down to cost or effort, but if what you are doing is profitable. If it is not, then it is simply unsustainable.
And even when you are profitable, you can count on Google helping others subvert that position.
On the topic of value add, I have even seen people buying AdSense ads to redistribute 3rd party works, where the only value "add" was lowering the retail price!
How much quality control is done on content?
A lot of the high ranking and much hyped social media networks like MySpace, Friendster, Twitter & Facebook are almost exclusively spam. A couple days ago I deleted over 75% of my Facebook "friends" because I was sick of getting daily email updates about how some dirtbag wanted to promote some autowealth MLM blaster unlimited downstream product on my wall.
That is not to say that everyone I deleted did anything wrong (most of them are likely good people) but there was no opportunity cost to spamming. The spammers who automate drive everything toward the tragedy of the commons. A paywall is perhaps the single best filter for quality, but if you use a paywall expect to deal with a lot of freetard rage & expect Google to pay some folks to steal it.
Google polices the web, but anything goes in their ad programs.
You can see how ridiculous the double standard is by simply considering that Google let their counterfeiting advertisers count grow to 50,000 strong before finally axing them when the US government pressured Google. Bizarrely, Google had the audacity to position themselves as good doers who were cracking down on spammers, when in fact they were taking their own longtime business partners out to the wood shed!
Does the article describe both sides of a story?
Mainstream media sources often like to share "both sides of a story" to seem unbiased. But the truth is that media by its very nature is biased toward the interest of advertisers & away from consumers. See, for example, either Manufacturing Consent or the BGH lawsuit.
Further, some well known corporations (LIKE GOOGLE) blackball media outlets that question them in certain ways. Google would never give exclusives to SEOBook & the sites that they do give exclusives to would lose the relationship if they were as blunt as we are.
Is the site a recognized authority on its topic?
Lots of recognized authorities have conflicting funding sources - something that was well highlighted in early Google research, and has been consistently exposed (years or decades after the fact) in the medical space.
Honesty is more important than authority, but then being bland & honest is not quite as remarkable (or profitable) as putting on a coat of spin.
Is the content mass-produced by or outsourced to a large number of creators, or spread across a large network of sites, so that individual pages or sites don’t get as much attention or care?
How would Google's efforts stand up when graded against this suggestion? Why does Google have Google Video, Youtube & Vevo?
Further, most market leaders do have large networks and multiple branded sites for purposes of branding, segmentation, and double dipping in the marketplace. Remember when Bankrate (which already owned Bankrate, Nationwide Card Services, Credit Card Search Engine, Bankaholic, etc.) bought out CreditCardsGuide.com & it got temporarily penalized for the spammy links it had? Well it ranks again & of course since then they have also bought out CreditCards.com. You see this sort of behavior amongst almost any big brand: from Amazon.com to Zappos. (Oh wait, Amazon.com now owns Zappos!)
Was the article edited well, or does it appear sloppy or hastily produced?
A lot of the best content comes from people who are subject matter experts. But those people may have only mastered their subject & may be new to: writing, website design, online publishing, etc.
For a health related query, would you trust information from this site?
Let's put it this way: the media people consume is in part responsible for the current state of health in the US where there is an obesity epidemic. Further, a lot of the leading health authority sites (like WebMD) run special advert sections in their site where it looks just like content but you have to read the small print to see it is an ad.
Going one step further on this front, it is worth mentioning that a number of the large pharmaceutical corporations have repeatedly sold drugs for off label purposes & yet none of their packaging is required to highlight those ill deed they did that have literally killed millions of people.
Would you recognize this site as an authoritative source when mentioned by name?
A lot of "authoritative" sites are simply sites with large ad budgets.
Quick, tell me which company advertises a clever gecko with a British accent. Other than as a mascot (& perhaps alliteration), how relevant is that gecko (or the accent) to their business? Not at all. But they do spend nearly a billion Dollars a year on ads.
Does this article provide a complete or comprehensive description of the topic?
Would users complain when they see pages from this site?
I'll complain about something I just saw. ;)
While searching for a link for a blog post I was writing today, the #1 Google result (not voted up by social circle stuff) was a Tweet linking to a Hootsweet framed page linking to a music industry site which posts RSS feed content and linked to a BusinessInsider article that referenced the TechCrunch article I was looking for.
If we want to get rid of unneeded duplication & noise then why is Google tying their bonus system to promoting more social media noise? After Amazon.com has done a great job with Kindle why is there a need for Google's ebook marketplace? After Yelp has created a strong community review site (with real editorial expenses) why is there a need for Google Places to scrape & displace its reviews?
If you look at what actually happens in reality (rather than what folks claim to support in their "ideals") it is anarchy. The bankers stole what they could and moved on. The pharmaceutical corporations create fear-driven propaganda about the dangers of drug re-importation, all the while pushing drugs for off label purposes. Google pays people to steal your content, then tells you to suck it up & it is your fault you are not a big brand.
Anarchy is here.
The only difference is that it is dressed up in suits and fancy language, where people perceive anarchists as like ripped jeans, megadeth shirt wearing, pyro's.
You see, tricking people is bad. Unless you are Google. In which case you have to hit the quarterly numbers.
Everyone else needs to read Google platitudes, create deep content, and pray to turn the corner before bankruptcy hits.
Matt Cutts stated that you should make your products like Apple products by packaging them nicely.
For illustrative purposes:
It was easy for Google to speak from a moral high ground when their growth was above 50% a year, but now that growth has slowed over the past couple years they have been willing to do things they wouldn't have. In November of 2009 when I saw the following I knew the writing was on the wall.
When Google Instant launched, we got to test Google's 50% content theory. And they hit the numbers perfectly. A full 50% of web users could see 2 organic listings above the fold when instant was extended (the other half of folks could only see one or none).
As if the massive Youtube promotion & the magically shrinking search results for everyone else were not bad enough, with Panda they suck at determining the original content source.
This site you are reading wasn't hit by Panda, which makes us lucky, as it allows us to rank as high as #3 for our own content (while Google pays dozens of other webmasters to snag it wholesale and wrap it in AdSense).
We got lucky though. If we had been hit by Panda (like 10,000's of other webmasters) we probably wouldn't even rank on the first page of the search results for our own content.
When Google screws up source attribution they are working counter to open culture, because they are having you bear 100% of the cost of content production, and then they are immediately paying someone else for your work. Do that long enough and the quality content disappears & we get a web full of eHow-like sites.
And yet Google tells us the secret recipe (which may or may not work at some unknown time) is to pour more money into content development.
The solution to this problem is more deep content. Keep feeding Google (and their AdSense scraper partners) and hope that after you pour $50,000 into your site that some small fraction of it ends up back in your bank account (while the larger share winds up in Google's and their AdSense partners).
As bad as all that is, I recently got selected as a lucky beta user for the next version of Google's search results. Notice the horizontal spacing that drives down the organic search results. After the top AdWords listings the organic listings start off 88 pixels lower on the screen.
I have a huge monitor. Less than 10% of people have a monitor as large as mine. Before this new search result I saw 8 organic search results above the fold on my large monitor. Now it is down to 5 (and that is with no Google video ad, no Google vertical comparison ad like the above credit card one, no browser toolbars, no browser status bar, and only 1 of the advertisers having ad sitelinks).
So how does Google score now on their ad to content ratio?
When Google's new search results roll out, there are some keywords where less than 1 in 3 searchers will be able to see a single organic listing above the fold! And lest you think that spacing is about improving user experience, notice how wide the spacing in the left column is, and how narrow the right rail AdWords spacing is. This is all about juicing revenues & hitting the number.
Which leads me to the Google Panda loophole I mentioned in the headline. It is an easy (but painful) one-step process.
All Google's propaganda about the horrors of paid inclusion look absurd when compared against the search result with 0 organic listings above the fold for half of desktop computer users.
The only "exploit" here is how Google is paying people to steal other's content, then ranking the stolen stuff above the original source.
With Google making the life of the SEO harder and harder, it pays to add as many marketing strings to our bows as we can. In this article, we'll look at a way to brand and position using stories. Hopefully, if we have a good story, and tell it well, people are more likely to remember us, and more likely to pass the story on.
Are We Special?
Most people think their site is special. But, by definition, few sites in a given niche can be special.
If we target a keyword term, that many other sites are targeting, we'll probably write a similar keyword-loaded page, including the same synonyms, derived from the same keyword tools, using the same headings in bold, in the hope of appearing in the top ten list of pages - which are just like the others.
We may distinguish ourselves by managing to rank in the top three, but, as we know, there are no guarantees we'll maintain this advantage.
We Need Something Else
If SEO is our only strategy, then this will only work if few other people are using SEO. How many niches worth fighting for are like that these days?
Generally speaking, the more mature the niche, the more you need something besides SEO. You need to make as much effort to stand out as possible, otherwise people will likely overlook and forget you. There are too many other sites and options.
Let's look at a differentiation strategy based on stories.
Why Use Stories?
Stories are universal.
The human race has been using stories for thousands of years. We use stories because they are informative, memorable, and easily spread to others. Isn't that what we want our sites to be, too?
Every news story is a tragedy. Every religion is a story of redemption. Politicians tell stories, some of which are true! The alternative would be to give people a string of disconnected data and facts. Such data and facts may be 100% true, but they are seldom memorable or easily repeatable. Telling a compelling story is one good way to contextualize information, and make it more meaningful.
A story isn't just words on a page, saying how great a company is, and what products they have, and if you want them, you should "click here". That's surface. Think of "story" as a sub-text, the underlying, perhaps unspecified tale of who you are, what you're doing, and how you can help people solve their problems. This is a form of positioning, and branding, but I find it's helpful to reduce those high concepts down into a simple narrative. It helps bring a lot of different, and sometimes complicated, marketing aspects together.
Everyone can tell a story, especially about themselves.
Every business has a story.
The Amazon is the largest river in the world, which is an appropriate name for a site which aimed to be the largest retailer on the planet. Amazon is huge. Amazon is huge because they took the shopping experience, made it easier, and people loved it. With one click, a customer could order a book, or a DVD, and many other products and have it sent to them. Amazon faced some huge challenges. Just how do you store and ship a vast array of products and still make money? Amazon do massive volume, and use unique, sophisticated tracking and packing systems to overcome these challenges. Amazon's cloud computing service alone has revenues in excess of $500m.
Amazon's story is mostly about "being big". All very well for Amazon, of course, but what about the little guy showing people how to build stuff? There's a story in that, too.
Tim Carter founded "Ask The Builder.com". Tim provides tips for DIY, answers building questions, and provides product and tool reviews. Rather than the home DIY enthusiast going out and buying manuals, or hiring an expensive builder, Tim provides his information for free, and his video's provide depth that printed books do not. Tim clearly cares about building, and the home DIY enthusiast. Tim's gone a step further, and told his life story.
Try boiling your site down into such a story. Once you have a story, you can then flesh out narratives that flow through everything you do, from your graphic design, to your copy, to your approach to customer services.
For example, Tim's story is a "small, personal" story. It is fitting that he doesn't have a glossy, corporate theme, as this would grate against the narrative. Rather, the site is a bit raggedy and amateurish, in a good way. He is providing one-to-one personal help, so it fits that he talks directly to camera. It fits that, unlike Amazon, you know who is behind the site. It fit's the the About Page is a personal history. It's approachable. It's all part of the "small, personal" story. It helps make the site more convincing, and hopefully more memorable, if common themes are repeated.
A story helps achieve focus, clarity and distinction.
How To Construct A Story
If you're having problems getting started, here's a work-plan.
1. Describe Your Brand
What do you do? Make it short and sweet i.e. "Provide advice to home DIY enthusiasts". "Sell books online".
2. Where Did You Come From, And Where Are You Now
How did you start? Why did you start? What did you do before you started? What position are you in now?
3. What Challenges Do You Face?
What problem do you solve? What are challenges have you overcome? It helps if these are the same challenges and problems your customers face.
4. Personify/Quantify These Challenges
Did you overcome people? Organizations? Time? Money? Lack of knowledge?
5. Who Is Your Target Market?
Who, exactly, are you trying to help? Where do they live? What is their time of life? What challenges do they face?
6. What Does Your Target Market Care About?
Security? Being first? Individual care? Low prices? Value?
7. Why Should They Buy From You?
What do you offer that other sites do not?
8. What is your end goal?
How do you know you're completed what you set out to do? What is the measure of victory?
Answer these questions, and it becomes easy to make decisions about design, positioning, branding, and marketing.
Hopefully it helps make your site more memorable, too.
Before Google's Panda update an effective SEO strategy was to "make a page for everything." If you are Wikipedia that strategy may still work, but for most websites that approach is a high risk & low return approach. Clustering like keywords together and using that to help set up your site's information architecture is a lower risk and higher return strategy.
Given Google's new approach to search (where dead weight can harm your good pages) organization is more important than ever.
Let's say you have a big list of keywords which is not well organized & you want a quick and dirty way to organize it. Here are a dozen different tips and tools to help you organize your keywords.
Ad Group Filter
We created an ad group organizer tool which aims to create a footprint for keywords by putting muti-word keywords in alphabetical order & stemming the keywords. The output is TSV, so you can copy it and paste it into a spreadsheet for further analysis. It also allows you to use stop words to filter the list. This can not only be used for organizing keywords for paid search, but also to help organize them for your SEO efforts down to a per-page level.
RKG Duck is a Perl clipboard extension which was the inspiration for our above tool. This tool works well in spreadsheets, but it takes some level of programming sophistication to get working.
Spyfu's Keyword Groupie allows you to look up a competing site's keywords & see them organized by root word. In addition to listing keywords by root word, they also give you the option of viewing the top 100, 500, or 1000 keywords for a site.
The big benefit with SpyFu is that you can view data on a competing site & use their performance as a bit of a filter for you, but the downside is that at times it can be a bit slower than the above tools. That is to be expected though, because it is searching through a database of records related to sites, rather than just applying a filter or returning results.
They allow you to browse keywords to drill down in areas of interest. Another nice feature is that they show you keywords they feel you are missing out on by putting them in bold, so this tool is great both for looking up competing sites & for looking up your own site to see what you missed.
Google AdWords Desktop
There are a couple different ways to use the Google AdWords editor to group keywords. Here is a semi-automated way (where you still have a bit of human editorial in the process to manually filter to find themes & create groups)
You could do similar to the above with Microsoft Excel's table filters. And if you are combining multiple data sources / tables in Excel this AbleBits merge tool is handy.
Google also offers an automated option inside Google AdWords editor to help organize keywords into fairly tight groups.
First create a new ad campaign (and its settings can be a bit arbitrary off the start as you are mainly using this to help automate data sorting). You only need to create 1 ad group in that ad campaign and then bulk upload a group of keywords into it.
Next use the keyword grouper tool, as shown here.
In the keyword grouper you can use the "generate common terms" option to automatically create keyword groupings. Note that in the right box you can add stop words & other words that you don't want them to cluster keyword groupings around.
Google then spits out a result set you can use, with the keywords clustered into tight groups. Note that sometimes they footprint geo-local keywords similarly even if they are for multiple different areas, but outside of that it is a pretty nice tool considering the amount of work it does in what amounts to a 2-minute process.
Put your site in rank checker and see how you rank for your target keyword list. If your site is nowhere to be found for a keyword then that may indicate a need to create more content pages around those new topics. If your pages already rank well then see how well they are optimized. A small amount of link building & on-page SEO can go a long way if you were already ranking for a keyword that you were not intentionally targeting.
If your site is brand new & has no authority (or you are researching a new market) you can search for the rankings of a popular website in your niche and see where they rank. Export the ranking data and you can sort the Excel spreadsheet by URL, which should help you cluster your keywords around a similar strategy that top ranked websites are using.
You can pull data on competing sites from competitive research tools like SEM Rush, Compete.com, Keyword Spy, SpyFu, and Alexa to help get an overview of some of the top keywords competing sites are ranking for.
Crawl Their Site
Do you have a well optimized competitor? You can crawl their site using tools like Xenu Link Sleuth or Screaming Frog & then export the data to a spreadsheet, using that as a baseline to start your information architecture strategy from. Xenu is free & Screaming Frog's SEO Spider is free for up to a 500 URL site.
Keyword List Cleaner
If you have a big and dirty keyword list where some of the words have multiple meanings you can try to filter the list down by using negative keywords on a keyword list cleaner.
Google Related Searches
Whenever you search on Google not only does their search box recommend tightly related keywords (which are good for late state optimization of on-page content), but in the left column they have a link to "related searches" which organizes related keywords. Within these lists of keywords you can click further into to drill down deeper.
Some folks scrape that data in bulk as well, but if you do that then you are back to having to organize it again. ;)
Google AdWords Keyword Tool
Many paid keyword tools like Wordtracker have advanced filtering & organization options, but I mainly wanted to show free options in the post. Google AdWords keyword tool has multiple helpful ways to organize data.
I tried to highlight key areas & options in the above image, but it sorta feels like I highlighted everything, as there are so many amazing options baked into it. You can get keyword data based on selecting a category, a site, or entering a root keyword. They allow further filtering by match type, tight or loose keyword groupings, location, and so on. Sometimes the data can be a bit inaccurate, but nonetheless it is a great starting point as it really is an amazing feature-rich tool.
Microsoft adCenter Plug-in for Excel
Earlier I mentioned how Excel tables have a bunch of handy filters in them. Taking that to the next level, try the adCenter Excel plug in (review here), offering you quick access to Microsoft's keyword data by root keyword, general topical category, ad campaign association, and so on.
Your Web Analytics
There are at least 4 amazing benefits to using your site's keyword data
This is the stuff that actually applies directly to your site. Rather than being some sort of academic exercise or a bunch of "what if" sort of stuff where there is a big margin for error, you have the data related to the actual business impact of these keywords.
Since your site is already ranking for these keywords you already have momentum behind them. Pushing a #5 to #2 is typically far easier than going from nowhere to #5. And it is not only easier, but it is also more profitable.
This data is organized by page already, and (since you know your site) you should be able to quickly tell if pages that are ranking should be further optimized for a keyword or if the user intent for that keyword is different and it deserves a different page.
If you have been tracking your site for an extended period of time you should know not only what pages are ranking, but also why. Sure Google aims to make this a bit more complex, but that is precisely why looking at data on your own site is so helpful: you already know so much about it.
The same types of benefits can be had by using a (phrase matched, broad matched, or modified broad match / with negative keywords) AdWords ad campaign to do keyword research. You are not only testing the search volume of the keywords, but also how your site performs for them.
WordTracker's Strategizer (review here) is a premium SEO-oriented extension of web analytics data, which helps make the data relationships easier to visualize. Concentrate is another paid application built on data wrangling & visualization front.
Free keyword cloud tools like Wordle & tools like Many Eyes can also be valuable for helping you see word relationships for a page and convey concepts to management. You can probably guess which page the following analytics-driven word cloud is for without even visiting it. ;)
Insurance is a popular, profitable area for some SEO's. Trying to find reputable insurance for an SEO business is not so popular because many insurance agents do not have the experience to make the distinction between what a web design shop does versus what an SEO or PPC business does.
Prior to entering this business I was an insurance agent and before that I was an underwriter and I still have my agent license (hey, you never know!!). There are policies out there which SEO's should consider purchasing as well as any web design or development shop.
There are a few different policies you might want to consider in this industry:
Professional Liability (Errors and Omissions)
Workers Compensation (if you have employees)
Short-Term and Long-Term Disability
As a business owner, you will have or not have the following conditions:
office space where you conduct business with clients and vendors
Workers Comp and STD/LTD
Workers Compensations and STD/LTD are fairly general insurance policies with respect to the policies not really being specific to the SEO business. Here in the US, Workers Comp is administered on the state level. Workers Comp is required in certain situations, depending on your state, so it is wise to check with your attorney and insurance agency regarding what is appropriate for you.
While you only have to worry about Workers Comp if you have employees (actual employees hired and placed on payroll not contracted labor or freelance arrangements) you should consider Short Term and/or Long Term Disability insurance even if you're a solo SEO.
Short Term and Long Term Disability
If you are coming from corporate America you likely had these policies under a group plan (which is why it's so cheap). Essentially, it breaks down as follows:
Coverage responds if you are injured and unable to work (this doesn't cover sick time and generally excludes maternity coverage)
Short Term Disability will cover you for a certain period of time (usually under 90 days) at 100% of your pre-tax (sometimes you can choose post-tax) income.
Long Term Disability kicks in either after Short Term has expired or if you decided not to purchase Short Term at all.
Long Term will typically cover you at 60% or so of pre-tax (or post-tax if you are given the choice) for an extended period of time.
Wages are usually determined based on your prior year's tax return in conjunction with a current Profit/Loss statement (another reason why you should report all your income!).
Sometimes it makes more sense to get a quote on LTD (as it's cheaper) and just build up a reserve of your own to cover what Short Term Disability would have covered (rather than spending money on premiums and losing it if you never file a claim).
If you advance a claim for either, there usually is a waiting period of a few weeks to a few months while a case manager is assigned to investigate the claim.
Be prepared to get put under the microscope much more than you would if you were part of a large group plan (if you work for a large company as an example) as you are no longer part of a protected herd, but this is where using an independent agent can be help in fending off the overzealous claims adjuster who might see you as an easy case :)
Different policies have different exclusions so it's wise to discuss all your extracurricular activities with your agent as things like sky-diving are usually not covered causes of injury.
This is one of the most common forms of business insurance. The meat of what this type of policy provides is:
Bodily Injury and Property Damage
Defense Costs while defending a suit
Most of this coverage is applicable when/if some of the following conditions occur:
Client is injured on your premises
Damages to property you are renting to other businesses
Injuries sustained by others on your defined premises due to the activities and operations of your business
Combining GL with Property Insurance (BOP Policies)
Many times, a GL policy is combined with Property Insurance to make what is called a BOP or business owners policy (BOP package).
Property Insurance is fairly standard and covers things like:
Equipment (probably computers for most of us)
Records and Documents
Other Real and Personal Property
Lost income due to a covered loss
A BOP is really geared towards business which have an office building, equipment in that office, meet clients on the premises, or rent out space to others.
Neither of these, or the BOP package, cover Workers Comp. A BOP is a good solution for a shop which has a physical location, clients on site, and has inventory/equipment on premises.
Professional Liability & Specialized Insurance
These types of policies is where a chunk of the specialized coverage for an SEO would come from. A BOP policy is meant to cover "products" with respect to liability so as an SEO, web designer, or web developer you'll need more specialized coverage which covers things like:
Malicious Code (say your Wordpress site gets hacked and distributes malware or keyloggers)
Loss of business income (say your host goes down and your client's e-commerce site goes offline, or it goes offline due to an error on your end)
Depending on your level of involvement with servers, software, and application development you may want to scale up and get a more specialized policy. Lots of larger insurers sell specialized, broad polices under the name of Technology Insurance or Information Technology Insurance.
Over the years these policies have developed to cover more and more specialized areas of tech insurance. In the beginning they were mostly geared towards straight IT companies but now cover all sorts of tech groups like:
Our industry is no different than any other, lots of snakes. If you are engaging in some kind of off the wall activity you really need to tell the agent. Tell them exactly what you do, if you are doing things that your state or other states consider illegal (fake reviews for instance) then don't expect your policy to respond to such things.
It's no different than getting a homeowners policy while saying you don't have a Pit Bull, even though you have a Pit Bull (which are excluded by all standard insurers), then expecting coverage when your Pit Bull bites the neighbor.
While the agent may not understand the nuances of the business, you are typically good to go if you take the time to fill out the application completely and accurately.
Why Use a Local, Independent Agent?
Most folks you get in the call centers of GEICO or Progressive are salaried or hourly employees, they don't live in your community, and they really don't care to get you the best deal (they can only give your theirs).
Having a local agent gives you access to more markets, someone close by to help you fight any injustices the carrier may try to perpetrate on you, someone who is making a living selling policies in a local market (less likely to burn you as they care about their reputation), and someone who can help insure all your personal and business needs.
A call center rep, if they are on commission, generally wants to churn and burn through the calls and probably won't meet with you face to face to go over anything and answer all your questions (or get answers). Plus, most local agencies need help with web marketing so it could be an easy in for you!
What Do You Need?
Scenario 1 (Work from home, no employees)
Get a business endorsement on your personal homeowners policies and schedule your equipment if you need to (might not need to if you have warranties in place already). This will also make your home office deduction look more official to the IRS.
Get a Professional Liability (E&O) policy and look into a specialized Technology policy depending on your business model. Consider STD/LTD insurance for lost wages if you can't work.
Scenario 2 (Home office, external office, no employees)
Same as above but add in a BOP (General Liability and Property Package) which is probably required if you are renting office space anyway.
You should check with your attorney about how you are defining employees and if that means they are actual employees versus free-lancers or contractors.
You would want to look at a BOP, Professional Liability and/or specialized Tech Insurance, and Workers Comp if required as well as employee benefit/insurance packages for things like STD/LTD.
Insurance is Boring
Yes it's boring but it's worthwhile for most of us from a cost/benefit standpoint (or legal liability standpoint). Solid, legal contracts reviewed by attorney's are also HIGHLY recommended.
Make time to sit with some local agents to really go over your business and your business activities. It's a really soft market right now for business insurance, especially small business, so agents are going to be more than happy to sit with you and go over what they have to offer.
Why Contracts are Important
There are many contracts available online, for free and for a fee. These contracts, even ones from places like LegalZoom have not been reviewed for your specific business by attorneys in your state.
A contract is no good if it's not enforceable. You can probably expect a fee for a good attorney to review your contracts, and make any necessary changes, to be in the high hundreds of dollars or low four figures.
There is a cost certainty to the cost for an attorney to give you the green light on a set of contracts (though, for really big deals you might still be wise to get a contract specific to that deal) but there is no cost certainty to the legal liability you could face if your contracts are essentially worthless in a court of law.
Another thing you'll want to watch out for is a client who tries to give you unlimited downside from a liability standpoint (in the contract) but severely limits the upside to your fees. You might be willing to take on the risk of downside so long as you are getting a decent %, or a few % points, of the upside on the deal. This is another case where an attorney reviewing the contracts can be well worth the cost.
Choosing the Right Business Entity
Your insurance policy is mutually exclusive from your business entity. If you are a sole-proprietor your personal assets are at risk even though your insurance policy covers defense costs. Choosing the right entity is another way to insulate yourself from liability.
In most states a single member LLC up to a full-blown corporation (and everything in between like a multi-member LLC, S-corp, and so on) will insulate your personal assets (home, savings, future personal earnings) from legal liability, whereas a sole proprietorship will leave your personal assets exposed.
The combination of a good insurance policy and the right business entity will cover defense costs (on a covered event) and protect your personal assets. Choosing the wrong set up can be financially disastrous.
The policy will cover defense costs if a claim is covered but if it is a frivolous lawsuit, or something personal, you might have to defend yourself. This is where having the right entity is key because your personal assets are not at risk even though you are probably going to incur your own legal costs.
The best way to protect yourself is to insure and to pick either an LLC or a corporation of some kind. Choosing neither, or one but not the other, can leave you and your business significantly exposed to liability.
The #1 goal for any organization is self-preservation. When people feel things are fairly just & they are just getting by they are fine with squeezing out more efficiency in what they do and figuring out ways to pay the bills. But when people feel the table is tilted at some point they stop caring and do whatever it takes.
Ex Post Facto
Some longtime AdWords advertisers have recently been punished for affiliate ads they ran 8 years ago where some of the sites they promoted at some point fell out of Google's graces through an ad system which never allows you to delete your history & offers ex post facto regulations that turn a regular advertiser arbitrarily into a spammer.
In 3 weeks it will have been 3 months since Google first launched Panda. Outside of bloggers with 50,000 RSS subscribers few (if any) reports of recovery from Panda have been seen. Some of the theories floating around what caused Panda attempt to tie it to AdSense & many of Google's AdSense case studies are now highlighting best practices to follow if you want to be just like the sites Google torched.
As if that wasn't conflicting enough, some of the webmasters that were torched by Panda received automated messages that they were missing out on revenues by not using the maximum allotted number of ad units. After the huge fall off from Panda, Google has been pushing AdSense so hard that many webmasters have been receiving unsolicited emails from Google suggesting they sign up for AdSense.
I won't run AdSense on our main sections of this site because it would be tacky and destroy perceived credibility (having a "submit your site to 2000 search engines for $29" ad next to the content doesn't inspire trust on an SEO site). I could create a content farm answers section of the site that mirrors Ask's strategy, but with a higher level of quality. I won't though, because it would be viewed as spam because I am me. Once again, SEOs should be held to a higher standard than search engines. ;)
That Which You Consume, Consumes You
Where this rubs wrong is not only the overt brand push, but also that some of Google's pushes at expansion down the search funnel have looked a lot like the spam they claim to fight.
In the Wall Street Journal there was an article about the Panda update highlighting that many small businesses were laying off their employees. The same article highlighted numerous cost extensive desperate marketing measures the firms were taking which may or may not work. Google didn't disclose much in the article other than:
The Google spokesman says the company doesn't disclose details about changes it makes to its algorithms because doing so "would give bad actors a way to game our systems."
Nobody likes bad actors, but most of the webmasters that were hit were not bad actors. Rather, most of them were naive & simply followed the Google guidelines thinking that was in their best interests and perhaps would allow them to stay competitive. Unfortunately, it wasn't.
If you adhere to guidelines, get beat down, are not told why, and are told that generally sites need to "improve their quality" that can be a pretty infuriating message. The presumption that your stuff isn't good enough when 3rd grade rewrites of your content now outrank you is both smug and obnoxious. What is worse about the update though now is that many scraper websites are outranking the original content sources, so the message is that your content is plenty good enough, but it is just not good enough when it is on your site. A large portion of those scraper sites are monetized via Google AdSense & would not even exist if it were not for AdSense.
So Google whacks your site, tells you to clean up your act (& increase your operating costs while decreasing your margins), lumps you in the bad actors group, offers no information about when the pain will (or even could) end, pays someone to steal your content, then ranks that stolen copy of your content above you in the search results.
Make Your Move
If a person has the pleasure to experience the above it doesn't take much critical thinking skills to develop a different perspective on search.
Ultimately this is going to lead to a "why not" approach to search for many folks in the search space.
If Google already dinged your website why wouldn't you remove AdSense & replace it with competing ad programs? Why not test those affiliate programs you have been meaning to test? If you have to rework your content anyway, why not move past AdSense/webmaster welfare?
If your AdWords budget was marginally profitable & you were buying ads to compliment your organic exposure, why wouldn't you stop buying ads with Google & test running ads on other websites? Google is fine funding an affiliate network that uses direct links, so why not use clean links on your ad buys? If you like run it through a self-hosted affiliate program so that you are just like Google.
If your site is already whacked why wouldn't you buy links to help boost its ranking back?
If your site earns nothing from search, why wouldn't you sell links if you have to do whatever it takes to make costs?
If your site gets penalized & someone copying your content & wrapping it in AdSense outranks you why wouldn't you create new mirror sites? Why wouldn't you create scraper websites to pollute Google with?
If rankings are unpredictable & one site is no longer enough, why wouldn't you create backup sites & projects of various levels of quality & effort? At this point diversity simply serves as a needed form of insurance.
If while running these purely scientific experiments you accidentally run into something that works really well that shouldn't, why not scale it to the moon?
I am not convinced that the search results are any cleaner today than they were a few months ago. However I am fairly certain things will soon head south. I am not advocating going out of your way to be extra spammy, but am just highlighting the cost-benefit analysis which is going through the heads of thousands of webmasters who Google just torched.
Google is betting that anonymous strangers will behave more kindly than Google has, but when an animal is backed into a corner it often acts in unpredictable (and even uncontrollable) ways.
The big problem for Google is this: "when innocence itself, is brought to the bar and condemned, especially to die, the subject will exclaim, it is immaterial to me whether I behave well or ill, for virtue itself is no security." - John Adams