A sweet new competitive research tool by the name SEMRush has hit the market. It can be seen as a deeper extension of the SEO Digger project (adding PPC data and tracking AdWords keywords), and a competitor to services like Compete.com and SpyFu (which recently launched SpyFu Kombat).
SEM Rush vs Compete.com
The big value add that SEM Rush has over a tool like Compete.com is that SEM Rush adds cost per click estimates (scraped from Google's Traffic Estimator tool) and estimated traffic volumes (from the Google AdWords keyword tool) near each keyword. Thus, rather than showing the traffic distribution to each site, this tool can list keyword value distribution for the sites (keyword value * estimated traffic).
Using these estimates does not provide results that are as accurate as Compete.com's data licensing strategy, but if you own a site and know what it earns, you can set up a ratio to normalize the differences (at least to some extent, within the same vertical, for sites of similar size, using a similar business model).
One of our sites that earns about $5,000 a month shows a Google traffic value of close to $20,000 a month.
5,000/20,000 = 1/4 = 0.25
A similar site in the same vertical shows $10,000
$10,000 * 0.25 = $2,500
Disclaimers With Normalizing Data
It is hard to monetize traffic as well as Google does, so in virtually every competitive market your profit per visitor (after expenses) will generally be less than Google. Some reason why..
In some markets people are losing money to buy marketshare, while in other markets people may overbid just to block out competition.
Some merchants simply have fatter profit margins and can afford to outbid affiliates.
It is hard to integrate advertising in your site anywhere near as aggressively as Google does while still creating a site that will be able to gather enough links (and other signals of quality) to take a #1 organic ranking in competitive markets...so by default there will typically be some amount of slippage.
A site that offers editorial content wrapped in light ads will not convert eyeballs into cash anywhere near as well as a lead generation oriented affiliate site would.
SEM Rush Features
Keyword Values & Volumes
As mentioned above, this data is scraped from the Google Traffic Estimator and the Google Keyword Tool.
Top Search Traffic Domains
A list of the top 100 domain names that are estimated to be the highest value downstream traffic sources from Google.
You could get a similar list from Compete.com's Referral Analytics by running a downstream report on Google.com, although I think that might also include traffic from some of Google's non-search properties like Reader.
Here is a list of sites that rank for many of the same keywords that SEO Book ranks for
Here is a list of a few words where Seo Book and SEOmoz compete in the rankings
Compare AdWords to Organic Search
These are sites that rank for keywords that SEO Book is buying through AdWords
And these are sites that buy AdWords ads for keywords that this site ranks for
Once Upon a Time...
I was going to create a tool similar to this one about a year ago, until I hired a programmer that was EPIC FAIL. The guy who managed that program is no longer selling programming services - and that makes the world a better place.
I actually had 3 attempts at such a tool. I bought a GREAT domain name, spec'd out the project, then planned on doing it...
investor backed, who decided to back out
self funded, but I hired... 1.) a programmer who mid-project decided he needed to make double what I make working part-time, then 2.) the worst programmers ever.
combination of heavily self funded with the guidance of a bad ass VC, but I backed out due to a need to focus on this site
I spent most of this year focusing on trying to build our community and raise our editorial quality (both goals are going well, but require significant maintenance). We have had 4 strong hires in a row, so it seems like our luck has changed on that front. Recently I started working with a programmer who really clicks with me, often taking my ideas and making them way better than I intended.
If these guys had not made this tool I was going to try to take another run at something like this early next year...which brings up a good point that a friend (and wicked intelligent open source programmer) named François Planque told me. He said all he had to do was think up a good idea but not do it, and within 6 to 12 months if he had not done it, someone else would have already launched it.
Entry cost is so low that a lot of great tools are going to get made in short order, but it is hard to win by sitting on a good idea. ;)
Wordtracker released a new keyword tool based around keyword questions. The information is quick and easy to export. Ken McGaffin said, “This is a fun tool that is a great source of inspiration for web content writers. You need never be short of creative ideas again." And it is a cool idea - good job Wordtracker!
They also did a comparison between their link counts and those found by Yahoo! Site Explorer and LinkScape. They claim to have more links in their database than Yahoo! is showing, but I have to wonder how they could do that economically, if they are counting more duplicates, and why they haven't bought a site design that reflects how much they must be spending on data.
A few years back search engines were in an ego based contest about who has the biggest index, and I find it a bit ironic that a couple SEO companies will likely be engaged in such a data war...but the marketplace competition should be good for all SEOs.
So my timing on that last post was a bit off, but I still think the general thesis is valid. But now that there has been so much negative feedback I figure it is my job to play devil's advocate and highlight reasons why most SEOs do not need to be too worried about Linkscape.
Unique Linking Domains
One of the coolest features of this tool is knowing the number of unique linking domains pointing links at a specific site, but that feature is for paying members only.
A competing tool by the name of Majestic SEO allows you to see that data as part of their free overview. Click on the image below for an example.
If your competitor has high authority links then you need more than just quantity to compete, but if most of their backlinks are garbage then this is a good stat to have, along with many other stats you can get from tools like SEO for Firefox.
Not that I advocate spam reporting (as the official guidelines have departed from reality so much that almost everyone that ranks is spamming and/or spammed in the past to get to their current market position), but for professional SEOs that own dozens of sites and like doing spam reports to Google this might be a good tool for outing competitors, since it makes it easy to find some noscript links, links from off the page, inbound 301 redirects, but the average webmaster probably does not need to worry about that.
A Bit Top Heavy
One of the biggest limitations in Linkscape is that you can only go 500 results deep unless you want to buy a custom report. They allow you to see various lenses of 500 at a time through search features and filters, but a big recommendation I can make on this front is for them to allow you to see all that data, even if it requires exporting data to CSV...they already spent the money to collect the data, so if you're a customer they may as well give it to you...it helps nobody if nobody sees it.
Majestic SEO appears to have a similar sized database as Linkscape, and they allow you to do a full data export for your own domain free of charge. Other domains they charge a scaling price for depending on the number of links to the domain.
More Cool Features?
Nick Gerner promised more features in the next version of Linkscape, but unless they start buying usage data and become more like Compete.com I am not sure if it will be a game changer. On to explaining why...
Personally, I'm not too worried. You want to compete with me and get links in places where I'm listed? We get listed in places where editorial rules. So just knowing where we're at doesn't get you in the door -- you have to be good enough to walk in. And if you are good enough, well, good I guess.
The highest quality links typically tend to be editorial in nature, with many of those being driven by social relationships. No matter how much one decides to analyze link patterns, they can't re-create most of the link relationships if they don't already have the content quality, market exposure, and awareness. And if you copy someone's idea after they already did it you need to greatly improve upon it to get credit for it.
2. Tons of Alternative Data Sources
Common link analysis questions...
How do I Get a Basic Competitive Overview of the Search Results?
Search Google with SEO for Firefox turned on. Make sure you are pulling data in the automatic mode while searching.
I Want to do Anchor Text Analysis. How do I Analyze Links?
Some options include...
SEO Link Analysis - a free Firefox extension that adds anchor text to Google Webmaster Central and Yahoo! Site Explorer.
Backlink Analyzer - a free desktop based tool I had created a few years ago that pulls data from the Yahoo! API. Make sure to watch the video on the download page before using it.
I Want to Find New Links to Competing Sites
If you want to find what someone's best ideas are all you have to do is subscribe to the Google Blogsearch feed for links to their site, like so. That should list many of the people who are talking about this site.
This is the same as competing sites, but you can also use your web analytics and server logs to dig up additional information. You can also look inside Google Webmaster Central to download backlink reports.
I Want to Find The Most Authoritative Links Pointing at a Site
Yahoo! Site Explorer generally orders backlinks roughly in terms of authority, with some of the most authoritative backlinks showing up at the top of their results.
What Sites Drive the Most Traffic to My Competitors?
The best way I have found to get this data is from Compete.com Referral Analytics, though it requires a $500 a month subscription...which is a nice chunk of change, unless you are already doing quite well!
Do I Have Any Broken Links?
Xenu Link Sleuth will crawl your site and help you find broken links, showing you which pages the broken links were on.
Each search engine has its own crawling priorities and own web graph. Google has probably spent hundreds of millions of dollars building and refining their crawling sequence. No two crawls are the same.
The irrelevant links at the bottom of a page, which will not be as valuable for a user, don’t add to the quality of the user experience, so we don’t account for those in our ranking. All of those links might still be useful for crawl discovery, but they won’t support the ranking.
5. Microsoft May be Looking to Heavily Incorporate Usage Data
Microsoft did research on BrowseRank, which aims to use actual usage data to augment (or perhaps replace) their link graph. Be default, Internet Explorer 8 sends usage data to Microsoft...when you know what 80% of web users are doing you do not need to rely on a random walk.
Think of having access to the majority of the web's usage data like this:
If Google's algorithms are more relevant than Microsoft, then putting weight on usage data allows Microsoft to quickly catch up by weighting whatever Google is weighting
Microsoft could theoretically be better than Google at filtering out paid links, as most paid links in a sidebar or footer do not send much traffic...and thus could easily be weighted less than links in content - though with Google owning so many products they could improve significantly on this front as well, if they decided to use their AdSense data, analytics data, Chrome browser data, Feedburner data, and toolbar data.
Beyond those editors there are many search engineers inside the webspam team offering a variety of techniques to throw off SEOs, including
stripping all PageRank from a site and killing all its rankings
stripping some portion of a site's PageRank and ranking abilities
stripping PageRank from the toolbar but still allowing sites to rank
showing full PageRank in the toolbar, but killing the ability of a link to pass PageRank
Without working inside of Google and/or buying and testing lots of links across a wide array of sites and verticals it would be hard to know if any particular site passes PageRank, and how much it might pass. For instance, a link from Text-Link-Ads.com's website is one of my highest MozRank links, but I doubt Google places much weight on that link since Google does not let Text Link Ads rank for their own brand.
7. Search Engine Editorial Policies are Selective, & Constantly Changing
According to Udi Manber, Google did 450 search algorithm updates last year. Even if you could somehow catch up with all the editorial stuff search engines were doing to manipulate their version of the link based web graph, you would have a hard time of keeping up with it - let alone accounting for the hoards of usage data the search engines have.
The status of a link (and its ability to pass PageRank) may arbitrarily change based on media exposure. In the past many websites were hijacked by 302 affiliate links (this even happened to Google's site, and this is still happening today to corporate sites as big as Snapnames).
Shockingly, when asked point blank if affiliate programs that employed juice-passing links (those not using nofollow) were against guidelines or if they would be discounted, the engineers all agreed with the position taken by Sean Suchter of Yahoo!. He said, in no uncertain terms, that if affiliate links came from valuable, relevant, trust-worthy sources - bloggers endorsing a product, affiliates of high quality, etc. - they would be counted in link algorithms. Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value.
A few years ago I set up my affiliate program to use 301 redirects to prevent hijacking, and get any link benefits I could. But right after I changed by business model to a membership site my affiliate program was featured/outed in this interview, and it no longer passes PageRank.
Watch the above video and see how at 2 minutes and 15 seconds in my site was put up for review to any Google engineer that happened to watch it.
The same set of links, to the same site, using the same format, under similar circumstances...
counts for most major corporations (and is allegedly an approved and legitimate strategy)
counted for this site for years
stopped counting around the time they were outed by a popular SEO blogger
8. Temporal Algorithms + Domains Expire, & May Lose PageRank
Search engines may place weight not only on the number of links pointing at a page, but also on the rate at which links are accumulated. Even if you know the raw number of links and the site age it still does not tell you how many links were built last month or in the last year.
Not only are links born, but some of them rot. The web graph as a whole is over a decade old. Linkrot was a big issue in 1998, and it is still a big issue today. In 1998 6% of links were broken, and the DotBot crawl shows 7% of links being broken.
To appreciate how bad linkrot is...
if you publish a large site with many outbound links, run Xenu Link Sleuth through your site and see how many broken links you find
Some domains that expire may keep their PageRank, but many expiring domains lose their PageRank. With how hard it is to build links today and 1 in 7 links broke there are SEO tools designed around trying to capture this link equity
The domains that die off may later be re-registered and re-purposed. And keep in mind that the 1 in 7 broken links number is actually much higher than that when you consider how many people buy expired domain names and build them out.
By creating an index of the web in 2008 a person would have no idea if...
the links occurred recently
if the links are old
if the site expired and potentially lost much of its link weight
And Matt Cutts generally hates re-purposing expired domain names. Why? The very first spam site he found was a high PageRank expired domain linked from the W3C. That site was converted to a porn site, and ever since then (before Matt was the head of the webspam group - before Google even had a webspam group) Matt has not liked expired domains.
Matt offers background on that story 30 seconds into this video:
9. Advancing Algorithms That Move Away From PageRank & Anchor Text
domain history (ie: spam infractions/penalties, etc.)
signals of locality (hosting location, TLD, link sources, etc.)
searcher intent (Google's Amit Singhal stated "the same query can mean entirely different things in different countries. For example, [Côte d'Or] is a geographic region in France - but it is a large chocolate manufacturer in neighboring French-speaking Belgium")
other forms of search personalization (past searches, user subscriptions, frequently visited sites, etc.)
editorial partnerships with news companies & other universal search categories (like Google Shopping Search and the maps local onebox)
usage data (especially with sites they host, like YouTube)
All of those links might still be useful for crawl discovery, but they won’t support the ranking. That’s what we are constantly looking at in algorithms. I can tell you one thing, that over the last few years as we have been building out our search engine and incorporating lots of data, the absolute percentage contribution of links and anchor text to the natural ranking of algorithms or to the importance in our ranking algorithms has gone down somewhat.
It is not that Linkscape is a bad tool, it is just aiming to do something incredibly complex, and as long as Yahoo! Site Explorer gives us a decent free sample (and other tools let us layer data on top of Yahoo!) we can get a good idea of the approximate level of competition for free. But with Yahoo! at $12 a share, if Yahoo! gets bought out and Site Explorer goes away then Linkscape (or Majestic SEO, depending on who does a better job of innovation) might be one of the best SEO investments one can make.
Compete.com quietly launched a referral analytics product as part of their advanced package ($499/month). Even as a free user you can see the top 3 results for any site, which can be used to see how reliant a site is on search. Why is % of search traffic an important statistic?
If search traffic (as a % of total traffic) is low (relative to other competing sites) then it could indicate that there are organic optimization opportunities that are currently being missed and/or that site has a large organic traffic stream that can be marketed to in order to help it improve any search related weakness.
If search traffic (as a % of total traffic) is high (relative to other competing sites) then it could indicate that the site is near its full search potential, that the site is not very engaging, and/or does not have many loyal users
Here are search stats for SEO Book. Note that Google controls a minority of the traffic to this site, which means they have limited direct influence on the revenue of this site. Some sites are closer to 90% Google, which makes it easy for Google to effectively remove them from the web!
This sort of data is important for considering the viability of a business model, the stability of a site, and what multiple a site should sell for. It can also be used when considering the CPM of an ad unit - search traffic is much more targeted and goal oriented than a person browsing a forum is.
Until everyone and their dog started looking at PageRank (and how to manipulate it) it was a rather sound way of finding the most valuable backlinks. But with the pollution of endless bought links, nepotistic links, and PageRank only being updated quarterly it is tough to glean much market data from only looking at PageRank. Tools like SEO for Firefox (especially when used on a Yahoo! backlink search) allow you to gather more data about the quality of link sources. But they all try to measure proxies for value rather than how people actually surf the web.
Microsoft BrowseRank research would use browsing data to supplement PageRank on determining relevancy. In Internet Explorer 8 (currently in beta) a person's browsing details are sent to Microsoft by default. With ~ 80% of the browser market, Microsoft does not need to use a random walk for the core of their relevancy algorithm - they know what people are actually doing, and can use usage data as a big part of their relevancy algorithms.
Using a tool like Compete.com Referral Analytics makes it far easier to poach top affiliates, discover the best ad buying locations, and replicate a competitor's best backlinks. Be forewarned that the tool only works at the domain level, so it is much better at analyzing Yahoo.com than shopping.yahoo.com.
Along with referral analytics Compete offers destination analytics, which let you know what websites people visit AFTER visiting a particular site...which should help you glean information about how sites are monetizing, what offers are working well, what sites are well referenced by another site, and what sites people go to if they can't get what they want on the current site.
At $500 a month, this tool is probably only going to be used by those who are already fairly successful rather than as an entry level tool.
Google has been changing the code used to display their search results a number of times over the past couple days. We recently updated SEO for Firefox and Rank Checker. Both should work as of now, and if any more SERP changes happen we will try to update the extensions as soon as possible.
If you need to update these tools you can do so within the Firefox browser by clicking on Tools in the top Firefox menu, then from the Tools menu click on Add-ons. This will pop up the Add-ons / extensions window. At the bottom of this window there is a Find Updates button you can click. That will bring in the new updates and then when you re-start the browser the extensions should be fully functional again.
James from Semvironment created a plug in to automatically email webmasters you link to from within Wordpress blog posts. When he launched it, the opening post sent me an email
Hi! We linked to your website in our post: Link Builder for Wordpress - Download it Now!. Please stop by and check it out, subscribe to our blog and if you find something useful on our site or blog - we would welcome a link back anytime ... no obligation - we're happy to link to high quality websites and blogs like yours! To your continued success, [Your Name Here]
That link was broken (pointing to a revisioned archive version of the post before the URL changed), but even beyond that I sorta do not like the idea. Why? Automated communications is the enemy of relationship building. And the worst people to offend are the people you find interesting / important / influential enough to want to talk about them. You can get the person's attention just as easily by clicking the link in the post a dozen times and by commenting on their blog. And most of them would even be up for lending their time to you if you stoke their egos.
As more people start using a wider array of automated link building tools the effectiveness of automation will drop. And you can't train an automated tool to be personal. A tool like this might work in some verticals for a year or two, but if people find it effective look out for the tragedy of the commons to be heading toward your inbox some time soon!
Google announcedAdPlanner, a tool to help ad agencies find where desired demographic audiences are active online. The WSJ highlighted how the new Google tool can help make the ad marketplace more efficient:
The Web-audience data could be combined with the ad-serving system, so that advertisers would be able to find out whether they would reach the right audience before they committed to placing an ad.
In addition to AdPlanner, Google will launch another tool that compares consumer response to ads against a control group of users who did not see the ad:
Separately, Google this week is expected to roll out a new tool aimed at showing how Web surfers respond to online ads. It will compare groups of people who are exposed to an ad with others who haven't seen it, taking into account such factors as search activity and site visitation.