Such lists should be taken with a grain of salt, but at free one can't complain about the price. As time passes free and good enough is going to force those selling tools and information to offer something that has a sustainable advantage over free.
At the same time...
the sea of information will become increasingly hard to navigate, increasing the value of filters (particularly those built around a shared perspective or bias)
hyped up salesmen will be able to build many business models out of selling such recycled information to the uninitiated, forcing others who sell information to add even more differentiators between themselves and the competition
this shows bid prices and search volume estimates next to keywords (like the Google Traffic Estimator)
this shows your current page titles and keywords
this shows the % of organic and paid traffic going to a URL
For any keyword, the Google Search-based Keyword Tool will show up to 800 related keywords with cost and search volume estimates. This tool also works to show you 100 keywords related to a site, and if you own a website they will show you thousands of keywords that they think you could bid on which are not already in your account. In addition they show your search share of voice (via ads and organic search results) for keywords. This data is easy to export using a handy export button.
There are a variety of cool extra filters that can be applied on this tool, including...
minimum or maximum search volumes
bid price range
low, medium, or high competition
keyword in URL
combining URL and keywords as filters
keyword + general category
Using a variety of different combinations for these filters you can see many different sets of 800 keywords even within the same subset. Export these different lists a variety of times and you can quickly build a list of thousands of high value keywords.
A sweet new competitive research tool by the name SEMRush has hit the market. It can be seen as a deeper extension of the SEO Digger project (adding PPC data and tracking AdWords keywords), and a competitor to services like Compete.com and SpyFu (which recently launched SpyFu Kombat).
Brief Tool Overview
Competitive research tools can help you find a baseline for what to do & where to enter a market. Before spending a dime on SEO (or even buying a domain name for a project), it is always worth putting in the time to get a quick lay of the land & learn from your existing competitors.
Seeing which keywords are most valuable can help you figure out which areas to invest the most in.
Seeing where existing competitors are strong can help you find strategies worth emulating. While researching their performance, it may help you find new pockets of opportunities & keyword themes which didn't show up in your initial keyword research.
Seeing where competitors are weak can help you build a strategy to differentiate your approach.
Enter a competing URL in the above search box & you will quickly see where your competitors are succeeding, where they are failing & get insights on how to beat them. SEMrush offers:
granular data across the global Bing & Google databases, along with over 2-dozen regional localized country-specific Google databases (Argentina, Australia, Belgium, Brazil, Canada, Denmark, Finland, France, Germany, Hungary, Japan, Hong Kong, India, Ireland, Israel, Italy, Mexico, Netherlands, Norway, Poland, Russia, Singapore, Spain, Sweden, Switzerland, Turkey, United Kingdom, United States)
search volume & ad bid price estimates by keyword (which, when combined, function as an estimate of keyword value) for over 120,000,000 words
keyword data by site or by page across 74,000,000 domain names
the ability to look up related keywords
the ability to directly compare domains against one another to see relative strength
the ability to compare organic search results versus paid search ads to leverage data from one source into the other channel
the ability to look up sites which have a similar ranking footprint as an existing competitor to uncover new areas & opportunities
historical performance data, which can be helpful in determining if the site has had manual penalties or algorithmic ranking filters applied against it
a broad array of new features like tracking video ads, display ads, PLAs, backlinks, etc.
While their tool is a paid service, the above search box still allows you to get a great sampling of their data for free. SEMrush is easily our favorite competitive research tool. We like their tool so much we also license their data to offer our paying subscribers a competitive research tool powered by their database.
SEM Rush vs Compete.com
The big value add that SEM Rush has over a tool like Compete.com is that SEM Rush adds cost per click estimates (scraped from Google's Traffic Estimator tool) and estimated traffic volumes (from the Google AdWords keyword tool) near each keyword. Thus, rather than showing the traffic distribution to each site, this tool can list keyword value distribution for the sites (keyword value * estimated traffic).
Using these estimates does not provide results that are as accurate as Compete.com's data licensing strategy, but if you own a site and know what it earns, you can set up a ratio to normalize the differences (at least to some extent, within the same vertical, for sites of similar size, using a similar business model).
One of our sites that earns about $5,000 a month shows a Google traffic value of close to $20,000 a month.
5,000/20,000 = 1/4 = 0.25
A similar site in the same vertical shows $10,000
$10,000 * 0.25 = $2,500
Disclaimers With Normalizing Data
It is hard to monetize traffic as well as Google does, so in virtually every competitive market your profit per visitor (after expenses) will generally be less than Google. Some reason why..
In some markets people are losing money to buy marketshare, while in other markets people may overbid just to block out competition.
Some merchants simply have fatter profit margins and can afford to outbid affiliates.
It is hard to integrate advertising in your site anywhere near as aggressively as Google does while still creating a site that will be able to gather enough links (and other signals of quality) to take a #1 organic ranking in competitive markets...so by default there will typically be some amount of slippage.
A site that offers editorial content wrapped in light ads will not convert eyeballs into cash anywhere near as well as a lead generation oriented affiliate site would.
SEM Rush Features
Keyword Values & Volumes
As mentioned above, this data is scraped from the Google Traffic Estimator and the Google Keyword Tool.
Top Search Traffic Domains
A list of the top 100 domain names that are estimated to be the highest value downstream traffic sources from Google.
You could get a similar list from Compete.com's Referral Analytics by running a downstream report on Google.com, although I think that might also include traffic from some of Google's non-search properties like Reader.
Here is a list of sites that rank for many of the same keywords that SEO Book ranks for
Here is a list of a few words where Seo Book and SEOmoz compete in the rankings
Compare AdWords to Organic Search
These are sites that rank for keywords that SEO Book is buying through AdWords
And these are sites that buy AdWords ads for keywords that this site ranks for
Once Upon a Time...
I was going to create a tool similar to this one about a year ago, until I hired a programmer that was EPIC FAIL. The guy who managed that program is no longer selling programming services - and that makes the world a better place.
I actually had 3 attempts at such a tool. I bought a GREAT domain name, spec'd out the project, then planned on doing it...
investor backed, who decided to back out
self funded, but I hired... 1.) a programmer who mid-project decided he needed to make double what I make working part-time, then 2.) the worst programmers ever.
combination of heavily self funded with the guidance of a bad ass VC, but I backed out due to a need to focus on this site
I spent most of this year focusing on trying to build our community and raise our editorial quality (both goals are going well, but require significant maintenance). We have had 4 strong hires in a row, so it seems like our luck has changed on that front. Recently I started working with a programmer who really clicks with me, often taking my ideas and making them way better than I intended.
If these guys had not made this tool I was going to try to take another run at something like this early next year...which brings up a good point that a friend (and wicked intelligent open source programmer) named François Planque told me. He said all he had to do was think up a good idea but not do it, and within 6 to 12 months if he had not done it, someone else would have already launched it.
Entry cost is so low that a lot of great tools are going to get made in short order, but it is hard to win by sitting on a good idea. ;)
Wordtracker released a new keyword tool based around keyword questions. The information is quick and easy to export. Ken McGaffin said, “This is a fun tool that is a great source of inspiration for web content writers. You need never be short of creative ideas again." And it is a cool idea - good job Wordtracker!
They also did a comparison between their link counts and those found by Yahoo! Site Explorer and LinkScape. They claim to have more links in their database than Yahoo! is showing, but I have to wonder how they could do that economically, if they are counting more duplicates, and why they haven't bought a site design that reflects how much they must be spending on data.
A few years back search engines were in an ego based contest about who has the biggest index, and I find it a bit ironic that a couple SEO companies will likely be engaged in such a data war...but the marketplace competition should be good for all SEOs.
So my timing on that last post was a bit off, but I still think the general thesis is valid. But now that there has been so much negative feedback I figure it is my job to play devil's advocate and highlight reasons why most SEOs do not need to be too worried about Linkscape.
Unique Linking Domains
One of the coolest features of this tool is knowing the number of unique linking domains pointing links at a specific site, but that feature is for paying members only.
A competing tool by the name of Majestic SEO allows you to see that data as part of their free overview. Click on the image below for an example.
If your competitor has high authority links then you need more than just quantity to compete, but if most of their backlinks are garbage then this is a good stat to have, along with many other stats you can get from tools like SEO for Firefox.
Not that I advocate spam reporting (as the official guidelines have departed from reality so much that almost everyone that ranks is spamming and/or spammed in the past to get to their current market position), but for professional SEOs that own dozens of sites and like doing spam reports to Google this might be a good tool for outing competitors, since it makes it easy to find some noscript links, links from off the page, inbound 301 redirects, but the average webmaster probably does not need to worry about that.
A Bit Top Heavy
One of the biggest limitations in Linkscape is that you can only go 500 results deep unless you want to buy a custom report. They allow you to see various lenses of 500 at a time through search features and filters, but a big recommendation I can make on this front is for them to allow you to see all that data, even if it requires exporting data to CSV...they already spent the money to collect the data, so if you're a customer they may as well give it to you...it helps nobody if nobody sees it.
Majestic SEO appears to have a similar sized database as Linkscape, and they allow you to do a full data export for your own domain free of charge. Other domains they charge a scaling price for depending on the number of links to the domain.
More Cool Features?
Nick Gerner promised more features in the next version of Linkscape, but unless they start buying usage data and become more like Compete.com I am not sure if it will be a game changer. On to explaining why...
Personally, I'm not too worried. You want to compete with me and get links in places where I'm listed? We get listed in places where editorial rules. So just knowing where we're at doesn't get you in the door -- you have to be good enough to walk in. And if you are good enough, well, good I guess.
The highest quality links typically tend to be editorial in nature, with many of those being driven by social relationships. No matter how much one decides to analyze link patterns, they can't re-create most of the link relationships if they don't already have the content quality, market exposure, and awareness. And if you copy someone's idea after they already did it you need to greatly improve upon it to get credit for it.
2. Tons of Alternative Data Sources
Common link analysis questions...
How do I Get a Basic Competitive Overview of the Search Results?
Search Google with SEO for Firefox turned on. Make sure you are pulling data in the automatic mode while searching.
I Want to do Anchor Text Analysis. How do I Analyze Links?
Some options include...
SEO Link Analysis - a free Firefox extension that adds anchor text to Google Webmaster Central and Yahoo! Site Explorer.
Backlink Analyzer - a free desktop based tool I had created a few years ago that pulls data from the Yahoo! API. Make sure to watch the video on the download page before using it.
I Want to Find New Links to Competing Sites
If you want to find what someone's best ideas are all you have to do is subscribe to the Google Blogsearch feed for links to their site, like so. That should list many of the people who are talking about this site.
This is the same as competing sites, but you can also use your web analytics and server logs to dig up additional information. You can also look inside Google Webmaster Central to download backlink reports.
I Want to Find The Most Authoritative Links Pointing at a Site
Yahoo! Site Explorer generally orders backlinks roughly in terms of authority, with some of the most authoritative backlinks showing up at the top of their results.
What Sites Drive the Most Traffic to My Competitors?
The best way I have found to get this data is from Compete.com Referral Analytics, though it requires a $500 a month subscription...which is a nice chunk of change, unless you are already doing quite well!
Do I Have Any Broken Links?
Xenu Link Sleuth will crawl your site and help you find broken links, showing you which pages the broken links were on.
Each search engine has its own crawling priorities and own web graph. Google has probably spent hundreds of millions of dollars building and refining their crawling sequence. No two crawls are the same.
The irrelevant links at the bottom of a page, which will not be as valuable for a user, don’t add to the quality of the user experience, so we don’t account for those in our ranking. All of those links might still be useful for crawl discovery, but they won’t support the ranking.
5. Microsoft May be Looking to Heavily Incorporate Usage Data
Microsoft did research on BrowseRank, which aims to use actual usage data to augment (or perhaps replace) their link graph. Be default, Internet Explorer 8 sends usage data to Microsoft...when you know what 80% of web users are doing you do not need to rely on a random walk.
Think of having access to the majority of the web's usage data like this:
If Google's algorithms are more relevant than Microsoft, then putting weight on usage data allows Microsoft to quickly catch up by weighting whatever Google is weighting
Microsoft could theoretically be better than Google at filtering out paid links, as most paid links in a sidebar or footer do not send much traffic...and thus could easily be weighted less than links in content - though with Google owning so many products they could improve significantly on this front as well, if they decided to use their AdSense data, analytics data, Chrome browser data, Feedburner data, and toolbar data.
Beyond those editors there are many search engineers inside the webspam team offering a variety of techniques to throw off SEOs, including
stripping all PageRank from a site and killing all its rankings
stripping some portion of a site's PageRank and ranking abilities
stripping PageRank from the toolbar but still allowing sites to rank
showing full PageRank in the toolbar, but killing the ability of a link to pass PageRank
Without working inside of Google and/or buying and testing lots of links across a wide array of sites and verticals it would be hard to know if any particular site passes PageRank, and how much it might pass. For instance, a link from Text-Link-Ads.com's website is one of my highest MozRank links, but I doubt Google places much weight on that link since Google does not let Text Link Ads rank for their own brand.
7. Search Engine Editorial Policies are Selective, & Constantly Changing
According to Udi Manber, Google did 450 search algorithm updates last year. Even if you could somehow catch up with all the editorial stuff search engines were doing to manipulate their version of the link based web graph, you would have a hard time of keeping up with it - let alone accounting for the hoards of usage data the search engines have.
The status of a link (and its ability to pass PageRank) may arbitrarily change based on media exposure. In the past many websites were hijacked by 302 affiliate links (this even happened to Google's site, and this is still happening today to corporate sites as big as Snapnames).
Shockingly, when asked point blank if affiliate programs that employed juice-passing links (those not using nofollow) were against guidelines or if they would be discounted, the engineers all agreed with the position taken by Sean Suchter of Yahoo!. He said, in no uncertain terms, that if affiliate links came from valuable, relevant, trust-worthy sources - bloggers endorsing a product, affiliates of high quality, etc. - they would be counted in link algorithms. Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value.
A few years ago I set up my affiliate program to use 301 redirects to prevent hijacking, and get any link benefits I could. But right after I changed by business model to a membership site my affiliate program was featured/outed in this interview, and it no longer passes PageRank.
Watch the above video and see how at 2 minutes and 15 seconds in my site was put up for review to any Google engineer that happened to watch it.
The same set of links, to the same site, using the same format, under similar circumstances...
counts for most major corporations (and is allegedly an approved and legitimate strategy)
counted for this site for years
stopped counting around the time they were outed by a popular SEO blogger
8. Temporal Algorithms + Domains Expire, & May Lose PageRank
Search engines may place weight not only on the number of links pointing at a page, but also on the rate at which links are accumulated. Even if you know the raw number of links and the site age it still does not tell you how many links were built last month or in the last year.
Not only are links born, but some of them rot. The web graph as a whole is over a decade old. Linkrot was a big issue in 1998, and it is still a big issue today. In 1998 6% of links were broken, and the DotBot crawl shows 7% of links being broken.
To appreciate how bad linkrot is...
if you publish a large site with many outbound links, run Xenu Link Sleuth through your site and see how many broken links you find
Some domains that expire may keep their PageRank, but many expiring domains lose their PageRank. With how hard it is to build links today and 1 in 7 links broke there are SEO tools designed around trying to capture this link equity
The domains that die off may later be re-registered and re-purposed. And keep in mind that the 1 in 7 broken links number is actually much higher than that when you consider how many people buy expired domain names and build them out.
By creating an index of the web in 2008 a person would have no idea if...
the links occurred recently
if the links are old
if the site expired and potentially lost much of its link weight
And Matt Cutts generally hates re-purposing expired domain names. Why? The very first spam site he found was a high PageRank expired domain linked from the W3C. That site was converted to a porn site, and ever since then (before Matt was the head of the webspam group - before Google even had a webspam group) Matt has not liked expired domains.
Matt offers background on that story 30 seconds into this video:
9. Advancing Algorithms That Move Away From PageRank & Anchor Text
domain history (ie: spam infractions/penalties, etc.)
signals of locality (hosting location, TLD, link sources, etc.)
searcher intent (Google's Amit Singhal stated "the same query can mean entirely different things in different countries. For example, [Côte d'Or] is a geographic region in France - but it is a large chocolate manufacturer in neighboring French-speaking Belgium")
other forms of search personalization (past searches, user subscriptions, frequently visited sites, etc.)
editorial partnerships with news companies & other universal search categories (like Google Shopping Search and the maps local onebox)
usage data (especially with sites they host, like YouTube)
All of those links might still be useful for crawl discovery, but they won’t support the ranking. That’s what we are constantly looking at in algorithms. I can tell you one thing, that over the last few years as we have been building out our search engine and incorporating lots of data, the absolute percentage contribution of links and anchor text to the natural ranking of algorithms or to the importance in our ranking algorithms has gone down somewhat.
It is not that Linkscape is a bad tool, it is just aiming to do something incredibly complex, and as long as Yahoo! Site Explorer gives us a decent free sample (and other tools let us layer data on top of Yahoo!) we can get a good idea of the approximate level of competition for free. But with Yahoo! at $12 a share, if Yahoo! gets bought out and Site Explorer goes away then Linkscape (or Majestic SEO, depending on who does a better job of innovation) might be one of the best SEO investments one can make.
Compete.com quietly launched a referral analytics product as part of their advanced package ($499/month). Even as a free user you can see the top 3 results for any site, which can be used to see how reliant a site is on search. Why is % of search traffic an important statistic?
If search traffic (as a % of total traffic) is low (relative to other competing sites) then it could indicate that there are organic optimization opportunities that are currently being missed and/or that site has a large organic traffic stream that can be marketed to in order to help it improve any search related weakness.
If search traffic (as a % of total traffic) is high (relative to other competing sites) then it could indicate that the site is near its full search potential, that the site is not very engaging, and/or does not have many loyal users
Here are search stats for SEO Book. Note that Google controls a minority of the traffic to this site, which means they have limited direct influence on the revenue of this site. Some sites are closer to 90% Google, which makes it easy for Google to effectively remove them from the web!
This sort of data is important for considering the viability of a business model, the stability of a site, and what multiple a site should sell for. It can also be used when considering the CPM of an ad unit - search traffic is much more targeted and goal oriented than a person browsing a forum is.
Until everyone and their dog started looking at PageRank (and how to manipulate it) it was a rather sound way of finding the most valuable backlinks. But with the pollution of endless bought links, nepotistic links, and PageRank only being updated quarterly it is tough to glean much market data from only looking at PageRank. Tools like SEO for Firefox (especially when used on a Yahoo! backlink search) allow you to gather more data about the quality of link sources. But they all try to measure proxies for value rather than how people actually surf the web.
Microsoft BrowseRank research would use browsing data to supplement PageRank on determining relevancy. In Internet Explorer 8 (currently in beta) a person's browsing details are sent to Microsoft by default. With ~ 80% of the browser market, Microsoft does not need to use a random walk for the core of their relevancy algorithm - they know what people are actually doing, and can use usage data as a big part of their relevancy algorithms.
Using a tool like Compete.com Referral Analytics makes it far easier to poach top affiliates, discover the best ad buying locations, and replicate a competitor's best backlinks. Be forewarned that the tool only works at the domain level, so it is much better at analyzing Yahoo.com than shopping.yahoo.com.
Along with referral analytics Compete offers destination analytics, which let you know what websites people visit AFTER visiting a particular site...which should help you glean information about how sites are monetizing, what offers are working well, what sites are well referenced by another site, and what sites people go to if they can't get what they want on the current site.
At $500 a month, this tool is probably only going to be used by those who are already fairly successful rather than as an entry level tool.
Google has been changing the code used to display their search results a number of times over the past couple days. We recently updated SEO for Firefox and Rank Checker. Both should work as of now, and if any more SERP changes happen we will try to update the extensions as soon as possible.
If you need to update these tools you can do so within the Firefox browser by clicking on Tools in the top Firefox menu, then from the Tools menu click on Add-ons. This will pop up the Add-ons / extensions window. At the bottom of this window there is a Find Updates button you can click. That will bring in the new updates and then when you re-start the browser the extensions should be fully functional again.
James from Semvironment created a plug in to automatically email webmasters you link to from within Wordpress blog posts. When he launched it, the opening post sent me an email
Hi! We linked to your website in our post: Link Builder for Wordpress - Download it Now!. Please stop by and check it out, subscribe to our blog and if you find something useful on our site or blog - we would welcome a link back anytime ... no obligation - we're happy to link to high quality websites and blogs like yours! To your continued success, [Your Name Here]
That link was broken (pointing to a revisioned archive version of the post before the URL changed), but even beyond that I sorta do not like the idea. Why? Automated communications is the enemy of relationship building. And the worst people to offend are the people you find interesting / important / influential enough to want to talk about them. You can get the person's attention just as easily by clicking the link in the post a dozen times and by commenting on their blog. And most of them would even be up for lending their time to you if you stoke their egos.
As more people start using a wider array of automated link building tools the effectiveness of automation will drop. And you can't train an automated tool to be personal. A tool like this might work in some verticals for a year or two, but if people find it effective look out for the tragedy of the commons to be heading toward your inbox some time soon!