SEM Rush Search Marketing Research - Review of SEMRush.com

Nov 8th

What is SEM Rush?

A sweet new competitive research tool by the name SEMRush has hit the market. It can be seen as a deeper extension of the SEO Digger project (adding PPC data and tracking AdWords keywords), and a competitor to services like Compete.com and SpyFu (which recently launched SpyFu Kombat).

SEM Rush vs Compete.com

The big value add that SEM Rush has over a tool like Compete.com is that SEM Rush adds cost per click estimates (scraped from Google's Traffic Estimator tool) and estimated traffic volumes (from the Google AdWords keyword tool) near each keyword. Thus, rather than showing the traffic distribution to each site, this tool can list keyword value distribution for the sites (keyword value * estimated traffic).

Normalizing Data

Using these estimates does not provide results that are as accurate as Compete.com's data licensing strategy, but if you own a site and know what it earns, you can set up a ratio to normalize the differences (at least to some extent, within the same vertical, for sites of similar size, using a similar business model).

One of our sites that earns about $5,000 a month shows a Google traffic value of close to $20,000 a month.
5,000/20,000 = 1/4 = 0.25

A similar site in the same vertical shows $10,000
$10,000 * 0.25 = $2,500

Disclaimers With Normalizing Data

It is hard to monetize traffic as well as Google does, so in virtually every competitive market your profit per visitor (after expenses) will generally be less than Google. Some reason why..

  1. In some markets people are losing money to buy marketshare, while in other markets people may overbid just to block out competition.
  2. Some merchants simply have fatter profit margins and can afford to outbid affiliates.
  3. It is hard to integrate advertising in your site anywhere near as aggressively as Google does while still creating a site that will be able to gather enough links (and other signals of quality) to take a #1 organic ranking in competitive markets...so by default there will typically be some amount of slippage.
  4. A site that offers editorial content wrapped in light ads will not convert eyeballs into cash anywhere near as well as a lead generation oriented affiliate site would.

SEM Rush Features

Keyword Values & Volumes

As mentioned above, this data is scraped from the Google Traffic Estimator and the Google Keyword Tool.

Top Search Traffic Domains

A list of the top 100 domain names that are estimated to be the highest value downstream traffic sources from Google.

You could get a similar list from Compete.com's Referral Analytics by running a downstream report on Google.com, although I think that might also include traffic from some of Google's non-search properties like Reader.

Top Competitors

Here is a list of sites that rank for many of the same keywords that SEO Book ranks for

Overlapping Keywords

Here is a list of a few words where Seo Book and SEOmoz compete in the rankings

Compare AdWords to Organic Search

These are sites that rank for keywords that SEO Book is buying through AdWords

And these are sites that buy AdWords ads for keywords that this site ranks for

Once Upon a Time...

I was going to create a tool similar to this one about a year ago, until I hired a programmer that was EPIC FAIL. The guy who managed that program is no longer selling programming services - and that makes the world a better place.

I actually had 3 attempts at such a tool. I bought a GREAT domain name, spec'd out the project, then planned on doing it...

  • investor backed, who decided to back out
  • self funded, but I hired... 1.) a programmer who mid-project decided he needed to make double what I make working part-time, then 2.) the worst programmers ever.
  • combination of heavily self funded with the guidance of a bad ass VC, but I backed out due to a need to focus on this site

I spent most of this year focusing on trying to build our community and raise our editorial quality (both goals are going well, but require significant maintenance). We have had 4 strong hires in a row, so it seems like our luck has changed on that front. Recently I started working with a programmer who really clicks with me, often taking my ideas and making them way better than I intended.

If these guys had not made this tool I was going to try to take another run at something like this early next year...which brings up a good point that a friend (and wicked intelligent open source programmer) named François Planque told me. He said all he had to do was think up a good idea but not do it, and within 6 to 12 months if he had not done it, someone else would have already launched it.

Entry cost is so low that a lot of great tools are going to get made in short order, but it is hard to win by sitting on a good idea. ;)

Lots of Marketing Goodies

PPC Stuff

My wife recently put together a PPC strategy flowchart. Check it out and please give her feedback.

Search Engine Land has a good post with interview snippets of Nick Fox about some of the recent Google AdWords changes.

Google announced they are ending the proposed partnership with Yahoo!

The FCC approved the wireless broadband whitespace plan, which in time should make for more online searchers.

SEO Stuff

Wordtracker released a new keyword tool based around keyword questions. The information is quick and easy to export. Ken McGaffin said, “This is a fun tool that is a great source of inspiration for web content writers. You need never be short of creative ideas again." And it is a cool idea - good job Wordtracker!

Majestic SEO did a major update, claiming to have crawled about 52 billion URLs and has nearly 350 billion unique URLs in their anchor index. Here is a list of their top URLs with inbound links.

They also did a comparison between their link counts and those found by Yahoo! Site Explorer and LinkScape. They claim to have more links in their database than Yahoo! is showing, but I have to wonder how they could do that economically, if they are counting more duplicates, and why they haven't bought a site design that reflects how much they must be spending on data.

A few years back search engines were in an ego based contest about who has the biggest index, and I find it a bit ironic that a couple SEO companies will likely be engaged in such a data war...but the marketplace competition should be good for all SEOs.

I recently did an interview with Patrick Altoft about link building for affiliates.

Jim Boykin started the WeBuildPages SEO blog.

In the weird department, have you heard the We Like SEO song yet?

Conversion Stuff

Conversion Rate Experts highlighted 14 cool conversion enhancing tools.

Avinash Kaushik showed how powerful Google Analytics segmentation is.

Content Stuff

The NYT is getting close to where it will be hard for them to service their debt. Who should buy them out?

Funny blog post about economic blog posts

Political Marketing

Obama won the election and gave one of his best speeches. But Seth Godin didn't even wait for the vote to happen before he deconstructed the campaigns from a marketer's perspective. XMCP also discussed the backchannel negative PR campaigns.

SEO for Firefox - Now With SEO X-ray

Oct 28th

We recently added an SEO X-ray feature to SEO for Firefox. You must use Firefox 3.0 or above to see these features, but if you want to see...

  • how the on page optimization of any page looks (headings, meta description, page title)
  • the keyword density of the page and popular phrases on the page
  • how many links point into a page (total links, or links from external resources)
  • how many links point out of a page (as well as the anchor text of these links, nofollow vs follow, internal vs external - all exportable in CSV format)

then this new feature makes it quick and easy to do all of that. Simply right click on the page you are viewing, scroll down to SEO for Firefox, and click on SEO X-ray.

That will show you an overlay on the screen like this

We are planning on doing another update in the next couple days, and may add...

  • the IP address of the site (and links to other sites on the same IP address)
  • character and word counts for page title and meta description body content
  • a link to the domain tools overview page for the associated site

If you are using Firefox 3 and SEO for Firefox please give this a try and let us know what you think.

SEOmoz's Linkscape: Why the Backlash is Overblown

Oct 21st

Right after I finished writing a post about how being likeable is a great business strategy, I went back to Sphinn and saw it erupted with controversy and negative feedback about SEOmoz's Linkscape. Since then threads have been open, closed, and open. People are worried about everything from the index size to how to remove your site to why you shouldn't label your site with an obvious SEO footprint.

So my timing on that last post was a bit off, but I still think the general thesis is valid. But now that there has been so much negative feedback I figure it is my job to play devil's advocate and highlight reasons why most SEOs do not need to be too worried about Linkscape.

Cool Features

Unique Linking Domains

One of the coolest features of this tool is knowing the number of unique linking domains pointing links at a specific site, but that feature is for paying members only.

A competing tool by the name of Majestic SEO allows you to see that data as part of their free overview. Click on the image below for an example.

If your competitor has high authority links then you need more than just quantity to compete, but if most of their backlinks are garbage then this is a good stat to have, along with many other stats you can get from tools like SEO for Firefox.

Spam Reporting

Not that I advocate spam reporting (as the official guidelines have departed from reality so much that almost everyone that ranks is spamming and/or spammed in the past to get to their current market position), but for professional SEOs that own dozens of sites and like doing spam reports to Google this might be a good tool for outing competitors, since it makes it easy to find some noscript links, links from off the page, inbound 301 redirects, but the average webmaster probably does not need to worry about that.

A Bit Top Heavy

One of the biggest limitations in Linkscape is that you can only go 500 results deep unless you want to buy a custom report. They allow you to see various lenses of 500 at a time through search features and filters, but a big recommendation I can make on this front is for them to allow you to see all that data, even if it requires exporting data to CSV...they already spent the money to collect the data, so if you're a customer they may as well give it to you...it helps nobody if nobody sees it.

Majestic SEO appears to have a similar sized database as Linkscape, and they allow you to do a full data export for your own domain free of charge. Other domains they charge a scaling price for depending on the number of links to the domain.

More Cool Features?

Nick Gerner promised more features in the next version of Linkscape, but unless they start buying usage data and become more like Compete.com I am not sure if it will be a game changer. On to explaining why...

1. Editorial Rules

When Linkscape was announced Danny Sullivan said:

Personally, I'm not too worried. You want to compete with me and get links in places where I'm listed? We get listed in places where editorial rules. So just knowing where we're at doesn't get you in the door -- you have to be good enough to walk in. And if you are good enough, well, good I guess.

The highest quality links typically tend to be editorial in nature, with many of those being driven by social relationships. No matter how much one decides to analyze link patterns, they can't re-create most of the link relationships if they don't already have the content quality, market exposure, and awareness. And if you copy someone's idea after they already did it you need to greatly improve upon it to get credit for it.

2. Tons of Alternative Data Sources

Common link analysis questions...

How do I Get a Basic Competitive Overview of the Search Results?

Search Google with SEO for Firefox turned on. Make sure you are pulling data in the automatic mode while searching.

I Want to do Anchor Text Analysis. How do I Analyze Links?

Some options include...

  • SEO Link Analysis - a free Firefox extension that adds anchor text to Google Webmaster Central and Yahoo! Site Explorer.
  • Link Diagnosis - another useful Firefox extension.
  • Link Analysis Tool - shows the PageRank and number of inlinks to each page on a site, though it requires you to set up a MySQL database.
  • Both Google Webmaster Central and Majestic SEO allow you to download backlink profiles for your own sites after you authenticate your sites.
  • Backlink Analyzer - a free desktop based tool I had created a few years ago that pulls data from the Yahoo! API. Make sure to watch the video on the download page before using it.

I Want to Find New Links to Competing Sites

If you want to find what someone's best ideas are all you have to do is subscribe to the Google Blogsearch feed for links to their site, like so. That should list many of the people who are talking about this site.

A paid option on this front is Advanced Link Manager. It costs $199 (or $299 if you package it with Advanced Web Ranking) and scrapes data from Yahoo!, keeping track of the date when the link was found.

I Want to Find New Links to My Site

This is the same as competing sites, but you can also use your web analytics and server logs to dig up additional information. You can also look inside Google Webmaster Central to download backlink reports.

I Want to Find The Most Authoritative Links Pointing at a Site

Yahoo! Site Explorer generally orders backlinks roughly in terms of authority, with some of the most authoritative backlinks showing up at the top of their results.

I Want to Find .edu Links

Yahoo! Search offers a wide array of advanced link operators. Here are .edu & .gov links pointing at searchengineland.com.

I Want to Get an Estimate of Unique Linking Domains

Majestic SEO offers a free estimate...though, like LinkScape, their crawl is not as comprehensive as Yahoo!'s.

I Want to Find Hub Links?

What Sites Drive the Most Traffic to My Competitors?

The best way I have found to get this data is from Compete.com Referral Analytics, though it requires a $500 a month subscription...which is a nice chunk of change, unless you are already doing quite well!

Do I Have Any Broken Links?

3. All Link Graphs Are Unique

Each search engine has its own crawling priorities and own web graph. Google has probably spent hundreds of millions of dollars building and refining their crawling sequence. No two crawls are the same.

Image from Google Touchgraph.

4. Yahoo! Search Counts Link Weight Differently Based on Page Segmentation

Google's PageRank was designed based on a random walk theory, where browsers click a random link on the page. But search engines are looking to move beyond the random walk model.

Yahoo! Search's Priyank Garg stated:

The irrelevant links at the bottom of a page, which will not be as valuable for a user, don’t add to the quality of the user experience, so we don’t account for those in our ranking. All of those links might still be useful for crawl discovery, but they won’t support the ranking.

5. Microsoft May be Looking to Heavily Incorporate Usage Data

Microsoft did research on BrowseRank, which aims to use actual usage data to augment (or perhaps replace) their link graph. Be default, Internet Explorer 8 sends usage data to Microsoft...when you know what 80% of web users are doing you do not need to rely on a random walk.

Think of having access to the majority of the web's usage data like this:

  • If Google's algorithms are more relevant than Microsoft, then putting weight on usage data allows Microsoft to quickly catch up by weighting whatever Google is weighting
  • Microsoft could theoretically be better than Google at filtering out paid links, as most paid links in a sidebar or footer do not send much traffic...and thus could easily be weighted less than links in content - though with Google owning so many products they could improve significantly on this front as well, if they decided to use their AdSense data, analytics data, Chrome browser data, Feedburner data, and toolbar data.

6. Google Does a Lot of Hand Editing

Google hires 10,000 remote quality raters.

Beyond those editors there are many search engineers inside the webspam team offering a variety of techniques to throw off SEOs, including

  • stripping all PageRank from a site and killing all its rankings
  • stripping some portion of a site's PageRank and ranking abilities
  • stripping PageRank from the toolbar but still allowing sites to rank
  • showing full PageRank in the toolbar, but killing the ability of a link to pass PageRank

Without working inside of Google and/or buying and testing lots of links across a wide array of sites and verticals it would be hard to know if any particular site passes PageRank, and how much it might pass. For instance, a link from Text-Link-Ads.com's website is one of my highest MozRank links, but I doubt Google places much weight on that link since Google does not let Text Link Ads rank for their own brand.

Read Eric Schmidt's perspective on brands to consider how Google holds different sites to different standards.

7. Search Engine Editorial Policies are Selective, & Constantly Changing

According to Udi Manber, Google did 450 search algorithm updates last year. Even if you could somehow catch up with all the editorial stuff search engines were doing to manipulate their version of the link based web graph, you would have a hard time of keeping up with it - let alone accounting for the hoards of usage data the search engines have.

The status of a link (and its ability to pass PageRank) may arbitrarily change based on media exposure. In the past many websites were hijacked by 302 affiliate links (this even happened to Google's site, and this is still happening today to corporate sites as big as Snapnames).

At an SEO conference about 3 or 4 months back someone highlighted that some large sites use 301 redirects on affiliate links. This topic came up once again at SMX East, where it was deemed an acceptable marketing practice:

Shockingly, when asked point blank if affiliate programs that employed juice-passing links (those not using nofollow) were against guidelines or if they would be discounted, the engineers all agreed with the position taken by Sean Suchter of Yahoo!. He said, in no uncertain terms, that if affiliate links came from valuable, relevant, trust-worthy sources - bloggers endorsing a product, affiliates of high quality, etc. - they would be counted in link algorithms. Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value.

A few years ago I set up my affiliate program to use 301 redirects to prevent hijacking, and get any link benefits I could. But right after I changed by business model to a membership site my affiliate program was featured/outed in this interview, and it no longer passes PageRank.

Watch the above video and see how at 2 minutes and 15 seconds in my site was put up for review to any Google engineer that happened to watch it.

The same set of links, to the same site, using the same format, under similar circumstances...

  • counts for most major corporations (and is allegedly an approved and legitimate strategy)
  • counted for this site for years
  • stopped counting around the time they were outed by a popular SEO blogger

8. Temporal Algorithms + Domains Expire, & May Lose PageRank

Search engines may place weight not only on the number of links pointing at a page, but also on the rate at which links are accumulated. Even if you know the raw number of links and the site age it still does not tell you how many links were built last month or in the last year.

Not only are links born, but some of them rot. The web graph as a whole is over a decade old. Linkrot was a big issue in 1998, and it is still a big issue today. In 1998 6% of links were broken, and the DotBot crawl shows 7% of links being broken.

To appreciate how bad linkrot is...

Some domains that expire may keep their PageRank, but many expiring domains lose their PageRank. With how hard it is to build links today and 1 in 7 links broke there are SEO tools designed around trying to capture this link equity

The domains that die off may later be re-registered and re-purposed. And keep in mind that the 1 in 7 broken links number is actually much higher than that when you consider how many people buy expired domain names and build them out.

By creating an index of the web in 2008 a person would have no idea if...

  • the links occurred recently
  • if the links are old
  • if the site expired and potentially lost much of its link weight

And Matt Cutts generally hates re-purposing expired domain names. Why? The very first spam site he found was a high PageRank expired domain linked from the W3C. That site was converted to a porn site, and ever since then (before Matt was the head of the webspam group - before Google even had a webspam group) Matt has not liked expired domains.

Matt offers background on that story 30 seconds into this video:

9. Advancing Algorithms That Move Away From PageRank & Anchor Text

Paid links have been an obvious weak spot in the relevancy algorithms for years. PageRank and anchor text are still both important, but Google also considers other factors like...

  • domain age / link age
  • domain name (and extension)
  • domain history (ie: spam infractions/penalties, etc.)
  • site authority
  • signals of locality (hosting location, TLD, link sources, etc.)
  • searcher intent (Google's Amit Singhal stated "the same query can mean entirely different things in different countries. For example, [Côte d'Or] is a geographic region in France - but it is a large chocolate manufacturer in neighboring French-speaking Belgium")
  • other forms of search personalization (past searches, user subscriptions, frequently visited sites, etc.)
  • editorial partnerships with news companies & other universal search categories (like Google Shopping Search and the maps local onebox)
  • usage data (especially with sites they host, like YouTube)
  • content age (read up on the Query Deserves Freshness algorithm)

Look at some of the search results from Google's 2001 index and compare them to current search results to see how much Google has moved away from a raw PageRank model. Yahoo! Search's Priyank Garg also stated that they have moved away from placing so much weight on links:

All of those links might still be useful for crawl discovery, but they won’t support the ranking. That’s what we are constantly looking at in algorithms. I can tell you one thing, that over the last few years as we have been building out our search engine and incorporating lots of data, the absolute percentage contribution of links and anchor text to the natural ranking of algorithms or to the importance in our ranking algorithms has gone down somewhat.

Final Thoughts

It is not that Linkscape is a bad tool, it is just aiming to do something incredibly complex, and as long as Yahoo! Site Explorer gives us a decent free sample (and other tools let us layer data on top of Yahoo!) we can get a good idea of the approximate level of competition for free. But with Yahoo! at $12 a share, if Yahoo! gets bought out and Site Explorer goes away then Linkscape (or Majestic SEO, depending on who does a better job of innovation) might be one of the best SEO investments one can make.

Firefox Rank Checker Extension Now With Pretty Graphs

Sep 29th

A member of the SEO Book community wanted to add graphs to the Rank Checker extension. Please give Site Rank Reporter a try, and leave feedback below. I have alerted him to this thread and he is anxious for your feedback.

A couple tips...

  • you must save the Rank Checker data to CSV before importing it to the Site Rank Reporter tool.
  • you have to have at least a few days worth of data to see the benefits of the charts.

New Backlink Analysis Strategy: Compete.com Referral Analytics

Sep 21st

Compete.com quietly launched a referral analytics product as part of their advanced package ($499/month). Even as a free user you can see the top 3 results for any site, which can be used to see how reliant a site is on search. Why is % of search traffic an important statistic?

  • If search traffic (as a % of total traffic) is low (relative to other competing sites) then it could indicate that there are organic optimization opportunities that are currently being missed and/or that site has a large organic traffic stream that can be marketed to in order to help it improve any search related weakness.
  • If search traffic (as a % of total traffic) is high (relative to other competing sites) then it could indicate that the site is near its full search potential, that the site is not very engaging, and/or does not have many loyal users

Here are search stats for SEO Book. Note that Google controls a minority of the traffic to this site, which means they have limited direct influence on the revenue of this site. Some sites are closer to 90% Google, which makes it easy for Google to effectively remove them from the web!

This sort of data is important for considering the viability of a business model, the stability of a site, and what multiple a site should sell for. It can also be used when considering the CPM of an ad unit - search traffic is much more targeted and goal oriented than a person browsing a forum is.

Until everyone and their dog started looking at PageRank (and how to manipulate it) it was a rather sound way of finding the most valuable backlinks. But with the pollution of endless bought links, nepotistic links, and PageRank only being updated quarterly it is tough to glean much market data from only looking at PageRank. Tools like SEO for Firefox (especially when used on a Yahoo! backlink search) allow you to gather more data about the quality of link sources. But they all try to measure proxies for value rather than how people actually surf the web.

Microsoft BrowseRank research would use browsing data to supplement PageRank on determining relevancy. In Internet Explorer 8 (currently in beta) a person's browsing details are sent to Microsoft by default. With ~ 80% of the browser market, Microsoft does not need to use a random walk for the core of their relevancy algorithm - they know what people are actually doing, and can use usage data as a big part of their relevancy algorithms.

Using a tool like Compete.com Referral Analytics makes it far easier to poach top affiliates, discover the best ad buying locations, and replicate a competitor's best backlinks. Be forewarned that the tool only works at the domain level, so it is much better at analyzing Yahoo.com than shopping.yahoo.com.

Along with referral analytics Compete offers destination analytics, which let you know what websites people visit AFTER visiting a particular site...which should help you glean information about how sites are monetizing, what offers are working well, what sites are well referenced by another site, and what sites people go to if they can't get what they want on the current site.

At $500 a month, this tool is probably only going to be used by those who are already fairly successful rather than as an entry level tool.

New Google SERP Changes

Aug 23rd

Google has been changing the code used to display their search results a number of times over the past couple days. We recently updated SEO for Firefox and Rank Checker. Both should work as of now, and if any more SERP changes happen we will try to update the extensions as soon as possible.

If you need to update these tools you can do so within the Firefox browser by clicking on Tools in the top Firefox menu, then from the Tools menu click on Add-ons. This will pop up the Add-ons / extensions window. At the bottom of this window there is a Find Updates button you can click. That will bring in the new updates and then when you re-start the browser the extensions should be fully functional again.

Wordpress Automated Email Link Building

Jul 22nd

James from Semvironment created a plug in to automatically email webmasters you link to from within Wordpress blog posts. When he launched it, the opening post sent me an email

Hi! We linked to your website in our post: Link Builder for Wordpress - Download it Now!. Please stop by and check it out, subscribe to our blog and if you find something useful on our site or blog - we would welcome a link back anytime ... no obligation - we're happy to link to high quality websites and blogs like yours! To your continued success, [Your Name Here]

That link was broken (pointing to a revisioned archive version of the post before the URL changed), but even beyond that I sorta do not like the idea. Why? Automated communications is the enemy of relationship building. And the worst people to offend are the people you find interesting / important / influential enough to want to talk about them. You can get the person's attention just as easily by clicking the link in the post a dozen times and by commenting on their blog. And most of them would even be up for lending their time to you if you stoke their egos.

As more people start using a wider array of automated link building tools the effectiveness of automation will drop. And you can't train an automated tool to be personal. A tool like this might work in some verticals for a year or two, but if people find it effective look out for the tragedy of the commons to be heading toward your inbox some time soon!

Robots.txt Overview, Generator, & Analyzer

Jul 21st

We recently compiled a 6 page robots.txt primer and added a robots.txt generator and robots.txt analyzer to our suite of free SEO tools.

Google AdPlanner Media Buy Planning Tool

Jun 24th

Google announced AdPlanner, a tool to help ad agencies find where desired demographic audiences are active online. The WSJ highlighted how the new Google tool can help make the ad marketplace more efficient:

The Web-audience data could be combined with the ad-serving system, so that advertisers would be able to find out whether they would reach the right audience before they committed to placing an ad.

In addition to AdPlanner, Google will launch another tool that compares consumer response to ads against a control group of users who did not see the ad:

Separately, Google this week is expected to roll out a new tool aimed at showing how Web surfers respond to online ads. It will compare groups of people who are exposed to an ad with others who haven't seen it, taking into account such factors as search activity and site visitation.

Update: Search Engine Land Reviewed AdPlanner.

Pages






    Email Address
    Pick a Username
    Yes, please send me "7 Days to SEO Success" mini-course (a $57 value) for free.

    Learn More

    We value your privacy. We will not rent or sell your email address.