Since Ayima launched in 2007, we've been crawling the web and building our own independent backlink data. Starting off with just a few servers running in our Directory of Technology's bedroom cupboard, we now have over 130 high-spec servers hosted across 2 in-house server rooms and 1 datacenter, using a similar storage platform as Yahoo's former index.
Crawling the entire web still isn't easy (or cheap) though, which is why very few data providers exist even today. Each provider makes compromises (even Google does in some ways), in order to keep their data as accurate and useful as possible for their users. The compromises differ between providers though, some go for sheer index size whilst others aim for freshness and accuracy. Which is best for you?
We need a website to analyze first of all, something that we can't accidentally "out". Search Engine Land is the first that came to mind, very unlikely to have many spam links or paid link activity.
So let's start off with the easy bit - who has the biggest result set for SEL?
The chart above shows MajesticSEO as the clear winner, followed by a very respectable result for Ahrefs. Does size matter though? Certainly not at this stage, as we only really care about links which actually exist. The SEOGadget post tried to clean the results using a basic desktop crawler, to see which results returned a "200" (OK) HTTP Status Code. Here's what we get back after checking for live linking pages:
Ouch! So MajesticSEO's "Fresh" index has the distinct smell of decay, whilst Mozscape and Ayima V2 show the freshest data (by percentage). Ahrefs has a sizeable decay like MajesticSEO, but still shows the most links overall in terms of live linking pages. Now the problem with stopping at this level, is that it's much more likely that a link disappears from a page, than the page itself disappearing. Think about short-term event sponsors, 404 pages that return a 200, blog posts falling off the homepage, spam comments being moderated etc. So our "Tenacious Tim" got his crawler out, to check which links actually exist on the live pages:
Ahrefs wins the live links contest, finding 84,496 more live links than MajesticSEO and 513,733 more live links than SEOmoz's Mozscape! I still wouldn't use Ahrefs for comparing competitors or estimating the link authority needed to compete in a sector though. Not all links are created equal, with Ahrefs showing both the rank-improving links and the crappy spam. I would definitely use Ahrefs as my main data source for "Link Cleanup" tasks, giving me a good balance of accuracy and crawl depth. Mozscape and Ayima V2 filter out the bad pages and unnecessarily deep sites by design, in order to improve their data accuracy and showing the links that count. But when you need to know where the bad PageRank zero/null links are, Ahrefs wins the game.
So we've covered the best data for "mentions", the best data for "link cleanup", now how about the best for competitor comparison and market analysis? The chart below shows an even more granular filter, removing dead links, filtering by unique Class C IP blocks and removing anything below a PageRank 1. By using Google's PageRank data, we can filter the links from pages that hold no value or that have been penalized in the past. Whilst some link data providers do offer their own alternative to PageRank scores (most likely based on the original Google patent), these cannot tell whether Google has hit a site for selling links or for other naughty tactics.
Whilst Ahrefs and MajesticSEO hit the top spots, the amount of processing power needed to clean their data to the point of being useful, makes them untenable for most people. I would therefore personally only use Ayima V2 or Mozscape for comparing websites and analyzing market potential. Ayima V2 isn't available to the public quite yet, so let's give this win to Mozscape.
So in summary
Ahrefs - Use for link cleanup
MajesticSEO - Use for mentions monitoring
Mozscape - Use for accurate competitor/market analysis
Juicy Data Giveaway
One of the best parts of having your own index, is being able to create cool custom reports. For example, here's how the big SEO websites compare against each other:
"Index Rank" is a ranking based on who has the most value-passing Unique Class C IP links across our entire index. The league table is quite similar to HitWise's list of the top traffic websites, but we're looking at the top link authorities.
Want to do something cool with the data? Here's an Excel spreadsheet with the Top 10,000 websites in our index, sorted by authority: Top 10,000 Authority Websites.
Rob Kerry is the co-founder of Ayima, a global SEO Consultancy started in 2007 by the former in-house team of an online gaming company. Ayima now employs over 100 people on 3 continents and Rob has recently founded the new Ayima Labs division as Director of R&D.
As the co-founder of an SEO Consultancy, my biggest hurdle in business is finding more staff. Clients are lining up at our door, we have no trouble there, it's finding the staff to work with them that becomes the issue. This may not sound like the worst dilemma for a business to face, especially during the current global economic decline, but the causation is a matter of great concern to me as both an SEO and a businessman.
Ayima's company structure is such that only highly skilled SEOs make it through to our interview stage and yet even then, less than 5% meet our skill requirements. This isn't me being picky, misjudging characters or sourcing bad candidates - this is a knowledge pandemic that is spreading through our industry. We've started apprenticeship programs to teach eager candidates from the ground up, but this can take several years to generate the finished article.
After looking back at our past 30 interview candidates, my opinion for the reason behind this issue may not be a popular one. I believe that celebrity SEOs, brands and blogs are feeding a generation of untested and poorly trained search marketers, who pass themselves off as SEO experts. I will of course explain my positioning…
The Pander Update
Some high profile SEO bloggers recently ceased client work and personal projects, in order to appear impartial and trustworthy to their community. This makes sense at first, after-all, who wants to use a link building tool operated by someone working for one of your client's competitors? It does however bring to light 2 much larger issues;
1) a reliance on tertiary information for SEO analysis, and
2) a reliance on search engineers to provide fresh and exclusive information/data.
Some SEO information sites may argue that they have access to the Web Analytics accounts of their partners and that they do study index changes, but nothing replaces the value of following a handful of websites every single day of the year. An absence of "boots on the ground" leads to misinformation and a distancing from the SEO practices and concerns that really matter. This in turn results in an information churn which newbies to the industry naturally perceive as important.
Moving away from servicing clients or running in-house/affiliate projects also causes a financial flux. Revenue no longer relies on problem solving, but on juicy search engine spoilers and interviews. Search Engines are businesses too though and it's in their best interest to only reward and recommend the publishers/communities that tow their line. A once edgy and eager SEO consultancy must therefore transition into a best practice, almost vanilla, publisher in order to pander to the whims of over-eager search reps.
How do we expect the next generation of SEO consultants to analyse a website and its industry competitors, when all they've read about is how evil paid links are and how to tweak Google Analytics?
I could directly link the viewpoints and understandings of some recent SEO candidates back to a single SEO community, word for word. They would be horrified to see the kind of broken and malformed SEOs that their community has produced.
OMG, Check Out My Klout
It's true that social media metrics will become important factors for SEO in the future, but this certainly does not negate the need for a solid technical understanding of SEO. Getting 50 retweets and 20 +1's for a cute cat viral is the work of a 12 year old schoolgirl, not an SEO. If you can't understand the HTML mark-up of a page and how on-page elements influence a search engine, pick up a HTML/SEO book from 2001 and get reading. If you don't know how to optimise site structure and internal linking, read a book on how the web works or even a "UNIX for Dummies" manual. If you're unable to completely map out a competitor website's linking practices, placement and sources, set up a test site and start finding out how people buy/sell/barter/blag/bate for links.
You may be thinking at this point, "Rob, I already know this - why are you telling me?". Well, the sad fact is that many SEOs, with several years of experience at major and minor agencies, fail to show any understanding of these basic SEO building blocks. There are SEOs who can't identify the H1 on a page and that seriously consider "Wordle" and "Link Diagnosis" as business-class SEO tools. It used to be the case that candidates would read Aaron Wall's SEO Book or Dan Thies' big fat Search Engine Marketing Kit from cover-to-cover before even contemplating applying for an entry level SEO role. These days, major agencies are hiring people who simply say that "Content is King" and "Paid Links are Evil", they have at least 50 Twitter followers of course.
"Certified SEO" is NOT the answer
In most other professional industries, the answer would be simple - regulate and certify. This simply does not work for SEO though. I die a little, each time I see a "Certified SEO" proclamation on a résumé, with their examining board consisting of a dusty old SEO company, online questionnaire or a snake-oil salesman. A complete SEO knowledgebase cannot be taught or controlled by a single company or organisation. No one in their right mind would use Google's guide to SEO as their only source of knowledge for instance, just as no self-respecting Paid Search padawan would allow Google to set-up their PPC campaigns. Google's only interest is Google, not you. Popular SEO communities and training providers have their own agendas and opinions too.
I do however concede that some learning should be standardised, such as scientifically proven or verified ranking factors. Just the facts, no opinions, persuasions or ethical stances.
My Plea To You, The Industry
I plea to you, my fellow SEOs, to help fix this mess that we're in. Mentor young marketers, but let them make up their own minds. Put pressure on SEO communities to concentrate on facts/data and not to be scared by controversy or those with hidden agendas. Promote apprenticeship schemes in your company, so that SEOs learn on the job and not via a website. Encourage people to test ideas, rather than blindly believing the SEO teachings of industry celebs and strangers.
An experienced SEO with, what I perceive to be basic skills, isn't too much to ask for is it?