There are three kinds of lies: lies, damned lies, and statistics - Disreli
We get presented with graphs and statistics every day. "Most SEOs think keywords in the title tag is an important ranking factor." "Spending on search to rise by $10b". Ever get that feeling that what you're being presented with sounds plausible, but the conclusion just doesn't make sense?
Here are a few common ways people try to pull the wool over your eyes with statistics. Some you'll be familiar with. If you've got more, add 'em to the comments :)
1. Built In Bias
The sample data supports an obvious agenda. For example, a company is hardly likely to show a graph that shows their product has produced negative results. Try to determine the bias of the person or organisation presenting the data - "what would they want me to hear"? then ask yourself: "what data are they not showing me?"
2. The Average
The media loves to state "the average", then neglect to tell you which average they are talking about.
For example, the average house price for an area could both be 500K and 200K, depending on what type of average is being used. They could be referring to either the mean, the median or the mode. They often mix these up, depending on what conclusion they want you to reach.
3. Inadequate Sample Size
20% of web designers make over $1M. That may be true if the sample size consisted of ten highest earning people in the industry, and two people just happen to have had a great year. But what if the sample size is all those who practice web design for a living? The outcome may be somewhat different.
4. Meaningless Differences
A difference is only a difference if it makes a difference. Potential employee Jill may have an IQ of 120, and potential employee Jack may have an IQ of 118, but does that really mean anything? What if Jill has an attitude problem, and Jack is a great conversationalist? Who would be the better hire?
5. Oh My God!
Al Gore loves this one. The graph that shows some astonishing change in the status quo. The impression is one of significant movement and is meant to shock an audience.
However, if the chart appears in a different context - say, over a longer time period - the rise may not look all that unusual. You often see this in stock price quotes. You could also change the measurement into smaller units, thus making any movement in the graph look even more impressive.
6. What You Infer Is Up To You
If you can't prove what you want to prove, prove something else and pretend they are the same thing. Often used in the alternative medicine industry. They may not be able to prove that their natural products cure cancer, but they can say that the plant extract has been used by some remote tribe, and they have a proven historical low incidence of cancer.
7. Post Hoc
A study found students who smoked got lower grades. The fallacy of one thing not following the other i.e. smoking doesn't cause bad grades. Frequently, other factors are left out i.e. the students who smoked also tended to be party animals. Look out for correlations that happen by chance.
8. Data Precision
Quoting specific numbers, especially including decimals points, can look authoritative. "Real estate values up 4.95%" Why would someone be so precise if they didn't know their stuff? The numbers can be wild guesses, but accuracy gives an air of authority.
General Tips For Spotting The Lies
- Ask "who says so?" Are they likely to be biased? If experts are cited, check to see if those experts actually agree with the conclusions. Often, they do not.
- Ask "How do they know"? Is the sample size really large enough, or relevant enough, to draw conclusions?
- Look To See If They Change The Subject. Look for a change between the raw data and the conclusion. Does one follow the other? For example, more reported incidences of crime do not necessarily mean there is more crime occurring.
- Ask "Does this make sense?" - are they trying to blind you with numbers? If the conclusion just sounds wrong, look for a disconnect between the data and the conclusion
If you want to delve deeper in to How To Lie With Statistics, grab the little book of the same name. It's getting a bit dated now - it was written in 1954 - but the advice and examples are great :)
Gain a Competitive Advantage Today
Your top competitors have been investing into their marketing strategy for years.
Now you can know exactly where they rank, pick off their best keywords, and track new opportunities as they emerge.
Explore the ranking profile of your competitors in Google and Bing today using SEMrush.
Enter a competing URL below to quickly gain access to their organic & paid search performance history - for free.
See where they rank & beat them!
- Comprehensive competitive data: research performance across organic search, AdWords, Bing ads, video, display ads, and more.
- Compare Across Channels: use someone's AdWords strategy to drive your SEO growth, or use their SEO strategy to invest in paid search.
- Global footprint: Tracks Google results for 120+ million keywords in many languages across 28 markets
- Historical data: since 2009, before Panda and Penguin existed, so you can look for historical penalties and other potential ranking issues.
- Risk-free: Free trial & low price.