“There are three kinds of lies: lies, damned lies, and statistics,” noted the British liberal critic Charles Wentworth Dilke. Two weeks ago, we received the first four of eight lessons on how to better deal with and interpret the myriad of financial and statistical data that we encounter daily in business. All lessons are taken from two highly recommended books, “The Tiger That Isn’t” by Michael Blastland and Andrew Dilnot, and “Innumeracy” by John Allen Paulos. Today, let’s take the last four lessons that help us to better play the numbers game and become more numerate.
What’s the sample here? Most statistical numbers are not derived by counting all cases. Instead, researchers count only very few cases in a “representative sample”, which is then extrapolated to the right size of an overall population of cases. Unfortunately, such “counting light” leads to skewed or flawed results if the selected few don’t mirror the many.
In the US, a survey found that Democrat voters were less satisfied with their sex lives than Republicans. Fortunately, someone recalled that more women tend to vote Democrat than Republican, and generally tended to report less satisfaction with their sexual lives. Insofar, the sample was flawed; instead of just having the same numbers of voters for both parties, it should have contained an equal number of both male and female voters for both parties to produce a valid sample.
Lesson 5: Always check if the few that are counted in a sample mirror the many others that are not counted, and if the sample is representative or flawed or lopsided. Ask: “Which few are to be included in the sample to make it representative? And what is an appropriate sample size here?”
Do storks cause babies? In Scandinavia, it was observed that storks are more likely to be seen on the rooftops where larger families live. Therefore, one may conclude: storks cause babies. Understand that correlation —the apparent link between two separate things or phenomena— does not prove causation; in other words, just because two things seem to go together does not mean that one causes the other. When two things change at the same time, the reason may well lie in a third. In the stork example, the real reason lies probably in a third factor, house size: Larger families live in larger houses, with more roof space, which attracts storks.
Lesson 6: Beware of confusing correlation with causation. When you encounter a correlation between two factors, always check if there is a third factor that explains the changes in two things. Ask: “What else could be true of the group, the numbers, the phenomena we are interested in? What other facets do they share? What else do we know about the things involved here that might help to explain the patterns that we see?”
Risks are ever increasing, aren’t they? “New medical research: long-term use of mobile phones doubles the risks of brain tumors”. Reports on newly discovered risks or risk increases, like this widely published newspaper story in 2005, cause fear and sell papers. But how scared should you really be?
Whenever you are confronted with a reported risk increase, fight your fear and focus on the facts and numbers. First of all, the said brain tumors (known as acoustic neuroma) are not cancerous or particularly dangerous; moreover, the study found they only appeared after at least 10 years of at least moderate mobile phone use. While you now know more facts, you still don’t know the most important number: What’s the baseline risk of getting these brain tumors? In fact, it is 0.001%, or 1 in 100,000. This means that 10 years of regular mobile phone use leads to 1 extra person in 100,000 developing these non-cancerous brain tumors, or 2 in 100,000. Now decide for yourself: Is this risk increase worth reducing your mobile phone use?
Lesson 7: Be careful when news reports —or insurance agents— bombard you with new research findings and stories on risk increases, particularly if they are expressed in relative terms or in percentages. Always look for the baseline, and translate a percentage number into what it means in terms of people. Ask: “How big is this risk? What’s the baseline here?” Then, express the risk increase in human terms: “How many extra people per 100 or 1,000 might this risk affect?” Only then can you make an informed decision on whether the risk increase is really big enough to start caring about.
This figure is shocking, isn’t it? Whenever you encounter a number that is totally out of line with others, it usually means one of three things: (1) this is an unexpected and extreme result, telling us something new and extraordinary, and it makes an astounding story; (2) the number is very inaccurate and false; or (3) the number has been misinterpreted. Be suspicious when you encounter extreme results that are introduced with words such as “may reach”, “could be as high as”, “may potentially affect” — the number is probably an outlier rather than a more plausible result. So every time you see the words “may potentially affect”, add in your mind the words “but probably won’t”.
Lesson 8: Whenever you come across a surprising or shocking number that is new and different, ask: “Does the very fact that this is new and different invite caution? Do the numbers mark a paradigm shift or a rogue result? Is this the most likely or the most extreme possibility (and therefore one of the most unlikely)?”
Conclusion: Poet Carl Sandburg remarks with a twinkling eye: “If you ask your mother for one fried egg for breakfast and she gives you two fried eggs and you eat both of them, who is better in arithmetic, you or your mother?” Dealing with numbers often requires more than meets the eye. So cultivate a healthy mix of realism, skepticism and detached curiosity toward numbers you encounter.
© Dr. Detlef Reis 2010. This article was published in parallel in the Bangkok Post under the title “How big? How high? How relevant? Don’t score an own goal in the numbers game” on 8 July 2010.