A Field Guide to Identifying Dubious Data

“Stat-Spotting: A Field Guide to Identifying Dubious Data” by Joel Best (2008) Section A: Spotting Questionable Numbers (pp. 1-6) ● “The statistics you don’t compile never lie” - Stephen Colbert ● A lack of being able to fully understand and visualize large numbers like millions, billions, and trillions leads to certain statistics to seem more daunting and common then they actually are Section B: Background (pp. 6-13) ● Statistical Benchmarks ○ Each year, about 4 million babies are born in the US, this is a good benchmark to remember because one can estimate that there are about 4 million kids at any age between 1-18 (so 72 million) ○ In 2011, 46.6% of the US died from cancer or heart disease ○ “Statistics about race and ethnicity are complicated because these categories have no precise meaning, due to mixed-race individuals and self claimed race/ethnicity on things like the Census ■ But, we can assume that African Americans account for ⅛ of the population at any given time for any given statistic ● Severity and Frequency ○ “Keep in mind one rule of thumb: in general, the worse things are, the less common they are” (11) ○ “Most social problems display this pattern: there are lots of less serious cases, and relatively few very serious ones” (11) ■ Take the idea of college underage drinking, there are examples in which there are fatalities and the media often portrays them as being serious issues and occuring in colleges across the nation, in reality, a very, very small percentage of those who drink underage in college will die Section C: Blunders (pp.14-26) ● The slippery decimal point ○ Moving a decimal point either way can mean 10 times more or less ○ “Factoid” vs. “Fictoid” ● Botched translations ○ When an individual recounts a statistic but represents it incorrectly ● Misleading graphs ○ “A snazzy graph isn't necessarily an accurate graph” (22) ● Careless calculations ○ Often the media only reports the final figure of the statistic, and not the numbers that went into its creation ■ Look for numbers that seem hard to produce: they can often be faulty or questionable statistics Section D: Sources - Who is Counted and Why? (pp.27-41) ● A good question to ask when seeing a statistic is “who produced this?” ● “The statistics that we consume are produced and distributed by a variety of different people, and those people may have very different agendas” (28) ○ The statistics that we are most exposed to have been selected, out of all the counts and numbers they found that one the most interesting, why? ● Big round numbers ○ Easy to remember and even shocking numbers ○ The experts see this as a problem and want their statistic to reflect it ● Hyperbole ○ “The greatest”, “the largest”, “the history-making” ● Shocking claims ○ Statistics like this make us scared of the world in a way ● Naming the Problem ○ It is hard to discuss a social problem without giving it a name ■ i.e. “binge drinking” ○ A chosen name can enact even stronger reactions to the statistic Section E: Definitions: What did they count? ● In general, advocates prefer to define social issues broader than they are: ○ They claim that they are drawing attention to issues and people that are unrepresented ○ Broader definitions allow for more people to be counted ● Definitions are constantly changing Section F: Measurements: How do they count? ● Unusual units of analysis can impact the answer/count ● Loaded questions Section G: Packaging: What are they telling us? ● The sources that create statistics are often competing for the attention of the media outlets and consumers ● Proportions draw attention well for topics that may not be as drastic ● Misleading samples instead of accurate counts ● The notion of statistical milestones ○ They need to be shared/ are new information ● Calling something an epidemic increases the worry/fear surrounding it Section H: Rhetoric: What do they want us to think? ● Using short term rates to correlate with long term issues is problematic ● Labelling a statistic occurrence as a myth tremendously discredits it Section I: Debates, What if they disagree? ● Casualty debates → the idea that it cannot be directly said that cigarettes cause cancer even though that is statistically correlated ● Equality debates → issues that often require statistics to back their arguments ● Policy debates → policy issues that disagree about statistics and whether or not they are true Section J: Summary, Common Signs of Dubious Data ● Background: ○ Numbers seen inconsistent with benchmark figures (B.1) ○ Severe examples are used to illustrate supposedly common problem (B.2) ● Blunders: ○ Numbers that seem too high or too low may be caused by a misplaced decimal point (C.1) ○ Botched translations convert statistics into simpler but incorrect language (C.3) ○ Misleading graphs distort the reader’s visual impression of data (C.3) ○ Errors in strings of calculations affect the final figures (C.4) ● Sources: ○ Big round numbers may be a sign of guessing (D.1) ○ Hyperbole (“the biggest”, the “worst”) may reveal exaggerations (D.2) ○ Claims that seem unbelievable shocking may indeed be unbelievable (D.3) ○ A problem is given a distributing name, calculated to arouse concern (D.4) ● Definitions: ○ Broad definitions lead to big numbers (E.1) ○ Expanding a problem's definition makes the problem seem larger (E.2) ○ Changing a problems’s definition distorts the measure of change (E.3) ○ A problem’s definition may exclude less disturbing cases (E.4) ● Measurements: ○ New statistics invite the question, how was this measure created? (F.1) ○ Unusual units of analysis can lead to questionable conclusions (F.2) ○ Changes in measurements may affect the resulting statistics (F.4) ○ Competing methods of measurement may produce different results (F.5) ● Packaging: ○ Numbers are presented in the most impressive format (percentages for the most common problems, absolute numbers for those less common (G.1) ○ Generalizations may be based on a biased or misleading sample (G.2) ○ Time frames are chosen to emphasize a particular trend (G.3) ○ An odd base is used to calculate percentages (G.4) ○ The number involves a selective comparison (looking only at those cases most likely to be affected) (G.5) ○ A claim reports that some statistical milestone has been passed (G.6) ○ The word average may refer to either the mean or the median (G.7) ○ Apparent epidemics may be caused by problems receiving closer attention than before (G.8) ○ Correlation is implied as a proof of causation (G.9) ○ Dramatic discoveries may prove incorrect (G.10) ● Rhetoric: ○ Debates in which one side talks about long-term problems, while their opponents counter with statistics about the short term (H.1) ○ warning s that long-term trends are about to be reversed (H.2) ○ Claims designating myths (H.3) ○ Dramatic numbers that seem hard to calculate (H.4) ● Debates: ○ Rival explanations identify different causes of the problem (I.1) ○ Opposing sides disagree about the nature of equality (I.2) ○ Advocates debate policy choices (I.3) Section K: Better Data, some characteristics 1. Better statistics come with data/information about the numbers they used to produce them 2. Better statistics seem to be subject to competing pressures 3. Better statistics tend to use consistent measures 4. Better statistics tend to be public (and so is the data)