Statistical evidence

Evidence comprised of or pertaining to statistical information – especially data collected by specific methods which have been found to be reliable – information which is or has been quantified, systematically collected, and ordered to convey the information meaningfully. For example, the voting records of senators belonging to a particular party, the percentage of car crashes involving intoxication, changes in patterns of immigration over a specified period of time, the proportion of teenage unwed mothers who have been brought up in single-parent – rather than two-parent – homes, or the education and income levels of various ethnic and racial groups in the population. 

Concerns involving statistical evidence include the selective citing of statistics: another version of suppressed evidence. The reliability of statistical results depends, amongst other things, upon the size and quality of the sample used. A ‘rule of thumb’ is that the smaller the sample, the weaker should be any conclusions drawn. Even a large sample might produce misleading results: to yield a greater reliability the sample must be chosen randomly. Is the sample representative of the target population? Is the sample biased? Who conducted the research?

Statistical studies may be subject to a non-random response rate – meaning people who choose to return completed survey forms may hold opinions which don’t accurately reflect the group of people sent survey forms. Likewise, people who volunteer for a scientific study, for example, a health-focused study concerning heart disease, may already differ from the population with the condition by acting in ways which do not parallel that population. Perhaps they are more conscious of actions which lessen or moderate the condition and are more likely to act on this knowledge (appropriate diet and exercise).

The manner in which questions are phrased in opinion surveys has a large effect upon the answers supplied by subjects, and therefore we should maintain a certain suspicion for statistical claims about the public’s opinions on various subjects, and more so when it is a controversial or contentious issue. Without knowing how a proposition was put to participants we have little basis to assess the ‘exact-sounding’ results. There are many different ways to present matters of policy: are we in favour of seeking to lift vulnerable children out of poverty or against a further government failure of wasting taxpayers’ money on welfare cheats? Reality is much more complex than either of these caricatures suggests.

Crime statistics derive from incident reports to law enforcement or justice authorities or arrests, not actual criminal acts – meaning that where there is under-reporting or non-reporting the accuracy of the statistical information will be diminished. Economic statistics are dependent upon the quality and completeness of data reported to government agencies, so in as much as reporters are hostile towards or apathetic about having to make such reports or simply dilatory, the reliability of the resulting information suffers.

Accurate statistical information may be used to mislead by leaving out proper context. So if in an election campaign a party indicates that it will increase spending on a particular government budget item popular amongst voters by 1.2 per cent in the coming fiscal year, marrying this information with the fact that inflation is then running at 4.5 per cent annually is something which would be necessary to see that in ‘real dollars’ terms spending would not increase, but rather be cut by such a proposal.

If we are interested in knowing whether spending on a particular area, say in high school-level education, is increasing or decreasing what is the comparison information we need to bring to bear? For a statistical comparison to be meaningful, the quantities being compared must be appropriate. For example, a company advertises that the consumers of its products get fewer teeth cavities, let’s say 35% fewer, and implies this is a (good) reason to buy its product. The question is compared with what, or in this case, whom? If the comparison is with consumers who use competitors’ products, assuming there is no other more likely reason for the greater part of the difference, then the statistic may have some worth in the company’s pitch to buy its product. If the company was instead comparing its customers with those who don’t brush their teeth or practice any kind of dental health regime, the comparison fails and the company’s statements cannot support the claim.

Drop-out rate: a company is promoting a training course of some kind – it might be in stock market analysis and investment strategy or learning to play a musical instrument. It reports that 95% of those who complete the course have a particular level of success, for example, in the stock market course successful participants increased the value of their investment portfolio by at least 23% in the twelve months following the end of the six-week course. What was the drop-out rate? If most people leave because the course becomes too frustrating or too challenging for them, the actual success rate for the training course is much lower than the company promises. It may simply be that those people for whom the difficulties in learning the subject or skill were too great were eliminated, while the remaining people already had a facility for the course material and needed little training.

The right statistic: Politicians or political candidates may make statements such as their particular party’s administration has created x number of jobs, say, three million, during its term (or a specified period of time), usually with the implication that electors should vote for the advocate or a member of the advocate’s party at the next electoral contest. However, what is likely to interest voters is whether there has been a net increase in employment during the time frame, and the quality of the jobs available. How many jobs were lost during the period under review – there may be a substantial deficit overall – and of course what is the nature and type of employment opportunities which have been created? For example, have mid-level paying jobs reduced or largely disappeared in some areas with a significant proportion of new jobs low-paying or ‘working poor’ positions.

Uniform statistics: It is reported that the number of journals available to students at a University’s library is fewer than fifteen-years ago (48 down from 56), while spending on journals has increased by twenty per cent. What is the spending situation beyond adjustment for inflation? Are all of the journals previously kept still extant and is the quality and relevance of the currently-accessible journals improved even if numbers have fallen?

How were the statistics presented obtained?

Has the proponent advocated a particular conclusion, yet provided statistics which appear to support a different conclusion (while failing to sustain the favoured conclusion)?

What’s missing in terms of good or relevant comparisons and context?

Has an estimate or educated guess been presented in the form of an ‘artificial’, but factual-sounding, statistic? (especially in cases where ascertaining the particular quantity is extremely difficult or next to impossible)

Statistical generalization

A generalization that the same proportion of the population as the sample inspected will have the property or properties of interest.

(see also: statistical significance, statistical analysis, Studies – Scientific or Empirical, statistic, sample, Law of large numbers, Margin of error, sample bias, survey)