There is ample evidence that humans are frequently bamboozled by statistics, and interpretation of risk factors is an area in which this is most apparent. As an example, research from a number of countries showing that about a quarter of the population cannot correctly answer the question “Which is the highest risk factor: 1 in 100, 1 in 1000, or 1 in 10?” (Galesic and Garcia-Retamero, 2010).
This was just one insight during a fascinating lecture on Quantifying Uncertainty given by David Spiegelhalter, Winton Professor of the Public Understanding of Risk at the University of Cambridge. Spiegelhalter, a Bayesian statistician, began his talk with two quotes which he said were very useful in setting an appropriate perspective:
“probability does not exist” Bruno de Finetti
“all models are wrong, but some are useful” George Box
In other words, probability is not an intrinsic property of the outside world, but something we apply to it. Risks associated with a number of activities can be compared using a standardised unit the micromort, invented by Stanford University statistician Ron Howard and defined as a 1-in-a-million chance of dying. For example – how far can you travel by different means for a risk of one micromort? Answer: driving = 250 miles, cycling = 20, walking = 17, motorcycling = 6, hang-gliding = 8, scuba-diving = 5 and skiing = 0.5. Comparisons of this sort lay at the heart of David Nutt’s famous assertion that taking ecstasy has the same risk as horse-riding.
In a healthcare context (Spiegelhalter is also a senior scientist in the Medical Research Council’s Biostatistics Unit) some of the relative risks are surprising. Setting aside any risks associated with a particular illness, simply having a general anaesthetic carries a risk of 10 micromorts. The risks associated with giving birth vary considerably between countries: in the UK the risk is 80 micromorts, in the USA it is 170. In some countries it would be considerably higher.
Perhaps the most surprising statistic given was the 75 micromort cost of simply spending a night in hospital. In this instance Spiegelhalter showed his workings: there are 3735 deaths in UK hospitals each year attributed to preventable reasons. On any given night, an average of 135000 people are staying in a UK hospital, hence the risk is 75 micromorts.
Much of the rest of the lecture was given over to consideration of ways to convey risk in order to make the statistics as easily intelligible as possible. Several interesting schemes were demonstrated, including Schwartz and Woloshin’s drug facts box to communicate benefits and harms. The key feature of this model is a side-by-side comparison of the new drug against a standard reference (e.g. aspirin) with relative risks per 1000 patients indicated.
A variety of visual representations were also reviewed, including traditional bar charts and survival curves. For the latter, the trial version of an interactive website was demonstrated; you can enter you heart rate, blood pressure, cholesterol levels and whether or not you are a smoker and see the potential impact in life expectancy that would arise from tackling one or more of these criteria.
Another graphical displays of risk used a set of red and green smiley faces to indicate how many out of 100 people would be expected to have a certain condition within the next ten years.
My favourite visual representation of risk utilised a series of stick men either side of a vertical line to indicate the number of people per thousand likely to have either the illness being treated or a known side-effect of the medication being described. One side of the line represented the control group, the other the drug of interest. What I particularly liked about this format was the potential to then deduct the less numerous side of the line (be it for the medicine or for the control) from the corresponding total on the other side of the line – leaving hollow stick-men to represent the original totals and filled stick-men to emphasise the difference. In this way, for example, the lives saved by the use of statins could be clearly pictured, but also the increased risk of cataracts, liver dysfunction and myopathy associate with taking the medicine.
Spiegelhalter moved on from risk statistics themselves to reflect on ways that we can report on the quality of the studies not simply the findings. In this regard he praised the Grading of Recommendations Assessment, Development and Evaluation (GRADE) scheme used for evidence-based medicine by the Cochrane collaboration and others. As an example of less good practice he cited the IPCC climate reports where different working parties seemed to be applying different criteria.
Lessons can be learnt from economics and social science which have been quicker to identify the differences between risk, uncertainty, indeterminacy, ignorance which, a la Donald Rumsfeld, can be subdivided into known unknowns and unknown unknowns.
“There are known knowns; there are things we know that we know. There are known unknowns; that is to say, there are things that we now know we don’t know. But there are also unknown unknowns; there are things we do not know we don’t know” Donald Rumsfeld
Spiegelhalter concluded his talk by drawing analogy between risk models and guide books; some can be out of date, some are wrong, some are too simple, others too complicated. Regardless of this they can still be useful.
Leave a comment
No comments yet.