1
march2011 35 © 2011 The Royal Statistical Society Too much or too little scepticism? In a world where we are constantly being bom- barded with lies, half-lies and statistics, it seems to me to be entirely appropriate that statisticians should be professional sceptics, people who say “show me the data!”, who follow and then test a chain of reasoning, who question the orthodox and the unorthodox alike. After all, we are trained to be aware of the many pitfalls associated with drawing valid conclusions from data. We know that correlation does not imply causation. For example, in children, reading skill is strongly cor- related with shoe size. However, we do not think that improving a child’s reading will make their feet grow larger; we quickly spot a third factor – age – strongly correlated with both reading level and shoe size in children, which gives rise to the non-causal association. We know that confound- ers abound, hampering our ability to make valid inferences. Indeed, if we think of shoe size as a disease, and reading skill as a risk factor which might cause that disease, then we would call age a confounding variable, as it is associated with both the risk factor and the disease. In this case, if we held age constant – that is, assessed the reading skills and took the shoe sizes of children at one age only, say 8 years old – we would not expect to see any correlation between the two. Also, we are conscious that many forms of bias can exist. For example, if we estimated disease incidence from self-selected samples, we might find an over-representation of cases among the respondents, in comparison with the population as a whole. Not all cases of spurious correlation, confounding or bias are as easily seen through as the examples above. Statisticians, sceptics that they are, are good at raising the possibility of hidden biases, unmeasured or even unknown confounders, casting doubt on apparently clear- cut cause-and-effect relationships. Recently a colleague asked me what I thought of evidence that extremely low-frequency electro- magnetic fields (EL EMF), such as those created by electric power lines, cause leukaemia in chil- dren. There was a proposal to build an electricity substation near her house, and her neighbours looked to her as a scientist to review the risks and advise them. It did not take me long to discover that some responsible authorities clas- sified EL EMF as presenting no risk, while others describe them as a “possible carcinogen”, and yet others recommended taking precautions to limit the extent to which children would be exposed to these risks. As well as studies which found no association, there was a large study finding evidence of an association, with the childhood leukaemia risk from living within 200 m being about double that of living more than 600 m from power lines. This conclusion was unlikely to have arisen “by chance”, but hidden biases and unmeasured confounders could not be ruled out. What did I think of the evidence for a causal link between EL EMF and childhood leukaemia? I was sceptical. That large study estimated the increased risk to be about 2-fold, compared with the more than 20-fold increase in the lung cancer risk for heavy smokers. There was a quite small total risk, an estimated 1–5 additional deaths in the UK, no clear mechanism, and the possibility that all or most of the apparent increased risk could be explained by confounding. My colleague then asked me whether I thought that governments and electric power au- thorities might want to downplay perceptions of the childhood cancer risks of EL EMF, in much the same way as tobacco companies did for so many years with the lung cancer risks of smoking. I acknowledged the possibility, but noted that the large study mentioned above had one author from the national grid company. The matter stayed in my mind: smoking, lung cancer, confounders, scepticism, statisticians. Are we too sceptical? In the 1950s three prominent statisticians (R. A. Fisher, J. Neyman and J. Berkson) challenged the evidence of R. Doll, A. B. Hill and others linking smoking and lung cancer. They pointed out pos- sible biases and potential confounders, when the evidence was already strong. We now know they were wrong. In a private letter written in 1997, the late Sir Richard Doll wrote: “I think the sceptical reaction of the medical and cancer research scientists was partly because they smoked themselves and partly because they were unaccustomed to the interpretations of epidemiologic data and tended to judge causality by Koch’s postulates. Advisors to the government were pathologically scared of caus- ing cancer phobia by undertaking any publicity about cancer, even to the extent of opposing education about the need for early diagnosis. Within government there was anxiety about the effects of reduced sales on tax income and there was certainly a desire to work with the industry rather than against it.” For the case of EL EMF and childhood leukae- mia, am I being too sceptical, or not sceptical enough? opinion In an uncertain world, how much should we doubt? Terry Speed is not sure.

Too much or too little scepticism?

Embed Size (px)

Citation preview

Page 1: Too much or too little scepticism?

march2011 35© 2011 The Royal Statistical Society

To o m u c h o r t o o l i t t l e s c e p t i c i s m ?

In a world where we are constantly being bom-barded with lies, half-lies and statistics, it seems to me to be entirely appropriate that statisticians should be professional sceptics, people who say “show me the data!”, who follow and then test a chain of reasoning, who question the orthodox and the unorthodox alike. After all, we are trained to be aware of the many pitfalls associated with drawing valid conclusions from data. We know that correlation does not imply causation. For example, in children, reading skill is strongly cor-related with shoe size. However, we do not think that improving a child’s reading will make their feet grow larger; we quickly spot a third factor – age – strongly correlated with both reading level and shoe size in children, which gives rise to the non-causal association. We know that confound-ers abound, hampering our ability to make valid inferences. Indeed, if we think of shoe size as a disease, and reading skill as a risk factor which might cause that disease, then we would call age a confounding variable, as it is associated with both the risk factor and the disease. In this case, if we held age constant – that is, assessed the reading skills and took the shoe sizes of children at one age only, say 8 years old – we would not expect to see any correlation between the two. Also, we are conscious that many forms of bias can exist. For example, if we estimated disease incidence from self-selected samples, we might find an over-representation of cases among the respondents, in comparison with the population as a whole. Not all cases of spurious correlation, confounding or bias are as easily seen through as the examples above. Statisticians, sceptics that they are, are good at raising the possibility of hidden biases, unmeasured or even unknown confounders, casting doubt on apparently clear-cut cause-and-effect relationships.

Recently a colleague asked me what I thought of evidence that extremely low-frequency electro-magnetic fields (EL EMF), such as those created

by electric power lines, cause leukaemia in chil-dren. There was a proposal to build an electricity substation near her house, and her neighbours looked to her as a scientist to review the risks and advise them. It did not take me long to discover that some responsible authorities clas-sified EL EMF as presenting no risk, while others describe them as a “possible carcinogen”, and yet others recommended taking precautions to limit the extent to which children would be exposed to these risks. As well as studies which found no association, there was a large study finding evidence of an association, with the childhood leukaemia risk from living within 200 m being about double that of living more than 600 m from power lines. This conclusion was unlikely to have arisen “by chance”, but hidden biases and unmeasured confounders could not be ruled out.

What did I think of the evidence for a causal link between EL EMF and childhood leukaemia? I was sceptical. That large study estimated the increased risk to be about 2-fold, compared with the more than 20-fold increase in the lung cancer risk for heavy smokers. There was a quite small total risk, an estimated 1–5 additional deaths in the UK, no clear mechanism, and the possibility that all or most of the apparent increased risk could be explained by confounding.

My colleague then asked me whether I thought that governments and electric power au-thorities might want to downplay perceptions of the childhood cancer risks of EL EMF, in much the same way as tobacco companies did for so many years with the lung cancer risks of smoking. I acknowledged the possibility, but noted that the large study mentioned above had one author from the national grid company. The matter stayed in my mind: smoking, lung cancer, confounders, scepticism, statisticians. Are we too sceptical? In the 1950s three prominent statisticians (R. A. Fisher, J. Neyman and J. Berkson) challenged the evidence of R. Doll, A. B. Hill and others linking

smoking and lung cancer. They pointed out pos-sible biases and potential confounders, when the evidence was already strong. We now know they were wrong. In a private letter written in 1997, the late Sir Richard Doll wrote:

“I think the sceptical reaction of the medical and cancer research scientists was partly because they smoked themselves and partly because they were unaccustomed to the interpretations of epidemiologic data and tended to judge causality by Koch’s postulates. Advisors to the government were pathologically scared of caus-ing cancer phobia by undertaking any publicity about cancer, even to the extent of opposing education about the need for early diagnosis. Within government there was anxiety about the effects of reduced sales on tax income and there was certainly a desire to work with the industry rather than against it.”

For the case of EL EMF and childhood leukae-mia, am I being too sceptical, or not sceptical enough?

opinion

In an uncertain world, how much should we doubt? Terry Speed is not sure.