The Economist, 04 November 2004.
A statistically based study claims that many more Iraqis have died in the conflict than previous estimates indicated
THE American armed forces have long stated that they do not keep track of how many people have been killed in the current conflict in Iraq and, furthermore, that determining such a number is impossible. Not everybody agrees. Adding up the number of civilians reported killed in confirmed press accounts yields a figure of around 15,000. But even that is likely to be an underestimate, for not every death gets reported. The question is, how much of an underestimate?
A study published on October 29th in the Lancet, a British medical journal, suggests the death toll is quite a lot higher than the newspaper reports suggest. The centre of its estimated range of death tolls—the most probable number according to the data collected and the statistics used—is almost 100,000. And even though the limits of that range are very wide, from 8,000 to 194,000, the study concludes with 90% certainty that more than 40,000 Iraqis have died.
Numbers, numbers, numbers
This is an extraordinary claim, and so requires extraordinary evidence. Is the methodology used by Les Roberts of the Johns Hopkins University School of Public Health, in Baltimore, and his colleagues, sound enough for reliable conclusions to be drawn from it?
The bedrock on which the study is founded is the same as that on which opinion polls are built: random sampling. Selecting even a small number of individuals randomly from a large population allows you to say things about the whole population. Think of a jar containing a million marbles, half of them red and half blue. Choose even 100 of these marbles at random and it is very, very unlikely that all of them would be red. In fact, the results would be very close to 50 of each colour.
The best sort of random sampling is one that picks individuals out directly. This is not possible in Iraq because no reliable census data exist. For this reason, Dr Roberts used a technique called clustering, which has been employed extensively in other situations where census data are lacking, such as studying infectious disease in poor countries.
Clustering works by picking out a number of neighbourhoods at random—33 in this case—and then surveying all the individuals in that neighbourhood. The neighbourhoods were picked by choosing towns in Iraq at random (the chance that a town would be picked was proportional to its population) and then, in a given town, using GPS—the global positioning system—to select a neighbourhood at random within the town. Starting from the GPS-selected grid reference, the researchers then visited the nearest 30 households.
In each household, the interviewers (all Iraqis fluent in English as well as Arabic) asked about births and deaths that had occurred since January 1st 2002 among people who had lived in the house for more than two months. They also recorded the sexes and ages of people now living in the house. If a death was reported, they recorded the date, cause and circumstances. Their deductions about the number of deaths caused by the war were then made by comparing the aggregate death rates before and after March 18th 2003.
They interviewed a total of 7,868 people in 988 households. But the relevant sample size for many purposes—for instance, measuring the uncertainty of the analysis—is 33, the number of clusters. That is because the data from individuals within a given cluster are highly correlated. Statistically, 33 is a relatively small sample (though it is the best that could be obtained by a small number of investigators in a country at war). That is the reason for the large range around the central value of 98,000, and is one reason why that figure might be wrong. (Though if this is the case, the true value is as likely to be larger than 98,000 as it is to be smaller.) It does not, however, mean, as some commentators have argued in response to this study, that figures of 8,000 or 194,000 are as likely as one of 98,000. Quite the contrary. The farther one goes from 98,000, the less likely the figure is.
The second reason the figure might be wrong is if there are mistakes in the analysis, and the whole exercise is thus unreliable. Nan Laird, a professor of biostatistics at the Harvard School of Public Health, who was not involved with the study, says that she believes both the analysis and the data-gathering techniques used by Dr Roberts to be sound. She points out the possibility of “recall bias”—people may have reported more deaths more recently because they did not recall earlier ones. However, because most people do not forget about the death of a family member, she thinks that this effect, if present, would be small. Arthur Dempster, also a professor of statistics at Harvard, though in a different department from Dr Laird, agrees that the methodology in both design and analysis is at the standard professional level. However, he raises the concern that because violence can be very localised, a sample of 33 clusters really might be too small to be representative.
This concern is highlighted by the case of one cluster which, as the luck of the draw had it, ended up being in the war-torn city of Fallujah. This cluster had many more deaths, and many more violent deaths, than any of the others. For this reason, the researchers omitted it from their analysis—the estimate of 98,000 was made without including the Fallujah data. If it had been included, that estimate would have been significantly higher.
The Fallujah data-point highlights how the variable distribution of deaths in a war can make it difficult to make estimates. But Scott Zeger, the head of the department of biostatistics at Johns Hopkins, who performed the statistical analysis in the study, points out that clustered sampling is the rule rather than the exception in public-health studies, and that the patterns of deaths caused by epidemics are also very variable by location.
The study can be both lauded and criticised for the fact that it takes into account a general rise in deaths, and not just that directly caused by violence. Of the increase in deaths (omitting Fallujah) reported by the study, roughly 60% is due directly to violence, while the rest is due to a slight increase in accidents, disease and infant mortality. However, these numbers should be taken with a grain of salt because the more detailed the data—on causes of death, for instance, rather than death as a whole—the less statistical significance can be ascribed to them.
So the discrepancy between the Lancet estimate and the aggregated press reports is not as large as it seems at first. The Lancet figure implies that 60,000 people have been killed by violence, including insurgents, while the aggregated press reports give a figure of 15,000, counting only civilians. Nonetheless, Dr Roberts points out that press reports are a “passive-surveillance system”. Reporters do not actively go out to many random areas and see if anyone has been killed in a violent attack, but wait for reports to come in. And, Dr Roberts says, passive-surveillance systems tend to undercount mortality. For instance, when he was head of health policy for the International Rescue Committee in the Congo, in 2001, he found that only 7% of meningitis deaths in an outbreak were recorded by the IRC's passive system.
The study is not perfect. But then it does not claim to be. The way forward is to duplicate the Lancet study independently, and at a larger scale. Josef Stalin once claimed that a single death is a tragedy, but a million deaths a mere statistic. Such cynicism should not be allowed to prevail, especially in a conflict in which many more lives are at stake. Iraq seems to be a case where more statistics are sorely needed.