What Numbers Can—and Can’t—Tell Us About the Pandemic

Data scientists identify common COVID-19 statistical pitfalls
14-Jul-2020 6:20 PM EDT, by New York University

Newswise — Currently, we are confronted, around the clock, with troubling data as reporters, public health experts, and elected officials seek to understand and describe the path and impact of COVID-19—poring over rates of infection, hospital admission, and death, to name just a few key indicators. 

With so many numbers to digest, it can be challenging to separate statistics that may mislead from those that illuminate—something that has complicated the decision-making of government officials, according to recent news accounts. And while the widespread suspicion that numbers can be manipulated to support almost any conclusion predates the pandemic, partisanship around the response to the virus has further undermined Americans’ trust in COVID-19 data, according to a recent Pew Research Center survey

But statistics are, of course, vital to understanding the current crisis, as well as other complex problems such as poverty, economic downturns, and climate change, and so researchers stress the importance of learning to distinguish what’s useful from what may be junk. 

“We suspect that statistics may be wrong, that people who use statistics may be ‘lying’—trying to manipulate us by using numbers to somehow distort the truth,” writes sociologist Joel Best in his book Damned Lies and Statistics. But, he explains, “[t]he solution to the problem of bad statistics is not to ignore all statistics, or to assume that every number is false. Some statistics are bad, but others are pretty good, and we need statistics—good statistics—to talk sensibly about social problems.” 

To help enhance our own statistical literacy as the pandemic continues, NYU News spoke with Andrew Gordon Wilson and Jonathan Niles-Weed, assistant professors at NYU’s Center for Data Science and Courant Institute of Mathematical Sciences, who outlined some principles to keep in mind when evaluating figures cited in the news. 

Their tips appear below, but both caution that training in data science alone isn’t enough to equip leaders to make perfect decisions.

“Many people—statisticians included—think that every problem can be solved by getting better data,” says Niles-Weed. “But even with perfect information, beating COVID will require politicians and public health experts to weigh very different considerations and make hard choices despite uncertainty. Data can help, but setting good policy also requires incorporating values and goals.” 

Be certain about the uncertainty in the data.

“Many of the facts and figures we see come with big unstated error bars,” warns Wilson. “Suppose the only person in a village tested for coronavirus tests positive. It could be reported that the incidence rate in that region is 100%. You might say, ‘Surely they need to test more people?’ But how many people should we test for an accurate incidence estimate? Ten people, 100 people, 10,000 people? What’s a reasonable sample size? And do we only test symptomatic people? What fraction of the population is asymptomatic? What constitutes ‘accurate’? Similarly, models predicting quantities such as incidence rate take many variables as input, such as case fatality rate. These inputs similarly have big uncertainty attached to them. We should be conscious of uncertainty in parsing numbers we see in the media—the point predictions, without reasonable estimates of the error bars, are often meaningless.” 

Separate real trends from random occurrences

“Random variation in data can easily be mistaken for a genuine trend,” says Niles-Weed. “Even if the underlying situation is static, data may change from day to day because of random noise. For example, if a state’s newly confirmed cases are particularly high during a given week and lower the next, it’s easy to interpret this as meaningful: perhaps the high caseload in one week made citizens more cautious, leading to a drop in cases the next week after behaviors changed. But it's just as likely that the first week was just a random outlier, and that nothing at all changed. By contrast, sustained day-over-day increases or decreases can indicate real trends.” 

Know what different probabilities can tell you—and what they can't. 

“It’s easy to confuse conditional probabilities, which is significant during a pandemic because it can lead to a misreading of testing data,” notes Wilson. “For example, in taking a test for coronavirus, we care about the probability that we have coronavirus given that we test positive—and not the probability that we test positive given that we have coronavirus."

We have to carefully interpret what a probability is telling us. For example, the sensitivity of a test tells us the probability that we test positive, given that we have the condition. Similarly, another measure—the specificity—is the probability of a negative result if we don't have the condition. If a test has a high sensitivity, and is thus reported as highly accurate, it does not mean testing positive means we are likely to have coronavirus, especially if the general rate of coronavirus in the population is low. Similarly, if the general rate of coronavirus is high, a negative test result may have high probability of being a false negative, even when the test has high specificity.” 

Is your sample biased?

“While a truly random sample can give precise information about the whole population, bias can arise if some people are more likely to be included than others,” explains Niles-Weed. “For example, if a research team performs antibody tests on a random set of people walking down a city street, they will invariably miss those too sick to leave their beds. Data collected in this way can fail to be representative when extended to the whole population.” 

What information is missing?          

“Many claims are factually correct but misleading due to crucial missing information,” says Wilson. “For example, it may be correct to say a majority of confirmed cases in a region are Asian, but if only a very small number of people had tested positive, that may not be a meaningful finding. Similarly, there are many correlations that can easily be explained away by missing causal factors. It was reported at one time that healthcare workers in New York have a slightly lower incidence of coronavirus than the general population. Does that mean social distancing is ineffective, since these workers will be more exposed to infected people? If we condition on the fact that healthcare workers are trained to be vigilant in mask wearing, hand-washing, distancing, and sanitization, it likely means the exact opposite!”


Register for reporter access to contact details

Damned Lies and Statistics

Filters close

Showing results

110 of 5419
Released: 15-Apr-2021 4:10 PM EDT
Penn Study Suggests Those Who Had COVID-19 May Only Need One Vaccine Dose
Perelman School of Medicine at the University of Pennsylvania

New findings from Penn suggest that people who have recovered from COVID-19 may only need a single mRNA vaccine dose. However, those who did not have COVID-19 did not have a full immune response until after a second vaccine dose, reinforcing the importance of completing the two recommended doses.

Released: 15-Apr-2021 4:00 PM EDT
June 2021 Issue of AJPH Comprises the Effects of COVID-19 on Drug Overdoses, E-cigarette Use, and Public Health Measures and Strategies
American Public Health Association (APHA)

June 2021 AJPH Issue highlights COVID-19 concerns in relation to fatal drug overdoses, drops in youth e-cigarette use, importance of public health measures, and strategies to protect correctional staff.

Newswise: 262150_web.jpg
Released: 15-Apr-2021 3:20 PM EDT
COVID-19 reduces access to opioid dependency treatment for new patients
Princeton University

COVID-19 has been associated with increases in opioid overdose deaths, which may be in part because the pandemic limited access to buprenorphine, a treatment used for opioid dependency, according to a new study led by Princeton University researchers.

Newswise: UGA to Establish National NIH-funded Center to Fight Flu
Released: 15-Apr-2021 2:45 PM EDT
UGA to Establish National NIH-funded Center to Fight Flu
University of Georgia

The National Institutes of Health has awarded the University of Georgia a contract to establish the Center for Influenza Disease and Emergence Research (CIDER). The contract will provide $1 million in first-year funding and is expected to be supported by the National Institute of Allergy and Infectious Diseases (NIAID), part of NIH, for seven years and up to approximately $92 million.

Released: 15-Apr-2021 2:15 PM EDT
Meatpacking plants increased COVID-19 cases in US counties
University of California, Davis

An estimated 334,000 COVID-19 cases are attributable to meatpacking plants, resulting in $11.2 billion in economic damage, according to a new study led by a researcher at the University of California, Davis.

Newswise: 262052_web.jpg
Released: 15-Apr-2021 1:55 PM EDT
How to build a city that prioritizes public health
Colorado State University

Most people by now have memorized the public health guidelines meant to help minimize transmission of COVID-19: wash your hands, wear a mask, keep six feet apart from others. That part is easy.

Released: 15-Apr-2021 1:45 PM EDT
Wake Forest School of Medicine Begins Study to Test New Mask for Healthcare Workers
Wake Forest Baptist Health

Open Standard Industries, Inc. (OSI), manufacturer of the OSR-M1 non-valved reusable elastomeric face mask, is pleased to formally announce the launch of its first Institutional Review Board (IRB)-approved user feasibility study. The trial is being led by the departments of Biomedical Engineering and Infectious Diseases at Wake Forest School of Medicine, part of Wake Forest Baptist Health. Recruitment in the study is underway, and enrollment is expected to be completed by May 28, 2021.

Newswise: Major clinical trial to test Pfizer-BioNTech COVID-19 vaccine opens for enrollment at UTHealth in Houston
Released: 15-Apr-2021 1:45 PM EDT
Major clinical trial to test Pfizer-BioNTech COVID-19 vaccine opens for enrollment at UTHealth in Houston
University of Texas Health Science Center at Houston

A large national clinical trial to evaluate the Pfizer-BioNTech COVID-19 vaccine for safety and efficacy in pregnant women is now open for enrollment at The University of Texas Health Science Center at Houston (UTHealth).

Released: 15-Apr-2021 1:15 PM EDT
For veterans, a hidden side effect of COVID: Feelings of personal growth
Yale University

The U.S. military veteran population is known to have abnormally high rates of suicide, so health officials have been concerned that the COVID-19 pandemic might elevate risk of psychiatric disorders, particularly among those suffering from post-traumatic stress and related disorders.

Newswise: 262026_web.jpg
Released: 15-Apr-2021 12:55 PM EDT
Significant spread of all coronavirus variants tracked in Houston area

In late 2020, several concerning SARS-CoV-2 variants emerged globally. They are believed to be more easily transmissible, and there is concern that some may reduce the effectiveness of antibody treatments and vaccines.

Showing results

110 of 5419