Election polls are 95% confident but only 60% accurate, Berkeley Haas study finds

Newswise — How confident should you be in election polls? Not nearly as confident as the pollsters claim, according to a new Berkeley Haas study.

 Most election polls report a 95% confidence level. Yet an analysis of 1,400 polls from 11 election cycles found that the outcome lands within the poll’s result just 60% of the time. And that’s for polls just one week before an election—accuracy drops even more further out.

“If you’re confident, based on polling, about how the 2020 election will come out, think again,” said Berkeley Haas Prof. Don Moore, who conducted the analysis with former student Aditya Kotak, BA 20. “There are a lot of reasons why the actual outcome could be different from the poll, and the way pollsters compute confidence intervals does not take those issues into account.”

 Many people were surprised when President Donald Trump beat Hillary Clinton in 2016 after trailing her in the polls, and speculated that polls are getting less accurate or that the election was so unusual it threw them off. But Moore and Kotak found no evidence of declining accuracy in their sample of polls back to 2008—rather, they found consistently overconfident claims on the part of pollsters.

 “Perhaps the way we interpret polls as a whole needs to be adjusted, to account for the uncertainty that comes with them,” Kotak said. In fact, to be 95% confident, polls would need to double the margins of error they report even a week from election day, the analysis concluded.

 As a statistics and computer science student on an undergraduate research apprenticeship in Moore’s Accuracy Lab during the 2019 presidential primary, Kotak grew curious about the confidence intervals included with polls. He noticed that polls’ margin of error was frequently mentioned as a footnote in news articles and election forecast methodologies, and he wondered whether they were as accurate as their margins of error implied they should be.

 Kotak brought the idea to Moore, who studies overconfidence from both a psychological and statistical perspective. Much of the research on polling accuracy considers only whether the poll correctly called the winner. To gauge poll confidence, they decided to take a retroactive look at polls based on how long before an election they were conducted, and consider not whether a candidate won or lost, but whether the actual share of the vote fell within the margin of error the poll had reported. For example, if a poll showed that 54% of voters favored a candidate, and it had a 5% margin of error, it would be accurate if the candidate garnered 49% to 59% of the vote, but would be a miss if the candidate won with more than 59% of the vote (or less than 49%).

 Moore and Kotak obtained 1,400 polls conducted ahead of the general elections of 2008, 2012, and 2016, as well as the Democratic presidential primaries in Iowa and New Hampshire from 2008 and 2016 and the Republican primaries in the same states from 2012 and 2016. Because some polls asked about multiple candidates, the sample included results of over 5,000 surveys of how people said they’d vote on particular candidates, as well as the accompanying margins of error.

Analyzing the polls in seven-day batches, they found a steady decline in accuracy the farther from an election the poll was conducted, with only about half proving to be accurate 10 weeks before an election. This makes sense, since unforeseen events occur—such as former FBI director James Comey announcing an investigation into Clinton’s emails just a week before the 2016 presidential election. Yet most polls, even weeks out, reported the industry standard 95% confidence interval.

Sampling error and confidence intervals

The confidence interval quantifies how sure one can be that the sample of people surveyed reflects the whole voter population. A 95% confidence interval, for example, means that if the same sampling procedure were followed 100 times, 95 of those samples would contain the true voter population. Therein lies the problem, however.

 The confidence level takes into account “sampling error,” a statistical term that quantifies how likely it is that by pure chance, the sample varies from the larger population of voters from which the sample was drawn. For example, not surveying a large enough group of voters would increase the sampling error. But sampling error does not include any other kinds of errors—such as surveying the wrong set of people to begin with.

 “People often forget that margins of error for polls only capture the statistical sources of error,” said David Broockman, an associate professor in Berkeley’s Department of Political Science. “This analysis shows just how large the remaining non-statistical sources of error are in practice.”

Added Prof. Gabriel Lenz, also of Berkeley Political Science, “This is a fascinating analysis, and future work could sort out the sources of the inaccuracy, such as low-quality pollsters, difficulty screening likely voters, last-minute changes in voter intentions, and more.”

It’s easy to take sampling error into account in polling statistics, but much harder to account for all the other unknowns, Moore said. It’s a lesson that goes far beyond polling.

“Because we base our beliefs on imperfect and biased samples of information, sometimes we will be wrong for reasons that we did not anticipate,” he said.


Filters close

Showing results

110 of 4147
Newswise: COVID-19 Peer Hub combats vaccine avoidance amid pandemic
Released: 27-Nov-2020 8:05 AM EST
COVID-19 Peer Hub combats vaccine avoidance amid pandemic
University of South Australia

UniSA researchers are evaluating a new vaccination education initiative – the COVID-19 Peer Hub – to help immunisation and public health professionals to tackle the emerging dangers of vaccine hesitancy amid the pandemic.

Newswise: Long-Term Impacts of COVID-19: Your Mental Health
Released: 25-Nov-2020 2:15 PM EST
Long-Term Impacts of COVID-19: Your Mental Health

The COVID-19 pandemic has shaped more than half a year of our lives, canceling plans, upending livelihoods and causing feelings of grief, stress and anxiety. And Cedars-Sinai mental health experts say the pandemic could be shaping our mental health well into the future.

Released: 25-Nov-2020 12:45 PM EST
SARS-CoV-2 mutations do not appear to increase transmissibility
University College London

None of the mutations currently documented in the SARS-CoV-2 virus appear to increase its transmissibility in humans, according to a study led by UCL researchers.

Newswise: COVID-19 vaccine candidate tested preclinically at UAB nears first clinical test in people
Released: 25-Nov-2020 11:05 AM EST
COVID-19 vaccine candidate tested preclinically at UAB nears first clinical test in people
University of Alabama at Birmingham

Maryland-based Altimmune Inc., a clinical stage biopharmaceutical company, has submitted an Investigational New Drug, or IND, application to the United States Food and Drug Administration to commence a Phase 1 clinical study of its single-dose intranasal COVID-19 vaccine candidate, AdCOVID.

Released: 25-Nov-2020 11:05 AM EST
BIDMC researchers reveal how genetic variations are linked to COVID-19 disease severity
Beth Israel Lahey Health

New research BIDMC-led sheds light on the genetic risk factors that make individuals more or less susceptible to severe COVID-19.

Newswise: blog-pandemic-scenario-planning-lg-feature2.jpg
Released: 25-Nov-2020 11:05 AM EST
Pandemic Ups Game on Scenario Planning in The Arts
Wallace Foundation

Researcher/Author of new toolkit and report seeks to help arts and culture organizations add scenario planning to their strategic toolbox

Released: 25-Nov-2020 10:30 AM EST
Young people's anxiety levels doubled during first COVID-19 lockdown, says study
University of Bristol

The number of young people with anxiety doubled from 13 per cent to 24 per cent, during the early stages of the COVID-19 pandemic and lockdown 1, according to new research from the University of Bristol.

Newswise: 249837_web.jpg
Released: 25-Nov-2020 10:20 AM EST
Tracking COVID-19 trends in hard-hit states
Louisiana State University

Currently, there are over 10 million confirmed cases and more than 240,000 casualties attributed to COVID-19 in the U.S.

Released: 25-Nov-2020 9:55 AM EST
More Health Systems Join National #MaskUp Campaign
Cleveland Clinic

Many more health systems are joining the national #MaskUp campaign encouraging Americans to stop the spread of COVID-19 by following safety guidelines. Over just a few days, another 19 health systems with hundreds of hospitals united with 100 health systems nationwide with hospitals numbering in the thousands. The public service campaign is critical to the health and well-being of all Americans. It is a plea from healthcare professionals everywhere: wear a mask and follow other precautions to save lives and help get our country back on its feet.

Newswise: delaterre_jpeg.jpg
Released: 25-Nov-2020 7:35 AM EST
Warwick scientists design model to predict cellular drug targets against Covid-19
University of Warwick

The covid-19 virus, like all viruses relies on their host for reproduction

Showing results

110 of 4147