Election polls are 95% confident but only 60% accurate, Berkeley Haas study finds
Analysis of 1,400 polls from 11 election cycles found that the outcome lands within the poll’s result just 60% of the time.
26-Oct-2020 4:25 PM EDT, by University of California, Berkeley Haas School of Business
Newswise — How confident should you be in election polls? Not nearly as confident as the pollsters claim, according to a new Berkeley Haas study.
Most election polls report a 95% confidence level. Yet an analysis of 1,400 polls from 11 election cycles found that the outcome lands within the poll’s result just 60% of the time. And that’s for polls just one week before an election—accuracy drops even more further out.
“If you’re confident, based on polling, about how the 2020 election will come out, think again,” said Berkeley Haas Prof. Don Moore, who conducted the analysis with former student Aditya Kotak, BA 20. “There are a lot of reasons why the actual outcome could be different from the poll, and the way pollsters compute confidence intervals does not take those issues into account.”
<<< Probability and Statistics Games for Kids >>>
Many people were surprised when President Donald Trump beat Hillary Clinton in 2016 after trailing her in the polls, and speculated that polls are getting less accurate or that the election was so unusual it threw them off. But Moore and Kotak found no evidence of declining accuracy in their sample of polls back to 2008—rather, they found consistently overconfident claims on the part of pollsters.
“Perhaps the way we interpret polls as a whole needs to be adjusted, to account for the uncertainty that comes with them,” Kotak said. In fact, to be 95% confident, polls would need to double the margins of error they report even a week from election day, the analysis concluded.
As a statistics and computer science student on an undergraduate research apprenticeship in Moore’s Accuracy Lab during the 2019 presidential primary, Kotak grew curious about the confidence intervals included with polls. He noticed that polls’ margin of error was frequently mentioned as a footnote in news articles and election forecast methodologies, and he wondered whether they were as accurate as their margins of error implied they should be.
Kotak brought the idea to Moore, who studies overconfidence from both a psychological and statistical perspective. Much of the research on polling accuracy considers only whether the poll correctly called the winner. To gauge poll confidence, they decided to take a retroactive look at polls based on how long before an election they were conducted, and consider not whether a candidate won or lost, but whether the actual share of the vote fell within the margin of error the poll had reported. For example, if a poll showed that 54% of voters favored a candidate, and it had a 5% margin of error, it would be accurate if the candidate garnered 49% to 59% of the vote, but would be a miss if the candidate won with more than 59% of the vote (or less than 49%).
Moore and Kotak obtained 1,400 polls conducted ahead of the general elections of 2008, 2012, and 2016, as well as the Democratic presidential primaries in Iowa and New Hampshire from 2008 and 2016 and the Republican primaries in the same states from 2012 and 2016. Because some polls asked about multiple candidates, the sample included results of over 5,000 surveys of how people said they’d vote on particular candidates, as well as the accompanying margins of error.
Analyzing the polls in seven-day batches, they found a steady decline in accuracy the farther from an election the poll was conducted, with only about half proving to be accurate 10 weeks before an election. This makes sense, since unforeseen events occur—such as former FBI director James Comey announcing an investigation into Clinton’s emails just a week before the 2016 presidential election. Yet most polls, even weeks out, reported the industry standard 95% confidence interval.
Sampling error and confidence intervals
The confidence interval quantifies how sure one can be that the sample of people surveyed reflects the whole voter population. A 95% confidence interval, for example, means that if the same sampling procedure were followed 100 times, 95 of those samples would contain the true voter population. Therein lies the problem, however.
The confidence level takes into account “sampling error,” a statistical term that quantifies how likely it is that by pure chance, the sample varies from the larger population of voters from which the sample was drawn. For example, not surveying a large enough group of voters would increase the sampling error. But sampling error does not include any other kinds of errors—such as surveying the wrong set of people to begin with.
“People often forget that margins of error for polls only capture the statistical sources of error,” said David Broockman, an associate professor in Berkeley’s Department of Political Science. “This analysis shows just how large the remaining non-statistical sources of error are in practice.”
Added Prof. Gabriel Lenz, also of Berkeley Political Science, “This is a fascinating analysis, and future work could sort out the sources of the inaccuracy, such as low-quality pollsters, difficulty screening likely voters, last-minute changes in voter intentions, and more.”
It’s easy to take sampling error into account in polling statistics, but much harder to account for all the other unknowns, Moore said. It’s a lesson that goes far beyond polling.
“Because we base our beliefs on imperfect and biased samples of information, sometimes we will be wrong for reasons that we did not anticipate,” he said.
OPINION – Texas conservatives have seen this movie before. A polished Republican talks tough on the Constitution, quotes the Founders on cue, rails against Washington corruption, and convinces voters he is one of the good guys. Then the pressure hits. The cameras come on. The media starts demanding blood. And suddenly the “fighter” voters elected folds faster than a lawn chair at a church picnic.
That is the growing fear surrounding Congressman Chip Roy as speculation intensifies over the Texas Attorney General race. For many grassroots conservatives, Roy is not simply another establishment Republican. He represents something more dangerous, a Republican who knows exactly how conservatives think, exactly what they want to hear, and exactly when to abandon them to protect his standing with the political class.
That perception hardened permanently after January 6.
While Democrats, corporate media, and anti Trump Republicans launched a coordinated political assault against President Donald Trump, Roy joined the feeding frenzy at the exact moment conservatives expected Republicans to stand firm. On January 13, 2021, Roy took to the House floor and declared Trump’s conduct was “clearly impeachable.” The comments were widely covered by outlets including CNN and The Texas Tribune.
At the time, Democrats were aggressively pushing impeachment while left wing media outlets painted millions of Trump supporters as domestic extremists. Conservatives across the country watched banks deplatform citizens, federal agencies ramp up investigations, and political dissent become increasingly criminalized. And there was Chip Roy, sounding almost indistinguishable from the Republicans conservatives had spent years fighting against.
Worse still, Roy’s rhetoric placed him in alignment with some of the most despised anti Trump Republicans in modern history, including Liz Cheney and Congressman Thomas Massie. Cheney ultimately became the public face of the January 6 Committee, a committee many conservatives viewed as less interested in truth than in politically destroying Trump and intimidating his supporters. Roy may not have joined that committee, but to many voters, he helped legitimize the narrative driving it.
This matters because the Attorney General’s office is not ceremonial. The Texas AG is often the final line of defense against federal overreach, politically motivated prosecutions, censorship efforts, and constitutional violations. Every time a city government wants to object to an open records request by a citizen, they need the permission of the AG. Conservatives are not looking for another Republican who caves once the editorial boards and Sunday shows begin screeching. They want someone willing to absorb political punishment without turning on the movement that elected him.
That is why Texas State Senator Mays Middleton is gaining traction among MAGA conservatives. Known by supporters as “MAGA Mayes,” Middleton has cultivated a reputation as an unapologetic America First conservative. He backed election integrity legislation, border enforcement measures, anti-ESG policies, and efforts to stop taxpayer funded lobbying by local governments. More importantly, he has not spent the past several years publicly distancing himself from the voters who dominate today’s Republican base.
To many conservatives, the contrast is glaring. Middleton looks like a man preparing for political combat. Roy increasingly looks like a man carefully managing his reputation with DC insiders while hoping Texas voters forget what happened in 2021.
And conservatives should ask themselves an uncomfortable question. If Roy was willing to publicly break with Trump during the biggest coordinated political attack against conservatives in modern history, what happens when the next crisis arrives? What happens when federal agencies pressure Texas? What happens when media outlets begin demanding prosecutions, investigations, or compromise? Does Roy suddenly rediscover his “constitutional concerns” while conservatives once again get thrown under the bus?
Roy’s defenders will point to his conservative voting record, and that’s fair. He has opposed Biden administration policies and marketed himself as a constitutional hardliner. But conservative voters are increasingly learning that voting scorecards mean very little when pressure reveals someone’s instincts.
And Roy’s instincts, at the defining moment, were not to protect the movement. They were to condemn it alongside people who openly despised it.
Texas conservatives have spent years warning about Republicans who campaign like MAGA warriors back home while quietly serving the priorities of the donor class and establishment once inside Washington. Many now fear Chip Roy fits that mold perfectly, polished, articulate, deeply ambitious, and ultimately unreliable when the stakes become uncomfortable.
The time has come to end the political careers of all who oppose the People, those who oppose the MAGA agenda.
OPINION – For years, Texas conservatives have watched Republicans campaign as fighters back home, only to return to Washington and govern like cautious corporate managers. That frustration is now boiling over in the growing divide between Texas Attorney General Ken Paxton and U.S. Senator John Cornyn, a battle that increasingly defines the Republican Party in Texas.
Paxton has become one of the most aggressive conservative legal figures in America. Cornyn, meanwhile, is increasingly viewed by grassroots Republicans as an establishment insider tied to the old Bush era wing of the GOP. The contrast could hardly be sharper.
Paxton built his reputation fighting the Biden administration on immigration, election disputes, COVID mandates, and federal overreach. Supporters say he has consistently used the Attorney General’s office to defend Texas sovereignty and conservative values. President Donald Trump praised Paxton during his 2022 reelection fight, calling him “a true warrior for conservative values” while endorsing him against challenger George P. Bush.
For many Texas Republicans, Trump’s support mattered because Paxton was already viewed as willing to confront Washington directly rather than negotiate with it.
Cornyn has found himself on the opposite side of many of those same debates. Conservatives sharply criticized his role in bipartisan gun negotiations after the Uvalde shooting, but immigration remains the biggest source of anger among the Republican base. Cornyn has long supported expansions of employment based immigration programs, including H1B visa policies favored by major corporations.
Critics argue those programs have displaced American workers in industries like engineering, healthcare, technology, and data services by allowing companies to import cheaper foreign labor. Over the years, outsourcing firms and tech companies have repeatedly faced backlash after replacing American employees with foreign visa workers, sometimes even requiring laid off staff to train their replacements before leaving.
Cornyn argues skilled immigration helps fill labor shortages and strengthens the economy. But many Texas conservatives increasingly see the system as benefiting multinational corporations while middle-class American workers fall behind.
Paxton has aligned himself almost entirely with border hawks and immigration enforcement advocates. He has repeatedly sued the Biden administration over border policies and backed Texas efforts to secure the southern border independently of federal action. Supporters argue those lawsuits helped slow federal policies they believed encouraged illegal immigration and weakened state sovereignty.
Some conservatives also frame the immigration debate in cultural and security terms, warning that unchecked migration and weak assimilation policies can destabilize communities and strain public resources. Paxton supporters often portray him as defending Texas from the kinds of social fragmentation seen in parts of Europe.
Cornyn’s critics increasingly label him a “RINO,” shorthand for Republican In Name Only, arguing that he represents donor class priorities rather than grassroots conservatives. Trump allies have also criticized Cornyn as part of the “old Republican guard” that voters rejected during Trump’s rise. Cornyn’s primary supporter is the Lone Star Freedom Project, a dark money 501c(4) operated by former Texas Governor Rick Perry.
Opinion sections are where political realities become unavoidable. The reality is this: many Texas Republicans no longer want cautious institutional Republicans who focus on compromise while Democrats aggressively push cultural and political change nationwide.
They want confrontation. They want resistance. They want politicians willing to fight publicly and relentlessly.
That explains why Paxton continues to maintain strong support despite years of legal and political attacks. Many conservatives interpret those attacks not as proof he should step aside, but as proof he threatens entrenched political interests.
Cornyn, meanwhile, increasingly represents a Republican era many grassroots voters believe failed to defend the border, protect American workers, or stand firmly against Washington’s expansion of power. In today’s Texas Republican politics, that perception may be impossible to overcome.
FATE, TX — The Fate City Council voted last night to censure Councilman Mark Hatley following a contentious ethics hearing that exposed deep divisions among elected officials.
The censure stems from two ethics complaints alleging Hatley improperly disclosed confidential information tied to internal discussions about the potential firing of former Department of Public Safety Chief Lyle Lombard. According to testimony, Hatley shared details with local journalist Michael Pipkins of PipkinsReports.com, including references to recorded conversations with City Manager Michael Kovacs.
The complaint was filed by outgoing councilman Scott Kelley, who played a central role throughout the proceedings and ultimately did not recuse himself and voted in favor of censure.
Monday’s meeting included a formal evidentiary hearing where Hatley, represented by attorney David Dodd, presented a defense and attempted to question fellow council members. The process, however, was repeatedly constrained by legal warnings from City Attorney Jennifer Richie, who advised council members not to answer questions related to Lombard’s termination due to ongoing litigation. That guidance, issued numerous times during the hearing, limited testimony and narrowed the scope of cross-examination.
The council ultimately split along familiar lines. Kelley was joined by outgoing councilman Mark Harper and recalled councilwoman Codi Chinn in supporting the censure. Mayor Andrew Greenberg and Councilman Rick Maneval opposed it, creating a 3–2 divide before the deciding vote was cast. Councilwoman Martha Huffman ultimately sided with the majority, breaking what would have otherwise been a tie, and would have quashed the censure.
Under Texas municipal norms, a censure is a formal statement of disapproval by a governing body against one of its own members. It carries no direct legal penalty, meaning Hatley retains his elected position and voting authority. However, such a reprimand can damage political standing, limit influence within the council, and shape future electoral prospects…if the electorate so decides.
The underlying controversy traces back to the dismissal of Lombard, which has since evolved into a broader legal dispute involving claims of wrongful termination. During Monday’s hearing, repeated references to that litigation underscored the complexity of the case and the limits placed on public disclosure. Richie’s guidance, aimed at protecting the city’s legal position, effectively curtailed testimony that might have clarified key details. Critics argue this dynamic left Hatley unable to fully defend himself against the allegations.
The political context surrounding the vote is difficult to ignore. This was Chinn’s last meeting, as she was recalled from office by the voters, in part due to her involvement in the Lombard matter. Kelley, who initiated the ethics complaint, participated fully in the decision-making process knowing that this was his last meeting. Harper has also been linked in prior discussions about leadership conflicts within city administration, and for he as well, this was his last meeting. Meanwhile, all three have supported recall efforts targeting Hatley, Greenberg, Maneval, and Huffman, for additional recall, along with two new councilmen who will take their seats at the next meeting.
From a procedural standpoint, the meeting reflected a council operating under significant strain. Testimony was fragmented, legal cautions were frequent, and the final vote appeared to follow established political alliances rather than shifting based on evidence presented during the hearing. Even Hatley’s legal representation struggled to gain traction within the constraints imposed by the city’s legal posture.
Opinion
The battle for power in Fate is very real. What unfolded Monday night was not merely an ethics hearing; it was the visible culmination of an ongoing political battle inside Fate’s leadership. When a complainant votes on his own accusation; when key witnesses are effectively shielded from cross examination; when you have councilmen under recall by the very people bringing charges against their opponents; the process begins to look less like a search for truth and more like a managed outcome. It’s cut-throat politics at its worst.
What’s changed due to this Hearing? Essentially, nothing. Hatley gets a political black eye, but that’s about it. The sides were already defined, and the votes exactly as expected. Councilmen whose terms were ending anyway are now gone after delivering one last poke in the eye to their opponents. And the City Manager, who is at the heart of this debacle because of his employee decisions, and his inability to stand up to influence from Council Members… is still employed.
For residents of Fate, the final result is an up-close view into how dirty local politics can get. It diminishes the desirability of the city to new residents, hurts economic growth, and the entire process gives citizens the perspective that their city government is completely dysfunctional.
Disclosure
The author of this article was referenced during the hearing as a recipient of information discussed in the ethics complaints. The reporting above is based on observations of the public meeting and review of the proceedings.