How Accurate Were Early Expert Predictions on COVID-19?

Who made more accurate predictions about the course of the COVID-19 pandemic – experts or the public? A study from the University of Cambridge found that experts such as epidemiologists and statisticians made far more accurate predictions than the public, but both groups significantly underestimated the true magnitude of the pandemic.

Researchers from the Winton Center for Risk and Evidence Communication surveyed 140 British experts and 2,086 British lay people in April 2020 and asked them to make four quantitative predictions about the impact of COVID-19 by the end of 2020. their predictions by providing upper and lower limits of where they were 75% sure the true answer would fall – for example, a participant would say they were 75% sure that the total number of infections would be between 300,000 and 800,000.

The results, published in the journal PLOS ONE, show how difficult it is to predict the course of the pandemic, especially in the early days. While only 44% of the expert group’s predictions fell within their own 75% confidence interval, the non-expert group fared much worse: only 12% of the predictions were within their range. Even if the non-expert group was limited to people with high math scores, only 16% of the predictions fell within the range of values ​​that 75% were sure would contain the real outcomes.

“Experts may not have predicted as accurately as we hoped, but the fact that they were much more accurate than the non-expert group is a reminder that they have expertise worth listening to,” said Dr. Gabriel Recchia of the Winton. Center for Risk and Evidence Communication, the paper’s lead author. Predicting the course of a brand new disease like COVID-19 just a few months after it was first identified is incredibly difficult, but most importantly, experts can recognize the uncertainty and adjust their predictions as more data becomes available. . “

Throughout the COVID-19 pandemic, social and traditional media have spread forecasts from experts and non-experts about its expected magnitude.

The opinion of experts is undoubtedly important in informing and advising those who make individual and policy decisions. However, because the quality of expert intuition can vary drastically depending on the field and the type of judgment required, it is important to conduct domain-specific research to determine how good expert predictions really are, especially in cases where they are have potential. to shape public opinion or government policy.

“People mean different things by ‘expert’: these are not necessarily people working on COVID-19 or developing the models to inform the response,” Recchia said. “Many of the people approached to comment or make predictions have relevant expertise, but not necessarily the most relevant.” Recchia noted that in the early COVID-19 pandemic, clinicians, epidemiologists, statisticians, and other individuals seen by the media and the general public as experts were often asked to provide standard answers to questions about how bad the pandemic would be. can be. to get. “We wanted to test how accurate some of these predictions were made by people with this kind of expertise, and more importantly, how they relate to the public.”

For the survey, participants were asked to predict how many people living in their country would die and be infected by the end of 2020; they were also asked to predict the death rate from infections, both for their country and for the rest of the world.

Both the expert group and the non-expert group have underestimated the total number of deaths and infections in the UK. The official death toll in the UK on December 31 was 75,346. The expert group’s median prediction was 30,000, while the median prediction for the non-expert group was 25,000.

In terms of death rates from infections, the experts’ average prediction was that 10 in 1,000 people with the virus worldwide would die from it, and 9.5 in 1,000 people with the virus in the UK would die from it. The median response from non-experts to the same questions was 50 out of 1,000 and 40 out of 1,000. The actual death rate for infections at the end of 2020 – as best as the researchers could determine, given that the actual number of infections remains difficult to estimate – was closer to 4.55 out of 1,000 worldwide and 11.8 out of 1,000 in the UK.

“There is a temptation to look at results that say experts are less accurate than we would hope and say we shouldn’t listen to them, but the fact that non-experts fared so much worse shows that it remains important to listen to experts, as long as we keep in mind that what is happening in the real world may surprise you, ”said Recchia.

The researchers caution that it is important to distinguish between research evaluating the forecasts of ‘experts’ – individuals who occupy professions or roles in fields of relevance, such as epidemiologists and statisticians – and research evaluating specific epidemiological models. although expert forecasts may be informed. by epidemiological models. Many COVID-19 models turn out to be reasonably accurate in the short term, but become less accurate as they try to predict the results further into the future.

Reference: May 5, 2021, PLOS ONE.
DOI: 10.1371 / journal.pone.0250935