Pandemic upends norms around peer-reviewed studies

The seemingly insatiable demand for medical information during the pandemic has accelerated the practice of making emerging science more quickly available to scientists, lawmakers and the public.

That’s putting a spotlight on preprint studies — papers that are published before being peer-reviewed — at a time when the world is eager to learn about even the slightest change in the coronavirus.

But while experts welcome the increased accessibility and speed of preprints, their growing popularity also poses challenges since the public is essentially left to digest troves of complicated data and findings mostly on their own, sometimes leading to misinterpretation or misinformation. 

Preprint studies are not a new phenomenon, but COVID-19 heightened public awareness of them because they provided information much sooner than waiting months or years for publication in a journal — a timeline that is not considered feasible during a pandemic when public health officials need to make decisions to prevent deaths.

“We cannot wait,” Leana Wen, an emergency physician and public health professor at George Washington University, said. “We don’t have that kind of time when there is a deadly virus, and months of delays would cost thousands or more lives.”

MedRxiv, one of the biggest sites that publishes preprints, had only been running for about six months by January 2020. That month 217 manuscripts, all unrelated to COVID-19, were posted, compared to in May, when the server had 1,625 preprint studies on the virus published.

“If you wait for the peer review publication, you’re far behind in the field,” said Harlan Krumholz, one of medRxiv’s co-founders and a cardiologist at Yale University.

“I can either stick my head in the sand or I can realize this is just a new way of communicating science and clinical research,” he added. 

MedRxiv said it screens the preprints before publishing them to ensure they’re not anonymous, and that they don’t contain any privacy violations, Krumholz said. The nonprofit, which is run out of Cold Spring Harbor Labs, also makes the call on whether it would be “harmful” to release the study before it’s peer reviewed.

While some have speculated that the focus on preprints could allow for unreliable research to be amplified, there’s no data to suggest the studies on COVID-19 are published at a lower rate or withdrawn at a higher rate than other research.

A total of 11,300 pandemic-related preprints were posted to medRxiv between January 2020 and March 2021. At least 38 percent of those studies have since been peer-reviewed and published, with more likely in the coming months. As a point of comparison, 38 percent of nonpandemic-related preprints posted to medRxiv since January 2020 have been peer-reviewed and published in journals.

The database Retraction Watch said there is the same ratio of retractions among studies related to COVID-19 as those on other topics. In total, it has documented 149 retractions on COVID-19, with about one-third being preprints.

Initially, the database did not include withdrawals of preprint studies, according to Ivan Oransky, co-founder of Retraction Watch.

“We quickly realized that that would be leaving out an important part of the picture if we left them out, so we’ve now started cataloging those,” he said.

Amid the surge of preprints, MIT Press launched Rapid Reviews: COVID-19 to speed up the peer review of COVID-19-related articles.

Still, the larger spotlight on COVID-19 preprints has raised concerns the media and public could misconstrue the results of early studies as finalized science and use it to make personal health decisions.

A study published in Health Communication in January found that about 43 percent of studied news articles on preprints did not indicate the study was preliminary or not yet peer-reviewed.

Scientists say the medical community and the media need to do a better job of communicating to the public the evolving nature of scientific knowledge and the fact that each preprint study does not provide unrefuted solutions. Experts also say the public should consider preprints as just one piece of the puzzle rather than the full picture.

MedRxiv includes a disclaimer on its website emphasizing that preprints are “preliminary” and “have not been certified by peer review.”

“They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information,” the website says.

“Science is a method, and it’s a process of thinking and exploring new ideas and building consensus over time,” Michael Cronce, an assistant editor for Rapid Reviews: COVID-19, said. “And so I think that you’re inevitably gonna run into conflicts when you’re trying to come to conclusions as quickly as possible with a minimal amount of information.”

Rapid Reviews: COVID-19 is designed to wade through the surge of coronavirus-related preprints, speed up the peer review of valuable studies and work to limit the spread of misleading information.

Prabhjot Singh, an associate clinical professor of medicine at Mount Sinai Health System in New York, said he thinks preprints will remain a major component of the research field following the pandemic, so scientists have to be aware their preprinted studies could be distorted.

“I think you can’t put preprints back into the box,” he said. “We need to just be increasingly aware that very fresh, scientific information that’s still in the process of science is going to be taken out of context or people will try to make interpretations about it. And part of what we have to do is educate the public about the strengths and limitations of the scientific process.”

The scientific community has come a long way in closing the “huge, huge gap of knowledge” on COVID-19, said Michael Mackert, director of the Center for Health Communication at the University of Texas at Austin.

But experts are still discovering and researching new data related to the pandemic, including on variants and booster vaccines, which officials continue to use to make decisions.

“I think preprints are gonna have a pretty important role to play throughout this pandemic, just because we have to keep learning new things and course correcting as we go, because there’s just a lot we still don’t know and can’t know,” Mackert said.