Seeing the Wood for the Trees: What Does the “Science Say” on COVID-19?

In March 2020, when governments across the world introduced strict measures in response to COVID-19, health experts Aaron Carroll and Ashish Jha noted that “flattening the curve,” or slowing the rate of infection, would not only reduce demand on healthcare systems to a manageable level, but also buy vital time for research to mitigate the impact of the deadly outbreak. When discovered in December 2019, neither the virus nor the disease it causes had a name, and its transmission was only believed to be zoonotic. Fast forward six months: research into COVID-19 has produced a wealth of information about its characteristics and natural history (infectiousness, symptom mapping); treatment and prevention options; and the societal and collateral impacts of the outbreak, among many other topics. This production of scientific knowledge is unprecedented in its scale and speed, and it already boasts many successes: for instance, vaccine development is progressing faster than for any other pathogen in history.

However, this tide of new evidence is daunting in scale and variety, challenging even the researchers who work full time on COVID, let alone decision-makers and their technical advisors, who work under substantial time constraints. Collabovid.org, a comprehensive repository of COVID-19 research indexing papers from the main print and pre-print servers (medRxiv, bioRxiv, arXiv, Elsevier, and PubMed), found 26,260 articles concerning COVID-19 from January 2020 to July 22 2020, with 1,528 articles published in the past week only. On July 20, 446 articles were published in a single day.

As we discuss below, practical investments that are urgently needed to ensure that decision-makers consult the best and most relevant evidence on COVID-19 before making a decision—a process known as “evidence-informed decision-making.”

An exponential growth in papers

Image

Source: Collabovid.org

Evidence informed decision-making (EIDM) in times of COVID

“What the science says” or “following the science” is one of the most common justifications employed by decision-makers when held to account for their decisions.

Decision-makers are held to account on the evidence base supporting their decisions

Source: The Guardian, March 13, 2020

Identifying, accessing, and critically appraising up-to-date evidence is, therefore, an important task for decision-makers and their advisors. However, regular expert reviews of all the evidence are hugely time demanding. Even countries with well-established evidence-to-policy mechanisms—such as the UK ‘s National Institute for Health and Care Excellence (NICE) or the Joint Committee on Vaccine and Immunisation (JCVI)—seem to have abandoned them in the face of the crisis.

Given the speed at which evidence is produced, many decision-makers are forced to rely on a handful of experts or advisory groups. A review of the governance of the COVID-19 response found that there is little transparency on who sits in those groups and what evidence/information they rely on. Peer review mechanisms and journals are also struggling to cope with the urgency and the high volume of submissions, leaving many to self-publish (although overall self-publishing has already proven to be very impactful). For instance, only 60 percent of COVID-19 papers listed on collabovid.org are peer-reviewed—and this database does not include the many pre-print servers other than the arXiv group and other report formats. The growing number of paper retractions also makes it hard to ascertain which information is current, as seen with what were considered two landmark studies on chloroquine and hydroxychloroquine and on masks. Add to this the proliferation of “fake news,” fragmentation of information sources, and growing distrust in science in many quarters of society, and we have the makings of confused public health decisions.

Poorly informed decision-making in this outbreak will have devastating consequences on the lives of thousands or even millions, as has been shown when it comes to making purchasing decisions for commodities to prevent or treat COVID-19.

Practical suggestions to support evidence-informed decision-making

Investments are needed to facilitate knowledge translation so that decision-makers can make sense of this growing body of research. Fortunately, this situation is not altogether new: evidence-informed decision making has been a topic of interest for decades. We suggest three solutions that strike a balance between what is practical in the current pandemic and the need for quality guidance:

1. Evidence synthesis is needed to increase the accessibility of research

There is an urgent need to summarise the literature and translate it into policy at speed. Systematic literature reviews and meta-analyses (e.g. Cochrane library) can play an important role in providing a transparent, reproducible, and unbiased assessment of all relevant quality research evidence. They are a trusted and unbiased source of information that helps decision-makers and other researchers efficiently navigate the vast quantity of literature. Several groups are currently working on evidence synthesis (see the useful EvidenceAid summaries), but their work should cover topic areas beyond the “medical aspects” of COVID.

In addition to systematic literature reviews and meta-analyses, we need to deploy an intentionally created system of information concentration and critical appraisal that uses simple and practical approaches to synthesizing large amounts of literature and acknowledges caveats and methods limitations transparently. For instance, one challenge will be how to best include pre-prints in those efforts. The medical profession offers an interesting example: doctors and public health professionals use the 4S framework (summaries, synthesis, synopsis, systems), designed to inform clinical practice.

2. Curated repositories of evidence organized by sector/field should be deployed to guide decision-makers

While writing this blog post, we tried to find up-to-date evidence on the effectiveness of masks in reducing onward transmission. Collabovid.org lists 203 resources from different servers. A Google search returned even more hits, with a variety of sources. With so many research sources available, the task of identifying relevant and good quality research is overwhelming.

A review of evidence-informed decision-making found strong evidence of the positive impacts of interventions to increase the accessibility of research (including through repositories, most effectively combined with targeted messages). There are many repositories that list COVID research (e.g., collabovid or this living map of evidence) in a comprehensive manner. However, we need trusted repositories of resources organized by field or sector that are curated by experts, frequently updated, and carefully organized to allow end-users to efficiently conduct searches. The American Society for Microbiology created a COVID-19 Research Registry, which is a rich resource built by over 50 experts curating research papers (involving reading 2500 articles each week). However, this resource only covers selected topics (e.g. basic virology, treatment, epidemiology) and we need many more of those efforts to cover the full range of disciplines that are important to the COVID response.

3. Greater collaboration between researchers and decision-makers is needed to ensure research is fit for decision-making

How much of this research is useful to decision-makers and addresses policy needs? Researchers and policymakers need to engage one another—as well as those their work impacts—regularly and in the early stages of policy or research development. Greater collaboration can help ensure that the research is relevant and trusted to decision-makers and increases the ability to interpret, translate and integrate the implications of findings. A practical suggestion from the UK is the use of multidisciplinary academic-policy-public teams in local pilots—for instance, the Southampton weekly testing program, an academic-health-public service pilot commissioned by the Department of Health.

Governments’ emergency preparedness plans should include collaborative efforts to pilot and generate evidence that is immediately fit for decision-making. However, they are often too focused on practical plans to tackle the emergency, at the expense of learning. Collaborative efforts have emerged worldwide, such as the COVID-19 Multi-Model Comparison Collaboration on global epidemiological models. In addition, funders and development partners should provide resources for dedicated knowledge brokers to help build relationships, foster collaborations between decision-makers and research, and advise decision-makers on evidence. Knowledge brokering supports evidence-based decision-making, not only by raising awareness of evidence, but also by providing support to develop policies based on it. But it often falls between two cracks, especially in the context of an emergency.

These are tall orders, but we may well be at the right moment to effect lasting change in the culture of research and decision-making. COVID-19 has revealed, alongside many resounding successes for the scientific community, a desperate need for better collaboration—and now is the time to address the issue.