Trust but verify: The US government needs to validate antibody tests

Serology tests, also called antibody tests, help to identify people who have had a past infection with SARS-CoV-2, the virus that causes COVID-19 disease. Hundreds of these tests have been developed for the detection of antibodies to SARS-CoV-2, but many on the market are poor quality. Information about the quality of these tests is important for public health decision-making and needs to be accessible to the public.

Most of the tests have not been independently validated, so purchasers of the tests have to hope that the manufacturers’ assessments are accurate. The National Cancer Institute (NCI) and FDA are now validating these tests to determine how accurate they are, but to date, their results have not been provided.

Only 18 serology tests of the hundreds being used in the US have been granted FDA Emergency Use Authorization (EUA). Early policies of the FDA to encourage the rapid development of serology tests led to an onslaught of tests with highly variable and often poor performance. The policy led to confusion over approval status, with some manufacturers fraudulently claiming FDA approval though they were in the policy section “IV.D.” This FDA regulatory category is comprised of tests that can be marketed in the US but have not yet submitted data to the FDA for a EUA and have not been reviewed or approved by the FDA.  

After pressure, the FDA has taken measures to increase the quality of the tests. On April 22, the FDA partnered with NCI to independently check their quality on a rolling basis. NCI has an extensive network of labs, including a serology laboratory at the Frederick National Laboratory for Cancer Research, that can expertly perform independent validation studies. This laboratory capacity will allow the FDA to review the independent validation of these tests to inform the EUA process better. The FDA now requires manufacturers to submit for a EUA also to submit their tests to the NCI for independent validation.

Some independent studies performed by university researchers have revealed very large discrepancies in the performance of serology tests, with some having accuracy levels far lower than reported by the manufacturer. For example, one manufacturer’s claim of 96.7 percent sensitivity for a test was found to only have 76.4 percent sensitivity at two weeks post symptoms. In 1,000 people who have antibodies, that means that it would miss 236 cases — even though the manufacturer listed sensitivity indicates it would miss only 33.

The effort by the NCI to independently validate tests is a major step forward to ensure quality serology tests. Understanding the performance of a test in different settings could help inform national and state decisions on which tests to use. But not all the results of these independent validation studies have been made public like the manufacturer results are. 

The FDA’s website on serology test performance does not state for each test if the results are from the manufacturer or the NCI, but that “experts selected the results they considered to be most representative of expected test performance.” They recently released results from selected tests, only 2of which have received EUA. The NCI has received significant federal funding to check test quality and should share the results of these trials publicly—just as any federal grant recipient should. All the NCI test results should be listed independently of the manufacturer’s listed test performance. This would provide two independent measures of performance for each test so that public health professionals can make the best possible decision in choosing a serology test.

At a minimum, these validation results should be made public for all current EUA approved tests — there are only 18, so listing these results should be feasible. Because all tests submitting for an EUA must also submit their tests to the NCI, the FDA could easily list results of the NCI validation after the approval decision is made. The FDA and NCI are generating this valuable data, so why not share it?

If serology test results are to be used in public health decision making—such as reporting the true prevalence of a disease or changing reported case fatality rates — the tests used must be as dependable as possible. The NCI and FDA have made important steps to ensuring the reliability and accuracy of serology tests by requiring independent validation. Providing the independent validation results of new serology tests is an easy, but impactful, a step that the NCI and FDA should take to ensure that our science is sound and that our public health decisions are based on strong foundations.

Dr. Gronvall is a Senior Scholar at the Johns Hopkins Center for Health Security and an Associate Professor in the Department of Environmental Health and Engineering at the Johns Hopkins Bloomberg School of Public Health. She is an immunologist by training. Dr. West is a post-doctoral scholar with the Center for Health Security and the Department of Molecular Microbiology and Immunology at the Johns Hopkins School of Public Health They recently authored the report, Developing a National Strategy for Serology (Antibody Testing) in the United States.