What a viral study criticizing cancer screening gets very, very wrong

A recent study prompted CNN to report, “Most cancer screenings don’t ultimately give someone extra time beyond their regular lifespan.” Does that mean it’s time to dismantle the cancer screening infrastructure in the United States?

Probably not. The complex math behind screening befuddled the researchers’ attempts to assess whether screening helped people live longer, leading to unsupportable conclusions. Too bad, because they were right that cancer screening must be assessed in terms of its ability to improve the health of populations.

It’s important to acknowledge that there are potential drawbacks to screening. Sometimes thousands of people must be screened to prevent just a handful of cancer deaths. Nearly 2,000 women aged 40 to 49 must receive mammograms to prevent one death from breast cancer, according to the United States Preventive Services Task Force.

With that kind of ratio, small harms from screening, accumulated over the many women who do not have cancer, could counterbalance the large benefits the few women alerted to breast cancer receive. For example, they might have to undergo invasive biopsies to evaluate screening findings that are neither cancer nor any other serious condition. Idle cancers detected by screening that would have never caused the patient harm can lead to treatments that are costly and sometimes toxic.

The study authors wanted to determine whether the multitudes of small harms from screening counterbalance the more tightly focused but substantial benefits to people who do have cancers that must be treated. That’s a tricky question to investigate. So they performed a meta-analysis where they combined data from studies of cancer screening, examining whether if collectively there was evidence it lowered rates of not just cancer deaths, but all deaths.

They thought they had their answer. For most cancer screenings, they did not find statistically significant differences in overall death rates. The editor in chief of JAMA Internal Medicine at the time the paper was accepted offered her summary: “despite the popular tagline, most cancer screening does not save lives.”

Not so fast. Proving that kind of negative isn’t just hard. It’s impossible.

Clinical studies ask whether treatments work, not if they don’t, and these aren’t two sides of the same coin. Rather, studies are designed to evaluate if a treatment works to a certain degree, often termed a clinically significant effect. The study is “negative” if the treatment’s benefits fall below that level. The treatment might still work, just not well enough.

A well-designed study enrolls the number of people it needs to identify this clinically significant effect if it is present (termed the study’s power). Meta-analyses, such as the cancer screening study, don’t have a specific power. They follow the “get what you get and don’t get upset” design, combining available studies, whether there are many or few.

People who perform meta-analyses know of this problem and rarely just take the pooled results at face value. But the authors of the cancer study apparently did. “The findings,” the authors wrote, “suggest that most individuals will not have any gain in longevity [from cancer screening].”

But what if the analysis lacked sufficient power? If the authors had paid more heed to this alternative explanation, they would have pointed out that for five of the seven categories of screening they examined, the results were pointing in the direction of an overall mortality benefit. In other words, that screening appeared to lengthen life.

If they were concerned about their study’s power, their choices don’t reflect that. They actually took steps to further reduce it. They dropped data from the National Lung Screening Trial (NLST), even though the NLST is the seminal study of lung cancer screening with low-dose computed tomography. It established guidelines and coverage for the approach. (I was the study lead for clinical practice guidelines that recommended lung cancer screening based on this study, and soon thereafter I formally requested Medicare coverage for lung screening. DELFI Diagnostics, where I work, is now developing a blood test aimed at identifying people most likely to benefit from lung screening.)

The NLST compared CT screening with chest X-ray screening and showed that CT screening reduced deaths from lung cancer and deaths overall. The authors say they excluded studies that evaluated chest X-ray screening, as the NLST did, because it is an outdated approach. But they did include a different study that evaluated chest X-ray screening. In another place they say they excluded studies that compared different screening approaches (another feature of the NLST). But they included a colorectal cancer screening study that did exactly that.

By my calculations using the authors’ methods, including the NLST data would have increased the certainty that lung cancer screening with low-dose CT lengthens life from 31% to 81%. This would align with an analysis across eight lung screening studies reporting that overall mortality is likely reduced by 4% — a study the authors fail to even mention.

There are other decisions the authors made that defy screening’s math. When reporting rates of death in groups that were and were not screened, their denominator is per 100 person-years, rather than per 100,000 person-years, which is the convention. This means rounding up event rates by 1,000-fold. That masks the reader’s ability to ascertain if there were important differences in event frequency between groups.

Cancer screening certainly isn’t perfect, nor is the authors’ question unimportant. Screening’s tradeoffs are complex and its math is challenging. Ensuring that it delivers benefits at the population level requires a focus on maximizing its benefits to those who harbor cancer, while minimizing the harms it visits on others. Cavalierly dismissing screening is ultimately harmful itself.

Peter B. Bach, M.D., is chief medical officer of DELFI Diagnostics. He is the immediate past chair of CMS’s MEDCAC and a member of the National Academy of Medicine.

Correction: A previous version of this essay misstated the role of an editor at JAMA Internal Medicine. It also misstated the number of lung cancer screening reports included in an analysis and the overall mortality reduction it found.