The CDC's Flu-Shot Effectiveness Figures Come Out of a Regression Model, Not a Trial

The page sits on the CDC’s website, filed under the 2022–2023 season, and the methods note beneath its tables is one sentence long where it matters: “VE was estimated using the test-negative design.” The agency then explains that it compared vaccination odds among people who tested positive for influenza with vaccination odds among people who tested negative, and that effectiveness was “calculated as (1 − adjusted OR) × 100%; ORs were estimated using logistic regression.” That regression stands behind the federal instruction that “everyone 6 months and older in the United States, with rare exception, should get a flu vaccine every season.” None of the estimates on that page comes from a randomized trial. They come from a model run on people who were already sick enough to walk into a clinic and get swabbed.

Take adults 65 and older, the group flu hospitalizes and kills most reliably. The page lists the shot as 41 percent effective against influenza A at outpatient visits in the VISION network and 41 percent against hospitalization in the same network. A second network, IVY, measured hospitalized adults in the same age group over the same season and reported 27 percent, with an interval running from 10 to 43 percent. Both figures sit on one federal page, produced by one design, for one population.

The design works like a sorting machine. Patients with respiratory symptoms come in, get tested, and land in one of two bins. If the vaccine works, vaccinated people should be scarce in the flu-positive bin, and the odds ratio measures how scarce. It’s cheap and fast, and no one has to be enrolled or randomized. Its theoretical groundwork, a 2016 paper in the American Journal of Epidemiology by Sheena Sullivan, Eric Tchetgen Tchetgen and Benjamin Cowling, describes the design as one that “is purported to reduce bias” from differences in who seeks care, and says studies “can avoid or minimize bias” when designed appropriately. That’s a conditional, and it comes from the method’s defenders. Whether the conditions held in a given season is something the published 41 percent does not tell the reader.

Eyal Shahar, an epidemiologist who has taught at the University of Arizona’s College of Public Health, tested those conditions against the VISION network’s 2022–2023 analysis in an essay Brownstone Institute published on October 1. His headline is his verdict: We Cannot Trust the CDC Estimates of Flu Vaccine Effectiveness. He doesn’t claim the shot does nothing, and he declines to offer a corrected figure. “The net bias remains unknown,” he writes. His argument is narrower and harder to wave off: the method can’t deliver the clean causal number the agency posts, and the study’s own tables show why.

He starts with the calendar. According to Shahar, flu risk that season “was high in October through December 2022 and low in January through March 2023,” while the vaccine rollout “typically follows the rising wave and is completed around the winter peak.” His tally puts 61 percent of the unvaccinated side of the comparison in October through December, against 47 percent of the vaccinated side. On his account, the unvaccinated were concentrated in the months when flu was spreading and the vaccinated in the months after it had faded, so part of the gap the model credits to the shot belongs to the calendar.

Then comes the exclusion rule. The study dropped “events among patients with documented vaccination <14 days before the index date,” on the theory that immunity takes two weeks to form. Almost 2,000 outpatient encounters were removed because of vaccination 1 to 13 days before the visit or unknown vaccination status, and the flowchart offers no breakdown of who they were. Shahar calls the exclusion of early events in the vaccinated “a well-known source of bias,” one that makes the vaccine look more effective than it is, and he estimates that “estimates of effectiveness in the range of 30% to 40% might change to 20% or lower.”

“The vaccinated were healthier than the unvaccinated,” he writes, and the study’s adjustment backs him up. Estimates came out lower after adjustment in most outpatient analyses and in every hospitalization analysis. If correcting for differences between the groups shrinks the number, the raw comparison was flattering the shot. Shahar argues the correction didn’t finish the job, because the healthy-vaccinee effect “cannot be completely removed by regression models.” Those models handled age and calendar time “as natural cubic spline variables” and added inverse-propensity weights “estimated via generalized boosted regression trees,” and Shahar says the published account of that machinery is too thin to check: “No one would have been able [to] replicate their analysis based on this description, even if they were given the dataset.”

The authors dealt with one number in a single word. Among hospitalized flu patients 65 and older, case fatality was 4.0 percent in the vaccinated and 2.7 percent in the unvaccinated, a crude risk ratio of about 1.5. The study described in-hospital severe outcomes as “similar across vaccination groups.” The record does not show the shot raises the risk of death. It shows that the vaccinated and unvaccinated patients in this dataset differed in ways that moved their outcomes by half again, and that the authors called the gap similar. The inference is ours, drawn from Shahar’s figures, but it’s hard to avoid: if the groups differ enough to open a gap that size in mortality, the same regression is being trusted to remove differences just as large from the effectiveness estimate.

Downstream, the figures travel without the machinery attached. A 2023 JAMA news brief, “Southern Hemisphere Outcomes Support Flu Vaccine Effectiveness”, relayed a roughly 52 percent cut in flu hospitalizations from five countries and passed along the advice that at-risk Americans “should receive their seasonal vaccination by September or October,” without describing the study design behind the number. This April, Pediatrics, the journal of the American Academy of Pediatrics, ran a piece titled “The Meaning of Vaccine Effectiveness: Candid Communication About the Flu Shot,” a title that treats candor about the shot’s effectiveness as something still owed.

The VISION investigators and the CDC hold the encounter-level data. Shahar had the published tables and a methods section he says no one could reproduce from. Readers get the 41 percent. Shahar’s own conclusion is bleak: “Since randomized trials will never be conducted, other designs should be sought.” The trial that would test that 41 percent against hospitalization and death in older Americans has not been commissioned. The decision to order it sits with the CDC and HHS, and until one of them makes it, this fall’s recommendation goes out to everyone 6 months and older resting on a regression that the people who built it have not shown can be checked.

Sources

1. CDC, 2022–2023 seasonal flu vaccine effectiveness estimates
2. Eyal Shahar, We Cannot Trust the CDC Estimates of Flu Vaccine Effectiveness (Brownstone Institute, 2026)
3. Sullivan, Tchetgen Tchetgen and Cowling, Theoretical Basis of the Test-Negative Study Design (American Journal of Epidemiology, 2016)
4. CDC, flu vaccine key facts
5. JAMA, Southern Hemisphere Outcomes Support Flu Vaccine Effectiveness (2023)
6. Pediatrics, The Meaning of Vaccine Effectiveness: Candid Communication About the Flu Shot (2026)
7. University of Arizona, Eyal Shahar profile