RR:C19 Evidence Scale rating by reviewer:
Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.
This paper uses an agent-based model to track COVID-19 progression and to estimate the costs of keeping the US economy open during the pandemic. The model is well-designed and provides valuable insights on the relative impact of societal choices regarding social distancing and school closure. However, some model parameters, such as the rate of asymptomatic infection, are not well-understood, introducing uncertainty. Additionally, the results are highly influenced by the value of a statistical life (VOSL), which is difficult to estimate with precision and is not a direct medical cost. Finally, the analysis does not account for benefits associated with reopening the economy, such as reduced unemployment. While the paper makes important contributions, readers should be aware of these limitations, particularly in interpreting the absolute magnitudes of results.
Below, we provide some questions and suggestions for the authors.
Is the model simulating all 330 million individuals in the US? This information is not mentioned. It is important to know the size of the network to confirm that 25 independent simulation replications are sufficient.
The model uses a single R0 value of 2.5 for the whole US taken from the current best estimates table of the CDC "COVID-19 Pandemic Planning Scenarios". However, R0 estimates vary across populations and geographic regions. We recommend that the authors discuss why the choice of a single value of R0 is appropriate to represent the whole US. Further, we recommend that the authors provide more detail on how the model was calibrated and which targets were used.
The parameters describing the transmission dynamics are taken from the current best estimates reported by the CDC, a justifiable approach. However, one of these parameters, the percentage of asymptomatic infections, ranges from 10 to 70 percent (CDC, 2020). We suspect that the medical costs will be sensitive to the value of this parameter, warranting additional sensitivity analysis.
The simulations represent dynamics from January 1, 2020 through December 31, 2020. As shown in Figure 2, interesting dynamics extend beyond the year’s end. For example, the 80 percent and 90 percent compliance scenarios with 30-day stay-home durations give daily new cases in December that have either just peaked or are yet to peak. We recommend that the authors discuss how the comparisons may change if the period of analysis were extended.
The title and discussion describe the paper as focusing on medical costs, but the major driver of the cost estimates is the value of lives lost. While important in estimating the total societal impact of COVID-19, the value of lost life is not a medical cost; the VOSL measure used in the analysis quantifies society’s willingness to pay for a reduction in mortality risk. The paper could be reframed to convey that it focuses on medical costs and the value of lives lost. It might be informative to present these estimates separately, particularly because pinpointing the medical costs of COVID-19 is more straightforward than valuing lives lost.
Describing these costs as representing “5 percent of GDP” could be misleading. Direct medical costs factor into GDP because they represent economic productivity. But the VOSL captures the intrinsic value of life beyond productivity contributions. Further, GDP is an annual number, while the VOSL is a cumulative amount reflecting losses across the remaining life span.
The statement in the limitations section that the cost estimates provide a lower-bound for the older population seems incorrect. A major driver of costs is the value of lives lost, and—as the authors note—there are a range of methodologies for estimating these costs. The authors assign the same VOSL to everyone, regardless of remaining lifespan. However, an age-dependent methodology could result in lower cost estimates for older adults. Further, the authors approximate the direct medical costs of COVID-19 using payment rates for large-group insurance. Payment for the Medicare population is substantially lower than commercial payment (White and Whaley, 2019), including payment for COVID-19 treatment (AHIP, 2020).
The assumptions about school closures need more explanation and should perhaps be modified to reflect current events. The authors assume schools remain closed through August 2020. When did the model assume that schools shut down, and what are the assumptions about how many schools would have open before September 1st? Ideally, the model would allow for geographic variation in school schedules and allow closures to extend later into the fall. The authors should also clarify what types of schools are assumed to close—are preschools and universities included?
The assumption that anyone who does not get a hospital bed will die seems overly restrictive. Particularly for the population that would not require an intensive care unit (ICU) stay or ventilator, some may recover independently. Could this assumption be revised so that only a fraction of those who would not require an ICU stay will die? It may also be worth considering that available beds may go to those with the greatest medical need, minimizing any mortality impact.
It would helpful to include context regarding how current stay-at-home compliance rates in the US compare to those used in the analysis. Additionally, the authors should explain how compliance is defined—e.g. what activities are fully off-limits, what activities are permitted with precautions, etc. Ideally, the model would account for heterogeneity across the country in the degree to which social distancing guidelines are followed.
The authors consider two possible hospital stay durations, and state that 14 days represents the most likely scenario. However, several studies report lower median values (Lewnard et al., 2020; Price-Haywood, et al., 2020), suggesting that 14 days may be closer to an upper bound.
The analysis considers only costs that decline as stay-at-home compliance increases. However, there are many costs that move in the opposite direction. Importantly, less-restrictive stay-at-home policies may increase employment outside of the medical sector and could have other benefits such as improving educational outcomes. While beyond the scope of this paper, these countervailing factors could be more clearly acknowledged.
America’s Health Insurance Plans (AHIP), 2020. COVID-19 Cost Scenario Modeling: Estimating the Cost of COVID-19 Treatment for US Private Insurers. Available at: https://www.ahip.org/wp-content/uploads/AHIP-COVID-19-Modeling.pdf
Lewnard, Joseph A., Vincent X. Liu, Michael L. Jackson, Mark A. Schmidt, Britta L. Jewell, Jean P. Flores, Chris Jentz, et al., 2020. “Incidence, Clinical Outcomes, and Transmission Dynamics of Severe Coronavirus Disease 2019 in California and Washington: Prospective Cohort Study.” BMJ, 369 (May). British Medical Journal Publishing Group. Available at: https://doi.org/10.1136/bmj.m1923.
Price-Haywood, Eboni, Jeffrey Burton, Daniel Fort, and Leonardo Seoane, 2020. “Hospitalization and Mortality Among Black Patients and White Patients with COVID-19.” The New England Journal of Medicine, 382(26). Available at: https://www.nejm.org/doi/full/10.1056/NEJMsa2011686
White, Chapin and Christopher Whaley, 2019. Prices Paid to Hospitals by Private Health Plans are High Relative to Medicare and Vary Widely. RAND: Santa Monica. Available at: https://www.rand.org/pubs/research_reports/RR3033.html
U.S. Centers for Disease Control and Prevention, 2020. Covid-19 Pandemic Planning Scenarios. Available at: https://www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios-h.pdf.