Review 2: "Delta variant and mRNA Covid-19 vaccines eectiveness: higher odds of vaccine infection breakthroughs"

This preprint is an important and timely study. The authors collected a dataset on infections with associated data on vaccination. They used a case-case design to assess the odds ratio of being vaccinated given infection with the Delta (versus the Alpha) variant. The authors report a higher probability of vaccination given infection with Delta than with the Alpha variant. The authors interpret this result to indicate that the Delta variant is less susceptible to vaccine-derived immunity, i.e., that the Delta variant may be an immune escape variant. This study is likely to contribute to public health decision-making. It is, therefore, a critical publication priority, and the primary analysis (based on the OR of vaccination among Delta/Alpha infections) is reasonably well-supported. However, I find some critical issues that need addressing before publication can occur.


Review:
This preprint is an important and timely study. The authors collected a dataset on infections with associated data on vaccination. They used a case-case design to assess the odds ratio of being vaccinated given infection with the Delta (versus the Alpha) variant. The authors report a higher probability of vaccination given infection with Delta than with the Alpha variant. The authors interpret this result to indicate that the Delta variant is less susceptible to vaccine-derived immunity, i.e., that the Delta variant may be an immune escape variant. This study is likely to contribute to public health decision-making. It is, therefore, a critical publication priority, and the primary analysis (based on the OR of vaccination among Delta/Alpha infections) is reasonably well-supported. However, I find some critical issues that need addressing before publication can occur.

Major comments:
The first primary concern is around the author's reported confidence intervals and statistical significance. In the following instances, the authors report overlapping confidence intervals and state the difference is significant, e.g., However, since, in this case, the 95% confidence intervals overlap widely, a statistical test is required to determine significance. To my reading, the authors did not report Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decisionmakers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
"Additionally, the Delta variant cases revealed a Ct-value mean increase of 2.24 (CI95% 0.85 to 3.64) between unvaccinated and fully vaccinated breakthrough cases contrasting with 4.49 (CI95% 2.07 to 6.91) in the Alpha VOC, suggesting a lower impact of vaccine on viral load of Delta cases." Rapid Reviews COVID-19 • Review 2: "Delta variant and mRNA Covid-19 vaccines e ectiveness: higher odds of vaccine infection breakthroughs 3 any additional tests to determine significance. Thus, the authors may need a more careful statistical analysis to support the authors' claims. Since this study is such an important and timely issue, I urge the authors to revisit the study, perhaps recruiting more substantial or more detailed statistical analytics or better reporting of statistical methodology.
Relatedly, in the first paragraph of the discussion, the authors write: However, since the authors do not report a test for significance and judging by the mean and SD in Table 3, this difference is not statistically significant. Without this significance, it appears inappropriate to claim such a difference without indicating that statistical confidence is lacking. I recommend that the authors make this clear.
In the discussion section, the authors write, That is not the case, as the authors write in the sensitivity analysis section of the results the following: "Restricting the analysis to the cases identified through WGS, we observed a drop in the adjusted OR point estimate of complete vaccination (OR=1.48; CI95% 0.75 to 2.93) with a loss of statistical significance…" The discussion must clarify this point.

Later, the authors write:
The authors suffer from the same problem. Since the 95% CI crosses 0, we cannot have confidence that the difference in Ct values the authors describe is reliable.
Therefore, I recommend that the authors make this distinction very clear. Again, a more careful statistical treatment of these data may yield more substantial support for the authors' conclusion. The discussion (for example, lines 345-355) and abstract (for example, lines 100-101) need to be edited to reflect changes made in response to the above comments.
The authors conduct a sensitivity analysis to understand the impact of reporting; however, they fail to account for the most apparent potential confounder: waning immunity. With increasing numbers of studies reporting that vaccine-derived immunity wanes over time, it is ideal if the authors could look at the time since vaccination.
Nevertheless, I understand that data may not be available, so a prominent discussion of the impacts of waning immunity on their primary result is warranted.
Additionally, the authors calculate the odds of breakthrough infection and discuss how this is impacted by vaccine effectiveness; I wonder why the authors did not calculate VE directly for Alpha and Delta? (using these methods: https://pubmed.ncbi.nlm.nih.gov/3879673/) The study's second objective is to compare a proxy for infectiousness (Ct values) between variants. The authors interpret the Ct value data in the results and discussion concerning the first objective, evaluating Delta-specific vaccine effectiveness through the mean difference between unvaccinated and vaccinated Ct values. I recommend that the Ct value results be analyzed, described, and interpreted following this stated objective to compare infectiousness between Delta and Alpha variants, rather than providing supplemental support for the first objective. Alternatively, this contribution could be described along with the other objectives early in the manuscript.
Lastly, I am calculating the crude odds ratio of being fully vaccinated given infection with delta, following methods in the supplement, as (162*517)/(28*777) = 2.84, not 1.96. I'm unsure why this discrepancy exists. If the authors used a different method to arrive at the crude OR, I recommend that they specify that in the methods.
Minor comments: 1. Throughout the manuscript, Ct values are interpreted as a measure of viral load, which is interpreted as an indicator of infectiousness. The authors should thoroughly support this interpretation with references since all researchers may not accept such an interpretation.
2. The supplemental methods seem better suited for the main text methods, contingent on the editor's approval.
3. Please add "B.1.1.7" after the first instance of Alpha in the abstract/background.