RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
Multiple previous trials (studies) failed to show any significant benefit for the anti-interleukin-6 receptor monoclonal antibody, Tocilizumab in COVID-19 hospitalized patients. In this Recovery trial, Tocilizumab was associated with improved survival in hospitalized COVID-19 patients with advanced disease.
A careful analysis of raw data showed randomization bias with unequal randomization of patients among the Tocilizumab arm and the control group (Usual care arm). Variances of 17 or 18 deaths among the Tocilizumab and Usual care arm respectively (1.3% - 1.4% of the total deaths) can invalidate the findings of this trial with a type I error. The randomization bias with excess randomization of patients with known variables that can independently cause more mortality in the Usual care arm group contributing to an excess of 1.6-2.8% deaths than the Tocilizumab arm (2.8% higher total deaths based on the age, 1.9% higher total deaths based on sex, 2.0% higher total deaths based on ethnicity , 2.4% higher total deaths based on respiratory support, and/or 1.6% higher total deaths based on days since symptom onset). Any one of these known variables can cause Type I error in favor of Tocilizumab with positive findings when none exist.
Careful reanalysis of the raw data among the compared groups in this trial is warranted to properly match the compared groups to assure the trial integrity and validity of the reported findings.
For any large trial of this magnitude, the authors could have just reported the findings of their conclusions reached in this trial and contrast them with other trials or studies about efficacy of tocilizumab for COVID-19 treatment. The investigators could have avoided an appearance of potential bias by not addressing the issues of meta-analysis of Tocilizumab efficacy in this publication and instead reported this as a separate review article with meta-analysis if necessary.
Results of the trial with potential for misinterpretation of the data and type I errors:
Analysis of the raw data of this Recovery-Tocilizumab publication notes, a total of 4116 patients were randomized with an overall mortality rate of 31.3% (n=1290 deaths) among the trial subjects. An unequal number of 72 excess patients (n=2094, 50.9%) were randomized to Usual care arm of this trial that can contribute an excess of 22.6 deaths (1.7% of total deaths) in this arm in comparison with the Tocilizumab arm (n=2022, 49.1%). Because of this unequal randomization, unless both compared groups are appropriately matched for all known risk factors that can independently affect the mortality, the trial will have randomization bias that can affect the validity of conclusions reached.
The following is the analysis of all other known variable that can independently affect the mortality regardless of the efficacy of the Tocilizumab due to unequal randomization among the following risk groups .
Age, years randomization among the Tocilizumab and Usual care arms
Age ≥ 80 years: Of the total (n=473 with mortality rate of 64.5%), 45.0% were randomized to Tocilizumab and 55.0% were randomized to Usual care arm. An excess of 47 patients randomized to usual care arm that can cause an excess of 30.3 deaths in this group creating a bias against this group.
Age ≥70 ≤80 years: Of the total (n=957 with mortality rate of 46.0%), 49.8% were randomized to Tocilizumab and 50.2% were randomized to Usual care arm. An excess of 3 patients randomized to usual care arm can cause an excess of 1.4 deaths in this group creating a bias against this group.
Age < 70 years: Of the total (n=2686 with mortality rate of 20.3.0%), 49.6% were randomized to Tocilizumab and 50.4% were randomized to Usual care arm. An excess of 22 patients randomized to usual care arm can cause an excess of 4.5 deaths in this group creating a bias against this group.
Randomization based on gender (male/female):
Of the total male (n=2772 with mortality rate of 32.6%), 48.2% were randomized to Tocilizumab and 51.8% were randomized to Usual care arm. An excess of 102 male patients randomized to Usual care arm can cause an excess of 33.3 deaths in the Usual care arm than the Tocilizumab arm.
Of the total female (n=1344 with mortality rate of 28.7%), 51.1% were randomized to Tocilizumab and 48.9% were randomized to Usual care arm. An excess of 30 female patients randomized to Tocilizumab arm that can cause an excess of 8.6 deaths in this arm than the Usual care arm.
Randomization based on ethnicity:
White: of the total (n=2782 with mortality rate of 34.1%), 48.7% were randomized to Tocilizumab and 51.3% were randomized to Usual care arm. An excess of 70 patients randomized to Usual care arm can cause an excess of 23.9 deaths in the Usual care arm than the Tocilizumab arm.
Black, Asian or Minority Ethnic: of the total (n=698 with mortality rate of 29.8%), 48.9% were randomized to Tocilizumab and 51.1% were randomized to Usual care arm. An excess of 16 patients were randomized to Usual care arm can cause an excess of 4.8 deaths in the Usual care arm than the Tocilizumab arm.
Unknown: of the total (n=636 with mortality rate of 21.1%), 51.1% were randomized to Tocilizumab and 48.9% were randomized to Usual care arm. An excess of 14 patients randomized to Tocilizumab arm can cause an excess of 2.9 deaths in the Tocilizumab arm than the Usual care arm.
Respiratory support at randomization:
Invasive Mechanical Ventilation: A total (n=562 with mortality rate of 47.5%), with 47.7% were randomized to Tocilizumab and 52.3% were randomized to Usual care arm. An excess of 26 patients randomized to Usual care arm can cause an excess of 12.4 deaths in the Usual care arm than the Tocilizumab arm.
Non-invasive ventilation: A total (n=1686 with mortality rate of 38.3%), with 48.6% were randomized to Tocilizumab and 51.4% were randomized to Usual care arm. An excess of 48 patients randomized to Usual care arm can cause an excess of 18.4 deaths in the Usual care arm than the Tocilizumab arm.
No ventilator support: of the total (n=1868 with mortality rate of 20.2%), 50.1% were randomized to Tocilizumab and 49.9% were randomized to Usual care arm. An excess of 2 patients randomized to Tocilizumab arm can cause an excess of 0.4 deaths in the Tocilizumab arm than the Usual care arm.
Days since symptom onset at Randomization:
≥ 7 days: A total (n=2787 with mortality rate of 30.0%), with 48.6% were randomized to Tocilizumab and 51.4% were randomized to Usual care arm. An excess of 79 patients randomized to Usual care arm can cause an excess of 23.6 deaths in the Usual care arm than the Tocilizumab arm.
≤ 7 days: of the total (n=1328 with mortality rate of 34.3%), 50.3% were randomized to Tocilizumab and 49.7% were randomized to Usual care arm. An excess of 8 patients randomized to Tocilizumab arm can cause an excess of 2.7 deaths in the Tocilizumab arm than the Usual care arm.
Supporting Statistical analysis for major revision: According to the data as analyzed by the authors, the Tocilizumab arm has a mortality rate of 29% (596/2022), and Usual care arm with 33% (694/2094) mortality rate.
The Recovery trials were powered to detect an absolute difference of 4 percentage points between the two compared arms. Any sampling errors that would have caused 17 fewer deaths than what was observed in the Tocilizumab arm or 18 more deaths than what was observed in the Usual care arm may result in the loss of statistical significance for Tocilizumab in this trial. These variances of 11 or 18 deaths among the two Tocilizumab and Usual care arms only account for 1.3% - 1.4% of the total deaths (n =1290) in this trial (a copy of the statistical analysis is enclosed).
Based on the above-presented data, nonuniform randomization of different age groups with varying mortality (<70 years, ≥70 ≤ 80 years and ≥ 80 years) among an excess of 72 patients randomized to the Usual care arm contributed to excess mortality with 36.1 extra deaths in the Usual care arm, and after adjusting for this bias, the trial will lose statistical significance vs. benefit with Tocilizumab treatment (p=0.20). Additionally, after adjusting for the excess deaths in the Usual care arm for each of the known variables that caused bias with excess deaths in the Usual care arm [Gender-male/female with an excess of 24.6 deaths (33.3 minus 8.6; p=0.07), ethnicity randomization bias with an excess of 25.7 deaths (23.9+4.8 minus 2.9, p=0.09), respiratory support at randomization bias with an excess of 30.3 deaths (12.4+18.4 minus 0.4, p=0.12), and/or days since symptoms onset with excess of 20.9 deaths (23.6 minus 2.7, p=0.06); this trial would lose the statistical significance for treatment benefit with Tocilizumab. If all of the above five known variables excess deaths were to be combined due to unequal randomization, there would be an excess of 152 deaths attributable to randomization bias in the usual care arm and 12 excess deaths attributable to the same in the Tocilizumab arm, with a mortality of 584/2020 in the Tocilizumab arm and 542/2094 in the Usual care arm (p=0.03, NNH 33.34, RR 1.11; CI 1.01-1.23).
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
