An independent appraisal and re-analysis of hydroxychloroquine treatment trial for COVID-19

DOI: https://doi.org/10.4414/smw.2020.20262

The open-label clinical trial of hydroxychloroquine in French patients diagnosed with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has received extraordinary attention amid the COVID-19 crisis [1]. Gautret et al. reported a significant decrease in viral load amongst hydroxychloroquine patients when compared with control patients as early as day 3 post-inclusion. In addition, they describe a synergistic effect in a subset of patients who received hydroxychloroquine with adjunctive azithromycin. The article concluded with a “recommendation that COVID-19 patients be treated with this combination to cure the infection and limit the transmission of the virus” [1]. We commend the authors for acknowledging certain limitations of the trial and for their approach in rapidly disseminating their preliminary data. However, now more than ever, it is paramount to exercise rigour and caution in reporting studies related to the COVID-19 pandemic. In this paper, we highlight a number of issues in the study’s design and methods and a re-analysis that does not agree with the results of Gautret et al.

Given the initiation of the study during the current pandemic, some limitations are understandable, such as nonrandomised allocation of participants to the control and treatment arms. However, one important consideration is that the treatment groups are approaching a significant difference in age (mean age 51.2 years vs 37.3 years in the control group, p = 0.06), which is not discussed or controlled for. Gautret et al. noted that they omitted patients’ clinical outcomes in this small-scale study, in order to rapidly disseminate their preliminary findings, and will discuss this in a follow-up study at the end of the trial. We look forward to reading these details in their future paper.

The study’s design limits its capacity to discern any synergistic effects of concurrent hydroxychloroquine-azithromycin therapy on the SARS-CoV-2 disease course. Only a small subset of patients (n = 6) received hydroxychloroquine with adjunctive azithromycin to treat SARS-CoV-2-related respiratory tract infections (RTIs). Of note, the clinical criteria for the administration of azithromycin in addition to hydroxychloroquine were vague, suggesting the antibiotic was given to prevent bacterial superinfection in this subset of patients. The lack of an azithromycin-only treatment arm precludes the ability to rule out azithromycin as exclusively responsible for the viral clearance observed. Furthermore, it is unclear whether these patients had pre-existing signs of a pneumonia superinfection, which could confound the patient’s viral clearance results. Therefore, the authors’ observations regarding the combination therapy are too preliminary to recommend systematic administration of both drugs concomitantly, particularly taking into account previously reported significant risks of interaction, including QT prolongation [2]. In addition, this patient group can neither be compared with nor collapsed within the hydroxychloroquine-treated patients to support hydroxychloroquine efficacy compared to controls.

An analysis of the methods also raises concerns regarding the study conclusions. Control patients were designated as those who refused treatment or patients from other centres who received supportive care only, many of whom did not have daily sampling to test for viral load. Throughout the study, the authors report an unusually high number of false negatives in their diagnostic tests. This may be due to the study’s lower ceiling for real-time qualitative polymerase chain reaction (RT-qPCR) cycle threshold: 35, rather than >40 as recommended by the protocols of the Laboratory Corporation of America, the US Centers for Disease Control and Prevention and the World Health Organization [3–5]. Further, we note the omission of several important details from the methods section. The authors failed to include primer sequences that were used to detect the virus by RT-qPCR. It has been observed that some primers made for clinical detection of SARS-CoV-2 form hairpin loops, secondary structures that can interfere with efficient detection of the virus [6, 7]. Additionally, it is unclear whether the viral load at treatment initiation or on days following was at comparable levels in the treatment and control groups. The methods included mention of electron microscopy for viral validation; however, no images were included in the article. The data analysis included proportion comparisons using chi-square and Fisher’s exact tests, which were not adjusted for age or patient presentation (asymptomatic, upper RTI, lower RTI). Finally, the authors did not comment on why more elaborate multivariate methods were not employed.

The most striking observation was the authors’ inconsistent exclusion and inclusion of treatment- and control-group subjects, respectively. The authors excluded a total of 6 patients from the hydroxychloroquine treatment group because of loss to follow-up, more specifically because of death (n = 1), intensive care unit (ICU) transfer (n = 3), adverse effects of treatment (n = 1) and early recovery (n = 1). Notably, the authors did not choose to exclude a hydroxychloroquine-treated patient for whom data are missing for days 5 and 6 of treatment (Gautret et al. supplementary table 1). It is also of interest that ICU transfer and patient fatality occurred exclusively in the hydroxychloroquine-treated group in this study, further emphasising the importance of additional safety and efficacy studies of hydroxychloroquine monotherapy in SARS-CoV-2 patients.

With respect to control patients, none were designated as “lost to follow-up”. Examination of the authors’ supplementary table 1 illustrates that diagnostic test results are missing on multiple days for nine control patients. The authors note that control patients did not undergo daily sampling, but were sampled every other day in most cases, and were considered positive for PCR when actually positive the day(s) before and the day(s) after the day(s) with missing data. However, a total of five control patients had no data on day 6 of the study, two of whom also did not have test values on day 5 either. Given the high incidence of false negative PCR readings throughout the study, the authors’ inference of virological status based on previous days’ results is inadmissible.

To address the implications of this oversight, we re-analysed the control and hydroxychloroquine-treated patient data from Gautret et al. table 3, excluding the missing datasets noted above (see table S1 in appendix 1). There were no significant differences between the virological clearance rates of control and hydroxychloroquine-treated patients on treatment days 3, 4, 5, or 6 (fig. 1A, table S1). This is in stark contrast to the analysis of Gautret et al, analysis, which reported significant differences between the groups starting as early as day 3 and persisting until day 6.

Figure 1 Gautret et al. [1] differences in PCR testing results between patient-groups are not significant when excluding missing patient data or using an intention-to-treat model for data analysis. (A) Percentage of patients with PCR-positive nasopharyngeal samples from day 3 to day 6 post-inclusion from control and hydroxychloroquine-treated groups, exclusive of all missing datasets. Multiple Fisher’s exact tests revealed no significant differences between the groups on any of the analysed treatment days. (B) Percentage of patients with PCR-positive nasopharyngeal samples from day 3 to day 6 post-inclusion from control and hydroxychloroquine-treated groups, inclusive of all patients using an intention-to-treat design. Multiple Fisher’s exact tests revealed no significant differences between the groups on any of the analysed treatment days. All statistical analysis was performed using Prism 8 software.

Another more standard way of evaluating the aforementioned patient data would be an intention-to-treat analysis, which generally accepts missing values and non-completion of treatment. We therefore performed a second revised analysis from data in Gautret et al. table 3, where we reintroduced the data from patients whom the authors described as lost to follow-up. This second analysis also failed to reveal significant differences in virological cure rates between control and hydroxychloroquine-treated patients on treatment days 3, 4, 5, or 6 (fig. 1B, table S2 in appendix 1).

As of April 7th, the burden of SARS-CoV-2 infections has exceeded 1.2 million cases and 72,000 deaths worldwide [8], highlighting the critical need for safe and efficacious therapies. Previous studies have explored the efficacy of hydroxychloroquine on SARS-CoV2 in vitro [9–11]; however, the study by Gautret and colleagues was the first to examine hydroxychloroquine treatment in patients. A recently published systematic review of published studies [12] and a review paper describing ongoing clinical trials [13] detail many groups that are examining hydroxychloroquine as treatment against SARS-CoV-2. A more recent randomised controlled trial of 30 patients was conducted in Shanghai to assess the efficacy of hydroxychloroquine in these patients. They observed no significant differences between control (n = 15) and hydroxychloroquine-treated (n = 15) patients across several outcome measures including viral load on day 7, median time to negative viral load, median time to body temperature normalisation and radiological progression [14].

The global public considers hydroxychloroquine a glimmer of hope for treating SARS-CoV-2. This undoubtedly warrants open sourcing of data, as well as greater rigour and closer scrutiny of any current trials to ensure its safe and efficient transition into clinical care for SARS-CoV-2 patients. In our re-analyses, hydroxychloroquine-treated groups approached significantly increased cure rates on day 6, demonstrating promising pilot data. We conclude that hydroxychloroquine has not yet demonstrated significant effects on decreasing viral load of SARS-CoV-2. Additional trials with larger sample sizes and more consistent analysis of patient data are needed for bona fide evaluation of hydroxychloroquine to treat SARS-CoV-2.

Appendix 1 Supplementary material

Extended statistical analysis of data from Gautret et al. [1].

Acknowledgments

The authors gratefully acknowledge Samantha Yammine, Lexy Schimmel, Sharjeel Arif, Mampay Myrthe, Viktor Yurevych, Stephanie Treat, and Tayler Kent for their valuable discussions.

References:

1 Gautret P , Lagier JC , Parola P , Hoang VT , Meddeb L , Mailhe M , et al. Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial. Int J Antimicrob Agents. 2020;105949. doi:.https://doi.org/10.1016/j.ijantimicag.2020.105949  

2 Schrezenmeier E , Dörner T . Mechanisms of action of hydroxychloroquine and chloroquine: implications for rheumatology. Nat Rev Rheumatol. 2020;16(3):155–66. doi:.https://doi.org/10.1038/s41584-020-0372-x  

3 Laboratory Corporation of America. 2020. https://www.fda.gov/media/136151/download  

4 Center for Disease Control and Prevention. 2020. https://www.fda.gov/media/134922/download  

5 World Health Organization. 2020. https://www.who.int/docs/default-source/coronaviruse/real-time-rt-pcr-assays-for-the-detection-of-sars-cov-2-institut-pasteur-paris.pdf?sfvrsn=3662fcb6_2  

6 Altman T . 2020. https://tomeraltman.net/2020/04/05/all-public-primers.html  

7 Chen SH , Lin CY , Cho CS , Lo CZ , Hsiung CA . Primer Design Assistant (PDA): A web-based primer design tool. Nucleic Acids Res. 2003;31(13):3751–4. doi:.https://doi.org/10.1093/nar/gkg560  

8 World Health Organization. 2020. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200407-sitrep-78-covid-19.pdf?sfvrsn=bc43e1b_2  

9 Liu J , Cao R , Xu M , Wang X , Zhang H , Hu H , et al. Hydroxychloroquine, a less toxic derivative of chloroquine, is effective in inhibiting SARS-CoV-2 infection in vitro. Cell Discov. 2020;6(1):16. doi:.https://doi.org/10.1038/s41421-020-0156-0  

10 Yao X , Ye F , Zhang M , Cui C , Huang B , Niu P , et al. In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2). Clin Infect Dis. 2020;ciaa237. doi:.https://doi.org/10.1093/cid/ciaa237  

11 Wang M , Cao R , Zhang L , Yang X , Liu J , Xu M , et al. Remdesivir and chloroquine effectively inhibit the recently emerged novel coronavirus (2019-nCoV) in vitro. Cell Res. 2020;30(3):269–71. doi:.https://doi.org/10.1038/s41422-020-0282-0  

12 Cortegiani A , Ingoglia G , Ippolito M , Giarratano A , Einav S . A systematic review on the efficacy and safety of chloroquine for the treatment of COVID-19. J Crit Care. 2020;S0883-9441(20)30390-7. doi:.https://doi.org/10.1016/j.jcrc.2020.03.005  

13 Gao J , Tian Z , Yang X . Breakthrough: Chloroquine phosphate has shown apparent efficacy in treatment of COVID-19 associated pneumonia in clinical studies. Biosci Trends. 2020;14(1):72–3. doi:.https://doi.org/10.5582/bst.2020.01047  

14 Chen J , et al. A pilot study of hydroxychloroquine in treatment of patients with common coronavirus disease-19 (COVID-19). Journal of ZheJiang University (Medical Sciences). 2020.