a Endocrinology, Diabetology and Metabolism, Department of Internal Medicine, University Hospital Basel, Switzerland
b Department of Clinical Research, University of Basel, Switzerland
c Bürgerspital Solothurn, Switzerland
d Medical University Clinic, Department of General Internal and Emergency Medicine, and Department of Endocrinology, Diabetology and Clinical Nutrition, Kantonsspital Aarau, Switzerland
AIM OF THE STUDY
The internal validity of double blinding in randomised placebo-controlled trials (RCTs) has become a target of criticism. The goal of this study was to investigate (a) how accurately the patients and their treating physicians were able to guess their assigned treatment, and (b) predictors for an accurate guess.
Data on treatment estimation from patients (n = 382) and their physicians (n = 358 guesses) in an RCT investigating the role of adjunct prednisone for community-acquired pneumonia in a tertiary care setting were analysed. At discharge, patients and their physicians had to guess whether they had been assigned to the prednisone or to the placebo group. The alternative possibility was “uncertain”. Percentages and confidence intervals (CIs) were calculated for the proportion of patients guessing correctly. Chance finding was defined as having 50% or less correct guesses. To test for predictors for prednisone treatment guess, a mixed effects logistic regression model was performed.
In the prednisone group, 28.9% (55/190; 95% CI 22.6–36.0%) of the patients made a correct guess and the majority (61.6%, 117/190) was uncertain. In the placebo group, 13.0% (25/192; 95% CI 8.8–18.8%) guessed correctly, with the majority being uncertain (69.8%, 134/192). Physicians guessed correctly in 48.3% (87/180, 95% CI 40.8–55.9%) of cases in the prednisone group and in 66.3% (118/178, 95% CI 58.8–73.2%) of cases in the placebo group, which was above chance for the placebo group. The physicians were uncertain in 21.7% (39/180) of cases in the prednisone group, and in 15.2% (27/178) of cases in the placebo group. Significant predictors for guessing prednisone were the occurrence of hyperglycaemia (odds ratio [OR] 3.77, 95% CI 2.39–5.95; p<0.001) and a shorter time to clinical stability (OR 0.95, 95% CI 0.91–0.99; p = 0.02).
We confirmed that patient blinding was achieved in this study. Physicians made correct guesses more often than patients. Treatment estimation by both patients and physicians was led not only by the expectations of treatment effects of the study drug but also by known side effects of prednisone.
Trial registration no.:
Keywords: randomised, controlled trials, double-blind, treatment assignment, guess
Randomised, placebo-controlled, double-blind studies (RCTs) have emerged as a gold standard in evaluating the effects of new drug treatments, but the internal validity of the process of double blinding has become a target of criticism . Several studies raised the question as to how accurately patients were able to guess their assigned treatment [2–7]. The accuracy of patients’ and physicians’ guesses was striking. Two studies evaluating the effects of two antidepressant drugs compared with placebo showed that the majority of patients and physicians were able to distinguish active drug from placebo [2, 3]. In these studies, the more certain the patients were about their guess, the higher was their probability of a correct guess. This is not only a problem for outcome assessment bias – which could be avoided by using hard or objective endpoints – but it is also suspected to be a source of bias concerning treatment effect. A study by Schnoll et al. evaluating smoking cessation drugs showed that a patient’s belief of being assigned to the intervention group positively influenced outcome . Even more striking is the effect of an antidepressant drug in chronic pain management: Pretreatment expectations of pain relief influenced the extent of pain relief in patients assigned to the intervention group. However, this was not observed in the placebo group, suggesting an effect beyond placebo .
Nevertheless, these studies were performed in the setting of psychiatric care [2, 3] or management of chronic disease (such as obesity, smoking cessation, chronic pain) [4–6] that have a longer course of disease and follow-up, more distinguishable treatment effects, and subjective outcomes especially influenced by personal behaviour and expectations. Currently, there are no publications available that examined the blinding in RCTs in acute clinical settings with short-term interventions in acute, life-threatening diseases. Therefore, we evaluated treatment estimation accuracy in a RCT investigating the effects of an adjunctive 7-day treatment with prednisone in patients hospitalised with community-acquired pneumonia and whether there were predictors for guessing prednisone rather than placebo.
Study procedures and definition of successful blinding
A detailed description of the study procedures has been published elsewhere . Briefly, patients with community-acquired pneumonia admitted to tertiary care hospitals in Switzerland were assigned to receive either 50 mg of prednisone or placebo daily for 7 days in a double-blind fashion. Before discharge, patients and their treating physicians (or the assigned study doctors if the treating physician did not answer) were asked to guess whether they had been allocated to the prednisone or the placebo group. The cut-off for guessing correctly by chance was defined as 50%, according to flipping a coin. An alternative possibility was to answer “uncertain”. Answering “uncertain” was equated with making a wrong guess because not being certain about treatment allocation is the ultimate proof that the patient/physician was blinded. Therefore, successful blinding was achieved if ≤50% of the patients made a correct guess, and the rest either made a wrong guess or were uncertain about treatment allocation. The same principle held true for the physicians’ guesses.
The following predictors for guessing prednisone were analysed: (a) Time to clinical stability (TTCS, with longer TTCS leading to the thought of placebo treatment); (b) occurrence of hyperglycaemia (defined as either mean blood glucose >10 mmol/l on at least one day during hospitalisation, or one blood glucose measurement >15 mmol/l, or need of new insulin treatment); (c) serious adverse events during hospitalisation that were possibly related to prednisone treatment; (d) the occurrence of nosocomial infections or empyema during hospitalisation; (e) persisting fever (being a predictor for placebo guess); and (f) relapse.
Guesses from patients and physicians according to treatment group are presented as summary statistics including 95% confidence intervals (CIs). Criteria for successful blinding were met if the lower 95% CI of the proportion that made a correct guess was below or equal to 50%. For the evaluation of predictors for guessing prednisone a mixed effects logistic regression model was fitted (GLMM). The outcome variable was guessing prednisone as binary variable (placebo = 0, prednisone = 1). Explanatory variables were the predictors mentioned above with participant identification number as random effect.
R software version 3.4.3 was used for the analyses.
Brief summary of original results
In this study, which was conducted from 1 December 2009 to 21 May 2014, 785 patients who were hospitalised with community-acquired pneumonia were randomly assigned to receive either prednisone (n = 392) or placebo (n = 393). The primary endpoint was time to clinical stability (TTCS). TTCS was shorter in the prednisone group (median 3.0 days, interquartile range [IQR] 2.5–3.4) than in the placebo group (4.4 days, 4.0–5.0; hazard ratio 1.33, 95% CI 1.15–1.50; p <0.0001). There was no difference between groups in pneumonia-associated complications. Patients in the prednisone group had a higher incidence of in-hospital hyperglycaemia requiring insulin treatment (76 [19%] vs 43 [11%]; odds ratio [OR] 1.96, 95% CI 1.31–2.93; p<0.01). Both groups had similar rates of adverse events that were compatible with corticosteroid use .
Treatment assignment guesses
Data were available from 449 patients from three centres in Switzerland (University Hospital Basel, Kantonsspital Aarau and Bürgerspital Solothurn). In 334 cases, guesses from both the patients and the physicians were available. Physicians’ and patients’ guesses were missing in 70 and 45 cases, respectively.
In the prednisone group, 28.9% of the patients (55/190) correctly guessed prednisone (95% CI 22.6–36.0%), which was below chance. Another 9.5% in this group (18/190) thought they had received placebo. The remaining 61.6% of patients (117/190) were uncertain about treatment allocation. The physicians in the same treatment group (prednisone) guessed correctly in 48.3% of cases (87/180, 95% CI 40.8–55.9%) which was within the expected range of chance. Thirty percent of the physicians’ guesses were wrong (54/180), and 21.7% (39/180) were “uncertain” (fig. 1, table 1).
|Treatment allocation||Patients’ guesses (n = 382)||Physicians’ guesses (n = 358)|
|Prednisone||55; 28.9% (22.6–36.0%)||18; 9.5% (5.7–14.6%)||117; 61.6% (54.3–68.5%)||190||87; 48.3% (40.8–55.9%)||54; 30.0% (23.4–37.3%)||39; 21.7% (15.9–28.4%)||180|
|Placebo||33; 17.2 (12.1–23.3%)||25; 13.0% (8.8–18.8%)||134; 69.8% (62.8–76.2%)||192||33; 18.5% (13.1–25.0%)||118; 66.3% (58.8–73.2%)||27; 15.2% (10.2–21.3%)||178|
In the placebo group, 13.0% of the patients (25/192) correctly guessed placebo (95% CI 8.8–18.8%), which was below the chance of 50%; 69.8% (134/192) were uncertain and 17.2% (33/192) falsely thought themselves to have been allocated to the prednisone group. Of the physicians’ guesses for the placebo group, 66.3% (118/178) were correct (95% CI 58.8–73.2%); 18.5% (33/178) of the guesses were wrong and 15.2% (27/178) were “uncertain”.
Predictors for treatment guess
The results for the linear regression model are summarised in table 2. The longer the time to clinical stability, the lower was the probability for the patients and their physicians to guess prednisone (OR 0.95, 95% CI 0.91–0.99; p = 0.02). The higher blood sugar values were during hospitalisation the higher was the probability to guess prednisone (OR 3.77, 95% CI 2.39–5.95; p <0.001). All other potential predictors did not show statistical significance: The ORs for guessing prednisone in patients who either had serious adverse events or experienced nosocomial infections and/or empyema were 2.52 (0.89–7.10; p = 0.08) and 1.63 (0.66–4.03; p = 0.29), respectively. In patients with persisting fever the OR was 0.86 (0.54–1.36, p = 0.51). In patients with relapse there was also no difference in guesses (OR 0.67 (0.22–2.04, p = 0.48).
|Predictor||Odds ratio (95% CI) for guessing prednisone||p-value|
|Time to clinical stability||0.95 (0.91–0.99)||0.02*|
|Hyperglycaemia (n = 90)||3.77 (2.39–5.95)||<0.001***|
|Serious adverse events (n = 13)||2.52 (0.89–7.10)||0.08|
|Infection or empyema (n = 23)||1.63 (0.66–4.03)||0.29|
|Persisting fever (n = 152)||0.86 (0.54–1.36)||0.51|
|Relapse (n = 15)||0.67 (0.22–2.04)||0.48|
Overall, we confirmed that patient blinding was achieved in this study, as successful blinding was defined as having 50% or less correct guesses . Blinding is especially important in clinical trials as the Hawthorne and/or placebo/nocebo effects could lead to an alteration in a patient’s health behaviour and, therefore, in outcome [11, 12]. An interesting finding in this study was that shorter time to clinical stability correlated with guessing prednisone, indicating patients’ and physicians’ expectations in this trial.
The majority of patients was uncertain or made a wrong guess about their treatment allocation, independently of their actual treatment. Even though physicians more often made a correct guess, this was not as pronounced as expected, even though physicians had the possibility to observe the patient at every stage of the disease and had full knowledge of their clinical and laboratory parameters. Moreover, in the prednisone group, only nearly half of the guesses by the treating physicians were correct. It was only in the placebo group that the treating physicians guessed correctly in two thirds of the cases. These results are surprising, as prednisone is known to dysregulate glycaemic control and to have immunosuppressive effects [13, 14]. Also, these findings stand in contrast to the results published by the authors mentioned in the Introduction section, in which 75–80% of patients or physicians correctly guessed their treatment [2–7]. There could be several reasons for this discrepancy. First, those studies were performed in long-term settings. It was shown that the longer a treatment is given, the clearer the treatment allocation becomes . Our trial tested a short-duration treatment of 7 days. Second, patients with chronic diseases have plenty of experience with the course of their disease, compared with patients who have suffered only once or twice from a certain disease, as is the case with community-acquired pneumonia. Third, antidepressant drugs are potent psychopharmaceuticals showing strong beneficial effects but also major side effects. Side effects are an important predictor for guessing accuracy [1, 4]. In contrast, prednisone given over a short period of 7 days does not have many side effects except for transient hyperglycaemia .
Interestingly, if patients in the prednisone group were certain about their guess, the probability of guessing prednisone was higher, which was also true for physicians. This is probably as a result of prednisone-induced hyperglycaemia and/or a psychological steroid effect for the patients. Although hyperglycaemia was more prevalent in the prednisone group, it also had a high prevalence in the placebo group . In conformity with these assumptions, we showed that hyperglycaemia is a predictor for guessing prednisone, independently of actual treatment allocation. Although our predictor analysis showed a tendency for physicians and patients to guess prednisone when corticosteroid-related adverse events such as nosocomial infection, delirium or gastrointestinal bleeding occur, these findings were not significant. However, considering the large confidence intervals, the lack of statistical significance could be due to the small number of patients experiencing serious adverse events.
This study has several limitations. First, this was a subanalysis and no sample size calculation was performed. However, the 95% CIs that do not include 50% provide evidence that the sample size was large enough for the primary research question of this article (treatment estimation accuracy). Furthermore, compared with the other trials cited in the introduction, this trial had the second largest sample size of treatment allocation estimates. Another limitation relates to prediction of the physicians’ treatment estimation accuracy. The n = 358 physician guesses did not come from individual physicians. Therefore, it would have been interesting to know whether making a correct guess was associated with the years of experience and other personal factors. However, physician-specific data are lacking.
In conclusion, we showed that sufficient patient and – to a lesser extent – physician blinding was achieved in a study of short-term prednisone vs placebo. These results may be extrapolated to similar interventional studies where administration time of the study drug is short, and drug-related side effects cannot clearly be distinguished from disease-specific symptoms. The predictor analysis showed that the influence of not only positive but also negative expectations about a study drug need to be prevented by blinding, as side effects may arise owing to not only the study drug but also the disease itself. We therefore propose that assessing potential unblinding through correctly guessing treatment allocation (due to effects or side effects of the verum) in the course of a double-blind RCT should be implemented when one of the following conditions are met: (a) investigation of a chronic disease (such as chronic psychiatric or pain conditions);(b) long treatment duration (for example, >7 days);(c) known drug-specific side effects (such as with antidiabetic, cytostatic, or diuretic agents); and (d) no comparison with an active substance with a similar therapeutic effect (placebo control only).
We gratefully thank the staff of the emergency departments and medical wards of all participating hospitals in supporting this study. Furthermore, we thank the many supporters, study and laboratory personnel at all participating centres who have made this trial possible, especially Cemile Bathelt, Katharina Regez, Ursula Schild, Sonja Schwenne, Merih Guglielmetti, Kristina Schumacher, Fausta Chiaverio, Renate Hunziker, Ursina Minder and Nicole Salvisberg.
MCC and CAB are joint senior authors.
The original study was supported by a grant by the Swiss National Foundation (PP0P3_123346) to MCC and the Nora van Meeuwen Häfliger Stiftung and the Gottfried Julia Bangerter-Rhyner Stiftung. CB is supported by the Swiss National Science Foundation (P155212) and a Young Independent Investigator Award of the Swiss Society of Endocrinology and Diabetology. MB is supported by santésuisse and the Gottfried and Julia Bangerter-Rhyner Foundation. PS is supported by the Swiss National Science Foundation (SNSF Professorship, PP00P3_150531 / 1). BM is supported by the Swiss National Science Foundation (Grant 407440_167376). BM, CB, and PS were supported for this study by the research funds from the Department of Endocrinology, Diabetology and Metabolism, Medical University Clinic of the Kantonsspital Aarau and the “Argovia Professorship” of the Medical Faculty of the University of Basel.
Potential competing interests
No potential conflict of interest relevant to this article was reported.
Dr Claudine Angela Blum, MD, Medical University Clinic, Kantonsspital Aarau, Tellstrasse 25, CH-5001 Aarau, claudine.blum[at]ksa.ch
1 White K, Kando J, Park T, Waternaux C, Brown WA. Side effects and the “blindability” of clinical drug trials. Am J Psychiatry. 1992;149(12):1730–1. doi:. http://dx.doi.org/10.1176/ajp.149.12.1730 PubMed
2 Margraf J, Ehlers A, Roth WT, Clark DB, Sheikh J, Agras WS, et al.How “blind” are double-blind studies?J Consult Clin Psychol. 1991;59(1):184–7. doi:. http://dx.doi.org/10.1037/0022-006X.59.1.184 PubMed
3 Rabkin JG, Markowitz JS, Stewart J, McGrath P, Harrison W, Quitkin FM, et al.How blind is blind? Assessment of patient and doctor medication guesses in a placebo-controlled trial of imipramine and phenelzine. Psychiatry Res. 1986;19(1):75–86. doi:. http://dx.doi.org/10.1016/0165-1781(86)90094-6 PubMed
4 Moscucci M, Byrne L, Weintraub M, Cox C. Blinding, unblinding, and the placebo effect: an analysis of patients’ guesses of treatment assignment in a double-blind clinical trial. Clin Pharmacol Ther. 1987;41(3):259–65. doi:. http://dx.doi.org/10.1038/clpt.1987.26 PubMed
5 Schnoll RA, Epstein L, Audrain J, Niaura R, Hawk L, Shields PG, et al.Can the blind see? Participant guess about treatment arm assignment may influence outcome in a clinical trial of bupropion for smoking cessation. J Subst Abuse Treat. 2008;34(2):234–41. doi:. http://dx.doi.org/10.1016/j.jsat.2007.04.004 PubMed
6 Turner JA, Jensen MP, Warms CA, Cardenas DD. Blinding effectiveness and association of pretreatment expectations with pain improvement in a double-blind randomized controlled trial. Pain. 2002;99(1-2):91–9. doi:. http://dx.doi.org/10.1016/S0304-3959(02)00060-X PubMed
7 Carroll KM, Rounsaville BJ, Nich C. Blind man’s bluff: effectiveness and significance of psychotherapy and pharmacotherapy blinding procedures in a clinical trial. J Consult Clin Psychol. 1994;62(2):276–80. doi:. http://dx.doi.org/10.1037/0022-006X.62.2.276 PubMed
8 Blum CA, Nigro N, Winzeler B, Suter-Widmer I, Schuetz P, Briel M, et al.Corticosteroid treatment for community-acquired pneumonia--the STEP trial: study protocol for a randomized controlled trial. Trials. 2014;15(1):257. doi:. http://dx.doi.org/10.1186/1745-6215-15-257 PubMed
9 Blum CA, Nigro N, Briel M, Schuetz P, Ullmer E, Suter-Widmer I, et al.Adjunct prednisone therapy for patients with community-acquired pneumonia: a multicentre, double-blind, randomised, placebo-controlled trial. Lancet. 2015;385(9977):1511–8. doi:. http://dx.doi.org/10.1016/S0140-6736(14)62447-8 PubMed
10 Desbiens NA. In randomized controlled trials, should subjects in both placebo and drug groups be expected to guess that they are taking drug 50% of the time?Med Hypotheses. 2002;59(3):227–32. doi:. http://dx.doi.org/10.1016/S0306-9877(02)00205-0 PubMed
11 Benedetti F, Carlino E, Piedimonte A. Increasing uncertainty in CNS clinical trials: the role of placebo, nocebo, and Hawthorne effects. Lancet Neurol. 2016;15(7):736–47. doi:. http://dx.doi.org/10.1016/S1474-4422(16)00066-1 PubMed
12 Jakovljevic M. The placebo-nocebo response: controversies and challenges from clinical and research perspective. Eur Neuropsychopharmacol. 2014;24(3):333–41. doi:. http://dx.doi.org/10.1016/j.euroneuro.2013.11.014 PubMed
13 Pagano G, Cavallo-Perin P, Cassader M, Bruno A, Ozzello A, Masciola P, et al.An in vivo and in vitro study of the mechanism of prednisone-induced insulin resistance in healthy subjects. J Clin Invest. 1983;72(5):1814–20. doi:. http://dx.doi.org/10.1172/JCI111141 PubMed
14 Busillo JM, Cidlowski JA. The five Rs of glucocorticoid action during inflammation: ready, reinforce, repress, resolve, and restore. Trends Endocrinol Metab. 2013;24(3):109–19. doi:. http://dx.doi.org/10.1016/j.tem.2012.11.005 PubMed
15 Popovic M, Blum CA, Nigro N, Mueller B, Schuetz P, Christ-Crain M. Benefit of adjunct corticosteroids for community-acquired pneumonia in diabetic patients. Diabetologia. 2016;59(12):2552–60. doi:. http://dx.doi.org/10.1007/s00125-016-4091-4 PubMed
Published under the copyright license CC BY-NC-SA: This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.