Funding characteristics of randomised clinical trials supported by the Swiss National Science Foundation: a retrospective cohort study

DOI: https://doi.org/10.4414/smw.2018.14587

Introduction

Randomised clinical trials (RCTs) are the gold standard for assesing healthcare interventions and a cornerstone of evidence-based medicine [1, 2]. Over the last decade, new regulations and guidelines for RCTs have been published, including initiatives to better protect research participants, improve trial methodology and harmonise research across countries [3–5]. These initiatives may have increased research quality, but also increased trial complexity and cost.

An estimated USD 99.6 billion was spent on biomedical research by the public sector worldwide in 2012 [6]. Funding in biomedical research has more than doubled from USD 37.1 billion in 1994 to USD 94.3 billion in 2003, and it has been estimated that clinical trial research costs are increasing by 7.5% per year above the rate of inflation [4, 7, 8]. Most clinical research is funded by industry (nearly 75% of US trials in 2000) [9]. Public-sector funded research has increased over the past 10 years. In 2012, it accounted for 37% of the global expenditure on biomedical research [6]. A UK study analysed 122 RCTs funded by two major public healthcare sponsors between 1994 and 2002. Grant sum per participant ranged from GBP 16 (CHF 36) up to GBP 4522 (CHF 10 245) with a median of GBP 641 (CHF 1452) [10].

At the same time, a substantial amount of public research money is wasted because results of RCTs (or lessons learned from discontinued RCTs) are not published. Across medical disciplines and countries, the proportion of unpublished trials is high and ranges between 30 and 60% [11]. Expressing the resulting waste in monetary terms is difficult because data on funding characteristics of unpublished RCTs are rarely available.

In a previous publication, we showed that 26 out of 101 (25%) RCTs supported by the Swiss National Science Foundation (SNSF) were prematurely discontinued owing to poor recruitment of study participants and 40% were not published in a peer-reviewed scientific journal [12]. In the present article we complement our findings with the funding characteristics of these 101 SNSF-supported RCTs. We aimed to describe funding characteristics of published and unpublished RCTs supported by the SNSF, to quantify the amount of money spent for unpublished studies, and to compare our results with a similar study performed in the UK.

Materials and methods

Ethics approval

None was necessary because this study examined proposals approved by the Swiss National Science Foundation and corresponding publications. Patients or patient data were not involved.

Study design and data source

We established a retrospective cohort of SNSF-supported RCTs for which recruitment and funding had ended in 2015 or earlier. We systematically searched titles and abstracts provided on the SNSF’s p³ research database (p3.snf.ch), which contains all research projects that the SNSF approved and supported fully or partially. We excluded RCTs that were supported through personal grants and that were carried out exclusively outside Switzerland (i.e., SNSF mobility grants), RCTs that were never started, and RCTs for which recruitment or funding was still ongoing at the time of the cut-off date (30 April 2015). The search strategy included synonyms for “randomisation” and “trial” and was limited to human healthcare categories using the database’s “advanced search” option. (appendix 1 ).

For each eligible RCT, two methodologically trained investigators independently searched corresponding publications through information available in the SNSF’s p³ research database, electronic databases (Medline, Embase, Cochrane Central Register of Controlled Trials, Google Scholar), and trial registries, using relevant keywords for study design, population, interventions, and outcome if applicable. Furthermore, we screened personal websites of principal investigators and co-investigators, and reference lists of related articles. In addition, we approached all principal investigators of included RCTs using an online questionnaire. We asked investigators for information about funding characteristics (additional funding sources, estimation of overall cost of their trial, estimation of percentage of SNSF support on the whole trial) and completion/publication status of their trial, including the reason for discontinuation. If a principal investigator did not respond, we sent several reminders by email, or a paper version of the questionnaire by regular mail, and eventually tried to call the investigator by telephone or approach a co-investigator. Sixty-seven of 101 investigators responded to all of the following questions: “Please estimate the total costs of the trial in CHF,” “Please estimate the proportion of total costs covered by the SNSF grant(s) in %,” and “If not 100% funding by the SNSF, please specify the sources of additional funding.” Consequently, the overall minimum response rate was 67%.

Data extraction

After data extractors signed appropriate confidentiality statements, the SNSF granted access to its proposals. In teams of two, data extractors trained in trial methodology extracted data from trial proposals and, if available, from corresponding publications independently and in duplicate. Disagreements were resolved by discussion and consensus. We used a password-protected web-based data extraction tool (www.squiekero.org) that included a detailed manual with instructions for each variable of interest.

Statistical analysis

We describe categorical variables as frequencies and percentages and continuous variables as medians and interquartile ranges (IQRs). We used complete case analysis. The SNSF’s p³ research database includes the approved grant sum per proposal but not per study. Some proposals combined RCTs with other studies such as pilot phases or observational studies. To account for this, we stratified the descriptive analyses by “SNSF grant proposal included a RCT and other projects” vs “SNSF grant proposal included one RCT only”. We were specifically interested in the latter proposals for which we knew that the total grant sum was allocated to the RCT. For SNSF grant proposals that included several projects, we were not able to reliably extract the amount of grant sum specifically used for the RCT.

Comparison with similar data from the UK

Our systematic review of the literature found only one comparable study that allowed us to extrapolate our results: a UK study that included RCTs supported by the Medical Research Council (MRC) or the Health Technology Assessment (HTA) programme from 1994–2002 [10]. To compare grant sums we (a) included SNSF-supported RCT proposals that comprised one RCT only and (b) included only SNSF-supported RCTs that started between 1994 and 2002. We used an exchange rate dating back to April 2006 (1 GBP = 2.265 CHF) when the UK study was published.

Results

Trial selection and characteristics of included trials

The search of SNSF’s p³ research database yielded 455 proposals, of which a total of 101 were eligible healthcare RCTs, conducted between 1986 and 2015 (fig. 1). Twenty-four proposals included other projects in addition to the RCT. The median study size was 138 (interquartile range [IQR] 76–400), and the median planned duration of recruitment was 14 months (IQR 12–21 months). Most RCTs were single-centre, parallel-group trials testing a medication or behavioural intervention in adults. Only 53% were published in a peer-reviewed scientific journal and 30% were prematurely discontinued owing to poor recruitment of study participants. Of the 52 trials approved after the ICMJE recommendation to prospectively register all RCTs (i.e., in 2005 or later) [13], only 40 (77%) were registered in a trial registry (table 1 ).

Figure 1 Trial inclusion and exclusion

Funding characteristics of included trials

Table 2 summarises the funding characteristics of the included RCTs. Investigator-initiated RCTs received a median total of CHF 222 000 (IQR 166 000–276 000) from the SNSF, with a median of CHF 1600 (IQR 600–3000) per planned participant. According to investigators’ statements in the survey, the median total cost of RCTs was CHF 428 000 (IQR CHF 282 000–900 000). The SNSF grant sum covered a median of 67% (IQR 40–80%) of the total trial cost. Seventy-four investigators responded to the specific survey question on whether additional funding was obtained. Fifty-two (70%) mentioned additional funding, mainly from their own institution or private foundations.

SNSF grant sum stratified by publication status

In a previous publication we reported on discontinuation and nonpublication of these 101 SNSF-supported RCTs [12]. For the analysis reported here, we limited our sample to the SNSF grant proposals that included one RCT only, resulting in a total of 74 eligible RCTs (fig. 1) and stratified the data by publication status. Table 3 summarises characteristics of the subsample of 74 RCTs. We identified all peer-reviewed publications through searching electronic databases with indexed publications. The search of trial registries did not yield any additional publications. Table 4 shows that CHF 6.7 million out of CHF 20 477 189 (33%) of SNSF grant sum spent over the last three decades for grant proposals including one RCT only did not result in a peer-reviewed scientific publication.

Comparison with the UK trial cohort

McDonald et al. reported that grant sums ranged from GBP 6 (CHF 36) to GBP 4522 (CHF 10 245) per participant, with a median of GBP 641 (CHF 1452) per participant. Twenty-nine percent of the RCTs had more than GBP 1000 (CHF 2270) per participant available, which the authors considered a sufficient level of funding.

To enhance the comparability of this exploratory analysis, we restricted the sample to the 14 proposals that included one RCT only and were started between 1994 and 2002 (fig. 1). The SNSF grant sum per RCT was similar to that of the MRC and HTA programme in the UK, with a median of CHF 1303 per participant and 3 of the 14 RCTs (21%) receiving more than the amount per participant deemed sufficient in the UK (table 5).

Discussion

Summary of findings

In the last three decades, SNSF-supported RCTs were granted, on average, CHF 222 000, covering 67% of the estimated total trial costs. Most applicants mentioned additional funding from their own institution or private foundations. CHF 6.7 Mio or 33% of the SNSF funding total granted for RCTs did not result in peer-reviewed publications.

Strengths and limitations

This is the first study assessing funding characteristics of publicly funded RCTs in Switzerland. After signing appropriate confidentiality statements, we had access to proposals for all healthcare RCTs supported by the SNSF without restriction by the funder or the applicants [14]. Methodologically trained reviewers systematically identified subsequent publications and extracted data independently and in duplicate. We combined information from several sources (grant proposals, trial registries, publications and investigator survey) in order to increase the accuracy of the data set.

Limitations of our study include data missing owing to the low reporting quality of some older RCT proposals, inability to extract individual RCT grant sums from combined proposals, or lack of response of investigators to our survey. However, our overall response rate of 67% is still sufficient to approximate average total cost per RCT and in line with other similar studies surveying biomedical researchers [15, 16]. Our comparison with the UK study had not enough power for inferential statistical comparisons because we restricted our sample to the same time period as the UK study. We were not able to compare our results with more recent data from the UK because more updated funding characteristics were not available [17, 18]. Apart from the multivariable logistic regression analysis in a previous publication [12], we did not further explore the reasons for nonpublication.

Implications and conclusion

The SNSF is mandated by the Swiss Federal Government and thus funded by the public. The SNSF invested CHF 6.7 million (not inflation-adjusted) in RCTs that were never published in peer-reviewed scientific journals. We were not able to quantify the absolute amount of money spent for these unpublished RCTs, including costs such as those associated with ethical review process. We were not able to elucidate how the money was spent in detail. We acknowledge that even if an RCT were not published, its conduct might have had positive and important consequences, such as capacity building within a research group or development of follow-up studies. However, the absence of results from these SNSF-supported RCTs in peer-reviewed scientific journals is worrisome. Nonpublication is not only a waste of precious resources, but also compromises systematic reviews and meta-analyses, and undermines the public’s and in particular the patients’ trust in clinical research and motivation to participate. We found no evidence that the proportion of unpublished RCTs has changed over the last 20 years [12].

Public funding agencies should have a genuine interest in ensuring that results, data and lessons learned from funded RCTs are made available to the scientific community and that ultimately the public will benefit from them. This can be achieved through dissemination of results or sharing of data in data repositories or trial registries and supporting initiatives such as AllTrials (www.alltrials.net). Unfortunately, even after the ICMJE official recommendation to prospectively register all RCTs, only 77% were registered in a trial registry. Public funding agencies should help endorse publication policies by making adherence to standards of publication and data sharing a condition of transfer of funds to investigators on (for example, adherence to a specific deadline after trial completion, such as 18 months, as suggested by the Institute of Medicine [19]).

The average grant sums per RCT spent by SNSF and MRC/HTA programme between 1994 to 2002 were similar. McDonald et al. showed that 45% of the MRC/HTA-supported RCTs failed to recruit 80% of the target sample size and hence were at risk for premature discontinuation [10]. The proportion of MRC/HTA-supported RCTs failing to recruit 80% of their target decreased in the period between 2002 and 2008, but still accounted for 22% [17]. Results from our study were similar, in that one out of four SNSF-supported RCTs were prematurely discontinued because of slow recruitment of study participants [12]. Since adequate funding is a crucial factor to support recruitment, one could hypothesise that in both Switzerland and the UK the grant sums provided by the main public funding bodies for RCTs were not sufficient. The SNSF has acknowledged the need to better fund RCTs and started a new funding track in 2015, called the Investigator Initiated Clinical Trial (IICT) programme, which might lead to better publication outcomes. Apart from sufficient funding, realistic projection of the target sample size and adequate planning of the recruitment process seem to be the main drivers for successful trial completion [20]. Other reasons for recruitment problems related to the Swiss healthcare system have recently been compiled by Briel et al. in a qualitative study [21].

In conclusion, publicly funded RCTs in Switzerland that were not published subsequently are common and used about a third of the SNSF funding granted for RCTs over the last three decades. Rigorous measures are necessary to ensure successful recruitment and publication. The new SNSF funding track for investigator-initiated RCTs aims to improve the situation but has yet to be evaluated.

Appendix 1 Search algorithm for Swiss National Science Foundation p³ research database

We searched titles and abstracts provided on the research project database of the SNSF (p³ research database, p3.snf.ch) for human healthcare RCTs using the “advanced search” option and the following key words: random* OR trial OR Zufallsprinzip.

We limited the results to the subject category “Biology and Medicine” and “Health” (under “Humanities and Social Sciences / Sociology”) and chose all “Funding Scheme” except “r4d – Ecosystems”, “r4d - Employment”, “r4d – Food Security”, “r4d - Open Call 1”, “r4d - Social Conflicts”, “Infrastructure”, “Science communication”. No further search restrictions were applied.

Acknowledgments

We would like to thank the administrative staff of Division III Biology and Medicine of the Swiss National Science Foundation for their support and assistance with the provision of eligible proposals. We are grateful to all applicants who responded to our survey and provided additional information about their trials.

References

1Ann B. Research Methods In Health: Investigating Health And Health Services. McGraw-Hill Education (UK); 2014. 

2 Devereaux PJ , Yusuf S . The evolution of the randomized controlled trial and its role in evidence-based decision making. J Intern Med. 2003;254(2):105–13. doi:.https://doi.org/10.1046/j.1365-2796.2003.01201.x  

3 Sinha G . European move affects academic trials research. J Natl Cancer Inst. 2006;98(16):1100–1. doi:.https://doi.org/10.1093/jnci/djj349  

4 Eisenstein EL , Collins R , Cracknell BS , Podesta O , Reid ED , Sandercock P , et al. Sensible approaches for reducing clinical trial costs. Clin Trials. 2008;5(1):75–84. doi:.https://doi.org/10.1177/1740774507087551  

5 Duley L , Antman K , Arena J , Avezum A , Blumenthal M , Bosch J , et al. Specific barriers to the conduct of randomized trials. Clin Trials. 2008;5(1):40–8. doi:.https://doi.org/10.1177/1740774507087704  

6 Chakma J , Sun GH , Steinberg JD , Sammut SM , Jagsi R . Asia’s ascent--global trends in biomedical R&D expenditures. N Engl J Med. 2014;370(1):3–6. doi:.https://doi.org/10.1056/NEJMp1311068  

7 Holler B , Forgione DA , Baisden CE , Abramson DA , Calhoon JH . Interactive financial decision support for clinical research trials. J Health Care Finance. 2011;37(3):25–37.  

8 DiMasi JA , Hansen RW , Grabowski HG . The price of innovation: new estimates of drug development costs. J Health Econ. 2003;22(2):151–85. doi:.https://doi.org/10.1016/S0167-6296(02)00126-1  

9 Bodenheimer T . Uneasy alliance--clinical investigators and the pharmaceutical industry. N Engl J Med. 2000;342(20):1539–44. doi:.https://doi.org/10.1056/NEJM200005183422024  

10 McDonald AM , Knight RC , Campbell MK , Entwistle VA , Grant AM , Cook JA , et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9. doi:.https://doi.org/10.1186/1745-6215-7-9  

11 Dwan K , Altman DG , Arnaiz JA , Bloom J , Chan AW , Cronin E , et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3(8):e3081. doi:.https://doi.org/10.1371/journal.pone.0003081  

12 Amstutz A , Schandelmaier S , Frei R , Surina J , Agarwal A , Olu KK , et al. Discontinuation and non-publication of randomised clinical trials supported by the main public funding body in Switzerland: a retrospective cohort study. BMJ Open. 2017;7(7):e016216. doi:.https://doi.org/10.1136/bmjopen-2017-016216  

13 De Angelis C , Drazen JM , Frizelle FA , Haug C , Hoey J , Horton R , et al.; International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet. 2004;364(9438):911–2. doi:.https://doi.org/10.1016/S0140-6736(04)17034-7  

14 Chan A-W , Upshur R , Singh JA , Ghersi D , Chapuis F , Altman DG . Waiving confidentiality for the greater good. BMJ. 2006;332(7549):1086–9. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1458595/. Accessed March 27, 2014. doi:.https://doi.org/10.1136/bmj.332.7549.1086  

15 Decullier E , Lhéritier V , Chapuis F . Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ. 2005;331(7507):19. doi:.https://doi.org/10.1136/bmj.38488.385995.8F  

16 Stern JM , Simes RJ . Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315(7109):640–5. doi:.https://doi.org/10.1136/bmj.315.7109.640  

17 Sully BGO , Julious SA , Nicholl J . A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013;14(1):166. doi:.https://doi.org/10.1186/1745-6215-14-166  

18 Walters SJ , Bonacho Dos Anjos Henriques-Cadby I , Bortolami O , Flight L , Hind D , Jacques RM , et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7(3):e015276. doi:.https://doi.org/10.1136/bmjopen-2016-015276  

19Incentives for Clinical Trialists to Share Data - NEJMp1608351. http://www.nejm.org/doi/pdf/10.1056/NEJMp1608351. Accessed December 4, 2017. 

20 Briel M , Olu KK , von Elm E , Kasenda B , Alturki R , Agarwal A , et al. A systematic review of discontinued trials suggested that most reasons for recruitment failure were preventable. J Clin Epidemiol. 2016;80:8–15. doi:.https://doi.org/10.1016/j.jclinepi.2016.07.016  

21 Briel M , Elger B , von Elm E , Satalkar P . Insufficient recruitment and premature discontinuation of clinical trials in Switzerland: qualitative study with trialists and other stakeholders. Swiss Med Wkly. 2017;147:w14556. doi:10.4414/smw.2017.14556.