Figure 1
Key elements of interrupted time series design.
Adapted according to Matowe [26].
DOI: https://doi.org/10.4414/smw.2015.14140
Evidence-based recommendations play an important role in medical decision-making, but barriers to adherence are common [1–3]. Conceptual models for clinical guideline adherence allow a better understanding of threats and success factors for guideline adherence [1]. Such frameworks are often based on collaboration of multidisciplinary groups of clinicians, social-psychologists and marketing experts, and linked to the “theory of planned behaviour” (knowledge – attitudes – behaviour) [4].
In Switzerland, the Swiss Medical Board produces health technology assessment (HTA) reports that conclude with recommendations ( http://www.medical-board.ch ). In these HTA reports, the Swiss Medical Board (SMB) examines diagnostic or therapeutic services if they are controversial but reimbursed under the Swiss social insurance scheme. To be in line with the legal demands, reimbursed services have to be “effective, appropriate and efficient”. Selection of HTA topics is made according to predefined criteria (such as high volume impact or highly debated cost-effectiveness). So far, 12 HTA reports have been published that cover different services (e.g. screening via mammography; statins for primary prevention of cardiovascular disease).
The SMB was founded in 2009 and is now active at the Swiss national level, but is supported by the cantons and not by the central government. The function of the SMB is to produce HTA reports. In the Swiss health care system, the SMB has not the legal competency to formulate mandatory decisions, but it formulates recommendations for clinical providers and other decision makers, as well as providing information for the public. The SMB has no other roles, such as direct involvement in reimbursement decisions. As with other HTA-agencies, such as the National Institute for Health and Care Excellence (NICE) in England [5, 6] or the Swedish Council on Health Technology Assessment (SBU) [7], the SMB needs to know if their recommendations have any impact on clinical practice.
Thus, the funding body of the SMB commissioned an independent impact evaluation to better understand opportunities and possible shortcomings of SMB recommendations. Two SMB reports were selected for impact evaluation as case studies:
– “Rupture of the anterior cruciate ligament of the knee: surgical or conservative treatment?” (published in 2009; updated in 2013)
– “The value of the PSA test for prostate cancer screening” (published in 2011)
Briefly, in the 2009 report about anterior cruciate ligament (ACL) rupture, the SMB concluded that surgical and conservative treatments are similarly effective, but surgical ACL repair is more costly. The SMB recommended the conservative approach, including physical therapy and removal of meniscus or cartilage tears if necessary, as the standard treatment option for most of the ACL rupture patients [8]. In the 2011 report about prostate-specific antigen (PSA) screening, the SMB’s recommendation was not to use PSA screening for prostate cancer in men without symptoms or without family predisposition for prostate cancer [9].
These reports were selected for impact evaluation for a variety of reasons: (i.) they represent a surgical and a non-surgical clinical service of high relevance, (ii.) they cover the therapeutic and the diagnostic (screening) domain, (iii.) the recommendations have both been under intense debate in the clinical community, and (iv.) a sufficiently long time period after publication has passed for impact data to be available.
With this paper, we report the methodological approach of the first SMB impact evaluation and the main quantitative findings.
Firstly, we established a Methodological Expert Group for methodological advice and validation of results. This group was independent from the commissioner and had no steering function. The expert group comprised clinicians (from general practice and traumatology), experts from the social insurance system (mandatory Swiss accident insurance scheme), epidemiological and health services research experts (research area: guideline implementation) as well as an expert on patient advocacy.
Figure 1
Key elements of interrupted time series design.
Adapted according to Matowe [26].
We aimed to assess the impact of the two selected SMB reports on the provision of services in the Swiss healthcare system.
We defined final impact as the number of services provided (PSA test; ACL repair), which may be due to behaviour of clinicians or due to the request of patients themselves. We did not assess behaviour of health professionals via self-declaration in questionnaires, as this may be prone to bias if someone does not agree with the guidelines under study.
We performed a quantitative observational study and assessed quantitative data over time via interrupted times series analyses. Additional details of the study are reported elsewhere [10].
We used two main data sources for services provided:
For the number of PSA screening tests provided, we used the claims data of a major Swiss health insurance company, covering about 20% of the Swiss population (Helsana Versicherungen AG). We analysed the quarterly number of PSA-tests from 2005–2013 in 662,874 outpatients.
For the rate of surgical repair of the ACL in patients with ACL-rupture, we used the claims data of the mandatory Swiss accident insurance scheme (Suva) and some private accident insurers (SSUV database). This database covers about 60% of the Swiss population. We analysed the rate of ACL repair from 1990–2011 in 101,737 patients with knee injury.
We tested the following hypothesis: in the case of a relevant impact of the respective SMB recommendation (i.e. intervention), the number of PSA tests (or the rate of ACL repair) will decline.
For the quantitative analysis, we applied a stepwise approach. First, we visually assessed a scatterplot of the number of performed services over time comprising the key elements of the interrupted time series analysis (fig. 1). We defined the time point of intervention as the publication date of each SMB report. We were interested in the secular trend before intervention (slope pre) and in the secular trend after intervention (slope post). In addition, the absolute change in the number of services (change in level) at the time of intervention was of interest.
Second, we performed a segmented regression analysis. For the PSA analysis, the dependent variable was “number of PSA screening tests per quarter” (explanatory variables: “intervention” [publication of SBM report PSA, November 2011; yes/no], “co-intervention” [publication of American Urologic Association, AUA, guidelines, November 2009; yes/no], “population at risk for PSA screening tests” [number of insured men aged 50‒70 years], “number of inpatients treated for prostate cancer” [to account for the prevalence of prostate cancer and thus for diagnostic PSA tests]).
For the ACL analysis, we calculated the “annual rate of surgical ACL repair” as the “number of patients with surgical ACL repair” divided by the “number of patients with ACL rupture”. This rate served as the dependent variable of the regression analysis (explanatory variable: “intervention” [publication of SBM ACL report, June 2009; yes/no]).
For each regression, we modelled a time trend by a second-order polynomial, included an interaction term for the main intervention, and controlled for seasonal fixed effects (if the data were collected at a higher frequency than annually). Models were optimised according to AIC-statistics, auto-correlation could be rejected (by a Durbin-Watson test), and residuals were inspected for key properties such as homoscedasticity. For inferential analysis we calculated 95% confidence intervals (CIs). No predictive intervals were needed, as we could fully rely on empirical data. Data were analysed with Stata SE 12 (StataCorp 2011, Stata Statistical Software, College Station, TX).
Ethical approval was not required, as our retrospective quantitative analysis was based on fully anonymised claims data.
For the PSA analysis, 34 quarterly data points from 2005 to 2013 were available (fig. 2). Data showed a relevant seasonal variability, with fewer PSA tests during Q3 (July to September), presumably as a result of summer vacations. The secular trend before the intervention showed a continuous but diminishing increase in the number of PSA tests over time, i.e. the growth rate had declined already before the intervention (first-order coefficient: 414, 95% CI: 312‒515; second-order coefficient: –5.44, 95% CI: –8.84 ‒ –2.04]). The trend predicts an increase in PSA tests from the previous period to the following one of 397 for 2005:Q2 and of 147 for 2011:Q1 (“slope pre”, discrete form).
Figure 2
Prostate specific antigen (PSA) tests over time (2005–2013).
Each data point represents the number of PSA tests over 3 months in Helsana Insurance (○ JAN-MAR; □ APR-JUN; ∆ JUL-SEP; ● OCT-DEC). The dashed line is corrected for seasonal effects. Vertical dashed line: Time point of SMB report released in 2011. Blue solid trend line: Trend (95% CI) without Swiss Medical board (SMB) recommendations up to 2013. Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2011.
During the quarter immediately after the intervention, a statistically significant reduction in PSA tests was estimated (“change in level”: –1556, 95% CI: –2679 ‒ –433]). The secular trend during the following periods, however, showed strong variation (coefficient of interaction between first-order time variable and intervention: 113, 95%-CI: –166 ‒ 392; p = 0.41). A return of the trend to its previous path cannot be ruled out. Accordingly, the 95% CIs of the two alternative trend functions, one including the intervention and one discarding it, overlap around the beginning of 2013. Thus, the sustainability of a possible causal effect of the intervention on the number of PSA tests remains unclear.
For the ACL analysis, 22 annual data points from 1990 to 2011 were available. The absolute number of patients with ACL rupture in our data set increased between 1990 (n = 3,341) and 2011 (n = 6,942). The absolute number of surgical ACL repairs increased as well (1990: n = 2,366; 2011: n = 3,481).
However, the rate of surgical ACL repair in patients with ACL rupture, initially reaching about 80%, declined after the late 1990s to approximately 55% in 2009 (fig. 3). This transition to a negative secular trend, which starts off clearly before the intervention, results in a negative second-order coefficient of the polynomial time trend (–0.13 percentage points, 95% CI: –0.16 ‒ –0.09) in combination with a positive first-order coefficient (1.9 percentage points, 95%-CI: 1.2‒2.7).
The rate of ACL repair continued to decline during the two years with available data after the intervention, but the trend after intervention is not significantly different from the secular trend before intervention (change in level: –5.7 percentage points, 95% CI: –16.2 ‒ 4.9; p = 0.27). Thus, no statistically significant relationship can be found between the intervention and an intensification of the downward time trend.
We assessed the impact of SMB recommendations in two case studies. Despite some evidence of a possible change we did not find a sustained and significant impact of SMB recommendations on annual PSA screening tests or on the already declining rate of surgical ACL repair for ACL rupture in Switzerland. However, we had only few post-intervention observations and longer term data are needed to confirm or refute the findings.
Figure 3
Operation rates of the ruptured anterior cruciate ligament (ACL) over time (1990–2011).
Each data point represents the annual operation rate among patients with ruptured ACL (source: SSUV database). Vertical dashed line: Time point of Swiss Medical Board (SMB) report released in 2009. Blue solid trend line: Trend (95% CI) without SMB recommendations up to 2011. Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2009.
Interestingly, already before the SMB recommendations there was a declining growth rate for PSA screening tests in our data set. Although we adjusted the PSA data for epidemiological trends and cointerventions, other less well defined factors, such as the ongoing debate about the benefit of PSA screening for prostate cancer, may have influenced the trend before intervention.
This possibility is even more pronounced for the operation rates due to ACL rupture. A negative secular trend with decreasing rates clearly started before the publication of the SMB report. No ACL guidelines, which might have acted as a cointervention, exist for Switzerland. However, the intensive debate about the suitable therapy for ACL rupture over several years in the scientific community during clinical symposiums and congresses may have led to changes in clinical practice and thus a decline in operation rates for ACL rupture already before the SMB report.
Whereas some impact evaluations have only assessed knowledge and attitudes [11, 12], others have focused on case load or frequency of services (e.g. for mammography screening [13, 14] or drug monitoring in epilepsy treatment [15]). For example, the analysis of self-reported mammography screening rates of women aged 40 to 49 years in the United States showed no decline from 2005 to 2011, despite the 2009 recommendation of the US Preventive Services Task Force not to screen this age group for breast cancer.
We are not aware of a published impact evaluation of recommendations concerning surgical ACL repair. A US study assessed the impact of a recommendation not to screen men over 75 years with PSA tests [16]. The author analysed data of a cancer registry and concluded that the recommendations led to a small decline in PSA screening rates. On the other hand, a study based on self-reported PSA screening tests of US men showed that the effect of recommendations against screening had been minimal at best [17].
The Swedish SBU agency has recently assessed the impact of a broad spectrum of their released HTA reports [7]. A mixed methods approach and time series data were used. The authors assessed different outcome domains and concluded that the selected HTA cases had a high impact on clinical guidelines. Concerning the impact on clinical practice they found some SBU HTA reports with high estimated impact (e.g. in dental care), but also some with low estimated impact (e.g. for vaccination of children) [7].
This study has for the first time assessed the impact of SMB recommendations on the provision of services in the Swiss healthcare system. In our model, we considered secular time trends for the quantitative analysis.
Our study has several limitations. First, it is an observational study and confounding, e.g. by cointerventions, cannot be excluded, despite our attempt to control for several variables. Second, the database for our impact evaluation is small. In Switzerland, routine data for medical services are often poorly documented or were not suitable for our purposes. Thus, no data could be used, for example, from the Swiss national statistics for in-hospital treatment, from other private accident insurances or from big medical laboratories that cover the outpatient sector for PSA tests. Furthermore, ACL data were only available until 2011. Thus, only a short time interval with two data points after intervention in 2009 could be analysed for a possible impact on care. Third, we had to make several assumptions, as codes for PSA tests, International Classification of Diseases (ICD) codes and coding of surgical procedures have been modified over the past years. However, we thoroughly scrutinised data together with the data owner to minimise misclassification. Finally, we only included two SMB recommendations and it is unclear if our findings are transferable to other SMB reports.
Change of clinical behaviour is subject to complex factors. In Switzerland, for example, an intense debate about the meaningfulness of the SMB recommendations for clinical practice emerged [18–20].
Several success factors have been discussed in the literature to increase the impact of recommendations and one conceptual model proposes a “knowledge-to-action” cycle [2], which includes impact evaluations, such as our study. Other approaches include efficient dissemination strategies (e.g. via adapted communication strategies for target groups); visibility of, and trust in, HTA bodies (e.g. via social marketing activities [21] and involvement of medical opinion leaders to create effective advocates for final reports [7]); involvement of citizens and patients (e.g. via cooperation with trustworthy consumer groups [21]); active stakeholder management and alliances with current activities against overtreatment (internationally: “choosing wisely” [22]; in Switzerland: “smarter medicine” [23]). A recent report of the SMB, for example, has fuelled the debate about overtreatment, also internationally [24, 25].
Improved accessibility of routine care data for impact evaluations remains another important issue, the aim being to close the gap from dissemination of recommendations to generation of reliable knowledge about their impact in practice. For example, follow-up impact assessments could include more post-intervention data and allow more precise impact estimations.
Impact evaluation is an important tool for evidence-based Swiss health services, as it allows a better understanding of the magnitude of impact of clinical practice recommendations. Further monitoring is needed to assess if recommendations disseminated by the SMB have a relevant impact on clinical practice in Switzerland.
Acknowledgements: We are grateful for the valuable support of the Methodological Expert Group for methodological advice and validation of results. We thank Oliver Reich and his team (Helsana Versicherungen AG), as well as Stefan Scholz (SSUV database) for the valuable cooperation in providing and preparing the PSA and ACL data. Thanks also to Marcel Dettling, Institute of Data Analysis and Process Design, Zurich University of Applied Sciences, for statistical advice; to Linda Tecklenburg for her contributions during literature search and data screening; to Armin Ledergerber, Institute of Marketing Management, Zurich University of Applied Sciences, for his useful advice in marketing issues; to Paul Kelly, Language Competence Centre, Zurich University of Applied Sciences, for English proof reading.
1 Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458–65.
2 Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.
3 Eichler K, Zoller M, Tschudi P, Steurer J. Barriers to apply cardiovascular prediction rules in primary care: a postal survey. BMC Fam Pract. 2007;8:1.
4 Gurses AP, Marsteller JA, Ozok AA, Xiao Y, Owens S, Pronovost PJ. Using an interdisciplinary approach to identify factors that affect clinicians' compliance with evidence-based guidelines. Crit Care Med. 2010;38(8 Suppl):S282–91.
5 Thornhill MH, Dayer MJ, Forde JM, Corey GR, Chu VH, Couper DJ, et al. Impact of the NICE guideline recommending cessation of antibiotic prophylaxis for prevention of infective endocarditis: before and after study. BMJ. 2011;342:d2392.
6 Roberts VI, Esler CN, Harper WM. What impact have NICE guidelines had on the trends of hip arthroplasty since their publication? The results from the Trent Regional Arthroplasty Study between 1990 and 2005. J Bone Joint Surg Br. 2007;89(7):864–7.
7 Rosen M, Werko S. Does health technology assessment affect policy-making and clinical practice in sweden? Int J Technol Assess Health Care. 2014;30(3):265–72.
8 Gesundheitsdirektion des Kantons Zürich. Ruptur des vorderen Kreuzbandes: operative oder konservative Behandlung? Fachbericht. Zürich: Medical Board, 2009.
9 Swiss Medical Board. Stellenwert des PSA-Wertes bei der Früherkennung des Prostatakarzinoms. Fachbericht. Zürich: Swiss Medical Board, 2011.
10 Eichler K, Hess S, Tecklenburg L, Ledergerber A, Riguzzi M, Brügger U. Swiss Medical Board: Impact Evaluation. Schlussbericht: Winterthurer Institut für Gesundheitsökonomie, Zürcher Hochschule für Angewandte Wissenschaften, 2014.
11 Overmeer T, Linton SJ, Holmquist L, Eriksson M, Engfeldt P. Do evidence-based guidelines have an impact in primary care? A cross-sectional study of Swedish physicians and physiotherapists. Spine. (Phila Pa 1976) 2005;30(1):146–51.
12 Sondergaard J, Andersen M, Stovring H, Kragstrup J. Mailed prescriber feedback in addition to a clinical guideline has no impact: a randomised, controlled trial. Scand J Prim Health Care. 2003;21(1):47–51.
13 Howard DH, Adams EK. Mammography rates after the 2009 US Preventive Services Task Force breast cancer screening recommendation. Prev Med. 2012;55(5):485–7.
14 Pace LE, He Y, Keating NL. Trends in mammography screening rates after publication of the 2009 US Preventive Services Task Force recommendations. Cancer. 2013;119(14):2518–23.
15 Minshall I, Berry D, Smith D. The impact of an educational intervention, the New GP Contract and NICE guidelines on anti-epilepsy therapeutic drug monitoring. Seizure. 2011;20(2):126–30.
16 Howard DH. Declines in prostate cancer incidence after changes in screening recommendations. Arch Intern Med. 2012;172(16):1267–8.
17 Sammon JD, Pucheril D, Diaz M, Kibel AS, Kantoff PW, Menon M, et al. Contemporary Nationwide Patterns of Self-reported Prostate-Specific Antigen Screening. JAMA internal medicine 2014;174(11):1839–41.
18 Recker F, Sulser T. Swiss Medical Board oder: “das Kind mit dem Bade ausschütten”. Schweiz Ärzteztg. 2011;92(51/52):2002–04.
19 Swiss Medical Board. Stellungnahme des Swiss Medical Board zur Kritik an seiner Empfehlung zum PSA-Test Swiss Medical Board oder: «von der Angemessenheit des PSA-Tests». Schweiz Ärzteztg. 2012(93: 3):83–85.
20 Dubler DmO. Briefe an die SÄZ; Keine Statistiken, sondern Menschen. Schweiz Ärzteztg. 2012(93: 8):281–83.
21 Hurley R. Can doctors reduce harmful medical overuse worldwide? BMJ. 2014;349:g4289.
22 Choosing Wisely. Unnecessary Tests and Procedures In the Health Care System. 2014. http://www.choosingwisely.org/wp-content/uploads/2014/04/042814_Final-Choosing-Wisely-Survey-Report.pdf.
23 Selby K, Cornuz J, Neuner-Jehle S, Meier C, Rodondi N, Gaspoz J. “Smarter Medicine”: 5 Interventionen, die in der ambulanten allgemeinen inneren Medizin vermieden werden sollten. Schweiz Ärzteztg. 2014;95(20):769.
24 Arie S. Switzerland debates dismantling its breast cancer screening programme. BMJ. 2014;348:1625.
25 Biller-Andorno N, Juni P. Abolishing mammography screening programs? A view from the Swiss Medical Board. N Engl J Med. 2014;370(21):1965–7.
26 Matowe L, Ramsay CR, Grimshaw JM, Gilbert FJ, Macleod MJ, Needham G. Effects of Mailed Dissemination of the Royal College of Radiologists’ Guidelines on General Practitioner Referrals for Radiography: A Time Series Analysis. Clinical Radiology. 2002(57):575–78.
Previous presentation: A preliminary version of this work was presented as a poster at the 3rd Symposium “Health Services Research”: November 04, 2014; Berne, Switzerland. This symposium was supported by the Swiss Academy of Medical Sciences.
Disclosures: The study was commissioned by the Funding Body of the Swiss Medical Board. The funding source had no influence on design of the evaluation; on the selection, analysis, and interpretation of the data; on the writing of the manuscript; and the decision to submit the manuscript for publication.