Impact evaluation of Swiss Medical Board reports on routine care in Switzerland : a case study of PSA screening and treatment for rupture of anterior cruciate ligament

QUESTIONS UNDER STUDY: Evidence-based recommendations play an important role in medical decisionmaking, but barriers to adherence are common. In Switzerland, the Swiss Medical Board (SMB) publishes evidence reports that conclude with recommendations. We assessed the impact of two SMB reports on service provision (2009: Recommendation of conservative treatment as first option for rupture of the anterior cruciate ligament of the knee; 2011: Recommendation against PSA screening for prostate cancer). METHODS: We performed an observational study and assessed quantitative data over time via interrupted times series analyses. The primary outcome was the quarterly number of performed prostate-specific antigen (PSA) tests and the annual rates of surgical ACL repair in patients with ACL rupture. Data were adjusted for time trends and relevant confounders. RESULTS: We analysed PSA tests in 662,874 outpatients from 2005–2013 and treatment data in 101,737 patients with knee injury from 1990–2011. For the number of PSA tests, the secular trend before the intervention showed a continuous but diminishing increase over time. A statistically significant reduction in tests was estimated immediately after the intervention, but a later return to the trend before the intervention cannot be ruled out. The rate of surgical ACL repair had already declined after the late 1990s to about 55% in 2009. No relevant additional change emerged in this secular trend after the intervention. CONCLUSIONS: Despite some evidence of a possible change, we did not find a sustained and significant impact of SMB recommendations in our case study. Further monitoring is needed to confirm or refute these findings.


Introduction
Evidence-based recommendations play an important role in medical decision-making, but barriers to adherence are common [1][2][3].Conceptual models for clinical guideline adherence allow a better understanding of threats and success factors for guideline adherence [1].Such frameworks are often based on collaboration of multidisciplinary groups of clinicians, social-psychologists and marketing experts, and linked to the "theory of planned behaviour" (knowledge -attitudes -behaviour) [4].In Switzerland, the Swiss Medical Board produces health technology assessment (HTA) reports that conclude with recommendations (www.medical-board.ch).In these HTA reports, the Swiss Medical Board (SMB) examines diagnostic or therapeutic services if they are controversial but reimbursed under the Swiss social insurance scheme.To be in line with the legal demands, reimbursed services have to be "effective, appropriate and efficient".Selection of HTA topics is made according to predefined criteria (such as high volume impact or highly debated cost-effectiveness).So far, 12 HTA reports have been published that cover different services (e.g.screening via mammography; statins for primary prevention of cardiovascular disease).The SMB was founded in 2009 and is now active at the Swiss national level, but is supported by the cantons and not by the central government.The function of the SMB is to produce HTA reports.In the Swiss health care system, the SMB has not the legal competency to formulate mandatory decisions, but it formulates recommendations for clinical providers and other decision makers, as well as providing information for the public.The SMB has no other roles, such as direct involvement in reimbursement decisions.As with other HTA-agencies, such as the National Institute for Health and Care Excellence (NICE) in England [5,6] or the Swedish Council on Health Technology Assessment (SBU) [7], the SMB needs to know if their recommendations have any impact on clinical practice.Thus, the funding body of the SMB commissioned an independent impact evaluation to better understand opportunit-ies and possible shortcomings of SMB recommendations.Two SMB reports were selected for impact evaluation as case studies: -"Rupture of the anterior cruciate ligament of the knee: surgical or conservative treatment?"(published in 2009; updated in 2013) -"The value of the PSA test for prostate cancer screening" (published in 2011) Briefly, in the 2009 report about anterior cruciate ligament (ACL) rupture, the SMB concluded that surgical and conservative treatments are similarly effective, but surgical ACL repair is more costly.The SMB recommended the conservative approach, including physical therapy and removal of meniscus or cartilage tears if necessary, as the standard treatment option for most of the ACL rupture patients [8].In the 2011 report about prostate-specific antigen (PSA) screening, the SMB's recommendation was not to use PSA screening for prostate cancer in men without symptoms or without family predisposition for prostate cancer [9].These reports were selected for impact evaluation for a variety of reasons: (i.) they represent a surgical and a nonsurgical clinical service of high relevance, (ii.) they cover the therapeutic and the diagnostic (screening) domain, (iii.) the recommendations have both been under intense debate in the clinical community, and (iv.) a sufficiently long time period after publication has passed for impact data to be available.With this paper, we report the methodological approach of the first SMB impact evaluation and the main quantitative findings.

Methods
Firstly, we established a Methodological Expert Group for methodological advice and validation of results.This group was independent from the commissioner and had no steering function.The expert group comprised clinicians (from general practice and traumatology), experts from the social insurance system (mandatory Swiss accident insurance scheme), epidemiological and health services research experts (research area: guideline implementation) as well as an expert on patient advocacy.

Figure 1
Key elements of interrupted time series design.Adapted according to Matowe [26].

Aim of the study
We aimed to assess the impact of the two selected SMB reports on the provision of services in the Swiss healthcare system.

Outcome
We defined final impact as the number of services provided (PSA test; ACL repair), which may be due to behaviour of clinicians or due to the request of patients themselves.We did not assess behaviour of health professionals via self-declaration in questionnaires, as this may be prone to bias if someone does not agree with the guidelines under study.

Design
We performed a quantitative observational study and assessed quantitative data over time via interrupted times series analyses.Additional details of the study are reported elsewhere [10].

Data collection
We used two main data sources for services provided: For the number of PSA screening tests provided, we used the claims data of a major Swiss health insurance company, covering about 20% of the Swiss population (Helsana Versicherungen AG).We analysed the quarterly number of PSA-tests from 2005-2013 in 662,874 outpatients.For the rate of surgical repair of the ACL in patients with ACL-rupture, we used the claims data of the mandatory Swiss accident insurance scheme (Suva) and some private accident insurers (SSUV database).This database covers about 60% of the Swiss population.We analysed the rate of ACL repair from 1990-2011 in 101,737 patients with knee injury.

Statistics
We tested the following hypothesis: in the case of a relevant impact of the respective SMB recommendation (i.e.intervention), the number of PSA tests (or the rate of ACL repair) will decline.For the quantitative analysis, we applied a stepwise approach.First, we visually assessed a scatterplot of the number of performed services over time comprising the key elements of the interrupted time series analysis (fig.1).We defined the time point of intervention as the publication date of each SMB report.We were interested in the secular trend before intervention (slope pre) and in the secular trend after intervention (slope post).In addition, the absolute change in the number of services (change in level) at the time of intervention was of interest.Second, we performed a segmented regression analysis.For the PSA analysis, the dependent variable was "number of PSA screening tests per quarter" (explanatory variables: "intervention" [publication of SBM report PSA, November 2011; yes/no], "co-intervention" [publication of American Urologic Association, AUA, guidelines, November 2009; yes/no], "population at risk for PSA screening tests" [number of insured men aged 50-70 years], "number of inpatients treated for prostate cancer" [to account for the prevalence of prostate cancer and thus for diagnostic PSA tests]).For the ACL analysis, we calculated the "annual rate of surgical ACL repair" as the "number of patients with sur-gical ACL repair" divided by the "number of patients with ACL rupture".This rate served as the dependent variable of the regression analysis (explanatory variable: "intervention" [publication of SBM ACL report, June 2009; yes/ no]).For each regression, we modelled a time trend by a secondorder polynomial, included an interaction term for the main intervention, and controlled for seasonal fixed effects (if the data were collected at a higher frequency than annually).Models were optimised according to AIC-statistics, auto-correlation could be rejected (by a Durbin-Watson test), and residuals were inspected for key properties such as homoscedasticity.For inferential analysis we calculated 95% confidence intervals (CIs).No predictive intervals were needed, as we could fully rely on empirical data.Data were analysed with Stata SE 12 (StataCorp 2011, Stata Statistical Software, College Station, TX).Ethical approval was not required, as our retrospective quantitative analysis was based on fully anonymised claims data.

Impact on number of PSA screening tests
For the PSA analysis, 34 quarterly data points from 2005 to 2013 were available (fig. 2).Data showed a relevant seasonal variability, with fewer PSA tests during Q3 (July to September), presumably as a result of summer vacations.The secular trend before the intervention showed a continuous but diminishing increase in the number of PSA tests over time, i.e. the growth rate had declined already before the intervention (first-order coefficient: 414, 95% CI: 312-515; second-order coefficient: -5.44, 95% CI: -8.84 --2.04]).The trend predicts an increase in PSA tests from the previous period to the following one of 397 for 2005:Q2 and of 147 for 2011:Q1 ("slope pre", discrete form).During the quarter immediately after the intervention, a statistically significant reduction in PSA tests was estimated ("change in level": -1556, 95% CI: -2679 --433]).The secular trend during the following periods, however, showed strong variation (coefficient of interaction between first-order time variable and intervention: 113, 95%-CI: -166 -392; p = 0.41).A return of the trend to its previous path cannot be ruled out.Accordingly, the 95% CIs of the two alternative trend functions, one including the intervention and one discarding it, overlap around the beginning of 2013.Thus, the sustainability of a possible causal effect of the intervention on the number of PSA tests remains unclear.

Impact on operation rates for ACL rupture
For the ACL analysis, 22 annual data points from 1990 to 2011 were available.The absolute number of patients with ACL rupture in our data set increased between 1990 (n = 3,341) and 2011 (n = 6,942).The absolute number of surgical ACL repairs increased as well (1990: n = 2,366; 2011: n = 3,481).However, the rate of surgical ACL repair in patients with ACL rupture, initially reaching about 80%, declined after the late 1990s to approximately 55% in 2009 (fig.3).This transition to a negative secular trend, which starts off clearly before the intervention, results in a negative secondorder coefficient of the polynomial time trend (-0.13 percentage points, 95% CI: -0.16 --0.09) in combination with a positive first-order coefficient (1.9 percentage points, 95%-CI: 1.2-2.7).The rate of ACL repair continued to decline during the two years with available data after the intervention, but the trend after intervention is not significantly different from the secular trend before intervention (change in level: -5.7 percentage points, 95% CI: -16.2 -4.9; p = 0.27).Thus, no statistically significant relationship can be found between the intervention and an intensification of the downward time trend.

Discussion
We assessed the impact of SMB recommendations in two case studies.Despite some evidence of a possible change we did not find a sustained and significant impact of SMB recommendations on annual PSA screening tests or on the already declining rate of surgical ACL repair for ACL rupture in Switzerland.However, we had only few post-intervention observations and longer term data are needed to confirm or refute the findings.Interestingly, already before the SMB recommendations there was a declining growth rate for PSA screening tests in our data set.Although we adjusted the PSA data for epidemiological trends and cointerventions, other less well defined factors, such as the ongoing debate about the benefit of PSA screening for prostate cancer, may have influenced the trend before intervention.This possibility is even more pronounced for the operation rates due to ACL rupture.A negative secular trend with decreasing rates clearly started before the publication of the

Original article
Swiss Med Wkly.2015;145:w14140 Swiss Medical Weekly • PDF of the online version • www.smw.chSMB report.No ACL guidelines, which might have acted as a cointervention, exist for Switzerland.However, the intensive debate about the suitable therapy for ACL rupture over several years in the scientific community during clinical symposiums and congresses may have led to changes in clinical practice and thus a decline in operation rates for ACL rupture already before the SMB report.

Context to existing evidence
Whereas some impact evaluations have only assessed knowledge and attitudes [11,12], others have focused on case load or frequency of services (e.g. for mammography screening [13,14] or drug monitoring in epilepsy treatment [15]).For example, the analysis of self-reported mammography screening rates of women aged 40 to 49 years in the United States showed no decline from 2005 to 2011, despite the 2009 recommendation of the US Preventive Services Task Force not to screen this age group for breast cancer.We are not aware of a published impact evaluation of recommendations concerning surgical ACL repair.A US study assessed the impact of a recommendation not to screen men over 75 years with PSA tests [16].The author analysed data of a cancer registry and concluded that the recommendations led to a small decline in PSA screening rates.On the other hand, a study based on self-reported PSA screening tests of US men showed that the effect of recommendations against screening had been minimal at best [17].The Swedish SBU agency has recently assessed the impact of a broad spectrum of their released HTA reports [7].A mixed methods approach and time series data were used.The authors assessed different outcome domains and concluded that the selected HTA cases had a high impact on clinical guidelines.Concerning the impact on clinical practice they found some SBU HTA reports with high estimated impact (e.g. in dental care), but also some with low estimated impact (e.g. for vaccination of children) [7].

Strengths and limitations of our approach
This study has for the first time assessed the impact of SMB recommendations on the provision of services in the Swiss healthcare system.In our model, we considered secular time trends for the quantitative analysis.Our study has several limitations.First, it is an observational study and confounding, e.g. by cointerventions, cannot be excluded, despite our attempt to control for several variables.Second, the database for our impact evaluation is small.In Switzerland, routine data for medical services are often poorly documented or were not suitable for our purposes.Thus, no data could be used, for example, from the Swiss national statistics for in-hospital treatment, from other private accident insurances or from big medical laboratories that cover the outpatient sector for PSA tests.Furthermore, ACL data were only available until 2011.Thus, only a short time interval with two data points after intervention in 2009 could be analysed for a possible impact on care.Third, we had to make several assumptions, as codes for PSA tests, International Classification of Diseases (ICD) codes and coding of surgical procedures have been modified over the past years.However, we thoroughly scrutinised data together with the data owner to minimise misclassification.Finally, we only included two SMB recommendations and it is unclear if our findings are transferable to other SMB reports.

Success factors for guideline adherence
Change of clinical behaviour is subject to complex factors.In Switzerland, for example, an intense debate about the meaningfulness of the SMB recommendations for clinical practice emerged [18][19][20].Several success factors have been discussed in the literature to increase the impact of recommendations and one conceptual model proposes a "knowledge-to-action" cycle [2], which includes impact evaluations, such as our study.Other approaches include efficient dissemination strategies (e.g. via adapted communication strategies for target groups); visibility of, and trust in, HTA bodies (e.g. via social marketing activities [21] and involvement of medical opinion leaders to create effective advocates for final reports [7]); involvement of citizens and patients (e.g. via cooperation with trustworthy consumer groups [21]); active stakeholder management and alliances with current activities against overtreatment (internationally: "choosing wisely" [22]; in Switzerland: "smarter medicine" [23]).A recent report of the SMB, for example, has fuelled the debate about overtreatment, also internationally [24,25].Improved accessibility of routine care data for impact evaluations remains another important issue, the aim being to close the gap from dissemination of recommendations to generation of reliable knowledge about their impact in practice.For example, follow-up impact assessments could include more post-intervention data and allow more precise impact estimations.

Conclusions
Impact evaluation is an important tool for evidence-based Swiss health services, as it allows a better understanding of the magnitude of impact of clinical practice recommendations.Further monitoring is needed to assess if recommendations disseminated by the SMB have a relevant impact on clinical practice in Switzerland.

Figures (large format)
Key elements of interrupted time series design.Adapted according to Matowe [26].

Figure 2
Figure 2 Prostate specific antigen (PSA) tests over time (2005-2013).Each data point represents the number of PSA tests over 3 months in Helsana Insurance (○ JAN-MAR; □ APR-JUN; ∆ JUL-SEP; • OCT-DEC).The dashed line is corrected for seasonal effects.Vertical dashed line: Time point of SMB report released in 2011.Blue solid trend line: Trend (95% CI) without Swiss Medical board (SMB) recommendations up to 2013.Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2011.

Figure 3
Figure 3 Operation rates of the ruptured anterior cruciate ligament (ACL) over time (1990-2011).Each data point represents the annual operation rate among patients with ruptured ACL (source: SSUV database).Vertical dashed line: Time point of Swiss Medical Board (SMB) report released in 2009.Blue solid trend line: Trend (95% CI) without SMB recommendations up to 2011.Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2009.

Figure 2
Figure 2 Prostate specific antigen (PSA) tests over time (2005-2013).Each data point represents the number of PSA tests over 3 months in Helsana Insurance (○ JAN-MAR; □ APR-JUN; ∆ JUL-SEP; • OCT-DEC).The dashed line is corrected for seasonal effects.Vertical dashed line: Time point of SMB report released in 2011.Blue solid trend line: Trend (95% CI) without Swiss Medical board (SMB) recommendations up to 2013.Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2011.

Figure 3
Figure 3 Operation rates of the ruptured anterior cruciate ligament (ACL) over time (1990-2011).Each data point represents the annual operation rate among patients with ruptured ACL (source: SSUV database).Vertical dashed line: Time point of Swiss Medical Board (SMB) report released in 2009.Blue solid trend line: Trend (95% CI) without SMB recommendations up to 2011.Red solid trend line: Trend (95% CI) after release of SMB recommendations in 2009.