A Swiss digital Delphi study on patient-reported outcomes

DOI: https://doi.org/https://doi.org/10.57187/smw.2023.40125

Introduction

Quality-of-care metrics continue moving away from a clinical focus (toxicity, survival) towards patient relevancy (functionality, perceived benefit). One such patient-relevant metric is patient-reported outcomes (PROs). The European Medicines Agency defines PROs as “any outcome evaluated directly by the patient himself and based on patient’s perception of a disease and its treatment(s)” [1].

PRO measures (PROMs) are receiving increasing attention with a view to their potential to provide valuable feedback to healthcare providers, to support medical product labelling claims as well as their conversion to quality-adjusted life years (QALYs) for economic evaluations [2]. As Weszl et al. recently observed in their review of value-based purchasing in Europe, “Since there is no requirement to include PROMs in device studies for regulatory purposes, it seems probable that their increasing use is driven by competitive market pressures”. Indeed, in Switzerland, PROMs’ presence continues to grow amid an ongoing shift toward value-based healthcare [3].

PROMs can be considered a widely recognised – although much less widely implemented – strategy to improve quality and appropriateness of healthcare. A cursory literature review illustrates the increasing attention being given to PROs in Switzerland (figure 1).

Figure 1Summary of Swiss PRO publications by 5-year periods. (Results are averaged. Combinatory keyword searches “patient-reported outcomes” AND “Swiss” across three databases: (1) PubMed, (2) Emcare and (3) Google Scholar.)

Interestingly, this increase coincides with both professional and federal recommendations for improving Swiss healthcare quality (table 1). However, the existence of a growing body of literature cannot be taken as a surrogate for successful implementation. As Vincent and Staines note in their recent report to the Federal Office of Public Health (FOPH), “… many health systems are interested in PROMs, but implementation is associated with significant investment, which may be beyond the reach of a single canton or health system” [5]. The objective of the present study is to describe and report on initial efforts of a Swiss Delphi study of PROs for the purpose of understanding content relevance of a generic PRO instrument.

Methods

The Swiss Delphi study on PROs aimed to found a comprehensive initiative for PROs in Switzerland. Such a founding has been recognised as necessitating coalescence of diverse PRO stakeholders towards endorsement of a generic PROM and generation of principal consensus statements. In particular, the digital Delphi method is justified by the pursuit of consensus-based processes to found a national initiative for PROs in Switzerland [8]. Specifically, a combination of pre-meeting informational materials, in-meeting presentations given by an expert from the field (external perspective), structured discussions, and voting, as well as post-meeting voting on consensus points was implemented.

Generic PROM selection

One primary distinction between PROMs pertain to their disease-specificity or generic format. For the purpose of the initial starting point, the internal working group deliberated that a generic PROM may best serve the interests of diverse stakeholders in Switzerland. This decision was also informed by a cursory scoping review of the literature of PROMs in Switzerland, where it was identified that approximately twice as many disease-specific PROMs as generic PROMs were in use (see stakeholder information materials, appendix 1). We decided to start out in this Delphi study with a focus on a generic PROM, with the perspective of future follow-up meetings to be open to disease-specific PROMs.

Several generic PROMs were identified as candidates for inclusion in the current digital Delphi study, among them traditional (“legacy”) PROMs such as Euro Quality of Life (Euro-QoL) and Health Outcomes Short Form (SF-36) measures, as well as the newer PROMIS instrument [9]. Ultimately, PROMIS was selected rather than traditional measures due to two advantages pertaining to substantive and methodological aspects. These two aspects are briefly elaborated on below.

First, the open-source ethos of PROMIS development aims to promote more equitable uptake by avoiding licensing costs associated with SF- and EQ- instruments [9]. Also, in contrast with traditional measures developed for specific diseases, PROMIS was purposively based on generic health domains applicable across diseases [10]. The wider distribution allows for general population health monitoring, as well as tracking of persisting conditions over time [11]. PROMIS’ broad, cross-disease design was considered as addressing an important gap in the use of PROMS in Switzerland, which had thus far focused more on disease-specific instruments [5].

Second, traditional measures developed on classical test theory have been superseded by the PROMIS basis on modern psychometric methods such as item response theory (IRT). For example, IRT’s interval scaling contributes to PROMIS’ more precise “responsiveness” to change over time (i.e. reduced floor/ceiling effects). Furthermore, IRT validation enables future creation of computer-adaptive tests for greatly improved efficiency in administration (lower response burden). For example, although the PROMIS-29 fixed form is shorter than the 36-item SF-36, computer-adaptive testing enables precise assessment of specific domains of interest with 3–5 items [12]. Finally, PROMIS’ methodological advantage of IRT enables scores to be converted into traditional metrics for the creation of quality-adjusted life years (QALYs) [13]. The ability to derive QALYs from PROMIS scores highlights the system’s flexibility for contributing to future health economics research in Switzerland.

Taken together, PROMIS was selected as a more appropriate generic PROM for Switzerland based on its cross-disease conceptualisation, its public population calibration and its advanced IRT methodology. It should be noted that PROMIS-29 is a 29-item, previously validated set. We did not select specific items for rating; rather, all items in the standard 29-item set were presented.

Consensus statement development

Key questions were originally drafted and revised via three iterative meetings among the internal working group (study authors) for approval. These key questions served as points of discussion during the meeting and were, subsequently, derived into consensus points for post-meeting voting (appendix 1). Specifically, the first author reviewed qualitative feedback and cluster-analysed it for synthesis and presentation to the fifth author. The first and fifth author jointly reviewed the content until consensus was reached on the number and content of points to be presented to stakeholders for consensus voting.

Recruitment of the expert panel

Initially recruited participants were encouraged to further circulate contact information for Swiss parties with interests in PROs for a community-referral sampling methodology. This was purposive, given the heterogeneity of cantonal and city health infrastructures. In addition, the panel sought to recruit stakeholders from distinct sources [14]. Although the sample primarily comprised a majority of academics and researchers, critical groups were identified as necessitating, at least, one representative (e.g. patient safety).

Survey voting

Recruited participants were invited via email to participate in an anonymous online survey. At the same time, they were briefed on and invited to an upcoming Swiss-PRO Stakeholder Meeting (pre-17 days). Informational materials regarding PROs accompanied the meeting and survey invitation (appendix 1). A reminder email for the survey was distributed one week later.

The 1^st survey comprised a demographic and biographical section, as well as presentation and voting of the tripartite health framework (physical, mental, social). Each part could be rated on a 4-point Likert-type scale: not important (1), slightly important (2), rather important (3) and very important (4). A definition of each framework part was provided as a reference during voting, with the stem for each item framed as “How important do you view XXXX for the average or typical person of Switzerland?” (copy of surveys, appendix 2).

“Consensus” was defined as ≥70% agreement in the rating of each item, with agreement defined as a rating of either 3 “rather agree” or 4 “strongly agree”. The 4-point response scale was retained for all subsequent survey rounds with the same consensus criteria for scoring. The wording of round 3 survey voting on specific items was slightly modified to ascertain relevancy of items (content validity) for people in Switzerland: not relevant (1), slightly relevant (2), rather relevant (3) and very relevant (4).

All surveys were administered in English. Feedback for the pre-meeting survey was administered at the start of the stakeholder meeting. In-meeting feedback was delivered real-time following 15-minute planned meeting pauses. Post-meeting consensus point feedback was delivered approximately 6 weeks following the stakeholder meeting.

Ethics approval and consent to participate

Ethics approval for the current study was non-applicable due to the non-intervention design and non-inclusion of patient data. Informed consent was sought from participants at the beginning of the pre-meeting survey, with the information that data will be used anonymously and not transferred to any third party, and that consent is given by proceeding in the survey form.

Results

A total of 28 stakeholders were recruited from 17 Swiss institutions. Of the 28 stakeholders, 21 from 8 institutions completed the framework survey, which corresponds to a 75% return rate for stakeholders and 47% for institutions.

Sample characteristics

Over a 2-week period, n = 28 stakeholders were invited to participate in the pre-meeting survey pertaining to the conceptual PROM framework (physical, mental, social health), with n = 21 responding and completing the survey. An additional n = 3 patients were added to the dataset post-meeting, but were given all materials for voting (framework, domains, items and consensus points). The organisational backgrounds of participating stakeholders were 17 from academia, 3 from professional associations and 1 governmental. The professional backgrounds comprised 6 physicians, 6 epidemiologists, 3 psychometricians, 2 economists, 2 data scientists, 1 nurse and 1 patient safety advocate. The vast majority had more than ten years of post-accreditation work experience (n = 17), while two had “some” (6–10 years) and two had “a little” (1–5 years). A summary of sample descriptive characteristics is shown in table 2 below.

Framework consensus (Round 1)

The average completion time for the framework survey was 4½ minutes.

All three aspects of the tripartite-health framework retained by PROMIS from the WHO reached consensus, ranging from 95–100%. Mental and social health was rated negligibly lower (95%) compared to physical health (100%).

Domain consensus (Round 2)

The average completion time for the domain survey was 2½ minutes.

All seven measurement domains of the PROMIS instrument reached consensus, ranging from 80–100%. Fatigue and Anxiety domains reached slightly lower levels of agreement (80%) compared to perfect agreement on Pain Interference, Depression and Ability to Participate in Social Roles and Activities (100%).

Item consensus (Round 3)

The average completion time for the item survey was 7½ minutes.

All 29 items from the PROMIS instrument reached consensus, ranging from 70–100% (figure 2). The highest-average consensus domain was Depression, which was also the only one to reach unanimous-perfect consensus (x̄ = 100%). The lowest average-consensus domain was observed for Sleep Disturbance (x̄ = 75%).

Figure 2Borderline agreement items from the PROMIS-29 instrument.

All consensus percentages by three survey rounds pertaining to the framework, domains and items are displayed in table 3 below.

Consensus for further development (Round 4)

Pre-selected questions for in-meeting discussion were used as prompts to generate potential consensus points for further development. In-meeting dialogue was transcribed and the content reviewed by two study authors (MK, MM) for deriving four preliminary consensus points for voting.

Approximately three weeks post-meeting, a final survey round was administered to solicit voting. Stakeholders were further invited to suggest modifications and nominate additional consensus points for further consideration. Two modifications were administered pertaining to terminology and ancillary example institutions. No new consensus points were suggested. All four consensus statements reached consensus, ranging from 90–100%. The four consensus statements are presented in table 4 below.

Discussion

This study sought to bring evidence from digital Delphi methods to bear on the potential future use of a generic PROM in Switzerland. Diverse stakeholders were recruited and four voting rounds were conducted with the aim of achieving consensus pertaining to: 1) theoretical tripartite framework, 2) PROM domains (conceptual model), 3) PROM items (measurement model) and 4) statements towards integration of PROs in Switzerland.

First, the tripartite framework as a PROM basis retained from the World Health Organisation reached high consensus [16]. The slightly lower importance for mental and social health aspects corresponds to the only other generic health-related quality of life data from Switzerland gathered with SF-36 v2, which has only five items on mental health and two items on social functioning [17]. Given that Switzerland tends to fare better on physical health but worse on mental health as compared to other countries [17], particular attention to mental health seems warranted. With the recent pandemic, the relevance of mental and social health has only increased. The PROMIS instrument would enable a wider scope of assessment by using more items to assess mental (8) and social (4) health.

Second, the seven primary domains were voted on to complete the conceptual model according to the “International Guidance Framework” for selecting PROMs [15]. Notably, three domains reaching perfect consensus (100%) represent each pillar of the overarching, tripartite framework: Pain Interference _(Physical), Depression _(Mental), and Ability to Participate in Social Roles and Activities _(Social). The adequate comprehensiveness in terms of the overarching framework supports content validity according to the consensus-based standards for the selection of health measurement instruments [18]. The “two lowest” consensus domains (80%) were Sleep Disturbance and Anxiety. Discussion among experts during structured moderation revealed substantive opinions that Anxiety is likely less familiar in the general population than Depression. Consequently, there was marginal scepticism regarding the applicability of the Anxiety domain for a general public survey. The domain was retained, however, after input from one of our PRO subject matter experts regarding dual-factor structures for negative affectivity (high- and low-energy).

Third, all 29 items from the PROMIS-29 instrument reached consensus. The lowest average-consensus domain for Sleep Disturbance may reflect rater confusion caused by the mixed-item scaling format. For example, half of the Sleep Disturbance items are positively scaled, the other half negatively. A scaling frame artifact may have simply cued ambiguity in the rating response process, ergo, lower agreement [19]. Alternatively, it may substantively correspond to recent findings from the SF-36’s usage in Switzerland [17]. Specifically, Vitality was found to have the least ceiling effect, whereas Emotional Role Functioning had the most. Finally, there was also substantive discussion regarding the potential redundancy of Sleep Disturbance with the Fatigue domain, although this was resolved through content review.

Finally, our curated key questions (see appendix 1) generated adequate discussion for deriving four consensus points for post-meeting voting. All four statements reached consensus, with only two suggested substantive changes. Both pertained to consensus point #2, with one addressing terminology (“activate stakeholders” added) and the other addressing the manifest example (“federal quality commission” instead of “within health-services research institutions”). Both suggested modifications were deliberated by study authors in considering their acceptance.

Limitations

There are a few study limitations that merit further consideration. First, in soliciting maximal responses to the initial survey, recruits were invited to respond regardless of their capacity to attend our stakeholder meeting, which led to considerable attrition between round one and two voting. Relatedly, our stakeholder sample is diverse, but its convenience sampling may have led to underrepresentation in important organisational areas, such as industry (insurer representatives). Also, although validated German, French and Italian versions of PROMIS have been published, each application in a new national culture would require correspondingly new demonstration of content validity. Indeed, further uptake within Switzerland may be facilitated by the promulgation of evidence via networking as an official PROMIS International partner. Finally, PROMIS was initially selected as an optimal generic PROM, but several other instruments may also suffice for use in Switzerland. For example, the International Consortium for Health Outcomes Measurement (ICHOM) may be a viable generic PROM option, as its Overall Adult Health measure is similarly rooted in the tripartite framework of physical, mental and social health [20]. One advantage of ICHOM compared to PROMIS is the apparent ease of translational services. One disadvantage, however, is its relative newness (2021 vs 2004) and seeming reliance on classical test theory approaches. It should also be noted that the content of PROMIS is non-exhaustive, and any future adoption of its secondary domains for more specific use in particular health conditions/samples should be subject to a similar “consensus-based” process as conducted here for the generic tool [21]. Finally, it should be noted that some medical specialties in Switzerland have begun systematic collection of PROMs, including PROMIS, for condition-specific purposes [22]. Similarly, some specific cantons have begun mandating routine collection of PROMs, although the specific instruments are not PROMIS (press release: https://www.gesundheitsversorgung.bs.ch/dam/jcr:b539da87-2f7c-4128-8889-dfa229e89d5f/2021-08-09_Pressemitteilung_QNS_heartbeat_finalisiert.pdf).

Also, although a health insurance representative originally agreed to participate, they were unable to join the meeting or send feedback due to loss of contact. More health insurance representatives should be sought and included in future stakeholder meetings to represent the payer side of the health system. Relatedly, it should be noted that three additional patients included here were only recruited after initial data collection. Although they did not participate in the initial stakeholder meeting, they were briefed by a study author in receiving and responding to the PROMIS framework, domains, scales, items and consensus statements. Finally, it should be noted that this is an initial effort to formalise the content validity process for a generic PROM in Switzerland, thus, it is non-exhaustive and follow-up stakeholders should be continuously engaged.

Conclusion

This consensus study provides a first strategic groundwork for adopting a generic PROM in Switzerland. Swiss-PRO stakeholders agreed with the tripartite health framework, as well as the seven primary domains, although Sleep Disturbance and Anxiety received less consensus regarding their importance (73%). Concomitantly, three of four “anxiety” items received lower “relevance” agreement (77%). Taken together, however, all items were above our 70% benchmark for consensus. Consequently, PROMIS-29 is potentially a suitable generic PROM for Switzerland. Finally, the current Swiss-PRO stakeholder sample agreed with four consensus statements derived from key questions posed to meeting attendees. To this goal, our findings reflect the agreement of diverse stakeholders, and we aim to include more patients in developing and advancing future PROM programmes. Furthermore, our research plans aim to empirically pilot PROMIS-29 in a Swiss adult population across all three language regions.

Acknowledgements

We would like to acknowledge and give our greatest thanks to all experts who joined the 1^st Annual Swiss-PRO Stakeholder Meeting. You represented diverse perspectives on a locally formative research topic (PRO in Switzerland). Through your voting and consensus points, you have initiated the potential for a common generic PROM to be adopted in Switzerland to support cross-regional quality improvement efforts. Furthermore, your contribution to and voting on consensus points brings principled guidance toward cohering the field of PROs in Switzerland. Thank you all, with special thanks to our presenter and moderator, Dr. Matthias Rose. Other attendees included Christoph Bosshard, Esther Kraft, Stefanie Hostettler, Katarzyna Wac, Anja Frei, Peter Berchtold, Francesco Cottone, Florian Rüter, as well as the submitting authors of the enclosed manuscript.

References

1. Erickson P, Patrick D. Health status and health policy: quality of life in health care evaluation and resource allocation. New York: Oxford University Press; 1993. 

2. Weszl M, Rencz F, Brodszky V. Is the trend of increasing use of patient-reported outcome measures in medical device studies the sign of shift towards value-based purchasing in Europe? Eur J Health Econ. 2019 Jun;20(1 Suppl 1):133–40. 10.1007/s10198-019-01070-1 

3. Teisberg E, Katz G, Deerberg-Wittram J. Patient-Reported Outcome Measures (PROMs). In: Rüter F, Biller-Andorno N, Steiger J, and Meier CA. The 3^rd Annual Value Based Health Care Symposium. 2021 June 17; University Hospital of Basel, Switzerland. 

4. Meier CA. Variations of care and diagnosis as markers for the quality of medical care in Switzerland. Swiss Med Wkly. 2019 Feb;149(0708):w20029. 10.4414/smw.2019.20029 

5. Vincent C, Staines A. Enhancing the Quality and Safety of Swiss Healthcare. A national report commissioned by the FOPH on the quality and safety of healthcare in Switzerland. Bern; 2019. doi: https://doi.org/10.13140/RG.2.2.22966.04160 

6. Allegranzi B, Björn B, Burnand B, Chopard P, Conen D, Pfaff H, et al. Qualität und sicherheit der Schweizerischen gesundheitsversorgung verbessern. Empfehlungen und vorschläge für die Bundesstrategie. Bundesamt für Gesundheit. 2017; 2. Bericht. https://www.bag.admin.ch/dam/bag/de/dokumente/kuv-leistungen/qualitaetssicherung/second-report-advisory-board-30-06-2017.pdf.download.pdf/second-report-advisory-board-30-06-2017-de.pdf 

7. Hostettler S, Kraft E, Bosshard C. Patient-reported outcome measures: die Patientensicht zählt. Schweiz Arzteztg. 2018;99(40):1348–52. 10.4414/saez.2018.17187 

8. Turoff M, Hiltz S. Computer based Delphi processes. In: Adler M, Ziglio E, editors. Gazing into the Oracle: The Delphi Method and its Application to Social Policy and Public Health. London: Jessica Kingsley Publishers; 1995. pp. 56–88. 

9. Bykerk VP. Patient-reported outcomes measurement information system versus legacy instruments: are they ready for prime time? Rheum Dis Clin North Am. 2019 May;45(2):211–29. 10.1016/j.rdc.2019.01.006 

10. Alonso J, Bartlett SJ, Rose M, Aaronson NK, Chaplin JE, Efficace F, et al.; PROMIS International Group. The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative. Health Qual Life Outcomes. 2013 Dec;11(1):210. 10.1186/1477-7525-11-210 

11. Hays RD, Spritzer KL, Fries JF, Krishnan E. Responsiveness and minimally important difference for the patient-reported outcomes measurement information system (PROMIS) 20-item physical functioning short form in a prospective observational study of rheumatoid arthritis. Ann Rheum Dis. 2015 Jan;74(1):104–7. 10.1136/annrheumdis-2013-204053 

12. Gulledge CM, Smith DG, Ziedas A, Muh SJ, Moutzouros V, Makhni EC. Floor and ceiling effects, time to completion, and question burden of PROMIS CAT domains among shoulder and knee patients undergoing nonoperative and operative treatment. JBJS Open Access. 2019 Dec;4(4):e0015.1-7. 10.2106/JBJS.OA.19.00015 

13. Thompson NR, Lapin BR, Katzan IL. Mapping PROMIS global health items to EuroQol (EQ-5D) utility scores using linear and equipercentile equating. PharmacoEconomics. 2017 Nov;35(11):1167–76. 10.1007/s40273-017-0541-1 

14. Calvert M, Kyte D, Price G, Valderas JM, Hjollund NH. Maximising the impact of patient reported outcome assessment for patients and society. BMJ. 2019 Jan;364:k5267. 10.1136/bmj.k5267 

15. Crossnohere NL, Brundage M, Calvert MJ, King M, Reeve BB, Thorner E, et al. International guidance on the selection of patient-reported outcome measures in clinical trials: a review. Qual Life Res. 2020;14:1–20. 10.1007/s11136-020-02625-z 

16. Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, et al.; PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007 May;45(5 Suppl 1):S3–11. 10.1097/01.mlr.0000258615.42478.55 

17. Roser K, Mader L, Baenziger J, Sommer G, Kuehni CE, Michel G. Health-related quality of life in Switzerland: normative data for the SF-36v2 questionnaire. Qual Life Res. 2019 Jul;28(7):1963–77. 10.1007/s11136-019-02161-5 

18. Terwee CB, Prinsen CA, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018 May;27(5):1159–70. 10.1007/s11136-018-1829-0 

19. Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: Implications for research and practice. Psychology Press; 1999. 10.4324/9781410601261 

20. Pan T, Mulhern B, Viney R, Norman R, Tran-Duy A, Hanmer J, et al. Evidence on the relationship between PROMIS-29 and EQ-5D: A literature review. Qual Life Res. 2021;28: 10.1007/s11136-021-02911-4 

21. Churruca K, Pomare C, Ellis LA, Long JC, Henderson SB, Murphy LE, et al. Patient-reported outcome measures (PROMs): A review of generic and condition-specific measures and a discussion of trends and issues. Health Expect. 2021 Aug;24(4):1015–24. 10.1111/hex.13254 

22. Audigé L, Bucher HC, Aghlmandi S, Stojanov T, Schwappach D, Hunziker S, et al.; ARCR_Pred Study Group. Swiss-wide multicentre evaluation and prediction of core outcomes in arthroscopic rotator cuff repair: protocol for the ARCR_Pred cohort study. BMJ Open. 2021 Apr;11(4):e045702. 10.1136/bmjopen-2020-045702 

Appendices

Appendix 1: Stakeholder meeting background informational documents.

Appendix 2: Copy of survey questionnaires.

The appendices 1 and 2 are available for download as separate files at https://doi.org/10.57187/smw.2023.40125