A Swiss digital Delphi study on patient-reported outcomes

mental and social health (95–100%). During the meeting, agreement was reached on all seven measurement domains of a generic PROM (PROMIS-29), ranging from 80% (Anxiety, Sleep Distur-bance) to 100% (Pain Interference, Depression, Ability to Participate in Social Roles). Consensus was also reached for all PROMIS-29 items, with average domain consensus ranging from 83% (Sleep Disturbance, Ability to Partici-pate in Social Roles) to 100% (Depression). Finally, four post-meeting consensus statements regarding PROs in Switzerland reached agreement. CONCLUSIONS: A Delphi method can help identify areas of need regarding PROMs in Switzerland. The current study identified a generic PROM as a missing quality indicator for the Swiss national health system’s value. A pre-meeting informational briefing, expert presentations and moderation supported three voting rounds to help identify PROMIS-29 as a PROM framework (round 1), measurement domains (round 2) and items (round 3) as a basis for further validation research. The empirical agreement among diverse stakeholders supports broad consensus towards preliminary feasibility of integrating generic PROMs into the Swiss health system based on content relevance.


Introduction
Quality-of-care metrics continue moving away from a clinical focus (toxicity, survival) towards patient relevancy (functionality, perceived benefit).One such patient-relevant metric is patient-reported outcomes (PROs).The European Medicines Agency defines PROs as "any outcome evaluated directly by the patient himself and based on patient's perception of a disease and its treatment(s)" [1].
PRO measures (PROMs) are receiving increasing attention with a view to their potential to provide valuable feedback to healthcare providers, to support medical product labelling claims as well as their conversion to quality-adjusted life years (QALYs) for economic evaluations [2].As Weszl et al. recently observed in their review of valuebased purchasing in Europe, "Since there is no requirement to include PROMs in device studies for regulatory purposes, it seems probable that their increasing use is driven by competitive market pressures".Indeed, in Switzerland, PROMs' presence continues to grow amid an ongoing shift toward value-based healthcare [3].
PROMs can be considered a widely recognised -although much less widely implemented -strategy to improve quality and appropriateness of healthcare.A cursory literature review illustrates the increasing attention being given to PROs in Switzerland (figure 1).
Interestingly, this increase coincides with both professional and federal recommendations for improving Swiss healthcare quality (table 1).However, the existence of a growing body of literature cannot be taken as a surrogate for successful implementation.As Vincent  PROMs, but implementation is associated with significant investment, which may be beyond the reach of a single canton or health system" [5].The objective of the present study is to describe and report on initial efforts of a Swiss Delphi study of PROs for the purpose of understanding content relevance of a generic PRO instrument.

Methods
The Swiss Delphi study on PROs aimed to found a comprehensive initiative for PROs in Switzerland.Such a founding has been recognised as necessitating coalescence of diverse PRO stakeholders towards endorsement of a generic PROM and generation of principal consensus statements.In particular, the digital Delphi method is justified by the pursuit of consensus-based processes to found a national initiative for PROs in Switzerland [8].Specifically, a combination of pre-meeting informational materials, in-meeting presentations given by an expert from the field (external perspective), structured discussions, and voting, as well as post-meeting voting on consensus points was implemented.

Generic PROM selection
One primary distinction between PROMs pertain to their disease-specificity or generic format.For the purpose of the initial starting point, the internal working group deliberated that a generic PROM may best serve the interests of diverse stakeholders in Switzerland.This decision was also informed by a cursory scoping review of the literature of PROMs in Switzerland, where it was identified that approximately twice as many disease-specific PROMs as generic PROMs were in use (see stakeholder information materials, appendix 1).We decided to start out in this Delphi study with a focus on a generic PROM, with the perspective of future follow-up meetings to be open to disease-specific PROMs.
Several generic PROMs were identified as candidates for inclusion in the current digital Delphi study, among them traditional ("legacy") PROMs such as Euro Quality of Life (Euro-QoL) and Health Outcomes Short Form (SF-36) measures, as well as the newer PROMIS instrument [9].Ultimately, PROMIS was selected rather than traditional measures due to two advantages pertaining to substantive  and methodological aspects.These two aspects are briefly elaborated on below.
First, the open-source ethos of PROMIS development aims to promote more equitable uptake by avoiding licensing costs associated with SF-and EQ-instruments [9].Also, in contrast with traditional measures developed for specific diseases, PROMIS was purposively based on generic health domains applicable across diseases [10].The wider distribution allows for general population health monitoring, as well as tracking of persisting conditions over time [11].PROMIS' broad, cross-disease design was considered as addressing an important gap in the use of PROMS in Switzerland, which had thus far focused more on disease-specific instruments [5].
Second, traditional measures developed on classical test theory have been superseded by the PROMIS basis on modern psychometric methods such as item response theory (IRT).For example, IRT's interval scaling contributes to PROMIS' more precise "responsiveness" to change over time (i.e.reduced floor/ceiling effects).Furthermore, IRT validation enables future creation of computer-adaptive tests for greatly improved efficiency in administration (lower response burden).For example, although the PROMIS-29 fixed form is shorter than the 36-item SF-36, computer-adaptive testing enables precise assessment of specific domains of interest with 3-5 items [12].Finally, PROMIS' methodological advantage of IRT enables scores to be converted into traditional metrics for the creation of quality-adjusted life years (QALYs) [13].The ability to derive QALYs from PROMIS scores highlights the system's flexibility for contributing to future health economics research in Switzerland.
Taken together, PROMIS was selected as a more appropriate generic PROM for Switzerland based on its crossdisease conceptualisation, its public population calibration and its advanced IRT methodology.It should be noted that PROMIS-29 is a 29-item, previously validated set.We did not select specific items for rating; rather, all items in the standard 29-item set were presented.

Consensus statement development
Key questions were originally drafted and revised via three iterative meetings among the internal working group (study authors) for approval.These key questions served as points of discussion during the meeting and were, subsequently, derived into consensus points for post-meeting voting (appendix 1).Specifically, the first author reviewed qualitative feedback and cluster-analysed it for synthesis and presentation to the fifth author.The first and fifth author jointly reviewed the content until consensus was reached on the number and content of points to be presented to stakeholders for consensus voting.

Recruitment of the expert panel
Initially recruited participants were encouraged to further circulate contact information for Swiss parties with interests in PROs for a community-referral sampling methodology.This was purposive, given the heterogeneity of cantonal and city health infrastructures.In addition, the panel sought to recruit stakeholders from distinct sources [14].
Although the sample primarily comprised a majority of academics and researchers, critical groups were identified as necessitating, at least, one representative (e.g.patient safety).

Survey voting
Recruited participants were invited via email to participate in an anonymous online survey.At the same time, they were briefed on and invited to an upcoming Swiss-PRO Stakeholder Meeting (pre-17 days).Informational materials regarding PROs accompanied the meeting and survey invitation (appendix 1).A reminder email for the survey was distributed one week later.
The 1 st survey comprised a demographic and biographical section, as well as presentation and voting of the tripartite health framework (physical, mental, social).Each part could be rated on a 4-point Likert-type scale: not important (1), slightly important (2), rather important (3) and very important (4).A definition of each framework part was provided as a reference during voting, with the stem for each item framed as "How important do you view XXXX for the average or typical person of Switzerland?"(copy of surveys, appendix 2).
"Consensus" was defined as ≥70% agreement in the rating of each item, with agreement defined as a rating of either 3 "rather agree" or 4 "strongly agree".The 4-point response scale was retained for all subsequent survey rounds with the same consensus criteria for scoring.The wording of round 3 survey voting on specific items was slightly modified to ascertain relevancy of items (content validity) for people in Switzerland: not relevant (1), slightly relevant (2), rather relevant (3) and very relevant (4).
All surveys were administered in English.Feedback for the pre-meeting survey was administered at the start of the stakeholder meeting.In-meeting feedback was delivered real-time following 15-minute planned meeting pauses.Post-meeting consensus point feedback was delivered approximately 6 weeks following the stakeholder meeting.

Ethics approval and consent to participate
Ethics approval for the current study was non-applicable due to the non-intervention design and non-inclusion of patient data.Informed consent was sought from participants at the beginning of the pre-meeting survey, with the information that data will be used anonymously and not transferred to any third party, and that consent is given by proceeding in the survey form.

Results
A total of 28 stakeholders were recruited from 17 Swiss institutions.Of the 28 stakeholders, 21 from 8 institutions completed the framework survey, which corresponds to a 75% return rate for stakeholders and 47% for institutions.

Sample characteristics
Over a 2-week period, n = 28 stakeholders were invited to participate in the pre-meeting survey pertaining to the conceptual PROM framework (physical, mental, social health), with n = 21 responding and completing the survey.An additional n = 3 patients were added to the dataset postmeeting, but were given all materials for voting (frame-work, domains, items and consensus points).The organisational backgrounds of participating stakeholders were 17 from academia, 3 from professional associations and 1 governmental.The professional backgrounds comprised 6 physicians, 6 epidemiologists, 3 psychometricians, 2 economists, 2 data scientists, 1 nurse and 1 patient safety advocate.The vast majority had more than ten years of postaccreditation work experience (n = 17), while two had "some" (6-10 years) and two had "a little" (1-5 years).A summary of sample descriptive characteristics is shown in table 2 below.

Framework consensus (Round 1)
The average completion time for the framework survey was 4½ minutes.
All three aspects of the tripartite-health framework retained by PROMIS from the WHO reached consensus, ranging from 95-100%.Mental and social health was rated negligibly lower (95%) compared to physical health (100%).

Domain consensus (Round 2)
The average completion time for the domain survey was 2½ minutes.
All seven measurement domains of the PROMIS instrument reached consensus, ranging from 80-100%.Fatigue and Anxiety domains reached slightly lower levels of agreement (80%) compared to perfect agreement on Pain Interference, Depression and Ability to Participate in Social Roles and Activities (100%).

Item consensus (Round 3)
The average completion time for the item survey was 7½ minutes.
All consensus percentages by three survey rounds pertaining to the framework, domains and items are displayed in table 3 below.

Consensus for further development (Round 4)
Pre-selected questions for in-meeting discussion were used as prompts to generate potential consensus points for further development.I n-meeting dialogue was transcribed and the content reviewed by two study authors (MK, MM) for deriving four preliminary consensus points for voting.
Approximately three weeks post-meeting, a final survey round was administered to solicit voting.Stakeholders were further invited to suggest modifications and nominate additional consensus points for further consideration.Two modifications were administered pertaining to terminology and ancillary example institutions.No new consensus points were suggested.All four consensus statements reached consensus, ranging from 90-100%.The four consensus statements are presented in table 4 below.

Discussion
This study sought to bring evidence from digital Delphi methods to bear on the potential future use of a generic PROM in Switzerland.Diverse stakeholders were recruited and four voting rounds were conducted with the aim of achieving consensus pertaining to: 1) theoretical tripartite framework, 2) PROM domains (conceptual model), 3) PROM items (measurement model) and 4) statements towards integration of PROs in Switzerland.
First, the tripartite framework as a PROM basis retained from the World Health Organisation reached high consensus [16].The slightly lower importance for mental and social health aspects corresponds to the only other generic health-related quality of life data from Switzerland gathered with SF-36 v2, which has only five items on mental health and two items on social functioning [17].Given that Switzerland tends to fare better on physical health but worse on mental health as compared to other countries [17], particular attention to mental health seems warranted.
With the recent pandemic, the relevance of mental and social health has only increased.The PROMIS instrument would enable a wider scope of assessment by using more items to assess mental (8) and social (4) health.
Second, the seven primary domains were voted on to complete the conceptual model according to the "International Guidance Framework" for selecting PROMs [15].Notably, three domains reaching perfect consensus (100%) represent each pillar of the overarching, tripartite framework: Pain Interference (Physical) , Depression (Mental) , and Ability to Participate in Social Roles and Activities (Social) .The adequate comprehensiveness in terms of the overarching framework supports content validity according to the consensus-based standards for the selection of health measurement instruments [18].The "two lowest" consensus domains (80%) were Sleep Disturbance and Anxiety.Discussion among experts during structured moderation re-vealed substantive opinions that Anxiety is likely less familiar in the general population than Depression.Consequently, there was marginal scepticism regarding the applicability of the Anxiety domain for a general public survey.The domain was retained, however, after input from one of our PRO subject matter experts regarding dual-factor structures for negative affectivity (high-and low-energy).
Third, all 29 items from the PROMIS-29 instrument reached consensus.The lowest average-consensus domain for Sleep Disturbance may reflect rater confusion caused by the mixed-item scaling format.For example, half of the Sleep Disturbance items are positively scaled, the other half negatively.A scaling frame artifact may have simply cued ambiguity in the rating response process, ergo, lower agreement [19].Alternatively, it may substantively correspond to recent findings from the SF-36's usage in Switzerland [17].Specifically, Vitality was found to have the least ceiling effect, whereas Emotional Role Functioning had the most.Finally, there was also substantive discussion regarding the potential redundancy of Sleep Disturbance with the Fatigue domain, although this was resolved through content review.
Finally, our curated key questions (see appendix 1) generated adequate discussion for deriving four consensus points for post-meeting voting.All four statements reached consensus, with only two suggested substantive changes.Both pertained to consensus point #2, with one addressing terminology ("activate stakeholders" added) and the other addressing the manifest example ("federal quality commission" instead of "within health-services research institutions").Both suggested modifications were deliberated by study authors in considering their acceptance.

Limitations
There are a few study limitations that merit further consideration.First, in soliciting maximal responses to the initial survey, recruits were invited to respond regardless of their capacity to attend our stakeholder meeting, which led to considerable attrition between round one and two voting.Relatedly, our stakeholder sample is diverse, but its convenience sampling may have led to underrepresentation in important organisational areas, such as industry (insurer representatives).Also, although validated German, French and Italian versions of PROMIS have been published, each application in a new national culture would require correspondingly new demonstration of content validity.Indeed, further uptake within Switzerland may be facilitated by the promulgation of evidence via networking as an offi-cial PROMIS International partner.Finally, PROMIS was initially selected as an optimal generic PROM, but several other instruments may also suffice for use in Switzerland.For example, the International Consortium for Health Outcomes Measurement (ICHOM) may be a viable generic PROM option, as its Overall Adult Health measure is similarly rooted in the tripartite framework of physical, mental and social health [20].One advantage of ICHOM compared to PROMIS is the apparent ease of translational services.One disadvantage, however, is its relative newness (2021 vs 2004) and seeming reliance on classical test theory approaches.It should also be noted that the content of PROMIS is non-exhaustive, and any future adoption of its secondary domains for more specific use in particular health conditions/samples should be subject to a sim-

Equity, diversity and inclusion
The application of PROs must incorporate strategies and specific tools to support equity, diversity and inclusion.Any Swiss PRO Group should adhere to the "International Guidance Framework" for selecting PROMs by including patients or public citizens in the selection of PROMs' conceptual (constructs/domains) and measurement model (items) [15].This statement principally coalesces with those underlying the SNF "Investigator-Initiated Clinical Trials" programme.

PROs support patients in becoming
PROs support patients in becoming informed partners in achieving better health outcomes.It is important to recognise that integrating PROs is not only a change for the health system, but a shift in shared responsibility for the individual (provideràßpatient) as well.To the extent that self-managed health is promoted, PROs should be seen as facilitating this goal.
Also, although a health insurance representative originally agreed to participate, they were unable to join the meeting or send feedback due to loss of contact.More health insurance representatives should be sought and included in future stakeholder meetings to represent the payer side of the health system.Relatedly, it should be noted that three additional patients included here were only recruited after initial data collection.Although they did not participate in the initial stakeholder meeting, they were briefed by a study author in receiving and responding to the PROMIS framework, domains, scales, items and consensus statements.Finally, it should be noted that this is an initial effort to formalise the content validity process for a generic PROM in Switzerland, thus, it is non-exhaustive and follow-up stakeholders should be continuously engaged.

Conclusion
This consensus study provides a first strategic groundwork for adopting a generic PROM in Switzerland.Swiss-PRO stakeholders agreed with the tripartite health framework, as well as the seven primary domains, although Sleep Disturbance and Anxiety received less consensus regarding their importance (73%).Concomitantly, three of four "anxiety" items received lower "relevance" agreement (77%).Taken together, however, all items were above our 70% benchmark for consensus.Consequently, PROMIS-29 is potentially a suitable generic PROM for Switzerland.Finally, the current Swiss-PRO stakeholder sample agreed with four consensus statements derived from key questions posed to meeting attendees.To this goal, our findings reflect the agreement of diverse stakeholders, and we aim to include more patients in developing and advancing future PROM programmes.Furthermore, our research plans aim to empirically pilot PROMIS-29 in a Swiss adult population across all three language regions.
and Staines note in their recent report to the Federal Office of Public Health (FOPH), "… many health systems are interested in
[6]e most important requirements for a successful national program are ability to measure and monitor safety and quality… It is important to identify reliable outcome measures and to include PROMs."[6]Allegranzietal. (2017).Qualität und Sicherheit der Schweizerischen Gesundheitsversorgung Verbessern.Empfehlungen und Vorschläge für die Bundesstrategie.2.PROs: patient-reported outcomes; PROMs: Patient-reported outcome measures.

Table 2 :
Summary sample descriptive characteristics by survey round.
* Patients were recruited and ratings added post-meeting.Discrepant total sample sizes and particular demographics resulted from respondent-missing survey data.Swiss Medical Weekly • www.smw.ch• published under the copyright license Attribution 4.0 International (CC BY 4.0)

Table 3 :
Stakeholder agreement by round.
* Response rate calculated on number of participants Accepted or Tentative for Meeting (n = 17).Agreement defined as >70% in top 2 response categories (Very or Rather Important).Swiss Medical Weekly • www.smw.ch• published under the copyright license Attribution 4.0 International (CC BY 4.0)

Table 4 :
Swiss Delphi Study PROs Consensus Statements.It is valuable to add patient reports (PROs) to the medical field.Assuming a simple value equation: [value = quality (clinical outcomes, functional status, PRPs) / cost], the Swiss-PRO Panel posits that the exclusion of PROs lowers quality, ergo, value.Conversely, including PROs should improve Swiss healthcare value by increasing quality.Swiss national PRO body consisting of PRO experts is needed to activate stakeholders and guide direction in all relevant areas of healthcare, policy and research.For example, this may manifest as a nationally mandated expert council (e.g.Federal Quality Commission), which should act in an advisory capacity and support PRO-related research interests.