Attitudes towards evaluation of psychiatric disability claims : a survey of Swiss stakeholders

a Swiss Academy of Insurance Medicine, University of Basel Hospital, Basel, Switzerland b Institute for Clinical Epidemiology and Biostatistics, University of Basel Hospital, Basel, Switzerland c School of Applied Psychology, University of Applied Sciences Northwestern Switzerland, Olten, Switzerland d University Psychiatric Hospital Basel, Basel, Switzerland e Department of Insurance Psychiatry, Suva, Swiss Accident Insurance Fund, Lucerne, Switzerland f Department of Anaesthesia, McMaster University, Hamilton, Canada g CLARITY Group, Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada h The Michael G. DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, Canada i Institute of Medical Disability Evaluations of Central Switzerland, Lucerne, Switzerland j Swiss Society of Insurance Psychiatry, SGVP, Basel, Switzerland


Introduction
Western societies have established social insurance systems to cover loss of income in workers who are unable to work owing to poor health.Insurers commission medical evaluations of work capacity when confronted with disputable claims.Disputes can arise from uncertainty about the underlying health impairments or the consequences of these health impairments.Disputes can also arise from uncertainty whether the insured individual is entitled to benefits and if so, to what extent [1].Disability claims involve a number of stakeholders: judges administering the law, lawyers defending claimants' rights or acting on behalf of insurers, treating physicians, employees of social or private insurers adjudicating claims, and independent medical experts providing views on claimants' medical restrictions and limitations.In a social security setting, all stakeholders should view the process for approving or denying disability claims as fair and equitable.Increasing restrictions in social insurance eligibility arising from attempts to address rising debts in disability insurance [2][3][4] have fuelled controversy among independent medical experts and criticism and discontent with the claim process from patients and the legal community.This controversy and expression of dissatisfaction has been particularly prominent in evaluations of disability arising from psychiatric conditions.
In face of these prevailing tensions in the system, it becomes important to gauge accurately stakeholders' perceptions of how the disability evaluation system is functioning.Therefore, in order to explore current perceptions of disability claim management and independent psychiatric evaluations, we conducted a national survey of clinical, administrative, and legal stakeholders in Switzerland.Our goal was to explore current concerns, to identify both conflicting and shared views, and to explore ideas and generate hypotheses regarding how the disability claim process and in particular medical evaluations could be improved.We focused on medical evaluations directed at determining work capacity among patients presenting with mental illness.

Methods
This study is part of the RELY research programme addressing the reliability of psychiatric medical evaluations (www.unispital-basel.ch/asim/RELY).

Defining stakeholders
We identified the following groups of clinical, administrative, and legal stakeholders who may be involved with disability claims and independent psychiatric evaluations: plaintiff lawyers ('lawyers') representing claimants or patient advocacy groups (e.g.Pro Mente Sana); treating psychiatrists ('psychiatrists') involved in patient care who perform fewer than five independent evaluations per year; independent psychiatric experts ('experts'), i.e. psychiatrists who conduct five or more evaluations of work capacity per year and are not employed by an insurer; judges at social courts ('judges'); and insurance employees involved in the claim process for disability benefits ('insurers'), e.g.managers, insurer lawyers, and administrative staff.We limited survey respondents to stakeholders who were involved with claimants with mental disorders who underwent an independent psychiatric evaluation to establish their work capacity, or were responsible for reviewing reports of such evaluations during the year preceding our survey (2012).

Questionnaire development
With the assistance of content experts, and reference to the previous literature [5], we developed a 37-item (original questions 5 to 37, see appendix E), German-language questionnaire to examine stakeholders' attitudes towards the disability claim process and independent psychiatric evaluations assessing work capacity.The final questionnaire framed response options with a 4-point Likert scale or discrete categorical response options as a previous report has shown that closed-ended questions result in fewer incomplete questionnaires than open-ended formats [6].Our questionnaire was designed to explore four areas.1. Perceptions of the current claim process (global appraisal, transparency, equitable treatment of claimants, appropriateness of allocated disability benefits), and the evaluation of work capacity (global appraisal, information covered, integration of involved professionals, three main weaknesses selected from a list of ten options).(Questions 5 to 12 in the original questionnaire.) 2. Characteristics of an optimal independent psychiatric evaluation (rating the importance of 12 quality items).(Questions 13 to 24 in the original questionnaire.) 3. Preferences for reporting of work capacity.Countries vary in their ways of expressing limitations in work capacity [7].We asked respondents to express their preferences regarding the options used in Switzerland, percentage reporting or narrative, on a number of domains including meaningfulness, amount and precision of information, and ease of interpretation and the importance of providing each method of expressing work capacity.(Questions 25 to 35 in the original questionnaire.) 4. Expectations of agreement among two experts independently evaluating the same claimant.Using a hypothetical case (a random number between 0 and 100 percent), we asked the stakeholders to provide the maximum difference in percentage work capacity they would find acceptable between two experts.In a second question that referred to the same hypothetical case, we asked respondents to specify the maximum difference they would find acceptable if the medical evaluation occurred under optimal conditions (complete medical file, detailed job description of the claimant's last job, interview with a cooperative claimant).(Questions 36 and 37 in the original questionnaire.)A professional translator who specialised in "insurance and health" translated the questionnaire from German to French.A bilingual senior researcher in insurance medicine reviewed the French version and discussed ambiguous text passages with the translator.A third reviewer, a bilingual independent psychiatric expert who was otherwise not involved in the project commented on clarity and consistency.In addition, we invited the presidents of the diverse stakeholder organisations to give feedback before distributing the final versions of the survey.

Disseminating the survey
We used multiple strategies to approach stakeholders across Switzerland (appendix A): medical and legal professional organizations, patient advocacy groups and their networks, mailing lists, conference participants, the national disability insurance, and the Swiss National Accident Insurance Fund (Suva).For psychiatrists and lawyers, we approached the president of the Swiss Society of Psychiatry and Psychotherapy, the secretary of the Swiss Society of Judges and the president of the Cantonal Social Security Courts who supported the survey by a personal note, the Chief Physician from the Swiss National Accident Insurance Funds, Suva, and the managing director of the conference of the disability insurance offices.Participants received a disclosure letter detailing the intent of the survey, the time for completion (≈15 minutes), assurance regarding the confidentiality of their response, and our intention to publish the results.The respondents received either a French-or German-language version of the questionnaire depending on prior information regarding their language preference.Limiting the dissemination to the target groups, and ascertaining the number of eligible individuals who received the survey, proved challenging.Most organisations circulated the survey through their membership lists or other internal communication channels that also reached individuals who did not meet eligibility criteria.On the other hand, not all eligible individuals were members of their professional organisations.We therefore supplemented the organisational lists with lists of individuals attending conferences (appendix A).In order to select eligible respondents, we included a screening question in the survey (question 1).The survey stopped immediately when a stakeholder had not seen any patient or claimant who was evaluated for work capacity during the previous year.We collected information from the stakeholder organisations that informed our approximation of the number of eligible stakeholders who received the questionnaire.We used this approximation to estimate the survey response rate (table 1).We circulated the questionnaire by e-mail using electronic forms in which, for questions in a "choose all options below that apply" format, we randomised the order of possible response options.We used an electronic check to prevent more than one response from the same computer.At 3 and 6 weeks after the initial e-mailing, stakeholders received a reminder to complete the questionnaire.

Analysis and reporting
We present our findings in accordance with the guidance proposed by Bennett and colleagues for survey research [8].We analysed all questionnaires that provided more than just demographic information, irrespective of the number of questions answered.We summarised responses by the proportion of respondents who chose each response option in the categories of lawyers, psychiatrists, experts, judges, and insurers.

Perception of current claim process and psychiatric evaluation of work capacity
Table 3 illustrates stakeholders' perceptions regarding the process of claim processing for patients with mental disorders from the time their claim is submitted to the final decision regarding whether the claim is accepted.Satisfaction with the claim process was lowest among lawyers (pooled: very/somewhat good; 7%), followed by psychiatrists (26%), experts (35%), and judges (72%) and highest among insurers (81%).Satisfaction with independent evaluation of work capacity revealed the same pattern: lowest among lawyers (7%), followed by psychiatrists (19%), experts (40%), and judges (54%) and highest among insurers (66%).The gradient in satisfaction across stakeholder groups also applied to treatment of claimants, transparency of the claim process, allocation of appropriate benefits, and completeness of information within reports (appendix B).Stakeholders reported common concerns regarding independent psychiatric evaluations and reports (table 4): a lack of transparency in the relationship between the medical expert report description of the patient's condition and the percentage of disability, medical evaluations inappropriately addressing legal issues, insurance administrative personnel or lawyers making inappropriate statements regarding medical conditions, and the long interval between the evaluation and the finalised report.Another concern, medical evaluations being an undue burden to claimants, was reported more frequently by psychiatrists (35%) and experts (24%) than by lawyers (7%), judges (2%) or insurers (7%).a Educated guess based on discussions with the president(s) of the organisation(s) b About 1,700 psychiatrists were registered with the Swiss Society of Psychiatry and Psychotherapy, possibly 300 were not.From these 2,000 psychiatrists, 500 were classified as experts and the majority of the remaining (≈1200) may regularly see patients at risk of work disability.The remaining psychiatrists are assumed to work in different fields such as child and adolescent psychiatry.c We might have missed a minority of eligible psychiatrists who were not registered in the professional organisations and did not attend psychiatric conferences.

Characteristics of an optimal psychiatric evaluation of work capacity
Stakeholders rated 12 characteristics of an expert report regarding their importance in determining the report's quality (summary in table 5, full analysis appendix C).Ratings of "very important" were chosen by almost all stakeholders for "transparent relationship between the experts' findings and their conclusions" (98%-89%), very frequently for "unambiguous statements" (91%-75%) and "consideration of the opinions of all professionals" (89%-62%) and somewhat less frequently for "explicit statement of the purpose of the evaluation" (80%-26%), "restriction to medical issues" (78%-50%), and "written in a way that is easy to follow by all parties" (63%-48%).For items that were rated as less important, such as the "report being indisputable", "accepted by all parties" and "of low burden to the claimant", there was also high agreement about the degree of importance both within and between stakeholder groups.

Preferences for reporting of remaining work capacity
Overall, lawyers, judges and insurers all expressed strong approval and preference for percentages over narrative statements regarding work capacity.This was not true of psychiatrists and experts, who in general had a more positive view of narrative than of percentage (table 6, appendix D).Specifically, all stakeholders agreed that it is important to express work capacity in psychiatric reports as narrative (87%-100% said "very important" or "somewhat important").The non-medical stakeholders (lawyers, judges, insurers) had a similar view of the importance of expressing work capacity as a percentage (81%-96%), but this was true of fewer psychiatrists (57%) and experts (53%).a Stakeholders differed very consistently in their response pattern of related questions 5-9 and 11+12.Here, we illustrate the pattern using two typical answers.Response patterns were similar when we asked whether claimants are treated equally, whether benefits are allocated appropriately, whether the claim process is transparent, whether relevant information is accounted for, and whether professional opinions are considered (appendix B).

Acceptability of inter-rater disagreement in expert judgment
In the hypothetical scenario, the median level of an acceptable absolute difference between the judgments of two independent experts differed between stakeholder groups: 10% (insurers), 15% (judges and lawyers), or 20% (psychiatrists and experts) (table 7).Referring to optimal circumstances for an evaluation, all stakeholders raised their expectations on agreement even further (median 10% difference in all groups, table 7).

Main findings
The most important finding in our survey was that lawyers, psychiatrists and experts are in general very dissatisfied with the current claims process whereas the majority of judges and particularly insurers are satisfied (see table 3).
The most important deficiencies identified were lack of clear conclusions in medical evaluations regarding claimants' work capacity and the inclination of experts to comment inappropriately on legal issues, and of insurers and legal personnel to make inappropriate statements regarding medical issues (see table 4).Other important findings included a clear divergence in preference for the method of expressing the extent of work incapacity: psychiatrists and experts saw considerable limitations in percentages and had much more positive views about narrative statements, while lawyers, judges and insurers reported a clear preference for percentages (table 6).All stakeholder groups had high expectations of agreement in medical evaluations by different raters, and this was particularly true of respondents who facilitate or interpret evaluations (lawyers, judges, and insurers) versus those who perform them (psychiatrists and experts) (table 7).

Strength and limitations
Strengths of our study include a large nationwide sample and inclusion of five relevant stakeholder groups.Our sample size was large and we asked a large number of ques-tions crucial to respondents' perception of the function of disability evaluation in Switzerland.Response rates, conservatively estimated, were over 45% in three of our five stakeholder groups (see table 1), a relatively high rate in current surveys in the medicolegal field.Limitations of our study include our inability to calculate precisely our response rate.The estimates we made were, however, conservative.Our response rate was under 45% in two of the stakeholder groups (experts and psychiatrists), weakening inferences for these participants.We did not list, among the possible problems with the medical evaluation process, financial conflicts of interest of the experts in judging work capacity.There is concern in the legal community regarding the possibility that experts render judgments in the interests of those who engage them to make their evaluations [9].In retrospect, this is a potentially important additional problem with the claim evaluation process.
Because many issues required professional insight and understanding about mental disorders, current practice of evaluation of work capacity, the claim process, and social law, we did not include claimants among our stakeholder groups.Claimants were, however, indirectly represented by treating physicians, and lawyers, including lawyers representing advocacy groups.We asked about attitudes towards evaluation of work capacity in general, thereby disregarding the fact that stakeholders see partly different cases: the judges see a minority of cases.

Implications
Evaluation of work capacity is a highly complex process: detailed information regarding the claimant's job, functioning at work, residual ability to perform job-specific skills, and self-perceived work ability need to be collected, selected, ranked and weighted.This process in itself involves innumerable implicit and explicit judgments [10].The experts' final judgment about work capacity is further determined by their interaction with the claimant, personal experience, education and training, personal norms and values.This complexity suggests the need for a rigorously struc- We report median and the interquartile ranges of the maximum acceptable differences.

Original article
Swiss Med Wkly.2015;145:w14160 Swiss Medical Weekly • PDF of the online version • www.smw.chtured approach to medical evaluations, with clear guidance on process and integration of information.Experts in Switzerland, like those in most European countries [11] and in North America [12] do not, however, use a consistent structured approach to evaluating and reporting work capacity.Requirements in Switzerland address mainly the format of the report [1,7,[13][14][15][16], leaving much room for variation in the evaluation and in the report's content.
Given the limited human ability to process highly complex information [17][18][19], one would anticipate such large variation using an unstructured approach.Further, evaluation of work capacity requires expertise in vocational rehabilitation, as medical restrictions do not correlate well with the ability to work.This skill is not part of traditional medical training, and as a result some regulatory bodies have argued that physicians should not adjudicate work capacity [20].
The dissatisfaction and limitations highlighted by our results suggest that experts need guidance and techniques that will help them to elicit trustworthy information from claimants regarding their functional limitations and remaining abilities.Communication skills training, an essential part of medical training for more than a decade [21], is just emerging in evaluation of work capacity.Notable in this area are innovations from researchers in the Netherlands [22] and Norway [23,24], who have developed interviewing techniques that tap a claimant's functional abilities and limitations specifically in the context of work.Processing functional information in a systematic way in relation to the outcome "work capacity" can clarify the key information that substantiates expert judgment.A recent randomised trial demonstrated the potential impact of structured processing: an evidence-based standardised evaluation of work capacity in claimants with posttraumatic stress disorder -not routinely taught and practiced in most countries -proved greatly superior to standard practice [25].

Narrative versus percentage formats for reporting work capacity
Preferences for reporting remaining work capacity varied.Stakeholders who are charged with interpreting and applying independent evaluations prefer to have work capacity presented as a percentage ("it's easy to interpret"), whereas stakeholders who treat patients and perform evaluations prefer narrative formats -probably because they recognise the difficulty in communicating complex concepts with a single number.Stakeholders who are charged with interpreting and applying independent evaluations have less tolerance for variability in work capacity evaluation among experts, whereas stakeholders who treat patients and perform evaluations are more forgiving -probably because they recognise the difficulty in assessing work capacity.

Lack of clarity in addressing legal and medical issues
Lawyers and judges expressed particular concerns about medical reports inappropriately addressing legal issues which, although still a concern, were less salient in the other stakeholder groups [26].The problem has several aspects.First, insurers often ask questions of experts that in-appropriately focus on non-medical issues or even request judgments on legal issues that are not in the expert's domain.Experts do not easily avoid answering such questions [1].Second, for certain terms the meaning may vary with the context.For instance, the Swiss social code book has a particular definition for "work capacity" while private insurance policies can vary in their definitions.To cite another example: the legal definitions of an accident or disease differ from the regular medical understanding.Interprofessional medicolegal skills training could improve experts' performance on what to include or not include in a report.Approximately a third of psychiatrists and experts expressed concern about confusion of medical and legal issues that may reflect legal authorities making inaccurate statements about medical issues.For instance, legal authorities may state that depression of mild to moderate severity is easily treated [27] when this is not in fact the case [28].
To cite other examples: legal professionals may deny the impact of a mental disorder (such as dysthymia) on work capacity [29] or disregard the inherent variation among patients in their response to medical treatment.Experts may be understandably perturbed when legal decisions are justified by a misunderstanding of medical evidence [30].

Unrealistic expectations of expert judgments
The stakeholders reported very high expectations of the consistency of expert judgment about work capacity when evaluating the same claimant.Available evidence suggests that the median standard for maximal disagreement of 10% to 20% (see table 7) is unrealistic.For example, one study found that experts' judgments of a videotape of a claimant with depression varied from her being fully able work to fully disabled, with a third of respondents concluding that the claimant could work more than 6 hours, a third 6 to 3 hours, and a third less than 3 hours [31].A second study requested 20 experts to provide disability rating on 42 files of patients with low back pain.Judgments of work disability varied widely.In mild cases, the difference between highest and lowest judgments of the same claimant ranged from 15% to 50%; among cases of moderate severity, the difference ranged from 30% to 85%; in severe cases, it ranged from 40% to 80%.Disagreement tended to rise with increasing severity of the claimants' condition [32].These results, consistent with the remainder of the relevant literature, suggest levels of agreement far lower than demanded by our respondents.

Conclusion
Lawyers, psychiatrists and experts perceive major problems in the psychiatric evaluation of work capacity.These problems relate primarily to lack of clarity in how experts come to their conclusions regarding work capacity, to inappropriate judgments of clinical and legal issues by those not qualified to make those judgments, and to delays in experts completing their reports.Judges and insurers see the process much more positively.Efforts to improve the process should include enhancing the clarity of the basis on which judgments are made, restricting judgments to areas of expertise, and ensuring prompt submission of evaluations.

Table 1 :
Eligibility and response rates.

Table 2 :
Characteristics of stakeholders.

Table 3 :
Perceptions of current claim process (full results in appendix B).

Table 4 :
Stakeholder concerns regarding independent psychiatric evaluation of work capacity.

Table 5 :
Characteristics of an optimal psychiatric evaluation of work capacity (full results in appendix C).
An optimal expert report should be of low burden to the claimant (greatest dissent on what stakeholders consider important) Swiss Medical Weekly • PDF of the online version • www.smw.ch

Table 6 :
Judgments about the relative merits of narrative vs. percentage approaches reporting a claimant's work capacity (full results in appendix D).

Table 7 :
Maximum acceptable difference between two experts performing a psychiatric evaluation of work capacity in the same claimant.
What is the maximum difference in percentage work capacity that stakeholders would find acceptable for two experts independently evaluating the same claimant …

psychiatric evaluation of work capacity considers the opinion of all professionals involved
* Participants were allowed to skip a question.Therefore, the number of answers varied from question to question.Swiss Medical Weekly • PDF of the online version • www.smw.chAppendix C:

be brought quickly to completion.
Participants were allowed to skip a question.Therefore, the number of answers varied from question to question.
Swiss Medical Weekly • PDF of the online version • www.smw.ch… be accepted by all parties.* Swiss Medical Weekly • PDF of the online version • www.smw.chAppendix D:

is more useful for the course of the claim process
Swiss Medical Weekly • PDF of the online version • www.smw.ch