DOI: https://doi.org/10.4414/smw.2012.13594
In 2011 the Swiss government published a report on homeopathy [1]. The report was commissioned following a 2009 referendum in which the Swiss electorate decided that homeopathy and other alternative therapies should be covered by private medical insurance. Before implementing this decision, the government wished to establish whether homeopathy actually works. In February 2012 the report was published in English and was immediately proclaimed by proponents of homeopathy to offer conclusive proof that homeopathy is effective. This paper analyses the report and concludes that it is scientifically, logically and ethically flawed. Specifically, it contains no new evidence and misinterprets studies previously exposed as weak; creates a new standard of evidence designed to make homeopathy appear effective; and attempts to discredit randomised controlled trials as the gold standard of evidence. Most importantly, almost all the authors have conflicts of interest, despite their claim that none exist. If anything, the report proves that homeopaths are willing to distort evidence in order to support their beliefs, and its authors appear to have breached Swiss Academies of Arts and Sciences principles governing scientific integrity.
The report takes the form of a Health Technology Assessment (HTA), which is different from a meta-analysis or a systematic review. The authors state that this is an “established scientific procedure which [...] examines not only the efficacy of a particular intervention, but especially also its ‘real-world effectiveness', its appropriateness, safety and economy.” This appears to be an attempt to lend their report objective credibility, but the report is very far from a typical HTA. HTAs normally conduct and commission new research in order to assess the effectiveness of a new technology or procedure [2], but the Swiss report simply reviews existing research. Furthermore, it is misleading to claim that meta-analyses and systematic reviews only consider efficacy; on the contrary, they often include information on cost and safety, and they invariably also include data on effectiveness (although the authors may have a different concept of effectiveness in mind; see below). Also, it is not clear what the authors mean by “appropriate”; if they mean “benefical to patients” then reviews and analyses also include such data.
The authors’ attempt to claim that their HTA exceeds the scope of a normal systematic review is perhaps explained by the fact that their report includes no new good evidence. It does, however, accept evidence that would normally be excluded from a typical analysis, including “observational studies, good case series and longitudinal cohort studies [1]”. The purported advantage of including these sources is that they are “politically more informative”, an ambiguous term that is not explained. However, the claim that HTAs are special in including such data is also misleading, since systematic reviews can include such sources of evidence. Perhaps the key difference between the evidence criteria for a systematic review and this report is that the former only includes high-quality evidence while the latter accepts lower-quality evidence.
The authors’ method of selecting literature for analysis is illuminating. They did not conduct an online review because “just searching online would not have been sufficient to supply a representative overview of homeopathic research” [1]. Hence they also included “expert contacts and scanning of bibliographic references.” This is a suspect research strategy for three reasons. First, any high-quality research would almost certainly be available online. Second, the use of “expert contacts” suggests biased and informal methodology. Third, references to papers that are not available online suggest research that is out of date or has not undergone peer review. Furthermore, it is not clear why an online search would not have provided a representative sample; it appears likely that it actually would have, but that the results would have shown that homeopathy is ineffective.
As already stated, no new high-quality evidence is presented in the report. Rather, it attempts to reanalyse in a more sympathetic light studies that are known to be flawed.
In fact, the authors base their conclusion that homeopathy is effective on four trials that are all more than a decade old and have been comprehensively exposed as weak, flawed studies (see reference 3 for details of the trials) [3]. Most of the authors of these studies acknowledge that they did not show homeopathy was effective, but this does not prevent the report’s authors from reanalysing the evidence in the light of their curious criterion of “real-world effectiveness” and certain preclinical studies the authors claim to remove any doubts about the plausibility of homeopathy. With regard to the latter, the authors claim that the mechanism of homeopathy “is supported by quite a large number of high-quality trials in fundamental preclinical research”, but that these studies “are unable to provide statements regarding the other mainstays of homeopathy: the simile principle and drug proving on the healthy subject” [1]. Thus the authors believe that homeopathy is plausible because of preclinical studies that do not establish one of the key tenets of homeopathy and do not show any efficacy in humans. How, then, can they conclude that homeopathy works? The authors effectively rewrite the normal rules of validity:
In contrast to the now customary view that the reliability of results grows with internal validity, we think that – roughly speaking – there is a risk of false positive results if the external validity is overrated and a risk of false negative results if the internal validity is overrated. From the homeopathic point of view, the external validity is low with most studies... because they tend to ignore the essential foundations of classical homeopathy [1].
This amounts to an apologia for homeopathy: rather than admit that normal methods of assessment show that homeopathy doesn’t work, the authors assume this means that the methods don’t work. This theme is continued in their use of “real-world effectiveness.”
We have already noted that systematic reviews and meta-analyses often take account of effectiveness. What, then, do the authors of the report mean by “real-world effectiveness”? The authors claim that this criterion takes account of safety while systematic reviews do not. We have also seen that reviews actually do take account of safety, but, somewhat ironically, the report itself does not. The authors state that “a systematic search for cases in the homeopathic and legal literature was not possible owing to problems of infrastructure, methodology and time” [1]. It is rather misleading of the authors to claim that their criterion includes elements that systematic reviews do not when reviews actually do and their own report does not. It is also ethically suspect to conclude that homeopathy is safe when no review was conducted.
A clue to the true nature of “real-world effectiveness” is provided through a question put by the authors themselves, viz.
If homeopathy is highly likely to be effective but this cannot be consistently proven in clinical trials, the question arises of what conditions are needed for homeopathy to show its effectiveness and realise its potential, and what conditions threaten to obscure this? [1]
It is somewhat surprising that the authors are so bold as to include this question in the report. Here they appear to be admitting that they seek to show that homeopathy is effective and help the practice to flourish, and that they are willing to select any research conditions that will achieve this aim. Rather than accepting that the scientific method shows that homeopathy is ineffective, and accepting this, they argue that the method itself is flawed and create their own method to produce the answers they want. In essence, this looks like an admission of research misconduct, and raises serious questions about the authors’ objectivity.
Not content with asserting that their criterion of effectiveness is superior when it is actually biased, the authors also attack the cornerstone of evidence-based medicine, the randomised controlled trial. In fact, they devote an entire chapter of the report to doing so, even though there was apparently no time to conduct a review of the literature on safety. Ironically, some of their concerns are of an ethical nature. They argue that it is unethical to repeat RCTs when a treatment is known to be effective, as some participants would be denied the best treatment. This is obviously true, but is no argument against RCTs. Who would want to repeat an RCT if the treatment is already known to be effective? They also argue that consent is no compensation for the fact that RCTs cannot guarantee equipoise, and conclude that authorities have no right to insist on RCTs because of these ethical problems. The authors are essentially arguing that it is unethical to establish whether a given treatment works; most people believe the opposite. (For homeopaths, of course, it might be convenient if they didn’t have to establish the effectiveness of their remedies.)
Note that the only point at which the word “ethics” is used in the document is when it is used to argue against RCTs. Given the large number of articles raising concerns about unethical aspects of homeopathy, it is troubling that the authors did not even mention the issues of potential harm to patients who choose homeopathy rather than effective conventional treatment. waste of resources through funding of ineffective treatment and the issue of deceiving patients in order to benefit them [4].
Given the preceding sections of this paper, it should come as no surprise that the majority of the report’s authors are homeopaths. In fact, only one of the medically qualified authors is not a homeopath or alternative medicine practitioner. Of course, the fact that someone is a homeopath does not mean they cannot be objective about homeopathy, but the potential for bias is substantial and that potential seems to have been fulfilled in the report. The authors of the report provide a conflict of interest statement:
Nobody involved in the compilation had any financial or other conflict of interest. Whenever expert advice was sought from a physician who himself uses the method in question, independent experts were also consulted [1].
Unfortunately, the first sentence is untrue. Homeopaths believe that homeopathy works, and as such have an inherent conflict of interest if they are involved in an assessment of homeopathy. Even if they are capable of objectivity, they have an obligation to declare this as a potential conflict of interest. But this is not the main issue here. As homeopaths, many of the authors would be professionally compromised if their report stated that homeopathy doesn’t work. Furthermore, the inclusion of homeopathy in insurance schemes, which required a favourable verdict from the report, potentially benefits many of the authors in terms of both prestige and money. The conflict of interest statement should actually have stated that many of the authors have financial and professional conflicts of interest. Note that the authors not only failed to declare these conflicts of interest but also denied that they existed. If a report endorsing a new drug was written by employees of a pharmaceutical company and they claimed they had no conflict of interest, they would be scorned; it is troubling that the same does not seem to have applied here. Note also that in the above quote the authors concede that a doctor’s use of homeopathy constitutes a conflict of interest; it is unclear why they do not apply this logic to themselves.
One particularly unusual facet of the authors’ conflicts of interest is that they actually form part of their argument in favour of homeopathy. For exemple, the report reinterprets Kleijnen’s famous study and argues that its conclusions were only negative because the authors believed that the mechanism of homeopathy was implausible. Given their acceptance (based on laboratory tests) that homeopathy is plausible, the authors argue that this means the results of the original study were positive. Thus their conflict of interest – they believe that homeopathy works despite evidence to the contrary – is key to their biased reinterpretation of results.
In addition to denying their conflicts of interest, we have already seen that the authors of the report seem to have cherry-picked and misinterpreted the evidence they wanted to back their contention that homeopathy is effective. This might seem understandable, given their faith in alternative medicine, but it is actually a serious ethical infraction. The Swiss Academies of Arts and Sciences have published guidelines on research integrity. On the subject of conflicts of interest, these state:
All persons participating in a research project must make their financial and other interests and relationships known to their superiors, to the responsible authorities and to other authorised persons, insofar as these could come into conflict with their research activity. Personal interests must not be allowed to influence an individual’s objectivity in the evaluation of projects or publications [5].
The authors of the report appear to be in breach of these requirements. They explicitly deny any conflicts of interest, despite having personal and financial stakes in the outcome of the report they are writing. The authors might protest that they were conducting not research but an assessment. However, the guidelines also condemn “deliberate concealment of conflicts of interests... negligent or intentional wrong assessment of projects, programmes or manuscripts... [and] unfounded judgments in order to create advantages, either personal or for the benefit of third parties” in expert appraisals and peer reviews. The biased selection of evidence and selective interpretation of the results in the report are at the very least careless and more likely to be intentional. This is nothing new in homeopathy, as Edzard Ernst has pointed out:
Homeopaths often argue that there are further systematic reviews which allegedly do show that homeopathy works. Examples are a recent [earlier] Swiss Health Technology Assessment or the review by Mathie. The problem is that these articles do not fulfil the formal criteria for a systematic review, originate from homeopaths, are open to bias and can be criticised on important methodological grounds [6].
The ethos of the research integrity guideline is that “Scientific misconduct must not be tolerated.” In this case, it has not only been tolerated but given the stamp of approval by the Swiss government and used to inform health policy: from January 2012, homeopathy will be included in health insurance cover, contributing in part to a rise in insurance premiums of as much as 4.4% [7]. In contrast to the Swiss report, the UK Government’s investigation of homeopathy heard evidence from both homeopaths and opponents of the practice, and the report was written by members of parliament on an independent committee. This report is available online for free and concluded that “Homeopathy should not be funded on the NHS and the MHRA should stop licensing homeopathic products” [8].
The present paper has established that the authors of this report adopted a very unusual strategy in what should have been an impartial evidence appraisal. It appears that their goal was not to provide an independent assessment but to choose criteria that would lead to their chosen conclusion that homeopathy is effective. To this end, they chose to adopt a highly questionable criterion of “real-world” effectiveness, ignore negative findings concerning homeopathy in favour of implausible reinterpretation of results, and attack RCTs. This use of a unique and suspect methodology in an appraisal designed to assess healthcare objectively gives cause for particular concern; one imagines that the Swiss government wanted homeopathy to be judged against existing standards rather than new ones created specially for the evaluation. In doing so the authors have distorted the evidence and misled the public; these actions, combined with their conflicts of interest, strongly suggest that they are guilty of research misconduct. It is extremely unfortunate that the Swiss government lent legitimacy to this report by attaching its name to it, and also unfair that the English-language text is not available free of charge to the public when it is being widely misrepresented all over the world as proof of the efficacy of homeopathy [9]. It remains possible that homeopathy is effective, but the authors of this report do the practice a grave disservice.
1 Gudrun Bornhöft and Peter Matthiessen (eds.) Homeopathy in healthcare: effectiveness, appropriateness, safety, costs. Springer, 2012.
2 NHS National Institute of Health. Health Technology Assessment programme. Available at: http://www.hta.ac.uk/ (accessed 9/3/12)
3 Ben Goldacre. Benefits and risks of homeopathy. The Lancet. 2007;370:1672–3.
4 David Shaw. Homeopathy is where the harm is. Five unethical effects of funding unscientific remedies. J Med Ethics. 2010;36(3):130–1.
5 Swiss Academies of Arts and Sciences. Integrity in scientific research: principles and procedures. http://www.akademien-schweiz.ch/en/dms/E/Publications/Guidelines-and-Recommendations/e_Integrity.pdf (accessed 9/3/12)
6 Edzard Ernst. Written submission. In: House of Commons Science and Technology Committee (see reference 8), p.88 (Evidence 28).
7 Swissinfo. Swiss make New Year’s regulations http://www.swissinfo.ch/eng/swiss_news/Swiss_make_New_Years_regulations.html?cid=31867422 (accessed 9/3/12)
8 House of Commons Science and Technology Committee. Evidence check 2: Homeopathy. 2009. Available at: http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/45/45.pdf
9 Dana Ullman. The Swiss government’s remarkable feport on homeopathic medicine. The Huffington Post. http://www.huffingtonpost.com/dana-ullman/homeopathic-medicine-_b_1258607.html The Swiss government’s remarkable report on homeopathic medicine (accessed 9/3/12).
Funding / competing conflicts of interest: The author has written papers critical of homeopathy in the past.