DOI: https://doi.org/10.4414/smw.2013.13720
Counterstatement to Shaw DM. The Swiss Report on homoeopathy: a case study of research misconduct. Swiss Med Wkly. 2012;142:w13594 http://www.smw.ch/content/smw-2012-13594/
David M Shaw recently published a critique of our Health Technology Assessment (HTA) on homoeopathy in this journal [1], which calls for a response. We expected a scientific debate to ensue on our chosen research methods and were surprised to be confronted with a biased mixture of false or incomplete citations, wrong matches and, as a consequence, incorrect conclusions that reveal a lack of familiarity and experience with both homoeopathy and the Swiss setting.
We will refute the allegations and rectify misunderstandings, in the order of appearance in Shaw’s text. Because of numerous repetitions in Shaw’s text we have subsumed some of the allegations thematically:
The HTA was prepared within the framework of PEK (“Programm Evaluation Komplementärmedizin” [Complementary Medicine Evaluation Programme]), which was launched by the Swiss Federal Office of Public Health (SFOPH) in 2005 [2, 3] and comprised several pre-defined project-parts, e.g., prospective observational studies, systematic literature reviews, meta-analyses and HTAs. Each project covered a part of the whole PEK-project, and it was intended that the HTA would cover aspects which were not addressed by the other parts.
Recently, a positive public vote in Switzerland for including complementary medicine in the constitution (19 May 2009) led to restricted provision by certified doctors within the statutory Swiss health insurance from 1 January 2012 until 31 December 2017 [4].
Coverage of complementary medicine provided by non-medical practitioners through private medical insurance is independent of the decision of the Swiss electorate.
Shaw, “In 2011 the Swiss government published a report on homoeopathy.“
Reply: Although the original HTA-report was shown for some time on the homepage of the BSV/BAG (part of SFOPH), neither the German booklet of the HTA report [5] nor the English summarised version [6] or the English translation of the report [7] were published by SFOPH. The report was prepared on behalf of SFOPH by the PanMedion Foundation, Zurich, as contractor with participation of external experts.
In his paper Shaw analyses the report and concludes that it is scientifically, logically and ethically flawed. […]
Shaw, “HTA normally conducts new research.”
Reply: A short glance at existing HTA reports contradicts this statement: A review of more than 1,000 HTA-reports from about 15 countries all over the world revealed that “All agencies consider data from published RCTs as preferred evidence base” [8]. The observation that in HTAs mainly a retrospective approach is used is also supported by another analysis [9], which reports that in more than 90% of HTAs, literature review is the chosen method of assessment. Randomised Controlled Trials (RCTs) are the gold-standard of study design for a methodologically defined setting and therefore deliver mainly information about the efficacy of the technology under investigation. Health technology assessment goes beyond this aspect, as it should also provide information about the effects of a technology in the “real world” [10]. Therefore, both aspects were taken into account by the inclusion of other study designs. This HTA followed pre-defined criteria: [7, p48ff, references p 66]: (ECHTA 2001, BSV 2001, DIMDI 2004, INHTA 2001, Heusser 2001) which, in addition, were specifically designed for the Swiss evaluation of complementary medicine by the SFOPH well in advance of the PEK-project [11]. Inclusion of study designs other than RCTs into comparative effectiveness research is accepted by other international HTA agencies, e.g., Scotland. [12, Chapter 5, p 11: “Clinical effectiveness” in the context of medicinal product assessment]. Apart from these methodological aspects, we would like to point out that this HTA was prepared within the framework of PEK which also included prospective observational research with published results [2, 3].
Shaw, “[...] RCT is the ‘gold standard’ [...]”
Reply: RCT, the “gold standard …” focuses on internal validity – and has never been validated for therapeutic systems like homoeopathy, especially concerning clinical effectiveness and cost-effectiveness. SFOPH therefore prepared a model that is better suited to the population's needs in terms of external validity, i.e., the non-experimental “real world” situation [11]. Today, this procedure is referred to as HIA (Health Impact Assessment).
The SFOPH’s Manual for the Standardisation of Clinical and Economic Evaluation of Medical Technologies [10], on which our assessment is also based, explicitly mentions as appropriate test methods those which
a) Evaluate the treatment method under consideration in its entirety
b) Take into proper consideration the realistic research possibilities in practice
c) Permit inference to the target population that is being treated in practice
It is obvious from the terminology (“entirety”, “in practice”) that RCTs do not comply with these three requirements.
Further reservations towards RCTs that we outlined in our HTA (chapter 5, pp 28ff) were:
1. The absence of a positive or any RCT result is no proof of ineffectiveness (“absence of evidence is not evidence of absence”, Altman and Bland 1995).
2. A negative RCT result is also not valid proof of ineffectiveness because many factors can be involved in causing false-negative RCT results.
3. Individualised medical care is more and more being replaced by standardised treatment methods to ensure comparability and reproducibility of study outcomes.
4. Trial results can be significantly positive even though only a small percentage of patients experience genuine benefit from the trial.
5. Reproducibility is surprisingly low even with “hard” RCTs (rigorous inclusion criteria, end points with minimal subjectivity).
6. There needs to be genuine openness (“equipoise”) at the beginning of a randomised trial, i.e., neither physician nor patient have a preference regarding a particular treatment. The fact that a patient has given his or her “informed consent” does not avoid the problem (of absent equipoise), since the responsibility cannot simply be placed on the patient, certainly not according to the Declaration of the World Medical Association.
7. In view of the ethical problems mentioned, it appears doubtful whether the authorities have the right to insist on randomised trials, i.e. evidence of inferior treatment and discrimination of control group patients, as a basis for decisions concerning health service reimbursement.
8. Different professional evidence-based reviews of identical clinical studies can arrive at different conclusions and even opposing recommendations on treatment. Not only the RCT results but also the results obtained from systematic reviews of RCTs can show considerable divergence.
9. The thematic orientation of RCTs is often not relevant to problems of health care or the needs of patients, but is driven by subjective interests (career, sponsors).
Shaw, “Rather than accepting that the scientific method shows that homoeopathy is ineffective, and accepting this, they argue that the method itself is flawed and create their own method to produce the answers they want.”
Reply: Shaw is quite wrong here and seems to stumble right into the trap that Altman and Bland warned against: “Absence of evidence is not evidence of absence” [13]. Even if Shaw sees the effectiveness of homoeopathy as not proven, there are no scientific studies that prove that it is not effective.
Having considered all arguments we state, “In clinical studies, taking internal and external validity criteria into account, the effectiveness of homoeopathy can be seen as clinically evident, and certified application as safe. From a methodological point of view the evidence of homoeopathic effectiveness is all the more remarkable if one considers that in most research studies basic rules of classical homoeopathy were violated.”
Shaw, “They did not conduct an online review.”
Reply: This statement is simply not true. Chapter 6 (pp 47ff) exactly describes the search strategies, searched databases (mainly online databases) used in the report and related results. Additional handsearching of other sources like journals and reference lists of relevant retrieved articles is a well-accepted approach, which is also recommended by the Cochrane collaboration [14]. The rationale for study selection is given in detail in chapter 10 (p 128).
Concerning the literature search, Shaw presumes also that the “... use of expert contacts” suggests biased and “informal methodology”. The consideration of expert opinions was intended within the PEK approach because the specific situation in Switzerland should be reflected. Consulting experts is a well-accepted practice within the HTA process, e.g., in Germany [15]. It is a matter of course that experts did not influence the assessment itself, but made their expertise available.
Shaw, “The report re-interprets Kleijnen’s famous study and argues that its conclusions were only negative because the authors believed that the mechanism of homoeopathy was implausible. Given their acceptance (based on laboratory tests) that homoeopathy is plausible, the authors argue that this means the results of the original study were positive.”
Reply: It is correct that we didn’t follow all conclusions of included studies (see also table 9.6. of our HTA, p. 116, where deviations are given in detail). There were two main reasons for different assessments in this HTA compared to the original authors’ conclusions:
1. The results were achieved by “vote counts”, e.g., “5 studies were negative, 4 positive, which results in a negative result” and
2. Original authors’ conclusions were based on criteria which we refused for good reasons or which were external (inappropriate, off topic), e.g., Kleijnen et al.’s conclusion: “The amount of positive evidence even among the best studies came as a surprise to us. Based on this evidence we would be ready to accept that homoeopathy can be efficacious, if only the mechanism of action were more plausible.” We concluded: “As their reservations are only based on the plausibility issue, […], we do not accept it and the result in favour of the effectiveness of homoeopathy stands.”
Shaw, “It is also ethically suspect to conclude that homoeopathy is safe when no review was conducted. The authors state that ‘a systematic search for cases in the homoeopathic and legal literature was not possible owing to problems of infrastructure, methodology and time.’ The authors did not even mention the issues of potential harm to patients who choose homoeopathy rather than effective conventional treatment. […]“
Reply: The first sentence is not true. Safety issues were part of the PEK project and were dealt with in our HTA as well (chapter 11, pp 159ff). The second sentence reads as follows in the HTA, “A systematic search for individual cases (single cases, ‘Einzelfälle’) in the homoeopathic and legal literature was not possible owing to problems of infrastructure, methodology and time.” (No emphasis in the original text). We furthermore pointed out in 11.4 (p 161) that “Homoeopathic physicians […] act in accordance with the general guidelines of medical responsibility”; so no essential medical treatment will be kept from patients.
Shaw: “The majority of the report’s authors are homeopaths. ... Homeopaths believe that homoeopathy works, and as such have an inherent conflict of interest. ... Even if they are capable of objectivity, they have an obligation to declare this as a potential conflict of interest. ... Being homeopaths themselves, they clearly had a strong interest in producing a report that might motivate the Swiss government to rule that homoeopathy will in future be covered by the Swiss health insurance system…”
Reply: Familiarity and, whenever possible, personal experience of the subject-matter are prerequisites of a trustworthy report. Systematic search and scientific analyses for this HTA were conducted exclusively by independent contractors. Of course, we collected information from experienced experts in homoeopathy (any other approach would have been careless and non-scientific). We felt no necessity to make a statement of conflict of interest, particularly as we gave full and frank disclosure of contributors, including CV details, interests and allegiances in Chapter 15. Nothing there renders it inappropriate for any of us to have undertaken this research work, as Shaw himself appears to acknowledge: “Of course, the fact that someone is a homeopath does not mean they cannot be objective about homoeopathy”[1]. We defined and substantiated our methodology, and applied it consistently.
We recognised the undoubted risk of bias, of which we were always mindful of, but considered it to be outweighed by the gain of expert knowledge of the subject of investigation. As a matter of course the experts consulted did not influence the assessment of systematically searched studies as such, but made their expertise available (mainly) in the form of relevant sources for systematic information retrieval and consultation on the selection of relevant therapies (e.g., the categorisation of homoeopathic approaches) and relevant outcomes (rather than surrogate parameters).
Reply: In this letter to the editor we refrain from discussion of the House of Commons Science and Technology Committee report. This has already been done by others [16].
In conclusion we state that the article by David Martin Shaw constitutes an accumulation of false claims, indefensible allegations and defamatory remarks without any justification in content. In our opinion, Shaw’s article and the arguments contained therein do not in any way meet the minimum requirements of a scientific paper. The author appears to have found it necessary to misrepresent or distort facts and attack the authors of the Swiss Report with untenable allegations and defamatory statements in order to support his belief of the ineffectiveness of homoeopathy. His allegations are clearly refuted by our counter-statement. We protest against the accusation that the Swiss Report was a ‘case of research misconduct’. David M. Shaw appears to attack homoeopathy, using pseudoscientific arguments and flouting intellectual integrity and honesty in a way that goes beyond the realm of scientific work.
Having considered all arguments we state: In clinical studies, taking internal and external validity criteria into account, effectiveness of homoeopathy can be seen as clinically evident, and certified application as safe.
1 Shaw DM. The Swiss Report on homeopathy: a case study of research misconduct. Swiss Med Wkly. 2012;142:w13594 http://www.smw.ch/content/smw-2012-13594/
2 PEK Review Board.Summary Consensus Statement of the PEK Review Board regarding the PEK process and the PEK products,27th September 2005 [in both, German and English].http://www.bag.admin.ch/kv/forschung/d/2005consensus_statement_review_board_d.pdf.
3 Final Report PEK [PEK Schlussbericht 24.4.2005; in German, summary in English].http://www.bag.admin.ch/themen/krankenversicherung/00305.02362/index.html
4 Information of Swiss Federal Office of Internal Affairs, 12.01.2011 [Mitteilung des Eidgenössischen Department des Innern EDI; in German]. http://www.edi.admin.ch/dokumentation/00331/01824/index.html?lang=de&msg-id=37173
5 Bornhöft G, Matthiessen P (eds.). Homeopathy in healthcare – effectiveness, appropriateness, safety and cost-effectiveness [Homöopathie in der Krankenversorgung – Wirksamkeit, Nutzen, Sicherheit und Wirtschaftlichkeit. VAS, Frankfurt/Main 2006; in German].
6 Bornhöft G, Wolf U, Ammon K, Righetti M, Maxion-Bergemann S, Baumgartner S, Thurneysen AE, Matthiessen PF (2006). Effectiveness, safety and cost-effectiveness of homeopathy in general practice – summarized health technology assessment. Forschende Komplementärmedizin 1 3 [Suppl 2]:19–29
7 Bornhöft G, Matthiessen P (eds.). Homeopathy in healthcare: effectiveness, appropriateness, safety, costs. Springer, 2011.
8 Charles River Associates. A comparative analysis of the role and impact of Health Technology Assessment. May 2011 (URL: http://www.phrma.org/sites/default/files/1661/hta_final_comparison_report_13_may_2011_stc1.pdf).
9 Draborg E, et al. International comparison of the definition and the practical application of health technology assessment. International Journal of Technology Assessment in Health Care. 2005;21(1):89–95.
10 Bingefors K, Pashos CL, Smith MD, Berger ML. Health care cost, quality, and outcomes. ISPOR Book of terms. Lawrenceville, NJ: International Society for Pharmacoeconomics & Outcomes Research, 2003. Cited from: Sean D. Sullivan, BSPharm, John Watkins, Brian Sweet, Scott D. Ramsey. Health Technology assessment in health-care decisions in the United States, 2009, value in Health Vol 12, Suppl. 2 URL:http://www.ispor.org/htaspecialissue/Sullivan.pdf
11 Federal Office of Social Insurance: Manual for standardisation of medical and economic assessment of medical goods and services, 1996 [Bundesamt für Sozialversicherung BSV. Handbuch zur Standardisierung der medizinischen und wirtschaftlichen Bewertung medizinischer Leistungen. 3. Entwurf, Bern, 1996; in German].
12 Scottish Medicines Consortium. New Product Assessment Form, revised March 2012 (URL: http://www.scottishmedicines.org.uk/About_SMC/Latest_News/News_Articles/New_Product_Assessment_Form_Guidance_For_Manufacturers
13 Altman DG, Bland JM. Absence of evidence is not evidence of absence. Br Med J. 1995;311:485.
14 Higgins Julian PT, Sally Green. Cochrane handbook for systematic reviews of interventions March 2011 (URL: http://www.cochrane-handbook.org/)
15 Institute for Quality and Efficiency in Healthcare: General methods, Version 4.0, 2011 [Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG). Allgemeine Methoden, Version 4.0, 2011; in both, German and English]. URL: https://www.iqwig.de/download/IQWiG_Methoden_Version_4_0.pdf
16 Science and Technology Committee of the UK Parliament. Report on homeopathy. Evidence Check Report on Homeopathy considered flawed by MPs and dismissed by Government. URL: http://www.homeopathyevidencecheck.org/index.html
Funding / potential competing interests: In connection with the submitted manuscript, there are not any funding sources or other financial or personal relationships than stated in: Bornhöft G, Matthiessen P (eds.). Homoeopathy in Healthcare: Effectiveness, Appropriateness, Safety, Costs. Berlin, Heidelberg: Springer; 2011.