The relevance of relevance in research

DOI: https://doi.org/10.4414/smw.2013.13792

Introduction

In Switzerland, a new law requires that any research involving humans must aim to answer “a relevant research question” [1]. There is no such requirement in UK legislation, and the recently harmonised guidance document for National Health Service (NHS) research ethics committees (RECs) states that they “need not reconsider the quality of the science”, which suggests that scientific relevance remains outside the purview of RECs [2]. There are many problems with this stance, and some REC members have argued that RECs must examine the design and relevance of studies in order to ensure that they are ethical. This paper explains the relevance of relevance in research, analyses both the Swiss and British governance documents, and proposes a framework for researchers and REC members that enables a clearer conception of the relevance of relevance in research.

Two types of relevance

There are two key types of relevance: scientific relevance, where a study increases our understanding of a disease or a process, and societal relevance, where society directly benefits as a result of this increased understanding. Emanuel, Wendler and Grady explain the importance of relevance in terms of value, as follows:

“To be ethical, clinical research must be valuable, meaning that it evaluates a diagnostic or therapeutic intervention that could lead to improvements in health or well-being; is a preliminary etiological, pathophysiological or epidemiological study to develop such an intervention; or tests a hypothesis that can generate important knowledge about structure or function of human biological systems, even if that knowledge does not have immediate practical ramifications” [3].

Thus a study can be scientifically relevant without being societally relevant; for example, a randomised controlled trial of parachutes would confirm their effectiveness and add to the sum of scientific knowledge, but it is unlikely that any societal benefit would result from such a study because parachutes are already so widely used [4]. While the criterion of scientific relevance is often accorded most attention, the social value of research is increasingly regarded as an essential ethical consideration, particularly in resource-poor settings in the developing world where there is a history of participants and their communities being denied the benefits of the clinical trials conducted there. Much ambiguity remains regarding the precise responsibilities of researchers with respect to the value they ought to provide to communities in resource-poor settings [5].

There are many reasons why relevance is an important ethical issue [3]. First, irrelevant research squanders valuable resources. Research is an expensive undertaking, and limited resources must be used efficiently. If hundreds of thousands of Swiss francs or British pounds are spent on a trial that adds nothing to the sum of scientific knowledge, the money has gone to waste, and another, more worthwhile study might have gone unfunded, compounding the inefficiency. Second, irrelevant research harms participants; even minimal risks are unethical if no good can come from the results of the study, and any significant harm caused during an irrelevant study is appalling. Even studies that merely waste participants’ time are unethical if the research question or results are not relevant. Finally, irrelevant research corrupts the evidence base. If a REC approves a poorly designed study that will nonetheless provide publishable results, there are serious potential implications for policy-making and the art of medicine. For example, if a REC were asked to approve an antismoking in pregnancy intervention that used nonsmoking after 10 weeks as the endpoint, the results would be irrelevant, as they would not reveal anything about pregnant women’s smoking behaviours beyond this point. If a REC were to approve such a study, the results might be used to attract funding for a public health intervention that was not supported by any evidence; this would both waste public funds and potentially harm (in this case) future children.

There is also another, often forgotten, way in which research can be relevant: if it directly benefits participants. This remains a sensitive issue because of concerns about the therapeutic misconception, but it is nonetheless true that patients sometimes enrol in studies (particularly first-in-human trials) because there is a small chance that doing so will prolong their lives or improve their quality of life. Of course, RECs should not approve otherwise irrelevant research because it may benefit the participants, but this factor should not be overlooked.

It is also important to note that there are several ways in which a study can be irrelevant. Even if the research question is interesting and potentially of value, poor scientific design can negate the potential relevance of a research project: “even research asking socially valuable questions can be designed or conducted poorly and produce scientifically unreliable or invalid results” [3]. A study can be scientifically sound without being scientifically relevant (if the design is good but the question is not important), but it cannot be scientifically relevant without being scientifically sound. In this sense, scientific relevance is dependent on design quality. Furthermore, even if the question is interesting and the design and results are sound, they are rendered irrelevant if the results are not published.

Relevance in the Swiss Research Act

The new federal law regarding research on human beings was adopted by the Swiss parliament in September 2011 and will come into force in 2014 [1]. Its 68 articles cover a wide range of issues, from a basic definition of research to the penalties for breaches of the law itself (a maximum of three years imprisonment). Article 5 of the law is entitled “Relevant scientific questions” and states that “research on human beings can take place only if the scientific question concerned is relevant for one of the following domains: understanding of human diseases; the structure and functioning of the human body; and public health” (French text below; translations used for all other quotations) Art. 5 “Problématique scientifique pertinente”:

“La recherche sur l’être humain peut être pratiquée uniquement si la problématique scientifique concernée est pertinente pour l’un des domaines suivants: la compréhension des maladies humaines; la structure et le fonctionnement du corps humain: la santé publique” [1].

Taken at face value, this appears to be a sensible article: no-one wants pointless research to take place, and this phrasing would also appear to rule out “me-too” studies that exist purely to aid the marketing of particular drugs. However, although Article 3 defines “research” and several other terms, it unfortunately does not define “relevance”. The obvious assumption is that only research which generates knowledge that is not only theoretically but also societally useful is to be regarded as relevant. While this sounds reasonable enough, a few questions remain. Article 3 defines research as “methodological research aiming to obtain generalisable knowledge”. There are three problems with this. First, it is a circular definition, as one cannot define research by referring to research. Second, “obtain” is slightly ambiguous, as one could obtain knowledge that is already possessed by someone else, but it seems reasonable to assume that “obtain new knowledge” is the intended meaning. Third, and most importantly, if research means “research leading to generalisable knowledge”, then research must by definition mean scientifically relevant research, rendering Article 5 redundant apart from its specification of particular domains. In other words, while article 5 makes sense taken on its own, Article 3’s definition of research as generating generalisable knowledge itself includes the idea of relevance. (However, it appears that “relevance” in Article 5 may also have a hidden meaning. An explanatory message from 2009 suggests that Article 5 refers to relevance not only in the sense of “scientifically relevant”, but also in the sense of ethical value; “appropriateness” might be a better translation [6]. Given that no mention of this is made in the law itself or its accompanying documentation, we will not discuss this further here.)

The Explanatory Report that accompanies the new law also mentions relevance (“pertinence”), but in a slightly different sense: in defining scientific quality, it states that “clinical trials should be designed to respond to a question that has been formulated in a relevant manner”, and elaborates that “rigorous literature review is required to guarantee that any results will contribute to the development of knowledge” [7]. This makes it even clearer that Article 5 is irrelevant in the sense that it does not add any new meaning: if scientific quality mandates scientific relevance, there is no need to have a separate article repeating this. To summarise, Article 3 defines research as studies that aim at obtaining generalisable knowledge, and Article 5 states that research must be relevant but does not specify what this means; the only answer is that it must obtain knowledge, but this was already stated in Article 3, rendering Article 5 redundant.

The Swiss Academy of Medical Sciences has a somewhat clearer position on relevance, stating that:

“A study must have sociological [sic] value. This first condition excludes studies that cannot provide any generalisable knowledge, studies that do not investigate any relevant questions, studies that are covered by already available confirmed data, or also studies for which publication is not envisaged” [8].

This closely echoes the arguments of Emanuel, Wendler and Grady; the fact that the new law is not quite so clear on this issue is unfortunate.

Other Swiss regulations also refer to relevance. Article 10 of the Ordinance relating to the Swiss law on therapeutic products (LPTh) imposes the obligation on RECs to check the relevance of clinical trials [9] (the LPTh has been in force since 2002) [10]. Three ordinances related to the new Swiss Research Act are currently under consultation; one on clinical trials, one on so-called nonclinical studies and a third “organisational ordinance”. Article 27 of the Clinical Trials Ordinance specifies that Swiss RECs must consider “the relevance of the scientific problem (as defined in Article 5), the choice of an appropriate scientific methodology, and good clinical practice” [11]. Of note here is the reference to methodology, which can be key in determining whether a study’s results are relevant, as we shall see.

Curiously, the new ordinance concerning nonclinical studies makes no reference whatsoever to relevance, suggesting that the Swiss Research Act’s relevance criterion applies only to clinical trials [12]. This in turn implies that it is acceptable to conduct irrelevant research if it only involves samples and data. The nonclinical ordinance states only that RECs must consider the scientific quality of studies (Article 12(c)1), which could be regarded as referring to relevance; nonetheless, the lack of a specific reference is puzzling, particularly given the claim in the 2009 message that relevance is one of the recognised evaluation criteria for ethics committees. (It may be that the reference to Article 5 was dropped from the nonclinical ordinance because of the three mentioned domains, which are quite specific, so as to avoid stifling nonclinical research in other areas.)

Relevance guidelines in the UK

In contrast to Switzerland, there is no specific law governing human research in the UK: the key guideline for research ethics committees is the recently harmonised Governance Arrangements for Research Ethics Committees (GAFREC) document. Section 5.4.2 states the following:

“A REC need not reconsider the quality of the science, as this is the responsibility of the sponsor and will have been subject to review by one or more experts in the field (known as ‘peer review’). The REC will be satisfied with credible assurances that the research has an identified sponsor and that it takes account of appropriate scientific peer review” [2].

The implication of this is that the scientific relevance of research, which is obviously a function of its scientific quality, is not something that RECs should consider (it is also questionable whether sponsors always have the competence to review research adequately, even if we disregard their conflicts of interest). However, Article 2.2.2 states that:

“Researchers must satisfy a research ethics committee that the research they propose will be ethical and worthwhile. The committee has to be assured that any anticipated risks, burdens or intrusions will be minimised for the people taking part in the research, and are justified by the expected benefits for the participants or for science and society” [2].

The contradiction is obvious; while Section 5.4.2 states that REC should limit themselves to being assured that the science is robust because it has already been reviewed, 2.2.2 invites committee members to decide whether the risks to research participants are outweighed by the benefits. There will only be any benefits to science or society if the research is relevant; this is actually hinted at by the use of the phrase “ethical and worthwhile”. If a study merely replicates existing knowledge, it cannot be worthwhile. Furthermore, if RECs did not concern themselves with addressing poorly designed studies, the results might be irrelevant even if the question was a good one [13]. For example, a study that sought to establish whether a new cancer drug was effective might well be relevant, but not if it aimed to recruit only 50 patients when 500 were required by the power calculation; this design flaw would render any results irrelevant. Ultimately, a study can be rendered irrelevant by validly answering an irrelevant question, or by invalidly answering a relevant question (or, of course, by invalidly answering an irrelevant question).

The Economic and Social Research Council guidelines, which govern nonclinical research in the UK, have also denied that RECs should review scientific design: “The scholarly or scientific standards/merits of the research are not the responsibility of the REC.” [14] However, the same document quotes the Research Integrity Office’s recommendation that researchers ask themselves the following question: “Does the proposed research address pertinent question(s) and is it designed either to add to existing knowledge about the subject in question or to develop methods for research into it?” [14].

RECs and relevance

The new Swiss Research Act focuses on scientific relevance, while the British GAFREC guidance states that science and society are both relevant, echoing the SAMW’s criterion of sociological value. (Of course, anything that is relevant to science is likely to be beneficial to society.) But how should RECs in these countries and elsewhere make decisions concerning relevance?

RECs should clearly reject irrelevant research in most cases, for the reasons given earlier in this paper: it squanders resources, harms participants, and corrupts the evidence base. However, it is often difficult for RECs to forecast accurately the prospective benefit of studies, making assessment of the scientific and societal relevance of a study difficult. Furthermore, both NHS and university RECs in the UK often review studies that are of questionable scientific relevance, often owing to small sample sizes or unoriginal research questions. However, in many cases these studies are approved, despite these committees’ adherence to the GAFREC or ESRC guidance. How can this be? In most cases the reason for approval is that it is pragmatically recognised that, while the results of such studies are unlikely to benefit science or society greatly, conducting a research project is an essential component most postgraduate educational courses. While the research in question might not be scientifically relevant, the education and qualification of a new generation of scientists and doctors is certainly societally relevant. Although this is a different type of societal relevance from that attached to the generalisable knowledge created as a result of most studies, such education is nonetheless highly valuable. If these studies were rejected because of hard-line adherence to the criterion of scientific relevance, young doctors and scientists would not receive the valuable experience of conducting research as part of their education. Furthermore, many patients are happy to be treated by student doctors despite the possibility of inferior clinical care; if the risks are minimal, there is no reason why people should not be able to participate in research that is of little direct scientific benefit if doing so aids the educational process. For example, if a student study aimed to evaluate whether the Nintento Wii entertainment system helped students to relax between exams, it is questionable whether this would qualify as a relevant research question. Even if this were regarded as an important issue, the study would be rendered irrelevant if the sample size was only five students, as it would be underpowered. Nonetheless, students could still learn a great deal about research methods and design by carrying out such a study.

Of course, this is only true of studies where there is minimal risk to participants; subjects should always be informed that research constitutes part of an educational course, and the researchers must not “overpromise” ground-breaking results. Ideally, of course, all student projects would also be scientifically relevant. In this sense, any law that focuses on the specific criterion of scientific rather than societal benefit would rule out some types of nonessential research that are relevant in others ways, such as educational research projects, despite their obvious value.

Sarah Edwards has argued that exceptions should not be made for student projects, arguing that “educational objectives cannot be met without laying down standards of good science…weak science is unnecessary for educational purposes …[and is]… unlikely to produce good researchers in the future” [16]. However, both RECs and students’ supervisors can set a standard of good science while being honest with the student that a particular project is flawed in some ways. Just as student researchers must not “overpromise” when explaining a study to potential participants, RECs and supervisors must make it clear when a study is scientifically flawed but still of educational benefit. And just as a study can be scientifically relevant without being societally relevant, scientifically irrelevant research can be of societal value through the educational experience it provides to students.

Conclusion

In summary, there are two types of relevance: scientific/medical relevance, and societal relevance, which should include the education of future researchers, scientists and doctors. And there are three types of scientific irrelevance: studies do not answer new interesting questions, studies that suffer from bad design, and, therefore, have invalid and irrelevant results, and studies that do generate new knowledge but do not make the information available (such as some trials by pharmaceutical companies [16]). The new Swiss law recognises the importance of relevance but does not define it or research very clearly, leading to some ambiguity. The UK guidelines state both that relevance is relevant and that it isn’t, but we have seen that the latter is false. RECs cannot ensure that research is ethical without assessing relevance, as stated in the GAFREC document: “RECs take into account the interests and safety of the researchers, as well as the public interest in reliable evidence affecting health and social care, and enables ethical and worthwhile research of benefit to participants or to science and society.” Other countries should make relevance a key requirement for ethical research while taking care to provide a clear definition of the concept.

References

  1 Swiss Federal Government. Human Research Act. (Loi fédérale relative à la recherche sur l’être humain.) http://www.bag.admin.ch/themen/medizin/00701/00702/07558/index.html?lang=fr (accessed 18/9/12)

  2 UK Department of Health. Governance arrangements for research ethics committees: a harmonised edition. http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_126474 (accessed 18/9/12)

  3 Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical? JAMA. 2000;283(20):2701–11.

  4 GCS Smith, JC Pell. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ. 2003;327:1459.

  5 Lairumbi GM, Parker M, Fitzpatrick R and MC English. Ethics in practice: the state of the debate on promoting the social value of global health research in resource poor settings particularly Africa. BMC Medical Ethics. 2011;12:22.

  6 Swiss Federal Government. Message sur la loi fédérale relative à la recherche sur l’être humain. http://www.admin.ch/ch/f/ff/2009/7259.pdf (accessed 18/9/12)

  7 Swiss Federal Government. Explanatory Report on the Human Research Act. (Rapport explicatif sur les ordonnances découlant de la loi relative à la re-cherche sur l’être humain.) http://www.bag.admin.ch/themen/medizin/00701/00702/12310/index.html (accessed 18/9/12)

  8 Swiss Academy of Medical Sciences. Research with Human Subjects: A Manual for General Practice. http://www.samw.ch/en/Projects/Research-in-Humans.html (accessed 18/9/12)

  9 Swiss Federal Government. Clinical trials ordinance. (Ordonnance sur les essais cliniques de produits thérapeutiques.) http://www.admin.ch/ch/f/rs/8/812.214.2.fr.pdf (accessed 18/9/12)

10 Swiss Federal Government. Therapeutic Products Act. (Loi fédérale du 15 décembre 2000 sur les médicaments et les dispositifs médicaux). http://www.admin.ch/ch/f/rs/c812_21.html (accessed 18/9/12)

11 Swiss Federal Government. Clinical trial ordinance. (Ordonnance sur les essais cliniques) http://www.admin.ch/ch/f/rs/812_214_2/index.html (accessed 18/9/12)

12 Swiss Federal Government. Non-clinical ordinance. (Ordonnance sur les projets de recherche sur l'être humain à l’exception des essays cliniques.) http://www.bag.admin.ch/themen/medizin/00701/00702/12310/index.html http://www.bag.admin.ch/themen/medizin/00701/00702/12310/index.html?lang=fr&download=NHzLpZeg7t,lnp6I0NTU042l2Z6ln1ae2IZn4Z2qZpnO2Yuq2Z6gpJCKfH96fWym162epYbg2c_JjKbNoKSn6A-- (accessed 18/9/12)

13 Shaw D. The Ethics Committee as Ghost Author. J Med Ethics. 2011;37:706.

14 Economic and Social Research Council. Framework for research ethics. http://www.esrc.ac.uk/about-esrc/information/research-ethics.aspx (accessed 18/9/12)

15 Goldacre B. Bad Pharma. Fourth Estate, London, 2012.

16 Edwards S. Student Projects in Medicine: A Lesson in Science and Ethics. Accountability in Research: Policies and Quality Assurance 2009;16:285–306.