Lost in translation: the impact of publication language on citation frequency in the scientific dental literature

DOI: https://doi.org/10.4414/smw.2011.13148

D Poomkottayil, MM Bornstein, P Sendi

Summary

PRINCIPLES: Citation metrics are commonly used as a proxy for scientific merit and relevance. Papers published in English, however, may exhibit a higher citation frequency than research articles published in other languages, though this issue has not yet been investigated from a Swiss perspective where English is not the native language.

METHODS: To assess the impact of publication language on citation frequency we focused on oral surgery papers indexed in PubMed MEDLINE that were published by Swiss Dental Schools between 2002 and 2007. Citation frequency of research papers was extracted from the Institute for Scientific Information (ISI) and Google Scholar database. An univariate and multivariate logistic regression model was used to assess the impact of publication language (English versus German/French) on citation frequency, adjusted for journal impact factor, number of authors and research topic.

RESULTS: Papers published in English showed a 6 (ISI database) and 7 (Google Scholar) times higher odds for being cited than research articles published in German or French.

CONCLUSIONS: Our results suggest that publication language substantially influences the citation frequency of a research paper. Researchers should publish their work in English to render them accessible to the international scientific community.

Introduction

One objective measure to assess the influence and scientific merit of a research article is the frequency with which the study is cited in subsequent publications [1–4] . The vast majority of research articles are published in English [5]. Only a minority of scientific journals publish their articles in other languages such as German or French. And, due to language restrictions, only a minority of researchers read these articles. Publishing research articles in languages other than English diminishes the research results’ accessibility and visibility to a wider audience [6].

In previous research from other fields such as ecology, no association between national language and citation frequency was found [7]. However, articles with a larger number of authors were cited more frequently by subsequent research papers [3, 7, 8] . In a similar study in the field of epidemiology, much-cited articles were more likely to be published in high-impact journals and by authors from US institutions, suggesting that publication language (or country) may be an important factor in determining the citation frequency of a research paper [2, 3]. Similarly, Callaham reported an association between journal impact factor and citation frequency of research papers in the field of emergency medicine [9].

To the best of our knowledge, the impact of publication language on citation frequency in the scientific literature has not yet been investigated from a Swiss perspective. In the present study we focused on oral surgery papers published by Swiss dental schools between 2002 and 2007, to assess the impact of publication language (English versus German/French) on citation frequency, adjusted for potential confounders including number of authors, journal impact factor and research topic. The null hypothesis in our study is therefore that there is no difference in citation frequency among oral surgery papers published in English or German/French by Swiss dental schools.

Methods

Publication retrieval

To extract publications in oral surgery that were authored or co-authored by dental surgeons in Switzerland, where the native language is not English, the online PubMed MEDLINE database (link: http://www.ncbi.nlm.nih.gov/pubmed ) was searched by the names of senior staff members of the oral surgery departments at the four Swiss dental schools listed on the departmental website in 2007. This included three German-language dental schools at the Universities of Basel, Bern and Zurich, and one French-language dental school at the University of Geneva. We focused on oral surgery papers, an exploratory MEDLINE search having indicated that in oral surgery a substantial proportion of research papers were published in German or French, facilitating statistical analysis. We limited our search to all publications indexed in Pubmed MEDLINE between the years 2002 and 2007.

All publications were categorised with respect to publication language (English versus Native), number of authors, publication type (original article, review, case report) and research topic (implantology versus other) by reading the abstract and index data provided by the PubMed MEDLINE database. Marginalia such as editorials or book reviews were not included in our study. In addition, we used the Journal Citation Reports database in the publication year of a research paper to extract the respective journal impact factor (http://thomsonreuters.com/). Journals that were not indexed in the Journal Citation Reports database and therefore had no impact factor yet were arbitrarily assigned a value of 0.5, the lowest indexed impact factor in the “dentistry, oral surgery & medicine” category for the year 2007. The journal most frequently assigned an IF of 0.5 was the “Schweizerische Monatsschrift für Zahnmedizin.”

To estimate the citation frequency of research articles included in our study, we accessed the Institute for Scientific Information (ISI) Web of Science database (http://thomsonreuters.com/) and the Google Scholar database (http://scholar.google.ch) between 25 and 27 May 2009 and recorded the number of times each research paper was cited until that respective date [10, 11]. We included papers published between 2002 and 2007 for further analysis in our study, to allow sufficient follow-up time and also capture citations of the more recent papers published in 2007. The Google Scholar database was used in addition to the ISI database, since Google Scholar is less restrictive with respect to the citations that are indexed, yielding more citations per article.

Statistical analysis

Data were first analysed descriptively and then supplemented with inferential statistical methods. Between-group differences of continuous and categorical variables were compared using the Wilcoxon rank sum test and chisquare test where appropriate. Variables with a statistically significant between-group difference were included for further analysis in multivariate models.

In a multivariate logistic regression model we analysed the association between publication language and likelihood of citation (cited at least once versus no citation), adjusted for number of authors, journal impact factor (categorised as < 1, >= 1 – 1.99, >= 2) and research topic (implantology versus other) that were considered potential confounding variables. We performed this analysis twice, once for citation data extracted from the ISI database and once for citations extracted from the Google Scholar database.

In a further analysis we excluded all papers that were not cited at all and used a multivariate linear regression model to analyse the association between publication language and citation frequency, adjusted for number of authors, journal impact factor and research topic that were considered to be potential confounding variables. This analysis was also performed twice, once for citation data extracted from the ISI database and once for citations extracted from the Google Scholar database. The fit of the model was evaluated by checking residuals as well as by comparing the results with robust regression methods. We used S-Plus Professional (Insightful Corp., Seattle, USA) for all statistical analyses.

Results

A total of 301 oral surgery research papers published between 2002 and 2007 were extracted from the MEDLINE database for further analysis. A total of 212 papers (70%) were published in English and 89 (30%) in German or French (table 1). The majority of papers represented original articles, including 82% and 72% of papers published in English and German/French respectively. Substantially more papers on implantology were published in English (38%) than in German or French (9%), this difference being statistically significant (table 1). Furthermore, the average journal impact factor of English language papers was three times higher than those published in German or French. Papers published in English were cited on average 12.9 times (ISI database) or 17.8 times (Google Scholar) whereas papers published in German or French were cited only once on average (table 1). Of the 301 papers evaluated, 72 (ISI database) and 63 (Google Scholar) had zero citations respectively.

In a multivariate logistic regression model, papers published in English had a 6 times (ISI database) and 7 times (Google Scholar) higher odds of being cited at least once compared to papers published in German or French (table 2). This association was statistically significant and independent of number of authors, journal impact factor and research topic.

In a multivariate linear regression model including papers that were cited at least once, papers published in English were cited 5 times more often than papers in German or French when the ISI database was used, this figure being non-significant (p = 0.10). However, when the Google Scholar database was used, the same figure was statistically significant (p = 0.009) (table 3). These figures were adjusted for number of authors, journal impact factor and research topic. Papers published in the field of implantology were on average cited 11 times (ISI database) and 13 times (Google Scholar) more often than papers in other fields of oral surgery (table 3).

Table 1: Characteristics of oral surgery research papers published in English and native language (German or French).
  English (n = 212) Native (German or French) (n = 89) p-value
Number of authors       mean (SD) 4.5 (2.0) 3.3 (1.3) p <0.0001
      Case report 173 (82%) 19 (9%) 20 (9%) 64 (72%) 17 (19%) 8 (9%) p = 0.0531
Research topic – n (%)       Implantology       Other 81 (38%) 131 (62%) 8 (9%) 81 (91%) p <0.0001
Journal impact factor       mean (SD)       median (IQR*) 1.66 (1.29) 1.42 (1.02–1.92) 0.49 (0.49) 0.5 (0.5–0.5) p <0.0001
Citations ISI database       mean (SD)       median (IQR) 12.85 (16.96) 7.50 (2.0–18.0) 1.08 (1.67) 1.07 (0.0–2.0) p <0.0001
Citations Google Scholar       mean (SD)       median (IQR) 17.75 (21.16) 11.00 (3.00–26.00) 1.08 (1.43) 0.0 (0.00–2.00) P <0.0001
*IQR denotes interquartile range
Table 2: Multivariate analysis of the association between publication language and likelihood of citation (n = 301).
  Odds ratio Wald Z p-value
ISI database      
Publication language (English versus Native) 6.09 4.08 < 0.0001
Number of authors 1.21 1.75 0.0800
Journal impact factor       1-1.9 versus <1       >=2 versus <1 1.06 1.80 0.11 0.81 0.9104 0.4199
Research topic (Implantology versus other) 1.52 1.00 0.3168
Google Scholar      
Publication language (English versus Native) 7.07 4.14 < 0.0001
Number of authors 1.63 3.52 0.0004
Journal impact factor       1-1.9 versus <1       >=2 versus <1 2.36 0.88 1.36 -0.17 0.1727 0.8618
Research topic (Implantology versus other) 1.07 0.16 0.8722
Table 3: Multivariate analysis of the association between publication language and frequency of citation.
  Mean difference in citation t-value p-value
ISI database (n = 229)      
Publication language (English versus native) 5.11 1.63 0.1043
Number of authors 1.14 2.20 0.0288
Journal impact factor       1-1.9 versus <1       >=2 versus <1 4.08 2.18 1.50 0.67 0.1346 0.5032
Research topic (Implantology versus other) 11.31 5.22 <0.0001
Google Scholar (n=238)      
Publication language (English versus Native) 5.11 2.63 0.0090
Number of authors 1.25 1.94 0.0540
Journal impact factor       1-1.9 versus <1       >=2 versus <1 1.22 0.06 0.73 0.05 0.4650 0.9534
Research topic (Implantology versus other) 13.57 5.06 <0.0001

Discussion

In our study we have found that (i) scientific papers published in English have a 6–7 times higher odds for being cited at least once and that (ii) among those papers that are cited at least once, papers in English are cited 5 times more often that those published in German or French.

Our results stress the importance of English as today's scientific language [6]. Every era has had its dominant scientific language(s). German and French, equally important national languages since the 19th century, have suffered a steep decline since World War II – or least since the 1970s [5, 12]. In Germany, Austria and German-speaking Switzerland, English became the dominant language of science in the early 1970s [5, 12]. The present study confirms this trend with regard to publications in the field of oral surgery, originating from dental schools in Switzerland.

Like native English speakers, authors in countries where English is not the first language prefer to publish in English – possibly because such articles have a higher impact than those in their native tongue [6]. They also tend to cite English-language articles even in non-English language publications [13]. In our study English language papers had a 6 (ISI database) and 7 (Google Scholar) times higher odds for being cited than research articles published in German or French. These results are in line with the widespread opinion that research articles need to be published in English to render them accessible to the international scientific community [6]. Even when considering articles that were cited at least once, articles in English were cited 5 times more often than those published in German and French (table 3).

Although our study focused on oral surgery research papers, it seems reasonable to assume that the results also hold true for papers published in other areas of medicine. It is clear that, because of language restrictions, articles not published in English are only read by a minority of researchers worldwide. In the present study, 72% of the papers published in German or French were original articles, indicating that in dentistry important research results are still communicated in a native language rather than English. In an exploratory PubMed MEDLINE search, this figure was tentatively substantially lower for other areas of medicine such as internal medicine, where important original articles are usually published in English. Thus, a statistical analysis on the association between publication language and citation frequency among internal medicine papers would have been complicated by a loss of statistical power.

We considered impact factor, research topic and number of authors potentially important confounders. Journals with a higher impact factor are usually English language journals and therefore also attract articles of greater importance that are also cited more frequently. In oral surgery, implantology represents an important research area with a large number of frequently cited papers that are also preferably published in English language journals. We therefore considered implantology as compared to other areas of oral surgery (e.g., third molar surgery) as an important potential confounder. Finally, the number of authors may also be a confounder since articles with a larger number of authors are often published in English and cited more frequently.

Our results also show that papers with more authors are cited more frequently (table 2 and 3). These results are in line with findings in other fields such as ecology, where citation frequency increases with number of authors, suggesting that the number of authors is an important attribute of a publication [3, 7]. There may be several explanations for this finding. As the number of authors increases, opportunities for self-citations concomitantly increase [7]. An increase in citation frequency with number of authors may reflect (1) an overall favourable effect of collaboration on the quality or scientific merit of an article, perhaps due to additional editing or the study authors’ greater diversity in experience and background, or (2) that the community perceives greater merit for an article when more authors are listed, i.e. “they can’t all be wrong” [7].

Our study has several limitations. First, we limited our search to articles written by oral surgeons who are members of Swiss dental schools. Our results therefore reflect the situation in Switzerland, where multiple national languages exist. Our results may not necessarily be transferable to other countries where English is not the native language. Second, our findings may also depend on the research area. We chose oral surgery as the focus of our literature search since a substantial proportion of papers are still published in German or French and therefore allowed a better statistical analysis of this issue. It may well be that in other research areas such as linguistics, where research results are published as monographs rather than research papers, publishing in English may not be as important as in medicine or basic sciences. Third, we used two databases, Google Scholar and the ISI database, as the source for our citation analysis. The results of the multivariate analyses are not identical for the two databases. However, the differences are small and may be due to the fact that Google Scholar searches generally yielded more citations per article than ISI searches.

In conclusion, the results of our study suggest that research papers published in English are cited more frequently by other authors, independently of journal impact factor, number of authors and research topic. Researchers from countries where English is not the native language would substantially benefit from publishing in English to render their research findings accessible to the international readership.

Study funding / potential competing interests

No financial support was available for this study. The authors have no competing interests.

Correspondence: Pedram Sendi, MD DDS PhD Institute for Clinical Epidemiology & Biostatistics Basel University Hospital Hebelstrasse 10 CH-4031 Basel Switzerland pedram.sendi@unibas.ch

References

  1 Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002:2847–50.

  2 Filion KB, Pless IB. Factors related to the frequency of citation of epidemiologic publications. Epidemiol Perspect Innov; 2008:3.

  3 Leimu R, Koricheva J.What determines the citation frequency of ecological papers? Trends in Ecology & Evolution. 2005;20(1):28–32.

  4 Bhandari M, Busse J, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta Iii P, et al. Factors associated with citation rates in the orthopedic literature. Can J Surg. 2007:119–23.

  5 Winkmann G, Schlutius S, Schweim HG. Publication languages of Impact Factor journals and of medical bibliographic databanks. Dtsch Med Wochenschr. 2002:131–7.

  6 Bakewell D.Publish in English, or perish? Nature. 1992;356:648.

  7 Borsuk R, Budden A, Leimu R, Aarssen L, Lortie C.The influence of author gender, national language and number of authors on citation rate in ecology. The Open Ecology Journal. 2009;2(1):25–8.

  8 Kostoff R.The difference between highly and poorly cited medical articles in the journal Lancet. Scientometrics. 2007;72(3):513–20.

  9 Callaham M, Weber E, Wears R. Citation characteristics of research published in Emergency Medicine versus other scientific journals. Ann Emerg Med. 2001:513–7.

10 Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of Citations in Web of Science, Scopus, and Google Scholar for Articles Published in General Medical Journals. JAMA: The Journal of the American Medical Association; 2009:1092–6.

11 Noruzi A.Google Scholar: The new generation of citation indexes. LIBRI-COPENHAGEN-. 2005;55(4):170.

12 Winkmann G, Schlutius S, Schweim HG. Citation rates of medical German-language journals in English-language papers – do they correlate with the impact factor, and who cites? (reprint). Klin Monbl Augenheilkd. 2002:72–8.

13 Chew M, Villanueva EV, Van Der Weyden MB. Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994–2005) and their Editors’ views. J R Soc Med. 2007:142–50.