Impact factor trends for general medical journals: non-English-language journals are lagging behind

DOI: https://doi.org/10.4414/smw.2012.13572

Siri Vinther, Jacob Rosenberg

Summary

BACKGROUND: The impact factor (IF) is a common citation metric used for evaluating and comparing scientific journals within a certain field. Previous studies have shown that IFs are increasing. However, rates may depend on journal publication language.

The aim of this study was to determine IF values and trends for general medical journals, comparing non-English-language with English-language journals.

METHODS: For all journals categorised as “medicine, general and internal” (n = 150) in the Journal Citation Reports (JCR), publication language, country of origin and IFs for the last 10 years were recorded (2001–2010).

Data were classified, analysed descriptively and compared using non-parametric tests.

RESULTS: From 2001 to 2010, IFs increased for English-language and non-English-language journals (p <0.001). During the 10-year study period, IFs were higher for English-language than for non-English-language journals (p <0.001).

The proportion of non-English-language journals included in the JCR was 12.2% in 2001 and 18.0% in 2010 (p = 0.28).

INTERPRETATION: From 2001 to 2010, IFs increased significantly for English-language and non-English-language journals. When comparing IF values year-by-year (2001–2010), IFs were significantly higher for English-language than for non-English-language journals.

In an international scientific community with English as the universal language of science, non-English-language journals should consider changing publication language, and adopt either a bi- or a monolingual approach. Publishing in English will increase citation counts and thus IFs, but, more importantly, scientific findings will be accessible to a much wider audience.

Introduction

Every year, Thomson Reuters publishes impact factors (IF) for more than 7,350 science journals in the Journal Citation Reports® (JCR) [1]. The IF is used as a proxy measure for the relative importance of scientific journals, reflecting the relationship between citing and cited articles. More specifically, IF is the number of citations appearing in publications in a given year to articles published in a given journal in the previous two years, divided by the number of citable articles published in those two years [2]. Previous studies have demonstrated that IFs are increasing [3–5], but rates may depend on publication language. Relatively few non-English-language journals are included in the JCR, but the non-English-language journals which are included seem to have lower IFs compared with English-language journals [6–16]. Despite being one of the largest subject categories in the JCR, IF values and trends for the JCR subject category “medicine, general and internal” – to which general medical journals belong – have not been studied thoroughly.

The aim of this study was to determine IF values and trends for general medical journals, comparing non-English-language with English-language journals as well as journals published in different continents.

Methods

All journals classified as “medicine, general & internal” in the JCR were analysed (150 journals in September 2011). For each journal, IFs for the last 10 years were recorded (2001–2010). Publication language and country of origin were also recorded.

Data were grouped into separate categories: language (English-language, non-English-language or multi-language) and continent of origin (North America, Europe, Australia, Africa, Asia and South America)).

Each category was analysed descriptively (number of journals, median IFs and IF inter-quartile range (IQR)).

For English-language, non-English-language and multi-language journals, Friedman’s test (the non-parametric equivalent to the one-way ANOVA with repeated measures) was used to test for IF trends. This was done for the time periods 2001–2010, 2001–2005, and 2006–2010.

For English-language, non-English-language and multi-language journals, Wilcoxon’s signed-ranks test (the non-parametric equivalent to the dependent t-test) was used to compare 2001-IFs with 2010-IFs.

For English-language versus non-English-language journals, Mann-Whitney U test (the non-parametric equivalent to the independent t-test) was used to compare IFs for all registered values during the study period 2001–2010 (year-by-year comparison).

The proportions of English-language versus non-English-language journals included in the JCR in 2001 and 2010 were compared using Fisher’s exact test.

For statistical analysis, we used IBM SPSS Statistics 19®.

Results

Characteristics of the 150 journals classified as “medicine, general and internal” in the JCR are shown in tables 1a and 1b. IF trends for both categories (language and continent of origin) are shown in figures 1a and 1b.

Figure 1a

From 2001 to 2010 (and especially from 2008 onwards), the total number of journals included in the JCR increased (from 90 to 150). Increases were relatively higher for European, Asian and South-American journals as well as for non-English-language journals.

The proportion of non-English-language journals increased from 2001 to 2010 (from 12.2% [11/90] to 18.0% [27/150]) (p = 0.28). The four major non-English languages in 2010 were Spanish (21.4%), German (14.3%), Serbian (14.3%) and French (10.7%).

From 2001 to 2010, IFs varied significantly for English-language journals (p <0.001), non-English-language journals (p <0.001) and multi-language journals (p <0.001). From 2001–2005, IFs varied significantly for English-language journals (p <0.001) and for multi-language journals (p = 0.034) but not for non-English-language journals (p = 0.290). From 2006–2010, IFs varied significantly for all three language groups (p <0.001 for English-language and non-English-language journals; p = 0.005 for multi-language journals).

For all three language groups (English-language, non-English-language and multi-language journals), there was a significant difference between IFs in 2001 and IFs in 2010 (p <0.001 for English-language and non-English-language journals; p = 0.008 for multi-language journals).

When comparing all IF values year-by-year (English-language versus non-English-language journals), IFs were significantly higher for English-language journals (p <0.001).

Table 1a: Characteristics of journals (classification: language).
  2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
English-language journals
     No of journals 71 75 76 76 78 78 79 84 102 114
     English:all journals 79.8% 80.6% 80.9% 80.9% 80.4% 80.4% 80.6% 80.0% 78.5% 76.0%
     Median IF 0.78 0.92 1.02 1.22 1.25 1.34 1.55 1.72 1.49 1.45
     25% quartile IF 0.46 0.57 0.58 0.65 0.66 0.75 0.86 1.00 0.83 0.73
     75% quartile IF 2.07 1.98 1.95 2.18 2.49 2.55 2.72 2.76 2.44 2.53
Non-English-language journals
     No of journals 11 10 10 11 11 11 11 12 19 27
     Non-English:all journals 12.4% 10.8% 10.6% 11.7% 11.3% 11.3% 11.2% 11.4% 14.6% 18.0%
     Median IF 0.33 0.35 0.32 0.31 0.36 0.34 0.36 0.46 0.42 0.29
     25% quartile IF 0.22 0.30 0.26 0.25 0.26 0.28 0.28 0.32 0.16 0.13
     75% quartile IF 0.40 0.40 0.44 0.47 0.41 0.45 0.48 0.60 0.51 0.47
Multi-language journals
     No of journals 8 8 8 8 8 8 8 9 9 9
     Multi:all journals 9.0% 8.6% 8.5% 8.5% 8.2% 8.2% 8.2% 8.6% 6.9% 6.0%
     Median IF 0.36 0.36 0.37 0.38 0.76 0.67 0.74 1.16 1.19 0.70
     25% quartile IF 0.31 0.28 0.34 0.31 0.34 0.34 0.33 0.56 0.51 0.52
     75% quartile IF 0.60 0.58 0.55 0.78 1.08 1.09 1.18 1.28 1.23 1.40
                     
Total no of journals 90 93 94 95 97 97 98 105 130 150
Table 1b: Characteristics of journals (classification: continent of origin).
  2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
North-America
     Number of journals 36 37 37 36 38 39 39 42 42 44
     North-American:all 40.0% 39.8% 39.4% 37.9% 39.2% 40.2% 39.8% 40.0% 32.3% 29.3%
     Median IF 1.05 1.23 1.40 1.49 1.63 1.90 1.88 2.11 2.11 1.97
     25% quartile IF 0.67 0.72 0.86 0.86 1.04 1.12 1.09 1.37 1.35 1.40
     75% quartile IF 2.56 2.75 2.90 3.32 3.74 3.91 4.45 3.71 4.21 4.15
Europe
     Number of journals 35 38 38 38 38 38 40 42 58 64
     European:all 38.9% 40.9% 40.4% 40.0% 39.2% 39.2% 40.8% 40.0% 44.6% 42.7%
     Median IF 0.73 0.73 0.73 0.85 0.94 0.96 1.26 1.27 0.98 0.70
     25% quartile IF 0.36 0.37 0.37 0.47 0.46 0.41 0.50 0.57 0.41 0.34
     75% quartile IF 1.37 1.40 1.25 1.56 1.70 1.83 1.83 2.21 1.70 1.87
Australia
     Number of journals 1 2 2 2 2 2 2 2 3 4
     Australian:all 1.1% 2.2% 2.1% 2.1% 2.1% 2.1% 2.0% 1.9% 2.3% 2.7%
     Median IF 1.79 1.15 1.37 1.52 1.82 1.99 2.05 2.67 1.79 1.14
     25% quartile IF 1.79 0.88 1.19 1.28 1.67 1.70 1.81 2.35 1.18 0.58
     75% quartile IF 1.79 1.41 1.56 1.76 1.97 2.29 2.30 3.00 2.34 1.89
Africa
     Number of journals 2 1 1 2 2 1 1 1 1 3
     African:all 2.2% 2.2% 2.1% 2.1% 2.1% 2.1% 2.0% 1.9% 1.5% 1.3%
     Median IF 0.57 1.02 0.99 1.11 1.07 1.23 1.27 1.04 1.33 0.52
     25% quartile IF 0.57 1.02 0.99 1.11 1.07 1.23 1.27 1.04 1.33 0.34
     75% quartile IF 0.57 1.02 0.99 1.11 1.07 1.23 1.27 1.04 1.33 1.10
Asia
     Number of journals 13 12 13 14 14 14 13 15 19 26
     Asian:all 14.4% 12.9% 13.8% 14.7% 14.4% 14.4% 13.3% 14.3% 14.6% 17.3%
     Median IF 0.34 0.41 0.42 0.46 0.56 0.67 0.64 0.84 0.82 0.71
     25% quartile IF 0.11 0.27 0.37 0.33 0.43 0.46 0.50 0.61 0.53 0.27
     75% quartile IF 0.42 0.51 0.55 0.59 0.64 0.84 0.82 1.00 0.98 1.06
South America
     Number of journals 3 3 3 3 3 3 3 3 7 9
     South-American:all 3.3% 3.2% 3.2% 3.2% 3.1% 3.1% 3.1% 2.9% 5.4% 6.0%
     Median IF 0.34 0.29 0.33 0.28 0.35 0.27 0.32 0.44 0.51 0.41
     25% quartile IF 0.32 0.27 0.30 0.28 0.31 0.26 0.26 0.33 0.41 0.20
     75% quartile IF 0.35 0.31 0.34 0.30 0.36 0.34 0.33 0.44 0.58 0.55
                     
Total number of journals 90 93 94 95 97 97 98 105 130 150

Interpretation

The main finding of this study was that non-English-language journals had significantly lower IFs than English-language journals. Although not significant, it is clear from the graphic presentation (fig. 1a) that IFs of English-language journals have increased at a higher rate than those of non-English-language journals during the last decade. This could suggest that scientific findings published in non-English languages do not have the same international impact as those published in English.

Figure 1b

From 2008, median IFs decreased for both categories (language and continent of origin). This could be due to the large increase in the number of journals with relatively lower IFs (i.e., non-English-language journals and/or journals from “non-English-speaking” continents).

Within clinical medicine, several studies have examined different subject categories and possible associations between IF and language and/or continent of origin [6, 7, 9, 10, 13, 14]. In brief, findings suggest that IFs are increasing with rates depending on publication language and/or country of origin.

Trends for journals classified as “medicine, general and internal” have not been studied thoroughly, despite the fact that it is one of the largest subject categories in the JCR. One study demonstrated increasing IF trends for seven high-impact general medical journals [7]. Another study analysed possible associations between 2003 IFs and publication language [15]. This study concluded that IFs of English-language general medical journals were significantly higher than those of non-English-language journals.

In this study of general medical journals, IF values and trends over a ten-year period were analysed for different categories (language and continent of origin). Despite being more comprehensive than some previous studies, this one has some limitations, one of which may be that only general medical journals were included. Yet, IFs should not be used for comparison across different subject categories [5, 17, 18]. For instance, high IFs can be much more difficult to achieve in “small” specialities with few researchers, since the number of citations will be lower and there may be different citation traditions in different clinical fields.

The IF has been the subject of major debate ever since this citation metric was devised. Most people agree that the IF reflects journal popularity and prestige, but the use of the IF to evaluate journal quality has been heavily criticised. The criticism mainly concerns “citation inflation” and the fact that IFs can be editorially manipulated [2, 19, 20]. In brief, IF increases when the number of “cited” articles increases – and/or the number of “citable” articles decreases. As regards the former, some aspects are difficult to control: the amount of scientific literature is getting larger, reflecting the fact that the number of scientists as well as the number of scientific journals are increasing. Accessing scientific literature is getting easier because of “open access” and the fact that the quantity and quality of indexing and search engines are growing. Moreover, reference lists are getting longer: authors cite more articles and proportionately more of these citations are of recent articles (cf. the IF equation). In addition, trans-disciplinary citations and self-citations (authors citing their own previous work) are getting more common [18, 20–24].

Editorial policies that may affect IFs include recruitment or selection of popular/controversial researchers whose articles will receive special attention and thus more citations. Favouring large and scientifically active research groups, thus increasing the potential for author self-citation, will also increase IF. Moreover, selecting specific article types or articles with specific outcomes are other options: review articles generally receive more citations compared with original articles and especially case reports, and articles with favourable outcomes receive more citations than negative or confirmatory studies. Reducing publication volume and/or publishing fewer “citable” articles will also increase the IF (it is not known how “citable” articles are defined by Thomson Reuters; however, publications without an abstract and/or with a limited number of references are generally not regarded as citable [e.g. editorials, letters to editors and errata]). Finally, changing publication language into English will expand a journal's readership and thus the potential for receiving more citations [5, 25–28].

The importance of publication language for journal visibility is interesting. Relatively few non-English-language journals are included in the JCR. The inclusion criteria of the JCR are not official, but publication language and thus international orientation are important parameters. Thomson Reuters states that the JCR is not and should not be “all-inclusive”, albeit “comprehensive”: “English is the universal language of science at this time in history. It is for this reason that Thomson Reuters focuses on journals that publish full text in English or at the very least the bibliographical information in English. […] However, going forward, it is clear that the journals most important to the international research community will publish full text in English” [29].

This declaration is somewhat in contrast to the fact that the number of non-English-language journals included increased by 145% from 2007 to 2010 (after having been stagnating during the previous six years, cf. table 1). Also, the proportion of non-English-language journals to English-language journals increased, though not significantly. It seems that Thomson Reuters has found it necessary to moderate their inclusion criteria, given the scale of criticism of the (previous) “exclusive” focus. There may also be a financial incentive to include journals, regardless of international orientation. It should however be mentioned that the increase in the number of non-English-language journals included may also reflect the fact that non-English-language journals do play an important role in the scientific community – and that scientific work published in non-English languages does have an impact.

Of course, non-English-language journals with high local circulations and established readerships are of great utility in disseminating knowledge of national/local importance. But publishing in a non-English language simply does not make sense if the ambition is to reach an international readership. Non-English-language journals are underrepresented in bibliographical indexing databases (such as MEDLINE, Embase etc.) and this means that articles published in these non-indexed journals are not accessible to (or citable for) a wider audience [16, 23].

Apart from the fact that scientific work published in English is cited more often, authors prefer to submit to high-impact and thus English-language journals [30]. Whether or not IF is accepted as a valid measure of a journal’s scientific quality, one must assume that articles published in these high-impact journals are generally associated with higher quality.

For the reasons given above, non-English-language journals desirous of contributing to the international pool of knowledge should consider changing publication language, adopting either a bi- or a monolingual approach. Publishing in English (possibly in addition to the national language) will increase journal visibility, expand readership and thus increase the potential number of citations.

The potential increase in IFs (or potential inclusion in the JCR) might have been a contributing factor for national journals that have already changed publication language into English [15, 31–33]. However, increasing IFs should not be the sole argument for changing publication language. From an ethical perspective, it could be argued that original research findings should not be published in small, local/national languages and thus “hidden” from the international scientific community [33]. Instead, authors and editors should seek to communicate original research findings to as many readers as possible.

In conclusion, this study demonstrated that non-English-language general medical journals had significantly lower IFs than English-language general medical journals. In an international scientific community with English as the universal language of science, non-English-language journals should consider changing publication language, adopting either a bi- or a monolingual approach. Publishing in English will increase citation counts and thus IFs, but, more importantly, scientific findings will be rendered accessible to a much wider audience.

References

  1 Reuters T. Journal Citation Reports. http://wokinfo.com/media/pdf/jcrwebfs.pdf 2010.

  2 Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90–3.

  3 Neff BD, Olden JD. Not so fast: inflation in impact factors contributes to apparent improvements in journal quality. Bioscience. 2010;60:455–9.

  4 Stock WG. The inflation of impact factors of scientific journals. Chemphyschem. 2009;10:2193–6.

  5 Althouse BM, West JD, Bergstrom CT, Bergstrom T. Differences in impact factor across fields and over time. J Am Soc Inf Sci Technol. 2009;60:27–34.

  6 Chen M, Zhao MH, Kallenberg CG. The impact factor of rheumatology journals: an analysis of 2008 and the recent 10 years. Rheumatol Int. 2011;31:1611–5.

  7 Chew M, Villanueva EV, Van Der Weyden MB. Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994–2005) and their Editors’ views. J R Soc Med. 2007;100:142–50.

  8 Falagas ME, Kouranos VD, Michalopoulos A, Rodopoulou SP, Batsiou MA, Karageorgopoulos DE. Comparison of the distribution of citations received by articles published in high, moderate, and low impact factor journals in clinical medicine. Intern Med J. 2010;40:587–91.

  9 Karageorgopoulos DE, Lamnatou V, Sardi TA, Gkegkes ID, Falagas ME. Temporal trends in the impact factor of European versus USA biomedical journals. PLoS One. 2011;6:e16300.

10 Lee C-H, Shih C-P, Chang Y-C, Chaou C-H. The evolution of academic performance in emergency medicine journals: Viewpoint from 2000 to 2009 Journal Citation Reports. Acad Emerg Med. 2011;18:898–904.

11 van Leeuwen TN, Moed HF, Tijssen RJW, Visser MS, van Raan AFJ. First evidence of serious language-bias in the use of citation analysis for the evaluation of national science systems. Research Evaluation. 2000;9:155–6.

12 van Leeuwen TN, Moed HF, Tijssen RJW, Visser MS, van Raan AFJ. Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics. 2001;51:335–46.

13 Lopez-Abente G, Munoz-Tinoco C. Time trends in the impact factor of public health journals. BMC Public Health. 2005;5:24

14 Smith DR. Citation analysis and impact factor trends of 5 core journals in occupational medicine, 1985–2006. Arch of Environmental Occupat Health. 2008;63:114–22.

15 Mueller PS, Murali NS, Cha SS, Erwin PJ, Ghosh AK. The association between impact factors and language of general internal medicine journals. Swiss Med Weekly. 2006;136:441–3.

16 Poomkottayil D, Bornstein MM, Sendi P. Lost in translation: the impact of publication language on citation frequency in the scientific dental literature. Swiss Med Weekly. 2011;141:w13148

17 Currie GM, Wheat JM. Impact factors in nuclear medicine journals. J Nucl Med 2007;48:1397–400.

18 Andersen J, Belmont J, Cho CT. Journal impact factor in the era of expanding literature. J Microbiol Immunol Infect. 2006;39:436–43.

19 Garfield E. The impact factor and its correct usage. Anaesthesist. 1998;47:439–40.

20 Kumar V, Upadhyay S, Medhi B. Impact of the impact factor in biomedical research: its use and misuse. Singapore Med J. 2009;50:752–5.

21 Andrade A, Gonzalez-Jonte R, Miguel Campanario J. Journals that increase their impact factor at least fourfold in a few years: The role of journal self-citations. Scientometrics. 2009;80:515–28.

22 DeLisa JA. Evaluating journal quality – the role of citation data and the impact factor. J Spinal Cord Med. 2004;27:1.

23 Falagas ME, Kouranos VD, Michalopoulos A, Rodopoulou SP, Batsiou MA, Karageorgopoulos DE. Comparison of the distribution of citations received by articles published in high, moderate, and low impact factor journals in clinical medicine. Int Med J. 2010;40:587–91.

24 van Leeuwen TN, Moed HF. Characteristics of Journal Impact Factors: The effects of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics. 2005;63:357–71.

25 Falagas ME, Alexiou VG. The top-ten in journal impact factor manipulation. Arch Immunol Ther Exp 2008;56:223–6.

26 Ha TC, Tan SB, Soo KC. The journal impact factor: Too much of an impact? Ann Acad Med Singapore. 2006;35:911–6.

27 Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498–502.

28 Joseph KS. Quality of impact factors of general medical journals. BMJ. 2003;326:283–3.

29 Reuters T. The Thomson Reuters journal selection process. http://thomsonreuters.com/products_services/science/free/essays/journal_selection_process/.

30 Kaltenborn KF, Kuhn K. The journal impact factor as a parameter for the evaluation of researchers and research. Med Klin (Munich). 2003;15:153–69

31 Schaffner A. Journal impact factors depend on more than just publication language. Swiss Med Weekly. 2006;136:411–2.

32 Smith DR, Rivett DA. Bibliometrics, impact factors and manual therapy: Balancing the science and the art. Man Ther. 2009;14:456–9.

33 Rosenberg J. Danish versus English in Ugeskrift for Laeger. Ugeskr Laeger. 2009;171:1911.

Notes

Funding / potential competing interests: No financial support and no other potential conflict of interest relevant to this article was reported.