Scrambling for Higher Metrics in the Journal Impact Factor Bubble Period: A Real-World Problem in Science Management and its Implications

Universities and funders in many countries have been using Journal Impact Factor (JIF) as an indicator for research and grant assessment despite its controversial nature as a statistical representation of scientific quality. This study investigates how the changes of JIF over the years can affect its role in research evaluation and science management by using JIF data from annual Journal Citation Reports (JCR) to illustrate the changes. The descriptive statistics find out an increase in the median JIF for the top 50 journals in the JCR, from 29.300 in 2017 to 33.162 in 2019. Moreover, on average, elite journal families have up to 27 journals in the top 50. In the group of journals with a JIF of lower than 1, the proportion has shrunk by 14.53% in the 2015–2019 period. The findings suggest a potential ‘JIF bubble period’ that science policymaker, university, public fund managers, and other stakeholders should pay more attention to JIF as a criterion for quality assessment to ensure more efficient science management.


INTRODUCTION
Over the past few years, academia has debated about the use (and misuse) of journal metrics in evaluating scientific quality with many criticisms directed toward the most widely used Journal Impact Factor (JIF) provided by Web of Science. In the beginning, citation index was conceived by Eugene Garfield as a tool to help the academic community track notes and references to earlier works (Garfield, 1955); and only later that he suggested "frequency and impact of citations" can potentially be used to assess journals as well (Garfield, 1972). Science Citation Index (SCI) and Journal Impact Factor (JIF) had gradually become useful tools for librarians, editors, and policymakers to evaluate important journals in the fields and identify a potential rise of a research topic (McKiernan, Schimanski, Muñoz Nieves, Matthias, Niles, & Alperin, 2019). However, managers at universities, research institutions, and science funding agencies have been using JIF as an important criterion for evaluation in many aspects including research quality (Moustafa, 2015), career promotion and grant application (McKiernan et al., 2019), prediction of scientific impact (Berenbaum, 2019), or even distribution of funding (Moustafa, 2015). As a conse-

LITERATURE REVIEW
Since the concept of JIF was for sorting journals in the library, the metrics have many shortcomings that make it insufficient in reflecting the quality of a journal or a researcher. As a statistical measure, changes in the field, time of citation count, journal's type and size affected the JIF significantly, thus, making an out-of-context usage impossible and misleading (Amin & Mabe, 2000). Moreover, the editors of a journal can manipulate JIF by asking under review manuscript to cite papers from the journal, or publishing a review of the previous articles in the journal frequently (Arnold & Fowler, 2011). Recently, Larivière and Sugimoto (2019) summed up most of the JIF flaws in their chapter. Firstly, the citation count for news, editorials, obituaries, articles is inflating the citation of many journals.
Moreover, the standard two-year period seems to be arbitrary, thus, putting at a disadvantage the fields that require a long time to accumulate citation. Generally, differences among fields make the comparison harder. The significant effect of a few highly cited papers also muddles the citation count. Finally, the drastic rise of impact factor in recent years appears to be inflation.
Scientists, university leaders, and policymakers have been fighting the usage of JIF as an indicator of quality, and the Declaration on Research Assessments or DORA (https://sfdora.org/) is a notable initiative. DORA emphasizes on the need to stop using JIF as a proxy for quality, to change the current research evaluation methods (DORA). The plan to achieve these goals includes the development and promotion of alternative methods, new tools, and the process of research evaluation and extending the impact of DORA (June 27, 2017).
However, Tregoning (2018) asks an important question that remains unanswered: If not JIF, then what Tregoning views JIF as a quick, immediate, and easy-to-understand method to grasp the essence of a person's work, especially for early-career researchers. In a career that celebrates longevity and seniority, using the number of publications, citations, or h-index, which can only accumulate in time, is rather unfair to young researchers. A recent study in bioRxiv also suggests that tenured and senior academics value journal prestige and metrics less than their younger and untenured counterparts (Niles, Schimanski, McKiernan, & Alperin, 2019). It is hard to ignore the fact that even though higher ranking might not mean quality, it does bring the reputation to the journal and attract readers (Langin, 2019).
Moreover, the introduction of journal ranking and JIF in the research evaluation did help to improve the overall productivity of scientists (Bornmann, 2011;Götz, 2019). Thus, the academic community still assesses the quality of an individual paper or a scientist based on the impact factor. Firstly, as Tregoning (2018) suggests, besides the number of citations, JIF is still one of the most familiar qualitative indicators that can show a journal's achievement. Moreover, comparing to article-level citation counts, JIF attracts users because they are quicker to obtain. Finally, JIF has established a clear indicator of the invisible hierarchy of academic journals (Bordons, Fernández, & Gómez, 2002).
These advantages lead to JIF being widely used in evaluating research quality (Moustafa, 2015), ca-reer promotion, and grant application (McKiernan et al., 2019), prediction of scientific impact (Berenbaum, 2019), or even distribution of funding (Moustafa, 2015). For instance, a paper studying over 860 review, promotion, and tenure documents from universities in the United States and Canada finds that 40% of doctoral, research-oriented institutions include JIF in their documents (McKiernan et al., 2019). Moreover, the study also suggests 60% of the institutions equated JIF with quality, while 40% mentioned it with impact, and 20% suggested reputation and prestige in close relation with JIF (McKiernan et al., 2019). In Spain, JIF and citations are important criteria to review a scientist's performance because they provide objective indicators to help the board of experts (Bordons, Fernández, & Gómez, 2002). Being tangible and measurable in a short time also makes impact factor an important research output indicator, along with citation in policy documents, debates, media (Mabiso, Rheenen, & Ferguson, 2013). In some countries, university managers and science funders are using JIF as a basis for providing a cash bonus. In China, a paper published in top journals such as Cell, Science, or Nature was paid an RMB 500,000 cash bonus (Nature Editoral, 2017). Similarly, in Vietnam, an ISI/Scopus article can be worth up to USD 2,000, while a publication in a journal that has a JIF higher than two can earn the author a sum of USD 8,600 (Vuong, 2019b).
Even though this practice can exacerbate fieldbased inequality and, by extension, other structural discrimination, skew the perception of success in academia and complicate science management, JIF continues to be one of the decisive elements in research careers. Brown (2007) suggests medical schools in the UK have lost many faculty members to impact factor because clinical researchers cannot compete with their laboratory-based counterparts in terms of journal ranking. Meanwhile, in Japan, domestic researchers find it harder to cooperate internationally if the outcomes are in low impact journals (Shibayama & Baba, 2015). Osterloh and Frey (2020) argue that most of the authors who got their papers accepted eventually benefit from the JIF, which leads to more effort in keeping the JIF.
In this article, based on a comparative view of changes in JIFs over the past five years, the authors will discuss how this technical aspect of JIF will affect the way universities and science funding agencies use it as a tool for science management.

METHODS
The subsequent descriptive analysis employs two types of data: a) JIFs provided by yearly Journal Citations Reports (JCR) from Clarivate Analytics; b) Counts of journals in the predefined JIF ranges.
The first type of data is readily understandable as each journal that has an Impact Factor is given the figure in the JCR. For instance, the 2019 JCR shows that the top-tiered general Science Magazine (https://sciencemag.org) has its 2018 IF standing at 41.037.
In theory, JIF is computed by dividing the number of citations to the articles that a journal has in a year by the number of total articles that the journal has in the two preceding years. For instance, the 2019 JIF of a journal can be computed as follows (Garfield, 1994 The second type is a little less straightforward. Data of this type are count data from some predefined ranges, which one would like to observe the "behaviors" of the corresponding data. For instance, if one wishes to know whether it is true that very few journals can attain a two-digit JIF, a JIF range with a starting value of 10. In principle, one can choose arbitrary intervals of JIF. But in practice, only certain intervals are meaningful for our audiences.
The process of collecting these data involves scanning both paper-based and pdf reports, cleaning up duplicates, and correcting for easy-to-misunderstand abbreviations of journal titles. These tasks have been performed with the help of our home-grown AI tools for detecting probable du-plicates and recognizing/suggesting titles using fuzzy strings.
The clean data were then saved into the CSV format, and SQL Server 2016 (Microsoft®, Seattle, WA, USA) was used to perform descriptive statistics. An example of the SQL code is shown on the

RESULTS
The authors start with Table 1, providing lists of top 50 among those 'elite journals' over the recent three years, using data from JCR 2017-2019. It is noteworthy that those most famous journals such as Nature (highlight in yellow) and Science (highlight in green) do not have the highest JIF, nor do their JIF always increase over time. The Editorin-Chief of Proceedings of the National Academy of Sciences (PNAS) -May R. Berenbaum -noted this interesting phenomenon when she moved from the Annual Review of Entomology to PNAS in 2017 (Berenbaum, 2019). Even though PNAS is considered more prestigious, its actual JIF (2017 JIF = 9.661) is lower than the Annual Review of Entomology (2017 JIF = 12.867). In Table 1, it is notable that PNAS is not in the top 50.
It is also clear that an increase in JIF for a particular journal does not guarantee their higher position because some other journals may show bigger jumps. Nonetheless, all the journals in this group have their JIF of higher than 20, with CA-Cancer J Clin being an exception. Next, Figure 2 gives a feel of how journals are distributed against some major JIF ranges (indicated by the legends inside the chart), using JCR 2019 data. It is not a surprise that the journals with a JIF of 10 or higher constitute the smallest group among all groups (2.19% of the JCR 2019 population). The next group (5 ≤ JIF < 10) accounts for a little less than 6% of the population. That being said, all the journals with a JIF of 5 or higher account for just 8.17% in JCR 2019. Journals, which have a JIF of lower than 2, account for a staggering majority of 57% all journals. Figure 2 uses only 6 JIF ranges for better visualization of the data. However, since we are also interested in the equal intervals (except for the highest, i.e., JIF ≥ 10), Table 2 provides such breakdowns for the recent five-year data. For instance, the authors read the line 5+, which counts the number of journals with a JIF of 5 or higher, together with the corresponding proportion (against the total number of journals present in a specific year of JCR). The number of journals increases from 617 (2015) to 969 (2019). Their proportion also increases from 5.59% (2015) to 8.17% (2019).  It seems clear that the proportion of journals with a JIF of lower than 1.0 drops from 40.16% (2015) to 25.63% (2019). Therefore, one may suspect that we are living in a "JIF bubble period." While this type of inflation may look pleasing to the authors, for now, it would potentially lead to a decrease in the value of JIF in the long run.
From another analytical angle, one may wish to learn the kind of JIF threshold for certain por-tions of journals, using the percentage of highest JIF journals. Table 3 is presented for that purpose.
In Table 3, the lowest JIF for each group is listed as a kind of threshold. Let us take a look at two groups, the top 1% and 10% journals in the 2015-2019 period. The threshold increases from 13.555 to 15.548 for the top 1%, and from 3.775 to 4.524 for the top 10%. All other groups experience certain degrees of increase, too.   The increasing thresholds across all groups of journals also provide a bit more support to the "JIF bubble" suspicion.

DISCUSSION
The analysis suggests a potential "JIF bubble period," to which stakeholders in the science community should pay attention. The median JIF for the top 50 journals has increased from 29.300 in 2017 to 33.162 in 2019. The presence of elite journal families is also notable, as Nature family has 20 journals in the top 50 highest JIF journals on average. Considering some major JIF ranges, journals with a JIF of 5 or higher occupy only 8.17% in 2019 JCR, while those with a JIF of lower than two account for about 57% of all. There is a significant drop in the proportion of journals with a JIF of lower than one from 40.16% in 2015 to only 25.63% in 2019. Moreover, the threshold for identifying the top percentage of journals also increases over the year.
It should also be noted that since November 2014, the Emerging Sources Citation Index (ESCI) has become part of the WOS core collection, focusing on growing journals. With more than 8,000 ESCI journals having been included for impact factor calculation, the "JIF bubble period" could also be a product of this expansion. However, unlike a bubble in the financial market, which is that eventually causing a market crash (Sornette & Cauwels, 2015), the JIF bubble period is more likely to continue.
Currently, as the academic world is under the pressure of 'publish or perish' (Editoral, 2015;Vuong, 2019a), this 'JIF bubble period' seems to benefit the elite group, and those with a JIF of 4.52 or higher, due largely to the supply-demand imbalances. There are two major implications, which science policymakers and publishers/editors will have little choice but to ponder heavily. First, this type of "Matthew's law" in attaining higher JIF and generating a higher demand will not end in any foreseeable horizon. Our findings have suggested an increase from 617 to 969 journals in the 2015-2019 period for journals with a JIF of 5 or higher, and this trend also happens in other JIF ranges as well. It looks like the "JIF bubble period" will further expand. When the authors would put more effort into keeping the high JIF because they find it helpful in advancing their career in the organization (Osterloh & Frey, 2020), they enjoy the "JIF bubble period." As the financial bubble is often "driven by sentiment and no longer reflects any real underlying value" (Sornette & Cauwels, 2015), JIF, which has been controversial since the beginning, would not be less controversial in the current "JIF bubble period." Therefore, reliance on JIF to promote or award science in China or Vietnam universities is gradually becoming an unstable method (Nature Editoral, 2017; Vuong, 2019b), which could adversely affect the ethical management of scientific funds. Universities and research institutions should use different methods and metrics to evaluate science. Second, the heated race will render many more debates, and criticisms on abuse of JIF, only to lack in both relevance and impact as voices from journal representatives mostly come from those who have been enjoying the privilege of being already in the elite group. These paradoxes are unsolvable and will remain unsolvable for quite some time. Sumpter (2019) shares a similar concern since most of the current metrics such as citation, JIF, or h-index -following Hirsch (2005) -are in favor of senior scientists. While evaluating, the early-career researchers will have to wait for more research data to come. Rather than criticizing the JIF, the senior scientists, especially those who are in managerial positions, should focus on creating a fairer guideline and policy, or finding an alternative method of evaluation. Otherwise, just as the elite families of scientific journals have consistently presented in the list of journals with high JIF (see Table 1), those who benefit from JIF will continue to enjoy the comfort, while others struggle to climb the rank -an act that would only further fortify the abuse of this one metric as a criterion for academic prestige.

CONCLUSION
The article has analyzed the yearly JIFs from Clarivate Analytics' Journal Citations Reports and counts of journals in certain JIF ranges. Notably, the results have shown signs of a "JIF bubble period," which can be found in the rise of the median JIF from 2017 to 2019, or the increase of the JIF limit for categorizing the top percentage of journals.
The situation should not be taken lightly by university governing body and science policymaker since science will, in the long run, no longer be the place for the type of "soul-touching research" that humanity has been longing for (Trinh et al, 2019). The academic scene will instead be dominated by the stonecold performance metrics, of which JIF can be the single most intimidating representative (Neuberger, & Counsell, 2002). Policymakers in science management would benefit from more sensible considerations regarding the disproportionate use of a singular metric -the JIF -in the multi-faceted task of evaluating the research careers and scientific credentials (Snoek, 2019).