Skip to main navigation menu Skip to main content Skip to site footer
Bitácora Académica

Artículos

Vol. 8 No. 1 (2021): Use and misuse of the journal impact factor for evaluating researchers

Use and misuse of the journal impact factor for evaluating researchers

DOI
https://doi.org/10.18272/ba.v8i1.3324
Submitted
May 16, 2024
Published
2021-04-12

Abstract

While the journal impact factor was originally developed to help librarians decide which journals to subscribe to, it has been increasingly used for the evaluation of the performance of individual researchers. The reasons why this practice should never be encouraged are reported in this study based on a literature review. This manuscript presents a critical overview on the international use, by governments and institutions, of the journal impact factor and/or journal indexing information for the assessment of individual researchers. The criticism on this practice is also illustrated through comments provided by Nobel laureates and others. This paper then proposes a rational method for evaluating researchers. As such, this work proposes an improved context for the use of the journal impact factor, so that this metric can be used for the right purpose.

viewed = 27 times

References

  1. Garfield, E., The history and meaning of the journal impact factor. JAMA, 2006. 295(1): p. 90-93
  2. Small, H., Citation Indexing Revisited: Garfield’s Early Vision and Its Implications for the Future. Frontiers in Research Metrics and Analytics, 2018. 3(8)
  3. Clarivate Analytics. Journal Citation Reports. 2018 [cited 2018 September 26th 2018]; Available from: http://ipsciencehelp
  4. thomsonreuters.com/inCites2Live/indicatorsGroup/aboutHandbook/usingCitationIndicatorsWisely/jif.html
  5. Eston, R., Editorial: The Impact Factor: a misleading and flawed measure of research quality. Journal of Sports Sciences, 2005. 23(1): p. 1-3
  6. Quan, W., B. Chen, and F. Shu, Publish or impoverish: An investigation of the monetary reward system of science in China (1999-2016). Aslib Journal of Information Management, 2017. 69(5): p. 486-502
  7. The Analogue University, Calling all journal editors: Bury the metrics pages! Political Geography, 2019. 68: p. A3-A5
  8. Kaltenborn, K.-F. and K. Kuhn, The journal impact factor as a parameter for the evaluation of researchers and research. Revista Española de Enfermedades Digestivas, 2004. 96: p. 460-476
  9. Authors, Research Counts, Not the Journal. OSF Preprints, 2018. Abbasi, K., Let's dump impact factors. BMJ, 2004. 329(7471)
  10. McKiernan, E.C., et al., Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. eLife, 2019. 8: p. e47338
  11. Pudovkin, A.I., Comments on the Use of the Journal Impact Factor for Assessing the Research Contributions of Individual Authors. Frontiers in Research Metrics and Analytics, 2018. 3(2)
  12. Smeyers, P. and N.C. Burbules, How to Improve your Impact Factor: Questioning the Quantification of Academic Quality. Journal of Philosophy of Education, 2011. 45(1): p. 1-17
  13. University of Lisbon (UL), Reitoria – Despacho n.º 3855/2017 (in Portuguese). Diário da República, 2017. 88(2.ª série — 8 de maio de 2017): p. 8628-29
  14. University of Lisbon (UL), Edital 655/2017 (in Portuguese). Diário da República, 2017. 173(2.ª série — 7 de setembro de 2017): p. 19764-7
  15. University of Lisbon (UL), Edital 688/2017 (in Portuguese). Diário da República, 2017. 178(2.ª série — 14 de setembro de 2017): p. 20201-3
  16. University of Lisbon (UL), Edital 662/2017 (in Portuguese). Diário da República, 2017. 174(2.ª série — 8 de setembro de 2017): p. 19856-8
  17. Ostravská univerzita v Ostravě. Open position for a postdoctoral fellow. 2018 [cited 2018 September 19, 2018]; Available from:
  18. http://www1.osu.cz/~elias/open_postdoc_position.html
  19. Muthamilarasan, M. and M. Prasad, Impact of impact factor in quantifying the quality of scientific research. Current Science, 2014. 107(8): p. 1233-1234
  20. Ferreira, R., F. Antoneli, and M. Briones, The hidden factors in impact factors: a perspective from Brazilian science. Frontiers in Genetics, 2013. 4(130)
  21. de Meis, L., M.S. do Carmo, and C. de Meis, Impact factors: just part of a research treadmill. Nature, 2003. 424: p. 723
  22. UMET Universidad Metropolitana. Senescyt establece lineamientos para categorización de investigadores (in Spanish). 2018 [cited 2018 October 12th, 2018]; Available from: http://www.umet.edu.ec/senescyt-establece-los-lineamientos-para-la-categorizacion-yrecategorizacion-de-investigadores/
  23. Universidad Tecnica Particular de Loja. https://procuraduria.utpl.edu.ec/sitios/documentos/NormativasPublicas/Reglamento%20de%20Concurso%20de%20M%C3%A9ritos%20y%20Oposici%C3%B3n.pdf. 2017 [cited 2019 November 11, 2019]; Available from: https://procuraduria.utpl.edu.ec/sitios/documentos/NormativasPublicas/Reglamento%20de%20Concurso%20de%20M%C3%A9ritos%20y%20Oposici%C3%B3n.pdf
  24. Universidad y Sociedad: Foro Ecuador. El model de evaluacion de las unversidades Ecuatorianas - Apuntes criticos para el debate. 2014 [cited 2018 October 12, 2018]; Available from: https://universidadsociedadec.wordpress.com/manifiesto-modelo-deevaluacion-
  25. ceaaces/
  26. Moed, H.F., Eugene Garfield’s Influences upon the Future of Evaluative Informetrics. Frontiers in Research Metrics and Analytics, 2018. 3(5)
  27. Ranjan, C.K., Bibliometric Indices of Scientific Journals: Time to overcome the obsession and think beyond the Impact Factor. Medical Journal Armed Forces India, 2017. 73(2): p. 175-177
  28. Robert, R. and S. Gnanavel, Impact factor: The holy grail of research. Indian Journal of Psychological Medicine, 2015. 37(2): p. 248-249
  29. Vinkler, P., Introducing the Current Contribution Index for characterizing the recent, relevant impact of journals. Scientometrics, 2009. 79(2): p. 409-420
  30. Seglen, P.O., Why the impact factor of journals should not be used for evaluating research. BMJ, 1997. 314(7079): p. 497
  31. Seglen, P.O., How representative is the journal impact factor? Research Evaluation, 1992. 2(3): p. 143-149
  32. Anonymous, Coping with peer rejection. Nature, 2003. 425: p. 645
  33. North, G., Q & A. Current Biology, 2004. 14(4): p. R143-R144
  34. Paine, C.E.T. and C.W. Fox, The effectiveness of journals as arbiters of scientific impact. Ecology and Evolution, 2018. 8(19): p. 9566-9585
  35. Brembs, B., Prestigious Science Journals Struggle to Reach Even Average Reliability. Frontiers in Human Neuroscience, 2018. 12(37).
  36. Fraley, R.C. and S. Vazire, The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power. PLOS ONE, 2014. 9(10): p. e109019
  37. Stöckle, U. and D.C. Wirtz, Journal Impact-Faktor: zur wissenschaftlichen Qualitätsbewertung ungeeignet. Z Orthop Unfall, 2014. 152(06): p. 553-553
  38. American Society for Cell Biology. The San Francisco Declaration on Research Assessment (DORA). 2018 [cited 2018 September 13th 2018]; Available from: https://sfdora.org/
  39. Curry, S., Let’s move beyond the rhetoric: it’s time to change how we judge research. Nature, 2018. 554(February): p. 147
  40. Hallberg, L., Can the impact factor measure the quality of research? International Journal of Qualitative Studies on Health and Well-being, 2012. 7(1): p. 19772
  41. Campbell, P., Escape from the impact factor. Ethics in Science and Environmental Politics, 2008. 8(1): p. 5-7
  42. Tregoning, J., How will you judge me if not by impact factor? Nature, 2018. 558(June): p. 345
  43. van Noorden, R., Metrics: A profusion of measures. nature, 2010. 465(June): p. 864-866
  44. Glänzel, W., et al., Springer Handbook of Science and Technology Indicators. 2018, Cham, Switzerland: Springer International Publishing
  45. Sample, I. Nobel winner declares boycott of top science journals 2013 [cited 2018 September 20th 2018]; Available from: https://www.theguardian.com/science/2013/dec/09/nobel-winner-boycott-sciencejournals.
  46. Nobel Media AB. Impact factors are ‘a substitute for judgement’ – Nobel Laureate Michael Brown (YouTube video). 2011 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=PKWJ5gmTuAw&t=0s&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F&index=6
  47. Nobel Media AB. Michael S. Brown – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/medicine/1985/brown/facts
  48. Nobel Media AB. Impact factors? What counts is the data – Nobel Laureate Joseph Goldstein (YouTube video). 2014 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=6MffmKuvBHs&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F&index=4
  49. Nobel Media AB. Joseph L. Goldstein – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/medicine/1985/goldstein/facts
  50. Nobel Media AB. Nobel Laureate Peter Doherty: Impact factors are skewing science (YouTube video). 2012 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=Nhqv3DeDriE&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F&index=7
  51. Nobel Media AB. Peter C. Doherty – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/medicine/1996/doherty/facts
  52. Nobel Media AB. Sir Paul M. Nurse – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/medicine/2001/nurse/facts
  53. Nobel Media AB. There's a lot of rubbish published in high-profile journals’ Paul Nurse, Nobel Laureate (YouTube video). 2014 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=YETmLPgeGT8&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F&index=6
  54. Nobel Media AB. ’I can categorically say I hate impact factors!’ Nobel Laureate Martin Chalfie (YouTube video). 2014 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=sCAsAKgNPjs&index=8&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F
  55. Nobel Media AB. Martin Chalfie – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/chemistry/2008/chalfie/facts.
  56. Nobel Media AB. Bruce A. Beutler – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/medicine/2011/beutler/facts/.
  57. Nobel Media AB. The impact factor really isn't that important’ Bruce Beutler, Nobel Laureate (YouTube video). 2015 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=2Ep5YYYSi5Q&index=3&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F
  58. Nobel Media AB. Brian K. Kobilka – Facts 2018 [cited 2018 September 17, 2018]; Available from: https://www.nobelprize.org/prizes/chemistry/2012/kobilka/facts/.
  59. Nobel Media AB. ’I am very troubled by the importance of impact factors’ Brian Kobilka, Nobel Laureate (Youtube video). 2016 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=eOd20-lFCaE&index=2&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F
  60. Nobel Media AB. Impact factors are taking us in the wrong direction - Brian Kobilka, Nobel Laureate (Youtube video). 2016 [cited 2018 September 21, 2018]; Available from: https://www.youtube.com/watch?v=TRyLZK_yq0I&index=1&list=PLnOZUvRwNnkyrHf5Es8R5-je_YubRdD0F
  61. Fassin, Y., A New Qualitative Rating System for Scientific Publications and a Fame Index for Academics. Journal of the Association for Information Science, 2018. 69(11): p. 1396-1399
  62. Traag, V.A. and L. Waltman, Systematic analysis of agreement between metrics and peer review in the UK. arXiv.org, 2018
  63. Aksnes, D.W. and R.E. Taxt, Peer reviews and bibliometric indicators: a comparative study at a Norwegian university. Research Evaluation, 2004. 13(1): p. 33-41
  64. Frontiers. AI-enhanced peer review: Frontiers launches next generation of efficient, highquality peer review. 2018 Dec 19th 2018]; Available from: https://blog.frontiersin.org/2018/12/14/artificial-intelligence-peer-review-assistant-aira/
  65. Van Diest, P.J., et al., Impactitis: new cures for an old disease. Journal of Clinical Pathology, 2001. 54(11): p. 817-819
  66. Norris, M. and C. Oppenheim, Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 2003. 59(6): p. 709-730.
  67. Oppenheim, C., Do citations count? Citation indexing and the Research Assessment Exercise (RAE). Serials, 1996. 9(2): p. 155-161
  68. Lantsoght, E.O.L., et al., Interviewing and Hiring Practices in Brazilian Academia: Proposals Towards Improvement. Societies, 2019. 9(3): p. 57
  69. Bornmann, L. and H.-D. Daniel, Does the h-index for ranking of scientists really work? Scientometrics, 2005. 65(3): p. 391-392
  70. Smith, D.R., “Platinum H”: Refining the H-Index to More Realistically Assess Career Trajectory and Scientific Publications. Archives of Environmental & Occupational Health, 2015. 70(2): p. 67-69