Should experts guide research merit assessment? Discrepancies between Journal Impact Factor and national expert-based journal ranking list for evaluation of scientific achievement in Poland – a bibliometric analysis

Abstract

ABSTRACTObjectivesTo assess discrepancies between Journal Citation Reports (JCR) Journal Impact Factor (JIF) and expert-assigned journal ranks used to evaluate achievements of medical scientists in Poland.Design and settingA bibliometric analysis based on data obtained from JCR and expert journal ranking from the Polish Ministry of Education and Science.Main outcome measuresWe provided descriptive analysis of all journals listed in JCR Clinical Medicine group and their respective ranks (20, 40, 70, 100, 140 or 200 points) assigned by experts in the category Medical Sciences. For each of 59 JCR Clinical Medicine categories we ranked journals from the highest JIF to the lowest (JCR category ranking) and correlated these data with points given by experts. Additionally, we analysed 4352 journals not listed in JCR but assigned expert ranks. Data collection occurred from August to September 2023.ResultsWe extracted data on 7441 journals listed in JCR (5908 after deduplication), of which 5315 were scored by experts. Across all 6 ranks the minimum JIF was comparable and did not exceed 0.2, variances were large and outliers with JIF of 20 and above were prevalent. In 12 out of 59 JCR categories no journal was assigned 200 points. The correlation between JCR category ranking and expert ranking varied with Spearman’s r between -0.18 for Medical Informatics and -0.93 for Neuroimaging. In less than half of categories (19/59) the correlation coefficient was -0,7 or stronger. Some journals assigned 200 points in the Medical Sciences category were not related to medicine or were Polish journals with low JIF.ConclusionsIn Poland, the system for assessing scientific merit in medicine lacks transparency, may encourage publishing in low-quality journals and discriminate top scientists in undervalued fields. A comprehensive review of the system is necessary to promote transparent, methodologically sound, and clinically relevant research.What is already known on this topicAssessment of scientific merit is challenging and often based on the quality of journals, in which studies are published. Metrics like Journal Impact Factor (JIF) are surrogates of journal quality but in some countries the quality of journals is ranked by experts. Whether expert-based assessment systems promote high quality research, remains unknown.What this study addsJournals with JIF lower than 1 were found in all expert-assigned ranking groups and 10% of journals from Clinical Medicine JCR group were not considered in Medical Sciences Ministerial category by experts.Journal ranking within a study field is often not reflected by ranks assigned by experts. Scientists in every fifth JCR Clinical Medicine category cannot achieve 200 points (top expert rank) for their work despite publishing in the most prestigious journals.How this study might affect research, practice, or policyCurrent assessment system may promote publishing low-quality studies in high-rank, low-JIF journals and discourage Scientists from engaging in solving more complex but likely more relevant clinical issues.We advocate for a change in the assessment system, which should focus on promoting transparent, high-quality research, as well as balance discrepancies between different medical study fields.