Responsible assessment of what research? Beware of epistemic diversity!

Abstract

Schönbrodt et al. (2022) and Gärtner et al. (2022) aim to outline in the target articles why and how research assessment could be improved in psychological science in accordance with DORA, resulting in a focus on abandoning the impact factor as an indicator for research quality and aligning assessment with methodological rigor and open science practices. However, I argue that their attempt is guided by a rather narrow statistical and quantitative understanding of knowledge production in psychological science. Consequently, the authors neglect the epistemic diversity within psychological science, leading to the potential danger of committing epistemic injustice. Hence, the criteria they introduce for research assessment might be appropriate for some approaches to knowledge production; it could, however, neglect or systematically disadvantage others. Furthermore, I claim that the authors lack some epistemic (intellectual) humility about their proposal. Further information is required regarding when and for which approaches their proposal is appropriate and, maybe even more importantly, when and where it is not. Similarly, a lot of the proposed improvements of the reform movement, like the one introduced in the target articles, are probably nothing more than trial and error due to a lack of investigation of their epistemic usefulness and understanding of underlying mechanisms and theories. Finally, I argue that with more awareness about epistemic diversity in psychological science in combination with more epistemic (intellectual) humility, the danger of epistemic injustice could be attenuated.