Responsible Research Assessment (Parts I and II): Responses to the Commentaries
DOI:
https://doi.org/10.15626/MP.2024.4603Keywords:
Metascience, Incentive, EvaluationAbstract
We give a brief overview of our deliberations in responding to the commentaries on our two target papers presenting the RESQUE (Research Quality Evaluation) framework. While we were able to incorporate many suggestions for improvement directly, we acknowledge that other areas (e.g., quality in theorizing) will require further elaboration. In this paper, we specifically touch on the following issues: (a) eligible types of publication, (b) measurability, (c) quality criteria for software and datasets, (d) theoretical rigor, (e) quantity, (f) authorship, (g) potential bias (against certain methodologies, types of research contributions, or subdisciplines), (h) overall rigor score, (i) weighting of individual indicators, (j) types of data, (k) impact, (l) interdisciplinary value, (m) teaching, (n) expertise, (o) gaming the new metrics, and (p) representativeness. The RESQUE framework has met with largely positive reception so far, but continues to evolve and will thrive best when community involvement stays strong.
Metrics
References
Abele-Brehm, A., & Bühner, M. (2016). Wer soll die Professur bekommen? Eine Untersuchung zur Bewertung von Auswahlkriterien in Berufungsverfahren der Psychologie [Who should receive the professorship? A research on the evaluation of different hiring criteria for appointments in academic psychology]. Psychologische Rundschau, 67(4), 250–261. https://doi.org/10.1026/0033-3042/a000335
Aksnes, D., Langfeldt, L., & Wouters, P. (2019). Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open, 9(1), 2158244019829575. https://doi.org/10.1177/2158244019829575
Auger, V., & Claes, N. (2024). Comment on "Responsible Research Assessment: Implementing DORA for hiring and promotion in psychology”. Meta-Psychology, 8, 2158244019829575. https://doi.org/10.15626/MP.2023.3779
Blashfield, R. K., & Reynolds, S. M. (2012). An Invisible College View of the DSM-5 Personality Disorder Classification. Journal of Personality Disorders, 26(6), 821–829. https://doi.org/10.1521/pedi.2012.26.6.821
Boessel-Debbert, N., Kluge, A., Leising, D., Mischkowski, D., Phan, L. V., Richter, F., & Stahl, J. (2025). An analysis of functional relationships between systemic conditions and unethical behavior in german academia [Preprint]. https://doi.org/10.31234/osf.io/xj2m6_v1
Bornmann, L. (2012). The Hawthorne effect in journal peer review. Scientometrics, 91(3), 857–862. https://doi.org/10.1007/s11192-011-0547-y
Brandmaier, A. M., Ernst, M., & Peikert, A. (2024). Commentary: ’Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology’. Meta-Psychology, 8. https://doi.org/doi.org/10.15626/MP.2023.3715
Brandt, H., Henninger, M., Ulitzsch, E., Kleinke, K., & Schäfer, T. (2024). Responsible research assessment in the area of quantitative methods research: A comment on Gärtner et al. MetaPsychology, 8. https://doi.org/10.15626/MP.2023.3796
Brown, G. (2024). A broader view of research contributions: Necessary adjustments to DORA for hiring and promotion in psychology. Meta-Psychology, 8. https://doi.org/10.15626/MP.2022.3652
Campbell, R., Javorka, M., Engleton, J., Fishwick, K., Gregory, K., & Goodman-Williams, R. (2023). Open-Science Guidance for Qualitative Research: An Empirically Validated Approach for De-Identifying Sensitive Narrative Data. Advances in Methods and Practices in Psychological Science, 6(4), 25152459231205832. https://doi.org/10.1177/25152459231205832
Chapman, C. A., Bicca-Marques, J. C., Calvignac-Spencer, S., Fan, P., Fashing, P. J., Gogarten, J., Guo, S., Hemingway, C. A., Leendertz, F., Li, B., Matsuda, I., Hou, R., Serio-Silva, J. C., & Stenseth, N. C. (2019). Games academics play and their consequences: How authorship, h index and journal impact factors are shaping the future of academia. Proceedings of the Royal Society B: Biological Sciences, 286(1916), https://doi.org/20192047.10.1098/rspb.2019.2047
Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135. https://doi.org/10.1017/S0140525X00065675
Dames, H., Musfeld, P., Popov, V., Oberauer, K., & Frischkorn, G. T. (2024). Responsible research assessment should prioritize theory development and testing over ticking open science boxes. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3735
Dougherty, M., & Horne, Z. (2022). Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences. Royal Society Open Science, 9(8). https://doi.org/10.1098/rsos.220334
Etzel, F. T., Seyffert-Müller, A., Schönbrodt, F. D., Kreuzer, L., Gärtner, A., Knischewski, P., & Leising, D. (2025). Inter-rater reliability in assessing the methodological quality of research papers in psychology. https://doi.org/10.31234/osf.io/4w7rb_v2
Fink-Lamotte, J., Hilbert, K., Bentz, D., Blackwell, S., Boehnke, J. R., Burghardt, J., Cludius, B., Ehrenthal, J. C., Elsaesser, M., Haberkamp, A., Hechler, T., Kräplin, A., Paret, C., Schulze, L., Wilker, S., & Niemeyer, H. (2024). Response to responsible research assessment I and II from the perspective of the DGPs working group on open science in clinical psychology. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3794
Fong, E. A., & Wilhite, A. W. (2017). Authorship and citation manipulation in academic research. PLOS One, 12(12), e0187394. https://doi.org/10.1371/journal.pone.0187394
Frischkorn, G. T. (2024). Responsible Research Assessment requires structural more than procedural reforms. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3734
Gärtner, A., Leising, D., Freyer, N., Musfeld, P., Lange, J., & Schönbrodt, F. D. (2022). Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology [Preprint]. https://doi.org/10.31234/osf.io/5yexm_v1
Gärtner, A., Leising, D., & Schönbrodt, F. (2023). Empfehlungen zur Bewertung wissenschaftlicher Leistungen bei Berufungsverfahren in der Psychologie [Recommendations for the evaluation of academic performance for hiring and promotion in psychology]. Psychologische Rundschau, 74(3), 166–174. https://doi.org/10.1026/0033-3042/a000630
Gärtner, A., Leising, D., Freyer, N., Musfeld, P., Lange, J., & Schönbrodt, F. D. (2025). Responsible Research Assessment II: A Specific Proposal for Hiring and Promotion in Psychology. Meta-Psychology, 9. https://doi.org/10.15626/MP.2024.4604
Greenhalgh, T., Raftery, J., Hanney, S., & Glover, M. (2016). Research impact: A narrative review. BMC Medicine, 14, 78. https://doi.org/10.1186/s12916-016-0620-8
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19
Hansen, M., Beitner, J., Horz, H., & Schultze, M. (2024). Indicators for teaching assessment. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3763
Hostler, T. (2024). Research assessment using a narrow definition of “research quality” is an act of gatekeeping: A comment on Gärtner et al. (2022). Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3764
Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84(1), 7120. https://doi.org/10.5688/ajpe7120
Karhulahti, V.-M. (2024). Interdisciplinary value. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3679
Karhulahti, V.-M., Branney, P., Siutila, M., & Syed, M. (2022). A primer for choosing, designing and evaluating registered reports for qualitative methods. MetaArXiv. https://doi.org/10.31222/osf.io/2azkf
Lange, J., Degner, J., Gleibs, I. H., & Jonas, E. (2024). Faires und valides shortlisting in Phase 1 [Fair and valid shortlisting in phase 1]. Psychologische Rundschau, 74(3), 187–189. https://doi.org/10.1026/0033-3042/a000641
Lange, J., Freyer, N., Musfeld, P., Schönbrodt, F., & Leising, D. (2025). A checklist for incentivizing and facilitating good theory building. Zeitschrift für Psychologie, 233. https://doi.org/10.1027/2151-2604/a000604
Leising, D., Grenke, O., & Cramer, M. (2023). Visual Argument Structure Tool (VAST) Version 1.0. Meta-Psychology, 7. https://doi.org/https://doi.org/10.15626/MP.2021.2911
Leising, D., Liesefeld, H., Buecker, S., Glöckner, A., & Lortsch, S. (2024). A tentative roadmap for consensus building processes. Personality Science, 5. https://doi.org/10.1177/27000710241298610
Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. (2022a). Ten steps toward a better personality science – a rejoinder to the comments. Personality Science, 3. https://doi.org/10.5964/ps.7961
Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. (2022b). Ten steps toward a better personality science – how quality may be rewarded more in research evaluation. Personality Science, 3. https://doi.org/10.5964/ps.6029
Meehl, P. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press. https://doi.org/10.1037/11281-000
Niessen, C., Melchers, K. G., Ohly, S., Fay, D., Handke, L., & Kern, M. (2023). Ein Plädoyer für breit gewählte und anforderungsbezogene Leistungsindikatoren [a call for using broad and demand-focused achievement indicators]. Psychologische Rundschau, 74(3), 180–182. https://doi.org/10.1026/0033-3042/a000637
Ortner, T. M., Kretzschmar, A., Rauthmann, J. F., & Tibubos, A. N. (2013). Berufungsverfahren unter einer diagnostischen Perspektive durchführen. Psychologische Rundschau, 74(3), 187–189. https://doi.org/10.1026/0033-3042/a000638
Pruschak, G., & Hopp, C. (2022). And the credit goes to—Ghost and honorary authorship among social scientists. PLOS One, 17(5), e0267312. https://doi.org/10.1371/journal.pone.0267312
Sandoval-Lentisco, A. (2024). Commentary: “Responsible Research Assessment: Implementing DORA for hiring and promotion in psychology”. Meta-Psychology, 8. https://doi.org/10.15626/MP.2022.3655
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., & Leising, D. (2022). Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology [Preprint]. https://doi.org/10.31234/osf.io/rgh5b_v1
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., Phan, L. V., Schmitt, M., Scheel, A. M., Schubert, A.-L., Steinberg, U., & Leising, D. (2025). Responsible Research Assessment I: Implementing DORA and CoARA for Hiring and Promotion in Psychology. Meta-Psychology, 9. https://doi.org/10.15626/MP.2024.4601
Schwartz, B., Szota, K., Schmitz, J., Lueken, U., & Lincoln, T. (2023). Mehr Differenzierung nach Fachgebieten [More differentiation by subdisciplines]. Psychologische Rundschau, 74(3), 184–185. https://doi.org/10.1026/0033-3042/a000639
Sparfeldt, J. R., Spörer, N., Greiff, S., & Schneider, R. (2024). Ein Plädoyer für valide(re) Bewertungen der wissenschaftlichen Leistungen in Berufungsverfahren [A call for (more) valid evaluations of scientific achievement in hiring processes]. Psychologische Rundschau, 74(3), 185–187. https://doi.org/10.1026/0033-3042/a000640
Stenbacka, C. (2001). Qualitative research requires quality concepts of its own. Management Decision, 39(7), 551–556. https://doi.org/10.1108/EUM0000000005801
Stengelin, R., Bohn, M., Sánchez-Amaro, A., Haun, D., Thiele, M., Daum, M., Felsche, E., Fong, F., Gampe, A., Giner Torréns, M., Grueneisen, S., Hardecker, D., Horn, L., Neldner, K., Pope-Caldwell, S., & Schuhmacher, N. (2024). Responsible Research is also concerned with generalizability: Recognizing efforts to reflect upon and increase generalizability in hiring and promotion decisions in psychology. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3695
Stroebe, W., & Strack, F. (2023). Zweierlei Maß? Warum manche Psychologen den Gebrauch von quantitativen Indikatoren der Forschungsqualität ablehnen[A double standard? Why some psychologists reject the use of quantitative indicators of research quality]. Psychologische Rundschau, 74(3), 175–189. https://doi.org/10.1026/0033-3042/a000631
Syed, M. (2024). Valuing preprints must be part of responsible research assessment. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3758
Ulpts, S. (2024). Responsible assessment of what research? Beware of epistemic diversity! Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3797
Witte, E. (2024). Comment on: responsible research assessment I and responsible research assessment II. Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3685
Published
Issue
Section
License
Copyright (c) 2025 Daniel Leising, Anne Gärtner, Felix Schönbrodt
This work is licensed under a Creative Commons Attribution 4.0 International License.
