A broader view of research contributions: Necessary adjustments to DORA for hiring and promotion in psychology.

Recently Schönbrodt et al. (2022) released recommendations for improving how psychologists could be evaluated for recruitment, retention, and promotion. Specifically, they provided four principles of responsible research assessment in response to current methods that rely heavily on bibliometric indices of journal quality and research impact. They build their case for these principles on the San Francisco Declaration on Research Assessment (DORA) perspective that decries reliance on invalid quantitative metrics of research quality and productivity in hiring and promotion. The paper makes clear the tension panels have to address in evaluating applications—too little time to do an in-depth evaluation of an individual’s career and contribution, so reliance on easy to understand, but perhaps invalid, metrics. This dilemma requires an alternative mechanism rather than simply a rejection of metrics. To that end, the authors are to be congratulated for operationalising what those alternatives might look like. Nonethe-less, the details embedded in the principles seem overly narrow and restrictive.


Diversifying Research Outputs?
In Principle 1, the authors correctly point out the importance of considering the teaching, leadership, service and social impact of an academic's career when making judgments about hiring and promotion.The paper then proceeds to focus on how to better evaluate the research component of an academic profile.To that end Figure 1 focuses on three kinds of output (i.e., journal articles, data sets, and research software).The authors provide indicators of rigour for each category, which seem well-aligned with open science frameworks (e.g., registered reports, independently verified replication, FAIRness, independent review, etc.).In addition, they provide suggestions for how the impact could be determined using citation counts (an extant quantitative metric in the H-index), the number of reuses, GitHub stars, and Digital Science's Altmetric score.For quantity, they seem to suggest the sum and frequency of use are appropriate.These recommendations resemble strongly current quantitative metrics.However, the real focus of the manuscript is a narrowing of research outputs to journal articles at the expense of other kinds of research publication and a broadening of research outputs to open science artefacts of data sets and software applications.The inclusion of data sets and software are welcomed additions to the underlying idea that contribution to research methods is an important way to discern whether a scientist has added value to a field.My concern is about the unintended negative impact of the narrowing of valid research outputs to journal articles.

Conference Papers
Not every field of psychology values most highly the journal article.For example, research in human factors in computing values most highly conference papers in highly selective associations (e.g., the ACM CHI Conference on Human Factors in Computing Systems Proceedings have an H-index=74).Psychologists working with computers will report their work in these supposedly lowly regarded conference proceedings.Hence, evaluators need to know that within this discipline journal articles are not the gold standard.Thus, disciplinary norms for publication need to be considered.

Handbooks
This concern for disciplinary norms extends to my own field of educational psychology.Despite the importance of journal articles, education as a field still values the book and the book chapter.This is even more valued if authors contribute to highly regarded handbooks, such as the APA Handbook of Research Methods in Psychology, the APA Handbook of Multimethod Measurement in Psychology, the APA Educational Psychology Handbook, or the NCME/AERA/APA Educational Measurement series.These chapters provide authoritative guides to the field both pedagogically but also as 2 pointers to further research.Authorship is an indication of both peer esteem but also deep competence.

Limitations of Principle 1
I suspect the real objective of this principle is not to restrict research value to journal articles, data sets, or software.I hope these are meant to be examples of how research science can be adjudged, rather than a narrow specification of value.Nonetheless, as it stands, this principle provides a very narrow a way of construing what it means to add value as a research scientist.Valuable methodological contributions to any domain include tests, instruments, and protocols.Indeed, publication of protocols is a key principle of open science, but in themselves they do not produce new knowledge.In many psychological fields, developing and validating new ways to collect data from humans through standardised tests or self-report instruments and making those instruments available for wider use is an important contribution.For example, a recent followup study of the treatment of neo-natal hypoglaecemia upon educational outcomes in mid-childhood (Shah et al., 2022) made use of a standardised educational testing system of reading comprehension and mathematics (Hattie et al., 2004).Under Principle 1, that methodological development (a way to measure student learning in schooling) would not be valued, despite its contribution to medical research.
A key mechanism for exploring the psychology of learners is the self-report inventory.Well-developed inventories, both theoretically and empirically, have considerable power to reveal insights into the mind of participants and the impact of those insights onto behaviour (e.g., self-perceptions of feedback; Brown and Zhao, 2023).The invention, development, and publication of such research tools is a real contribution to any field of investigation.Unfortunately, the current statement of Principle 1 would exclude useful methodological contributions that do not exist as data sets or software.

Pedagogical Contributions to Research
While it is important that new software and methods are made available, pedagogical or instructional resources are needed to ensure the methods are understood and used.Hence, this narrowing does a disservice to research and scientists.Unless, new researchers learn how to use software and master the intricacies of new data collection or analytic methods, the scientific field will not advance robustly.Pedagogical resources are often disseminated in books, chapters, online instructional videos, and other media.
Major textbooks on general and specific methods (e.g., Bollen, 1989;Field, 2005;Fitzmaurice et al., 2004;Goldstein, 2011;Hancock and Mueller, 2010;Little and Rubin, 2002;Tabachnik and Fidell, 2007) make complex methodologies accessible to junior researchers and for experienced researchers who are new to a specific method.This type of work makes invaluable contribution to the field and while this might be treated as contribution to teaching, in my view there is a strong cross-over to the notion of research contribution.A well-developed technique that no one knows about, can understand, or follow is not a valuable contribution.In contrast, a text that makes psychological methods accessible is an indicator of both competence and contribution to the field.Indeed, it would be a rare committee that would not hire or promote the author of a methods textbook that had more than 100,000 citations on Google Scholar (i.e., Andy Field's 2013 Discovering Statistics Using IBM SPSS Statistics).In addition to print-based publications, online open-access resources such as Lakens ( 2022) demonstrate both competence and contribution of research scientists and should be taken as a sign of research competence when being evaluated.
Most research journals understandably publish cutting-edge content-focused research.Highly regarded methodological journals (e.g., Psychometrika, Psychological Methods, or Advances in Methods and Practice in Psychological Science) are committed to publishing new methods that improve how data can be collected or analysed.However, few methodological journals specialise in pedagogical instructional texts.An important exception in my view is Practical Assessment, Research, & Evaluation, a platinum (no APC, no subscription charges) journal, listed in Scopus with a 2021 CiteScore of 2.1 and a rank at the 61st percentile.This suggests that contributions to this journal are read and cited for their methodological guidance.However, the journal does not publish basic research and is not listed by the Web of Science and is likely to be ignored in researcher evaluations.Consequently, the potential to ignore the impact of contributions in this kind of journal would do a disservice to research scientists and psychology itself.
Although the authors state that indicators are not a valid way to assess research contribution, I was disconcerted to see reference to Altmetrics in Figure 1 as a way of quantising the impact of a research publication.In my experience, robust Altmetrics scores can be obtained if there is a good marketing department surrounding an academic.Few research academics are rewarded for bothering to write a PR piece from a new research output, and fewer institutional marketing departments un-derstand how psychological research could be turned into material for the media.In my own career, a piece with Altmetric = 227 (as of 25 May, 2023) had the initial media release written by the lead scientist and a piece with Altmetric = 168 (as of 25 May, 2023) appeared in a high-ranking medical journal and was the subject of the journal's editorial.Hence, this metric seems to be sensitive to factors that may have little to do with the quality of the fundamental research.Further, it relies on Twitter as a source, which has seen an exodus of research scientists to other platforms such as Mastodon (Stokel-Walker, 2022).It would be tragic if this metric led to puffery and exaggeration by rewarding research that is surrounded by motivated public relations teams, rather than reflecting real merit.
Because so many research psychologists work in universities, it is short-sighted to remove pedagogical resources from research evaluation.Universities teach and so research scientists must develop resources to help future researchers become research scientists.To ignore that aspect of a researcher's career would not help any employer in deciding to hire or promote a psychological scientist.Likewise, peer-reviewed research contributions that do not easily fit into the three prioritised categories in Principle 1 are important in many sub-disciplines of psychology.Not including them would do a disservice to psychological scientists.Nonetheless, extending valued research outputs to include data sets and software applications is a good thing.But this should not be done at the expense of traditional outputs and especially pedagogical outputs.