Reviewers ’ Decision to Sign Reviews is Related to Their Recommendation .

Surveys indicate that researchers generally have a positive attitude towards open peer review when this consists of making reviews available alongside published articles. Researchers are more negative about revealing the identity of reviewers. They worry reviewers will be less likely to express criticism if their identity is known to authors. Experiments suggest that reviewers are somewhat less likely to recommend rejection when they are told their identity will be communicated to authors, than when they will remain anonymous. One recent study revealed reviewers in five journals who voluntarily signed their reviews gave more positive recommendations than those who did not sign their reviews. We replicate and extend this finding by analyzing 12010 open reviews in PeerJ and 4188 reviews in the Royal Society Open Science where authors can voluntarily sign their reviews. These results based on behavioral data from real peer reviews across a wide range of scientific disciplines demonstrate convincingly that reviewers’ decision to sign is related to their recommendation. The proportion of signed reviews was higher for more positive recommendations, than for more negative recommendations. We also share all 23649 text-mined reviews as raw data underlying our results that can be re-used by researchers interested in peer review.

As technology advances, science advances. The rise of the internet has made it possible to transparently share all steps in the scientific process (Spellman, 2015). This includes opening up the peer review process. An increasing number of journals have started to make peer review reports available alongside published articles as part of ongoing experiments that aim to improve peer review (Bruce, Chauvin, Trinquart, Ravaud, & Boutron, 2016). Open peer review can be implemented by making peer reviews available, but also by revealing the identity of reviewers during or after the peer review process. An important argu-ment in favour of revealing the identity of reviewers is that they can receive credit for their work (Godlee, 2002). However, scientists do not feel these benefits outweigh possible costs, and are worried that criticism on manuscripts might lead to backlash from the authors in the future. Some reviewers might accept these negative consequences, while other might choose to strategically reveal their identity only for positive reviews they write.
Researchers self-report that they would be less likely to review for a journal if their identity is made public, and anecdotally mention that signed reviews would make it more difficult to be honest about manuscripts they believe are poor quality (Mulligan, Hall, & Raphael, 2013). A more recent survey found that 50.8% of almost 3000 scientists believe that revealing the identity of reviewers would make peer review worse (Ross-Hellauer, Deppe, & Schmidt, 2017). Almost twothirds of respondents believed reviewers would be less likely to deliver strong criticisms if their identity became known to the authors.
These self-report studies are complemented by experiments in which reviewers are randomly assigned to a condition where their identity would be revealed during the peer review process (Walsh, Rooney, Appleby, & Wilkinson, 2000). Reviewers in the condition where their identity was revealed were less likely to recommend rejection (n = 30) than reviewers who remained anonymous (n = 51). This suggests that a causal effect exists between knowing your identity will be revealed, and the recommendation that is made during the peer review process. Based on a small-scale meta-analysis of four studies Bruce and colleagues (2016) found support for the conclusion that reviewers are somewhat less likely to recommend rejection when they have to sign their reviews.
Although the self-report studies and the experiments clearly suggest that reviewers worry about having their name attached to more critical reviews they write, so far little is known about what reviewers actually do when given the opportunity to sign their reviews. The tradeoff between the benefit of getting credit when performing peer reviews and the risk of negative consequences when signing critical reviews might lead to strategic behavior where authors become more likely to sign reviews the more positive their recommendation is. If this strategic behavior occurs in practice, we should see a different pattern of recommendations for signed and unsigned reviews. One recent study revealed such a pattern when analyzing data from an Elsevier trial on publishing peer review reports in the journal Agricultural and Forest Meteorology, Annals of Medicine and Surgery, Engineering Fracture Mechanics, the Journal of Hydrology: Regional Studies, and the International Journal of Surgery (Bravo, Grimaldo, López-Iñesta, Mehmani, & Squazzoni, 2019). Although only 8.1% of reviewers voluntarily disclosed their identity in these reviews, the data revealed a clear difference between the recommendations by reviewers who chose to sign their reviews, compared to reviewers who did not sign.

The Current Study
We examined the relationship between the recommendations peer reviewers made and the proportion of signed reviews in two large open access journals, PeerJ (including reviews for PeerJ Computer Science) and Royal Society Open journals (Royal Society Open Science and Royal Society Open Biology). We ignored more recently launched PeerJ journals in the field of Chemistry due to the small number of articles published to date in these journals. PeerJ and Royal Society Open journals publish articles across a wide range of scientific disciplines, including biology, chemistry, engineering, life sciences, mathematics, and medicine, thus allowing us to replicate and extend the analysis by Bravo and colleagues (2019). PeerJ launched in 2012 and PeerJ Computer Science launched in 2015. PeerJ provides reviewers the possibility to sign, and authors the possibility to make peer reviews available with the final publication. Royal Society Open Science (RSOS) launched in 2014 and strongly encouraged authors to make the peer reviews available with the final publication, and made this mandatory in January 2019. Royal Society Open Biology (RSOB) made sharing reviews with the final publication mandatory in May 2017. Peer reviewers have the option to make their identity known when submitting their review to RSOS or RSOB. Because of their broad scope, the large number of publications in each journal, and their early focus on open reviews, the reviews for PeerJ and Royal Society Open journals provide insights into the peer review behavior of scientists across a wide range of disciplines.

Accessing Open Reviews
PeerJ assigns all articles a number, increasing consecutively with each published manuscript. Reviews are always accessible in HTML (e.g., reviews for the first article published in PeerJ are available at https:// peerj.com/articles/1/reviews). For Royal Society Open journals reviews are published online as a PDF file. A list of Digital Object Identifiers (DOIs) for every article published in RSOS and RSOB was retrieved through Scopus. All available reviews were downloaded, and the PDF files were converted to plain text files using pdftools for R (Ooms, 2020;R Core Team, 2013). These text files were mined for recommendations, reviewer names, submission and acceptance dates, and the review content, using the stringr package in R (Wickham, 2019).
For each article we extracted the number of revisions, and for each revision we saved whether each of the reviewers signed, the word count for their review, and their recommendation for that review round. Note that for PeerJ the editor makes the recommendation for each submission based on the reviews. We therefore do not directly know which recommendation each reviewer provided, but we analyze the data based on the assumption that the decision by the editor is correlated with the underlying reviews. For Royal Society Open journals reviewers make a recommendation, which may be to "accept as is", "accept with minor revisions", "major revision", or "reject". Because PeerJ and Royal Society Open journals only share reviews for published articles there are few "reject" recommendations for Royal Society Open journals and no "reject" recommendations by editors among PeerJ reviews. Searching all reviews for PeerJ for the words "appealed on" revealed 47 articles that were initially rejected, appealed, received a "major revision" recommendation, and were eventually published. We have coded these papers as "major revisions". All scripts to download and analyze the reviews, and computationally reproduce this manuscript, are available at https://osf.io/9526a/.

Results
We examined 8155 articles published in PeerJ (7930 in PeerJ, 225 in PeerJ Computer Science), as well as 3576 articles from Royal Society Open journals (2887 from RSOS, 689 from RSOB, 81 of which were editorials or errata without reviews) published up to October 2019. We retrieved all reviews when these were made available (the reviews were available for 5087 articles in PeerJ, and 1964 articles in Royal Society). Articles can, of course, go through multiple rounds of review. However, we focus only on the first review round in our analyses as this review reflects the initial evaluation of reviewers, before the handling editor has made any decision, following Bravo et al. (2019). On average initial submissions at PeerJ received 2.36 reviews. Articles in the Royal Society Open journals received on average 2.13 reviews for the original submission.

Signed reviews as a function of the recommendation
For all 5087 articles published in PeerJ where reviews were available we retrieved 12010 unique reviews for the initial submission (as each article is typically reviewed by multiple reviewers). In total 4592 reviewers signed their review for the initial submission, and 7418 reviewers did not. In Royal Society Open journals we analyzed 1964 articles for which we retrieved 4186 unique reviews for the first submission, where 1547 reviewers signed their review and 2639 did not. The percentages of people who signed (38.23% for PeerJ, 36.96% for Royal Society Open journals) are slightly lower than the 43.23% reported by Wang, You, Manasa, and Wolfram (2016) who analyzed the first 1214 articles published in PeerJ.
To answer our main research question we plotted the signed and unsigned reviews for PeerJ as a function of the recommendation in the first review round (see Figure 1). Remember that for PeerJ these recommendations are made by the editor, and thus only indirectly capture the evaluation of the reviewer. For minor revisions, a greater proportion of reviews was signed than unsigned, but for major revisions, more reviews were unsigned than signed. Too few articles are immediately accepted after the first round of reviews in PeerJ (22 in total) to impact the proportions in the other two categories.
To examine the extent to which there is variation in signed and unsigned reviews across subject areas in PeerJ, Figure 3 shows the number of signed and unsigned reviews, as a function of the recommendation provided by the editor, across 11 subject areas (and one left-over category for articles not assigned a section). We see that the general pattern holds across all fields. In general, reviews are less likely to be signed than unsigned, except for the paleontology and evolutionary science section, where reviews are more likely to be signed than unsigned. Beyond this main effect, we see a clear interaction across all sections, where the ratio of signed versus unsigned reviews is consistently greater for major revisions than for minor revisions. Across all fields covered by PeerJ, it is relatively less likely to see a signed review for major revisions than for minor revisions, suggesting that the observed effect is present across disciplines. Figure 4 shows the number of signed and unsigned reviews as a function of the recommendation for each year in which articles were published (since 2012 for PeerJ, and since 2014 for Royal Society Open journals). As already noted, the proportion of signed reviews reduced over time compared to earlier analyses by Wang et al. (2016). Figure 4 reveals that this conclusion holds for both PeerJ and Royal Society Open journals. It seems that in the early years of their existence more reviewers who accepted invitations to review for PeerJ and Royal Society Open journals also agreed to sign their reviews. We don't have the data to explain this pattern, but one might speculate that these open access journals attracted authors and reviewers especially interested in transparency and openness when they started to publish manuscripts. As the journals became more established, and the number of submissions increased, the reviewer population increased beyond the early adopters of open science, to include reviewers less likely to sign their reviews.
Analyzing the reviews at Royal Society Open journals provides a more direct answer to our question, since each individual reviewer is asked to provide a recommendation of "accept", "minor revisions", "major revisions", or "reject". We can therefore directly compare Proportion of "accept", "minor revisions", "major revisions", or "reject" recommendations by reviewers of Royal Society Open journals conditioned on whether reviews were signed or unsigned.
how recommendations are related to the decision to sign reviews (see Figure 2). The overall pattern clearly shows that the proportion of signed reviews is larger for more positive recommendations (accept and minor revisions) whereas the proportion of unsigned reviews is larger for more negative reviews (major revisions and reject).
We cannot draw causal conclusions based on this correlational data. It is possible that reviewers are less likely to sign more negative reviews. It is also possible that people who sign their reviews generally give more positive recommendations, and therefore the distribution of signed reviews differs from non-signed reviews. These are just two of many possible explanations for the observed pattern. Based on the literature reviewed in the introduction we know researchers are hesitant to voice criticism when their identity will be known, and experimental evidence suggests that if identities are shared with authors, recommendations become somewhat more positive. Therefore, it seems plausible that at least part of the pattern we observed can be explained by reviewers being more likely to sign more positive reviews. Although we had access to few "reject" recommendations because we could only access reviews for published manuscripts, the difference between signed and unsigned reviews for major revisions, minor revisions, and accept recommendations replicates the findings by Bravo et al. (2019) across a larger range of research fields, based on a larger dataset, and in journals where a larger percentage of reviewers volunteer to disclose their identity. This replication suggests that the difference in recommendations depending on whether reviews are signed or not is a rather reliable observation.
Both PeerJ and Royal Society Open journals publish articles in a wide range of disciplines. The open reviews at PeerJ specify the subject area the article was submitted to, but this information is not available from metadata or the bibliographic record for Royal Society Open journals.

Discussion
Our analysis shows that when authors are given the choice to sign their reviews, signed reviews have more positive recommendations than unsigned reviews. This pattern is clearly present for reviews in Royal Society Open Science and Open Biology, a large multidisciplinary journal that publishes articles across a wide range of scientific domains. The pattern is also visible in a second large multi-disciplinary journal, PeerJ, under the assumption that recommendations by editors at PeerJ are correlated with the recommendations by reviewers. Our results replicate and extend earlier findings by Bravo et al. (2019), and complement self-report and experimental results in the literature.
It might be that some reviewers consistently sign their reviews, while others sign selectively. Some researchers have contributed multiple signed reviews to our dataset, but it is unknown whether they also contributed unsigned reviews. The confidentiality of peer review makes it impossible to retrieve whether reviewers selectively sign from open reviews, and this question will need to be studied by surveying researchers directly. It is possible to indicate suggested and opposed reviewers when submitting a manuscript at PeerJ and RSOS. It is unknown how often suggested reviewers end up reviewing submissions, or whether suggested or . Total number of "minor revisions" or "major revisions" recommendations by the handling editor at PeerJ conditioned on whether reviews were signed or unsigned across different scientific disciplines.  non-suggested reviewers differ in how often they sign their reviews, depending on their recommendation. It therefore is unknown whether the current results will generalize to journals that do not ask authors to recommend reviewers. It is interesting to note that PeerJ asks reviewers for a recommendation, but only shares the reviews and the editor's decision with the authors. It would be interesting to examine whether this feature of the review process impacts reviewer's decision to sign.
Regrettably, neither PeerJ nor Royal Society Open journals make peer reviews available for manuscripts that were rejected. As a consequence, we have analyzed a biased sample of the literature. Few scientific journals make peer reviews available for all submitted articles (two notable exceptions are Meta-Psychology and F1000). Although open reviews enable us to look in more detail at the peer review process, it would be extremely interesting to be able to follow manuscripts through the peer review process even when they are rejected at one specific journal. Despite this limitation, the pattern of results we observe is very similar to that reported by Bravo et al. (2019) who had access to the reviews for accepted and rejected manuscripts.
Peer review is generally seen as an important quality control mechanism in science, yet researchers can rarely evaluate the quality of peer review at journals. Open reviews allow researchers to examine meta-scientific questions that give insights into the peer review process. The dataset we are sharing has information about the recommendations of reviewers (RSOS and RSOB) or editors (PeerJ) after each round of peer review, the names of reviewers who signed their review, and the time in review (114 days for PeerJ, 132 days for Royal Society Open journals). Using the DOI, researchers can link this data to other sources of information such as citation counts. Because the reviews themselves are included in our dataset, researchers can use the text files to answer more detailed questions about the content of peer reviews across different domains. Since we know the individual recommendation of each reviewer for Royal Society Open journals, one example of the insights that open reviews provide is how often reviewers agree. For the 1961 papers where the reviews were published, all reviewers agreed on the recommendation for 829 articles (42.27% of the time). For 41.71% of the manuscripts the maximum deviation was one category (e.g., minor and major revisions), for 14.38% of the manuscripts the maximum deviation was two categories (e.g., accept and major revision), and for 1.63% of the manuscripts the maximum deviation was three categories (i.e., accept and reject). There were 3 articles where researchers received all four possible recommendations (accept, minor revisions, major revisions, re-ject) from at least four different reviewers. We hope the dataset we share with this manuscript will allow other meta-scientists to examine additional questions about the peer review process.
Reviewers might have different reasons to sign or not sign their reviews. They might be worried about writing a positive review for a paper that is later severely criticized, and receiving blame for missing these flaws when reviewing the submission. Reviewers might not want to reveal that their recommendations are unrelated to the scientific quality of the paper, but based on personal biases or conflicts of interest. Finally, reviewers might fear backlash from authors who receive their negative reviews. Our data supports the idea that reviewers' decisions to sign are related to their recommendation across a wide range of scientific disciplines. Due to the correlational nature of the data, this relationship could emerge because reviewers who are on average more likely to sign also give more positive recommendations, reviewers are more likely to sign more positive recommendations, or both these effects could be true at the same time. Together with self-report data and experiments reported in the literature, our data increase the plausibility that in real peer reviews at least some reviewers are more likely to sign if their recommendation is more positive. This type of strategic behavior also follows from a purely rational goal to optimize the benefits of peer review while minimizing the costs. For positive recommendations, reviewers will get credit for their reviews, while for negative reviews they do not run the risk of receiving any backlash from colleagues in their field.
It is worthwhile to examine whether this fear of retaliation has an empirical basis, and if so, to consider developing guidelines to counteract such retaliation (Bastian, 2018). Based on all available research it seems plausible that at least some reviewers hesitate to sign if they believe doing so could have negative consequences, but will sign reviews with more positive recommendations to get credit for their work. Therefore, it seems worthwhile to explore ways in which reviewers could be enabled to feel comfortable to claim credit for all their reviews, regardless of whether their recommendation is positive or negative.