Report results
The full report can be found here. The raw data files, survey files, R-scripts, excel sheets, and additional documents related to transparency and replication can be found on our OSF page.
Given the length and level of detail of the report, we also present below a short summary of each section containing some of our key findings and conclusions.
2. SURVEY DESIGN AND SAMPLE
Survey content
We conducted a 15-20 minute anonymous survey to gather the views of economists on the current peer review system and how it could be improved. Researchers could participate if, over the last two years, they had:
[Peer review survey PDF]
Our population of respondents
Data collection took place between July 2020 and January 2021, with two recruitment waves. The first wave targeted behavioral and experimental economists, while the second wave targeted all fields. The survey was advertised to various forums (e.g., the ESA discussion forum) and networks (CESifo, CEPR), and disseminated via two mail merges. In total, 1,497 researchers at least partially completed the survey, while 1,459 fully completed it.
We conducted a 15-20 minute anonymous survey to gather the views of economists on the current peer review system and how it could be improved. Researchers could participate if, over the last two years, they had:
- written at least one referee report
- received referee reports on at least one submission made to a journal.
[Peer review survey PDF]
Our population of respondents
Data collection took place between July 2020 and January 2021, with two recruitment waves. The first wave targeted behavioral and experimental economists, while the second wave targeted all fields. The survey was advertised to various forums (e.g., the ESA discussion forum) and networks (CESifo, CEPR), and disseminated via two mail merges. In total, 1,497 researchers at least partially completed the survey, while 1,459 fully completed it.
3.1 THE ECOSYSTEM OF PEER REVIEW
Unlike traditional markets, the peer review process is an ecosystem where the demand side (authors) and supply side (reviewers) come from the same pool of researchers, although not all researchers participate equally.
The demand side of peer review: authors
Over the past two years, our survey respondents made a total of 10,468 new submissions (from 1,484 researchers), with a mean (median) of 3.5 (3) submissions per year. [Exhibit 1: number of submissions histogram] However, there is heterogeneity, with most papers coming from a minority of authors (50% of papers are produced by about 20% of authors). What authors expect most from the peer review process is "receiving a reasonable and well-substantiated decision" followed by "obtaining a timely decision" and finally "getting useful feedback." In terms of individual peer review reports, authors generally believe that their primary objective should be to "help editors reach an informed decision" followed by "making precise suggestions," rather than providing general comments or detailed feedback.
The supply side of peer review: referees
Looking at the past two years, our survey respondents wrote about 15,060 referee reports per year (from 1,483 researchers), with a mean (median) of 10.2 (8) reports written per year. [Exhibit 2: number of reports histogram] There is also heterogeneity on the supply side, with the top 10% most active referees writing at least 20 reports per year. More active referees tend to be senior researchers, researchers working in the US/Canada, and researchers with some editorial experience. We also find heterogeneity in terms of the journals respondents referee for. Only about half of respondents had written at least one report for a top 5 journal over the designated period, and 80% of reviews for top 5 journals were produced by only 25% of respondents. Researchers tend to spend a lot of time on reports, with the top 10% of the distribution spending 25 working days or more per year on referee reports (based on our estimates). In line with this, researchers consider their role as referees to be quite important. Our respondents report that the main benefits of refereeing are the ability to "ensure the right papers are published" and the ability to "read papers attentively."
The demand side of peer review: authors
Over the past two years, our survey respondents made a total of 10,468 new submissions (from 1,484 researchers), with a mean (median) of 3.5 (3) submissions per year. [Exhibit 1: number of submissions histogram] However, there is heterogeneity, with most papers coming from a minority of authors (50% of papers are produced by about 20% of authors). What authors expect most from the peer review process is "receiving a reasonable and well-substantiated decision" followed by "obtaining a timely decision" and finally "getting useful feedback." In terms of individual peer review reports, authors generally believe that their primary objective should be to "help editors reach an informed decision" followed by "making precise suggestions," rather than providing general comments or detailed feedback.
The supply side of peer review: referees
Looking at the past two years, our survey respondents wrote about 15,060 referee reports per year (from 1,483 researchers), with a mean (median) of 10.2 (8) reports written per year. [Exhibit 2: number of reports histogram] There is also heterogeneity on the supply side, with the top 10% most active referees writing at least 20 reports per year. More active referees tend to be senior researchers, researchers working in the US/Canada, and researchers with some editorial experience. We also find heterogeneity in terms of the journals respondents referee for. Only about half of respondents had written at least one report for a top 5 journal over the designated period, and 80% of reviews for top 5 journals were produced by only 25% of respondents. Researchers tend to spend a lot of time on reports, with the top 10% of the distribution spending 25 working days or more per year on referee reports (based on our estimates). In line with this, researchers consider their role as referees to be quite important. Our respondents report that the main benefits of refereeing are the ability to "ensure the right papers are published" and the ability to "read papers attentively."
3.2 ALLOCATING PAPERS TO JOURNALS, REFEREES, AND EDITORS
This section focuses on the assignment of manuscript submissions to journals, editors, and reviewers.
Mismatch on the quantity dimension
To assess the extent of misallocation in the volume of reports assigned to each referee, we proceed in two complementary ways:
Mismatch on the quality dimension
We consider various aspects of the quality of the match between submissions and the journals, referees, and editors they are allocated to.
Relatedly, we discuss the fact that editorial and refereeing responsibilities are highly concentrated among certain types of researchers, which likely creates an imbalance in the system and leads to the underrepresentation of other researcher demographics.
Solving the allocation problem
The report discusses various potential solutions for improving the allocation process, and makes note of their limitations and the level of support they received from respondents. Among them, we note a few here:
Mismatch on the quantity dimension
To assess the extent of misallocation in the volume of reports assigned to each referee, we proceed in two complementary ways:
- Actual vs. reasonable number of reports: We compare the number of reports that respondents currently write to the number they think is reasonable to write. At the aggregate level, we find a substantial amount of overlap between these two distributions and a strong correlation in answers (Spearman rho = 0.73). However, there is considerable heterogeneity. Only 25% of respondents write the number of reports they deem reasonable. [Exhibit 3: actual and reasonable number of reports relationship] On average, tenured professors report writing more reports than they consider reasonable, while PhD candidates and postdocs report writing fewer. These findings (cautiously) suggest that some refereeing work could be reallocated to early-career researchers who might benefit from more practice and exposure to emerging research.
- Number of submissions vs. reports: To assess whether the allocation of reports is balanced across researchers, we also compare respondents' individual supply of reports to their individual demand, though we acknowledge that this approach has limitations. The correlation between the two quantities is quite low (Spearman rho = 0.22). The ratio of average reports written to average submissions is 2.8, with substantial heterogeneity: while about 40% of respondents write at least 3 times as many reports as they make submissions, about 15% write fewer reports than they make submissions (ratio less than 1). Researchers based in the US/Canada and editors and referees for top 5 journals tend to have a higher reports-to-submissions ratio. [Exhibit 4: number of reports to submissions heterogeneity] We offer a calculator (see our OSF page) for those interested in studying what the reports-to-submissions ratio should be for the system to be in balance, depending on various parameters (e.g., the desk rejection rate).
Mismatch on the quality dimension
We consider various aspects of the quality of the match between submissions and the journals, referees, and editors they are allocated to.
- First, we discuss the idea that there might be a mismatch between manuscripts and the journals they are submitted to, because researchers are strongly incentivized to publish in the most prestigious journals.
- Second, we look at the mismatch between submissions and reviewers' skills. Over the last two years, about 60% of our respondents rejected more than 1 in 10 requests, while 30% rejected 1 in 4 requests or more. The correlation between their rejection rate and volume of requests is weak, suggesting that other factors matter beyond current peer review load. In particular, 50% of respondents who rejected (or were tempted to reject) a request mentioned that the paper was "too remote from [their] research field" and 25% mentioned their "inability to judge the paper" (multiple reasons could be selected).
- Third, an important dimension of mismatch is conflict of interest, which is mentioned by 1 in 6 respondents as a reason for rejecting a request. About 40% of respondents think that reviewing the manuscripts of friends or co-authors should "happen as little as possible but cannot be avoided sometimes," while a similar share say that it "should never happen." We note that journals often make editor-specific conflict guidelines available, but reviewer-specific guidelines are quite rare.
Relatedly, we discuss the fact that editorial and refereeing responsibilities are highly concentrated among certain types of researchers, which likely creates an imbalance in the system and leads to the underrepresentation of other researcher demographics.
Solving the allocation problem
The report discusses various potential solutions for improving the allocation process, and makes note of their limitations and the level of support they received from respondents. Among them, we note a few here:
- The creation of a centralized platform: a centralized platform for peer review could track refereeing workloads and allow researchers to signal their availability, research interests, skills, and professional networks. A majority of respondents would be favorable to the use of such a platform. [Exhibit 5: views on the use of a centralized platform]
- Centralizing information and pooling efforts in other ways: Since setting up a (widely-adopted) centralized platform might be difficult, we also examine more immediate action steps e.g., referencing peer reviews via ORCID, making submissions to families of journals, and re-using reports from past submissions more often.
- Introducing formal procedures for managing conflicts of interest: Besides providing clear guidelines, we consider the possibility that authors could suggest or oppose the assignment of particular referees to their submission. Respondents were largely favorable to a policy allowing authors to request the disqualification of certain referees. Meanwhile, most had unfavorable views about the use of double-blind reviewing.
3.3 CONTENT OF THE REPORTS
After the initial allocation of papers to referees and journals, the report examines the production of reports, with a focus on their content.
Perceived quality of referee reports
Our respondents seemed moderately satisfied with the quality of the reports they receive, with considerable variance: on average, 43% of reports were judged to be of high quality, while 27% were considered to be of low quality. [Exhibit 6: perceived quality of reports received] When asked to consider the characteristics of low-quality reports, 75% of respondents mentioned "inaccurate statements about what the paper does or does not do." Other sources of complaints included "very vague and unconstructive comments" (two-thirds of respondents) and reports "written with an aggressive tone" (one-third of respondents).
When are referee reports most useful (if at all)?
The comments found most useful were those that help "clarify the contribution of the paper relative to the literature" and "improve the existing analysis" (both selected by about 75% of respondents). On the other hand, very few respondents found "robustness checks" and "possible extensions" particularly helpful.
Improving the quality of reports
The report examines various solutions for improving the content of reports, including:
Perceived quality of referee reports
Our respondents seemed moderately satisfied with the quality of the reports they receive, with considerable variance: on average, 43% of reports were judged to be of high quality, while 27% were considered to be of low quality. [Exhibit 6: perceived quality of reports received] When asked to consider the characteristics of low-quality reports, 75% of respondents mentioned "inaccurate statements about what the paper does or does not do." Other sources of complaints included "very vague and unconstructive comments" (two-thirds of respondents) and reports "written with an aggressive tone" (one-third of respondents).
When are referee reports most useful (if at all)?
The comments found most useful were those that help "clarify the contribution of the paper relative to the literature" and "improve the existing analysis" (both selected by about 75% of respondents). On the other hand, very few respondents found "robustness checks" and "possible extensions" particularly helpful.
Improving the quality of reports
The report examines various solutions for improving the content of reports, including:
- Supplying guidelines and doctoral training: Very few journals currently provide official guidelines for how reports should be structured and formulated. Furthermore, training on how to write good reviews is typically not included in doctoral programs. Most respondents would support the provision of clearer guidelines and doctoral training, particularly women and early-career researchers. [Exhibit 7: views on doctoral training]
- Offering feedback to reviewers via formal and informal mechanisms: Although a common practice, not all editors systematically share their decision letter and reports with all referees. Respondents judged this practice to be very useful and important for improving the reviewer's experience and quality of work. As another form of feedback, respondents were supportive of setting up a system for grading reviews and rewarding referees who write high-quality reports.
- Allowing authors to respond to reviewers: Most respondents would be favorable to a formal appeal procedure allowing authors to appeal in case of inaccurate or aggressive comments made by a reviewer; this is also true of editors and active reviewers, despite the potential extra time costs that such a procedure would imply for those most involved in the review process.
3.4 REVIEWING PROCESS AND DECISION TIMES
This section of the report focuses on current peer review processing times, which are particularly long in economics as compared to other disciplines and to its own past performance.
Delays in first-response times
We start by discussing potential delays at every stage of the review process, with a focus on first-response times, since the majority of new submissions end up being rejected. These include:
Delays in returning reviews
Using our survey data, we next focus on the delays faced by referees in returning their reports (Point #3):
Delays from additional review rounds
While our analysis mostly focuses on first-response times, we also discuss the practice of conducting multiple, prolonged review rounds, which can sometimes more than double the total review duration. We note that many respondents expressed their frustration with the repeated rounds of revision and the lack of editorial guidance about necessary changes.
Reducing delays
We discuss various potential solutions for reducing delays at each stage of the evaluation process. Among them, we note:
Delays in first-response times
We start by discussing potential delays at every stage of the review process, with a focus on first-response times, since the majority of new submissions end up being rejected. These include:
- Delays in the initial editorial decision to desk reject a paper or not
- Delays due to frictions in the matching of submissions to reviewers
- Delays in receiving all completed reports
- Delays in the editor's final decision after all reports are received
Delays in returning reviews
Using our survey data, we next focus on the delays faced by referees in returning their reports (Point #3):
- Only about a third of respondents reported always being punctual, with about a quarter acknowledging being late at least 50% of the time. [Exhibit 8: percentage of delays] Delays are also non-trivial: the median delay is 1-2 weeks, and the 90th percentile of the distribution is late by 3 weeks or more.
- Views on the appropriate turnaround time to submit a report are quite heterogeneous, with fairly similar proportions of respondents favoring 4, 6 and 8 weeks. Many reviewers are late even relative to the deadline they would find most appropriate.
- The relationship between the number of reports written and the propensity to be late is mostly weak, except that the least active reviewers are significantly more on time. Respondents with more delays do not appear to turn down requests at a much higher rate, even though they tend to write relatively more reports than they deem reasonable. While we cannot completely speak to the nature of these delays, the overall evidence seems to suggest that many late referees are overcommitted.
Delays from additional review rounds
While our analysis mostly focuses on first-response times, we also discuss the practice of conducting multiple, prolonged review rounds, which can sometimes more than double the total review duration. We note that many respondents expressed their frustration with the repeated rounds of revision and the lack of editorial guidance about necessary changes.
Reducing delays
We discuss various potential solutions for reducing delays at each stage of the evaluation process. Among them, we note:
- Increasing desk rejection rates: While many journals have responded to rapidly rising submissions by increasing the number of desk rejections, there is considerable variation in desk rejection rates across journals, even after controlling for journal quality. Our respondents appear generally favorable to the practice of journals desk rejecting manuscripts, although there is likely some heterogeneity in views about how this parameter should be set.
- Setting shorter turnaround times: Changing the default turnaround time requested for returning reviews could reduce response times from referees, as could soliciting more expedited reviews for manuscripts that should clearly be rejected.
- Making referee payments or charity donations: Three-quarters of our respondents said that referees would do a better job if they were better rewarded for their effort. Among them, about 75% indicated that referees should be paid for timely completion of the report. This payment could take many forms e.g., a donation to a charity or research fund.
- Providing non-monetary incentives and public recognition: Among respondents who thought that referees should be better rewarded, about two-thirds indicated that excellence in refereeing awards are an appropriate recognition mechanism. The report also discusses the possibility of imposing negative incentives, such as delaying the consideration of a late referee's own submissions or making delays public.
- Expanding the use of the AER: Insights model: Respondents showed overwhelming support for the short-paper, accept-or-reject, one-revision round format of AERI. Certain components of this format have been implemented at other journals, suggesting they could have broad appeal. [Exhibit 9: support for AER: Insights model]
- Limiting and better orienting revisions: Limiting the number of revision rounds and the requested turnaround time given to authors for returning their revisions could speed up the process considerably, as could better editorial direction about necessary revisions.
3.5 INNOVATIONS IN PEER REVIEW
Beyond examining discrete problems within the current system, this section explores ideas that are more transformative and controversial in nature, with the aim of fostering experimentation and innovation.
The key dimensions of peer review
To contextualize our current system within the realm of possible models of peer review, we present a taxonomy created by the Committee on Publication Ethics (COPE). Within this taxonomy, the top 5 journals are unanimous in following what COPE calls the "standard'' approach to peer review:
Transparency in peer review
We first discuss innovations pertaining to the transparency of the peer review process, focusing on: (i) the identifiability of peer reviewers and (ii) the publication of peer review documents (anonymously or not). Our respondents were generally unfavorable to an open peer review policy where "referees sign their reports and the entire review history (including responses to referees) is disclosed," whether this applies to all referees or to more senior referees only. [Exhibit 10: attitudes towards open peer review (for all referees)] To separate the issue of identifiability from the issue of publication, we also asked respondents for their views on "making the history of reports/responses to referees publicly available in an anonymized way unless the reviewers choose to disclose their identity." A near majority of respondents expressed favorable views, with junior researchers being the most favorable. [Exhibit 11: support for disclosing review history] However, respondents were not very convinced that this practice would improve review quality.
Other peer review models
Our current peer review system relies on the feedback of a limited number of ad-hoc referees, given after a full manuscript was produced. We consider several changes that could be made to this model, including:
The key dimensions of peer review
To contextualize our current system within the realm of possible models of peer review, we present a taxonomy created by the Committee on Publication Ethics (COPE). Within this taxonomy, the top 5 journals are unanimous in following what COPE calls the "standard'' approach to peer review:
- Timing: Peer review occurs pre-publication.
- Identifiability: Peer review is single-blind, with referees (but not authors) remaining anonymous.
- Mediation: Editors are the sole mediators of all interactions between reviewers and authors.
- Publication: Referee reports are unpublished.
- Facilitation: Reviews are facilitated by journals (not authors or third parties).
- Ownership: Reviewers retain ownership of their referee reports (not journals or authors).
Transparency in peer review
We first discuss innovations pertaining to the transparency of the peer review process, focusing on: (i) the identifiability of peer reviewers and (ii) the publication of peer review documents (anonymously or not). Our respondents were generally unfavorable to an open peer review policy where "referees sign their reports and the entire review history (including responses to referees) is disclosed," whether this applies to all referees or to more senior referees only. [Exhibit 10: attitudes towards open peer review (for all referees)] To separate the issue of identifiability from the issue of publication, we also asked respondents for their views on "making the history of reports/responses to referees publicly available in an anonymized way unless the reviewers choose to disclose their identity." A near majority of respondents expressed favorable views, with junior researchers being the most favorable. [Exhibit 11: support for disclosing review history] However, respondents were not very convinced that this practice would improve review quality.
Other peer review models
Our current peer review system relies on the feedback of a limited number of ad-hoc referees, given after a full manuscript was produced. We consider several changes that could be made to this model, including:
- Post-publication peer review: Submissions could be published immediately and then subjected to peer review, or they could be subject to continued evaluation at the conclusion of the standard peer-review process.
- Peer review of registered reports: Empirical papers could be conditionally accepted before the results are known, based on their research question and design. A limited number of journals have started to offer publication tracks for registered reports.
- Crowdsourced peer review and prediction markets: Rather than relying on a small number of referees, the wisdom of crowds could be leveraged to provide assessments of a manuscript's merits.
- Non-economists and non-academics as referees: Besides enlarging the size of the pool of referees who assess a paper, the diversity of the pool could be increased by seeking the opinion of researchers from other disciplines or non-academics, such as policy makers.
- Collaborative peer review platforms: Communication between authors, reviewers, and editors could be made more interactive, with the implementation of new channels for real-time discussion. Collaborative platforms could also be set up to solicit feedback before journal submission occurs.
4. TAKING A STEP BACK... but also moving forward
In this final section, we summarize the main takeaways of the report. We also acknowledge several limitations of our work and the many open questions that we left unaddressed, with the hope that they will be pursued in future work.
Challenges facing our peer review system
Several concerns about peer review emerged in the course of writing this report. First, we noted that the current distribution of refereeing work might be both inequitable and inefficient. In addition, submitted manuscripts might be inappropriate fits for the journals and referees that they end up with, in terms of submission quality, topic, and/or potential conflict. Next, lower-quality reports and review delays emerged as common complaints. Finally, concerns about transparency in decision-making and the lack of reproducibility of published research have raised questions about the appropriateness of the status quo in peer review.
Potential solutions and respondent attitudes
With input from respondents and a review of existing literature and journal policies, we presented a diverse set of potential solutions to the issues identified. These included doctoral training programs, enhanced reviewer guidance, modified default deadlines, redesigned incentive structures, information-sharing partnerships, and many other tools. Some proposals appear relatively uncontroversial and could be implemented almost immediately, such as editors sharing reports and decision letters with referees. Other proposals e.g., the sharing of anonymous review histories, commanded support from slim majorities or pluralities, which might mean that they require experimentation at a journal before being implemented more broadly. Finally, we noted that some changes might either be unfeasible or undesirable at this stage, such as the creation of a centralized platform.
Gaps in our knowledge
Our study presents several limitations that should be kept in mind when assessing the content of this report:
Changing both how we produce and evaluate research?
There are many positive and normative questions that this report left largely untouched. Given the interplay between the production and the evaluation side of research, a more comprehensive approach would have considered them jointly. In this respect, it is worth noting the following trends in production:
Next steps
As we could neither address every perspective nor discuss every proposal, we took the additional step of preparing a list of over 160 proposals, built from our research and from the input of our respondents. We stress that we do not necessarily endorse the proposals presented in the report or in the table. In fact, some proposals directly contradict each other, which is reflective of the diversity of views in our profession. We hope that these proposals and new ones will be the subject of active community discussion, both on the forum we set up and elsewhere. While the issues raised in this report are far from straightforward, the economics profession seems uniquely well-equipped to take them on in a productive manner. We will do our best to facilitate further constructive discussion in the future, so we hope that you will join the conversation.
Challenges facing our peer review system
Several concerns about peer review emerged in the course of writing this report. First, we noted that the current distribution of refereeing work might be both inequitable and inefficient. In addition, submitted manuscripts might be inappropriate fits for the journals and referees that they end up with, in terms of submission quality, topic, and/or potential conflict. Next, lower-quality reports and review delays emerged as common complaints. Finally, concerns about transparency in decision-making and the lack of reproducibility of published research have raised questions about the appropriateness of the status quo in peer review.
Potential solutions and respondent attitudes
With input from respondents and a review of existing literature and journal policies, we presented a diverse set of potential solutions to the issues identified. These included doctoral training programs, enhanced reviewer guidance, modified default deadlines, redesigned incentive structures, information-sharing partnerships, and many other tools. Some proposals appear relatively uncontroversial and could be implemented almost immediately, such as editors sharing reports and decision letters with referees. Other proposals e.g., the sharing of anonymous review histories, commanded support from slim majorities or pluralities, which might mean that they require experimentation at a journal before being implemented more broadly. Finally, we noted that some changes might either be unfeasible or undesirable at this stage, such as the creation of a centralized platform.
Gaps in our knowledge
Our study presents several limitations that should be kept in mind when assessing the content of this report:
- Our respondent pool is unlikely to be representative of the entire profession, so care should be taken when extrapolating our results to the entire economics research community.
- Our analysis makes use of self-reported data, which is typically subject to measurement error. In addition, we acknowledge that the ambiguous wording of some survey questions likely left an interpretational gap.
- Our analysis is merely correlational and provides limited information on the factors driving the observed data patterns.
- Constraints on survey space meant that we had to leave some important topics aside. In particular, we focused on the role of referees in the peer review process, without giving as much consideration to the role of editors.
Changing both how we produce and evaluate research?
There are many positive and normative questions that this report left largely untouched. Given the interplay between the production and the evaluation side of research, a more comprehensive approach would have considered them jointly. In this respect, it is worth noting the following trends in production:
- Submissions have become longer and more technical in recent years.
- Submissions have increased in number.
Next steps
As we could neither address every perspective nor discuss every proposal, we took the additional step of preparing a list of over 160 proposals, built from our research and from the input of our respondents. We stress that we do not necessarily endorse the proposals presented in the report or in the table. In fact, some proposals directly contradict each other, which is reflective of the diversity of views in our profession. We hope that these proposals and new ones will be the subject of active community discussion, both on the forum we set up and elsewhere. While the issues raised in this report are far from straightforward, the economics profession seems uniquely well-equipped to take them on in a productive manner. We will do our best to facilitate further constructive discussion in the future, so we hope that you will join the conversation.