Communication Skills for Academics Writing Peer Reviews of Manuscripts

Teaching Note Writing Peer Reviews of Manuscripts The peer review process -- both reviewing and being reviewed -- is an integral part of any academic career. While reviewing well is a demanding activity that takes time and intellectual effort, that activity pays you back in several ways. Reviewers who can write fair and constructive reviews become known to and valued by journal editors and conference chairs, and some eventually become editors or associate editors of prestigious journals. Reviewing also gives you an early glimpse of (and even influence on) new directions in your field, helping you to keep current. Finally, good reviewing skills may help you avoid making enemies. Peer reviewing is typically a double blind process – in theory, you don’t know whose paper you are reviewing and the author does not know who his/her reviewers are. In reality, however, in some cases reviewers can guess the identity of the author or the author can guess the identity of the reviewer. Thus it is important to develop your skills in constructive reviewing, so if the writer guesses who you are, s/he will appreciate your hard work and fairness, even if the review is a negative one. Of course, peer reviewing has drawbacks, as well as pay-offs. The biggest drawback is that reviewing can easily become a time sink. Since good reviewers are much in demand, you may find yourself with more requests to review than you can handle. The answer to this problem is not to write poor reviews, but to find other ways of imposing limitations to reviewing. One way of doing so is to refuse some of the requests to review. You might, for example, choose the journals that are most central to your interests, and then review only for them, turning down other requests. You can also limit the time you spend on reviewing very weak papers, recognizing that discussing all the weaknesses in detail is time consuming, unlikely to make the paper publishable, and overwhelming to the authors. Keep reviews of the weakest papers relatively brief. The Purpose of a Review The purpose of a review is not to tell the writers how you would have written the piece, nor to demonstrate your own knowledge of an area; the purpose is to guide the editor’s judgement on the manuscript and to give the authors comments that will be useful to them in revising this paper as well as in conducting future research. Overly positive reviews do not help the profession in general and the authors in particular to improve. Conversely, overly harsh reviews may simply leave the authors demoralized, without giving them guidance for future efforts around this and subsequent papers. Journal reviews tend to be much longer than conference reviews (they can be up to 5 pages long), since journal articles are expected to be thoughtful and polished end products. Conference papers, on the other hand, are often work in progress, and can include the initial stages of thinking about particular data or theories. Thus conference paper reviewers are not expected to hold conference papers to as high a standard, and reviews of conference papers are not expected to be as long (typically from one paragraph to two pages) and detailed. The longest reviews are usually written for major journals and about papers that have potential, but will require major revisions. It is critical that such reviews be helpful, and not just negative. If you consider the paper hopeless, do not provide extensive destructive comments. Keep the review shorter, highlighting major weaknesses and suggesting a different approach for further work on this subject. Whatever the length, reviews have the same essential structure and content, but at different levels of detail. Most reviews consist of responses (often on a numerical scale) to a set of structured questions, and comments to authors. Responding to Scaled Questions In responding to the scaled questions, you should take into account the intended audience for the paper – that is, the type of journal or conference to which it has been submitted -and the other papers likely to be compared to the paper at hand. While you would hold a paper for ASQ or the AER to very high standards, you might respond to the same questions with a different number on a 5-point or 7-point scale if the paper has been submitted to a lesser, regional journal, and differently yet if the paper has been submitted to a conference, such as the Academy of Management. Frequently, one item will ask you to recommend a disposition of the paper (accept, accept with revisions, revise and resubmit, reject). Most good journals very rarely accept a paper outright on first submission. Revisions are almost always required, and “revise and resubmit” (R&R) is a very commonly used category. Be careful about using it too often, however, especially for very weak papers. Remember that you will usually have to keep reviewing successive versions of a paper until it is rejected or accepted. The rating form usually does not go to the author, though in some cases it may. When this first part does not go to the authors, it often has a space in which you can put any comments intended for the editor but not the authors. This allows you to say, for example, that you think a paper cannot be revised to meet the journal’s standard, while still being constructive, if critical, in the comments to the authors. Writing Comments to Authors The time consuming part of the review is writing the open-ended comments to authors and the editor. The comments to authors may be organized into three parts: an introduction giving an overall judgement, one or more main sections discussing the big issues that you think need attention by the authors, and a final set of comments and questions organized by page number. The comments may be aimed directly towards the authors, addressing them as “you.” This stylistic approach will also help you keep your main audience in mind, rather than focusing solely on the editor or conference chair as audience. The introductory section typically gives a brief (one or two sentence) summary of the paper, then provides an overview of your judgement. For a journal article, that judgement may say, for example, that the paper has great promise and you think can be made publishable by revisions in the following two areas; or that the paper raises interesting issues, but will need an extensive overhaul if it is to become publishable in this journal; or that you are sorry to say that this paper is not appropriate to or publishable in this journal in its current form. Even if your overall judgement is fairly negative, however, it should be tactfully worded and the rest of the review should be constructive, not just pointing out problems but also suggesting remedies and new directions. And if you find aspects of a paper interesting or worthwhile, say so. Reviews are not limited to pointing out problems and weaknesses; they may note strengths, as well. The main section(s) should address major issues raised by the manuscript or major areas for revision (or, in the unlikely event that you find the paper successful as it stands, the major strengths of the manuscript). This means that you must go back to the manuscript you have just read and decide which of the marginal comments and notes you’ve made are most important. It is easy to provide a page-by-page list of problems, but harder to decide which problems are most important and to synthesize your linear thoughts as you read the paper into a coherent discussion of the key problem(s). Sometimes reviewers skip this section and go straight to the page-by-page list, but such a structure is often the sign of a weak review. It is also less useful to the authors, since a list puts all comments and problems on the same level, structurally, and gives little guidance to the authors about where they should focus their efforts. A main section starts with a general statement of the issue or problem, then provides examples from the paper to illustrate it, and finally suggests ways to fix it. Such a section could focus, for example, on the positioning of the research, dealing with ambiguities about whether the paper is exploratory and theory building or attempts to test a theory. Alternatively, it could address a problem of focus in the paper, or the inadequacy of the evidence provided to support the main argument of the paper. It could suggest that the analysis ignores a key alternative interpretation, or that a quantitative analysis needs to be done using a different statistical technique. It could suggest an area of the scholarly literature that is relevant but that the authors have not considered in the paper (but resist the temptation to cite your own work too much). Depending on the paper, the journal or conference, and your assessment of how useful the authors are likely to find your advice, you may have from one to three or four such main sections. If you cover more than one key point, subheads (e.g., “Positioning of the Paper,” “Quantitative Analysis”) are useful to help the authors see your structure and readily locate each section during their revisions. The final, page-by-page comments are typically a list of comments and questions organized by page number(s). These comments can be briefer, and may point out ambiguities, ask questions, suggest alternative wordings, and so on. They may refer to the major sections, too, highlighting a particular passage as part of a problem discussed above. In shorter, conference reviews, you may not provide such page-by-page comments, but simply an introduction and one or two main sections. In all cases, avoid nitpicking too much and focusing on minor matters that may change during any revisions. A long list of very minor points may be intimidating and not particularly useful. The Ethics and Norms of Reviewing There are commonly recognized standards of behavior vis-a-vis reviewing. The most important one is that reviewing is confidential. You may not pass the paper on to others or cite it or use material from it in your own papers. If you suspect you know who wrote the paper, and you would like to cite it, find another way to request it (e.g., request any recent working papers from that person). If you don't know who wrote a paper but have reviewed it one or more times previously and think it is close to publication, you may ask the editor to contact the author for you about possible citation of it. Another important standard involves conflict of interest. If you have been involved in a paper or the research project it reports in any significant way, you should decline to review the paper, noting the reason to the editor. Similarly, you should not review papers by colleagues in your own department or by your own students. What to do about papers that do not fit these categories, but that are familiar papers -- e.g., papers by friends or acquaintances that you have read and commented on for the authors at an earlier point -is a gray area. You can always avoid the review by stating that you have a conflict of interest, and you should probably do so if you are biased (especially negatively biased) about it. Sometimes you are sent a paper to review for one journal when you have already reviewed it (perhaps negatively) for a different journal. Academics do not all agree on how to handle this situation. Some return the manuscript, saying that they have a conflict of interest because they have reviewed it for another journal. This is certainly an acceptable course of action, especially if the problem is one of clashing paradigms (e.g., you are an experimentally oriented social psychologist and the paper is based on quasiethnographic research). It is legitimate to review a paper you have already reviewed if it was rejected as inappropriate or not good enough for one journal, but has now been submitted to a more appropriate and/or less selective journal. In any case, recycling your previous review without stating to the editor (in the section for comments to the editor) that you have previously reviewed this paper is considered inappropriate by most academics. A more acceptable action is to send the editor the previous review as well as your assessment of whether if it has changed significantly from the earlier version. Even if the paper has been sent to a lesser journal after rejection from a top journal, it should show some signs of having been revised based on the comments. Given the variance in what action is considered appropriate in this case, you should read the journal or conference guidelines and perhaps consult the editor before you decide what to do. Timeliness of reviewing is another important issue for reviewers. Typically, when you receive a review you are given a deadline by which the review is supposed to be returned. Try very hard to stick to that schedule, since when you are on the other end of the process (awaiting the editor's response to a paper under review), you certainly do not want the process to stretch out. In some cases, you will be asked to return the paper if you cannot meet the deadline, and it is better to do so than to delay the process by weeks or months. Finally, you will save everyone time if you look at each article you've been asked to review as soon you receive it. If you keep it in a sealed envelope until the review is due, then discover that you have a conflict of interest, you will delay the entire reviewing process for the manuscript. Moreover, if the paper seems to be a very weak one, you should try to turn it around quickly, rather than leaving the task to hang over you. If you run into a question about how to handle a review, think about how you would feel as the author. This policy will help you avoid potential lapses. You may also ask a more senior colleague (without providing details that will reveal too much about the paper or author) for advice in handling such situations. A Final Word about Reviewing Reviewing well is a valuable skill that will help you advance in your profession. It is also a core practice essential to the research and publication process as it is enacted in academic professions today. Be careful not to let it become a time sink, however; institutions don’t give tenure solely for good reviewing. Try to balance your activities so that you do enough reviewing to be a good citizen of your academic community, but not so much that you cannot also be a good researcher and teacher yourself.