How to write a peer review

Back in 2018, I wrote a post about being on the receiving end of peer review, where I talked about what one can learn from good feedback, from bad feedback, and from the occasional review that reads as though it came into being to settle a grudge. I wrote that post from the position of an author. Since then I have spent a fair amount of time on the other side of the process, both as a reviewer and as a journal editor for Studies in Second Language Learning and Teaching in the past and the newly established European Journal of Education and Language Review more recently. This post aims to restore some symmetry and share some of the things editors would like you to know.

My starting point, in writing this, is the following. We are trained, at considerable length, to write papers. We are trained, perhaps with less care, to respond to reviews. But we are seldom trained to write them.¹ The first review most of us produce imitates, most likely very poorly, the reviews we have received, and this means that whatever was wrong with those reviews lives on, uncorrected, in the next generation of the literature. I think we can do better, and what I would like to do, in the paragraphs that follow, is outline what such an attempt might look like.

What the review is actually for

The most fundamental issue I see in reviews seems to be a confusion about audience. It’s hardly the reviewers’ fault, though. It’s not very clear from the guidance we receive, but a review has two readers, who each want different things.

What editors look for

The first reader of a review is the editor, who needs to decide what happens to the manuscript. As an editor, I am not asking the reviewers to rank the paper against the field, or to certify its importance. What I am asking is a much narrower question: “given what you know that I do not, is there a version of this paper that this journal should publish?” and “what will it take to transform this manuscript into that version?” For this purpose, an answer along the lines of “yes, but the analysis section needs rebuilding” is useful. An answer of “the authors don’t seem very capable in their handling of statistics” is not, because it does not tell me what to do next.

What authors need

The second reader of a review is the author, who will have to act on what the reviewers write. For them, the review is a set of instructions. “Add a positionality statement”, “explain what you mean in paragraph 3”, and “verify this reference” are instructions that one can use to improve a paper, even if this manuscript is not a good fit for this journal. Everything a reviewer writes that they cannot act upon is, from the author’s point of view, noise (and occasionally, injury).

---

Almost all the practical advice that follows stems from that distinction. General comments about the rigour of the paper, its relation to the field, your reservations about the intellectual tradition where the paper belongs, and so on are best placed in the confidential comments to the editor. Specific instructions are most useful in the box with comments to the author. I know that writing two sets of comments can be tedious, but it’s not very helpful when reviewers announce an acceptance or rejection in the author-facing section, because doing so pre-empts a decision that is not theirs to make, and because this just makes the editor’s job harder if they disagree.

Some peer reviewer practices editors love

Decline quickly

Most editors understand heavy academic workloads, and will not begrudge you declining an invitation, provided you do it within a day or two (and if you could suggest a couple of alternative names, that would be fantastic). The thing is, as an editor, I can work around a “no” and move down the list of potential reviewers. What I cannot work around as easily is silence. Even worse is a “yes” that turns into a non-delivery six weeks later. If you know that the next two months are impossible, dropping a line or two to say so helps immensely.

Read the entire paper before writing anything

I will concede that I am writing this in order to make myself accountable, because it’s an issue I often face, both when reviewing and when reading student work. Sometimes objections that I raise on page two are resolved by page ten: for example, I might find a limitation in the methodology, and go ahead to write a paragraph explaining why findings cannot be generalised, only to discover that the authors are quite aware of this and have written as much in the discussion section.

When I assemble a review sequentially, as the reader’s eye goes, there’s a danger that the review just documents my shifting confusion rather than a settled judgement of the manuscript. This means that I then have to spend much time re-editing the document, which is not a very efficient use of time.

Ask what the paper is trying to do before asking whether you like it

The question the editor is asking me when I get a paper to review is not whether this is the study I would have designed. What they want to know is whether the study in this particular manuscript is coherently conceived, competently executed, and honestly described, and whether the claims are in line with what was done.

Not all the papers I reviewed have been to my taste. I believe, for example, that one can learn more about language teaching and learning² by carefully attending to people’s stories than by reducing them to numbers. But my job as a reviewer, I have tried to keep in mind, is not to make the authors write what I would have written; it is to help them write their paper in a way that removes all obstacles to clarity and all methodological objections, ultimately making their insights useful to those who will learn from them. Fas est et ab hoste doceri and all that…

But what about those papers where the difference in perspective is so fundamental that it cannot be reconciled? I hesitate to discuss real cases in much detail, but what if, say, I had to review a study in which students were zapped with a non-lethal level of electricity from their smartwatch whenever they used their home language at school? I don’t think that one can or should extend the general principle of tolerance beyond what one’s ethics allow – and that’s one instance where declining a review, even one you have accepted, seems justified.

Sort your comments by weight

I find three tiers sufficient: (a) issues that the authors need to resolve for the paper to be publishable at all, (b) issues that would substantially improve it, and finally (c) lesser matters of presentation. This helps editors judge whether the paper requires a major or a minor revision, or whether they need to reject it outright.

This helps when the revised manuscript comes back: a revision that addresses 90% of the presentation suggestions, but does not engage with the core conceptual problem that destroys the authors’ thesis is not ready for publication — and making it easier for the editor to assess what the major issues are is helpful all around.

Be specific enough to be actionable

Writing that “the literature review is inadequate” does not tell an author much, apart from communicating your displeasure. “The review moves from a discussion of self-determination theory to the L2MMS without signalling how the two theories differ in their treatment of causality, on which the hypothesis depends” tells them what the problem is, why it is important and how to correct it.

The same applies to every part of a manuscript. “The methodology is unclear” leaves the authors guessing, and –later in the process– makes it harder for the editor to know if the revisions addressed your concerns. “The paper does not explain how the twelve interviewees were selected, or whether the teachers who declined to take part differed from those who agreed, which makes it hard to assess whether these findings are typical for the region” gives the authors something to do, and the editor a box to tick.

Say what works, and mean it

This is not a politeness ritual, carried over from the ‘sandwich’ model of feedback. Naming a paper’s strengths is how we signal that any criticisms that will follow come from someone who has understood the work.

Conversely, when this section is not effective, authors (who often have to add new information to their paper while adhering to the word limit) are likely to remove the best parts of their paper during revision, because nobody told them these were the best parts.

Write in a voice you would be willing to sign

Let me start this with a disclaimer: Personally, I do not think reviews should be open. There are good reasons for anonymity, particularly for early-career reviewers assessing the work of powerful people.

A practical test is to reread your review as though the authors were sitting across the table from you. “It is hard to believe that several co-authors somehow agreed to submit this“³ would not survive that test. “The manuscript does not yet seem ready for review: the results section refers to tables that are actually missing from the manuscript” would, and (believe me!) it is no less critical. The difference is that the second sentence is about the manuscript, while the first is about the people who wrote it.

What it boils down to is that the privilege of anonymity exists so that we can be candid, not so that we can be unkind without consequence. If you would not put your name to the sentence, the problem is the sentence.

And some practices to avoid when peer reviewing

Do not review the paper you would have written

Yes, I did mention this above, but I would like to reiterate it here, firstly because some people just skim, and secondly because it is the dominant pathology of academic publishing.

Often this kind of criticism, much of which is well-meaning, comes dressed as an expectation of methodological rigour. Let’s be very clear: it is the reviewers’ job, with the editor, to engage in the kind of gate-keeping that preserves the quality (and hence the value) of the scholarly record. If there are flaws, we must name them and demand that corrections are made before the paper moves forward to publication.

But asking an ethnographer for a control group, or a small-scale qualitative study for generalisability, or an interpretive paper for hypotheses, is not a defence of standards. It is a request that the authors abandon their study and conduct the kind of work that fits the reviewers’ expectations of rigour instead.

Do not require citations to your own work

Demanding that authors include your work in theirs is one of the most reliable signatures of people who abuse the review system. As an editor, I often ignore such reviews entirely or ask authors to disregard that aspect of the comments.

If a body of work is genuinely missing and it happens to be yours, the best way to approach this is to describe the argument the authors have overlooked and trust them to find relevant work (presumably including yours) that fits their argument.

Do not treat language as a proxy for quality

A lot of the scholarship in applied linguistics and language education comes from people composing in an additional language. Non-standard English is a fixable surface feature, not evidence of weak thinking. Reviews which conflate the two perpetuate disadvantages for scholars outside the Anglophone centre, and constitute a form of epistemic injustice (i.e., unjustly questioning someone’s competence as a ‘knower’).

In more practical terms, this involves limiting comments on language unless it gets in the way of meaning, and saying exactly where. “p.7, l.13: I could not tell whether ‘the teachers’ refers to the entire school or only to the study participants” is a comment the authors can act on; “the English needs substantial improvement” is not.

Where the language needs attention throughout, it belongs in the third tier of your comments, alongside other matters of presentation, and the appropriate recommendation is professional copy-editing. Being a ‘native speaker’, by the way, is not an editorial qualification.

Do not use the review to teach the authors a lesson

Sloppy work is genuinely irritating, and the temptation to answer carelessness with contempt is real. But the power to recommend rejection does not include a licence to punish, and the author on the other end is often a doctoral student who will remember your paragraph for a decade.

None of this means pretending that carelessness is not there. If the references are unreliable, the tables do not match the text, or the manuscript requires detail-oriented proofreading, do make it clear once, plainly, and give two or three instances, so that the authors can see the pattern and find the rest themselves. But do not burden yourself with doing the line-by-line editing that the authors, or their copyeditors should do. That is not your job as a reviewer, and it’s not a good use of your time.

When you’ve established the problem, place it in the appropriate tier: a manuscript with pervasive errors may well be unpublishable in its current form. If that is the case, it’s perfectly justified to say as much. What is not helpful is a paragraph of commentary on the authors’ professionalism (“Unless the authors manage to produce on demand a copy of the book, with the words Narr Publications where my copy writes Cambridge University Press, I suggest that they buy one, read it and refrain from resubmitting before they can credibly show familiarity with at least the cover page” was fun to write, but you shouldn’t send it out). The editor gains nothing from it, and the authors learn only that peer review is a place where people are unkind.

Do not try to unmask the author, or act on it if you have guessed

You frequently will guess, especially in a small field. What matters is that the guess does not enter the review.

This is more important than it seems, because a guess changes how we read. A review written for someone we believe to be an established name tends to be either deferential (or combative, but that’s a story for another time). One written for someone we believe to be a doctoral student can easily become patronising. Neither is a review of the manuscript content alone. So resist the urge to look for the preprint or the conference abstract, and if you recognise the work anyway, keep your comments as close as you can to what is on the page.

If you are fairly sure you know who the authors are,⁴ by the way, and you have a connection with them (a former student, a co-author, a colleague in your department, or someone you have publicly disagreed with), tell the editor in the confidential comments and let them decide whether you should continue. Often the right answer is to decline.

Do not bluff beyond your expertise

Reviewing outside your competence, on the assumption that general scholarly acumen will carry you, is how we end up with methodologically illiterate reviews.

This does not mean that one should decline every paper that falls partly outside one’s area(s) of methodological expertise. Few papers sit neatly within a single reviewer’s competence, and editors assemble reviews as a set, precisely so that different reviewers can cover different parts of a manuscript: for example, if I have to assign a paper proposing a new epistemological perspective on Amazonian languages, I will likely assign it to someone who knows something about those languages and a methods expert. Neither of them is a match for the entirety of the paper, and that is fine.⁵

What it does mean is that as reviewers, we should limit our comments to the things we really know about, and let the editor know what we don’t (e.g., “I have been unable to review the ANOVA results because I do not have statistical expertise, and have therefore limited myself to a review of the underlying theory and the way findings build on it”).

Where your expertise covers part of the paper, say which part; editors would far rather have a bounded review than a confident one. And if, having read the manuscript, you realise that the part you can assess is too small to be useful, it is perfectly fine to tell the editor so and step back, as quickly as possible.

And then there is AI

Everyone involved in academic publishing knows that the reviewer shortage is not imaginary. As editors, we are chasing acceptances further down the list every year, and the realistic alternative to an AI-assisted review is often not a fine human review but a three-sentence one, or none at all. Against this background, harnessing artificial intelligence to make peer review more efficient and fair is not unreasonable. The question is finding ways to do this that do not compromise the core functions of the process.

I have chosen to discuss artificial intelligence in peer review separately from the rest of the document, because much of what follows is uncharted territory. So, unlike the advice above, which is –more or less– uncontroversial, what follows is more of an open invitation to discussion, and subject to revision.

Potentially legitimate uses of AI-assisted peer review

Once you have formed your own judgement, a model is a reasonable editor of your prose, particularly if you are writing your review in an additional language and want to be sure that your criticism is firm without being brusque. It can talk you through an unfamiliar analytic procedure in the abstract, so that you can decide whether you are competent to review the paper at all.

What these uses have in common is that the judgement stays with the reviewer and the manuscript stays inside the review system. That is the line I would draw, and it is roughly where the publishers have drawn it too. And whatever you do, tell the editor you did it. Undisclosed assistance turns a manageable question about tools into an unmanageable question about trust.⁶

Disclosure need not be elaborate. A sentence in the confidential comments to the editor is usually enough: “The evaluation in this review is my own. After drafting my comments, I used [tool] to check the English; no part of the manuscript was shared with it.” If you cannot write a sentence like that truthfully, that is probably a sign that your use of AI has crossed a line it should not have.

Confidentiality in AI-assisted peer review

A manuscript under review is a confidential document that the journal entrusts to a reviewer. Pasting it into a general-purpose chatbot discloses someone else’s unpublished work to a third party, which is quite unacceptable. This is why the policies converge as firmly as they do: Elsevier prohibits it outright, and Springer Nature, Wiley and Taylor & Francis all forbid uploading manuscripts to generative tools, although some permit AI assistance for copyediting. Before you use anything, read the policy of the journal that invited you. The variation between publishers is real, and the default assumption that “everyone does this now” is not a defence.

Accountability in AI-assisted peer review

What an editor is buying from a reviewer is a judgement: a claim, by a named person, that this work does or does not hold up. A language model can produce something that reads exactly like such a judgement. That is precisely the problem with fluent, well-organised, superficially specific commentary that exists independently of a person’s actual reading of the manuscript. As an editor, you can tell when a human has not invested in reading the paper, and act accordingly; this is not possible with weak machine reviews.

We already have a documented problem of undisclosed AI in manuscripts. If reviews are also products of AI, we arrive at a situation where a text nobody quite wrote is evaluated by a text nobody quite read, and the human contribution to the whole process is reduced to routing.

As I said at the start of this section, I do not think of these questions as settled, and I would be keen to hear where readers would draw the line. Is it acceptable, for example, to use a tool to check whether the references in a manuscript exist, if nothing but the reference list is shared? Should journals provide reviewers with approved, secure tools, rather than simply prohibiting general-purpose ones? And what should an editor do with a review that reads as machine-written but has not been disclosed as such? If you have views on any of this, or know of journal policies that handle it well, the comments are open.

A closing thought

In the 2018 post I ended by suggesting that, since peer review gives authors an opportunity to learn, writing one is an opportunity to teach. I still think that, but I would now add something less comfortable. Every review you write is also a model. An author who has never been taught to review will read it, and in a year or two will write their first review by remembering yours.

That is a slightly alarming thought, and on balance I find it a useful one.

---

Notes

1. There are honourable exceptions: doctoral colloquia that run reviewing workshops, journals that operate mentored review schemes, a few graduate programmes that build it into research training. They are exceptions, and they are also, tellingly, almost always optional. ↩︎
2. And about ourselves… ↩︎
3. True story: a reviewer actually wrote that, but that’s all I can say about that. ↩︎
4. And sometimes, they make it easy, as an established scholar once did when they included in the manuscript something along the lines “In a publication that appeared in 2005, Author 2 proposed viewing motivation as a system consisting of two self-images and the language learning experience’. We had a good laugh when I told him the story, after the paper was published (“That was not very clever of us, was it?”) — because that’s the kind of person he was… ↩︎
5. Because I don’t have too many submissions, I have the luxury to be able to explain to reviewers exactly what I am looking for. This may not be the case for larger journals, but the principle is the same. ↩︎
6. This is the same principle I apply as an editor when reference problems appear in a submission. The integrity question is rarely about the tool; it is about whether the people involved were transparent about what they did. ↩︎

---