Peer review: The good, the bad and the ugly

I am currently preparing a workshop, intended to help postgraduate students deal with peer-review, and in the process of looking up sources, I came across this older article in Times Higher Education, where six senior academics shared the worst advice they had received from reviewers. Peer review, which involves getting anonymous feedback from experts as a condition for publication, has a rather bad reputation in academic writing, and there is even a Facebook group dedicated to stopping the infamous Reviewer 2 (the academic equivalent of the bad cop).

The difficulty in dealing with peer review stems in part from the fact that, after investing time and effort in a study, receiving criticism is not easy. It also connects to the unfortunate fact that the anonymity of reviewers tends to encourage comments that are not consistently constructive. But the main aim of the upcoming workshop is to raise awareness that there is always something useful to learn, even from misguided suggestions. So what follows in this post is a short list of potential lessons prompted by peer-reviewer’s comments I have come across over time.

The good…

The best piece of advice I ever received from a peer-reviewer came in one of my first publications. At the time, I had just completed the pilot study of my PhD, presented the findings at a conference, and submitted the write-up to a regional journal. The reviewers made some encouraging remarks about the actual study, but to my shock one of them commented that there were problems with the way the content was presented.

The paper is over-referenced, and reads like the work of a post-graduate student. The multiple citations that follow every sentence are very distracting and disrupt the flow of the text. The author should limit himself to the most important citations, and trust the readers to know the more trivial details.

The truth is that I was initially infuriated when I read this comment, particularly because I was actually quite proud of my writing style. Academic writing is about being able to back up your thinking, and that’s what I was doing, and doing it very well, I thought! Maybe this reviewer was just jealous that I was able to cite so many sources, because I was educated in a great university, and they were just reviewing for a small regional journal. So, I was clearly dealing with professional envy, and that seemed like a much more plausible explanation than the insulting suggestion that there was something wrong with my paper.

Cartoon character, enraged at the state of today's academia

On the other hand, if I wanted to get the paper published, I had to comply with the reviewers’ instructions, so a couple of weeks later I began to revise. When I was done, I somewhat reluctantly admitted that the paper did look much better, my argument was quite tighter, and I even had more space to develop a couple of points that I had previously left out because of the word-limit. More importantly, my voice as an author was easier to hear, and it sounded more confident. The reviewer was right to point out, just like Voltaire had, that “the easiest way to be boring is to say everything”. But perhaps they were wrong in saying that I needed to trust the readers more. The person I needed to learn how to trust was myself.

The bad…

More recently, I had a less pleasant encounter with the peer-review process. A colleague in our department was about to retire, and someone suggested that it would be a nice idea to put together a Festschrift, or tribute volume, for them. Knowing that the Festschrift genre is relatively informal, another colleague and I went about collecting narratives from academics in the research area, with which we wanted to trace the influence of the retiring colleague’s thinking. Adding our own narratives to the corpus, we used a semi-formal procedure loosely informed by narrative analysis to document the spread of the retiring scholar’s seminal contrubutions, and we went on to discuss how she passed on the torch to an emerging academic community, who now worked with these ideas.

The reviewers were livid. Reviewer 1 demanded to know how we had chosen the interview participants, suggested that we should have chosen academics in a research group working in a similar field, and insisted that the paper needed more citations from that research group.

The idea of inviting experts to reflect on their experiences and histories is a good idea, though it is unclear how the informants were selected. Their stories are interesting and add to our understanding of how new theories emerge and spread. There is one BIG problem. (…) Important references are missing. I would have suggested to include the work by the [redacted] group extensively.

The editor of the volume, whose academic expertise was in primate anthropology rather than the social sciences, also chimed in, and commented that there were “too many quotations” in this narrative study. Blurring the distinction between data and discussion, she suggested that the authors’ narratives should be disguised as discussion or findings, but only reported opaquely. Apparently, this would make the study look more rigorous, by adding a veneer of ‘objectivity’ to the investigation.

You cannot include yourself in your own study. I have therefore suggested that you remove all your own citations and maybe rephrase them as main text of your article. This actually ‘kills two birds with one stone’, as it shortens the direct quotations in your manuscript (which are a bit overwhelming) and at the same time expands your discussion of the findings, which I suggested you do in my last email.

Picture of handsome academic
This may have been my social media picture at the time

In the ensuing discussion, positions became increasingly entrenched, and our chapter had to be removed from the edited collection. As is often the case in failed relationships, the question of whether we voluntarily withdrew or were rejected is a matter of different perspectives. However, a revised version of the paper, using the same narratives, was accepted for publication elsewhere, and I have some hope that the tribute volume – now over a year behind schedule – might also be published, eventually.

So what is the lesson from this chain of events? I think that it is important to remember that every piece of writing has a specific purpose and a specific audience. The mistake I made in this case was that I assumed that my writing purpose and perception of audience aligned with those of others involved in the publication of the book. As an editor of a forthcoming book myself, I now make an extra effort to communicate to the authors of individual chaptes how I envisage the entire project, and to negotiate the purpose of the chapter with them.

And the ugly…

The truth is that, although peer review comments can occasionally get brutal, I have so far been spared from the worst examples. I have, however, engaged in the somewhat perverted practice of collecting nasty comments I encounter in my role as editor of Studies in Second Language Learning and Teaching, or in various internet groups. Here are some of my favourites:

  • If the authors are so keen on ‘solving a problem’ of little practical significance to the field, maybe they should apply their skills to solving sudoku.
  • Several sources are cited, but there is little evidence to suggest that they have been consulted, let alone understood.
  • The only conceivable contribution this paper can make is by providing the academic community with an alternative to counting sheep.
  • I would suggest either activating the spell-checker on Word or finding a way to keep your cat from walking on the keyboard.
  • If the author is comfortable with having his/her name on THIS, then I won’t stand in the way of publication. 
img_2457
Also, I need to get a life…

And, yes, there are some lessons to be learnt even from these comments. In the most obvious sense, it means that – unless you actually received comments as extreme as the ones listed above – the reviewers probably do not hate your paper (not very passionately, at least). It also means that, as an author, one might need develop the skills of disconnecting their sense of personal worth from the evaluation of the paper, and disconnecting one’s own evaluation of one’s work from the perspectives of others.

There is also a useful lesson in the above, for when we have our reviewer hat on. We know, from deindividuation theory, that anonymity can bring out the darker aspects of our self. And we know, from teaching and reviewing experience, how frustrating it is when authors do not seem to have put enough effort in their work – it just seems so disrespectful that one is tempted to lash out. But the privilege of reviewer anonymity is not granted with a view to coupling cruelty with immunity, and the power to reject a substandard paper does not extend to punishing authors with unjustifiably harsh critique. The previous two examples showed that receiving peer feedback is an opportunity for authors to learn, so the corollary must be that providing feedback should be an opportunity to teach. Good teachers point at their learners’ strengths, and if a paper made it past the journal’s editor it means that there must be at least some redeeming features to it. Building on these strengths is often a better way to improve the standards of academic publications.

*

More to explore

  • Here’s some advice on how to respond to peer-reviewer comments.
  • If you’re interested in writing peer reviews, the LSE Impact of Social Sciences blog has some good tips.
  • Richard Smith, the former editor of the British Medical Journal strongly feels that peer-review doesn’t work and needs to be abolished.

*

Was it as good for you?

I hope that your own experiences with peer review have been more fortunate, or at least that you have found ways of deriving from them useful lessons, which helped you to develop your papers and yourselves as academics. By the way, I’d really be keen to read any stories about peer review that you might want to share, which might be relevant to at my forthcoming workshop. Do feel free to join in, either by adding a comment in the space below, or by sending me an email.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s