I am was [in May 2018] in the process of preparing a workshop, intended to help postgraduate students deal with peer-review, and this post is my way of procrastinating. Peer review, which involves getting anonymous feedback from experts as a condition for publication in an academic journal, serves an essential function in preserving the standards of academic publishing, but it has a rather bad reputation in academic writing. In fact, there is even a Facebook group dedicated to stopping the infamous Reviewer 2 (the academic equivalent of the bad cop).
The difficulty in dealing with peer review stems in part from the fact that, after investing time and effort in a study, receiving criticism is not easy. It is also quite unfortunate that the anonymity of reviewers tends to encourage comments that are not consistently constructive. But the main aim of the upcoming workshop is to raise awareness that there is always something useful to learn, even from misguided suggestions. So what follows is a short list of lessons I learnt (or should have learnt), from peer-reviewer’s comments that I received at one time or another.
The good…
The best piece of advice I ever received from a peer-reviewer came in one of my first publications. At the time, I had just completed the pilot study of my PhD, presented the findings at a conference, and submitted the write-up to a regional journal. The reviewers made some encouraging remarks about the study, and that pleased me enormously. But – to my shock – one of them commented that there were problems with the way I was presenting the content:
The paper is over-referenced, and reads like the work of a post-graduate student. The multiple citations that follow every sentence are very distracting and disrupt the flow of the text. The author should limit himself to the most important citations, and trust the readers to know the more trivial details.
The truth is that I was more than slightly upset when I read this comment, particularly because I was actually quite proud of my writing style. Academic writing is about being able to back up your thinking, and that’s what I was doing, and doing it very well, I thought! Maybe this reviewer was just jealous because I was able to cite so many sources. Plus, I was educated in a great university, and they were just reviewing for a small regional journal. So, I decided, I was clearly dealing with professional envy. That seemed like a much more plausible explanation than the insulting suggestion that there was something wrong with my paper.
On the other hand, if I wanted to get the paper published, I had to comply with the reviewers’ instructions, so a couple of weeks later I began to revise. When I was done, I somewhat reluctantly admitted that the paper actually looked much better, my argument was tighter, and I even had more space to develop a couple of points that I had previously left out because of the word-limit. More importantly, my voice as an author was easier to hear, and it sounded more confident. The reviewer was right to point out, just like Voltaire had done before him, that “the easiest way to be boring is to say everything”. But perhaps they were wrong in saying that I needed to trust the readers more. The person I needed to learn how to trust was myself.
The bad…
More recently, I had a less pleasant encounter with the peer-review process. A colleague in my (then) department was about to retire, and someone suggested that it would be a nice idea to put together a Festschrift, or tribute volume, for her. Knowing that a Festschrift is more about honouring a friend and less about breaking new scientific ground, another colleague and I decided to write a chapter mapping out the influence of the retired colleague’s thinking. We therefore went about collecting narratives from academics who were working in the same research area; then, after adding our own narratives to the corpus, we used a semi-formal procedure loosely informed by narrative analysis to document the spread of the retiring scholar’s seminal contributions. We went on to discuss how she passed on the torch to an emerging academic community, who carried on working with these ideas, and suggested some directions for future research.
The reviewers were livid. Reviewer 1 demanded to know how we had chosen these particular interview participants, and suggested that we should have chosen academics from a different research group working in a similar field. He also insisted that the paper needed more citations from that particular research group. In his view:
The idea of inviting experts to reflect on their experiences and histories is a good idea, though it is unclear how the informants were selected. Their stories are interesting and add to our understanding of how new theories emerge and spread. There is one BIG problem. (…) Important references are missing. I would have suggested to include the work by the [redacted] group extensively.
The editor of the volume, whose academic expertise was in primate anthropology rather than the social sciences, also chimed in, and commented that there were “too many quotations” in this narrative study. Blurring the distinction between data and discussion, she suggested that the authors’ narratives should be disguised as discussion of findings, but only reported opaquely. Apparently, this would make the study look more rigorous, by adding a veneer of ‘objectivity’ to the investigation. She wrote:
You cannot include yourself in your own study. I have therefore suggested that you remove all your own citations and maybe rephrase them as main text of your article. This actually ‘kills two birds with one stone’, as it shortens the direct quotations in your manuscript (which are a bit overwhelming) and at the same time expands your discussion of the findings, which I suggested you do in my last email.
In the ensuing discussion, positions became increasingly entrenched, and our chapter was eventually removed from the edited collection. As is often the case in failed relationships, the question of whether we voluntarily withdrew or were rejected is a matter of differing perspectives. However, a revised version of the paper, using the same narratives, was accepted for publication elsewhere, and the tribute volume is also in print, a couple of years behind schedule. It is actually a good book.
So what is the lesson from this chain of events? I think that it is important to remember that every piece of writing has a specific purpose and a specific audience. The mistake I made in this case was that I assumed that my writing purpose and perception of audience aligned with those of others involved in the publication of the book. As the editor of a forthcoming newly published book, I made an extra effort to communicate to the authors of individual chapters how I envisaged the entire project, and to negotiate the purpose of their chapter with them.
And the ugly…
The truth is that, although peer review comments can occasionally get brutal, I have so far been spared from the worst examples. I have, however, engaged in the somewhat perverted practice of collecting nasty comments that I come across in my (former) role as a journal editor, or in various internet groups. Here are some of my favourites:
- If the authors are so keen on ‘solving a problem’ of little practical significance to the field, maybe they should apply their skills to solving sudoku.
- Several sources are cited, but there is little evidence to suggest that they have been consulted, let alone understood.
- The only conceivable contribution this paper can make is by providing the academic community with an alternative to counting sheep.
- I would suggest either activating the spell-checker on Word or finding a way to keep your cat from walking on the keyboard.
- If the author is comfortable with having his/her name on THIS, then I won’t stand in the way of publication.
And, yes, there are some lessons to be learnt even from these comments. In the most obvious sense, it means that – unless you actually received comments as extreme as the ones listed above – the reviewers probably do not hate your paper (not very passionately, at least). It also means that, as an author, one might need develop the skills of disconnecting their sense of personal worth from the evaluation of the paper, and disconnecting one’s own evaluation of one’s work from the perspectives of others.
There is also a useful lesson in the above, for when we have our reviewer hat on. We know, from deindividuation theory, that anonymity can bring out the darker aspects of our self. And we know, from teaching and reviewing experience, how frustrating it is when authors do not seem to have put enough effort in their work – it just seems so disrespectful that one is tempted to lash out. But the privilege of reviewer anonymity is not granted with a view to coupling cruelty with immunity, and the power to reject a substandard paper does not extend to punishing authors with unjustifiably harsh critique.
If there is something I have taken away from the experiences I described above, it is that peer feedback gives authors an opportunity to learn. I take this to mean that that providing feedback should be an opportunity to teach. Good teachers point at their learners’ strengths, and the papers we receive as reviewers must have some redeeming features, if they made it past initial screening by the journal’s editor. Building on these strengths is often a better way to improve the standards of academic publications than pointing out weaknesses.
More to explore
- This post of mine describes what happens when an article is submitted to a journal.
- Here’s some advice on how to respond to peer-reviewer comments.
- Sometimes Reviewer 2 will ask you to cite their papers. Stephen Heard (Scientist Sees Squirrel) has some useful suggestions for dealing with this situation.
- If you’re interested in writing peer reviews, the LSE Impact of Social Sciences blog has some good tips.
- Finally, if you think that your reviewers have been harsh, you may want read this older article in Times Higher Education, where six senior academics share the worst advice they had received from reviewers.
Before you go…
I hope that your own experiences with peer review have been more fortunate, or at least that you have found ways of deriving from them useful lessons, which helped you to develop your papers and yourselves as academics. When I originally wrote this post, in May 2018, I asked readers to chime in with your own peer review experiences, which I wanted to use in the upcoming workshop. Although the workshop is now on the past, I am still keen on reading about any experiences that you might want to share, so please feel free to add a comment in the space below or get in touch.
About this post: This post was originally written in May 2018. It was updated in February 2020, at which time I updated various links, and removed or updated information that was no longer accurate. With the exception of the Disney image, which is promotional material, all other images are my own work.
Leave a Reply