Peer Review, Peer Exposure

At the end of Pink Floyd's 'The Wall', during 'The Trial', the Judge decrees to the main character Pink that:

I sentence you to be exposed before your peers!

This has some personal resonance as I am currently attempting a part-time 'thesis by publication' thus constantly exposing myself to the scholarly (and sometimes ruthless) trial by peer review.

To recap, I began ostensibly a MSc in July 2010, assessing the impact of 1:1 laptops and the Digital Education Revolution on student performance in science. The MSc was a mutual get-out clause with the agreed intention to ultimately complete a PhD. As such I had to conduct my time as if it was a PhD. This study had been in the planning for a couple of years (while I awaited permanent residency). Consequently, I was fortunate enough to achieve human ethics approval almost immediately and begin surveying students and teachers. With a very healthy return rate this was a great start. The following year was spent reading, learning multiple regression, playing with the 2010 data and collecting 2011 data. I started work on my first paper, presenting some preliminary findings and developing some analytical tools for future use. This was when I first entered the world of peer review.

Towards the end of 2011 was the deadline for submission to present at a very large and famous 'international' (American) edtech conference. In my naïveté I hoped to kill two birds with one stone and submit a research article, thereby contributing to my publishing commitment, and thus warranting my attendance at this gargantuan event (that I had presented at previously ). My superviser wasn't that impressed with the idea but indulged me (what I should have realised was that appearing in conference proceedings has nowhere near the kudos of appearing in a journal). So accordingly I submitted a proposal. This required cutting the 5,000 words I already had down to 2,000 needed to meet the proposal guidelines. Here came my first real mistake - in reducing the word count I decided to whet the reviewers appetite by alluding to my useful analytical tools rather than presenting them for their discernment. Still, it was a decent proposal and I waited with bated breath for the reviewers' comments.

The reviewers were prompt, and the result ... my first rejection. Now, I have no problem with rejection (essential in this game!), but here is the curious feedback. There were 9 criteria, each to be awarded a mark out of 9. The results:

REVIEWER 1 scores: 9,8,7,7,8,9,7,6,8

REVIEWER 2 scores: 9,1,1,1,1,3,3,1,1

REVIEWER 3 scores: 9,8,8,7,7,8,7,6,7

So, overall, Reviewers 1 and 3 seemed to value my paper, awarding 3 perfect 9's and 6 near-perfect 8's between them. Yet Reviewer 2 slammed the proposal with 6 diabolical 1's!!! As already stated, I have no problem with rejection but a 1 is the lowest score possible. This 'peer' considered that in those six criteria I couldn't get any worse, in total contradiction with the other two reviewers. (I have a theory about the identity of Reviewer 2 but it would appear stageringly paranoid!). When teachers engage in pilot marking such inconsistency is not permitted. How is it then that this is allowed to occur for a massive conference? Whilst accepting up front the rejection I put this to the Chair of Review. It turns out that overturning a rejection would require at least two 8's in the final column (Final Recommendation). I had 8, 7, 1. Essentially, I missed out by one mark with the added comments mentioned earlier that I should have provided the analytical tools rather than alluding to them. So endeth Rejection #1.

In the meantime, I had been writing my second paper. This was on a slight tangent, with admittedly low level analysis (comparative frequencies), but interesting findings nevertheless. In early 2012, we found out late in the day about a conference on our very doorstep that might be appropriate for my research findings. As if history was repeating itself (although this time at the instigation of my superviser) I had to chop down my paper into a brief proposal and submit at the eleventh hour. The reviewers were very prompt and well and truly slated the proposal. In the rush, neither myself nor my superviser had quite done enough research into the conference and its conflicting ideology. To quote:

Definite reject: I would argue strongly for rejecting this proposal.

Rationale for the sake of my pride: not much time put into the submission due to time constraints, not too much time lost. Scholarly learning: academics in different departments often have quite opposing ideologies.

After this brutal initiation into peer review, understanding the err in my ways in opting for conferences and actually benefiting from some of the more constructive criticism, I set about improving both of my initial papers for submission to quality journals. After a heavy re-edit of the original first paper, it was submitted to a well considered, regional international journal in February 2012. After a bit of a hiatus the editor replied with recommendations from the reviewers - recommendations not rejections! After responding to every recommendation and query we received some final recommendations which again we addressed. Then, finally, in late August I received this message:

Our decision is to accept your manuscript for publication. Congratulations!

This was sensational news! After the initial setbacks with conference proceedings, I finally had some runs on the board (and good runs at that) with a quality journal. I could now say that I was 'in press'. We were hoping that the paper would make it in time for a 2012 edition but this was not to be. Hopefully the first edition in 2013 will include the paper. This will of course be featured in this blog when it happens.

Whilst the first paper was in review, the second paper was seriously re-edited and submitted to the same journal in May. After the approval of the first paper it wasn't long before the second paper was reviewed. The result - rejection (#3). One of the choice comments was:

this article adds little to what is already known.

To be fair, the feedback was much appreciated even though the right to reply was removed with the rejection, I would have loved to address some of the issues and misunderstandings with the reviewers. Same journal, different paper, different editor and reviewers - different result.

After our previous success this caught us by surprise, but we had the backup plan of submitting to an alternative, recently promoted international journal. I had a good feeling about this submission with the editor being extremely efficient and an open line of communication. However, the final result - rejection (#4). After the initial hope this was quite gutting being rejected for the second (arguably third) time for the same paper. What was particularly disappointing was the minimal feedback from the reviewers, yet still with the choice line:

The title and repeated comments about Australia being the first country to explore 1:1 laptops seem both xenophobic and parochial. 

Australia being the first country to explore 1:1 laptops over 20 years ago is simply a well published fact. Any article about the Australian Digital Education Revolution and 1:1 laptops has to mention Australia, a lot. At worst this could be considered 'parochial' (a common feature of Australia and Australians) but in this instance I would argue it is simply the relevant context. I had thought a journal that includes articles on Taiwan and Singapore would also be interested in an Australian story. As for 'xenophobic',  I took issue with this and received a formal apology from the editor and publishers. Whomever this peer and reviewer is, he/she does not know the meaning of the word. As the editor highlighted in his apology, if there had been even a whiff if bigotry or racism it would have been rejected out of hand immediately.

So the current situation is that my second paper has been resubmitted yet again, this time more locally with a good chance of success (my superviser is the editor!). I am currently working on the 'biggy', a multiple regression analysis of the unique 2011 data (all of this resubmitting and re-editing has put me behind in the prompt reporting of findings from the data). I did finally upgrade to PhD, but only after my paperwork was lost and the resulting resubmission being briefly rejected by 'new protocols' that surfaced in the interim.

In conclusion, I do totally understand and value the process of peer reviewing. In fact, if society, particularly the tabloids and shock-jocks, understood the merit and subsequent quality thanks to peer reviewing we would not have such inane, ignorant arguments and diatribe regarding much studied issues e.g. global warming. That being said, the necessary discrimination in peer review leaves many of us licking our wounds and sometimes perplexed by the reviews. As my good friend @effectsofNAPLAN pointed out, both teachers and researchers need a thick skin ... and any teacher involved in research requires "crocodile thick" skin :-)

It is important to note that reviewers do so voluntarily in their own time. This is an honourable task that I hope to aspire to one day, when I truly consider myself a peer. In the meantime, I endeavour to improve the quality of my work and take any knock-backs on the 'chin'.