Peer review — and particularly the oft-sorry state it seems to sink to — is a frequent topic of conversation at the water-coolers and espresso machines of scientific institutions the world over.
Of course, every researcher freshly wounded by a rejection has strong opinions about reviews and reviewers. These are often of the sort that are spectacularly unfit to print, but they are widely held nonetheless.
Yet these same wounded researchers typically serve as reviewers themselves, and write reviews which other authors receive.
And I can assure you that “other authors” all too frequently regard the remarks contained in the reviews of these erstwhile wounded researchers with the same low esteem.
So if we play out this vicious cycle to its logical conclusion, in a dystopian view peer review boils down to the following:
- We trash one another’s work.
- Everything gets rejected.
- And we all decide to pack up our toys and go home.
That’s not much of a recipe for scientific progress.
But what fuels this vicious cycle and what can be done about it?
As reviewers, how can we produce Excellent Reviews that begin to unwind this dispiriting scientific discourse?
As authors, how should we interpret the comments of referees, or (ideally) write papers that will be better received in the first place?
When I pulled together the program committee for the annual MobileHCI conference last year, I found myself pondering all of these issues, and really wondering what we could do to advance the conference’s review process with a positive footing.
And particularly because MobileHCI is a smaller venue, with many of the program committee members still relatively early in their research careers, I really wanted to get them started with the advice that I wished someone had given me when I first started writing and reviewing scientific papers in graduate school.
So I penned an essay that surfaces all of these issues. It describes some of the factors that lead to this vicious cycle in reviews. It makes some very specific recommendations about what an excellent review is, and how to produce one. And if you read it as an author (perhaps smarting from a recent rejection) who wants to better understand where the heck do these reviews come from anyway? and as a by-product actually write better papers, then reading between the lines will give you some ideas of how to go about that as well.
And I was pleased, if not more than a bit surprised, to see that my little rant essay was well-received by the research community:
And I received many other private responses with a similar tenor.
So if you care at all about these issues I hope that you will take a look at what I had to say. And circle back here to leave comments or questions, if you like.
There’s also a companion presentation [Talk PPTX] [Talk PDF], which I used with the MobileHCI program committee to instill a positive and open-minded attitude as we embarked on our deliberations. I’ve included that here as well in the hope that it might be of some use to others hoping to gain a little insight into what goes on in such meetings, and how to run them.
Hinckley, K., So You’re a Program Committee Member Now: On Excellence in Reviews and Meta-Reviews and Championing Submitted Work That Has Merit. Published as “The MobileHCI Philosophy” on the MobileHCI 2015 Web Site, Feb 10th, 2015. [PDF] [Talk PPTX] [Talk PDF].
Like this:
Like Loading...
Related
The Past and Present Future 
Commentary: On Excellence in Reviews, Thoughts for the HCI Community
Peer review — and particularly the oft-sorry state it seems to sink to — is a frequent topic of conversation at the water-coolers and espresso machines of scientific institutions the world over.
Of course, every researcher freshly wounded by a rejection has strong opinions about reviews and reviewers. These are often of the sort that are spectacularly unfit to print, but they are widely held nonetheless.
Yet these same wounded researchers typically serve as reviewers themselves, and write reviews which other authors receive.
And I can assure you that “other authors” all too frequently regard the remarks contained in the reviews of these erstwhile wounded researchers with the same low esteem.
So if we play out this vicious cycle to its logical conclusion, in a dystopian view peer review boils down to the following:
That’s not much of a recipe for scientific progress.
But what fuels this vicious cycle and what can be done about it?
As reviewers, how can we produce Excellent Reviews that begin to unwind this dispiriting scientific discourse?
As authors, how should we interpret the comments of referees, or (ideally) write papers that will be better received in the first place?
When I pulled together the program committee for the annual MobileHCI conference last year, I found myself pondering all of these issues, and really wondering what we could do to advance the conference’s review process with a positive footing.
And particularly because MobileHCI is a smaller venue, with many of the program committee members still relatively early in their research careers, I really wanted to get them started with the advice that I wished someone had given me when I first started writing and reviewing scientific papers in graduate school.
So I penned an essay that surfaces all of these issues. It describes some of the factors that lead to this vicious cycle in reviews. It makes some very specific recommendations about what an excellent review is, and how to produce one. And if you read it as an author (perhaps smarting from a recent rejection) who wants to better understand where the heck do these reviews come from anyway? and as a by-product actually write better papers, then reading between the lines will give you some ideas of how to go about that as well.
And I was pleased, if not more than a bit surprised, to see that my
little rantessay was well-received by the research community:And I received many other private responses with a similar tenor.
So if you care at all about these issues I hope that you will take a look at what I had to say. And circle back here to leave comments or questions, if you like.
There’s also a companion presentation [Talk PPTX] [Talk PDF], which I used with the MobileHCI program committee to instill a positive and open-minded attitude as we embarked on our deliberations. I’ve included that here as well in the hope that it might be of some use to others hoping to gain a little insight into what goes on in such meetings, and how to run them.
Share this:
Like this:
Related