Reviewing Scientific Papers
I don't have any sort of claim to vast experience with reviewing papers or even a claim to producing useful scientific papers. Yet, I do have a claim to needing to take a break from a paper I'm reviewing to let my thoughts simmer down into something coherent. In the meantime, I thought I'd jot down a few thoughts on paper reviews. These thoughts aren't unique and it's unclear there's much of a contribution in stating them here, but perhaps a recasting of ideas in different words might inspire thought. (Or not, I keep re-reading this and it seems more rambling and disorganized than thought provoking. One day I'll learn to do drafts of things I write before hitting that submit button.)

Reviewing a scientific paper can be incredibly multifaceted. You can take a paper that demonstrates an amazing piece of technology that provides substantial improvements that the authors have data to back up and end up deciding it doesn't make a very good paper. The reasons why this is so can sometimes be hard to express. Especially to the poor disappointed authors who eventually receive your review and read it carefully for clues on how to vanquish whatever it was that turned you off from the research they've put so much time into.

Sometimes a paper's problem is that while one can take a piece of technology and construct an amazing system that works well in a lab, it can be highly unlikely that system will work well in real life. The thing is, you never know whether this is the problem. No one really has any clue what types of systems nicely transition from a lab to real-life. We all have ideas on which types of things will make the transition, but determining whether or not this is going to be a problem with a particular paper is not easy. Real life and reality have been frequently known to adopt sub-optimal solutions just as easily as they've been known to adopt triumphs of the research community. Completely accurate determinations as to which systems will succeed involve market forces, timing and a bunch of other factors not easily grasped, predicting the likelihood research turns to reality requires a type of foresight we haven't isolated. Since this problem is impossible, publication venues for papers generally base review criteria on the contribution a paper provides.

Generally, this means that in addition to creating a good system, one must bring something new to the table. The standard thought is that a paper that doesn't make some sort of contribution is just about unlikely to turn into reality as the work done before it. So even if the authors end up building a good system, if a very similar system has already been built, or the authors only solve a few simple issues along the way, we generally assume that solving those issues was not the limiting factor preventing that type of technology from finding it's way out into the world and making everyone's lives better. So when you read paper reviews or you end up reviewing papers, you hear a lot of discussion about what the contribution of each paper might be. The magnitude of the contribution, that is—the actual advancement in the paper—is the defining factor that usually determines a paper's ability to get published.

While this might seem to make sense, let me re-emphasize: The value of the paper is in it's contribution. No one cares what amazing things the system does if it doesn't also bring a contribution to the table. There are many papers which have laid out what should be really nice solutions to really pressing problems that never go anywhere because of niggling reasons reality cares about and academia doesn't. Unless a paper can show that they've overcome a specific problem with their system any future papers that build equivalent amazing systems aren't useful contributions to the field and will face rejection.

To make this all just a little more complex is the problem that no one really has a clear idea on what merits a good contribution. The type of contribution a paper might bring to the table is entirely ill-defined and often only begins to make sense through lots of practice. A contribution can sometimes simply be combining things together in a different way that uses some undefinable yet recognizable spark of innovation and/or insight to transform a series of most theoretical papers into a robust and deployable system. Another type of contribution is providing a fundamental building blocks that don't really yield any immediate benefit but will eventually (hopefully) be used by later systems to change the world. There are many other types of contributions. While all the types of contributions are important, everyone has slightly different ideas on how to reconcile the wide variety of contributions into a coherent scheme which dictates what papers are truly important and which are not.

It can be frustrating to end up giving poor marks to systems you think are good while giving good marks to systems you think are bad. But it's about the contribution, not about the system.

The end conclusion? Paper reviews are hit and miss. A lot of papers are easy to review and most people who read them agree on the outcome. Others however, are much less clear. Often I think, the more interesting papers tend to fall into the latter category.