I spent nearly a full day bidding for papers to review for a couple of conferences last week, which led me to think about bidding strategies. As brief background, program committee (PC) members for a conference are often asked to bid on paper submissions to help the PC chair assign reviewers to papers. Typically, possible bids are something like:
- high interest
- moderate interest
- definitely not interested (i.e., block the paper)
- neutral
By default, a neutral bid is entered for all papers. A reviewer could be assigned any paper that she hasn’t blocked, but the PC chair attempts to line up the assignments with reviewer interests as much as possible, subject to other constraints (e.g., a minimum number of reviewers per paper).
One question that comes up is when to block papers. I tend to not block papers: rather than deciding which papers to block, I try to enter enough moderate+high bids that it’s unlikely I’ll be assigned many papers that I haven’t bid on. This strategy has worked ok thus far, but I wonder if it would be worth the additional effort to distinguish papers that I really don’t want to review.
I also wonder how to balance current interests vs. expertise. Say you’re an expert in some research topic A, but your current research interests are in other areas. Should you bid on papers on topic A since you can give an expert review, even though you’re not so interested in that area anymore? Or should you not bid on those papers, to increase the likelihood of being assigned papers you are more interested in? In such cases, I’ve mostly been giving moderate interest bids to topic A papers, particularly if I’ve seen papers on A accepted to other conferences that I would have reviewed negatively. But, I’ve heard of reviewers going as far as simply blocking papers they don’t want to review, independent of expertise level.
Anyway, I’m relatively new to this process, so I’d be interested to hear thoughts on bidding strategies that are good for reviewers, good for overall paper review quality, etc. Pointers to existing resources are also welcome (I found this interesting discussion via a quick search).
Update: See Mike Hicks's comment below on a cool solution to the interests vs. expertise issue he devised for POPL 2012.
Update (12/4): Check out the Toronto paper matching system, which does an algorithmic paper assignment based both on bids and by matching submitted papers against a "publication profile" for each PC member.
5 comments:
I find that unless I block things I will get papers that I don't feel I'm even qualified to review. Then again, I've moved areas quite a bit. Reviewing a paper you don't really want to read/understand is a real chore, so I want to avoid it as much as possible.
For POPL'12, we separated bids for *preference* from bids for *expertise* and I designed an assignment algorithm that aims to maximize preferences for reviewers and expertise for papers; you will only be assigned a paper you don't (or only moderately) prefer if we can't otherwise find sufficient experts. I think this approach worked really well. For more, see http://www.cs.umd.edu/~mwh/papers/hicks12popl.html
@ergodicity.net: it seems that if the PC has sufficient expertise to review the submitted papers, and the reviewers bid for papers where they have expertise, then your reason for blocking papers shouldn't occur. I wonder: for the papers you block, do you think other PC members are entering high/moderate bids? Or is there a general lack of interest in those papers, so by blocking them you end up sticking someone else with the unpleasant reviewing job?
@Mike: I was on the POPL'12 ERC and I remember the bidding scheme. It seems like an even better idea in retrospect! :-) Hope it gets adopted by other conferences.
Principles: bid carefully, place many bids. Blocking is optional for PLDI, but if on a theory PC then block mercilessly! This also goes for heavy-duty theorists. Otherwise there will be a lot of stuff that nobody wants and you'll get assigned to it. (If Easychair then you are screwed: there is no "block" option.)
"Oh, this pi-mu-intuitionistic-lambda-calculus/separation-temporal-logic/category-theory-with-session-dependent-types paper is really about concurrent programming and I got you on the PC because you are an expert on that."
Post a Comment