I spent nearly a full day bidding for papers to review for a couple of conferences last week, which led me to think about bidding strategies. As brief background, program committee (PC) members for a conference are often asked to bid on paper submissions to help the PC chair assign reviewers to papers. Typically, possible bids are something like:
- high interest
- moderate interest
- definitely not interested (i.e., block the paper)
- neutral
By default, a neutral bid is entered for all papers. A reviewer could be assigned any paper that she hasn’t blocked, but the PC chair attempts to line up the assignments with reviewer interests as much as possible, subject to other constraints (e.g., a minimum number of reviewers per paper).
One question that comes up is when to block papers. I tend to not block papers: rather than deciding which papers to block, I try to enter enough moderate+high bids that it’s unlikely I’ll be assigned many papers that I haven’t bid on. This strategy has worked ok thus far, but I wonder if it would be worth the additional effort to distinguish papers that I really don’t want to review.
I also wonder how to balance current interests vs. expertise. Say you’re an expert in some research topic A, but your current research interests are in other areas. Should you bid on papers on topic A since you can give an expert review, even though you’re not so interested in that area anymore? Or should you not bid on those papers, to increase the likelihood of being assigned papers you are more interested in? In such cases, I’ve mostly been giving moderate interest bids to topic A papers, particularly if I’ve seen papers on A accepted to other conferences that I would have reviewed negatively. But, I’ve heard of reviewers going as far as simply blocking papers they don’t want to review, independent of expertise level.
Anyway, I’m relatively new to this process, so I’d be interested to hear thoughts on bidding strategies that are good for reviewers, good for overall paper review quality, etc. Pointers to existing resources are also welcome (I found this interesting discussion via a quick search).
Update: See Mike Hicks's comment below on a cool solution to the interests vs. expertise issue he devised for POPL 2012.
Update (12/4): Check out the Toronto paper matching system, which does an algorithmic paper assignment based both on bids and by matching submitted papers against a "publication profile" for each PC member.
