Friday, November 23, 2012

On Conference Paper Bidding

I spent nearly a full day bidding for papers to review for a couple of conferences last week, which led me to think about bidding strategies. As brief background, program committee (PC) members for a conference are often asked to bid on paper submissions to help the PC chair assign reviewers to papers. Typically, possible bids are something like:

  • high interest
  • moderate interest
  • definitely not interested (i.e., block the paper)
  • neutral

By default, a neutral bid is entered for all papers. A reviewer could be assigned any paper that she hasn’t blocked, but the PC chair attempts to line up the assignments with reviewer interests as much as possible, subject to other constraints (e.g., a minimum number of reviewers per paper).

One question that comes up is when to block papers. I tend to not block papers: rather than deciding which papers to block, I try to enter enough moderate+high bids that it’s unlikely I’ll be assigned many papers that I haven’t bid on. This strategy has worked ok thus far, but I wonder if it would be worth the additional effort to distinguish papers that I really don’t want to review.

I also wonder how to balance current interests vs. expertise. Say you’re an expert in some research topic A, but your current research interests are in other areas. Should you bid on papers on topic A since you can give an expert review, even though you’re not so interested in that area anymore? Or should you not bid on those papers, to increase the likelihood of being assigned papers you are more interested in? In such cases, I’ve mostly been giving moderate interest bids to topic A papers, particularly if I’ve seen papers on A accepted to other conferences that I would have reviewed negatively. But, I’ve heard of reviewers going as far as simply blocking papers they don’t want to review, independent of expertise level.

Anyway, I’m relatively new to this process, so I’d be interested to hear thoughts on bidding strategies that are good for reviewers, good for overall paper review quality, etc. Pointers to existing resources are also welcome (I found this interesting discussion via a quick search).

Update: See Mike Hicks's comment below on a cool solution to the interests vs. expertise issue he devised for POPL 2012.

Update (12/4): Check out the Toronto paper matching system, which does an algorithmic paper assignment based both on bids and by matching submitted papers against a "publication profile" for each PC member.

Wednesday, February 29, 2012

Online Supplements for Alex Ross's Books

I've been slowly working my way through Listen to This, Alex Ross's excellent second book. His first book, The Rest Is Noise, was also mind-blowingly good. I wanted to point out that the online audio guides for these books are also highly worthwhile, with many music excerpts to go along with each chapter of each book. (The only problem is you may spend a lot more time on those pages than you intended.) I've already bought a couple recordings based on the excerpts he linked, and I still have many chapters to go.

Saturday, February 25, 2012

Ranking Shows by Season using IMDB

I enjoyed this recent post by Matt Zoller Seitz ranking the first 14 seasons of The Simpsons. I wondered how this ranking would compare to a ranking based on episode ratings on IMDB. I did some tinkering with node.js and jsdom and came up with this. Sorting the output (and eliding non-existent seasons) yields:

8.34 average rating for season 6
8.33 average rating for season 5
8.31 average rating for season 7
8.26 average rating for season 4
8.18 average rating for season 8
8.16 average rating for season 3
8.01 average rating for season 2
7.91 average rating for season 9
7.82 average rating for season 1
7.63 average rating for season 10
7.39 average rating for season 12
7.35 average rating for season 11
7.17 average rating for season 23
7.10 average rating for season 16
7.09 average rating for season 18
7.04 average rating for season 13
6.98 average rating for season 19
6.96 average rating for season 17
6.96 average rating for season 15
6.92 average rating for season 20
6.91 average rating for season 22
6.90 average rating for season 21
6.90 average rating for season 14

I didn't do anything fancy like try to weight ratings by the number of votes, but the results are still interesting, and pretty similar to Zoller Seitz's ranking. To try another show, just change the URL in the script to the IMDB URL for the show with /eprate tacked on the end.