Review for From Mice to Men – 24 years of Evaluation in CHI

This paper reviews trends in HCI evaluation methods used in CHI conference papers. It does this by sampling the various kinds of evaluation used in 4 sampled years from its 24 year history. The results show that CHI had became rather rigid in the kind of paper it accepts, tending toward quantitative evaluations of system with students and male subjects.

This evaluation is an interesting quantitative evidence of what we have known all along---the trend in CHI conference to accept papers that contain quantitative evaluations done using a particular style deemed to be the most rigorous. Problems include the tendency to use student and male subjects. Moreover, the number of subjects in studies have decreased.

Positives:
- These trends point to differences in academic style evaluation and industry accepted discount usability methods and evaluations used in iterative design.
- These results, while already known by everyone in the community and confirms it, points to a serious debate that needs to happen in the CHI community about the nature of our field and what should be done about the direction of the research.

Negatives:
- One issue about this paper is that, while giving specific numbers and quantifying this problem in certain aspects, it does little to really further the debate. It points out trends that we already know, but doesn't say much about what is needed to move the evaluation field forward.

- How do we have a constructive debate about this issue? How do we make a decision? As a position paper, it needs to take a stand and argue convincingly what needs to happen. Simple suggestions such as "don't use student subjects and have gender balance" does little to really change the trend. If the authers wish to join the debate, then a stronger stance appears to be needed.

- It is also not clear to me that a paper (even an alt.chi paper) is the best way for the debate to be carried forward. Alt.Chi doesn't appear to be the best form for this to happen, since it is meant as a venue for presentation of results, rather than a panel for debating a particular issue. I urge the authors to bring this issue to a CHI conference panel and use that as an outlet for the data here, and call attention to the issue, and most importantly, invite people to have a debate about the nature of this problem and to come up with consensus and solutions.

Finally, as a side comment, this paper should be considered along side with Joseph Kaye's submission. It appears to be on the same debate.