Review for From Mice to Men – 24 years of Evaluation in CHI

This submission analyzed selected CHI papers over the last 24 years. It focuses on the evaluation types and concludes with a critique on participants’ gender and numbers, as well as other findings.

While reading the paper, I could not help wondering why it was not accepted to the full paper track, where it was initially submitted to. Later during the paper, it became (a little) clear to me: the paper is very strong in the beginning, with a clear focus on the types of evaluations over the last 24 years. Although only a limited subset of papers was investigated, and it focuses on a very specific aspect, the paper cannot be criticized for that: it simply shows focus. However, some (not all) conclusions were not very strong:

The authors critique the use of students as participants, for example: very strong, backed up by the evidence demonstrated earlier in the paper.

On the other hand, the authors mention the lack of female participants: although there is probably no one who would argue that this is not an issue, the authors’ data and description cannot strongly back this up: for most evaluations, there is simply no data about what gender the participants are. The Figure 5 compares 1/3 and 2/3 participant rates, however, a 50% partitioning would shed more light on this issue; for example, what is “an unacceptable low proportion of females”? (p. 10) How many are “many women”?

Another (obvious) point that is hard to back up is the matter of “evaluation-requirement” for a successful CHI submission. I fully agree with the authors that innovative interfaces have a hard time getting accepted to CHI simply because they are not or cannot be evaluated with traditional methods. I agree with the authors that this causes an issue that prevents interesting work to be presented at CHI.

Why is this? CHI guidelines require an evaluation component for CHI submissions, as the authors point out. They also, however, state that almost all recent submissions have an evaluation aspect, hindering the acceptance of non-evaluated interfaces. It could also be the case, however, that only evaluated interfaces have been submitted. In this case, not the review process permits non-conventional work, but the Call for Papers. As I mentioned, I cannot agree more with the authors that CHI misses out on interesting work because of the evaluation focus, however, I am not convinced that their work on analyzing prior papers led them to this conclusion.

Overall, a very interesting paper, and very valuable to the CHI community. I would like to see the conclusions revisited, with a sole focus on what the data shows. There is enough data without straying too far for an interesting contribution.

Minor points:
[18] is cited very often
P. 6: “se saw”
P. 12: “a great tool” what makes it “great”?
Acknowledgements: The authors mention a practitioner: some of their arguments make general statements about the industry, is their argument based on his insights? If so, it would benefit from a footnote a la “personal discussion with industry representative”
Ref 14: misses year