Review for From Mice to Men – 24 years of Evaluation in CHI

The authors analyze the role of evaluation in CHI over several decades, where their analysis reveals possible weaknesses in how CHI uses and/or insists on evaluation as part of the paper contribution.

This paper sparked my interest because I have become increasingly concerned over the insistance of the CHI community (indeed, as spelled out in the CHI submission process) that all work must include an evaluation component. My belief is that this is negatively impacting our community: innovation is not valued as much as it should be; work that is not yet ready for evaluation is being evaluated (i.e., the evaluation does not have much meaning), and that evaluation methods on hand are inadequate or just too expensive for some emerging domains. Indeed, I am currently writing a paper on this theme.

This particular paper introduces some of the same concerns, but backs it up by reviewing the role of evaluation in CHI over several decades. What is clear is that the number of papers with evaluations have increased significantly over the years, and that the quality of evaluations may have declined e.g., by lower n, by emphasing student participants, by being gender-biased.

I would really like to see this paper published. I would use it immediately as a key reference for further discussions of the role of evaluation in CHI. The details provided are strong evidence of what is going on. Indeed, I would have liked to have seen this published in the main conference.

Note: the only reason I gave the paper a 4 vs a 5 is that the 4 year time intervals between the CHI conferences studies are perhaps too long; much could have happened in the intervening years *especially in the more recent years, as the CHI review process has changed considerably). A 3 year interval (or preferably 2) would have been much better. I don't think the results would have changed much, but I would have liked that detail.

Note there ars some small typos on page 4, 5, and 6. On p 10, there is a small grammatical error : "to both trends"