Review for From Mice to Men – 24 years of Evaluation in CHI

The paper analyzes the trend in CHI papers regarding qualititative and quantitative methods and their characteristics. In particular, it notes that CHI papers are becoming too rigid and that the statistical power of quantitative evaluations is decreasing due smaller samples sizes.

Overall, the paper is very interesting and makes many valid points. However, it does not provide an alternative approach to studying HCI. It also does not go into any details regarding statistical power and how reducing the number of subjects in a quantitiative study affects power.

It also alleges gender bias and provides evidence for such. I think this is a valid point and studies should at least include a statement of the gender distribution and if there's a significant difference between the populations.

I am concerned that the paper did its sampling by using equal time intervals versus a random sample. If a random sample were drawn one could possibly draw statistical conclusions from the data, such as whether the decrease in sample size is statistically significant.

The paper notes that the sample size is going down, which is not necessarily bad; it is simply more efficient to reduce sample size. In fact, other fields have started to use dynamic sampling to get the smallest number of scores needed to make a statistically valid inference. The issue is that papers do not publish the "power" of their samples.