Review for From Mice to Men – 24 years of Evaluation in CHI

This article is a good place to start a debate about how HCI methods can be suffering from a research “template” syndrome. As indicated by the authors, HCI methods have evolved an acceptable set of evaluative research tools in the past 24 years starting with methods derived from cognitive psychology.

HCI as a discipline heavily emphasizes the use of experimental design. While mentioned in the Emergence of Computing System Evaluation section, there is no discussion of the differences between the three main types of experiments: true; quasi; and natural. The point here is that true experiments manipulate variables in controlled environments, while quasi designs manipulate variables with less control. Natural experiments do not intervene and manipulate variables. Practical system evaluation seeks to measure the natural human machine interaction. I would like to see how these experimental distinctions relate to the evolution of HCI evaluation methods.

While the main taxonomy was a priori, not completely emergent, the authors did engage in iteration and refinement of the taxonomy. This process strengthens the credibility of the analysis.

The reliance on students as a convenient sample is a general problem of social science

In the section, Changes in Empirical Evaluation, the center issue of evaluative measurement cutting across quantitative, qualitative, and mixed method evaluation is the focus on task analysis. Could this lead to a discussion of how the criticism of statistical significance with small samples should not apply? Task analysis is more akin to time and motion studies, classic Tayloristic measurement techniques, than true experimental research designs requiring study replication until enough participants are included to reach statistical validity. Is the HCI community looking to measure the functional fit between man/machine or seeking to manipulate variables to uncover variables that to influence man/machine interaction?

There are excellent points to be considered. The authors’ reference that as a field matures certain methodologies and methods become preferred in a field of colleague. This is in part found by the analysis put forth by the authors. The danger of not accepting new ways of measuring and researching can diminish a field of study. The authors warn of this dilemma.

This paper is constructing the foundational argument for the needed to engage in alternative methods. The next step would be the discussion of what alternative methods would take the place of existing evaluation methods. There is no significant discussion of alternative methods, which is the basis of CHI Alt forum.