Evaluation of decision aids

This paper uses the experience gained from several of the evaluation studies of person-machine system performance in the nuclear industry to describe some of the characteristics of good evaluation studies (e.g., Pew et al., 1981; Woods et al., 1982; Rouse, 1984; Rouse et al., 1984). Examples from actual evaluations are used to illustrate these characteristics. The techniques discussed are of special relevance to measuring the performance of person-machine systems on cognitive tasks such as monitoring process and goal status, fault management, and planning. The characteristics of good evaluation studies presented here include: the role of a conceptual framework or model of relevant decision problems; the importance of measures of the user's decision process as well as the outcome of that process; an emphasis on mixed fidelity simulation for cost effective testing