Human Error Analysis Project (HEAP) - The Fourth Pilot Study: Scoring and Analysis of Raw Data Types

Pilot study No. 4 rounded off the series of pilot studies by looking at the important issue of the quality of the various data sources. The preceding experiments had clearly shown that that it was necessary to use both concurrent and interrupted verbal protocols, and also that information about eye movements was very valuable. The effort and resources needed to analyse a combination of the different data sources is, however, significant, and it was therefore important to find out whether one or more of the data sources could replace another. In order to determine this issue, pilot study No. 4 looked specifically at the quality of information provided by different data sources. The main hypotheses were that information about operators' diagnosis and decision making would be provided by verbal protocols, expert commentators, and auto-confrontation protocols, that the data sources would be valid, and that they would complement each other. The study used three main data sources: (1) concurrent verbal protocols, which were the operators' verbalisations during the experiment; (2) expert commentator reports, which were descriptions by process experts of the operators' performance; and (3) auto-confrontation, which were the operators' comments on their performance based on a replay of the performance recording minus the concurrent verbal protocol. Additional data sources were eye movement recordings, process data, alarms, etc. The three main data sources were treated as independent variables and applied according to an experimental design that facilitated the test of the main hypotheses. The pilot study produced altogether 59 verbal protocols, some of which were in Finnish. After a translation into English, each protocol was analysed and scored according to a specific scheme. The scoring was designed to facilitate the evaluation of the experimental hypotheses. Due to the considerable work involved, the analysis process has only been partly completed, and no firm results can be reported at this time. The analysis will be continued and completed later in 1996. The pilot study has produced valuable experience about the conduct of a large scale study of this nature. and has yielded a well-structured analysis and scoring method. (author)