The Operator Performance Assessment System (OPAS)

Systematic experiments in complex, rapidly changing operating environments like HAMMLAB, put special demands on operator performance measurement. These demands are presumably impossible to fulfil by traditional indicators of operator performance, as for example, temporal measures, accuracy measures, modelling techniques, or subjective expert judgement. The need for a special instrument resulted in the development of OPAS, the Operator Performance Assessment System. This system prescribes optimal solutions to scenarios through discussion with process experts. For every scenario, expert solutions are represented diagrammatically and hierarchically. During the experiment, a process expert registers operator activities in real time, concurrent with operator performance. Performance scores are then calculated, estimating the discrepancy between the expert analysis and operator solutions to scenarios. In OPAS, performance is comparable across scenarios. The system accounts for individual solution paths in dynamic operating environments, and can be applied to crews or single operators without adjustments. Real-time registration of operator activities is very effective, and the performance assessment can be accomplished in a reasonable time with reasonable resources. An empirical evaluation indicated that OPAS complies with most established standards for human performance measurement. (author)