Evaluation of the use of expert judgment to estimate human error probabilities

As part of a broad research program to develop human reliability data and estimates to support probabilistic risk assessment (PRA), the Nuclear Regulatory Commission has supported a program to develop, test, and evaluate the use of expert judgment to estimate human error probabilities (HEPs). During FY 84, this program conducted an empirical test of previously developed procedures based on psychological scaling. This test evaluated the practicality, acceptability, and usefulness of two specific procedures: paired comparisons and direct numerical estimation. A complete report of this research is provided in NUREG/CR-3688. This paper briefly summarizes the two procedures and how they would be implemented. It then describes the empirical test and its results. The results of this test provided a positive evaluation of the use of expert judgment. Judgments were shown to be consistent and to provide HEP estimates with a good degree of convergent validity. Of the two techniques tested, direct numerical estimation appears to be preferable in terms of ease of application and quality of the results. The fact remains, however, that actual relative frequencies of errors are not available, so predictive validity against such a criterion has not been established