Development of a Diagnostic Complexity Questionnaire

The HRP human error analysis project has for some time been investigating what makes certain fault scenarios difficult for operators. One line of research has been to develop a questionnaire to measure diagnostic complexity. This report concerns some theoretical and experimental work underpinning the development of the questionnaire. A study of the literature reviewed the factors or components thought to contribute to difficulty in diagnosing and problem-solving. Two experimental studies of complexity were carried out using two versions of a questionnaire based on the review. The studies were simulator based, using scenarios designed to be diagnostically challenging. A factor-analytic approach to the analysis of the study data was suggested in the literature review. This is reported here (together with other analyses) though the factor analysis did not produce so clear results as was hoped. The present analysis found no clear factor structure with the first version of the complexity questionnaire used in experiment I. Partly because of this result, a factor-analytic approach to a second version of the questionnaire used in experiment II was not considered appropriate. A descriptive and qualitative analysis of the two questionnaire studies and a synthesis of the results from them both was promising. There were indications of components of complexity and some indications of what contributes to a personal perception of high or low diagnostic difficulty in fault scenarios. Components adding to diagnostic difficulty were tentatively named 'severity', 'need for co-operation', 'stress' and 'spread of changes'. Components not adding to difficulty were 'directness of indications', 'familiarity' and 'lack of stress'. There was some evidence of different responses to these components in a comparison of rule-based vs. knowledge-based diagnostic scenarios. These findings and experience with analysis techniques will feed into the design of further work on the human error analysis project. (author)