Expert judgements in performance assessments. Report of an SKI/SSI seminar

Expert judgements are an important element of all performance assessments and are made when alternative approaches to decision-making are not available or are not feasible. Decisions regarding the scope of a particular assessment or the type of modelling approach adopted must be made through judgements because there are no observations that can be made. Similarly, any assumptions concerning human activities in the far future are essentially speculative and must be based on expert judgement. Observations of spatial heterogeneity within a disposal system may, on the other hand, be theoretically possible but not be feasible because of excessive cost or because they would adversely affect the system they were intended to characterise. Because there is a wide range of judgements made within a performance assessment, there are several ways of making the judgements and of assessing how they have been made. Judgements may, for example, be made by individuals or by groups, and they may be formally elicited or made without formal elicitation. Dialogue with stake holders can be an important means of assessing judgements, as can peer review. Documentation is a key element throughout the process of making and reviewing judgements, and appropriate quality assurance procedures can also build confidence in judgements. In order to develop an understanding of the processes of making and assessing judgements, the Swedish Nuclear Power Inspectorate (SKI) and the Swedish Radiation Protection Institute (SSI) jointly sponsored a seminar entitled 'The Use of Expert Judgements in Performance Assessments'. The seminar was held in Norrtaelje, Sweden, on 17-19 January 2000. The seminar was organised by Galson Sciences Ltd (GSL) on behalf of SKI, and conducted jointly by GSL and Professor Steve Hora of the University of Hawaii. Participants at the seminar included SKI and SSI staff and independent experts. A key element of the seminar was an illustrative expert elicitation session, designed to give participants experience and insight into the advantages and disadvantages of one important means of making judgements in performance assessments. For the purposes of the trial elicitation exercises, participants formed two Working Groups to explore how elicitation could be used to derive numerical values and to make judgements concerning concepts or assumptions. The topics explored by the Working Groups were canister performance and cost calculations/longevity of funding. Prior to the elicitation exercises, the seminar included presentations to familiarise participants with the concepts of expert judgement and elicitation, and participants also undertook a training exercise on probability theory. This report presents summaries of the discussions held within the Working Groups. Key lessons learned by participants included the importance of appropriate training for experts prior to formal elicitation, the usefulness of the structured approach to decomposing problems into manageable issues, and the importance of clear problem definition. Although the concept of probability as a means of characterising uncertainty was unfamiliar to many participants, the potential for elicitation as one means of making judgements was recognised. Topics that participants felt could be considered for further work included the development of guidelines for elicitation, structured evaluation of canister failure modes, structured decision analysis in areas such as funding, and the use of probabilities and the treatment of uncertainty in the calculation of costs