Failure detection and recovery in distributed systems: Experience with REDNET and recommendations

A number of advantages have been attributed to distributed system structures, not the least of which is increased reliability through better 'fail-soft' characteristics by enabling work load to migrate to other processors as failures occur; distributed systems therefore have found increasing favour, both in literature and in recent and proposed installations. In the nuclear power plant environment, however, careful attention will have to be paid to failure detection and recovery issues in distributed structures in order to achieve a system availability higher than 99.88%, which is already possible with the centralized systems using the hot stand-by approach currently in use in CANDU power plants. The paper describes the experience acquired in implementing failure detection and recovery features in REDNET, a distributed data acquisition system for nuclear research reactors. While some of the problems encountered are peculiar to the selected hardware and software configurations, the conclusions and recommendations presented are applicable to highly reliable distributed systems of the type needed in nuclear power plants. (author)