Experience report: System management at the ALICE HLT cluster

The ALICE HLT cluster is responsible for the first analysis and compression of the data from the ALICE experiment at CERN. The processing is performed using hardware accelerators like FPGAs, GPUs and computer nodes with commodity hardware. The mixture of hardware accelerators and several types of nodes causes an increased configuration and system management effort. To handle this effort, we are using a combination of three tools: Chef for the configuration management, Ganglia for the real time monitoring and SysMES for unattended system management, i.e. automatic problem recognition and solution. The tools help to minimize the manpower needed to administrate the cluster by reducing the time needed to recognize and identify problems or even by solving problems automatically. In this talk, we give an insight into our setup and report on the experience we have gained with the heterogeneous, on-line processing cluster during the last four years.