Distributed task-based in situ data analytics for high-performance simulations

A widening performance gap is separating CPU performance and IO bandwidth on large-scale systems. In some fields, such as weather forecast and nuclear fusion, numerical models generate such amounts of data that classical post hoc processing is not feasible anymore due to the limits in both storage capacity and IO performance. In situ approaches are attractive to bypass disk accesses in these cases and fully leverage the HPC platform. They are, however, often complex to set up and can require to re-develop parallel versions of the analysis from scratch. In our work, we propose a hybrid model that is well-suited for in situ workflows that combine regular simulations and irregular analytics. We couple the bulk synchronous parallel paradigm for simulation with a distributed task-based one for analysis. This reduces complexity and leverages the best of each of these two powerful paradigms. We propose a bridging model between the two paradigms and validate it through a prototype called DEISA, which supports coupling MPI parallel codes with analyses written using Dask. The bridging model requires minimal modifications of both the simulation and analysis codes compared to their post hoc counterpart. It gives access to an already existing rich ecosystem to be used in situ, such as the parallel versions of Numpy, Pandas and scikit-learn. We introduce new concepts in Dask distributed to support the in situ analytics natively. The approach has been evaluated and compared to post hoc analytics in two supercomputers, and DEISA has been used in production use cases. The results are quite interesting and show good performance with minimum coding efforts. (author)