Parallelization of gyrokinetic PIC code for MHD simulation

The gyrokinetic PIC (particle-in-cell) code for MHD simulation, Gpic-MHD, was installed on SR16000 ('Plasma Simulator' in NIFS), which is a state-of-the-art scalar SMP (symmetric multiprocessing) cluster system consisting of 8,192 logical cores (128 nodes, each node includes 32 physical cores with SMP architecture, and one physical core is equivalent to two logical cores with multithreading technology). Gpic-MHD assumes a cylindrical coordinate system corresponding to the lowest order tokamak ordering. The hybrid parallel programming model of thread parallel (auto-parallelization) and process parallel (MPI) is used. The total simulation domain (cylinder) is decomposed in one (1d) or two (2d) directions. Replicas of field quantities are used to utilize logical cores larger than the number of decomposed domains (parallelization due to 'particle decomposition'). Each process is responsible to one decomposed domain and includes the approximately same number of particles. Gpic-MHD with 1d domain decomposition in an axial direction, demonstrated a good scaling up to 8,192 logical cores. However this scaling will saturate for more than several tens of thousands of logical cores because the communication time between processes will increase as the number of replicas increases. To overcome this deterioration of the scaling, Gpic-MHD with 2d domain decomposition was made, in which the total domain is decomposed in axial and radial directions. The 2d domain decomposed version also showed a good scaling, but the computation time was a little bit longer than the 1d domain decomposed version for the relatively small number of meshes and cores studied in this work; the faster computation time was obtained for the 1d domain decomposed version. However, for the future simulation with much larger meshes and logical cores, it is expected that the 2d domain decomposed version with further optimization will exhibit better parallelization performance. (author)