Parallelization of gyrokinetic PIC code for MHD simulation

The gyrokinetic PIC (particle-in-cell) code for MHD simulation, Gpic-MHD, was installed on SR16000 ('Plasma Simulator' in NIFS), which is a state-of-the-art scalar SMP (symmetric multiprocessing) cluster system consisting of 8192 logical cores (128 nodes, each node includes 32 physical cores with SMP architecture, and one physical core is equivalent to two logical cores with multithreading technology). Gpic-MHD assumes a cylindrical coordinate system corresponding to the lowest order tokamak ordering. The hybrid parallel programming model of thread parallel (auto-parallelization) and process parallel (MPI) is used. The total simulation domain (cylinder) is decomposed in one (1d) or two (2d) directions. The replicas of field quantities are used to utilize logical cores larger than the number of decomposed domains (parallelization due to 'particle decomposition'). Each logical core is responsible to one decomposed domain and includes the approximately same number of particles. Gpic-MHD with the 1d domain decomposition in the axial direction, demonstrated a good scaling up to 8192 logical cores. However this scaling will saturate for more than several tens of thousands of logical cores because the communication time between logical cores will increase exponentially as the number of replicas increases. To overcome this deterioration of the scaling, the Gpic-MHD with 2d domain decomposition was made, in which total domain is decomposed in axial and radial directions. The 2d domain decomposed version also showed a good scaling, but the computation time was slower than the 1d domain decomposed version for the relatively small number of meshes and cores studied in this work; the faster computation time was obtained for the 1d domain decomposed version. However, for the future simulation with much larger meshes and logical cores, it is expected that the 2d domain decomposed version with further optimization will exhibit better parallelization performance. (author)