Status of J-PARC operation data archiving using Hadoop and HBase

J-PARC (Japan Proton Accelerator Research Complex) is controlled with a lot of equipment, and we archive the operation data of about 64000 EPICS records in Linac and RCS. PostgreSQL is used in the data archiving system now, but it has a problem of extendibility and data migration and so on. In order to cope with the problem, we have examined the next-generation archive system using Hadoop of a distributed processing framework and HBase of distributed database. We developed the test system and use it in a tentative way. It is mentioned that a master node is SPOF (single point of failure) as a problem of Hadoop and HBase. So, we make two servers into HA (High Availability) cluster structure using Heartbeat and Pacemaker, and raise system availability. Moreover, important metadata for data management is protected by the data replication between two servers using DRBD (Distributed Replicated Block Device). About 50 TB of HDFS (Hadoop Distributed File System) is built using nine slave nodes, and HBase is worked on it. About 6,500 polling type data in a cycle of 1-60 seconds and about 13,000 the event type data in Linac are collected in a tentative way now, and it can be archiving stably. Moreover, about data retrieval, there is a case where response time is shortened by about 1/5 as compared with the present system. This paper reports the present status of this archive system, and the view of a subject and future. (author)