Published 2019 | Version v1
Journal article

Columnar data processing for HEP analysis

  • 1. Princeton University (United States)
  • 2. National Institute of Technology, Silchar (India)

Description

In the last stages of data analysis, physicists are often forced to choose between simplicity and execution speed. In High Energy Physics (HEP), high-level languages like Python are known for ease of use but also very slow execution. However, Python is used in speed-critical data analysis in other fields of science and industry. In those fields, most operations are performed on Numpy arrays in an array programming style; this style can be adopted for HEP by introducing variable-sized, nested data structures. We describe how array programming may be extended for HEP use-cases and an implementation known as awkward-array. We also present integration with ROOT, Apache Arrow, and Parquet, as well as preliminary performance results.

Availability note (English)

Available from https://www.epj-conferences.org/articles/epjconf/pdf/2019/19/epjconf_chep2018_06026.pdf; https://doaj.org/article/f9f0e3c141c04becb657778b3f1ffb04

Additional details

Publishing Information

Journal Title
EPJ. Web of Conferences
Journal Volume
214
Journal Page Range
vp.
ISSN
2100-014X

Conference

Title
23. International Conference on Computing in High Energy and Nuclear Physics
Acronym
CHEP 2018
Dates
9-13 Jul 2018
Place
Sofia (Bulgaria)

INIS

Country of Publication
France
Country of Input or Organization
France
INIS RN
53095595
Subject category
S97: MATHEMATICAL METHODS AND COMPUTING;
Resource subtype / Literary indicator
Conference
Descriptors DEI
DATA ANALYSIS; HIGH ENERGY PHYSICS; PERFORMANCE; PYTHON
Descriptors DEC
DATA PROCESSING; PHYSICS; PROCESSING; PROGRAMMING LANGUAGES