Offline data processing in the First JUNO Data Challenge

The Jiangmen Underground Neutrino Observatory (JUNO) is a multipurpose neutrino experiment with the primary goals of the determining the neutrino mass ordering and precisely measuring oscillation parameters [1, 2, 3]. Currently under construction in Southern China, it comprises a central detector (CD) for neutrino detection, a water pool (WP) and a top tracker (TT) for cosmic ray muon measurement. The innermost of CD is 20 kton liquid scintillator (LS), surrounded by 17,612 20-inch and 25,600 3-inch photomultiplier tubes (PMTs). The WP is equipped with 2,400 20-inch PMTs, served as a veto system for the cosmic ray muons. On the top of WP, the TT is used to measure the muons as well. The installation of all the PMTs and readout electronics will be completed by the end of 2024 and that afterwards, during detector filling, JUNO will start commissioning all the readout channels to test the full DAQ and data processing chains.

Figure 1 shows a schematic view of data processing in JUNO. When data taking starts, the event rate is about 1 kHz. The detector produces thousands channels of waveforms at a sampling rate of 1 GHz. To reduce the huge data volume, an additional system named Online Event Classification (OEC) is applied after trigger to reduce the event size according to the event types. Unlike the trigger system discards events, OEC retains all events. Approximately 60 MB/s of byte-stream RAW data, amounting 2 PB per year, is expected to be produced. The RAW data is transferred from the onsite to the IHEP data center via a dedicated network. The RAW data is preprocessed and converted to the ROOT-based RAW (RTRAW) data, using JUNO Event Data Model [4] and ROOT I/O[5]. Both types of data are replicated to the other data centers through the Distributed Computing Infrastructure (DCI) [6]. To minimize disk volume, the RAW data is archived to a tape library, while the RTRAW data is stored on a disk. After event reconstruction of the RTRAW data, the output is stored in ESD (Event Summary Data) format. The data processing involves several critical components, including data quality monitoring (DQM), keep-up reconstruction (KUP) and physics production (PP).

A series of JUNO Data Challenges (DC) have been proposed to evaluate and validate the complete data processing chain in advance. The JUNO DC serves not only to test the event reconstruction software, but also functions as a system test for the database, the Kafka-based [7] data pipeline, DQM, KUP and PP etc. The estimation of computing and storage capacities are validated through JUNO DC.

The JUNO DC-1 is focus on data processing within the central detector. About one week of inclusive datasets are produced and then reconstructed. Radioactivities, cosmic ray muons and neutrino events are simulated in advance. The rates of radioactivities and muons are set to the rates expected in the real data, while the rate of neutrino events is increased from 60 events per day to 4 Hz. By increasing the event rates of neutrinos, the reconstruction algorithms can be tested with higher statistics. In order to test the conditions database later, seven sets of time offsets are added to the channels during simulation.

As shown in Figure 2, there are two major steps in the workflow. The first step involves using simulation software to generate RTRAW files. The subsequent step involves reconstruction of these RTRAW files in the different systems and producing ESD files. Additionally, the conditions database is used for testing.

The JUNO simulation software [8, 9] incorporates Geant4-based [10, 11, 12] detector simulation and electronics simulation with OEC. As previously mentioned, the existing detector simulation datasets, which serve as inputs of the electronics simulation, are produced prior to the electronics simulation and are mixed according to the event rates. The electronics simulation generates pulses for all detector channels, which are subsequently digitized into waveforms after trigger. Unlike collider experiments, time correlation is curcial, which requires that the events could not be discarded. To reduce the data volume, OEC is employed to classify the event types using fast reconstruction results and to select a storage strategy for each event. Multiple events within a given time window are used for event type classification, a process known as time correlation analysis. As shown in Figure 3, the waveforms are reconstructed to time and charge (t/q) information in the OEC first. This t/q information is then calibrated, and the events are reconstructed with the calibrated t/q information. The OEC determines whether the waveform or t/q information should be stored in the final output file. For instance, waveforms are stored for neutrino events, while only the t/q information is stored for the other types. For the t/q stream, only the uncorrected data is stored into file, permitting offline correction with the conditions database at a later stage.

JUNO Hackathon was organized with the aim of migrating the reconstruction algorithms and conditions database from a serial version to a multi-threaded version. Any issues that encountered during testing have been addressed and resolved. Profiling tools, such as Intel VTune [13], were used to identify bottlenecks. Figure 4 illustrates one of the issues related to low CPU usage, as well as the CPU usage after optimization. A significant issue was the internal use of locks during the event processing, which resulted in multiple threads being blocked when attemping to access the locks.

Dedicated computing resources are employed in JUNO DC-1. Since the data taking has not yet started, the dedicated DQM cluster is used for the testing purposes. This cluster comprises 36 computing nodes with a total of 2304 cores, which are managed by the HTCondor system [14]. A total of 576 job slots are allocated, with each slot equipped with 4 cores and 15 GB memory.

The JUNO DC-1 is the first time to test the data processing chain, beginning with RTRAW, mimicking the real data processing. Multi-threaded algorithms have been developed and tested. The database has also been used in both the local cluster and DCI. All primary goals have been successfully met, and ongoing checks on the produced data are in progress.

However, certain aspects of DC-1 still require enhancement. For instance, only the reconstruction algorithms for the central detector have been tested. These work will be addressed in the upcoming rounds of the JUNO DC.

This work is supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDA10010900), National Natural Science Foundation of China (12375195), the Youth Innovation Promotion Association, CAS.