A fast and flexible machine learning approach
to data quality monitoring

Modern high-energy physics experiments consist of complex detectors where hundreds of millions of sensors are read out as frequently as every few nanoseconds. The electrical signals are amplified, processed and combined before the trigger selection and the final storage. At each of these steps some errors can occur and invalidate the whole process. Monitoring systems are deployed to assess the quality of the data and keep the flow under control throughout all the stages of the acquisition. Data quality monitoring (DQM) is a challenging task from a statistical point of view due to its high intrinsic dimensionality and the high level of human supervision required. The unforeseen events incoming during an experimental run can be several. Some of them can be anticipated and recognised while they occur, others cannot. The detection of well known dysfunctions could be nonetheless missed due to the huge amount of channels that should be simultaneously monitored. Developing flexible highly automatized techniques to supervise multiple variables at once is thus fundamental to reduce the risk of failures [1, 2, 3, 4]. This work proposes the use of a recent machine learning approach to compare collected data with a sample of reference events that depicts the correct detector readings. This reference sample can be, for instance, a set of measurements in a controlled scenario or simulated events. The basic idea is to perform a hypothesis test powered by a fast and flexible machine learning (ML) algorithm. In practice, we leverage the ability of binary classifiers to implicitly model the underlying data-generating distributions and estimate the likelihood ratio test statistics. This is then used to assess whether the hypothesis underlying the observed data (alternative hypothesis) agrees with the assumption of normal behavior (null hypothesis). If a set of measurements significantly deviates from the reference sample, the learned likelihood ratio can be used to characterize the anomalies in the feature space. To increase the model’s sensitivity, we take a large reference sample and reduce its statistical fluctuations. This work is inspired by the ML model introduced in Ref. [5] in the context of model-independent new physics searches in high-energy physics. In that work, the authors propose an algorithm based on kernel methods with clear advantages compared to similar neural network implementations in terms of training time while obtaining similar performance. This feature makes the kernel approach ideal for DQM, where fast training is essential for online analyses.
The paper is organized as follows. In the next section we introduce the experimental setup and the algorithm input variables. These include a reference data set collected under standard conditions and smaller samples with anomalous controlled behaviors. The ML model and our core strategy are then described in Section 3, whereas an overview of the results is given in Section 4. Finally, the last section is devoted to conclusions and further developments.

Each detector channel provides an (absolute) time stamp once a signal is recorded. Since we are interested in muons crossing the detector, we group the hits corresponding to such events and we reconstruct the muon track out of the observed hits pattern.
The performances of the muons track reconstruction deteriorate as soon as the detector conditions are anomalous. The following list of relevant quantities can be exploited to assess the quality of the recorded data:
$\bullet$ Drift times ($t_{i}$): the four drift times associated with the muon track.
$\bullet$ Slope ($\phi$): the angle formed by the muon track in the plane orthogonal to the anodic wires (measurement plane) with respect to the vertical axis.
$\bullet$ Number of hits ($n_{hits}$): the number of hits recorded in a time window of one second around the muon crossing time.
To test the performances of the DQM approach presented in this work, we reproduce two of the typical failures that occur during detector operations. In particular, we consider the eventuality of the cathodic strips voltage reduced to $25\%$ of their nominal value (i.e., to $-300$ V) and that of the front-end thresholds reduced to $75$, $50$ and $25$ mV, respectively 75%, 50% and 25% of the nominal value. A dedicated data collection campaign has been launched.²²2https://doi.org/10.5281/zenodo.7128223 As a result, a set of data samples has been collected, one consisting of $3\times 10^{5}$ events being the reference and the other four of approximately $10^{4}$ events, each corresponding to one of the anomalous conditions mentioned above.

Model design. To estimate the likelihood ratio, we train a binary classifier on a reference sample $S_{0}=\{x_{i}\}_{i=1}^{n_{0}}$ and on a set of measurements $S_{1}=\{x_{i}\}_{i=1}^{n_{1}}$ using a weighted logistic loss

with $n=n_{0}+n_{1}$ and where $y$ is the class label that takes the zero value for $S_{0}$ and one for $S_{1}$. The model parameters $w$ are going to be learned from the data. The weights $n/n_{y}$ are introduced to compensate for the data imbalance $(n_{0}\gg n_{1})$ while keeping the statistical advantage of a large reference sample. The learned classifier $f_{\hat{w}}$ is then obtained by minimizing the average loss over the training set. By a standard computation, the learned function can be shown to approximate the likelihood ratio $f_{\hat{w}}\approx f^{*}(x)=\log\frac{p(x|1)}{p(x|0)}$ (see, for instance, appendix A of Ref. [5]). The log-likelihood ratio test statistics can then be easily evaluated on the measurements as

In this work, we consider a learning algorithm based on kernel methods [8] of the form $f_{w}(x)=\sum_{i=1}^{n}w_{i}k_{\sigma}(x,x_{i})$, where $k_{\sigma}(x,x_{i})=\exp{(-\|x-x^{\prime}\|^{2}/2\sigma^{2})}$ is the Gaussian kernel function and the width $\sigma$ is a hyper-parameter. Kernel methods are universal in the large sample limit [9, 10] but they scale poorly with data size. For this reason, we consider here Falkon [11], a modern library which makes use of a number of algorithmic ideas, such as column sub-sampling, and takes full advantage of GPU hardware to extend the use of kernel methods to large scale scenarios with fast training and strong statistical guarantees.
Model selection and training. The training is performed by considering the empirical risk based on Eq. (1) with an added L2 regularization term [12] weighted by a hyper-parameter $\lambda$. While the machine learning model is at its core a binary classifier, we do not monitor typical classification metrics such as binary accuracy or the AUC, and our metric of interest is the likelihood ratio test statistics given in Eq. (2). Hyper-parameters are tuned following the prescription given in Ref. [5], which includes a mix of heuristics, statistical optimality and efficiency requirements.
Anomaly detection. The analysis strategy for the detection of anomalies in the measurements is composed of three steps:
$\bullet$ The distribution of the test statistics under the null (reference) hypothesis is estimated by training the model multiple times ($\mathcal{O}(100)$) on the reference sample $S_{0}$ and on different data samples $S_{1}$ also following the reference distribution, e.g., anomaly free.
$\bullet$ A single training is performed on the reference $S_{0}$ and the actual measurements $S_{1}$ to estimate the test statistic $t_{\hat{w}}(S_{1})$ of the measurements.
$\bullet$ The $p$-value corresponding to $t(S_{1})$ is computed with respect to the test statistic distribution under the reference hypothesis, learned in the first step. If a statistically significant deviation from the reference data is found (for instance with respect to a threshold p-value $p^{*}$), the nature of the discrepancy can be analyzed in feature space by re-weighting the reference sample with the learned density ratio $\exp(f_{\hat{w}})$, as described in the following section.
If a database of observations characterizing known anomalous behaviors is available, different strategies can be employed to further characterize anomalous readings. We tested different ML approaches for this task, among which supervised binary classifiers, multi-class classifiers and parametrized regression models. These models are trained in a fully supervised fashion to separate reference data from specific types of anomalies. This step is not discussed in this manuscript and will be presented in details elsewhere.

In this work we propose a fast machine learning approach for data quality monitoring. The algorithm compares collected measurements with a reference dataset describing the standard detector readout performing a multidimensional likelihood-ratio hypothesis test. The significance of an eventual discrepancy in the data is quantified via a frequentist p-value. Some solutions for the location and characterization of the anomaly in the space of the input features are presented. Those could be relevant in order to design an automatic tool for the system control that allows to reset the apparatus to normal condition acting on the specific source of malfunctioning. The model is fast and can be used for quasi-online monitoring. The performances of the model can be tuned varying the size of the experimental data to be tested, the algorithm parameters and the computational resources. The experimental setup used to perform this study is simple but representative of the type of detectors installed in LHC experiments. A dedicated study tailoring more specific use cases will be performed in order to fit the requirements for the online deployment.

M.L and M.R. acknowledge the financial support of the European Research Council (grant SLING 819789). G.G. is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement no 772369). A.W. acknowledges support from the PRIN grant 2017FMJFMW.