Robust and Provably Monotonic Networks

The sensor arrays of the LHC experiments produce over 100 TB/s of data, more than a zettabyte per year. After drastic data-reduction performed by custom-built read-out electronics, the annual data volumes are still $O(100)$ exabytes, which cannot be stored indefinitely. Therefore, each experiment processes the data in real-time and decides whether each proton-proton collision event should remain persistent or be discarded permanently, referred to as triggering in particle physics. Trigger classification algorithms must be designed to minimize the impact of effects like experimental instabilities that occur during data taking—and deficiencies in simulated training samples. (If we knew all of the physics required to produce perfect training samples, there would be no point in performing the experiment.) The need for increasingly complex discriminators for the LHCb trigger system LHCb-DP-2012-004 ; LHCb-DP-2019-001 calls for the use of expressive models which are both robust and interpretable. Here we present an architecture based on a novel weight normalization technique that achieves both of these requirements.

The goal is to develop a neural network architecture representing a scalar-valued function

that is provably monotonic in any subset of inputs and whose gradient (with respect to its inputs) has a constrained magnitude in any particular direction. In an experimental setting, this latter property is a measure of robustness to small changes in experimental conditions or to small deficiencies in the training samples.

Constraints with respect to a particular $L_{p}$ metric will be denoted as Lip^p. We start with a model $g(\bm{x})$ that is Lip¹ with Lipschitz constant $\lambda$ if $\forall\,\bm{x},\bm{y}\in\mathbb{R}^{n}$ (we show below how to train such a model)

The choice of 1-norm is crucial because it allows a well defined maximum directional derivative for each input regardless of the gradient direction. This has the convenient side effect that we can tune the robustness requirement for each input individually. Note that rescaling the inputs $x_{i}$ allows for $\lambda$ directional dependence.

Before applying our new architecture to real-time data-processing at the LHC, we first demonstrate that it behaves as expected on some simple toy problems.

The architecture presented in Sec. 2 has been developed with a specific purpose in mind: The classification of the decays of heavy-flavor particles produced at the Large Hadron Collider, which are bound states that contain a beauty or charm quark that live long enough to travel an observable distance $\mathcal{O}(1\,{\rm cm})$ before decaying. The dataset used here is built from simulated proton-proton ($pp$) collisions in the LHCb detector. Charged particles that survive long enough to traverse the entire detector before decaying are reconstructed and combined pairwise into decay-vertex (DV) candidates.

The task concerns discriminating between DV candidates corresponding to the decays of heavy-flavor particles versus all other sources of DVs. The signatures of a heavy-flavor DV are substantial separation from the $pp$ collision point, due to the relatively long heavy-flavor particle lifetimes, and sizable transverse momenta, $p_{\rm T}$, of the component particles, due to the large heavy-flavor particle masses. There are three main sources of background DVs. The first involves DVs formed from particles that originated directly from the $pp$ collision, but where the location of the DV is measured to have non-zero displacement due to resolution effects. These DVs will typically have small displacements and small $p_{\rm T}$. The second source of background DVs arises due to particles produced in the $pp$ collision interacting with the LHCb detector material, creating new particles at a point in space far from the $pp$ collision point. Such DVs will have even larger displacement than the signal, but again have smaller $p_{\rm T}$. The third source involves at least one fake particle, i.e. a particle inferred from detector information that did not actually exist in the event. Since the simplest path through the detector (a straight line) corresponds to the highest possible momentum, DVs involving fake particles can have large $p_{\rm T}$.

In the first decision-making stage of the LHCb trigger, a pre-selection is applied to reject most background DVs, followed by a classifier based on the following four DV features: $\sum p_{\rm T}$, the scalar sum of the $p_{\rm T}$ of the two particles that formed the DV; ${\rm min}[\chi^{2}_{\rm IP}]$, the smaller of the two increases observed when attempting to instead include each component particle into the $pp$-collision vertex fit, which is large when the DV is far from the $pp$ collision point; the quality of the DV vertex fit; and the spatial distance between the DV and $pp$-collision locations, relative to their resolutions. N.b., the threshold required on the classifier response when run in real time during data taking is fixed by the maximum output bandwidth allowed from the first trigger stage.

Unfortunately, extremely large values of both displacement and momentum are more common for backgrounds than for heavy-flavor signals. For the former, this is easily visualized by considering a simplified problem using only the two most-powerful inputs, $\sum p_{\rm T}$ and $\chi^{2}_{\rm IP}$. Figure 4 (left) shows that an unconstrained neural network learns to reject DVs with increasing larger displacements, corresponding to the lower right corner in the figure. Figure 5 (left) shows that this leads to a dependence of the signal efficiency on the lifetime of the decaying heavy-flavor particle. Larger lifetimes are disfavored since few heavy-flavor particles live more than $\mathcal{O}(10\,{\rm ps})$. While rejecting DVs with the largest displacements does maximize the integrated signal efficiency in the training sample, this is undesirable because in many cases studying the longest-lived heavy-flavor particles is of more interest than simply collecting the largest decay sample integrated over lifetime (see, e.g., HFLAV:2022pwe ). Furthermore, many proposed explanations of dark matter and other types of new physics predict the existence of new particles with similar properties to heavy-flavor particles, but with longer lifetimes RF6 ; Graham:2021ggy . This classifier would reject these particles because it is unaware of our inductive bias that highly displaced DVs are worth selecting in the trigger and studying in more detail later.

Since the LHCb community is generally interested in studying highly displaced DVs for many physics reasons, we want to ensure that a larger displacement corresponds to a more signal-like response. The same goes for DVs with higher $\sum p_{\rm T}$. Enforcing a monotonic response in both features is thus a desirable property, especially because it also ensures the desired behaviour for data points that are outside the boundaries of the training data. Multiple methods to enforce monotonic behavior in BDTs already exist auguste2020better , and Figs. 4 (middle) and 5 (middle) show that this approach works here. However, the jagged decision boundary can cause problems, e.g., when measuring the heavy-flavor $p_{\rm T}$ spectrum. Specifically, the jagged BDT decision boundary can lead to sharp changes in the selection efficiency. If there is not perfect alignment of where these changes occur with where the interval boundaries of the spectrum are defined, then correcting for the efficiency can be challenging. Figure 4 (right) shows that our novel approach, outlined in Sec. 2, successfully produces a smooth and monotonic response, and Fig. 5 (right) shows that this provides the monotonic lifetime dependence we wanted in the efficiency.

Not only does our architecture guarantee a monotonic response in whatever features the analyst wants, it is guaranteed to be robust with respect to small changes to the inputs as governed by the constrained Lipschitz constant. Because calibration and resolution effects play a role in obtaining the features during detector operation, robustness is a necessary requirement for any classification performed online. Downstream analyses of these data depend on their stability. Figure 6 shows that the cost in terms of signal efficiency loss of enforcing monotonicity and robustness is small, even under the unrealistic assumption that the training data were, in fact, perfect. Therefore, the actual cost is likely negligible, while the benefits of the guarantees provided is hard to quantify but immediately obvious to the LHCb collaboration. Our algorithm runs in the LHCb trigger software stack and has been chosen to replace Refs. BBDT ; LHCb-PROC-2015-018 as the primary trigger-selection algorithm used by LHCb in Run 3. Due to its guaranteed robustness—and excellent expressiveness even for small networks—this architecture is being explored for other uses within the LHCb trigger system for Run 3, since robustness and monotonicity are ubiquitous inductive biases in experimental particle physics.

We are working on improving the architecture as follows. First, common initialization techniques are not optimal for weight-normed networks arpit2019initialize . Simple modifications to weight initialization could likely improve convergence significantly. Second, it appears from empirical investigation that the networks described in Eq. (6) with GroupSort activation could be universal approximators, as we have yet to find a function that could not be approximated well enough with a deep enough network. A proof for universality is still required and could be developed in the future. Note that universal approximation is indeed proven for a similar architecture that only differs slightly in the normalization scheme, see sorting2019 . Neither of these limitations has any practical impact on the example applications discussed in the previous sections.

In many scenarios, Lipschitz-constrained architectures are considered inferior to unconstrained architectures because of their inability to offer competitive performance on standard benchmarks. This low performance is partly due to the fact that standard losses (such as cross-entropy) are not an adequate proxy of the metric of interest (accuracy) for the Lipschitz-constrained models. At a fundamental level, for any maximally accurate unconstrained classifier $f(x)$ with Lipschitz constant $\lambda$, there exists a Lipschitz 1 classifier that replicates the former’s decision boundary, namely, $f(x)/\lambda$. In the following, we will demonstrate a basic toy setting in which a maximally accurate Lipschitz classifier exists but cannot be obtained using standard losses.

To understand the effect of the choice of objective function, we train a Lipschitz-constrained model to separate the two-moons dataset as shown in Fig. 7. This example is special in that the two samples do not overlap and can be completely separated by a Lipschitz-bounded function; however, that function cannot return the true label values for any data points due to the Lipschitz bound. Therefore, a loss function that penalizes any difference of the model output to the true label now faces a misalignment of the optimization target and the actual goal: While the classification goal is to have high accuracy, i.e. correct output sign, the optimization target is to minimize deviations of the output from the true label. This misalignment becomes irrelevant for a function with unbounded Lipschitz constant. We will show below that for examples such as this there is an important dependence on the objective used and its hyperparameters.

First, we note that losses with exponential tails (in the sense that they require large weights to reach zero) are in general not suitable for maximizing accuracy. In practice, this can be remedied in cross-entropy by increasing the temperature. Note that cross-entropy with temperature $\tau$ is defined as

where $\mathcal{L}^{\mathrm{BCE}}(y,\hat{y})$ is the usual binary cross-entropy loss on targets $y$ and predictions $\hat{y}$. Following PyTorch conventions, $\hat{y}$ are logits which will be normalized before computing the negative log-likelihood.

An accurate classification boundary comes at the expense of reduced margins when classes have small separation. A maximally robust accurate classifier will, however, have optimal margins if trained using the appropriate objective. In the case of separable data (i.e. when classes have disjoint support), the maximally robust accurate Lipschitz classifier is the signed-distance function (SDF) manyfaces defined, in the binary case as

where $C^{+1}$ and $C^{-1}$ are the sets of points for which $x$ has label $+1$ and $-1$, respectively, and $B$ is the boundary between classes defined as $B\equiv\left\{x\,\big{|}\,d(C^{+1},x)=d(C^{-1},x)\right\}$. For a closed set $S$, the distance to $x$ is defined as $d(S,x)=\min_{y\in S}d(y,x)$.

A naive objective minimized by the SDF is the hinge loss with margin given by $d(B,x)$. Because we do not have access to the true decision boundary a priori, as a proxy, we use the following objective:

where $\delta(x|y)=\frac{d(C^{-y},x)}{2}$ and $\mathcal{L}^{\mathrm{Hinge}}$ is defined as

While this objective produces the highest margins for an accurate classifier, as depicted in Fig. 7, it may encounter scalability issues when applied to higher-dimensional problems due to the unavoidable spareness of the training data. There are many possible alternative approaches that could resolve this issue, though this remains an open problem. For lower-dimensional problems with overlapping datasets—as studied in the various examples above and the most common scenario in scientific applications—this non-optimal loss issue does not appear to be relevant.

Another factor that restricts the perceived expressiveness of Lipschitz architectures is the lack of access to standard techniques that improve convergence in unconstrained networks. For example, batch norm cannot be directly used with Lipschitz architectures. If the variance is too small, it may exceed the Lipschitz bound, and if it is too large, it can reduce the effective Lipschitz constant substantially.

The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We developed a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. Our method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence.

Our implementation of Lipschitz constrained networks is minimally constraining compared to other weight-normed models. This allows the underlying architecture to be more expressive and easier to train while maintaining explicit robustness guarantees. We showed how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor decays in the LHCb trigger system. Furthermore, thanks to the expressive capacity of the architecture, we were able to shrink the number of model parameters to meet the memory and latency requirements of the LHCb trigger, which allows for faster event selection. This translates to higher sensitivity to the elusive physics phenomena we aim to observe. Our algorithm has been adopted for use as the primary data-selection algorithm in the LHCb trigger in the current LHC data-taking period known as Run 3.

Given that the desire for robustness and interpretability and benevolent out-of-distribution behavior is ubiquitous when performing experiments, we expect that our architecture could have wide-ranging applications in science. In addition, our architecture could also be used in various applications in which robustness is required such as safety-critical environments and those which need protection against adversarial attacks. Monotonicity is a desirable property in various applications where fairness and safety are a concern. There are many scenarios in which models which are not monotonic are unacceptable. For example, in Ref. ICLR-Mono we showed that our algorithm achieves state-of-the-art performance on benchmarks in medicine, finance, and other applications with monotonic inductive biases. In addition, in Ref. NEEMo we presented a new and interesting direction for the architecture developed here: Estimation of the Wasserstein metric (Earth Mover’s Distance) in optimal transport by employing the Kantorovich-Rubinstein duality to enable its use in geometric fitting applications. Therefore, we expect that our algorithm will have far-reaching impact well beyond experimental physics.

The authors would like to thank the LHCb computing and simulation teams for their support and for producing the simulated LHCb samples used to benchmark the performance of RTA software. This work was supported by NSF grants PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/) and OAC-2004645.