Learning to Identify Semi-Visible Jets

The microscopic nature of dark matter (DM) remains one of the most pressing open questions in modern physics Bertone and Hooper (2018); Bertone and Tait (2018), and a robust program of experimental searches for evidence of its interaction with the visible sector Aaltonen et al. (2012); collaboration (2013, 2012); Aad et al. (2013); Chatrchyan et al. (2012); Bai and Tait (2013); Petriello et al. (2008); Ferreira de Lima et al. (2017); Aaij et al. (2018) described by the Standard Model (SM). These experiments typically assume that DM is neutral, stable and couples weakly to SM particles; in collider settings this predicts detector signatures in which weakly-produced DM particles are invisible, evidenced only by the imbalance of momentum transverse to the beam. No evidence of DM interactions has been observed to date.

However, while these assumptions are reasonable, the lack of observation motivates the exploration of scenarios in which one or more of them are relaxed. Specifically, if DM contains complex strongly-coupled hidden sectors, such as appear in many Hidden-Valley models Strassler and Zurek (2007a); Han et al. (2008), it may lead to the production of stable or meta-stable dark particles within hadronic jets Cohen et al. (2015, 2017); Kar and Sinha (2021); Bernreuther et al. (2021). Depending on the portion of the jet which results in dark-sector hadrons, it may be only semi-visible to detectors, leading to a unique pattern of energy deposits, or jet substructure.

A robust literature exists for the identification of jet substructure Larkoski et al. (2014a, 2013); Kogler et al. (2019); Lu et al. (2022), with applications to boosted $W$-boson Baldi et al. (2016); Chen et al. (2020); collaboration (2019), Higgs boson Chung et al. (2021) and top-quark tagging collaboration (2019). Typically, observables are designed to discriminate between the energy deposition patterns left by jets which result from a single hard parton and the patterns left by jets which result from several collimated hard partons, as can be produced from the decay of a massive boosted particle. While these observables have some power when adapted to the task of identifying semi-visible jets Kar and Sinha (2021); Cohen et al. (2020), no observables have yet been specifically designed to be sensitive to the unique energy patterns of semi-visible nature of jets.

In parallel, the rapid development of machine learning to the analysis of jet energy depositions Larkoski et al. (2013); Komiske et al. (2018); Baldi et al. (2016) demonstrated that jet tagging strategies, including those for semi-visible jets, can be learned directly from lower-level jet constituents without the need to form physics-motivated high level observables Bernreuther et al. (2021); Dillon et al. (2022). Such learned models are naturally challenging to interpret, validate or quantify uncertainties, especially given the high-dimensional nature of their inputs. In the case of semi-visible jets, extra care must be taken when drawing conclusions from low-level details, many of which may depend on specific theoretical parameter choices as well as the approximations made during modeling of hadronization Cohen et al. (2020). However, techniques have been recently demonstrated Faucett et al. (2021); Collado et al. (2021a, b); Bradshaw et al. (2022) to translate the learned model into interpretable high-level observables, which can provide guidance regarding the nature and robustness of the information being used for classification.

In this paper, we present a study of machine learning models trained to distinguish semi-visible jets from QCD background jets using the patterns of their low-level jet constituents. We compare the performance of models which use low-level constituents to those which use the set of existing high-level observables to explore where the existing HL observables do and do not capture all of the relevant information. Where gaps exist, we attempt to translate Faucett et al. (2021) low-level networks into networks which use a small set of interpretable observables which replicate their decisions and performance. Interpretation of these observables can yield insight into the nature of the energy deposition inside semi-visible jets.

Following Refs. Cohen et al. (2015); Kar and Sinha (2021), we consider pair production of dark-sector quarks of several flavors $\left(\chi_{i}=\chi_{1,2}\right)$ via a messenger sector which features a $Z^{\prime}$ gauge boson in an $s$-channel process (Fig. 1(a)) or scalar mediator $\varphi$ in the $t$-channel process (Fig. 1(b)) that couples to both SM and DM sectors and leads to a dijet signature. The dark quarks produce QCD-like dark showers, involving many dark quarks and gluons which produce dark hadrons, some of which are stable or meta-stable and some of which decay into SM hadrons via an off-shell process.

The detector signature of the resulting semi-visible jet (SVJ) depends on the lifetime and stability of the dark hadrons, leading to several possibilities (see Fig. 2). Though the physics is complex and depends on the details of the dark sector structure, a description of the dark and SM hadrons produced by a DM model quark can be encapsulated in the quantity $r_{\textrm{inv}}$, the ratio of dark stable hadrons to all hadrons in the jet:

An invisible fraction of $r_{\textrm{inv}}=0$ corresponds to a dark quark which produces a jet consisting of only visible hadrons, as depicted in the Rapid Decay example given in Fig. 2(a). Alternatively, an invisible fraction of $r_{\textrm{inv}}=1$ describes a stable dark jet (Fig. 2(b)), in which the dark quark hadronizes exclusively in the dark sector. For any intermediate value of $r_{\textrm{inv}}$, jets contain a visible and invisible fraction, leading to $\cancel{E}_{\textrm{T}}$ along the jet axis(Fig. 2(c)).

Samples of simulated events with semi-visible jets are generated using the modified Hidden Valley Strassler and Zurek (2007b) model described in  Cohen et al. (2015, 2017) for both an $s$-channel (Fig. 1(a)) and $t$-channel (Fig. 1(b)) process. Samples of simulated $pp\to Z^{\prime}\to\chi_{1}\overline{\chi}_{1}$ and $pp\to\varphi\to\chi_{1}\overline{\chi}_{1}$ events are produced in proton-proton collisions at a center-of-mass energy of $\sqrt{s}=13\,\text{TeV}$ in MadGraph5 Alwall et al. (2011) (v2.6.7) with xqcut=100 and the NNPDF2.3 LO PDF set Skands et al. (2014). The mediator mass is set to $1.5\,\textrm{TeV}$ and the dark quark mass to $M_{\chi_{1}}=10\,\textrm{GeV}$. Up to two extra hard jets due to radiation are generated and MLM-matched Mangano et al. (2002), to facilitate comparison with existing studies. Invisible fraction, showering and hadronization are performed with Pythia8 v8.244 Sjöstrand et al. (2015). In particular, the following parameters are set: the dark confinement scale $\Lambda_{d}=5\,\textrm{GeV}$; the number of dark colors $N_{c}=2$; the number of dark flavors $N_{f}=1$; and the intermediate dark meson $\rho_{d}$ mass of $m_{\rho_{d}}=20\,\textrm{GeV}$ and the dark matter $\pi_{d}$ mass of $m_{d}=9.9\,\textrm{GeV}$. Distinct sets were generated for invisible fractions of $r_{\textrm{inv}}\in[0.0,0.3,0.6]$ via configurations of the branching ratio of decay process $\rho_{d}\to\pi_{d}\pi_{d}$. Detector simulation and reconstruction are conducted in Delphes v3.4.2 de Favereau et al. (2014) using the default ATLAS card. A sample of SM jets from a typical QCD process is generated from the $pp\to jj$ process. The same simulation chain as described above is applied to the SM jets.

Jets are built from calorimeter energy deposits and undergo the jet trimming procedure described in Ref. Krohn et al. (2010) with the anti-$k_{\textrm{T}}$ Cacciari et al. (2008) clustering algorithm from pyjet Cacciari et al. (2012), using a primary jet-radius parameter of $R=1.0$ and subjet clustering radius of $R_{\text{sub}}=0.2$ and $f_{\text{cut}}=0.05$. The threshold on $f_{\text{cut}}$ effectively removes constituents in subjets with $p_{\textrm{T}}$ that is less than 5% of the jet $p_{\textrm{T}}$. Leading jets are required to have $p_{\textrm{T}}\in[300,400]\,\text{GeV}$. For each event generated, the leading jet is selected and truth-matched to guarantee the presence of a dark quark within the region of $\Delta R<1$.

After all selection requirements, $2\times 10^{6}$ simulated jets remain with a 50/50 split between SVJ signal and QCD background. To avoid inducing biases from artifacts of the generation process, signal and background events are weighted such that the distributions in $p_{\textrm{T}}$ and $\eta$ are uniform; see  Fig. 7.

For both the low-level (LL) trimmed jet constituents and high-level jet substructure observables, a variety of networks and architectures are tested.

Due to their strong record in previous similar applications Faucett et al. (2021); Collado et al. (2021a, b); Bernreuther et al. (2021), a deep neural network using dense layers is trained on the high-level observables. We additionally consider XGBoostChen and Guestrin (2016), which has shown strong performance in training high-level classifiers with jet substructure Bourilkov (2019); Cornell et al. (2021), as well as LightGBMKe et al. (2017) which has demonstrated power in class separation on high-level features. LightGBM has the strongest classification performance among these networks which use high-level features; see Tab. 1.

In the case of classifiers which use low-level constituents, convolutional neural networks on jet images are considered Faucett et al. (2021); Collado et al. (2021a, b); Baldi et al. (2016); Cogan et al. (2015); de Oliveira et al. (2016). Given the specific task of classifying jet substructure observables and their use for a similar task in Ref. Collado et al. (2021b), Energy Flow Networks (EFN) and Particle Flow Networks (PFN) are also applied Komiske et al. (2019), and found to significantly out-perform convolutional networks, with the PFN emerging as the most powerful network; see Table 2.

Receiver operating characteristic (ROC) curves for both the PFN and LightGBM high-level models are given in Fig. 3. Additional details for the network training and hyperparameter selections are provided in App. C.

The studies above reveal that models which use low-level constituents as inputs provide the overall best classification performance. However, our objective is not merely to find the classifier with the optimal statistical performance with difficult-to-assess systematic uncertainties. Rather, we seek to understand the underlying physics used by the PFN and to translate this information into one or more meaningful physical features, allowing us to extract insight into the physical processes involved and assign reasonable systematic uncertainties. In this section, we search for these additional high-level observables among the Energy Flow Polynomials Komiske et al. (2018) (EFP), which form a complete basis for jet observables.

For each of the four scenarios in which a gap is observed between the AUC performance of the HL LightGBM model and the PFN, a guided search is performed to identify an EFP observable which mimics the decisions of the PFN.

The search results, shown in Table 4, identify in each of the four cases an EFP with $d\leq 3$ which boosts the classification performance of the LightGBM classifier when trained with the original 15 HL features as well as the identified EFP observable. In addition, the decision similarity (ADO) between the new HL model and the PFN is increased. Scans for a second EFP do not identify additional observables which significantly increase performance or similarity. A guided search was also performed with an identical set of $\kappa$ and $\beta$ parameters for EFPs with dimension $d\leq 5$ but no improvements were seen in the case of higher dimensional graph structure. While the performance and similarity gaps have been reduced, they have not quite been erased.

Given the persistent gap between the performance of the LL networks and the HL models augmented by EFP observables, we consider whether the EFP space lacks the needed observables, or whether the guided search is failing to identify it. We examine this question by taking a more comprehensive look at the space of EFPs we consider. Similar to the technique described in Ref. Faucett et al. (2021), we perform a greedy search in the same EFP space studied in the guided iteration approach explored above. In a greedy search, we explicitly train a new model for each candidate EFP, combining the EFP with the existing 15 HL features. Note that this is significantly more computationally intensive than evaluation of the ADO, as done in the guided search, and seeks to maximize AUC rather than to align decision ordering with the PFN. The candidate EFP which produces the best-performing model is kept as the 16th HL observable (Pass 1), and the process is repeated in search of a 17th (Pass 2), until a plateau in performance is observed.

The results of the greedy search across all choices of $r_{\textrm{inv}}$ in $s$-channel and $t$-channel scenarios where a gap between HL and LL exists are given in Table 5. Similar levels of performance are achieved as in the guided search, to within statistical uncertainties. In all cases, the HL and LL gap persists. Results are given for the IRC-unsafe selections with dimension $d\leq 3$. A similar greedy search was also performed on the IRC-unsafe $d\leq 5$ EFP set with no performance differences observed.

The greedy search selects similar EFP graphs as the guided search, with the exception of the 4-node graph selected in pass 2 for the $r_{\textrm{inv}}=0.6$, $t$-chanel scenario. No IRC-safe graphs ($\kappa=1$) are selected and we again see frequent sensitivity to low $p_{\textrm{T}}$ parameters (i.e. $\kappa=-2,-1,\tfrac{1}{2}$) and a variety of both narrow and wide angle features (i.e. $\beta=\tfrac{1}{10},2,4$). A repeat of the greedy search with only IRC-safe observables achieves no performance improvements over the original HL features, suggesting that missing information may be strongly tied to IRC-unsafe representations of the information in the jet constituents.

The results of both the guided search and the greedy search strongly suggest that the full performance gap which persists between the HL and LL representation of the jet contents cannot be compactly expressed in terms of a small number of EFP observables in the set that have been considered. This raises the question of what feature of the LL constituents can be credited with this performance improvement and why that information does not translate compactly to our EFP observables. The clues from the guided and greedy search point to sensitivity to low-$p_{\textrm{T}}$ constituents, as can be generated from soft radiation. Figure 6 shows the distribution of constituent $p_{\textrm{T}}$ relative to the jet $p_{\textrm{T}}$ for the six scenarios, in which SVJs appear to have more constituents at low relative $p_{\textrm{T}}$. Recall that jet constituents are trimmed if the subjet they belong to has a $p_{\textrm{T}}$ below a threshold of 5% of the $p_{\textrm{T}}$ of the jet, corresponding to $f_{\textrm{cut}}=0.05$.

To consider whether the distinguishing information is contained in these low-$p_{\textrm{T}}$ constituents, we explore a broader range of thresholds, both lowering it to 0% and raising it to 10% and 15%. The HL features are re-evaluated on the newly trimmed constituents, and used as inputs to a new LightGBM model, whose performance is compared to a PFN trained on the trimmed constituents. Results for networks with varying $f_{\textrm{cut}}$ thresholds are shown in Table 6.

In each case, raising the $f_{\textrm{cut}}$ threshold decreases the classification performance, as might be expected due to the removal of low-$p_{\textrm{T}}$ information. Perhaps more interesting is the variation in the gap between the performance of the HL LightGBM model and the PFN operating on low-level constituents. In nearly every case, the gap grows as more low-$p_{\textrm{T}}$ information is included, supporting the hypothesis that this is the origin of most of the information missing from the HL models. The details of the low-$p_{\textrm{T}}$ constituents are likely to be very sensitive to modeling uncertainties and subject to concerns about infrared and collinear safety.

However, even in for the most aggressive value of $f_{\textrm{cut}}=0.15$, a persistent gap of $\Delta$AUC=0.021 remains in the $s$-channel, $r_{\textrm{inv}}=0.6$ scenario, which cannot be explained by low-$p_{\textrm{T}}$ constituents. We therefore examine this in more detail.

First, we consider the EFPs selected in the study above where $f_{\textrm{cut}}=0.05$, including EFPs selected by the guided search (Table 4) and the greedy search (Table 5). For all combinations of HL and identified EFPs, no performance gain is seen. Next, we perform a fresh guided search on models trained from the $f_{\textrm{cut}}=0.15$ constituents. In contrast to previous cases, no large improvement in training performance is obtained on the first selected EFP. After 200 iterations, the gap is reduced to $\Delta$AUC=0.010 with the addition of 200 EFPs. We note that this far exceeds the mean number of constituents of these trimmed jets, 60. We conclude that there there does not appear to be a compact representation of the remaining information in the EFP space we have explored.

We have analyzed the classification performance of models trained to distinguish background jets from semi-visible jets using the low-level jet constituents, and found them to offer stronger performance than models which rely on high-level quantities motivated by other processes, mostly those involving collimated hadronic decays of massive objects.

While models operating on the existing suite of HL quantities nearly match the performance of those using LL information, a significant gap exists which suggests that relevant information remains to be captured, perhaps in new high-level features. To our knowledge, this is the first study to compare the performance of constituent-based and high-level semi-visible-jet taggers, and to identify the existence of relevant information uncaptured by existing high-level features. Jets due to semi-visible decays are markedly different in energy distribution than those from massive objects, so it is not unexpected that existing features may not completely summarize the useful information.

Using a guided strategy, we identify a small set of new useful features from the space of energy-flow polynomials, but these do not succeed in completely closing the performance gap. In most cases, the remaining gap seems to be due to information contained in very low-$p_{\textrm{T}}$ constituents, which is likely to be sensitive to modeling of showering and hadronization and may not be infrared and collinear safe. This highlights the importance of interpretation and validation of information used by constituent-based taggers. As demonstrated by Ref. Cohen et al. (2020), the specific pattern of energy deposition may depend sensitively on both the parameters of the theoretical model as well as the settings chosen for the hadronization model. It is therefore vital that the information being analyzed be interpreted before being applied to analysis of collider data.

In one case studied here, a gap persist between low- and high-level-based models even when low-$p_{\textrm{T}}$ constituents are aggressively trimmed, suggesting the possibility that a new high-level feature could be crafted to capture this useful high-$p_{\textrm{T}}$ information. Our efforts to capture this information with the simpler energy-flow polynomials was not successful, suggesting that more complex high-$p_{\textrm{T}}$ observables may exist which provide useful discrimination between QCD and semi-visible jets. The studies presented here can inform and guide theoretical work to construct such observables specifically tailored to this category of jets. Whether such observables can be efficiently represented using alternative basis sets of observables, and whether they are robust to Shimmin et al. (2017) or explicitly dependent on Ghosh et al. (2021) uncertainties while providing power over large regions of theoretical parameter space is an important avenue for future work.

The authors thank Po-Jen Cheng for his assistance in sample validation. This material is based upon research supported by the Chateaubriand Fellowship of the Office for Science & Technology of the Embassy of France in the United States. T. Faucett and D. Whiteson are supported by the U.S. Department of Energy (DOE), Office of Science under Grant No. DE-SC0009920. S.-C. Hsu is supported by the National Science Foundation under Grant No. 2110963.