Estimation of Photometric Redshifts. II. Identification of Out-of-Distribution Data with Neural Networks

In $step_{sup}$ using only ID data, we bin redshift ranges and classify samples into the redshift bins instead of conducting typical regression of redshifts as we name the model the multiple-bin regression with the neural network used in Paper I. We adopt 64 independent bins with a constant width following the performance result in Paper I. Then, the model estimates probabilities that the photometric redshifts of input samples lie in each bin. Using the model output probabilities, we may obtain point-estimation of photometric redshift $z_{phot}$ by averaging the central redshift values of the bins with the output probabilities.

As a training loss of $step_{sup}$, we use anchor loss $L_{ANCH}$. The loss is designed to measure the difference between two given probability distributions considering prediction difficulties due to various reasons, e.g., the lack of data or the similarities among samples belonging to different classes, i.e., redshift bins. The anchor loss assigns the difficult samples for prediction high weights governed by a weighting parameter $\gamma$. For an in-depth definition of the loss, refer to Ryou et al. (2019). Since we train two networks with the same architecture for the dual tasks, the training loss of $step_{sup}$, which should be minimized, is set as follow:

where $p_{1}$ and $p_{2}$ are the model output probability distributions coming from $F_{1}$ and $F_{2}$, respectively, ${\bf z}$ is a redshift bin vector, and ${\bf x_{ID}}$ is an ID input vector.

$step_{unsup}$ uses the UL data, presumably containing both ID and OOD samples, in training models and employs the disparity of the results from two networks to evaluate the OOD score of the samples. The two networks are trained to maximize disagreement between them in the photometric redshift estimation for the potential OOD samples, pushing the OOD samples outside the manifold of the ID samples. The two networks, which are identically structured and trained on the same ID data via supervised learning, might have different results on OOD samples because of stochastic effects, although the networks are not optimized for OOD detection. Then, by defining discrepancy loss $L_{DCP}$ to measure the disagreement and maximizing it, the two networks can be forced to produce divergent results for the potential OOD samples, which generally correspond to the samples in the class (i.e., redshift bin) boundaries and/or are misclassified (i.e., assigned to an incorrect redshift bin). The $L_{DCP}$ defined for this purpose is as follows:

where $H(\cdot)$ is the entropy for given probability distributions. As the networks are trained to maximize the discrepancy $L_{DCP}$, the entropies from $F_{1}$ and $F_{2}$ output increase and decrease, respectively. The network $F_{1}$ outputs a flat probability distribution (i.e., high entropy; high uncertainty), and the network $F_{2}$ generates a peaked one (i.e., low entropy; low uncertainty). Notably, the $L_{DCP}$ is available in the unsupervised approach using the UL samples since it can be computed without sample labels (i.e., redshifts). The potential ID samples make it difficult for the two networks produce discrepant inference results for photometric redshifts since the inference for the ID samples has a strongly peaked distribution in both networks $F_{1}$ and $F_{2}$ as explained below. Meanwhile, the potentially OOD samples can increase $L_{DCP}$ as the two networks become easily divergent in the training process.

To prevent divergent results on the ID samples, we employ the sum of $L_{sup}$ and $L_{DCP}$ as the training loss of $step_{unsup}$. Although the unsupervised approach using the $L_{DCP}$ diverge the model outputs on the probable OOD examples included in the UL dataset, it can also cause discrepant results on the ID samples since the UL data includes ID and OOD samples. Hence, defining the training loss of $step_{unsup}$ as the sum help the two network outputs on the ID samples stay as similar as possible since minimizing $L_{sup}$ encourages the network outputs unimodal probability distributions with a peak at the true class on ID samples. Namely, $step_{unsup}$ diverges model outputs for OOD samples more than ID ones because OOD samples are out of support of the $L_{sup}$. The training loss of $step_{unsup}$, which should be minimized in training, is defined as follows:

where ${\bf x_{UL}}$ is an UL input vector and $m$ is a margin. Notably, $L_{DCP}$ is maximized as $L_{unsup}$ is minimized due to the negative sign. The use of the margin $m$ is required to prevent overfitting of the networks by replacing the second term of the unsupervised loss with zero if $L_{DCP}>m$. We use $m=4.3$ found as the best parameter in multiple trial runs.

Our usage of $L_{unsup}$ as the combination of $L_{sup}$ and $L_{DCP}$ can be understood as multi-task learning of both estimating photometric redshifts and evaluating OOD scores. Particularly, this multi-task learning model is a case of transferring the knowledge of supervised learning (i.e., the model with $L_{sup}$ in estimating photometric redshifts) to unsupervised learning (i.e., the model with $L_{unsup}$ in evaluating OOD scores) (Pentina & Lampert, 2017). The combined loss in our dual-task learning adopts the parameter $m$, which regularizes the contribution of $L_{DCP}$ to the total loss $L_{unsup}$. This is a simple strategy to find balance among multiple tasks as commonly used as weight parameters with multiple losses in multi-task learning (Sener & Koltun, 2018; Gong et al., 2019; Vandenhende et al., 2021).

We proceed with the three-stage training of the two networks. Each training stage comprises aforementioned $step_{sup}$ and $step_{unsup}$ as summarized in Figure 1. Training stage-1 is a pre-training of the two networks with the $step_{sup}$. The purpose of this stage is to obtain a stable performance of the networks in estimating photometric redshifts. In training stage-2, the two networks are trained for OOD detection using an iterative training procedure comprising one $step_{sup}$ and two $step_{unsup}$s. Although $L_{sup}$ included in $L_{unsup}$ helps the disparity between two network outputs on ID samples stay small, the disparity increases as the models are trained by the optimization using UL samples and $L_{DCP}$. Composing the stage-2 as the iterative training procedure incorporating $step_{sup}$ further precludes the disparity on the ID samples from increasing as $L_{sup}$ included in $L_{unsup}$ affects the total loss (see Equation 3). Therefore, as the training continues, the disagreement of the network outcomes for the probable OOD samples becomes larger than that for the ID samples because the OOD samples are outside the support of the $step_{sup}$, which is only applied to the ID galaxies. Then, the measure of disagreement (i.e., $L_{DCP}$) can be used as an OOD score to flag OOD candidates. At the inference level, the trained networks in training stage-2 are used to score the test samples as OOD.

In addition to training stage-1 and -2, we proceed to an additional supervised training stage, training stage-3, with $step_{sup}$ for photometric redshift estimation. The unsupervised training for OOD detection affects the model performance of the original task (i.e., photometric redshift estimation) because the model in training stage-2 is optimized for two losses described in Equation 3. In practice, training stage-3 might not be needed if the models trained in stage-1 are saved and used for the task of photometric redshift estimation. In this work, we train the two networks again using only $L_{sup}$ after training stage-2. Then, the outputs of the two stage-3 networks are combined by averaging them to estimate photometric redshifts.

Since our model performs the two tasks together, we need separate metrics to assess model performances on both tasks. As point-estimation metrics in estimating photometric redshifts with photometric redshifts as mean value with probabilistic inference by the trained model, we adopt the ones used in Paper I: bias, MAD, $\sigma$, $\sigma_{68}$, NMAD, and $R_{cat}$. For a detailed explanation of the metrics, refer to Paper I. In short, the bias is measured as the absolute mean of $\Delta z$ which is difference between photometric redshifts and true spectroscopic redshifts. Similarly, the MAD is the mean of the absolute difference $|\Delta z|$. $\sigma$ and $\sigma_{68}$ corresponds to the standard deviation of $\Delta z$ and the 68th percentile of $|\Delta z|$, respectively. NMAD is defined to be 1.4826 $\times$ median of $\Delta z$. $R_{cat}$ presents the fraction of data with $|\Delta z|>0.15$. Notably, the lower the point-estimation metrics are, the higher the performance is.

Before explaining OOD detection metrics, we first define four measures: true negative (TN), true positive (TP), false negative (FN), and false positive (FP). The Boolean values (i.e., true and false) in the terms indicate whether the predicted class of the sample is correct or incorrect. The negative and positive in the terms mean the true class in which the given sample is included. We, respectively, set ID and OOD as negative and positive since the model performs OOD detection as a task. Hence, for example, the number of true positive cases is the number of correctly classified OOD samples. The numbers of these four cases can be varied with respect to the adopted threshold of the OOD score. Notably, these metrics are used with the LOOD samples not in training the model but in validating the trained model.

We use the central value between the minimum and maximum $L_{DCP}$ as the threshold to compute the metrics. Note that this threshold is chosen simply because it can be easily estimated for model assessment. The following is a brief explanation of the detection metrics.

• •

  Accuracy: ratio of the correctly classified samples to the entire samples $\frac{{\it TP}+{\it TN}}{{\it TP}+{\it TN}+{\it FN}+{\it FP}}$,

• •

  Precision: ratio of the correctly classified positive samples to the entire positively classified samples $\frac{{\it TP}}{{\it TP}+{\it FP}}$,

• •

  True Positive Rate (TPR) or recall: fraction between the correctly predicted positive samples and the entire positive samples $\frac{{\it TP}}{{\it TP}+{\it FN}}$,

• •

  $F_{\beta}$: harmonic mean of precision and recall weighted by $\beta$ $\frac{(1+\beta^{2})*Precision*Recall}{\beta^{2}*Precision+Recall}$. The lesser and larger $\beta$ gives more weight to precision and recall, respectively. Typical values for $\beta$ is 0.5, 1.0, and 1.5. In this work, we set $\beta=1.5$ ($F_{1.5}$) to focus on more recall than precision, paying attention to the number of the correctly classified OOD samples among all OOD samples,

• •

  $AUC_{ROC}$: area under the receiver operating characteristic (ROC) curve. The ROC curve is a visualized measure of the detection performance for the binary classifier. In the curve, the TPR is plotted as a function of the false positive rate for different thresholds of the OOD score. False Positive Rate is a fraction between the incorrectly predicted negative samples and entire negative samples $\frac{{\it FP}}{{\it TN}+{\it FP}}$. The $AUC_{ROC}$ offers a comprehensive measure of the detection performance of the model.

• •

  $AUC_{PR}$: area under the precision-recall (PR) curve. Although $AUC_{ROC}$ offers a representative measure of binary detection performance generally, it may produce misleading results for imbalanced data because TPR (i.e., the vertical axis of the ROC curve) depends only on positive cases without considering negative cases. In the PR curve, precision is plotted as a function of recall. Because precision considers both positive and negative samples, $AUC_{PR}$ may be more informative for imbalanced data.

The higher values of the detection metrics indicate better detection performance of the model. Notably, these OOD detection metrics are evaluated with the labeled ID (i.e., galaxies) and OOD (i.e., stars and QSOs confirmed spectroscopically).

We find that the networks in training stage-3 outperform the networks in the stage-2 in terms of photometric redshift estimation. To verify the effectiveness of the additional stage training stage-3, we compare the quality of the photometric redshifts derived from the models in training stage-2 and -3. Table 4 shows the point-estimation metrics evaluated in the high and low entropy models (high and low entropy models, i.e., $F_{1}$ and $F_{2}$, respectively) in training stage-2 and the averaged model in stage-3. Except for the bias, the training stage-3 model outperforms others. It proves that unsupervised training of the networks in training stage-2 for OOD detection affects the quality of photometric redshifts as expected for multi-task learning.

The networks in training stage-2 hold noticeable peculiarities showing deteriorated photometric redshifts. As shown in Figure 2, the photometric redshifts from the high entropy model are overestimated for a large fraction of samples. Meanwhile, the photometric redshifts from the low entropy model show concentrations at $z_{phot}\sim 0.05$ and $z_{phot}\sim 0.2$. The peculiarities in the derived photometric redshifts arise from the minimization of $L_{unsup}$ during training stage-2, which makes the output probability distributions of the high and low entropy models flat and peaked, respectively. These oddities of the training stage-2 model vanish in the averaged stage-3 model (see Figure 3). The Pearson correlation coefficient $\rho=0.959$ affirms that the estimated photometric redshifts match spectroscopic redshifts well.

As we design and expect, training stage-2 model shows better separation between the ID and OOD samples (i.e., physically OOD samples of stars and QSOs) than the stage-3 model. We compare the OOD detection performance of the networks in training stage-2 and -3. The detection metrics of the networks in training stage-2 and -3 are tabulated in Table 5, showing that the OOD detection model is optimized in training stage-2.

Interestingly, the training stage-3 model has higher precision than the stage-2 model, which is due to the high TP values with the small number of samples flagged as OOD by the training stage-3 model; 352,055 samples ($\sim 7\%$) among 5,006,223 true OODs are correctly classified as OOD. This small number results in unexpectedly high precision, $AUC_{ROC}$, and $AUC_{PR}$ for the training stage-3 model.

The stage-2 model separates well the labeled ID and OOD samples as shown in Figure 4 displaying the $L_{DCP}$ distributions of the ID, LOOD, and UL samples in training stage-2. Although most ID samples appear in low $L_{DCP}$ ranges, most LOOD samples are distributed in the vicinity of the highest $L_{DCP}$. In addition, most UL samples show high $L_{DCP}$ values. There are small portions of the UL and LOOD samples with the low $L_{DCP}$, which represents samples with the input properties similar to those of the ID samples. We infer that the highest proportion of UL samples are more likely to be OOD objects (including physically OOD objects such as stars and QSOs) than ID galaxies although we cannot specify the classes of the samples. The distribution of the $L_{DCP}$ in the UL samples makes sense because the largest fraction of sources should be stars in our Galaxy.

The prediction difficulties of photometric redshift estimation and OOD scores for the ID data are related as presented in Figure 5. Although most samples with non-catastrophic (hereinafter, non-cat) estimation (i.e., $(z_{spec}~{}-~{}z_{phot})/(1+z_{spec})\leq 0.15$) are assigned to the low $L_{DCP}$ group, most samples with catastrophic (hereinafter, cat) estimation of photometric redshifts belong to middle or high $L_{DCP}$ groups. High $L_{DCP}$ samples generally more deviate from the slope-one line than low $L_{DCP}$ samples, indicating that the samples with high OOD score can be viewed as OOD-like ID samples and tend to have incorrect photometric redshifts.

The distributions of $L_{DCP}$ for the ID non-cat and cat samples shown in Figure 6 also affirm our interpretation of the relation between $L_{DCP}$ and the error in photometric redshifts. The distributions show that a higher fraction of cat samples populates in the high $L_{DCP}$ ranges than non-cat samples. Although the number of non-cat samples decreases as the $L_{DCP}$ increases, the cat samples are almost uniformly distributed over the entire $L_{DCP}$ range. The difficulties in estimating photometric redshifts are related to the OOD-like ID samples²²2In Paper I, we have already found that the lack of training samples causes high errors in photometric redshifts..

The training stage-3 model is unreliable concerning cat samples with high $L_{DCP}$. As shown in Figure 6, the number of non-cat samples with high confidence on photometric redshifts decreases as $L_{DCP}$ increases. The cat samples mostly have low confidence on photometric redshifts throughout low and middle $L_{DCP}$ ranges. It also shows that the networks in the training stage-3 are well calibrated in estimating photometric redshifts for the cat samples with low and middle $L_{DCP}$ presenting the low confidence on them. However, the cat samples with high $L_{DCP}$ extend to cases with high confidence which is the indication of overconfident results on the wrong photometric redshift estimation for OOD-like ID cat samples. It also emphasizes the importance of the OOD detection algorithm since we cannot filter the OOD samples simply based on confidence on photometric redshifts without the OOD score.

The distribution of $L_{DCP}$ in the space of input features allows us to visually inspect the OOD-like ID samples. In Figure 7³³3 This visual presentation often used hereafter displays the projected dimensions of the input space with degeneracy of the input features in the higher dimension., most high $L_{DCP}$ samples reside in the low-density regions of the training samples. The training stage-2 model is optimized to assign high OOD scores to under-represented galaxies, meaning that these high $L_{DCP}$ ID samples are under-represented galaxies from the model perspective.

As we find the ID samples with high $L_{DCP}$, there are LOOD samples with low $L_{DCP}$ (i.e., the ID-like LOOD samples). As presented in Figure 8 depicting the $L_{DCP}$ distributions for the two types of LOOD objects, i.e., QSOs and stars, a fraction of LOOD samples have low $L_{DCP}$ although most samples have high $L_{DCP}$. The points to be noted are as follows: 1) both QSO and star distributions have peaks at the lowest $L_{DCP}$ bin, 2) the ratio of the lowest $L_{DCP}$ peak to the highest $L_{DCP}$ peak for QSO is larger than that of star, and 3) more QSOs are distributed in the low and middle $L_{DCP}$ ranges than stars.

The ID-like LOOD samples locate around the ID sample distribution in the input space. Figure 9 shows that the QSO samples with high $L_{DCP}$ overwhelm the ones with low $L_{DCP}$ outside the ID sample density contours. However, the number of the low $L_{DCP}$ samples for QSOs increases as they approach the ID sample density peak. These ID-like QSOs contribute to the peak at the lowest $L_{DCP}$ bin of Figure 8.

As found in Figure 9, some stars certainly reside inside the contours of the ID galaxy samples with respect to the projected input dimension, corresponding to the stars with low $L_{DCP}$ (Figure 8). However, most stars have input features significantly deviating from the typical properties of the ID galaxy samples in any one dimension of the input features. Compared with the QSO distribution shown in Figure 9, more stars than QSOs can be sorted out from the ID samples in higher dimensions of input features since the star distribution significantly differs from that of the ID samples. Therefore, the relatively smaller peak of stars at the lowest OOD score and a lower fraction of ID-like stars than QSOs arise as shown in Figure 8.

Filtering out the QSO and star samples with low/middle $L_{DCP}$ enables further reliable usage of the trained model for photometric redshifts. As depicted in Figure 10, although the confidence of ID samples gradually reduces as $L_{DCP}$ increases, the LOOD objects with high $L_{DCP}$ have higher confidence than those with low $L_{DCP}$. Using this distribution difference in the space of $L_{DCP}$ and confidence, a considerable number of LOOD samples with low and middle $L_{DCP}$ can be filtered by checking their higher confidence than that of ID samples.

Because the UL data have no labels and spectroscopic redshifts, we assess the model performance on the UL samples indirectly. To measure the performance of detecting the physically OOD samples in the UL dataset, we first extract the point-source score (hereinafter, ps-score) from Tachibana & Miller (2018) for the UL samples. The ps-score is a machine-learning-based probabilistic inference on being point sources for PS1 objects with the range of [0, 1]. The samples with low ps-score are more likely to be extended sources, i.e., mainly galaxies, and we expect our OOD detection algorithm to produce low $L_{DCP}$ for samples with low ps-score.

The distributions of $L_{DCP}$ with respect to ps-score are subject to the expected pattern. Figure 11 shows the $L_{DCP}$ distribution of the samples under-sampled from the UL dataset within the RA ranges [0, 10], [100, 110], and [170, 180]. The samples within the lowest ps-score interval are mostly distributed near the minimum and maximum $L_{DCP}$ values, and the distributions of the higher ps-score samples shift toward high $L_{DCP}$ ranges. We conjecture that the middle/high $L_{DCP}$ samples with low ps-score are highly likely to be under-represented galaxies residing in the outskirts or low-density parts of the ID distribution (see discussion for Appendix A). Except for this understandable difference, the two machine-learning models reach a consensus on other UL samples which are mainly point-source stars and QSOs.

Figure 12 indirectly proves $L_{DCP}$ to be a reliable OOD score for a broad range of input space for the UL samples. The UL samples with low $L_{DCP}$ have the most similar distributions to those of the ID samples because the networks assign low $L_{DCP}$ to the UL samples with input features comparable with the ID samples. The UL samples with middle $L_{DCP}$ begins to display discrepancies from the ID samples in input features distributions. The samples in the high $L_{DCP}$ group have a completely different distribution from the ID samples. In the increasing order of the OOD score (i.e., $L_{DCP}$), the UL data density distributions in the input color spaces deviate more from those of ID samples. As expected from the typical colors of stars (e.g., Jin et al., 2019), the peak in the color distribution for the high-$L_{DCP}$ UL samples in Figure 12 corresponds to common stellar colors.

The training stage-3 model adequately produces photometric redshifts for the UL samples. As examined with the ID and LOOD samples, we expect the photometric redshift distributions to be similar to those of ID samples for the UL samples with low $L_{DCP}$ and random guessing, i.e., uniform distribution, for the UL samples with high $L_{DCP}$, respectively. As presented in Figure 13, the samples in the low $L_{DCP}$ range mostly reside inside the ID sample density contours. On the other hand, the photometric redshifts of the OOD-like samples randomly spread in the spaces without any strong systematic patterns. It certainly shows that the photometric redshift distribution for OOD-like samples are closer to a uniform distribution than ID-like samples. However, most redshifts are in the region $z_{phot}\,<\,0.8$. It arises from the fact that approximately $99.82$% of the training samples for photometric redshifts are found in the region $z_{spec}<0.8$.

We examine the UL samples with the low ps-score and their $L_{DCP}$ in the input feature space. These samples generally correspond to galaxies not well represented by the training data. Naturally, the UL samples include more faint galaxies than the spectroscopic training samples because the procedure of selecting spectroscopic targets is generally biased to bright samples. Furthermore, the color and photometric-redshift distributions of the UL samples with the low ps-score and high $L_{DCP}$ affirms that these UL samples have input features that are not well represented by the training samples (see Figure 14).