PNet – A Deep Learning Based Photometry and Astrometry Bayesian Framework

Choosing appropriate performance evaluation criteria is crucial for properly assessing the performance of a framework. It is essential to select evaluation criteria that align with the objectives of developing the framework. Our algorithm focuses on four key aspects: detection of point-like targets, regression of their positions and magnitudes, and estimation of photometric uncertainties. Hence, the following performance evaluation criteria have been selected for our algorithm:

• •

  The recall rate and the precision rate, and mean Average Precision (mAP) are chosen to evaluate the performance of target detection results.

• •

  The astrometry accuracy in pixels is used to assess the accuracy of the astrometry results.

• •

  The photometry accuracy in magnitudes is used to evaluate the accuracy of the photometry results.

• •

  The outlier fraction ($\eta$), Normalized Median Absolute Deviation (NMAD), Median Absolute Deviation (MAD), and the mean value of the photometry results $1\sigma$ ($\bar{E}$) are utilized to evaluate the uncertainty of the photometry results.

In the following, we will provide a detailed description of the aforementioned performance evaluation criteria. When evaluating the detection results, we consider four possible scenarios:

• •

  True Positive (TP): This occurs when a point-like celestial object is correctly identified as a point-like celestial object.

• •

  True Negative (TN): This occurs when targets other than point-like celestial objects are correctly identified as non-point-like celestial objects.

• •

  False Positive (FP): This occurs when targets other than point-like celestial objects are wrongly identified as point-like celestial objects.

• •

  False Negative (FN): This occurs when point-like celestial objects are wrongly identified as non-point-like celestial objects.

Given that our framework is capable of directly outputting the center coordinates of the point-like celestial objects, we assess the detection results by calculating the Euclidean distance between the predicted position and the corresponding position in the label. The Euclidean distance can be defined using the following equation 2:

the variables $x_{i}$ represent the predicted positions, $y_{i}$ represent the label positions, and $n$ represents the number of coordinates used to describe the positions (which is 2 in this paper). If the Euclidean distance between the predicted positions and the label positions is below a certain threshold and the classification result is in accordance with the label, it will be considered true positive (TP) or true negative (TN). Otherwise, it will be classified as a false positive (FP) or false negative (FN).

Based on the aforementioned definition, we can assess the performance of our framework in target detection by utilizing the Precision and Recall metrics, as defined in the following Equation 3:

Precision and Recall are widely used metrics for evaluating target detection results. Precision represents the percentage of true positives among all positive detection results, indicating the performance of the detection algorithm in minimizing false alarms. Recall, on the other hand, represents the percentage of true positives among all actual targets, describing the ability of the detection algorithm to identify all positive instances. Precision and Recall are both influenced by the chosen detection threshold. A higher threshold leads to higher precision, but lower recall, and vice versa. To comprehensively evaluate the performance of a detection algorithm, we can vary the detection threshold and generate a precision-recall curve (P-R curve). The area under the P-R curve is known as the average precision (AP). The mean average precision (mAP) is calculated by averaging the AP values across all categories, providing an overall assessment of the detection algorithm’s performance. In this paper, the mAP is obtained using Equation 4, where $n$ represents the number of categories (which is 1 in this case). Furthermore, it is worth noting that the astrometry accuracy can be measured by calculating the distance between the predicted position and the corresponding position in the label for all true positive (TP) detection results.

The Bayesian neural network is employed to estimate the uncertainty inherent in the data. When a star image is fed into the Bayesian neural network, it generates a predictive posterior distribution for the star’s magnitude, along with the mean value of that distribution. The mean and standard deviation of this distribution, which cannot be directly obtained, are typically estimated through multiple Monte Carlo sampling iterations. The standard deviation $\sigma_{mag}$ of this distribution can be interpreted as a quantitative measure of uncertainty. Consequently, we consider two aspects of the output: the deviation of the true value from the mean of the predicted distribution and the level of uncertainty in the prediction results. To evaluate the deviation of the true value from the mean, we employ the relative error $RelativeError$ and the absolute error $\delta_{mag}$, defined in Equation 5:

By considering the relative error for all targets, we can determine the fraction of outliers ($\eta$) by establishing a threshold value for the relative error and calculating the proportion of prediction results exceeding this threshold ($\sigma_{mag}$ in this study). Additionally, we employ the normalized median absolute deviation (NMAD) to evaluate the relative errors obtained. The median absolute deviation (MAD) is a statistical measure that characterizes the sample bias of one-dimensional numerical data. To obtain the NMAD, we normalize the MAD by multiplying it by a factor of 1.4826 (Rousseeuw & Croux, 1993), as demonstrated in Equation 6,

The absolute error is assessed through the Mean Absolute Error (MAE), which represents the average of the absolute differences between the mean of the predictive posterior distribution and the corresponding true value. This metric provides a visual indication of the error level. Regarding uncertainty, we initially estimate the magnitude distribution for each star using the Bayesian neural network. We then compute the mean value of these distributions, denoted as $\bar{E}$, which serves as a reference indicator for the uncertainty distribution.

The Bayesian photometry neural network is utilized to estimate both the magnitude of stars and the associated photometry uncertainties. In traditional neural networks, the weights are fixed, leading to a lack of uncertainty estimation and excessive confidence in the predicted results. To address this issue, Bayesian neural networks employ the Variational Bayes (VB) method (Blundell et al., 2015) to introduce uncertainty into the network weights. However, before delving into the methods for capturing uncertainty, it is crucial to comprehend the origins of uncertainty.

In the field of machine learning, two main types of uncertainty are commonly recognized: aleatoric uncertainty (also known as data uncertainty) and epistemic uncertainty (also known as model uncertainty) (Kiureghian & Ditlevsen, 2009). Aleatoric uncertainty (AU) stems from the inherent noise present in the dataset itself (Gal et al., 2016). Since this noise is natural and unpredictable, aleatoric uncertainty cannot be eliminated. On the other hand, epistemic uncertainty (EU) arises from insufficient training of the network, resulting in a lack of knowledge about the system’s behavior. In principle, this uncertainty can be reduced as the training data approaches infinity (Hora, 1996). As mentioned earlier, the predictive uncertainty (PU) we aim to capture can be expressed as the combination of AU and EU (Abdar et al., 2021), as illustrated in Equation 7:

By defining the $PU$, we can establish the underlying principle of the Bayesian Photometry Neural Network (BPNN). Initially, we define the weights of the BPNN as $\omega\in\Omega$, where $\Omega$ represents the parameter space of the BPNN. The training dataset is denoted as D, and within this dataset, we have data pairs X and Y. Similarly, in the test dataset, we have data pairs x and y. The distribution of weights $\omega$ learned by the network from the dataset D is represented as $p(\omega|D)$. Additionally, $p(y|x,\omega)$ signifies the probability that the neural network yields output y when given input x and weight $\omega$. Lastly, $p(\omega)$ represents the prior weight distribution of the network. With these definitions, we can derive the probability distribution of the output y given the input x when the neural network is trained using dataset D, as illustrated in Equation 8:

The prediction uncertainty captured by the BPNN is represented by $p(y|x,D)$. However, obtaining $p(\omega|D)$ through analytical calculations is challenging in real applications, often requiring approximation methods to perform the inference task (Hortúa et al., 2020). In this study, we employ variational inference to approximate the solution for $p(\omega|D)$. Initially, we assume a variational distribution $q(\omega|\theta)$, where $\theta$ denotes a set of variational parameters. Subsequently, we calculate the Kullback-Leibler (KL) divergence between the variational distributions $q(\omega|\theta)$ and $p(\omega|D)$. Finally, we determine a set of variational parameters $\theta^{\ast}$ that minimizes the KL divergence, as shown in Equation 9:

Since $p(\omega|D)$ cannot be obtained analytically, we further introduce the Bayesian formula below:

We could obtain the KL divergence according to Equation 9 and Equation 10, as shown in Equation 11:

Since $\ln{p(D)}$ is only related to properties of data, we could minimize the KL divergence through minimizing the following Equation 12:

It is important to note that the term $F(D,\theta)$ in Equation 12 corresponds to the negative value of the Evidence Lower Bound (ELBO), as discussed in Blei et al. (2017). By employing Equation 12, we can transform the analytically challenging calculation into a practical optimization problem for the variational parameter $\theta$. For a comprehensive understanding of the approximation methodology, we refer readers to the work of Hortúa et al. (2020), while providing a concise overview here.

To begin, let us assume that we can obtain a set of variational parameters $\hat{\theta}$ that minimizes $F(D,\theta)$ through an optimization process. In this study, we perform such optimization using the backpropagation algorithm (Rumelhart et al., 1986) and employ the Adam gradient descent algorithm (Kingma & Ba, 2014). With the obtained $\hat{\theta}$, we can derive the predictive distribution $q_{\hat{\theta}}$. By combining this with Equation 8, we can determine the predictive distribution of the output variable y given the input x:

Although Equation 13 provides an analytical solution for the predictive distribution, calculating the integral can be challenging in practical applications. Hence, we employ Monte Carlo sampling (Gal et al., 2016) and series addition method to obtain the final results:

where $N$ denotes the number of samples, and $\hat{\omega}_{n}$ represents the nth sampled value of the weights obtained from $q(\omega|\hat{\theta})$. Equation 14 is equivalent to Equation 13 as the number of samples $N$ tends to infinity. This equation indicates us that we could obtain a Bayesian estimation through Monte Carlo sampling of the weights for a predefined neural network.

In the study by Hortúa et al. (2020), the authors use the total variation principle to derive the analytical results for the variance of the predicted distribution on a fixed input x. They further simplify these results to obtain:

In the equation above, $T$ denotes the total number of forward passes of the network. The terms $\sigma_{t}$ and $\mu_{t}$ refer to the standard deviation and mean of the distribution obtained during the t-th forward pass, respectively, while $\bar{\mu}$ represents the mean of all $\mu_{t}$ values. The first term in Equation 15 corresponds to the aleatoric uncertainty discussed earlier, whereas the second term corresponds to the epistemic uncertainty. In real applications, we build a BPNN with the flipout method discussed in Wen et al. (2018) to generate pseudo-independent weight perturbation on minibatches, which could simulate the Bayesian interference process. Then the distribution between predicted results and prior distribution could be evaluated to provide reference to photometry uncertainty. We will discuss details of the method in Section 3.3.2.

In this subsection, we will provide a comprehensive overview of the neural network structure employed in our framework. The neural network comprises three interconnected components that collaboratively predict the final outcomes. The first component, known as the Photometry-Detection Net, is designed to detect point-like celestial objects, accurately determine their subpixel positions, and calculate their flux. The second component is responsible for estimation of the photometry results and their uncertainties. Lastly we carry out the photometry calibration, utilizing reference stars to ensure precise calibration of the photometric measurements derived from the flux values. This framework allows us to obtain reliable magnitudes and positions for point-like celestial objects. It is important to emphasize that the primary focus of this paper is not the detection of various celestial objects such as galaxies or quasars. Neural networks specifically tailored for the detection of celestial objects from multicolor images are more suitable for these targets (Jia et al., 2023b). Once these targets have been detected by aformentioned methods, we proceed to mask them and carry out point-like celestial object detection, astrometry, and photometry using single-band images with our framework. As a result, the final output of our framework includes the positions of point-like celestial objects, their corresponding magnitudes, and the associated photometry uncertainties, which could be directly used to analyze celestial objects with light variation and moving celestial objects.

During the training phase, certain targets that are not utilized in the training data, such as dense star fields and stars close to galaxies, are masked. As previously mentioned, the detection of these targets from multiple color images requires specialized processing methods, which are beyond the scope of this paper. For optimization, we employ the Adam Optimizer (Kingma & Ba, 2014) and evaluate the classification results using the Focal Loss (Lin et al., 2017). The Focal Loss dynamically adjusts the impact of each training example on the overall loss, ensuring a balanced approach to the detection results of celestial objects with varying magnitudes. The Focal Loss is formulated by introducing a modulating factor into the Cross Entropy (CE) loss, as originally proposed by Lin et al. (2017). In this study, we employ an alpha-balanced version of the focal loss, as defined in Equation 19:

where $\alpha_{t}$ is the weighting factor used to adjust the weights of positive and negative samples, $p_{t}$ is the output of the network, and $\gamma>0$ is an adjustable focusing parameter (2 in this paper). The Mean Absolute Error (MAE) loss is utilized to assess position error, while the Mean Square Error (MSE) loss is employed to evaluate flux prediction outcomes. The computation of MSE and MAE is demonstrated in Equation 20, where $n$ represents the total number of targets, $\hat{y}_{i}$ denotes the predicted value of the i-th target, and $y_{i}$ denotes the corresponding true value. The aggregate of these three losses constitutes the loss function employed to train the Photometry-Detection Net, as depicted in Equation 20:

It takes approximately 324 seconds to complete a single epoch when training the Photometry-Detection Net on a computer equipped with a 3090Ti GPU. The batch size consists of 40 images, each with dimensions of $512\times 512$ pixels. After approximately 40 to 50 epochs of training with randomly initialized weights, the Photometry-Detection Net starts to converge. The convergence process is accelerated when using pre-trained weights. Since the instrumental magnitude measurements exhibit a relatively consistent pattern, when employing pre-trained weights and training on new data batches, it is common practice to freeze the feature extraction network (the Hourglass Network) and the photometry branch network. After training for a specific number of epochs, these two networks are then unfrozen, and the overall network undergoes fine-tuning.

After training, we have tested the performance of Photometry-Detection Net in detecting star targets and measuring magnitues using a batch of $512\times 512$ pixel SDSS astronomical images and simulated images generated by Skymaker. When the network outputs classification results, it provides a confidence level, and when this level is higher than a certain threshold, we consider that there exist star targets. Generally, the lower the threshold, the higher the recall, but the lower the precision, and vice versa. We have shown the performance of Photometry-Detection Net using different confidence thresholds, as shown in Figure 6. When the confidence threshold is set to 0.3, the specific results obtained from testing on SDSS data and simulated data are shown in Table 1. The table also presents the detection results from SExtractor. It’s evident that our method achieves a higher level of precision and recall rate compared to SExtractor.

During the star target detection process, the Photometry-Detection Net also conducts photometry and astrometry. For the SDSS data, the photometry branch utilizes the calibration method described in Section 3.3.3 to calibrate the magnitudes. However, since the simulated data does not account for airmass effects or other effects, calibration is not necessary for these data. When using SExtractor to process SDSS data, a fixed offset between the photometry and astrometry results and their ground truth values may exist. This offset can be determined by statistically analyzing the measurement results. In our study, we have adjusted the results obtained from SExtractor by subtracting this offset. Aside from this adjustment, no further processing has been performed on the measurement results derived from SExtractor for the SDSS data. The photometry and astrometry results are depicted in Figure 7 and Figure 8.

Figure 7 reveals that the magnitude error is mostly within 0.5 magnitudes for the majority of stars, with approximately $50\%$ of the stars exhibiting magnitude measurement errors within $2.5\%$. These measurement errors tend to increase gradually as the magnitude of the stars increases. This behavior aligns with theoretical analysis, as the impact of noise with the same level has a smaller effect on stars with lower magnitudes. For every 5-magnitude decrease, the flux of the star increases by a factor of 100. Additionally, compared to the errors observed in the SDSS data, the measurement errors of star magnitudes in the simulated data are generally smaller and more concentrated. This difference primarily stems from the presence of noise in real data, which not only interferes with the algorithm’s flux measurements but also affects the process of calibrating star magnitudes. Additionally, as illustrated in Figure 7, it’s worth noting that the photometric error of our framework when applied to SDSS data exhibits some asymmetry, primarily due to the calibration process. However, when looking at the overall performance, our framework shows more consistent and reliable photometric results, outperforming those obtained by SExtractor on the SDSS data.

Meanwhile, we compared the astrometry results obtained by the Photometry-Detection Net and SExtractor in Figure 8. The astrometry error manifests as a roughly symmetrical circle. The contour lines marked with values 3 represent a diameter of 0.15 pixels, while those with values 4 represent a diameter of 0.21 pixels. In contrast to SExtractor, our framework demonstrates superior astrometry accuracy when applied to the SDSS data, with a higher number of stars exhibiting astrometry error smaller than 0.1. Nevertheless, it’s worth noting that the astrometry error for the SDSS data remains somewhat larger than that observed in the simulated data, primarily due to the introduction of other noise during real observations.

As mentioned earlier, the detected results in the images will be segmented into smaller patches to estimate photometry results and uncertainties. We assume the Gaussian distribution as the prior distribution for the photometry results, and the loss function can be derived directly from Equation 12. For optimization during the training stage, we utilize the Adam optimizer (Kingma & Ba, 2014). However, during the training stage, there is a possibility of encountering the issue of exploding gradients due to the random weight sampling, which can lead to unstable training stage. To address this problem, we implement gradient clipping by setting a threshold. If the gradient surpasses this threshold, it is truncated to a specific value. On average, training one epoch typically takes approximately 12 seconds for a batch of 2000 small images.

Upon completing the training phase, we proceed to assess the performance of the Bayesian Photometry Neural Network using a dataset of 14,000 target samples. To obtain the distribution of estimated magnitudes for each star target, we employ Monte Carlo sampling with 50 discrete samples. It is worth noting that due to the large number of parameters involved and the challenges inherent in optimizing Bayesian Neural Networks, the estimated uncertainty distribution may not closely align with the true distribution. In order to address this, we propose utilizing the method introduced by Chung et al. (2021) to calibrate the predicted uncertainty results. This calibration algorithm, based on isotonic regression (Kuleshov et al., 2018), adjusts the uncertainty represented by the standard deviation by determining an appropriate calibration coefficient. After measuring the magnitudes of all star targets, the Bayesian Photometry Neural Network provides corresponding mean values and standard deviations to characterize the uncertainty distribution. We employ Equation 21 to standardize these distributions:

Here, $\mu$ represents the mean of the predicted distribution, $y_{true}$ represents the true value of the corresponding target, and $\sigma$ denotes the standard deviation of the predicted distribution. By utilizing the aforementioned method, we are able to obtain the distribution of all targets on a standardized evaluation scale. Subsequently, we compare the standardized distribution of magnitudes for each target with a standard normal distribution having a mean of 0 and a standard deviation of 1. This comparison allows us to assess the accuracy of our Bayesian network’s quantification of uncertainty. The results are depicted in Figure 9.

More specifically, we utilize the standard normal distribution as the probability density function (PDF) to conduct a symmetrical search for boundaries from the center outward. This search aims to identify the boundaries where the ratio of the integral of the PDF within the boundary to the infinite integral of the PDF equals the horizontal axis value in the Average Calibration plot of Figure 9. The vertical axis represents the ratio of the number of standardized targets within the observed boundary to the total number of targets. Since the PDF follows a standard normal distribution, the infinite integral value is 1. In an ideal scenario where uncertainty can be accurately quantified, these two ratios should align, resulting in a diagonal line with a slope of 1 in the Average Calibration plot of Figure 9. When the curve bends downwards, indicating that the Predicted proportion in the interval exceeds the Observed proportion in the interval, it signifies an overconfident state of uncertainty estimation (where the predicted uncertainty is too small). Conversely, when the curve bends upwards, indicating that the Predicted proportion in the interval falls below the Observed proportion in the interval, it suggests an under-confident state of uncertainty estimation (where the predicted uncertainty is too large).

Overall, the Bayesian Photometry Neural Network has the ability to predict the uncertainty distribution of magnitude estimation results. Once we have obtained the magnitude distribution predictions for all targets using this network, we can assess these predictions using the evaluation criterion described in Section 3.1. By defining the boundary of the relative error as positive and negative $3\sigma_{relative}$, we can examine the results presented in Table 2. It is worth noting that the calculation method for $\sigma_{relative}$ involves initially calculating the relative error of all targets using the procedure outlined in Equation 6, and subsequently determining the standard deviation of these relative errors as $\sigma_{relative}$. This calculation method differs from the $\sigma$ value that represents the uncertainty computed from the distribution of an individual target. Based on the data presented in Table 2, it becomes apparent that the outlier fraction obtained from the simulated data is lower compared to that from the real data, suggesting that the algorithm demonstrates greater stability when applied to simulated data. Furthermore, the $sigma_{NMAD}$ and MAE values, assessed for both simulated and real data, indicate that the dispersion of predictions from this algorithm is lower, resulting in overall smaller errors when dealing with simulated data. Additionally, the parameter $\bar{E}$ indicates that the Bayesian Photometry Neural Network expresses higher confidence in magnitude measurements for simulated data. In general, the results obtained from simulated data exhibit higher accuracy and confidence compared to real data. This discrepancy could potentially be attributed to the effects induced by noise present in real observational data. On one hand, the presence of noisy pixels can interfere with the Resnet50 model’s extraction of image features, particularly when stars have low signal-to-noise ratios, intensifying such interference. On the other hand, noise can also disrupt the statistical photometry results obtained by the Bayesian layer when leveraging image information. This interference manifests as increased uncertainty in the output results, consequently indicating lower reliability and credibility.