Discovering Faint and High Apparent Motion Rate Near-Earth Asteroids Using A Deep Learning Program

The classifier that we use in this paper to find asteroid streaks is a convolutional neural network (CNN), specifically a modified version of EfficientNet-B1 (Tan & Le, 2019). CNNs have had much success in being able to differentiate between different classes of images and achieve state-of-the-art performance on classification tasks like ImageNet, which involves classification of images into object categories like birds, cars, and screwdrivers (Russakovsky et al., 2015). CNNs accomplish this through combination of many different “layers" which are trained to recognize images using training data and gradient descent.

EfficientNet is a family of CNN models that currently has one of the highest accuracy to performance ratios, which is important as we want a model that runs quickly and has high accuracy at the same time. EfficientNet is able to achieve its high efficiency by computationally searching for the best model configuration and method for scaling up the size of the CNN (Tan & Le, 2019).

We chose to use EfficientNet-B1 (the 2nd smallest version of the model) in order to allow for higher accuracy than the B0 model (the smallest version) while still allowing the CNN to be run quickly on most GPUs. Because the EfficientNet model is intended to be used on RGB images from the ImageNet dataset, we modify the inputs to contain two 80 by 80 images – the science and reference cutouts – and the output to be a probability from 0 to 1 – representing the probability the image contains an asteroid streak. This is done using an EfficientNet-B1 backbone and then modifying the final layers to use global pooling and fully connected layers to process the feature vector output by EfficientNet (see Table 1).

The raw image data can not be passed directly into the machine learning model as the pixel values can vary widely due to extremely bright stars. If we input these numbers directly into a neural network, we may run into overflow errors. Moreover, extremely bright stars will dominate images and make dim streaks almost invisible in comparison (see Figure 7). However, the full range of brightnesses is not necessary, especially since this paper focuses on fainter asteroids.

To normalize both the science and reference images, the image background intensity and background RMS is computed using the sep library, which is based upon the popular Source Extractor library (Barbary, 2016; Bertin & Arnouts, 1996). This allows us to standardize each pixel in the image by computing the difference between the pixel and the background intensity and then dividing that difference by the standard deviation. Pixel values below $-5$ and above $5$ are clipped to $-5$ and $5$, respectively, since extremely low and high pixel values do not provide much useful information, allowing us to focus on dimmer pixels which may belong to streaks. This gives us the following equation for normalizing each pixel $p$:

To reduce overfitting, where the CNN “memorizes" the training data rather than learning how to extrapolate to unseen samples, we apply a dropout rate of 0.2, which causes 20% of neurons in the CNN to be randomly omitted during each iteration of training. Dropout has been shown to be effective in reducing overfitting by making it harder for neural networks to reliably memorize training data (Srivastava et al., 2014).

This model was implemented using the Keras (Chollet et al., 2015) and EfficientNet Python libraries³³3https://github.com/qubvel/efficientnet. To train the CNN, we used the binary cross-entropy loss function, a smaller batch size of size 32 to further reduce overfitting (Masters & Luschi, 2018) and the Adam optimizer (Kingma & Ba, 2014). We used an NVIDIA 2080 TI GPU to increase training speed.

To reduce the number of false positives the model outputs, artifacts which may be detected as a false positive must be removed. The three types of false positives prevalent in ZTF data are cosmic rays, ghosts, and crosstalk.

Cosmic rays are produced when highly energized particles strike CCDs. This produces transient streak-like objects in the images (see Figure 8). One of the weaknesses of our normalization algorithm is that it is hard for the CNN to differentiate asteroid streaks and cosmic rays since cosmic rays are distinct in that they are very bright and their pixels are not distributed according to a standard point spread function, but these nuances are lost due to the clipping of bright pixels. However, there have been many algorithms developed that can effectively exploit the unique appearances of cosmic rays to remove them from images. To remove cosmic rays, we employ the AstroSCRAPPY library, which is based upon the LA Cosmic algorithm (McCully et al., 2018). The LA Cosmic algorithm uses Laplacian edge detection to differentiate cosmic rays from other sources of light (van Dokkum, 2001). This allows us to detect cosmic rays in an image, and then mask them out, as is done in Figure 8.

Ghosts are created by charge spillages from saturated pixels in the same image. These ghosts are horizontal streaks located at a set distance in front of or behind a cluster of saturated pixels and are typically up to 50 pixels in width (Zwicky Transient Facility, 2020). To remove these, we mask out pixels which are a certain distance in front of or behind a saturated pixel and the 50 pixels in front and behind it to account for the width (see Figure 9). Although this sometimes leads to masking of pixels without ghosts, since charge spillage does not always occur, the number of pixels masked in an image is extremely small compared to the total number of pixels in the image (<1%).

Crosstalk is an artifact produced when a strong signal in one CCD quadrant (channel) is electronically replicated onto another quadrant. To remove such artifacts, the pixel locations of strong signals (e.g. saturated pixels) are masked in both quadrants affected by crosstalk (see Figure 10). We observed that the CCD pairs affected by bi-directional crosstalk are quadrants 1-2 and 3-4.

The components used for training the neural network are combined together to create a pipeline for detecting asteroids. To process one night of data, we use the following procedure on every single triplet of the corresponding science, reference, and difference images to search for asteroid streaks. The result of this is a large set of potential asteroid streak candidates.

1. 1.

   The difference image preprocessing pipeline is used to extract transient objects, which will include asteroid streaks but mostly artifacts and other non-streak transients. The positions of these transients are projected from the difference image onto the reference image.

2. 2.

   The science image is aligned to the reference image using the SWarp program (Bertin, 2010). This is to account for differences in rotation and position of the science and reference images. The positions obtained from Step 1 are used to extract 80 by 80 pixel crops from the aligned science and reference images, where the center of the crop corresponds to the center of the transients.

3. 3.

   All the crops are normalized using the normalization process described in Section 2.2 and Equation 5. These normalized images are then inputted into the trained convolutional neural network, which outputs probabilities for each science-reference pair.

4. 4.

   These probabilities are thresholded using a threshold of 0.85, so that probabilities above 0.85 are considered to be positive detections and ones below 0.85 are considered to be negative detections. This gives us a set of images which the CNN believes contain asteroid streaks.

5. 5.

   Cosmic rays, crosstalk, and ghosts are removed from the original un-normalized images of the positive detections if present and the cleaned images are re-normalized and re-inputted into the CNN. This allows us to reduce computation time by only running the false positive reduction on the small subset of positive detections from the CNN.

Each streak candidate is then visually checked and false positives are removed. Detections with the same orientation and length along with a corresponding motion rate and movement direction are linked together. The astcheck⁴⁴4https://www.projectpluto.com/astcheck.htm program is then used to check if the detection corresponds to a previously discovered asteroid and the sat_id⁵⁵5https://www.projectpluto.com/sat_id.htm program is used to ensure it does not correspond to a known artificial satellite.

The validation dataset that we use consists of a subset of our dataset which is set aside and not trained on, so that we can evaluate the performance of our model on unseen data. We generate additional positive and negative samples so that in total, 30% of our data is used for the validation set, which is abnormally high for most machine learning research. However since our false positive rate is extremely low ($\sim$0.02%), we must ensure that we have enough data to obtain an accurate estimate of the false positive rate.

The final performance of the CNN on the validation dataset can be evaluated on the metrics in Table 2 and the receiving operator characteristic (ROC) curve which plots the true positive rate against the corresponding false positive rate (see Figure 11). Overall, the accuracy for the validation dataset is extremely high at 98.7% while still maintaining an extremely low false positive rate of just 0.02%. This low false positive rate is critical to reducing the number of images outputted by the CNN per night since the vast majority of images do not contain asteroid streaks.

We collected a set of real streaks from ZTF’s submissions to the Minor Planet Center. The submissions falling within the date range of ZTF’s publicly available data were then kept, and cutouts of the corresponding streaks are extracted. This dataset of real streaks differs from the one used to generate the distributions of our simulated data since it only uses streak detections which ZTF has verified and reported to the MPC. This allows us to directly compare our model’s performance on streaks which ZTF’s DeepStreaks model (Duev et al., 2019a) has successfully detected. We incorporate data from detections of confirmed asteroids and unconfirmed asteroids (which are objects with a short observational arc located in the MPC’s Isolated Tracklet File). Since this research focuses on shorter streaks, we remove streaks longer than 50 pixels in length.

After running our CNN on the original images that contain these previously detected streaks, we plotted the visual magnitude and motion rates (lengths of streaks) of the correct and incorrect classifications (see Figure 12). Overall, our detection completeness is very high, roughly matching the true positive rate for simulated data. The detection completeness for confirmed asteroids is 97.4% and 95.6% for unconfirmed asteroids. For simulated data the true positive rate is 97.0%. We plot the missed detections in Figures 14 and 14.

Our pipeline was run on the following nights of data from the ZTF public data releases: 2019/06/05-08. The DeepStreaks algorithm began making discoveries in November of 2018, so the data from 2019 allows us to verify that the streaks which we detect were not found by DeepStreaks, thus demonstrating improvement over their algorithm.

Our new discoveries of asteroids and detections of previously discovered ones are summarized in Table 3. In just four nights of data, we were able to find six new NEOs and detections of five asteroids ZTF and others have discovered previously. Images of these new asteroids are shown in Figure 15. Note that only two streaked detections of an asteroid are needed to consider it a discovery since each streak contributes two sets of positions and time (one at each endpoint of the streak, which correspond to the start and end of the exposure). This gives us four sets of astrometry which is enough to fit a preliminary orbit.

For each of our streaks, we fit a Gaussian PSF model (see Equation 1) using the Markov chain Monte Carlo (MCMC) method to minimize the mean squared error between the pixels in the difference image and the model. To accomplish this, the approximate locations of the ends of the streak are identified by hand. Then, a pill shaped mask is generated by finding all the pixels within 5 pixels of the line segment connecting the two points. The pixels in the mask are then used to perform the MCMC least-squares fit, which is done using the emcee library (Foreman-Mackey et al., 2013). Using MCMC allows us to quantify the uncertainties in each of the parameters. An example of a corner plot created from the MCMC sampled streak fitting parameters can be found in Figure 16.

We also use the mask as a pill aperture for measuring the visual magnitude of our streaks. This is done by directly summing over the difference image (which roughly has a background of 0) pixels that are within the aperture using Equation 7, where $F_{ij}$ refers to difference image pixel intensity (in ADUs), ZP is the photometric zero point, and AP_COR is the visual magnitude correction required for an aperture with a radius of 5 pixels. These photometric parameters are calculated by ZTF and included in headers of the difference image FITS files. The uncertainty calculation is shown in Equation 9, where $F_{sci,ij}$ is the science image pixel intensity (which produces the shot noise), $n$ is the number of pixels, $R$ is the read noise (in e^-), $G$ is the gain of the CCD (in e^-/ADU), $D$ is the dark current (in e^-/sec), and $t$ is the exposure time (in seconds).

From the MCMC fits and photometry, we have obtained the astrometry, the length, the $\sigma$ parameter of the Gaussian PSF kernel, and the visual magnitude of each streak. We then use the astrometry and visual magnitude to fit orbits to each of our asteroids. Since our observation arcs are extremely short, there is significant uncertainity in these orbits, and so we use the OpenOrb library, which uses statistical orbital ranging to generate thousands of possible orbits (Granvik et al., 2009). These orbits can be then sampled to find the probability distributions of the closest approach distances and H magnitudes. The H magnitudes can then be converted to an approximate diameter using the following formula from Bowell et al. (1989), where $d$ is the diameter, $a$ is the albedo, and $H$ is the H magnitude. This formula assumes that the asteroid is roughly spherical in shape and has a uniform surface. We sample the albedo values from the distrbution derived from WISE data in Wright et al. (2016) (see Figure 17).

The attributes of each new streak detection is shown in Table 4. The probability distributions of the H-magnitudes, diameters, and close approach distances are also shown in Figures 18, 19, and 20, respectively. Due to the short observation arcs and the large variability in the possible albedos, the diameter and close approach distances have very high uncertainty. In Figure 18, we also compare the H-magnitudes of our asteroids to the discovery completeness of asteroids at those H-magnitude ranges. From the figure, we can see that our asteroids mostly correspond to small asteroids with high H-magnitudes that have a low detection completeness. By specifically targeting faint streaks, our algorithm is able to find these smaller asteroids, most of which currently have very low discovery completeness of less than around 10%.

Because archival data was used in our search for NEOs, we were unable to obtain the follow-up observations required for most of our streak detections to be confirmed by the Minor Planet Center (MPC). If we had access to more recent data, it is possible that other observatories would have done follow-up observations of these streaks, satisfying the MPC’s 24 hour observation arc requirement. However, we were able to find additional observations of Asteroid $\#$6 on 2019/06/09, partly due to the asteroid being located in a high cadence field which allowed us to obtain over 100 observations on the first night (see Kupfer et al. (2021) for more details). This NEO has now been given the designation 2019 LH27 by the MPC. In the future we plan to continue exploring high cadence data from ZTF’s archives to find asteroids which may potentially appear across multiple nights in ZTF data.

In this work, we only focused on shorter and fainter streaks. However, if we expanded our simulated dataset to include more bright and long streaks we likely would have better detection completeness for the ZTF discoveries we missed, as described in Section 3.2. For longer streaks however, it may become necessary to model the brightness fluctuations of asteroids due to rotation as small asteroids tend to have very short rotation periods on the order of minutes that would become more pronounced with longer streaks (Hergenrother & Whiteley, 2011).

While using the science and reference images rather than the difference images potentially allows our model to have access to more information, our normalization algorithm can cause bright pixels to be clipped, making it difficult to detect streaks significantly overlapping with bright stars. Because of this, it may be helpful to provide science, reference, and difference images to the machine learning model, as the difference images would subtract out stars that are overlapping with a streak. Increasing the clipping range to be larger than -5 to 5 could also allow for better detections.

It is important to note that ZTF’s algorithm uses a different preprocessing algorithm for extracting potential streak candidates to be fed into their CNN model. Their algorithm focuses on longer asteroid streaks and also has stricter thresholds to reduce false positives. Because of this difference in preprocessing, our CNN and synthetic training approach is not necessarily better at streak detection compared to ZTF’s algorithm. In particular, the 5th and 6th asteroids which we found were likely missed by ZTF since they are shorter than the streaks they tended to focus on during that time period of the survey. Nevertheless, the fact that we are able to recover most of ZTF’s discoveries using no real image data for the training set and are also able to detect undiscovered objects, suggests that our approach allows for the very robust detection of asteroid streaks based on the distributions that we target our dataset towards. Our work has shown that real image data is not necessary to train an effective asteroid streak detection system and that a purely simulated dataset may allow for the detection of asteroid streak distributions like faint streaks or short streaks for which detection completeness is low.

Given different strengths for both algorithms, we would suggest to ZTF that our pipeline could be a potential supplement which runs in addition to their current DeepStreaks algorithm by targeting specific distributions of streaks which may be harder to detect. Moreover, we believe the main benefit of our algorithm is derived from the removal of the need for real asteroid streak data collection, which allows other surveys besides ZTF to much more easily adopt streak detection.

As all the data we use to train the model is simulated, this approach can be more easily applied to other transient surveys than approaches that require real data. For instance, ZTF is a prototype for the future Vera C. Rubin Observatory. Applying our method to the Rubin Observatory, which has a higher limiting magnitude of roughly 24.7 for the red band filter as opposed to 20.4 for ZTF, fainter asteroids would appear at a higher SNR, making it much easier for our algorithm to detect them. Quantitatively, we can compare the étendue and volumetric survey speed⁶⁶6The volumetric survey speed roughly quantifies the volume of space where an object of a certain absolute magnitude can be detected divided by the time per exposure, allowing us to take into account survey speed (see Bellm (2016) for details) of both telescopes. The Rubin Observatory has an étendue of $319.5\text{ m}^{2}\text{ deg}^{2}$ and a volumetric survey speed of $3.7\times 10^{5}\text{ Mpc}^{3}\text{/s}$ while ZTF only has an étendue of $53.1\text{ m}^{2}\text{ deg}^{2}$ and a volumetric survey speed of $2.5\times 10^{4}\text{ Mpc}^{3}\text{/s}$ (Bellm, 2016). Using our method, faint asteroid streaks can be more robustly detected, allowing us to take advantage of the increased detection capabilities to the fullest extent possible.