Radio Transient Detection with Closure Products and Machine Learning ¹¹1Released on

One of the specific difficulties in this task is to separate suspected transient signals from the noise-corrupted sky background signal and radio frequency interference (RFI) without the production of a sky-image. Differential signal analysis, i.e. subtracting a reference from a target to remove the background, is a common strategy to study object variability in time domain astronomy (Riess et al., 1998; Bramich, 2008; Offringa et al., 2014; Stewart et al., 2016; Zackay et al., 2016). Instead of subtracting images, we difference the visibilities. The individual visibilities data dump, $V_{ij}(t_{k})$, is the output of correlation on the baseline between antennas $i$ and $j$. Hence the difference between the visibilities of two consecutive data dumps is:

This approach leaves residual sources that were not present in the previous time-step, largely removing the relatively slow-changing sky background and thus improving the transients’ detectability.

We use the closure product (also called bispectrum or closed triangle) as the desired feature of for machine learning, (Wilson et al., 2013), which is formed by multiplying three correlated data streams from different baselines on a closed loop. We can write this as:

where $V_{ij}(t)$ follows the notation from the above, and $V_{ik}(t)^{*}$ is the conjugate of the output of correlation on the baseline between antennas $k$ and $i$. This type of closure product is coherently sensitive to point sources anywhere in the FoV and is incoherent for local interference (Law & Bower, 2012; Cornwell, 1987; Kulkarni, 1989; Rogers et al., 1995). Thus in closure products, transients and RFI behave in different ways. The transient source is a delta function in the sky with no dependence on closure triangles, which would be the same for all closure products. RFI resolves on longer baselines as the source is located in the near-field of the array. Thus the signals do not have a planar wavefront, but are expanding ripples and the correlated signal degrading with antenna separation. The bispectrum is also calibration independent. It can be applied to improve the quality of visibility functions by canceling all instrumental residual phase terms, such as receiver delays and phases (Wilson et al., 2013), thus making it computationally efficient for the detection of transients in raw data from the correlator.

The major problem with closure products is that they are sensitive to signals only with significant SNR in the bispectrum per closure product. This is because the closure product is formed by the multiplication of several visibilities, which implies that we might lose some weaker transients. But in fact, it is impractical to attempt to observe such weak events in all cases. If the signals are too weak, it means that we need more collection time, which is not possible for transients and the signal would not reach detectable levels for any method. Therefore, we can expect only the non-faint pulses as detectable targets (see discussion in Dodson et al. 2014). This ensures that the closure product is the optimal responsive strategy to be applied in transient detection with radio interferometers.

The residual of the closure product in our data files could be defined as:

The characteristics of the bispectrum residual of noise, RFIs and transients in data segments are listed below. The real and imaginary components representation of the complex information in Figure 1 show the distinction among these three main catergories of components.

• •

  Noise: the complex of bispectrum reside near the origin of the real-imaginary plane.

• •

  RFI: the bispectrum of RFI shifts (potentially) to an arbitrary value and is widely spread across the complex plane, since the emissions of RFI are near-field or multi-path propagations, which generate random complex information.

• •

  Transients: the bispectrum of transient migrate toward positive real values with a small spread. This is true for all of the closure products, while such behaviour is not the case for RFI.

To detecting transients we trained a supervised machine learning model to classify the signal components from the data stream automatically and efficiently. Neural networks are a class of very flexible and non-linear models to perform classification tasks. The Artificial Neural Network (hereafter ANN) we use simply consists of an input layer, multiple hidden layers, and an output layer, which are all fully connected. In the prediction process of the model, when the input signal is received at the input layer, the information flows in the direction of the arrow in the network model diagram to the output layer unidirectionally, a process called “forward pass”. The outputs of the output layers are compared with their true labels to calculate a defined distance, bias or loss.

Our MLP architecture is listed in Table 1. To simplify our classification model as much as possible, we have only three hidden layers in our MLP model, and they are fully connected to the input and output layers. The ReLU is used as the activation function after each hidden layer; the Softmax is used as the activation function for the output layer. We use Pytorch to implement the MLP and accelerate the training of the MLP via NVIDIA RTX 3090 GPU. We used the Adam optimizer (Kingma & Ba, 2014) for training, which is an extension of the stochastic gradient descent method. The initial learning rate is set to $\alpha=10^{-3}$ and the batch size is set to 64. The MLP is then trained on the training set, and the training period is set to 1000 epochs when the model has reached convergence.

Unlike deterministic prediction with fixed weights, the Bayesian methods treat the model parameters as random variables when making predictions. Given a training dataset $\mathbb{D}$, a model distribution can be built that contains multiple weights with different parameters. This distribution is referred to in the Bayesian framework (Jaynes, 2003) as the posterior distribution, denoted as $p(\omega|\mathbb{D})$, where $\omega$ is a set of weight parameters of the model.

To present the prediction uncertainty of the model more clearly, we consider the distribution of the probabilities of the softmax output of category instead of their mere pseudoprobability values. The variance on the predicted category probabilities should be more informative than presenting the uncertainty of the integer labels. Let a new input be $x^{*}$, and the corresponding softmax layer output probability vector be $y^{*}$, which contains three prediction category probabilities, being given as

Logical analysis of the weight posterior distribution $p(\omega|\mathbb{D})$ is usually infeasible, especially for complex structures like neural networks. However, Gal & Ghahramani (2016) applies ‘dropouts’, a regularization technique, to analyze the uncertainty of a model and shows that it can be approximated as a Bayesian neural network. The dropout rate $p$ is a hyperparameter of the model, taking a value between 0 and 1, which reflects the probability of each neuron being dropped out during the implementation. The stochastic behavior of each neuron follows the same Bernoulli distribution. As mentioned above, neuron rejection generally occurs during the training phase. Here, the dropout is also turned on in the prediction phase. Performing multiple prediction passes completes the Monte Carlo sampling in the approximate distribution of $p(\omega|\mathbb{D})$, thus

where $T$ is the number of stochastic forward passes through the neural network using dropout. The above method is known as the Monte Carlo Dropout approximation. More specific details can be found in  Gal & Ghahramani (2016).

With the Monte Carlo Dropout, we can analyze the uncertainty of our model. By comparing uncertainties of models with different dropout, we can select the appropriate value of the dropout rate. In many papers on practical applications (e.g. Walmsley et al., 2020), the dropout value is chosen as 0.5 or 0.25 for convenience, and such a common choice does reduce the overfitting of the model. A value like 0.5 can give good classification results on more complex networks. However, networks without calibration containing dropout are not optimized. In particular, for networks with different applications, their structures and depths are not quite the same. If the same dropout rate is used for matching the uncertainty of the predictions without optimization, the optimal performance of the classifier cannot be guaranteed. Our investigation, in addition to finding the optimal dropout rate to maximize the score for detection, also provides probability matching.

Since the number of real transients on the aforementioned time scale is relatively low, we lacked labelled datasets containing a large number of short-duration transients to train our model, which means annotated data did not exist and thus we used simulations. We created a range of simulated datasets for use with our model. The one is purely simulated data, which was used to train the transient classifier and to investigate the feasibility of the dropout-base Bayesian network. The other was semi-real data, which was a real MWA observation that provided a background sky to which we added simulated transients to verify the performance of the well-trained neural network.

The telescope configuration used for the pure simulation matched the Australia Telescope Compact Array (ATCA). The ATCA has sufficient baselines to develop the model in a rapid manner. It would be trivial, but time consuming, to scale this work up to the SKA but there are no obvious benefits from this. We simulated observations of the six antennas of ATCA. In our simulations, we observed a local sky area of RA=01h11m13.0s and DEC=-45d45m00.0s using the central frequency of 1.42 GHz and a field of view of 15 arcmins wide. 100 different static sky models were produced in 10 surrounding frequency channels with a width of 0.1 MHz. Each sky model contains a random number of 4 to 8 sky background sources with random locations and varying strength. We randomly selected one of the 100 sky models and added a point source at a random location. The parameters of the transients include arrival time of the pulse and a random width (between 1 and 10-time steps). The RFI was simulated with a random (complex) number added to the baseline data at a random time step. The correlated data streams produced were then differenced, and then the closure product formed with the residual information (see Equation 3).

The noise in practice here contains two aspects. One is the “thermal noise” of the observational system. The other is the “measurement error”, which comes from the simple differencing method we implemented. “Measurement error” could be reduced by interpolating over points, rather than differencing. Although an interpolation method would be more accurate compared to the differencing method, in practice, the errors introduced by both are swamped by the “thermal noise”. We keep the “measurement error” and the “thermal noise” at a constant level and gradually suppress the signal-to-noise ratio to test the ability of the machine learning model. Our simulated “thermal noise” is generated by a Gaussian distribution with a standard deviation of 0.01 arbitrary units in strength, and the measurement error is kept at the same level of intensity as the thermal noise.

The high SNR transient is well characterized and easy to find, but the weak signals will pose a challenge to model training and prediction. In our experiments, we trained and tested several datasets with different SNR quality. Here we present only the dataset with the lowest SNR that maintains the high performance of the model, which we call the high SNR dataset; and the dataset with the lowest SNR that has significantly decreased the performance of the model, yet still can be a good training dataset to maintain a working model, which we call the low SNR dataset. For the high SNR dataset, the SNR ratio is 7$\sigma$, which includes a Gaussian distribution simulation with a transient source intensity of 0.1, “thermal noise” and “measurement error” of 0.01. For the low SNR dataset, the SNR ratio is 4$\sigma$, where the noise is kept constant, and the transient source intensity is reduced to 0.06.

In each observational data segment, time steps containing transient sources, RFI, and noise are extracted respectively to form an available group of simulated samples. For each of these sample data, we selected three frequency channels with 20 different closure products in each frequency channel. Since the machine learning model deals with real numbers, we separate the real and imaginary parts of the data and then “deconvolve” the 3 $\times$ 20 complex data points of each sample into a real vector of length 120. We set the closure product $CP_{1f}=a_{1f}+ib_{1f}$ in one frequency channel with 20 closure product components. Thus the input vector is:

We generated a total of 5,000 such sets of simulated samples for each category. They are labeled and then fused into a large dataset containing a total of 15,000 data samples. Three splits of the data into training, validation and test set, are required to train and evaluate the MLP classification. One-tenth of the data (which contains equal amounts of the three signal components of transient source, RFI, and noise) is taken as the validation set, another tenth is taken as the test set, and the remaining datasets were used as the training set.

In the first experiment, we will build up the simple ANN to perform a simple classification. Following the usual labeling convention, we label 0, 1, and 2 of three categories as noise, transients, and RFI, respectively. These are the hard integer labels for the samples.

In the second experiment, we improved the representation of categorical data in prediction analysis by changing the way to encode categorical variables. The predicted output of the model after the softmax activation layer is retained as a probability vector instead of the one-hot label. Meanwhile, we also use the probability vector soft labels to label the samples of the data. As implemented, we attribute a certain degree of uncertainty to the probability of each category in the vector after converting the one-hot binary labels into probability label vectors, which approximates the uncertainty of the data and facilitates the comparison between the predicted output of the probability vector and the true labels of the probability vector. Such manipulation allows the representation of categorical data to be more expressive and provides more detailed information than a single hard label.

For each training epoch, the probability values are randomly drawn from some probability distribution, which is the ground truth distribution of the soft labels. In practice, in each training epoch, for each instance, after converting its hard label into an one-hot label vector, the element at the position corresponding to its own category is a probability randomly extracted from the normal distribution (2/3, 0.01) as the value. In contrast, two probability values are assigned to the corresponding heterogeneous category from the normal distribution (1/6, 0.01) at the position of the two classes. Then the formed probability vector is normalized and corrected to satisfy the sum of 1 so that each instance is given a suitable soft label. Thus, the experimental procedure is to train the model from data with random probability soft labels. We propose to determine the optimal dropout rate by analyzing the statistical approximation of the probabilistic output distribution of each predicted data about the category to its original true probabilistic label distribution.

The probability output of the classifier for each instance includes three classes of probability distributions. To evaluate the performance of these probability outputs, we can examine the similarity of the output probability distributions to the input ground truth distributions in two different ways. The first is that we generalize the three categories of probability distributions by overlapping them into a one-dimensional curve of the output distribution. In this way, we can then use the commonly used KL-divergence that measures the distance between two 1-dimensional distributions. KL-divergence is defined as follows:

where $s$ is the soft label of ground truth and $\widehat{s_{t}}$ is the soft label of prediction.

However, applying only KL-divergence to analyze the overall distribution cannot take into account the origins of those sub-distributions. When the multiple distributions in the right panel of Figure 2 are superimposed into one distribution, the KL-divergence cannot distinguish between color class distributions (the left panel in Figure 2). This would result in the KL-divergence making the same judgments, even if we switch the labels corresponding to the color distributions (i.e., it would no longer match ground truth). Therefore, we added an evaluation metric to ensure a trustworthy analysis.

The second way is to keep the soft label output as a multivariate probability distribution and use the Dice distance (Dice, 1945) to measure the dissimilarity of the distribution across multiple dimensions (see representations in the right panel of Figure 2. Dice distance is a simple and useful aggregated measure of spatial overlap in a multi-category context in semantic segmentation. It is complementary to the Dice similarity coefficient (DSC) and is obtained by subtracting the DSC from 1 (Dice, 1945). There are multiple definitions for Dice coefficient (Milletari et al., 2016; Sudre et al., 2017). Since they are numerically equivalent, we choose one of the definitions, which is defined as follows:

where $\mathbf{p}\equiv{p_{i}}$ and $p_{i}\in\left[0,1\right]$ is a set of continuous-valued probabilistic outputs. And therefore the Dice distance is $1-D(\mathbf{p},\mathbf{l})$. In semantic segmentation, $p_{i}$ represents the probability vector of the $i$-th pixel. Here we can let it represent the set of predicted probability distributions for the location of the $i$-th element in the soft label, and $\mathbf{l}\equiv{l_{i}}$ is the corresponding set of ground truth label distributions. The Dice distance is minimum when $\mathbf{p}\rightarrow\mathbf{l}$.

The results of the neural network on the high SNR level (7$\sigma$) we reached can be seen in Figure 3(a), where the columns represent the instances of the predicted classes, and the rows represent the instances of the actual classes. The diagonal cells in these matrices contain percentage values, indicating the proportion of samples correctly assigned, while the non-diagonal cells contain error rate values. The model can do an excellent job in identifying noise and RFI, as they yield classification scores higher than 96%. The classification performance can be considered a promising result despite the confusing inference in distinguishing transients from noise. At such a low SNR level, our transient signal is classified as noise 19.8% of the time. On the other hand, if it is noise, there is only a 3.6% probability of being classified as transient and not likely to be confused with RFI. It clearly shows that our idea is effective for classifying RFI, transients, and noise.

Since the quality of the real data in the task to be classified is hardly the same as the one of the training data used for training the classifier, we used both high (7$\sigma$) and low (4$\sigma$) SNR training data to train the classifiers separately, and then cross-test the low and the high SNR validation datasets to explore the best model training and validating process. Specifically, for example, in Figure 3(b), during training with a high SNR (7$\sigma$) dataset, we used 12,000 samples for training and 1,500 ones for validating. The parameters of the model were saved according to the best MCC overall score, which includes all the entries (namely false positives, false negatives, true positives, and true negatives) of the confusion matrix in its calculation, and is generally regarded as a reliable, balanced measurement (Grandini et al., 2020). Then the inference was performed with a low SNR test dataset (including 1,500 samples). For Figure 3(c), the operation was just the opposite. And for the experiments on the diagonal, we performed regular training and testing with the same SNR dataset.

With the high SNR test data, we found that the model trained at low SNR (4$\sigma$) predicted transients with slightly higher accuracy than the model trained at high SNR (7$\sigma$) dataset (comparison between the centers of panels). While for the prediction of noise and RFI, the model trained at low SNR (4$\sigma$) is worse than the model trained at high SNR (7$\sigma$). Most of the reduction in noise prediction (96% to 94%) and some of the misclassification of RFI go into the confusion of transients, which causes an increase in false positives for transient prediction (comparison along the second column in Figure 3(a), 3(c)). Similar false detections of noise as transients are also more likely to occur when the SNR of the test data decreases (second column comparison in Figure 3(b) and 3(d). Although the prediction accuracy of transients is a metric of interest, we prefer to minimize the occurrence of noise and RFI false detection rates as transients. Therefore, we favour the model under high signal-to-noise data training.

Furthermore, comparing along the rows in Figure 3(a) & 3(b), and Figure 3(c) & 3(d), we find that the model trained with high SNR (7$\sigma$) also controls false positives well when applied to different SNR test datasets. In contrast, the model trained at low SNR (4$\sigma$) yields surprisingly higher false positive rates as the SNR of the test data increases. In order to tackle the slightly higher false negatives caused by noise and the slightly lower true positives for transients, we will suggest ways to improve the SNR of the observations operationally. Furthermore, a method of evaluating uncertainty in the model inference can also help us to exclude some unreliable predictions, as will be discussed next.

In this experiment, the batch size, initial learning rate, and stochastic gradient descent optimization method among the training retained the training settings of the original experiments, and the training process lasts for 1,000 epochs. After training, the training result of the iteration with the best global metric MCC performance is saved as the best model. We applied dropout to the prediction process by implementing 100 Monte Carlo forward passes, which meant we had 100 Monte Carlo approximation networks that produced 100 posterior responses to each categorical element in predicted probability vector for each test data.

Here for the dropout rate to be evaluated, we choose the random search method, which is a common method of hyper-parameter tuning for machine learning. When multiple hyper-parameters to be tuned in machine learning, there are tens of thousands of combinations required. Random search is a more efficient and fast strategy than the traditional grid search method (Bergstra & Bengio, 2012). We also apply this strategy to our study of finding the optimal dropout rate. Although it is a one-dimensional search, as shown in the result, it is the application of the random search strategy that allows us to find the optimal dropout rate in the logarithmic space of parameters more quickly.

Figure 4 shows the KL divergence of our 20 random points with dropout rates between 0 and 0.5. These points include the 10 points found by random search and the 10 points taken out equivalently in their logarithmic space after localising to a smaller interval. Lower KL divergence and Dice distance values represent the more similar the two distributions are. We can see from the figure that the overfitting caused by the too-small dropout rate makes the scatter of the predicted posterior distribution with the ground truth distribution large, and the KL divergence value reaches a stable minimum as the dropout increases. Then, as the dropout rate increases, it leads to a lower capability of the model, which in turn leads to a predicted distribution that does not match the ground truth distribution. We find that the dropout can be taken to have the minimum scatter at a low SNR (4$\sigma$) equal to 0.03; while the optimal dropout value to minimize the scatter is 0.021 for high SNR (7$\sigma$), and the range of the optimal dropout can be extended to 0.021-0.3 for high SNR (7$\sigma$). The curves of the Dice distance also supported this suggestion for the dropout rate choice.

Next, in order to qualitatively provide examples of the posterior distribution matching, we randomly selected one example from each category of the noise, transient source, and RFI, and compared the posterior results for each category when we took the optimal dropout value as well as the dropout = 0.5. Figure 5 show the high SNR results, and Figure 6 show the low SNR results.

As an example, in Figure 5, each row is a randomly picked noise, transient, and RFI datum, respectively. The first column on the left shows the signal examples of closed products in the complex plane. The second column shows the predicted posterior distribution of soft labels after 100 forward passes when the dropout takes the optimal value of 0.021 (the horizontal axis indicates the probability values of the possible posteriors, and the vertical axis indicates the corresponding PDF). The fixed orange probability distribution curve (center value of 2/3 and variance of 0.01) is the distribution of the maximum pseudo-probability values in the real soft labels of the test data, i.e., the pseudo-probability values of the transient category locations; the other orange distribution curve with the center value of 1/6 and variance of 0.01 is the distribution of the corresponding noise and RFI location in the ground-truth soft labels of the test data. Blue is the probability posterior distribution of the category of transient source in the predicted soft labels; red is the probability posterior distribution of the noise category in the predicted soft labels; green is the probability posterior distribution of the RFI category in the predicted soft labels. The third column is similar to the second column, but it is the predicted results for dropout=0.5. Note that the Gaussian probability distribution of soft labels does not represent the physical properties of each signal component, and the prior information of such labels is only set to Gaussianity here for the convenience of the simulation. Ultimately, the degree to which the predicted PDFs of the soft labels match the ground truth PDFs is examined.

By examining the independent examples, it can be seen that the centers of all posterior distributions are very close to or overlap with the ground truth distribution, when dropouts are taken to their optimal values in both high and low SNR cases, and the uncertainty of each distribution is also very similar to the ground truth distribution. Excellent matching is obtained for the inferences of transient source and RFI when dropout is taken as 0.021 in the high SNR case (7$\sigma$). It also holds for vast majority of their counterparts when dropout is taken as 0.03 in low SNR case (4$\sigma$), but some individual events become confused with noise. See Figure 6. This suggests that maximizing the SNR of the training data can improve classification accuracy. As for the noise inference, although it does not fit perfectly, the centers of the distributions of the noise category are also close to the center of the distribution of the ground truth soft labels, and its uncertainty is basically the same as the ground truth distribution. It is worth noting that the posterior values of the noise category for the two SNR levels discussed here (Figure 5 and 6) do not perfectly reproduce the input ground truth, and the results in the low SNR case is worse than that in the high SNR case; however they are significantly better than the results for a drop out of 0.5. This is additional evidence that high SNR data can improve the model’s capability.

Further, and more importantly, the models with the best dropout values for both high and low SNRs gave overall better predictions than the model with a standard dropout rate of 0.5. Dropout rates taking 0.5 typically show inconsistency with the ground truth distribution and more significant uncertainty in the posterior distribution. For the former bias, either can be seen in Figure 5, when transient sources are classified with RFI at high SNR (7$\sigma$), which shows incredibly high confidence with deviation from the actual distribution (output classification probability values close to 1, coupled with small uncertainty), or can be found as more mis-classifications (transient is incorrectly classified as noise in Figure 6). With regard to the latter variance, the greater predicted uncertainty can be mainly seen in both plots. It indicates that with the dropout rate set to 0.5, all these inferences behave far from the ground truth. This suggests that a uniform standard dropout rate value does not provide correct posterior distribution matching in a given scenario and with a given neural network structure, and that calibrating the dropout rate is a necessary step.

In our previous experiments, we built and optimized artificial intelligence networks to achieve high detection rates for test signals in a high discrimination feature space of closure product signals under simulated Gaussian noise and simulated transient source conditions, as well as to give predictions similar to ground truth labels. Due to the difficulty in obtaining large amounts of pertinent real labelled data, we can further test our trained models with more challenging data.

In this experiment, we consider observational data generated from a wide field-of-view transient source observation scenario using the MWA array. The dataset was constructed as a semi-real dataset containing real sky noise and a simulated transient model of single rising and falling flare. It is designed to mimic the recent discovery by Hurley-Walker et al. (2022). In this demonstration, we set aside RFI. In the experiment presented here, we simulated 26 separate synthetic observation files each containing a transient source in the same sky noise background. The transient rose from zero to 8$\sigma$ over the sky noise on the 4th time step, then a further 4, then 2 and then 1$\sigma$ in the subsequent time steps. Then it rapidly fell by 8, then 4, 2, and 1$\sigma$, back to zero. This profile of inverse exponential rise and exponential decay we called the Shark-Fin. Each data file contains a single transient source in different locations that were spread over the sky to reflect more realistic observational scenarios. At no time were these new MWA data included in the previous model training or validation experiments.

We tested this semi-real data with our best model, which had a dropout rate of 0.03 and was trained by the data with soft labels. Using the soft-labelled training data with slight uncertainty as augmented data will result in a more robust final model, and the inferred best confidence output made with this model would be close to the previously selected probability distributions for soft-label category elements, which is centered at 0.66 as well as the ones centered at 0.17.

In our experiments, we needed to manipulate the test MWA data prior to input to fit the requirements of the model’s input dimension. The MWA data here are observed by 101 effective antennas with 512 frequency channels, which form 101 $\times$ 100/2 baseline pairs and 101 $\times$ 100 $\times$ 99/6 closure products. To best fit our fine-tuned model’s input requirements and keep a high level of sensitivity of the signal, we made the following adjustments to the MWA test data. The 101 $\times$ 100 $\times$ 99/6 closure products were combined into 20 composite ones, binned to have an equal number in each bin, sorted on perimeter length. The 512 frequency channels were combined into three composite channels. However, if we had done this directly, the signal of the transient source far from the center of the observed phase of the MWA would have been severely diluted. The further away from the center, the more severe the signal loss. This is due to bandwidth smearing caused by the averaging of the frequency channels, and signals from longer baselines lose a significant amount of sensitivity. To reduce the loss of effective field of view due to the frequency averaging, we maximized the pre-averaging of the baseline channels by 16 to 32 composite frequency channels and then to form the composite closure products (each integrated closed product contains approximately 8332 original closure products). We expect the transients around the phase center to work well, but perhaps not so successfully for those further from the centre. In this demonstration, we hope to learn about the detection limit as a function of the positions of transient sources as well.

Figure 7 gives confidence curves for the evolution of the prediction categories over observation time for a part of 26 files containing transient sources. For the 3rd differenced time step at all angular separations, the model gives a steep increase in the probability that this signal is a transient source, which means that the appearance of the transient source is accurately predicted. However the weaker transient signals are only detected for the smaller angular separation. During the decay of the transient source the (7th differenced time step), the transient is misidentified as a RFI spike. This is because the simulation used for training had no labels for decaying signals; there was a transient or there was not. Comparing Figure 7 (left) with Figure 7 (right) which is for a dropout of 0.5, we see that it gives correct predictions for the appearance of transient sources close to the phase center, but the predictions have much greater uncertainties.

Figure 8 gives the model prediction of the transient source as a function of angular separation and as a function of SNR. The model predicts the highest probability of the transient source when it appears at the observational phase center, for all SNR. For the strongest transient (8$\sigma$), the probability decreases as the transient source’s location gradually moves away from the phase center, and the uncertainty gradually increases until the signals at the edge fail to be recognized as transient. For the mid-strength transient (4$\sigma$) the transient is detected close to the phase centre, but not at larger separations.

We need to point out here that the current model does not accurately predict the falling portion of the transient signals even for the source in the phase center, but this is not to say that our solution is wrong or pointless. Instead, it is because we focused on the rising of transient sources during the initial training process, which is the only one that matters. Remember that in Equation 3, each timestep of closure product difference is generated by consecutive time instances $t_{k}$ and $t_{k-1}$. Only transient samples with positive features are labelled in our simulated training data and our current model can only detect the positive differences. In reality, as long as we can accurately detect the onset of transient sources, the fall is not that important.

Therefore, we conclude that the optimal choice for one of the hyperparameters, the dropout rate, obtained by MC dropout approximating Bayesian optimization, allows the model to give small uncertainty and accurate inferences when performing transient source detection. The success of this experiment is significant for the practical construction of workflows. Rarely do we have a large number of similar yet labelled observations from different observation projects with different configurations and conditions for a neural network model to learn. In contrast, we can now build up a model by simulating close to real data. The resulting best model can perform efficient inference tasks in more severe scenarios, rather than requiring labelling to acquire large training sets of real observational data.

Given that the model can be changed and retrained for its input features, subsequent increases in the number of frequency channels will further reduce the effect of bandwidth smearing and allow for a larger effective field of view available for inference, GPU computing power permitting.