Out-of-Distribution Dynamics Detection:
RL-Relevant Benchmarks and Results

The OODD benchmarks are all derived from common RL benchmark environments. These include three OpenAI Gym classic control environments (Acrobot, Cartpole, and LunarLander), and four Bullet physics environments (Ant, Hopper, HalfCheetah, and Walker2D). For all environments, the observations used for OODD are the continuous environment state features, which vary in dimension depending on the environment (see Table 5), ranging from 4 (Cartpole) to 28 (Ant). We refer to the each unaltered environment using the default parameters as the nominal environment, which is governed by the nominal dynamics and nominal observations.

In order to generate environment trajectories that persist for non-trivial durations, we require control policies that achieve non-trivial performance. For this purpose, we train good policies using the IQN algorithm (Dabney et al., 2018) for the OpenAI Gym environments and TD3 (Fujimoto et al., 2018) for the Bullet environments. This set of policies, one per environment, are referred to as the nominal policies.

Finally, for each environment, we generate 10,000 nominal trajectories, each a sequence of observations, by executing the the nominal policy in the nominal environment for a finite horizon. These trajectories characterize the behavior of each learned policy under normal environmental conditions, through the lens of observations. If the test time trajectory distribution matches the nominal distribution, then test time performance can be expected to mirror the nominal performance. The goal of OODD is to detect when test-time conditions change by noticing anomalies, which may forecast a possible degregation in policy performance.

For each environment, anomalous trajectories are created by first executing the nominal policy in the nominal environment until a random time point and then switching to an anomalous variant of the environment and continuing executing until a horizon. As described below, we consider two main classes of anomalous variants: sensor-injected and dynamics-injected. Sensor-injected anomalies correspond to different ways of corrupting the environment observations before sending them to the policy, which can model different types of sensor failure and degradation. Note that such modifications can impact the behavior of the controllers, which in turn can impact the evolution of the trajectories. Dynamics-injected anomalies directly change the dynamics of the environment by modifying key physical parameters of the environment simulator.

Our benchmarks include the following 8 types of anomalous variants for each environment–4 sensor-injected and 4 dynamics-injected. Table 1 provides relative details for each environment and anomaly type described below.

IID Noise. One or more observation sensors is corrupted by IID Gaussian noise. For each environment, the mean and standard deviation are selected as discussed below to avoid near-term failure.

Sensor Shutdown. One or more sensors are clamped to zero immediately after the anomaly.

Sensor Calibration Failure. One or more sensors are multiplied by a constant throughout the anomalous run.

Sensor Drift. Relative to the time step in a trajectory, a small amount of noise is injected into the chosen sensor(s). As the trajectory progresses, the magnitude of the injected noise increases.

Wind Simulation (L2R and R2L). Simulated winds starts to blow after injection from either left-to-right (L2R) or right-to-left (R2L), which impacts the environment dynamics. Wind is simulated in the discrete action environments by changing a percentage of the left/right action to noop and reinforcing either the right/left action depending on the direction of the wind. For the continuous-action Bullet environments and appropriate value is added to the action to approximate the impact of wind.

Gravity Manipulation. The value of gravity is changed from the nominal value of 9.8 by a magnitude that depends on the environment.

Manipulating Components. Each agent has some physical characteristics, for example, its components’ length and masses in Acrobot and CartPole, its engines’ powers in LunarLander, or its power in Bullet physics environments. These anomalies randomly change the value of one or more attributes within pre-defined environment-dependent ranges.

Importantly, we avoid injecting anomalies that cause the nominal policy to completely fail soon after injection (e.g., falling or crashing). This is because such failures are trivial to detect and do not allow testing the ability of detecting anomalies before “disasters”, which is a primary goal. To achieve this, for sensor-injected anomalies, we only modify a small percentage of the features (at most 20%) chosen for each environment based on tests with the trained controllers. Each anomalous trajectory is generated by randomly selecting the appropriate size subset of features to modify, which provides diversity across the anomalous trajectories. For the dynamics-injected anomalies, we change the physical characteristics of each environment with an environment-relevant value based on pilot tests. Table 2 gives the average reward over 1000 trajectories of the nominal policy in each nominal and anomalous environment. We see a decrease in reward for anomalous environments but not catastrophic levels of decrease. We see that the dynamics-injected anomalies tend to lead to larger performance degradation than the sensor-injected anomalies.

Recurrent Implicit Quantile Networks. To capture aleatoric uncertainty, we ideally want a model that provides a distribution over future observations rather than just a point estimate. For this purpose, we draw on Implicit Quantile Networks (IQNs) (Dabney et al., 2018), which were proposed to represent value-distributions in distributional RL. IQNs learn an implicit representation of a value or feature distribution using the quantile Huber loss. At a functional level, once trained, it can be used to generate samples from the learned distribution conditioned on the input. We refer the reader to the original paper for detailed coverage of IQN. Our model is a recurrent variant of IQN (RIQN) and Figure 1 illustrates the architectural differences compared to IQN with additional details in Appendix B. At a high-level RIQN differs from IQN in three ways: 1) Since we are interested in dynamics prediction in problems where observations may not fully capture the state (e.g. nominal policies can have internal memory), we add recurrent memory to the IQN network via a GRU (Cho et al., 2014) layer. We found this to be an important extension, which is evaluated in Appendix 4.2 by a memoryless version of the model. 2) Rather than predicting action values from state-action pairs as input, we train an RIQN for each observation feature to predict the feature distribution at time $t$ based on the previous values of the features. 3) Rather than training via noisy RL-derived target values, we follow a supervised learning approach.

More specifically, for supervised learning, we minimize an asymmetric variant of the Huber loss (from prior work (Dabney et al., 2018)) on predictions of each feature value at time $t$ given all observation before time $t$. In addition, since we use the RIQN to generate auto-regressive predictions of future observation sequences (see below), we potentially face the problem of compounding prediction error. That is, since the network’s outputs are fed back to the input during auto-regressive prediction, error accumulation can occur at test time. To address this, we use the Scheduled Sampling approach (Bengio et al., 2015), which conducts training using inputs from ground truth and auto-regressive samples.

For anomaly detection, when $\Delta>1$, we use the RIQN to generate predicted future observations for $\Delta$ time steps. At time step $t$ given prior observations $o_{1:t}$, we consider two ways to generate a future prediction sequence $\hat{o}_{t+1:t+\Delta}$. First, use the RIQN to sample $\hat{o}_{t+1}$ given $o_{1:t}$, followed by sampling $\hat{o}_{t+2}$ given $o_{1:t};\hat{o}_{t+1}$, and so on until sampling a value for $\hat{o}_{t+\Delta}$. By repeating this process for RIQNs, we can generate multiple sampled sequences that for any time $t+\Delta$ yields a set of samples $\{\hat{o}^{(i)}_{t+\Delta}\}$ that can be used to represent the predictive distribution at time $t+\Delta$.

The second approach we consider is referred to as mean sampling (RIQN+MS), which is intended as a lower variance alternative to pure auto-regressive sampling. This proceeds like regular auto-regressive sampling, but at each step $t+k$ a set of $M$ samples for $t+k+1$ are generated. Next the mean of those samples is computed and used as the next observation for auto-regressive sampling of time step $t+k+2$. This approach also yields a set of observation samples for each time-step, but typically has less variance due to collapsing to the mean during sampling.

Non-Probabilistic Networks. In contrast to RIQNs, non-probabilistic networks (NPNs) output a deterministic prediction value at each time step. Since NPNs do not learn distributional representations implicitly, they can be learned using a simple mean-squared loss. Our NPNs have a similar recurrent architecture to the RIQNs, except that they output only a deterministic value given an input. This is achieved by removing the last fully-connected component and the cos transform from the RIQN in Figure 1b. NPNs can be used to generate deterministic predictions of future trajectories via auto-regressive sampling.

Random Forest. We also use the well-known random forest model (Breiman, 2001) for predicting the environment dynamics. The model is trained to take a history window of recent observations and predict the next observation. As with NPNs future trajectories can be predicted through auto-regressive sampling.

The above RIQN model accounts for aleatoric uncertainty, while none of the above models directly account for epistemic uncertainty. For purposes of anomaly detection, it can be argued that epistemic uncertainty is critical, since it more directly captures out-of-distribution observation sequences. We consider a common approach to addressing this issue by using ensembles of the above models instead of single models. Each model (either RIQN, NPN, or RF) is trained starting from different initial conditions and different hyperparameter settings, attempting to encourage diversity. In this work, we did not attempt to optimize the ensemble training approach, nor did we attempt to quantify how well the ensembles capture epistemic uncertainty.

Given an ensemble of $e$ dynamics models, an observation sequence $o_{1:H}$, and a detection horizon $\Delta$, we can now define a simple anomaly score to assign to any time $t\in\{\Delta,\ldots,H\}$, denoted by $A_{t}$. First, use each model model to produce a set of $M$ auto-regressive trajectory samples ($M=1$ for NPN and RF) at time $t$ given the sub-sequence $o_{1:t-\Delta}$, resulting in a set $\{\hat{f}_{t}^{(1)},\ldots,\hat{f}_{t}^{(M\times e)}\}$ of $M\times e$ samples. Given these samples the anomaly score is given by: $A_{t}=\frac{1}{M\times e}\sum_{i}\left|f_{t}-\hat{f}_{t}\right|_{1},$ which is just the average L1 distance between the samples and the actual observation at time $t$. There are many other ways to define an anomaly score given an ensemble of models, and we leave a deeper comparison for future work. Our initial investigation did not identify an approach significantly better than the above.

Part of the evaluation in Section 4 involves evaluating the raw anomaly detection scores (via AUC) and does not require an anomaly detection mechanism. When evaluating detection performance there is a choice of how to combine anomaly scores across time to trigger an alarm. In this work, we use the classic cumulative sum (CUSUM) method (Page, 1954) for this purpose. We used a CUSUM threshold of 0.01 and drift of 0.0018. The threshold sets the amplitude value for the change in the data, and the drift term prevents any change detection in the absence of change.

A common anomaly-detection metric is the area under the ROC curve (AUC), which is independent of detection thresholds and depends only on the quality of the anomaly scores of a detector. Here we evaluate the anomaly scores of our OODD baselines via AUC. To do this, for each test trajectory we calculate the anomaly scores for each time point. Since each an anomaly is injected at a random time in each test sequence and persists to the end, we can also label each time point by whether it is nominal or anomalous. This allows for computing the AUC for each test sequence and then reporting the average AUC across all sequences.

Overall AUC. Figure 2 shows results using RIQNs for detection horizons $\Delta=1,10,20$ for the different environments and different anomaly types. We see that, with respect to AUC averaged over anomaly types (last bar in each group), the performance for different horizons ($\Delta=1,10,20$) are quite similar and that the baseline RIQN approach yields non-trivial overall performance across the benchmarks. However, for the majority of anomaly types it appears that there is significant room for improvement in future work.

Impact of Horizon. Here, we consider the performance RIQN for the individual types of anomalies when the horizon $\Delta$ is changed. First, consider sensor-injected anomalies, where the sensor-drift variant is much more slowly evolving than the other types. This suggests the expectation that longer horizons would be more advantageous for sensor-drift compared to the others. The results in Figure 2 agree with this expectation showing that for $\Delta=1$ sensor-drift leads to significantly worse AUC compared to other sensor anomalies. Further, we see that increasing $\Delta$ significantly improves performance for sensor drift. In contrast, the other sensor-injected anomalies all have immediate impacts and in the vast majority of environments the AUC is best at $\Delta=1$. Indeed we see that for the sensor shutdown anomaly, the performance degrades significantly for larger values of $\Delta$. This can be partly explained by the fact that larger values of $\Delta$ cause a smearing effect of the anomaly score, which can decrease the AUC. These results indicate the importance of varying the qualitative types of anomalies in OODD benchmarks to highlight differences in approaches and parameters.

Regarding the dynamics-injected anomalies, there is less variation in AUC with different horizons $\Delta$. However, there does appear to be a small advantage to using $\Delta=1$ for these anomalies. This indicates that the dynamics anomalies we have introduced may have quite sudden impacts that make longer horizons unnecessary. On the other hand, there is significant room to improve the AUC in all cases, which could suggest that our current baselines are unable to leverage the larger $\Delta$ values to improve performance.

Comparing Dynamics Models. Next, Figure 3 shows the AUC performance of different models (RIQN, RIQN+MS, NN, and Random Forest) averaged over all of the benchmarks. We see that for all horizons $\Delta$ that RIQN (without mean sampling) perform the best overall, followed by RIQN+MS, then NPN, and finally Random Forest. The result that RIQN and RIQN+MS outperforms NPN, suggests that there is an advantage to using a probabilistic model that can directly capture aleatoric uncertainty. The observation that RIQN outperforms RIQN+MS indicates that the variance reduction induced by mean sampling has an overall negative impact. The increased diversity of sampled trajectories from RIQN appears to allow for the L1 distances to better discriminate anomalies. The difference between NPN and Random Forest is likely due to the higher expressiveness of the NPN model and also that the NPN model is recurrent and hence can learn to selectively attend to the history. Rather, Random Forest is inherently an atemporal model and requires processing a history window of predetermined size to make its predictions.

The difference between NPN and Random Forests was suggestive that memory is an important element to include in the dynamics models. However, there are too many other confounding difference between those two models. To better observe the impact of memory we trained a model that was identical to RIQN except that: 1) the RNN memory cells were removed, and 2) a history window was provided as input to the model rather than just the current observation. We refer to this model as N-RIQN and a comparison to RIQN is provided in Table 3 for two representative environments averaged across anomaly types. We see that N-RIQN is significantly worse in terms of AUC compared to RIQN for both horizons shown, noting that $\Delta=20$ is qualitatively similar. This provides further evidence that recurrent models with memory are advantageous for OODD. We note that a reasonable effort to improve the performance of N-RIQN by varying certain hyperparmeters was not successful, though future work may identify non-recurrent architectures that exceed the performance achieved in this paper.

Impact of High-Level Anomaly Types. In Section 2, we introduced two primary types of anomalies, sensor-injected and dynamics-injected, which have a total of 8 different anomaly types. From Figure 2 we can see a clear trend that for RIQN the dynamics-injected anomalies lead to significantly worse AUCs compared to sensor-injected anomalies.

For a more quantitative analysis across dynamics-model types, Table 5 gives AUC performance (in addition to other metrics) for the different models when averaged across sensor-injected anomalies and averaged across dynamics injected anomalies. Further, the columns labeled AVG provide the average performance across models for the two high-level types. We see a clear trend that across all environments, for individual models and the averages, the dynamics-injected anomalies lead to significantly worse AUCs compared to sensor-injected anomalies. This suggests that better handling dynamics-injected anomalies and understanding the challenges they pose is a key area of future work in OODD.

Impact of Ensemble Size. To study the effect of having an ensemble of models instead of just one model, we experimented an ensemble of five models versus only one model. According to our experiments, we can see that the ensemble perform better regarding the obtained AUC compared to only one model, as presented in Table 4. The reason for such an improvement is because with an ensemble of models, it is easier to address the epistemic uncertainty.

In many applications, it is important not just to provide anomaly scores but also to raise an alarm when an anomalous shift in dynamics is suspected. The alarms should have a minimal delay, measured by the number of time steps after injection the alarm is raised, to allow for time-critical interventions when necessary. A small delay, however, must be traded-off with the false alarm rate (FAR), which is the fraction of nominal time points where an alarm is raised. FAR is a traditionally challenging metric in anomaly detection and it is critical have small FARs to support use in practical applications. Here, we use the sequence of anomaly scores for an observation sequence to produce alarms using the classic cumulative sum (CUSUM) method (Page, 1954) (see Section 3.2). Quantitative results for the average delay and FAR for each environment averaged across the high-level anomaly types (sensor-injected and dynamics-injected) are reported in Table 5. In general we see that the FAR rates are not particularly sensitive to the high-level type of anomaly. Given that the AUCs were sensitive, this suggests that false negatives may be dominating the AUC scores. Overall, the FAR values are quite large (on the order of 10%) for practical applications and improving the FAR metric by orders of magnitude is a critical direction for future work. Further tuning of the CUSUM algorithm may be able to achieve some improvement without significantly impacting other metrics, but it is likely that fundamentally different approaches will be needed. The results for $\Delta>1$ (not shown) are qualitatively similar.

Comparing delays across anomaly types we see that there is typically slightly more delay for dynamics-injected anomalies, which corresponds to the relative AUC performance. The results for $\Delta>1$ (not shown) are qualitatively similar, except that the delays are typically large by the constant $\Delta$. That is, the inherent increase in delay implied by using a larger $\Delta$ did not payoff in terms of reducing the FAR, AUC, or overall delay.