Sensor Control for Information Gain in Dynamic, Sparse and Partially Observed Environments

The problem involves finding and tracking multiple signals that have activity patterns of varying complexity involving several unknown parameters. A controller agent is trained and tested using multiple episodes (trials), where each episode has a new signal environment sampled from the same stochastic environment model (spec), and the episode involves a sequence of control (frequency selection) and observation (signal detection) steps. The specific signal patterns of each new environment are unknown to the agent and must be inferred through tracking and monitoring during the interaction. To solve this problem, we extend the original DAN architecture and rewards.

Problem Formulation. Following the DAN approach in (Satsangi et al., 2020), we consider a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., 1998) to model the dynamics of our RF environment. We denote $s\in S$ to represent the hidden state of the environment, $y\in Y$ to denote a target variable of interest that depends only on $s$, $z\in\Omega$ to denote a partial observation that is correlated with $y$ and $a\in A$ to denote the action taken by the agent. At each discrete timestep, $t$, the agent takes an action $a^{t}$, the environment transitions to state $s^{t+1}$ and the agent receives an observation $z^{t+1}$. The goal of the agent is to correctly predict the target variable $y^{t}$ given the history of previous actions and observations denoted by $h^{t}=(a^{0},z^{1},...,a^{t-1},z^{t})$. We denote by $\hat{y}$ the agent’s prediction of $y$. At each step, the agent receives a reward, denoted by $r(y^{t},\hat{y}^{t})$, that indicates how similar $y^{t}$ and $\hat{y}^{t}$ are.

RF Environment. In the context of RF spectrum monitoring, $s$ represents the state of the whole RF spectrum, that contains a set of unknown entities interacting with each other and transmitting signals spread across frequency and varying in time. Further, $y$ is the vector representing signal activity ($0$ for none and $1$ for activity) at all frequency bands, $a$ represents the frequency band(s) that the agent samples, and $z$ represents observed detections (the presence/absence of signals at the sampled bands); $h$ is the history of sampled frequencies and observations. The agent’s actions can only sub-sample the spectrum at specific frequencies. The goal of the agent is to select actions at each timestep in order to accurately report the presence/absence of signals in all frequency bands at that timestep. Fig. 1 shows an example of a partially observed RF environment. One or more bands can be active (contain signals) at any given time. Here, a sequential scan is performed across the spectrum, which misses the short signal bursts around $1{,}200$MHz.

Environment Spec Details. An environment spec defines an environment population model as a range of possible values for the following parameters: (i) number: number of interacting signal pairs; (ii) width: total number of spectral bands spanned by the pairs; (iii) period: period of the signal pair interaction; (iv) duty cycle: duration of one signal over the other during an interaction cycle; (v) frequency: the frequency of the lowest of the signal pair; (vi) start: timestep when signal pair appears in spectrum. Table 1 lists some of the environment specs used in our experiments.

Deep Anticipatory Networks (DANs) (Satsangi et al., 2020) were developed for information-gathering tasks under partial observability e.g., tracking people using cameras. A DAN makes control decisions that maximize the information gathered (minimize the uncertainty) about a target variable of interest $y$ that cannot be fully-observed. The key point is that, for this type of task, DAN avoids complex belief state updating and instead uses state prediction rewards to guide the agent behavior.¹¹1This has been demonstrated in Satsangi et al. (2020) to be equivalent in effect to belief updating. DAN uses two neural networks to train a policy, as shown in Fig. 2. The first, $\bm{Q}$, takes the history of previous actions and observations, $h_{t}$, and produces $Q$-values for the different actions. A second network, $\bm{M}$, takes the history $h_{t}$ plus the last action and observation to predict $\hat{y}$, i.e., an estimate of target variable $y$. $M$ is trained in a supervised manner with ground truth data, i.e., using the actual target values $y$. $Q$ is a standard value function network trained using RL but here it is rewarded when $M$’s predictions are accurate, i.e., when $\hat{y}$ is similar to $y$. $Q$ is trained to maximize cumulative discounted reward, $\sum_{t}{r_{t}\gamma^{t}}$. The $Q$ and $M$ networks are trained simultaneously and, after training, only $Q$ is used for autonomous control for information gain. In all of our DAN variants, $M$ is trained using weighted binary cross entropy (WBCE) as a loss, weighted to emphasize our metrics; see the discussion on rewards and metrics in Sec. 3.2.2.

Often, the lab spec used for training does not entirely cover the field (ground-truth) population of environments — for example, the domain knowledge employed may be outdated. To address this challenge, we introduce an experience feedback loop. Once trained on the lab spec, the agent is then deployed in field environments (here, simulated by a field spec) for a limited time, during which it collects samples that might be sparse. In this fielded phase, unlike the lab training, the agent does not have access to the underlying full state required to train the $M$ network via prediction loss. Rather, these experiences are used to estimate the environments’ dynamics, which are then used to update a lab spec/model. The agent is then retrained using the updated spec/model and re-deployed for more samples, and the process repeats for a number of times. We develop two model-based RL approaches to implement this feedback loop: (i) field spec estimation uses collected experiences to estimate the field spec parameters directly , which is then sampled to retrain the controller; (ii) field state estimation uses them to build a deep, generative model of the field state sequence population, from which we sample to retrain the controller.

We hand-coded four controllers that follow simple, but often effective, behavior rules for RF environments to serve as baselines to evaluate the DAN-based system robustness:

Random:

randomly selects bands in the spectrum. For prediction, it uses a persistent state scheme: each band’s signal state (active/inactive) is set to the last observation in it. Initially, all bands are considered inactive.

Scan:

sequentially selects bands in the spectrum, uses the same persistent state scheme for prediction.

Scan-and-Dwell:

estimates the different spec parameters governing the dynamics of the environment from the sampled observations. The controller is given ranges for the parameters but does not know their true values. It traverses the spectrum sequentially and whenever an action triggers a signal detection, it subsequently samples the same band until the true parameters controlling the dynamics in that band are learned, via a process of elimination. The predicted state is computed for each from the estimated parameters.

Expert:

uses the same process as the above controller to estimate the spec parameters from experience. At each step, it selects the band with the highest associated uncertainty, i.e., whose current range of possible values is the largest.

In particular, the Scan-and-Dwell and Expert controllers serve as reasonable upper bounds on performance since in our experiments they are given relatively small ranges for the different spec parameters around their ground-truth values. In other words, an ML agent that has been trained in environments whose spec parameters are similar to those governing the environment on which it is being evaluated can be expected, at best, to attain a performance similar to that of the best hand-coded controller.

Fig. 6 compares the performance of hand-coded controllers and ConvLSTM-DAN trained in only one environment spec (SpecA) and tested in multiple environment specs (SpecA and SpecB1). In Fig. 6(a) the controllers were tested in environments from SpecA. The Expert and Scan-and-Dwell controllers (which are given SpecA to estimate each sampled environment parameters) correctly predict the signal from $~{}t=35$. ConvLSTM-DAN achieves a good performance because it was trained on SpecA environments. Then, when tested on SpecB1 environments while being given SpecA to estimate the parameters (Fig. 6(b)), the hand-coded solutions fail to correctly predict the out-of-distribution signals. In contrast, the ConvLSTM-DAN, trained only in SpecA, shows more robustness to changes in environment dynamics.

ConvLSTM-DAN vs. original DAN. We compare our ConvLSTM-DAN model (a shared architecture) with the models used in the original DAN: Separate Dense ($M$-net and $Q$-net are separate networks with dense layers followed by an LSTM layer) and Shared Dense (a single network with shared initial layers and separate outputs for $M$-net and $Q$-net). The ConvLSTM-DAN layers can scale arbitrarily with respect to number of frequencies without a corresponding increase in the number of model parameters and as a result it converges much faster to a better accuracy as shown in Table 2.

Predictive, InfoMax vs. ConvLSTM-DAN. We evaluate their performance in environments with distinct challenges: (i) Stationary (simplest): a range of patterns (period, duty-cycle) and signal numbers (1-3); (ii) Non-stationary (more challenging and realistic): narrower range of patterns, but signal patterns and locations randomly change during episode; (iii) Multi-class wide spectrum environment (most realistic): multiple signal classes with different behaviours (signal modulations), aperiodic, random activity and more bands (100 vs. 20). In (iii), observation and internal state includes objectness (is signal present) and signal class score (softmax score vector). Also added multi-class loss, metrics and rewards.

Table 3 shows the results of our approaches. Notice that InfoMax performs best in stationary environments, where the signal parameters vary a lot from episode to episode, but within the episode, they are stationary. Predictive-DAN is best at non-stationary environments, and RL methods (as opposed to non-RL based InfoMax) seem to perform best there. InfoMax-DAN seems to perform best at complex multi-class environments. It also shows that our designs can scale in frequency.

We now cover our results in leveraging experience feedback via field spec estimation and controller retraining using the estimated spec, while field state estimation is covered in Sec. 4.4. For these experiments, our Predictive-DAN is the ML-based controller used.

Estimating Spec from Feedback. In Fig. 7(a) we compare our approach (4. Field Est on Field) to a scan-and-dwell hand-coded control where field spec is within-distribution (5.Expert on Field) and also to our ML controller alternatively trained (1.-3.). The hand-coded controller takes a long time to reduce uncertainty (large search space). A DAN controller trained on lab alone cannot cope with field (2. Lab on Field). By learning estimated field spec without GT, our approach has similar performance to training on field spec with GT (3.Field on Field, see also analogous 1.Lab on Lab).

Estimating Spec from Retraining. In Fig. 7(b), we compare a controller (4. Re Lab Field Est) trained on Lab spec, then re-trained on a mix of estimated field spec ($0.7$ weight) + Lab spec ($0.3$ weight) to a scan-and-dwell hand-coded controller (5. Expert) and also to our ML controller alternatively trained (1.-3.). For this experiment, Field Spec and Lab Spec differ in number of signals (1 vs. 2, respectively) and width between signals (3 vs. 2), and we sample a mix of both specs for testing. Controllers trained on only one spec (1. Lab, 2. Field) cannot cope with environments from multiple specs. The controller retrained only on estimated field spec (3. Re Field Est) suffers from “forgetting” and cannot cope with environments from it’s initial training spec. By retraining on both specs (weighted), the controller has the best performance.

Boostrapping. In Fig. 8, we consider a bootstrap loop with 3 iterations, where the ground-truth field spec parameters differ slightly from the previous iteration’s (lab spec used is A and the three field specs are F1-F3 in Table1). The idea is to expose a controller to environments of variable complexity. The results for the last iteration are shown in Fig. 8, where we evaluate against all environment specs (lab + 3 field). The detection rate ($\approx 60\%$) of the controller with estimated specs (1. Retrain Estimated) is close to that of controller using GT specs (3. Retrain Ground Truth) $\approx 65\%$. Hence, training with GT loses advantage over training with estimated specs over multiple iterations as the agent is exposed to more and more types of environments. We can also see that re-training DAN controller leads to smooth adaptation and using only current estimated spec (2.Cur Estimated) results in poor performance.

Instead of estimating the parameters of a field spec, which is then used to train a controller, we can perform Field State Estimation: do ML-based estimation of the full field/spectrum state (2. in Fig. 4) from partially observed field observations (1. in Fig. 4), and then use these reconstructed full-state episodes directly to retrain our ML controller, as described in Sec. 3.3.2. The ML-based estimation is done by a generator that is trained offline, separately from DAN. Once trained, it then generates one full-state episode from one partially observed field episode. In Table. 4, we compare the field state estimation approach to a controller exhaustively trained on an (anticipated) lab spec and one trained on the ideal training source: the field spec with complete ground truth information. We also study the performance after training with different numbers of reconstructed full-state episodes (25, 50, 100). Finally, we compare the approach with fine-tuning a controller in the field alone using partially observed prediction rewards—reward only what you observe in the band sampled and no error information using GT in other bands. For these experiments, we use SpecC1 (Table 1) for the lab spec and SpecC2 for the field spec. Throughout these experiments, evaluations of all training configurations are done on the same 100 field spec episodes (not used in training) and Predictive-DAN is the controller used.

The first row in Table. 4 corresponds to DAN trained with the lab spec, with no field experience and 1000 unique episodes. We can see that the performance is poor. The next three rows (2-4) correspond to DAN retrained with $25/50/100$ estimated full-state field episodes mixed with $25/50/100$ lab spec episodes, respectively. We can see that performance improves when the lab controller is retrained with more estimated field episodes. The fifth row in Table. 4 corresponds to training from the field spec with full ground truth and for 1000 unique episodes. This result roughly corresponds to the best possible performance of DAN given GT. As we can see, in this case, retraining DAN with 100 estimated field episodes (using no GT) has similar performance to training DAN with field GT and 1000 episodes.

The result of evaluating the DAN controller fine-tuned directly on 100 partially observed field episodes alone is shown in the last row (6) of Table. 4. For this case, we modified training for DAN accordingly to accommodate partially observed states, i.e., we have ground truth information only in the particular band sampled instead of all the bands previously—hence the reward obtained by $M$-network is limited to the prediction only in this band. We can see that its performance is inferior compared to training with full-state episodes, even when those complete states are estimated (row 4 of Table. 4). Partial observations in sparse, dynamic signal environments makes training difficult.

In summary, we show that retraining a DAN with full-state sequences estimated using ML-based generators from partially observed field observations is a viable and promising approach, while fine-tuning a DAN directly on partially observed field experiences produces poor results in general.