Unsupervised Learned Kalman Filtering

We consider state estimation in dt, linear, Gaussian ss models. Letting $\mathbf{x}_{t}$ denote the $m\times 1$ hidden state vector at time instance $t\in\mathbb{Z}$, which evolves in time via

$\displaystyle\mathbf{x}_{t}$

$\displaystyle=\mathbf{F}\cdot{\mathbf{x}_{t-1}}+\mathbf{w}_{t},$

$\displaystyle\mathbf{w}_{t}\sim\mathcal{N}\left(\mathbf{0},\mathbf{Q}\right),$

$\displaystyle\quad\mathbf{x}_{t}\in\mathbb{R}^{m}.$

(1)

Here, $\mathbf{F}$ is the $m\times m$ state evolution matrix, while $\mathbf{w}_{t}$ is awgn (awgn) with covariance $\mathbf{Q}$. The corresponding observation $\mathbf{y}_{t}$, is related to $\mathbf{x}_{t}$ via

$\displaystyle\mathbf{y}_{t}$

$\displaystyle=\mathbf{H}\cdot{\mathbf{x}_{t}}+\mathbf{v}_{t},$

$\displaystyle\mathbf{v}_{t}\sim\mathcal{N}\left(\mathbf{0},\mathbf{R}\right),$

$\displaystyle\quad\mathbf{y}_{t}\in\mathbb{R}^{n},$

(2)

where $\mathbf{H}$ is the $n\times m$ measurement matrix, and $\mathbf{v}_{t}$ is an awgn with covariance $\mathbf{R}$. We focus on the filtering problem, where one needs to track the hidden state ${\mathbf{x}}_{t}$ from a known initial state $\mathbf{x}_{0}$. At each time instance $t$, the goal is to provide an instantaneous estimate $\hat{\mathbf{x}}_{t}$, based on the observations seen so far $\left\{\mathbf{y}_{\tau}\right\}_{\tau=1}^{t}$. We consider scenarios where one has partial domain knowledge, such that the statistics of the noises $\mathbf{w}_{t}$ and $\mathbf{v}_{t}$ are not known, while the matrices $\mathbf{F}$ and $\mathbf{H}$ are known. To fill the information gap we assume that we have access to an unlabeled ds containing a sequence of observations from which one has to learn to recover the hidden state.

kn is a hybrid mb/dd implementation of the kf. The latter utilizes full knowledge of the ss model to estimate $\mathbf{x}_{t}$, based on the current observed $\mathbf{y}_{t}$ and the previous estimate $\hat{\mathbf{x}}_{t-1}$. This is achieved by first predicting the next state and observation based solely on the previous estimate via

	$\displaystyle\hat{\mathbf{x}}_{t\left|t-1\right.}$	$\displaystyle=\mathbf{F}\cdot{\hat{\mathbf{x}}_{t-1}},$		(3a)
	$\displaystyle\hat{\mathbf{y}}_{t\left|t-1\right.}$	$\displaystyle=\mathbf{H}\cdot{\hat{\mathbf{x}}_{t\left|t-1\right.}},$		(3b)

while computing the second-order moments of these estimates as $\boldsymbol{\Sigma}_{t\left|t-1\right.}=\mathbf{F}\cdot\boldsymbol{\Sigma}_{t-1}\cdot\mathbf{F}^{\top}+\mathbf{Q}$, and $\mathbf{S}_{t\left|t-1\right.}=\mathbf{H}\cdot\boldsymbol{\Sigma}_{t\left|t-1\right.}\cdot\mathbf{H}^{\top}+\mathbf{R}$. Next, the kf computes the kg (kg) $\mathbfcal{K}_{t}$ as $\mathbfcal{K}_{t}={\boldsymbol{\Sigma}}_{t\left|t-1\right.}\cdot{\mathbf{H}}^{\top}\cdot{\mathbf{S}}^{-1}_{t\left|t-1\right.}$, which is used to update the estimation covariance ${\boldsymbol{\Sigma}}_{t}={\boldsymbol{\Sigma}}_{t\left|t-1\right.}-\mathbfcal{K}_{t}\cdot{\mathbf{S}}_{t\left|t-1\right.}\cdot\mathbfcal{K}^{\top}_{t}$, and provide the state estimate $\hat{\mathbf{x}}_{t}$ via

$$\hat{\mathbf{x}}_{t}=\hat{\mathbf{x}}_{t\left|t-1\right.}+\mathbfcal{K}_{t}\cdot\Delta\mathbf{y}_{t},$$

(4)

where $\Delta\mathbf{y}_{t}$ is the innovation process computed as

$$\Delta\mathbf{y}_{t}=\mathbf{y}_{t}-\hat{\mathbf{y}}_{t\left|t-1\right.}.$$

(5)

kn learns to implement the kf from labeled data in partially known ss models. This is achieved by noting that the available knowledge allows to compute the predictions in (3), while the missing domain knowledge is needed to compute the kg. Thus, kn augments the flow of the kf with an rnn, which estimates the kg and implicitly tracks the second-order moments computed by the kf, while the state estimate is obtained via (4) (see [16] for a detailed description). The kn architecture, depicted in Fig. 1, is trained to estimate the state in a supervised manner based on labeled data. The data set comprises $N$ pairs of hidden state trajectories and its corresponding observations of the form $\mathcal{D}=\left\{\left(\mathbf{Y}_{i},\mathbf{X}_{i}\right)\right\}_{i=1}^{N}$, where

$\displaystyle\mathbf{Y}_{i}$

$\displaystyle=\left[\mathbf{y}_{1}^{\left(i\right)},\ldots,\mathbf{y}_{T_{i}}^{\left(i\right)}\right],\hskip 5.0pt\mathbf{X}_{i}=\left[\mathbf{x}_{0}^{\left(i\right)},\mathbf{x}_{1}^{\left(i\right)},\ldots,\mathbf{x}_{T_{i}}^{\left(i\right)}\right],$

(6)

and $T_{i}$ is the length of the $i$th training trajectory. The training procedure aims at minimizing the regularized $\ell_{2}$ loss. Letting $\mathbf{\Theta}$ be the trainable parameters of the rnn and $\hat{\mathbf{x}}_{t}^{\left(i\right)}\left(\mathbf{\Theta}\right)$ be the output of kn with parameters $\mathbf{\Theta}$ at time $t$ applied to $\mathbf{Y}_{i}$, the loss is computed for the $i$th trajectory as

$\displaystyle l_{i}(\mathbf{\Theta})=\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}\left\|\hat{\mathbf{x}}_{t}^{\left(i\right)}\left(\mathbf{\Theta}\right)\!-\!\mathbf{x}^{\left(i\right)}_{t}\right\|^{2}+\gamma\cdot\left\|\mathbf{\Theta}\right\|^{2},$

(7)

where $\gamma>0$ is a regularization coefficient. The loss measure (7) is used to optimize $\mathbf{\Theta}$ via sgd (sgd) optimization combined with the bptt (bptt) algorithm [18].

kn, described in Subsection 2.2, as well as other previously proposed dnn-based state estimators such as [14], are designed to estimate $\mathbf{x}_{t}$, and thus are trained so that the output approaches the ground-truth hidden state sequence. kn admits an interpretable architecture owing to its hybrid mb/dd design, which preserves the flow of the kf. We exploit this fact to propose a training algorithm for kn that does not rely on ground-truth labels.

Unsupervised loss: kn uses its state estimates to predict the next observation via (3b) as an internal feature. While the accuracy of this prediction, e.g., the squared magnitude of the innovation process (5), depends on the accuracy in estimating $\mathbf{x}_{t}$ and the observation noise, (5) can be computed based solely on the observed sequence. Consequently, one can adapt kn in an unsupervised manner by training it to minimize $\left\|\Delta\mathbf{y}_{t}\right\|^{2}$. This quantity can be used to compute the gradient with respect to the parameters of the rnn, which outputs the learned kg, by the derivative chain rule. Indeed,

	$\displaystyle\frac{\partial\|\Delta\mathbf{y}_{t}\|^{2}}{\partial\mathbfcal{K}_{t-1}}\stackrel{{\scriptstyle(a)}}{{=}}\frac{\partial}{\partial\mathbfcal{K}_{t-1}}\left\|\mathbf{H}\cdot\mathbf{F}\cdot\mathbfcal{K}_{t-1}\cdot\Delta\mathbf{y}_{t-1}-\Delta\mathbf{y}_{t}^{-}\right\|^{2}$
	$\displaystyle=2\cdot\mathbf{H}^{\top}\cdot\mathbf{F}^{\top}\cdot\left(\mathbfcal{K}_{t-1}\cdot\Delta\mathbf{y}_{t-1}-\Delta\mathbf{y}_{t}^{-}\right)\cdot\Delta\mathbf{y}_{t-1}^{\top},$		(8)

where $\Delta\mathbf{y}_{t}^{-}\triangleq\mathbf{y}_{t}-\mathbf{H}\cdot\mathbf{F}\cdot\hat{\mathbf{x}}_{t-1\left|t-2\right.}$. In (8), $(a)$ holds since

	$\displaystyle\hat{\mathbf{y}}_{t\left|t-1\right.}$	$\displaystyle=\mathbf{H}\cdot\mathbf{F}\cdot\hat{\mathbf{x}}_{t-1\left|t-1\right.}$
		$\displaystyle=\mathbf{H}\cdot\mathbf{F}\cdot\left(\hat{\mathbf{x}}_{t-1\left|t-2\right.}+\mathbfcal{K}_{t-1}\cdot\Delta\mathbf{y}_{t-1}\right).$		(9)

The gradient in (8) indicates that the $\ell_{2}$ norm of the innovation process can be used to learn the computation of the kg, which involves the trainable parameters of kn $\mathbf{\Theta}$. Similarly to (7), the resulting loss for the $i$th trajectory is

$\displaystyle\tilde{l}_{i}(\mathbf{\Theta})=\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}\left\|\hat{\mathbf{y}}_{t\left|t-1\right.}^{\left(i\right)}\left(\mathbf{\Theta}\right)\!-\!\mathbf{y}^{\left(i\right)}_{t}\right\|^{2}+\gamma\cdot\left\|\mathbf{\Theta}\right\|^{2}.$

(10)

Unsupervised kn is thus trained using solely observed trajectories based on the loss measure (10) using sgd variants combined with bptt for gradient computation.

Offline versus online training: The ability to train kn without providing ground-truth state sequences gives rise to two possible training approaches: a purely unsupervised offline training scheme, and an online semi-supervised strategy. The offline approach follows conventional unsupervised learning using unlabeled data of the form $\tilde{\mathcal{D}}=\left\{\left(\mathbf{Y}_{i}\right)\right\}_{i=1}^{N}$. This data set is used to optimize $\mathbf{\Theta}$ via mini-batch sgd-based optimization, where for every batch indexed by $k$, we choose $M<N$ trajectories indexed by $i_{1}^{k},\ldots,i_{M}^{k}$, computing the mini-batch loss as $\mathcal{L}_{k}\left(\mathbf{\Theta}\right)=\frac{1}{M}\sum_{j=1}^{M}\tilde{l}_{i_{j}^{k}}\left(\mathbf{\Theta}\right)$.

Online training builds upon the ability to learn without labels to adapt a pre-trained kn to dynamics that differ from those used during training. Pretraining can be done using labeled data obtained by mathematical modelling and/or past measurements without altering the architecture of kn. Then, the deployed model is further adapted in an unsupervised manner using observations acquired during operation to form training trajectories from realizations of the data. Such a training procedure provides kn with the ability to be adaptive to changes in the distribution of the data. Specifically, once every $\tilde{T}$ time steps, we compute the loss (10) over the last $\tilde{T}$ observations online and optimize the rnn parameters $\mathbf{\Theta}$ accordingly.

The ability to train kn in an unsupervised manner without relying on ground-truth sequences, follows directly from the hybrid mb/dd architecture of kn, where one can identify the observations innovation process as an internal feature and use it to compute the loss. The training procedure does not affect the kn architecture, and one can use the same supervised model designed in [16]. In the current work, we focus on partially-known ss models where $\mathbf{F}$ and $\mathbf{H}$ are available from, e.g., a physical model. While supervised kn was shown in [16] to operate reliably when using inaccurate approximations of $\mathbf{F}$ and $\mathbf{H}$, we leave such a study in unsupervised setups for future work.

The proposed online semi-supervised technique allows one to adapt a pre-trained kn state estimator after deployment, coping with setups in which the original training is based on data that does not fully capture the true underlying ss model. This gain, numerically demonstrated in Section 4, bears some similarity to online training mechanisms proposed for hybrid mb/dd communication receivers in [19, 20, 21]. Despite the similarity, the proposed technique, obtained from the interpretable operation of kn, is fundamentally different from that proposed in [19, 20, 21], where structures in communication data were exploited to generate confident labels from decisions. Nonetheless, both the current work and [19, 20, 21] demonstrate the potential of mb deep learning in enabling application-oriented, efficient training algorithms.