Unsupervised Learning of Hybrid Latent Dynamics:
A Learn-to-Identify Framework

Considering high-dimensional time-series $\mathbf{x}_{1:T}=[\mathbf{x}_{0},\cdots,\mathbf{x}_{t}\cdots,\mathbf{x}_{T}]$, we model its generation process as:

where function $\mathcal{F}(\mathbf{z}_{<z})$ describes the latent dynamics of the state variables $\mathbf{z}_{t}$, and function $g(\mathbf{z_{t}})$ describes the emission of the latent state variables to the observed data.

Latent dynamic functions: While Meta-HyLaD is agnostic to the type of dynamic functions used in Equation (1), we choose an ordinary differential equation (ODE) to describe the latent dynamics as a continuous process that can be emitted to the observation space only when needed. More specifically, we describe the latent dynamics as a UDE:

where $f_{\textrm{PHY}}(\mathbf{z}_{t};\mathbf{c}_{p})$ represents the known physics equation governing the data, and $\mathbf{c}_{p}$ its unknown parameters. $f_{\textrm{NN}_{\phi}}(\mathbf{z}_{t};\mathbf{c}_{n})$ represents the potential errors in the prior physics, modeled by a neural network with weight parameters $\phi$. Instead of identifying a single neural function $f_{\textrm{NN}_{\phi}}(\mathbf{z}_{t})$ that will only model a global discrepancy between $f_{\textrm{PHY}}$ and the true governing dynamics, we further allow it to change with a parameter $\mathbf{c}_{n}$ that can be identified from data: more specifically, we use $\mathbf{c}_{n}$ to generate the weight parameter $\phi$ of $f_{\textrm{NN}_{\phi}}$ via a hyper network $\phi=h_{\theta}(\mathbf{c}_{n})$.

Emission functions: The emission function $\mathbf{x}_{t}=g(\mathbf{z_{t}})$, i.e., the decoder, bridges the mapping from the latent state space to the observed data space. In most existing works considering latent physics-based functions, it has been considered critical for this emission function to be physics-based [25, 17]. Similarly, when the latent dynamic function is purely neural, it is customary to adopt a neural function as the emission function [2]. As a secondary objective of this paper, we will investigate the use of a physics-based vs. neural decoder for different types of latent dynamic functions. We denote them generally as $g_{\psi}$ where $\psi$ is known if $g$ is physics-based, and unknown if $g$ is a neural network parameterized with $\psi$.

With the pair of functions described in Equations (1-2), both state variables $\mathbf{z}_{0:T}$ and data $\mathbf{x}_{0:T}$ can be generated once the initial condition $\mathbf{z}_{0}$ and the parameters of the hybrid components – $\mathbf{c}_{p}$ and $\mathbf{c}_{n}$ – are properly identified.

Given an observed time-series $\mathbf{x}_{0:T}=[\mathbf{x}_{0},\cdots,\mathbf{x}_{t}\cdots,\mathbf{x}_{T}]$ in training data $\mathcal{D}$, a common approach to identifying its latent dynamic process is to infer its time-varying latent state $\mathbf{z}_{0:T}=[\mathbf{z}_{0},\cdots,\mathbf{z}_{t}\cdots,\mathbf{z}_{T}]$ to reconstruct $\mathbf{x}_{0:T}$ [23, 19, 8, 35]. We will show that – for neural functions with abstract latent states $\mathbf{z}_{t}$ – this reconstruction does not guarantee a correct identification of the dynamic function governing $\mathbf{z}_{t}$: an evidence is when the identified dynamic function, given different initial conditions, could not generate the correct time series governed by the same dynamics.

We propose that a proper identification of the latent dynamic functions requires us to go a step further: one must further separate $\mathbf{z}_{0:T}$ into two key ingredients generating it – the initial latent state $\mathbf{z}_{0}$ that is specific to the time-series sample, and parameters $\mathbf{c}_{\mathbf{z}}$ for the governing latent dynamic function. In a neural network where the learned functions can be arbitrary and the latent states abstract without direct supervision, this separation is not trivial. Below, we describe our learning strategies to facilitate this separation, and discuss how such identification strategies may be different for the physics-based versus neural functions.

Identifying initial latent states: The initial latent state $\mathbf{z}_{0}$ of an observed time series $\mathbf{x}_{0:T}$ is specific to that series and can be simply identified from the first several frames of observations $\mathbf{x}_{0:l}$, where $l\ll T$:

where $\mathcal{E}_{\phi_{z}}$ is a neural encoder with weight parameters $\phi_{z}$.

We consider common benchmarks including three simple physics systems of Pendulum [2], Mass Spring [8], and Bouncing Ball (under gravity) [2], and two more complex physics systems of Double Pendulum [2] and Rigid Rody [25]. To demonstrate feasibility towards more complex systems, we further consider an additional dataset of dynamic PET [27]. For each of the five physics system, we randomly sample the initial states and parameters of the governing dynamic function, and generate time-series of system states and the corresponding image observations by physical rendering in [25]. We refer to the dynamic functions used for the generation of data as full physics functions, and design partial physics functions to represent our imperfect prior knowledge about the observed data reflecting a variety of potentially additive and multiplicative errors as summarized in Table 1: in Pendulum and Mass Spring, partial physics lacks the knowledge about damping (additive error), in Bouncing Ball and Rigid Body, partial physics only considers gravity/force in one direction/plane (multiplicative error); in Double Pendulum, partial physics ignores the impact of the mass difference between two pendulums (multiplicative error).

To introduce heterogeneity in the underlying dynamics, in data generation we vary the parameters of the full physics equations in the components both present (blue) and absent (red) in the prior physics, with their training and test distributions and number of samples summarized in Table 1. More details on each dataset are in Appendix A. We leave descriptions of the dynamic PET dataset in Section 4.5.

We first evaluate the reconstruction vs. learn-to-identify objectives for identifying physics-based vs. neural latent dynamic functions, on the Pendulum dataset (see Table 1).

Models: Here we consider the three identification schemes for learning hybrid latent dynamic functions as described in Section 3.2 (Fig. 2). All three learning strategies share the same backbone architectures, with their architectural and implementation details provided in appendix G. We consider $k=1$ and $k=7$ for learn-to-identify strategies, to assess potential impact associated with the number of samples used to identify the dynamic functions.

Metrics: A successfully-identified latent dynamic function should be able to forecast a time series that is different from those used to identify it (but shares the same dynamics). To this end, we test the three learning strategies for two tasks: 1) a reconstruction task to identify $\mathbf{c}_{p}$ and $\mathbf{c}_{n}$ from a test time-series and reconstruct the seires itself, and 2) a prediction task where $\mathbf{c}_{p}$ and $\mathbf{c}_{n}$ are identified from a test time-series but used to predict another time-series that follows the same dynamics but different initial conditions.

For both tasks, we measured the mean squared error (MSE) between the ground-truth and estimated time series at both the observation $\mathbf{x}_{0:T}$ and the state variable $\mathbf{z}_{0:T}$ level . In addition, we consider two metrics to separately evaluate how well $\mathbf{c}_{p}$ and $\mathbf{c}_{n}$ are each identified: for $\mathbf{c}_{p}$, we directly measure the MSE between its identified and ground-truth value; for $\mathbf{c}_{n}$, because we do not expect its value to be physically meaningful, we instead measure the fit between the identified neural dynamic function $f_{\textrm{NN}_{\theta}}(\mathbf{z}_{t};\mathbf{c}_{n})$ and the ground truth residual function $-\beta\dot{\varphi}$ by their MSE.

Results: As summarized in Table 2, all three learning strategies results in strong performance to reconstruct the same time series used to identify the latent dynamic functions. However, latent dynamic functions identified with the reconstruction objective (A) fail when attempting to predict different time series sharing the same dynamics. A deeper dive reveals something interesting: the physics component ($\mathbf{c}_{p}$) is well identified by all identification strategies (A-C); in contrast, $\mathbf{c}_{n}$ is only successfully identified by the learn-to-identify strategy (B-C). Moreover, the mixed identification strategy (C) seems to facilitate a better identification of $\mathbf{c}_{n}$ when $k=1$, although both Meta- (B) and Mixed-HyLaD (C) achieve similar performance when $k$ is increased to 7. Interestingly, increasing $k$ for identifying $\mathbf{c}_{n}$ also improves the identification of $\mathbf{c}_{p}$ in Mixed-HyLaD (C). We provide visual examples of identified $f_{\textrm{NN}_{\phi}}$ in appendix B.

These experiments provide two important insights: 1) a good reconstruction is not sufficient evidence that the underlying latent dynamics is correctly identified, and 2) the inclusion of physics inductive bias or the learn-to-identify strategy are two effective solutions to address the issue of identifiability.

We now evaluate the benefits of hybrid latent dynamic functions on the five physics systems described in Section 4.1.

Models: We now fix the learn-to-identify strategy to that of Fig. 2B, and compare four alternatives of latent dynamic functions: using only the partial physics function (purely physics), using only a neural network (purely neural), hybrid as defined in Equation (2) but with a global $f_{\textrm{NN}_{\phi}}$ (Global-HyLaD), and Meta-HyLaD. We also obtain results on a latent dynamic function utilizing the full physics, setting a performance reference when prior knowledge is perfect.

Metrics: We consider MSE for both the observed images $\mathbf{x}_{0:T}$ and state variables $\mathbf{z}_{0:T}$, and Valid Prediction Time (VPT) that measures how long the predicted object’s trajectory remains close to the ground truth trajectory based on the MSE [2]. We separately evaluate these metrics for test time-series with parameters within (in distribution / ID) or outside (out of distribution / OOD) those used in the training data, as summarized in Table 1.

Results: Fig. 3 summarizes the results on three datasets, with complete results and visual examples provided in appendix C: note that the purely neural dynamic functions (yellow), due to the abstract nature of its latent state variables, results in poor MSE/VPT metrics on $\mathbf{z}_{0:T}$ at a a magnitude larger than alternative approaches; their values are thus omitted from Fig. 3 but can be found in appendix C. As shown, purely physics-based approaches, when given perfect knowledge, achieve excellent performance as a reference (dashed blue line). Their accuracy, however, deteriorates substantially given imperfect knowledge (aqua), often to a level on par with purely neural approaches (yellow). The use of hybrid functions helps, although to a limited extent if the neural components are globally optimized (green). Meta-HyLaD (red), with an ability to identify the neural components from data, substantially improves over all three alternatives – its performance is on par with, and sometimes better than, the performance achieved with perfect physics.

On the five physics systems, we then compare Meta-HyLaD with existing unsupervised latent dynamic models. Note that there is no existing hybrid latent dynamic models.

Models: We consider 1) ALPS [35] which considers a physics-based latent dynamic function (here the partial physics) and a reconstruction-based learning objective, and 2) three neural latent dynamic models ranging from those with minimal physics prior (LSTM and neural ODE / NODE) and strong prior (Hamiltonian generative network / HGN) [2]. We consider their original formulations [2] that optimize global latent dynamic functions as illustrated in Fig. 1B, as well as their extension to the presented meta-learning framework representative of recent works in meta-learning latent dynamics [18, 21]. Architecture details of all baselines are provided in appendix G. This provides a comprehensive coverage of prior arts in their choices of latent dynamic functions and identifications strateiges.

We further note that different decoders were used in these original baselines: the neural latent dynamic models utilize a neural network as the decoder (neural decoder) [2]; the latent-HGN utilizes a neural decoder but specifically uses the position latent state as the input (neural decoder with prior) [2]; ALPS uses both this and a physics-based decoder [35]. To isolate the effect of decoders, we further evaluate each of these baselines using each of the three different decoders.

Metrics: We consider MSE and VPT for $\mathbf{x}_{0:T}$, and omit metrics on $\mathbf{z}_{0:T}$ due to its abstract nature in most baselines.

Results: Fig. 4 summarizes the results on three datasets for physics-based vs. neural decoders, with the complete quantitative results in appendix D. Notably, across all datasets, Meta-HyLaD in general demonstrates a significant margin of improvements over all baselines including those utilizing meta-learning. This improvement is the most significant when used in combination of a physics-based decoder while, with a neural decoder with or without prior, Meta-HyLaD sometimes becomes comparable with the meta-extension of HGN – this again demonstrates the benefits of physics inductive bias in both models. Note that, as we will show in Section 4.5, Meta-HyLaD is more generally applicable beyond Hamiltonian systems for which HGN is designed.

Interestingly, Meta-HyLaD improves with using a physics-based decoder, while the other baselines deteriorates. The effect of using a prior with the decoder (appendix C) is less consistent and varies with the dataset for all models.

Finally, we consider recovering the spatial distribution and kinetics of radiotracer-labeled biological substrates from dynamic PET images. The regional tracker kinetics underlying PET can be described by the widely used compartment models in Table 1. The total concentration of radioactivity in all tissues, $C_{T_{i}}(t)$, on the scanning time interval $[t_{k-1},t_{k}]$ is measured as: $x_{ik}(t)=\frac{1}{t_{k}-t_{k-1}}\int_{t_{k-1}}^{t_{k}}C_{Ti}(t)e^{-\frac{t}{\tau}}dt$ at each voxel $i$. The activity image $\mathbf{x}_{k}$ is obtained by lexicographic ordering of $x_{ik}$ at different voxels. While raw PET data are sinograms, in this proof-of-concept we consider $\mathbf{x}_{0:T}$ as the observed image series, with the goal to identify the compartment models. More details are in appendix E.

Data: We use the two-tissue compartment model listed in Table 1 to generate $\textbf{x}_{0:T}$ of a brain with 5 regions of interesting for 0-40 minutes with a scanning period of 0.5 minutes, resulting in 80 frames per time series. 2000 time-series samples are generated with training/test and ID/OOD split of parameter ranges provided in appendix E.

We then assume the one-tissue compartment model to be our prior physics in Meta-HyLaD. Compared to the additive and multiplicative errors considered earlier, this setup provides a more challenging scenario where the prior physics represents a crude approximation, i.e., $C_{Ti}(t)=C_{Ei}(t)+C_{Mi}(t)$ , for the data-generating latent dynamics.

Model & Metrics: We compare Meta-HyLaD with purely physics, purely neural, and Global-HyLaD similar to Section 4.3. We consider MSE and peak signal-to-noise-ratio (PSNR) on $\mathbf{x}_{0:T}$, and MSE and VPT on $\mathbf{z}_{0:T}$, i.e., $C_{T}(0:T)$.

Results: Fig. 5A summarize the quantitative metrics along with visual examples, with the full results and examples of tracer kinetics in appendix E . The same observations in the physics systems still hold: Meta-HyLaD is the only model that is able to approach the reference performance (where data-generating physics is used in the identification), significantly outperforming the alternatives. This leaves exciting future real-world opportunities for Meta-HyLaD.