Forecasting causal dynamics with universal reservoirs

Our results apply to the discrete-time evolution of dynamical variables in a $d$-dimensional Euclidean space whose behavior is determined by a causal chain with infinite memory (CCIM). More specifically, we are interested in the dynamical properties of infinite sequences ${\bf y}=\left({\bf y}_{t}\right)_{t\in\mathbb{Z}}$, where ${\bf y}_{t}\in\mathbb{R}^{d}$, for all $t\in\mathbb{Z}$, for which there exists a functional $H:(\mathbb{R}^{d})^{\mathbb{Z}_{-}}\longrightarrow\mathbb{R}^{d}$ such that

$${\bf y}_{t}=H\left(\ldots,{\bf y}_{t-2},{\bf y}_{t-1}\right)=H\left(\underline{{\bf y}_{t-1}}\right),\quad\mbox{for all $t\in\mathbb{Z}$,}$$

and where the symbol $\underline{{\bf y}_{t-1}}\in({\mathbb{R}}^{d})^{\mathbb{Z}_{-}}$ denotes the semi-infinite sequence $\underline{{\bf y}_{t-1}}=\left(\ldots,{\bf y}_{t-2},{\bf y}_{t-1}\right)$. These dynamics are the deterministic counterparts of various structures with a widespread presence in time series analysis. See for instance [Dede 07, Douk 08, Alqu 12] and references therein.

The CCIM determined by the functional $H:({\mathbb{R}}^{d})^{\mathbb{Z}_{-}}\longrightarrow{\mathbb{R}}^{d}$ has a canonical causal and time-invariant filter $U_{H}:({\mathbb{R}}^{d})^{\mathbb{Z}}\longrightarrow({\mathbb{R}}^{d})^{\mathbb{Z}}$ associated determined by the equality

$$U_{H}({\bf y})_{t}:=H\left(\underline{{\bf y}_{t-1}}\right),\quad\mbox{for all ${\bf y}\in({\mathbb{R}}^{d})^{\mathbb{Z}}$ and $t\in\mathbb{Z}$.}$$

(2.1)

A classical problem in control and systems theory is the realization of filters like (2.1) as the unique solution (when it exists) of a state-space transformation

	$\displaystyle\mathbf{x}_{t}$	$\displaystyle=F(\mathbf{x}_{t-1},{\bf y}_{t-1}),$		(2.2)
	$\displaystyle{\bf y}_{t}$	$\displaystyle=h(\mathbf{x}_{t}),$		(2.3)

where $F:\mathbb{R}^{L}\times\mathbb{R}^{d}\longrightarrow\mathbb{R}^{L}$ and $h:\mathbb{R}^{L}\longrightarrow\mathbb{R}^{d}$ are a state and a readout map, respectively, $\mathbb{R}^{L}$ is, in this case, the state space and, typically, $N$ is much bigger than $d$. Some classical references in connection to this problem are [Kalm 59b, Kalm 59a, Kalm 60, Kalm 62, Baum 66, Kalm 10] or, more recently, [Sand 78, Sont 79, Dang 84, Nare 90, Matt 94, Lind 15, Grig 21b, Orte 24b]. Even though the realization problem does not always have a solution, it has been shown that if the domain of the functional $H$ is restricted to a space of uniformly bounded sequences of the type $K_{M}:=\left\{{\bf y}\in(\mathbb{R}^{d})^{\mathbb{Z}_{-}}\mid\left\|{\bf y}_{t}\right\|\leq M,\,t\in\mathbb{Z}_{-}\right\}$ for some $M>0$ and, additionally, it has a continuity condition called the fading memory property [Boyd 85, Grig 18b], then various families of state-space transformations of the type (2.2)-(2.3) can be used to approximate it uniformly. This is the case for linear systems with polynomial or neural network readouts [Boyd 85, Grig 18b, Gono 20b], state-affine systems with linear readouts [Grig 18b], or echo state networks [Grig 18a, Gono 20b, Gono 23a, Gono 21]. In other words, any of these families exhibit universality properties in the fading memory category. This allows us to conclude that given a fading memory functional $H:K_{M}\longrightarrow\mathbb{R}^{d}$ and $\varepsilon>0$, there exists a state-space system $(F,h)$ in that family to which a functional $H^{F}_{h}:K_{M}\longrightarrow\mathbb{R}^{d}$ can be associated and such that

$$\left\|H-H^{F}_{h}\right\|_{\infty}=\sup_{{\bf y}\in K_{M}}\left\{\left\|H({\bf y})-H^{F}_{h}({\bf y})\right\|\right\}<\varepsilon.$$

For future reference, a condition that ensures that a state-space system $(F,h)$ has a functional $H^{F}_{h}:K_{M}\longrightarrow\mathbb{R}^{d}$ associated is the so-called echo state property (ESP) [Jaeg 10]: the state equation $F$ has the ESP with respect to inputs in $K_{M}$ when for any ${\bf y}\in K_{M}$ there exists a unique $\mathbf{x}\in(\mathbb{R}^{L})^{\mathbb{Z}_{-}}$ such that (2.2) holds for any $t\in\mathbb{Z}_{-}$. In such a situation, $F$ has a filter $U^{F}:K_{M}\longrightarrow(\mathbb{R}^{L})^{\mathbb{Z}_{-}}$ associated that is uniquely determined by the relations $U^{F}({\bf y})_{t}=F\left(U^{F}({\bf y})_{t-1},{\bf y}_{t-1}\right)$. The filter $U^{F}$ induces the functional $H^{F}\colon K_{M}\longrightarrow{\mathbb{R}}^{N}$ given by $H^{F}({\bf y})=U^{F}({\bf y})_{0}.$ The functional $H^{F}_{h}:K_{M}\longrightarrow\mathbb{R}^{d}$ corresponding to the state-space system is given by $H^{F}_{h}({\bf y})=h\circ H^{F}({\bf y})$. The ESP has important dynamical implications [Grig 19, Manj 20, Manj 22]. In practice, the state space is often taken to be compact. In this case, if $F$ is a continuous map and a contraction on its first entry, the ESP is ensured. Indeed, for such a state map, it can be shown that with inputs from $\overline{B({\bf 0},M)}\subset\mathbb{R}^{d}$ there is a compact subset $X\subset\mathbb{R}^{L}$ such that $F(X\times\overline{B({\bf 0},M)})\subset X$. Stochastic generalizations of some of these statements can be found in [Manj 23, Orte 24a].

A particularly important family of CCIMs can be generated using the observations of some dynamical systems. Indeed, let $M$ be a manifold and let $\phi:M\longrightarrow M$ be a continuous map that determines a dynamical system on it via the evolution recursions

$$m_{t+1}=\phi(m_{t}),$$

using a given point $m_{0}\in M$ as initial condition. The data available for the forecasting of a dynamical system is given in the form of observations that are usually encoded as the values of a continuous map $\omega:M\longrightarrow{\mathbb{R}}^{d}$, that is, they are given by the sequences ${\bf y}_{0}:=\omega(m_{0}),\,{\bf y}_{1}:=\omega(m_{1}),\ldots,{\bf y}_{t}:=\omega(m_{t})$. The problem that we tackle in this work is the forecasting or continuation of those sequences using state-space transformations of the type (2.2)-(2.3) that have been learned/trained using exclusively temporal traces of $\omega$-observations.

We shall now show that the observations of a large class of dynamical systems are CCIMs. More specifically, we shall work in the context of invertible dynamical systems for which an invertible generalized synchronization map can be found. We briefly recall the meaning of these terms. First, we say that the dynamical system $\phi$ on $M$ is invertible when it belongs either to the set ${\rm Hom}(M)$ of homeomorphisms (continuous invertible maps with continuous inverse) of a topological space $M$ or to the set of diffeomorphisms ${\rm Diff}^{1}(M)$ of a finite-dimensional manifold $M$. The invertibility of $\phi$ allows us, given an observation map $\omega:M\longrightarrow{\mathbb{R}}^{d}$, to define the $(\phi,\omega)$-delay map $S_{(\phi,\omega)}:M\longrightarrow(\mathbb{R}^{d})^{\mathbb{Z}}$ as $S_{(\phi,\omega)}(m):=\left(\omega(\phi^{t}(m))\right)_{t\in\mathbb{Z}}$. Second, given a state-space map $F:\mathbb{R}^{L}\times\mathbb{R}^{d}\longrightarrow\mathbb{R}^{L}$, consider the drive-response system associated to the $\omega$-observations of $\phi$ and determined by the recursions:

$$\mathbf{x}_{t}=F\left(\mathbf{x}_{t-1},S_{(\phi,\omega)}(m)_{t-1}\right),\quad\mbox{$t\in\mathbb{Z},\,m\in M.$}$$

(2.4)

We say that a generalized synchronization (GS) [Rulk 95, Peco 97, Ott 02, Bocc 02, Erog 17] occurs in this configuration when there exists a map $f_{(\phi,\omega,F)}:M\longrightarrow\mathbb{R}^{L}$ such that for any $\mathbf{x}_{t}$, $t\in\mathbb{Z}$, as in (2.4), it holds that

$$\mathbf{x}_{t}=f_{(\phi,\omega,F)}(\phi^{t}(m)),$$

(2.5)

that is, the time evolution of the dynamical system in phase space (not just its observations) drives the response in (2.4). We emphasize that the definition (2.5) presupposes that the recursions (2.4) have a (unique) solution, that is, that there exists a sequence $\mathbf{x}\in(\mathbb{R}^{L})^{\mathbb{Z}}$ such that (2.4) holds true. When that existence property holds and, additionally, the solution sequence $\mathbf{x}$ is unique, we say that $F$ has the $(\phi,\omega)$-Echo State Property (ESP). The existence, continuity, and differentiability of GSs have been established, for instance, in [Star 99, Grig 21a] for a rich class of systems that exhibit the so-called fading memory property and that are generated by locally state-contracting maps $F$. In such a framework, the state equation (2.4) uniquely determines a filter $U^{F}:\left({\mathbb{R}}^{d}\right)^{\mathbb{Z}}\longrightarrow\left({\mathbb{R}}^{L}\right)^{\mathbb{Z}}$ that satisfies the recursion $U^{F}({\bf z})_{t}=F\left(U^{F}({\bf z})_{t-1},{\bf z}_{t-1}\right)$, for all $t\in\mathbb{Z}$, in terms of which the GS $f_{(\phi,\omega,F)}:M\longrightarrow\mathbb{R}^{L}$ can be written as

$$f_{(\phi,\omega,F)}(m)=\left(U^{F}\left(S_{(\phi,\omega)}(m)\right)\right)_{0}.$$

(2.6)

The following result, introduced in [Grig 21a, Grig 24], shows that the observations of a dynamical system in the presence of injective GS form a CCIM. More explicitly, we shall be assuming that the GS $f_{(\phi,\omega,F)}:M\longrightarrow\mathbb{R}^{L}$ is an embedding which means that $f_{(\phi,\omega,F)}$ is an injective immersion ($C^{1}$ map with injective tangent map) and, additionally, the manifold topology in $f_{(\phi,\omega,F)}(M)$ induced by $f_{(\phi,\omega,F)}$ coincides with the relative topology inherited from the standard topology in $\mathbb{R}^{L}$.

As we show in the next corollary, the last part of this proposition, together with Takens’ Embedding Theorem (or, more generally, the main result in [Grig 23]), implies that generic observations of a large class of dynamical systems form CCIMs.

An important tool in the study of dynamical systems is the concept of ergodicity, which allows us to translate statements about space averages to averages along trajectories of the system, thus relating global analytic properties of a map to the dynamics it induces. Of particular relevance to our work is the study of nonuniform hyperbolicity and the multiplicative ergodic theorem of Oseledets [Osel 68, Ruel 79, Barr 07]. We briefly introduce the theory as applied to our context, keeping to the details and results necessary for our work. Our overview is based largely on [Barr 07, Chapter 1 and Chapter 3].

Let $M$ be an $q$-dimensional $C^{1}$ submanifold of $\mathbb{R}^{L}$ and let $\phi\in\operatorname{Diff}^{1}(M)$ be a dynamical system acting on $M$. A concept that allows us to quantify the rate of divergence of two points initially close together is that of the Lyapunov exponent. For a point $m\in M$ the forward Lyapunov exponent along a vector $v\in T_{m}M$ in the tangent space $T_{m}M$ of $M$ at the point $m$ is defined as

$\displaystyle\lambda^{+}(m,v)=\limsup_{t\to\infty}\frac{1}{t}\log\lVert D\phi^{t}(m)v\rVert,$

(2.10)

where $D\phi^{t}(m):T_{m}M\longrightarrow T_{\phi^{t}(m)}M$ is the tangent map of $\phi^{t}$ at the point $m\in M$. Similarly, we define the backward Lyapunov exponent

$\displaystyle\lambda^{-}(m,v)=\limsup_{t\to-\infty}\frac{1}{|t|}\log\lVert D\phi^{t}(m)v\rVert.$

(2.11)

It can be shown that, for any given point $m\in M$, the forward Lyapunov exponent $\lambda^{+}(m,\cdot)$ satisfies the following three properties:

(i)

$\lambda^{+}(m,\alpha v)=\lambda^{+}(v)$ for all $v\in T_{m}M$ and all $\alpha\in\mathbb{R}\setminus\{0\},$

(ii)

$\lambda^{+}(m,v+w)\leq\max\{\lambda^{+}(m,v),\lambda^{+}(m,w)\}$ for all $v,w\in T_{m}M,$ and

(iii)

$\lambda^{+}(0)=-\infty,$ where we make use of the convention $\log 0=-\infty.$

As a result, $\lambda^{+}(m,\cdot)$ takes on finitely many values $\lambda_{1}^{+}(m)>\cdots>\lambda_{l^{+}(m)}^{+}(m)$ for some $l^{+}(m)\leq q$. These are related to the differing rates of convergence/divergence along different tangent directions. The properties (i)–(iii) also allow us to construct a filtration of vector subspaces in the tangent space using the ordered Lyapunov exponents to single out linearly independent tangent directions up to a given Lyapunov value. The filtration is $\mathcal{V}^{+}(m)\colon\{0\}\subsetneq V_{l^{+}(m)}^{+}(m)\subsetneq\cdots\subsetneq V_{1}^{+}(m)=T_{m}M,$ where we define

$\displaystyle V_{i}^{+}(m):=\{v\in T_{m}M\colon\lambda^{+}(m,v)\leq\lambda_{i}^{+}(m)\},$

(2.12)

for $i=1,\dots,l^{+}(m),$ that is, $V_{i}^{+}(m)$ is the subspace of the tangent space consisting of all directions with asymptotic rate of divergence at most $\lambda_{i}^{+}(m).$ The number of tangent directions having a given rate of divergence is $k_{i}^{+}(m)=\dim V_{i}^{+}(m)-\dim V_{i+1}^{+}(m)$ for $i=1,\dots,l^{+}(m).$ The value $k_{i}^{+}(m)$ is referred to as the multiplicity of the Lyapunov exponent $\lambda_{i}^{+}(m).$

In like manner, for $m\in M$ the backward Lyapunov exponent $\lambda^{-}(m,\cdot)$ takes on finitely many values $\lambda_{1}^{-}(m)<\cdots<\lambda_{l^{-}(m)}^{-}(m)$ for some $l^{-}(m)\leq q$ and has a filtration $\mathcal{V}^{-}(m)\colon\{0\}\subsetneq V_{1}^{-}(m)\subsetneq\cdots\subsetneq V_{l^{-}(m)}^{-}(m)=T_{m}M,$ defined by

$\displaystyle V_{i}^{-}(m):=\{v\in T_{m}M\colon\lambda^{-}(m,v)\leq\lambda_{i}^{-}(m)\},$

(2.13)

for $i=1,\dots,l^{-}(m).$ In a similar fashion to the forward Lyapunov exponent, the multiplicity of the backward Lyapunov exponent $\lambda_{i}^{-}(m)$ is defined as $k_{i}^{-}(m)=\dim V_{i}^{-}(m)-\dim V_{i-1}^{-}(m)$ for $i=1,\dots,l^{-}(m).$

While the forward Lyapunov exponent captures the growth of tangent vectors in forward time, the backward exponent measures this growth in backward time. An obstacle in the definitions (2.10) and (2.11) is the $\limsup,$ which makes the growth of individual terms in the sequences intractable. To deal with this, we relate the forward and backward Lyapunov exponents and their corresponding filtrations. This relation is expressed in the notion of coherence. The filtrations $\mathcal{V}^{+}(m)$ and $\mathcal{V}^{-}(m)$ are said to be coherent if

(i)

$l^{+}(m)=l^{-}(m)=:l(m),$

(ii)

there is a decomposition $T_{m}M=\bigoplus_{i=1}^{l(m)}E_{i}(m)$ such that

$\displaystyle V_{i}^{+}(m)=\bigoplus_{j=i}^{l(m)}E_{j}(m)\quad\text{and}\quad V_{i}^{-}(m)=\bigoplus_{j=1}^{i}E_{j}(m)\quad\text{for }i=1,\dots,l(m),\text{ and}$

(2.14)

(iii)

$\lim_{t\to\pm\infty}\frac{1}{t}\log\lVert D\phi^{t}(m)v\rVert=\lambda_{i}^{+}(m)=-\lambda_{i}^{-}(m)=:\lambda_{i}(m)$ uniformly over $\{v\in E_{i}(m)\colon\lVert v\rVert=1\}.$

The subspaces $E_{i}(m)$ are called the Oseledets subspaces and the decomposition $T_{m}M=\bigoplus_{i=1}^{l(m)}E_{i}(m)$ is called the Oseledets splitting. The number of distinct Lyapunov exponents and the Oseledets subspaces are invariant along a trajectory of the dynamical system in the sense that $l(m)=l(\phi(m))$ and $D\phi(m)E_{i}(m)=E_{i}(\phi(m))$ for $i=1,\dots,l(m).$ The Oseledets subspaces can be constructed by taking $E_{i}(m)=V_{i}^{+}(m)\cap V_{i}^{-}(m)$ for $i=1,\dots,l(m).$ Expressing them in this way is instructive: the set $V_{i}^{+}(m)$ filters out those tangent directions which, in forward time, have greater Lyapunov exponent than $\lambda_{i}(m)$, while the set $V_{i}^{-}(m)$ filters out those that, in backward time, have greater Lyapunov exponent than $-\lambda_{i}(m)$. Thus the tangent directions left in $E_{i}(m)$ are in some sense pure, in that their Lyapunov exponent in forward and backward time are the same.

We focus on a certain regular class of points whose properties allow us to establish a link between the Lyapunov exponent and the rate of divergence of points in a nearby neighborhood using the theory of nonuniform hyperbolicity. In particular, for these points the $\limsup$s in (2.10) and (2.11) become limits and they possess a splitting as in (2.14), allowing us to decompose the tangent space into subspaces where the rate of expansion/contraction under the action of the dynamical system is uniform. A point $m\in M$ is said to be forward regular if

$$\lim_{t\to\infty}\frac{1}{t}\log\lvert\det D\phi^{t}(m)\rvert=\sum_{i=1}^{l^{+}(m)}k_{i}^{+}(m)\lambda_{i}^{+}(m)<\infty.$$

Likewise, $m$ is said to be backward regular if

$$\lim_{t\to-\infty}\frac{1}{|t|}\log\lvert\det D\phi^{t}(m)\rvert=\sum_{i=1}^{l^{-}(m)}k_{i}^{-}(m)\lambda_{i}^{-}(m)<\infty.$$

In both cases, the expression on the right is the sum of the distinct values of the Lyapunov exponent counted with multiplicity. Intuitively, this says that a small volume of space around the point $m$ grows at an exponential rate controlled by the sum of the Lyapunov exponents. Finally, we are ready to define the set of regular points: a point is said to be Lyapunov-Perron regular, or simply regular, if it is both forward and backward regular and the filtrations $\mathcal{V}^{+}(m)$ and $\mathcal{V}^{-}(m)$ are coherent. This brings us to Oseledets’ Multiplicative Ergodic Theorem [Barr 07, Theorem 3.4.3].

While above we gave the standard definition for a regular point, we require a slightly stronger consequence of regularity found in [Barr 07, Theorem 1.3.11]. The result involves a certain type of basis which is conveniently adapted to the associated filtration at the point.

The result we need is in fact an equivalent condition for forward regularity of a point, quantifying how a small volume at a point expands along trajectories. It allows us to bound how the angles between the Oseledets subspaces in the Oseledets splitting evolve with time. For a set $\{v_{1},\dots,v_{r}\}$ of linearly independent vectors, we denote by $\operatorname{vol}(v_{1},\dots,v_{r})$ the volume of the $r$-parallelepiped formed by these vectors. The following result can be found in [Barr 07, Theorem 1.3.11].