Information-theoretical analysis of statistical measures for multiscale dynamics

First, we consider decomposing $C^{m}(r)$ by the difference in the indices of the embedding vectors. Let $M=\lfloor N/\tau_{A}\rfloor$ be the length of the coarse-grained time series, and $D^{m}(i,j)$ be $d_{{\rm cheb}}(\bm{v}^{m}(i),\bm{v}^{m}(j))$. Considering the expectation and symmetry of $i$ and $j$, and assuming $s(t)$ or its first-order difference $s(t)-s(t-1)$ is strongly stationary,

First we clarify the meaning of expectation. In the following, we regard the time series $s(t),~{}t\in\{1,2,\cdots,N\}$ under study as a sequence of random variables following a specific probability distribution function (PDF). In this sense, $C^{m}(r)$ and $u(r-D^{m}(i,j))$ are also random variables, and thus each of the possible values of $C^{m}(r)$ and $u(r-D^{m}(i,j))$ has some probability of occurrence. The expectation is the weighted average of the likelihood of every potential value.

Here we explain the transformation of the equations from (16) to (19) line by line. The only change made from Eq. (16) to (17) is that the expectation term $\langle u\left(r-D^{m}(i,j)\right)\rangle$ is changed to the probability term $\Pr\left(D^{m}(i,j)<r\right)$. We assume that a random variable sequence $s(t),~{}t\in\{1,2,\cdots,N\}$ follows a PDF, and calculate $u\left(r-D^{m}(i,j)\right)$. As previously mentioned, $u\left(r-D^{m}(i,j)\right)$ is also a random variable that is either 0 or 1 with the following probability:

Consequently,

holds.

From Eq. (17) to (18), the factor $2$ is added to the numerator, and the range of summation over $j$ is restricted to $j>i$. Here, we use the fact that $D^{m}(i,j)=D^{m}(j,i)$. From Eq. (18) to (19), we must assume that $\Pr\left(D^{m}(i,j)<r\right)$ depends only on the difference in the indices $k=j-i$. For example, $\Pr\left(D^{m}(i+1,j+1)<r\right)$ is the same as $\Pr\left(D^{m}(i,j)<r\right)$. This assumption is satisfied when the original signal $s(t)$ or its first-order difference $s(t)-s(t-1)$ is strongly stationary. This is a reasonable assumption when studying a time series from a dynamical system. The factor $(M-m-k)$ in Eq. (19) comes from the fact that, for each $k$, the number of pairs $(i,j)$ satisfying $k=j-i$ is $(M-m-k)$. For example, when $k=1$, the pairs are $(1,2),(2,3),\cdots,(M-m-1,M-m)$, as the total number of embedded vectors is $M-m$.

The probability in Eq. (19) can be decomposed further by considering conditional probabilities. Using the proposition (LABEL:iff:cheb),

holds. The probability of a product event, such as Eq. (22), can be represented as a product of conditional probabilities. For example, the probability that propositions $A$, $B$ and $C$ hold simultaneously is

In the same way,

holds.

In the following, we show that the above conditional probability behaves oppositely to the Allan variance regardless of $k$, while the unconditional probability does not. This point is the key to understanding why $-C^{m}(r)$, the MSE and Allan variance show similar $\tau_{A}$ dependency.

Figure 6(a) shows a logarithmic-scale color map of $\Pr(D^{1}(1,1+k)<r)$, which is the first term on the right-hand side of Eq. (24), whereas Fig. 6(b) shows that of $\Pr(D^{1}(2,2+k)<r\mid D^{1}(1,1+k)<r)$, corresponding to the conditional probability in the second term on the right-hand side of Eq. (24) for $i=2$.

Each probability was calculated for the LFF time series. For $k=1$, corresponding to the lowest row in Fig. 6(a) and the solid curve in Fig. 6(c), the unconditioned probability $\Pr(D^{1}(1,2)<r)$ behaves opposite to the Allan variance (see Fig. 4(b)). However, for larger $k$, the behavior differs from that of the $k=1$ case, as shown in the dashed plot in Fig. 6(c).

This can be understood as follows. First, the following equation holds:

For $k=1$, whether $D^{1}(1,2)=|\Delta\bar{s}^{(\tau_{A})}(2)|$ is smaller than $r$ is closely related to the Allan variance. If the Allan variance for a certain $\tau_{A}$ is smaller than that of another $\tau_{A}$, the $\Delta\bar{s}^{(\tau_{A})}(l)$ distribution is biased toward the center, because the mean value of $\Delta\bar{s}^{(\tau_{A})}(l)$ is zero by definition. Consequently, the probability $D^{1}(1,2)=|\Delta\bar{s}^{(\tau_{A})}(2)|<r$ is high for that $\tau_{A}$. For larger $k$, the $\sum_{j=2}^{k+1}\Delta\bar{s}^{(\tau_{A})}(j)$ distribution is affected by the time correlation of $\bar{s}^{(\tau_{A})}(l)$ that the time series under study inherently has, and the Allan variance $\operatorname*{Var}(\Delta\bar{s}^{(\tau_{A})}(l))$ cannot predict the $\sum_{j=2}^{k+1}\Delta\bar{s}^{(\tau_{A})}(j)$ distribution well, so $\Pr(D^{1}(1,1+k)<r)$ does not behave in a manner that is strongly correlated with the Allan variance.

Conversely, the conditional probability $\Pr(D^{1}(2,2+k)<r\mid D^{1}(1,1+k)<r)$ depicted in Fig. 6 (b) shows a similar trend to that of the Allan variance regardless of $k$, which we can also observe in Fig. 6(d). That is, $\Pr(D^{1}(2,2+k)<r\mid D^{1}(1,1+k)<r)$ is the main connection that explains the similarity between the MSE and Allan variance. To examine this further, we discuss the variance of the corresponding quantity, rather than considering the probabilities in the following sections.

In this section, we introduce an assumption called the Neighborhood-Likelihood-to-Variance-Relationship (NLVR) to connect the probability discussion to the variance discussion. First, define $\Delta\Pr(\tau_{A})$ as the difference between $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ for $\tau_{A}$ and $\tau_{A}-1$. In the same way, $\Delta\operatorname*{Var}(\tau_{A})$ is the difference between $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$ for $\tau_{A}$ and $\tau_{A}-1$. Please note that the conditional probability $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ is the same as that in Eq. (24), and the absolute value of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$, referred to in $\Delta\operatorname*{Var}(\tau_{A})$, is identical to $D^{1}(i,i+k)$ in the definition of $\Delta\Pr(\tau_{A})$.

Then, to connect the probability with the variance, we assume the Neighborhood-Likelihood-to-Variance-Relationship (NLVR), as follows:

The inequality (26) states that the signs of $\Delta\Pr(\tau_{A})$ and $\Delta\operatorname*{Var}(\tau_{A})$ are always opposite.

We will explain the NLVR in more detail. Both $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ and $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$ are statistics of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ under the same condition of $D^{i-1}(1,1+k)<r$. Considering the PDF of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$, $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ represents the area under the PDF in the range $[-r,~{}r]$. In addition, when there is no condition, the mean of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ is zero by definition. Here we also assume that the mean value of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ is close to zero under the condition $D^{i-1}(1,1+k)<r$. When $\Delta\Pr(\tau_{A})>0$, the distribution of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ is more centrally biased. Accordingly, when $\Delta\Pr(\tau_{A})>0$, $\Delta\operatorname*{Var}(\tau_{A})$ is likely to be negative.

Figure 5 schematically illustrates the concept of the NLVR. The black curves in Figs. 5 (a) and (b) represent the PDF of $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ under the condition $D^{i-1}(1+k,1)<r$ for $\tau_{A}-1$ and $\tau_{A}$, respectively. The red-filled areas and black bars show $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ and $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$, respectively. $\Delta\Pr(\tau_{A})$ and $\Delta\operatorname*{Var}(\tau_{A})$ are the difference in the red-filled areas and black bars between Figs. 5(b) and (a), respectively. Please note that the plots shown here, based on Gaussian distributions, are for explanatory purposes only and are not obtained from a time series. The validity of the NLVR is discussed in Sec. IV.3.

Here we further discuss the conditional variance $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$, instead of the conditional probability $\Pr(D^{1}(i,i+k)<r\mid D^{i-1}(1,1+k)<r)$ following the NLVR assumption, as mentioned in Sec. IV.1.1. By decomposing $\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)$ as

the conditional variance, $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$, can be decomposed as follows:

Here, the condition $D^{i-1}(1,1+k)<r$ is abbreviated by the symbol $\cdots$ on the right-hand side of Eq. (28).

There are six terms on the right-hand side of Eq. (28). The logarithmic scale color map of each term is shown in Figs. 7(a)–(f) to determine which term has the greatest influence on the conditional variance. Figure 7(g) represents the summation of each term, which is the original conditional variance. Meanwhile, Fig. 7(h) show cross-sectional profiles when $k=50$ regarding the first term (Fig. 7(a)), the third term (Fig. 7(c)) and the total conditional variance (Fig. 7(g)).

As discussed shortly below, the first through third terms in Eq. (28) are dominant, as shown in Figs. 7(a)–(c), whereas they show a similar $\tau_{A}$ dependency with the Allan variance regardless of $k$, as indicated in Fig. 7(h).

Please note that we plotted the absolute values of the covariance terms, because the covariance can be negative. However, the $\tau_{A}$ dependency of $\operatorname*{Cov}(\Delta\bar{s}^{(\tau_{A})}(i+k),\Delta\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$, shown in Figs. 7(c) and (h), is still similar to that of the Allan variance. Conversely, the remaining panels (Figs. 7(d)–(f)) exhibit small values, regardless of $k$ and $\tau_{A}$.

Therefore, we observe that the conditional variance $\operatorname*{Var}(\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)\mid D^{i-1}(1,1+k)<r)$, which is shown in Figs. 7(g) and (h), and the conditional probability $\Pr(|\bar{s}^{(\tau_{A})}(i+k)-\bar{s}^{(\tau_{A})}(i)|<r\mid D^{i-1}(1,1+k)<r)$, shown in Fig. 6(b) for the case of $i=2$, exhibit the opposite $\tau_{A}$ dependency to the Allan variance from the NLVR.

The first and second terms are the conditional variances of $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$, namely the conditional Allan variances, respectively. Although the condition somewhat affects the $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$ distribution, the size relationship of the variance of $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$ concerning $\tau_{A}$ is almost the same as the Allan variance.

However, the third term is strongly influenced by the condition. In the absence of the condition, if $k$ is sufficiently large, there would be little correlation between $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$; thus, the covariances are expected to be small. Fig. 8(a) shows the unconditional covariance of $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$, where the color scale is the same as in Fig. 7. Comparing to the conditional covariance plotted in Fig. 7(c), this covariance without the condition is smaller in most places.

By contrast, when the condition is satisfied, $i-1$ consecutive points up to $\bar{s}^{(\tau_{A})}(i+k-1)$ and $\bar{s}^{(\tau_{A})}(i-1)$ are close values. In that case, $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$, i.e., the difference between these points and their consecutively following point, often have the same sign, and the distribution of their magnitudes follows the Allan variance.

Figure 8(b) shows an example of a pair of $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$ satisfying the condition for $i=3$. Because we can regard $\bm{v}^{2}(7)$ and $\bm{v}^{2}(17)$, denoted by the blue dots, as identical under the tolerance $r$, the red arrows, representing $\Delta\bar{s}^{(\tau_{A})}(9)$ and $\Delta\bar{s}^{(\tau_{A})}(19)$ are likely to be similar. By the definition of covariance, if $\Delta\bar{s}^{(\tau_{A})}(i+k)$ and $\Delta\bar{s}^{(\tau_{A})}(i)$ are similar, $\operatorname*{Cov}(\Delta\bar{s}^{(\tau_{A})}(i+k),\Delta\bar{s}^{(\tau_{A})}(i))$ is close to $\operatorname*{Var}(\Delta\bar{s}^{(\tau_{A})}(i+k))$, namely the Allan variance. From these considerations, the conditional covariance behaves similarly to the Allan variance.

Please note that the form of the condition in the Fig. 8(b), namely $D^{2}(7,17)<r$ is slightly different from $D^{i-1}(1,1+k)<r$. However, as we assume stationary, it is allowed to shift the indices. For example, in this case, we examined $\Delta\bar{s}^{(\tau_{A})}(19)$ and $\Delta\bar{s}^{(\tau_{A})}(9)$ under the condition of $D^{2}(7,17)<r$, and this corresponds to the shifting of indices by six. To be more specific, we investigated $\Delta\bar{s}^{(\tau_{A})}(19)=\Delta\bar{s}^{(\tau_{A})}(3+10+6)$ and $\Delta\bar{s}^{(\tau_{A})}(3+6)$, under the condition of $D^{2}(7,17)<r=D^{2}(1+6,1+10+6)$, where $i=3$ and $k=10$.