On the entropy of rectifiable and stratified measures

The starting point of our considerations is the asymptotic equipartition property:

The proof of this proposition only depends on the weak law of large numbers, which ensures the convergence in probability of $-\frac{1}{n}\sum_{i=1}^{n}\ln f(x_{i})$ to its mean $H_{\mu}(\rho)$, see [11, Ch. 12] and [3, Ch. 8].

Shannon’s discrete entropy and differential entropy are particular cases of (2): the former when $\mu$ is the counting measure on a discrete set equipped with the $\sigma$-algebra of all subsets; the latter when $E_{X}=\mathbb{R}^{d}$, $\mathfrak{B}$ is generated by the open sets, and $\mu$ is the Lebesgue measure. Up to a sign, the Kullback-Leibler divergence, also called relative entropy, is another particular case, that arises when $\mu$ is a probability measure.

One can imagine other examples, geometric in nature, that involve measures $\mu$ on $\mathbb{R}^{d}$ that are singular continuous. For instance, $E_{X}$ could be a Riemannian manifold equipped with the Borel measure given by integration of its Riemannian volume form. The measure-theoretic nature of Proposition 1 makes the smoothness in this example irrelevant. It is more natural to work with geometric measure theory.

Geometric measure theory “could be described as differential geometry, generalized through measure theory to deal with maps and surfaces that are not necessarily smooth, and applied to the calculus of variations” [9]. The place of smooth maps is taken by Lipschitz maps, which are differentiable almost everywhere [2, p. 46]. In turn, manifolds are replaced by rectifiable sets, and the natural notion of volume for such sets is the Hausdorff measure.

Let $\rho$ be a locally finite measure and $s$ a nonnegative real number. Marstrand proved that if the limiting density $\Theta_{s}(\rho,x):=\lim_{r\downarrow 0}\rho(B(x,r))/(w_{s}r^{s})$ exists and is strictly positive and finite for $\rho$-almost every $x$, then $s$ is an integer not greater than $n$. Later Preiss proved that such a measure is also $s$-rectifiable in the sense of the following definition. For details, see e.g. [5].

The study of these measures from the viewpoint of information theory, particularly the properties of the entropy $H_{\mathcal{H}^{m}}(\rho)$ of an $m$-rectifiable probability measure $\rho$, was carried out relatively recently by Koliander, Pichler, Riegler, and Hlawatsch in [7]. We provide here an idiosyncratic summary of some of their results.

First, remark that in virtue of (5), an $m$-rectifiable measure $\nu$ is absolutely continuous with respect to the restricted measure $\mathcal{H}^{m}|_{E^{*}}$, where $E^{*}$ is countably $m$-rectifiable and has the form $\bigcup_{i\in\mathbb{N}}f_{i}(K_{i})$ with $f_{i}$ injective and $K_{i}$ Borel and bounded. (A refinement of this construction gives a similar set such that, additionally, the density of $\rho$ is strictly positive [7, App. A].) Although the product of an $(\mathcal{H}^{m_{1}},m_{1})$-rectifiable set and an $(\mathcal{H}^{m_{2}},m_{2})$ rectifiable set is not $(\mathcal{H}^{m_{1}+m_{2}},m_{1}+m_{2})$-rectifiable—see [6, 3.2.24]—the carriers behave better.

Let $\rho$ be an $m$-rectifiable measure, with carrier $E$ (we drop the $*$ hereon). It holds that $\rho\ll\mathcal{H}^{m}|_{E}$ and $\mathcal{H}^{m}|_{E}$ is $\sigma$-finite. If moreover $H_{\mathcal{H}^{m}|_{E}}(\rho)<\infty$, Proposition 1 gives estimates for $(\mathcal{H}^{m}|_{E})^{\otimes n}(\mathcal{W}^{(n)}_{\delta}(\rho;\mathcal{H}^{m}|_{E}))$. Lemma 2 tells us that $E^{n}$ is $mn$-rectifiable and that $(\mathcal{H}^{m}|_{E})^{\otimes n}=\mathcal{H}^{mn}|_{E^{n}}$, which is desirable because the Hausdorff dimension of $E^{n}$ is $mn$ and $\mathcal{H}^{mn}$ is the only nontrivial measure on it as well as on $W^{(n)}$, which as a subset of $E^{n}$ is $mn$ rectifiable too.

To apply the AEP we need to compute $H_{\mathcal{H}^{m}|_{E}}(\rho)$. In some cases, one can use the area formula (Proposition 2) to “change variables” and express $H_{\mathcal{H}^{m}|_{E}}(\cdot)$ in terms of the usual differential entropy. For instance, suppose $A$ is a bounded Borel subset of $\mathbb{R}^{k}$ of nontrival $\mathcal{L}^{k}$-measure and $f$ is an injective Lipschitz function on $A$. The set $f(A)$ is $k$-rectifiable. Moreover, if $\rho$ is a probability measure such that $\rho\ll\mathcal{L}^{k}|_{A}$ with density $r$, then the area formula applied to $u=r/J_{k}f$ and $E=f^{-1}(B)$, for some Borel subset $B$ of $\mathbb{R}^{d}$, shows that $f_{*}\rho\ll\mathcal{H}^{k}|_{f(A)}$ with density $(r/J_{k}f)\circ f^{-1}$, which is well-defined $(\mathcal{H}^{k}|_{f(A)})$-almost surely. A simple computation yields

There is a more general formula of this kind when $A$ is a rectifiable subset of $\mathbb{R}^{d}$.

Finally, we deduce the chain rule for the entropy of rectifiable measures as a consequence of our general theorem for disintegrations (Remark 1). Let $E$ be a countably $(\mathcal{H}^{m},m)$-rectifiable subset of $\mathbb{R}^{k}$, $f:\mathbb{R}^{k}\to\mathbb{R}^{d}$ a Borel function (with $d\leq m$), and $\rho$ a probability measure such that $\rho\ll\mathcal{H}^{m}|_{E}$. Because $\mathcal{H}^{m}|_{E}$ has an $(f,\mathcal{L}^{d})$-disintegration $\{F^{-1}\mathcal{H}^{m-d}|_{E\cap f^{-1}(t)}\}_{t\in\mathbb{R}^{k}}$, with $F=C_{d}d^{E}f$, then $f_{*}\rho\ll\mathcal{L}^{d}$, and

The probabilities $\rho_{t}$ are described in Remark 1. If one insists in only using the Hausdorff measures as reference measures, one must rewrite the integrand in (13) using the chain rule for the Radon-Nikodym derivative:

One recovers in this way the formula (50) in [7].

Thus $1$-stratified measures are rectifiable measures. If $\nu$ is $k$-stratified for some $k$ we simply say that $\nu$ is a stratified measure.

A fundamental nontrival example to bear in mind is a discrete-continuous mixture, which corresponds to $k=2$, $E_{1}$ countable, and $E_{2}=\mathbb{R}^{d}$. More generally, a stratified measure has a Lebesgue decomposition with a singular continuous part provided some $m_{i}$ is strictly between $0$ and $d$.

Let $\rho$ be a probability measure that is stratified in the sense above. We can always put it in the standard form $\rho=\sum_{i=1}^{k}q_{i}\rho_{i}$, where each $\rho_{i}$ is a rectifiable probability measure with carrier $E_{i}$ of dimension $m_{i}$ (so that $\rho_{i}=\rho_{i}|_{E_{i}}$), the carriers $(E_{i})_{i=1}^{k}$ are disjoint, $0\leq m_{1}<\cdots<m_{k}\leq d$, and $(q_{1},...,q_{k})$ is a probability vector with strictly positive entries. The carriers can be taken to be disjoint because if $E$ has Hausdorff dimension $m$, then $\mathcal{H}^{k}(E)=0$ for $k>m$, hence one can prove [13, Sec. IV-B] that $E_{i}\setminus(\bigcup_{j=1}^{i-1}E_{j})$ is a carrier for $\nu_{i}$, for $i=2,...,k$.

We can regard $\rho$ as the law of a random variable $X$ valued in $E_{X}:=\bigcup_{i=1}^{k}E_{i}$ and the vector $(q_{1},...,q_{k})$ as the law $\pi_{*}\rho$ of the discrete random variable $Y$ induced by the projection $\pi$ from $E_{X}$ to $E_{Y}:=\{1,...,k\}$ that maps $x\in E_{i}$ to $i$. We denote by $D$ the random variable $\dim_{H}E_{Y}$, with expectation $\mathbb{E}(D)=\sum_{i=1}^{k}m_{i}q_{i}$.

The measure $\rho$ is absolutely continuous with respect to $\mu=\sum_{i=1}^{k}\mu_{i}$, where $\mu_{i}=\mathcal{H}^{m_{i}}|_{E_{i}}$, so it makes sense to consider the entropy $H_{\mu}(\rho)$; it has a concrete probabilistic meaning in the sense of Proposition 1. Moreover, one can prove that $\frac{\mathrm{d}\rho}{\mathrm{d}\mu}=\sum_{i=1}^{m}q_{i}\frac{\mathrm{d}\rho_{i}}{\mathrm{d}\mu_{i}}\mathds{1}_{E_{i}}$ [13, Lem. 3] and therefore

holds [13, Lem. 4]. This formula also follows form the chain rule for general disintegrations (Remark 13), because $\{\rho_{i}\}_{i\in E_{Y}}$ is a $(\pi,\pi_{*}\rho)$-disintegration of $\rho$.

The powers of $\rho$ are also stratified. In fact, remark that

where $N(a;\mathbf{y})$ counts the appearances of the symbol $a\in E_{Y}$ in the word $\mathbf{y}$. Each measure $\rho_{\mathbf{y}}:=\rho_{y_{1}}\otimes\cdots\otimes\rho_{y_{n}}$ is absolutely continuous with respect to $\mu_{\mathbf{y}}:=\mu_{y_{1}}\otimes\cdots\otimes\mu_{y_{n}}$. It follows from Lemma 2 that for any $\mathbf{y}\in E_{Y}^{n}$, the stratum $\Sigma_{\mathbf{y}}:=E_{y_{1}}\times\cdots\times E_{y_{n}}$ is also a carrier, of dimension $m(\mathbf{y}):=\sum_{j=1}^{n}\dim_{H}E_{y_{j}}$, and the product measure $\mu_{\mathbf{y}}$ equals $\mathcal{H}^{m(\mathbf{y})}|_{\Sigma_{\mathbf{y}}}$. Therefore each measure $\rho_{\mathbf{y}}$ is rectifiable. We can group together the $\rho_{\mathbf{y}}$ of the same dimension to put $\rho^{\otimes n}$ as in Definition 3.

By Proposition 1, one might approximate $\rho^{\otimes n}$ with an arbitrary level of accuracy by its restriction to the weakly typical realizations of $\rho$, provided $n$ is big enough. In order to get additional control on the dimensions appearing in this approximation, we restrict it further, retaining only the strata that correspond to strongly typical realizations of the random variable $Y$.

Let us denote by $Q$ the probability mass function (p.m.f) of $\pi_{*}\rho$. Recall that $\mathbf{y}\in E_{Y}^{n}$ induces a probability law $\tau_{\mathbf{y}}$ on $E_{Y}$, known as empirical distribution, given by $\tau_{\mathbf{y}}(\{a\})=N(a;\mathbf{y})/n.$ Csiszár and Körner [4, Ch. 2] define $\mathbf{y}\in E_{Y}^{n}$ to be strongly $(Q,\eta)$-typical if $\tau_{\mathbf{y}}$, with p.m.f. $P$, is such that $\tau_{\mathbf{y}}\ll\pi_{*}\rho$ and, for all $a\in E_{Y}$, $|P(a)-Q(a)|<\eta$. We denote by $A_{\delta^{\prime}_{n}}^{(n)}$ the set of these sequences when $\eta_{n}=n^{-1/2+\xi}$. In virtue of the union bound and Hoeffding’s inequality, $(\pi_{*}\rho)^{\otimes n}(A_{\delta^{\prime}_{n}}^{(n)})\geq 1-\varepsilon_{n}$, where $\varepsilon_{n}=2|E_{Y}|\mathrm{e}^{-2n\eta_{n}^{2}}$; the choice of $\eta_{n}$ ensures that $\varepsilon_{n}\to 0$ as $n\to\infty$. Moreover, the continuity of the discrete entropy in the total-variation distance implies that $A^{(n)}_{\delta^{\prime}_{n}}$ is a subset of $W^{(n)}_{\delta^{\prime}_{n}}$ with $\delta_{n}^{\prime}=-|E_{Y}|\eta_{n}\ln\eta_{n}$, which explains our notation. See [13, Sec. III-D]

We introduce the set $T^{(n)}_{\delta,\delta^{\prime}_{n}}=(\pi^{\times n})^{-1}(A_{\delta^{\prime}_{n}}^{(n)})\cap W_{\delta}^{(n)}(\rho)$ of doubly typical sequences in $E_{X}^{n}$, and call $T^{(n)}_{\delta,\delta^{\prime}_{n}}(\mathbf{y})=T^{(n)}_{\delta,\delta^{\prime}_{n}}\cap(\pi^{\times n})^{-1}(\mathbf{y})$ a doubly typical stratum for any $\mathbf{y}\in A_{\delta^{\prime}_{n}}^{(n)}$.

The main result of [13] is a refined version of the AEP for stratified measures that gives an interpretation for the conditional term in the chain rule (14).

This gives a geometric interpretation to the possibly noninteger dimension $\mathbb{E}(D)=\sum_{i=1}^{k}q_{i}m_{i}$, which under suitable hypotheses is the information dimension of $\rho$ [13, Sec. V], thus answering an old question posed by Renyi in [10, p. 209].