The Information-Geometric Perspective of Compositional Data Analysis

To exploit the equivalence of probability distributions with compositional data, we only consider the manifold of discrete distributions $\mathcal{S}^{D}$. To emphasize the equivalence, we will denote the probabilities by the same symbol as our compositions, i.e., $\boldsymbol{x}\in\mathcal{S}^{D}$ is a vector of $D$ probabilities. To complete the probabilistic picture, we need a random variable $R$ which can take values $r\in\{1,\dots,D\}$ with the respective probabilities $x_{r}=\mathbb{P}\{R=r\}$, the coordinates of $\boldsymbol{x}$. The distribution of this random variable can now be written as

From the information-geometric perspective, there are two natural ways to parametrize the set ${\mathcal{S}}^{D}$ of all (strictly positive) distributions of $R$. The first possibility is quite obvious. The first $D-1$ probabilities in Eq. (16) that are free to be specified, that is $x_{1},\dots,x_{D-1}$, can be considered parameters, which we denote by $\boldsymbol{\eta}=(x_{1},\dots,x_{D-1})^{T}$. In these coordinates, probability distributions are written as

Alternatively, our distribution can be parametrized using what is known as the alr-transformation logratioTrans in CoDA:

With this, we can write our distribution in the form

where ${\mathbbm{1}}_{k}(r)=1$ if $r=k$, and ${\mathbbm{1}}_{k}(r)=0$ otherwise. The function $\psi$ ensures normalization, that is

The parametrization of Eq. (19) in terms of $\boldsymbol{\theta}\in\mathbb{R}^{D-1}$ can be used in order to define a linear structure on ${\mathcal{S}}^{D}$. The addition of two distributions $p(\cdot;\boldsymbol{\theta}_{1})$ and $p(\cdot;\boldsymbol{\theta}_{2})$ can simply be defined by taking their product and then normalizing. With this vector addition, denoted by $\oplus$, we obviously have

Thus, the operation $\oplus$ is consistent with the usual addition in the parameter space $\mathbb{R}^{D-1}$. The multiplication of a distribution $p(\cdot;\boldsymbol{\theta})$ with a scalar $\alpha\in\mathbb{R}$ can be correspondingly defined by potentiating and then normalizing. This defines a scalar multiplication $\odot$, and we have

Obviously, the scalar multiplication $\odot$ is consistent with the usual multiplication in the parameter space. Note that the vector space structure defined here, which is well known in information geometry, coincides with the structure defined by equations (2) and (3). Given the linear structure, we can consider affine subspaces of $\mathcal{S}^{D}$. These are well-known and fundamental families in statistics, statistical physics, and information geometry, the so-called exponential families. We basically obtain them from the representation of Eq. (19) if we replace the functions ${\mathbbm{1}}_{k}$ by $d$ ($d\leq D-1$) functions $X_{k}:\{1,\dots,D\}\to\mathbb{R}$, and shift the whole family by some reference measure $p_{0}$:

Here, $\psi(\boldsymbol{\theta})$ ensures normalization, but it does not reduce to the simple structure of Eq. (20) in general. In statistical physics, the function $\psi$ is known as the free energy (in other contexts it is also known as the cumulant-generating function). Note that, given the same linear structure on $\mathcal{S}^{D}$, the exponential families coincide with the linear manifolds of Eq. (4), which were introduced into the field of Compositional Data Analysis more recently.

In what follows we restrict attention to the parametrizations of equations (17) and (19) of the full simplex $\mathcal{S}^{D}$ as one instance of the general structure that underlies information geometry. The function $\psi$, given by Eq. (20), is a convex function, and we can get back the $\boldsymbol{\eta}$ coordinates from it via a Legendre transformation AmariNagaoka

where $\nabla$ denotes the partial derivatives $(\partial\psi/\partial\theta^{i})_{i=1}^{D}$. The Legendre dual of $\psi(\boldsymbol{\theta})$ is another convex function defined by

which is given by the negative Shannon entropy

In equivalence to Eq. (24), from $\phi(\boldsymbol{\eta})$ we can get back the $\boldsymbol{\theta}$ coordinates:

There is thus a fundamental duality mediated by the Legendre transformation which links the two types of parameters $\boldsymbol{\eta}$ and $\boldsymbol{\theta}$ as well as the convex functions $\psi$ and $\phi$. Legendre transformations are well known to play a fundamental role in phenomenological thermodynamics. Their importance for information geometry was established by Amari and Nagaoka AmariNagaoka .
In what follows, we use the parameters $\boldsymbol{\theta}$. Like $\boldsymbol{\eta}$, they define a (local) coordinate system of the manifold $\mathcal{S}^{D}$. From each point $\boldsymbol{\theta}$, $D-1$ coordinate curves $\gamma_{i}:t\mapsto\gamma_{i}(t)$ in ${\mathcal{S}}^{D}$, $\gamma_{i}(0)=p_{\boldsymbol{\theta}}$, emerge when holding $D-2$ of the $\theta_{i}$ constant. Consider their velocities

In the point $\boldsymbol{\theta}$ itself, the $D-1$ vectors $\mathrm{e}_{i}$ pointing in the direction of each coordinate curve form the basis of the so-called tangent space ${T}_{\boldsymbol{\theta}}\mathcal{S}^{D}={\mathcal{T}}^{D}$ of this point, see Figure 1a). Similarly, we can define vectors $\mathrm{e}^{i}$ with respect to the coordinates $\boldsymbol{\eta}$, which also span the tangent space.

For a Riemannian manifold a metric tensor $\mathbf{G}$ is defined. This metric is obtained via an inner product on the tangent space:

which depends on the coordinates chosen. The coordinate system is Euclidean if $g_{ij}=\delta_{ij}$ (this is the case for the parametrization achieved by Eq. 13). For the Riemannian metric of exponential families, the basis vectors can be identified with the so-called score¹¹1The score function plays an important role in maximum-likelihood estimation. function $\partial\log p(r,\boldsymbol{\theta})/\partial\theta^{i}$. The resulting Riemannian metric is known as the Fisher information matrix:

with $E$ denoting the expectation value, and

Note that this is the covariance matrix of the random vector $(\mathbbm{1}_{1},\dots,\mathbbm{1}_{D-1})$, as expressed by Eq. (31). Convex functions have positive definite Hessian matrices. Here their elements are given by the Fisher metric itself:

The second matrix is the inverse of the first, which follows from their Legendre duality. This means the second matrix is an inverse covariance, an important object in the theory of graphical models graphMod . Although the Fisher metric is not Euclidean, i.e., $\left<\mathrm{e}_{i},\mathrm{e}_{j}\right>\neq\delta_{ij}$, we do have a generalization of this when mixing the dual coordinates: $\left<\mathrm{e}_{i},\mathrm{e}^{j}\right>=\delta_{ij}$.

Note that for a general exponential family of the form of Eq. (23), the Fisher information matrix can be expressed as a covariance matrix, which generalizes Eq. (31):

Our affine structure on $\mathcal{S}^{D}$ can be reformulated additively via an exponential map $\mathcal{S}^{D}\times\mathcal{T}^{D}\to\mathcal{S}^{D}$, $(\boldsymbol{x},\boldsymbol{v})\mapsto\boldsymbol{y}$:

using the notation introduced in section 2, and shown here together with its inverse. This map is used in AyErb , where ordinary linear differential equations with the time derivative defined for $\mathrm{vec}(\boldsymbol{x}(t_{0}),\boldsymbol{x}(t))$ letting $t\to t_{0}$ are considered. These turn out to be replicator equations with special properties that are known from population dynamics. Note that, e.g., with the center of the simplex $\boldsymbol{n}$ as defined before, we have $\mathrm{vec}(\boldsymbol{n},\boldsymbol{x})=\mathrm{clr}(\boldsymbol{x})$. With the exponential map, we can interpret $\mathrm{vec}(\boldsymbol{x},\boldsymbol{y})$ as the difference vector between two compositions, see Figure 1b). The set of all such difference vectors for a given point can be interpreted as the gradient field of a convex function (a.k.a. a potential). In order to highlight the generality of this concept in information geometry, we consider a general convex function $U$ with respect to some parameters $\boldsymbol{\eta}$ from a convex domain. In the following, we will denote the compositions for which the parameters are to be evaluated by subscripts. The linearization of $U$ in $\boldsymbol{\eta}_{\boldsymbol{y}}$ is given by

The graph of this linearization is a hyperplane of dimension $D-1$ touching the graph of $U$ in the point $(\boldsymbol{\eta}_{\boldsymbol{y}},U(\boldsymbol{\eta}_{\boldsymbol{y}}))$, see Figure 2. The difference between $U$ and its linearisation $\overline{U}$ in $\boldsymbol{\eta}_{\boldsymbol{y}}$, evaluated at $\boldsymbol{\eta}_{\boldsymbol{x}}$, defines a so-called Bregman divergence, a class of divergences that plays an important role in information geometry. More precisely,

Divergences are similar to distance functions but they are not necessarily symmetric and need not fulfill the triangle inequality. As an example, let us consider the potential naturally associated with the structure of equations (39) and (40), the squared Aitchison norm

with ilr_i the $i$-th ilr coordinate $z_{i}$, see Eq. (13), and $\boldsymbol{z}_{\boldsymbol{x}}$ the vector of coordinates $z_{i}$. We then have

coinciding with the squared Aitchison distance. This is the special case of a Euclidean divergence, which is also a (squared) distance function.

Let us now come to the divergences that arise when replacing $U(\boldsymbol{\eta})$ in Eq. (41) by our dually convex functions $\psi(\boldsymbol{\theta})$ and $\phi(\boldsymbol{\eta})$. They turn out to be the relative entropies (a.k.a. Kullback-Leibler divergences)

Thus the symmetry we had in the Euclidean case finds its generalization for our dual case in

Moreover, one can show that we can “complete the square” via

Of course, symmetrizations of relative entropy exist, with the Jensen-Shannon divergence perhaps the most prominent example. Also, a symmetric “compositional” version of relative entropy has been proposed because of its additional properties that are often regarded as indispensable²²2As an example, the translation invariance of distance measures, known under the name of perturbation invariance in CoDA, has its information-geometric analog in the invariance of the inner product of tangent vectors $\mathbf{u}$, $\mathbf{v}$ under parallel transport $P$ and its dual $P^{*}$: $\left<\mathbf{u},\mathbf{v}\right>=\left<P(\mathbf{u}),P^{*}(\mathbf{v})\right>$. in CoDA symmDiv . While such measures have some interesting properties, they do not make use of the duality of our parametrizations and are therefore less suitable for our approach. Indeed, although a dual divergence is not a distance measure in the strict sense, it can quantify the distance between points along a curve in a similar way, and generalizations of well-known relationships from Euclidean geometry are available. Geodesic lines connecting two compositions can be constructed via convex combinations of the parameters $\boldsymbol{\theta}$, and the corresponding dual geodesics from convex combinations of the dual parameters $\boldsymbol{\eta}$. Such geodesics are orthogonal to each other when the inner product of their tangent vectors, with respect to the Fisher metric, vanishes in the point of intersection. In this case, a generalized Pythagorean theorem holds for the corresponding dual divergence, e.g.,

where the geodesics $t\boldsymbol{\eta}_{\boldsymbol{x}}+(1-t)\boldsymbol{\eta}_{\boldsymbol{z}}$ and $t\boldsymbol{\theta}_{\boldsymbol{z}}+(1-t)\boldsymbol{\theta}_{\boldsymbol{y}}$ intersect orthogonally.

We have seen that the potential of Eq. (42) led to a divergence that is also a Euclidean distance. Let us now consider a generalization of our dual divergences that includes as a special case a Euclidean distance that is related to the Fisher metric. The so-called $\alpha$-divergence (closely related to the Renyi Renyi and Tsallis Tsallis entropies) is defined as

In the limit, the cases $\alpha=\pm 1$ correspond to $D_{\psi}$ and $D_{\phi}$. The case $\alpha=0$ (where the divergence is self-dual, i.e., symmetric) corresponds to the Euclidean distance of the points $(\sqrt{x_{1}},\dots,\sqrt{x_{D}})$ and $(\sqrt{y_{1}},\dots,\sqrt{y_{D}})$:

This is the so-called Hellinger distance. It is closely related to the Riemannian distance³³3The Riemannian distance between two points on a manifold is the minimum of the lengths of all the piecewise smooth paths joining the two points. between two compositions with respect to the Fisher metric. This so-called Fisher distance can be expressed explicitly Nihat as

and its relation to the Hellinger distance is given by

These distances can be better understood when noting the role that plays the angle between the two points on the sphere, i.e., when comparing Eq. (52) with the cosine of the angle $\varphi$ between the rays going from the origin through the transformed compositions (see Fig. 3), also referred to as Bhattacharyya coefficient Bcoeff :

Let us now come back to Aitchison distance and discuss in which structural aspects it differs from divergences obtained from the Fisher metric. In fact, in data analysis, parametrized classes of distance measures are common green3 . In the same way as in Eq. (49), they are mediated by the Box-Cox transformation, which has the limit

This has been applied in CoDA to obtain log-ratio analysis as a Correspondence Analysis of power-transformed data green1 . There, we have the following family of distance measures:

where $\omega_{i}$ are suitable weights. For the case $\beta=1$, this is the (square of the) symmetric $\chi^{2}$-distance used in Correspondence Analysis, while $\beta=0$ gives⁴⁴4This can be seen as the high-temperature limit in statistical physics. Aitchison distance (when $\omega_{i}=D^{2}$ for all $i$), see the Appendix for a proof. Although the case $\beta=1/2$ has a direct relationship with it, Hellinger distance does not form part of this family because of the closure operation that makes us stay inside the simplex. Similarly, Aitchison distance cannot be obtained from the alpha divergences of Eq. (49). Alpha divergences are included in a more general class of divergences known under the name of $f$-divergence. They have the form

where $f$ is a convex function. It is a well-established result in information geometry that $f$-divergences are the only decomposable⁵⁵5A divergence is decomposable if it can be written as a sum of terms that only depend on individual components. divergences that behave monotonically under coarse-graining of information Amari , i.e., when compositional parts are amalgamated into higher-level parts. This important invariance property is called information monotonicity. Aitchison distance is not decomposable, as in each summand we use information from all compositional parts when evaluating their geometric mean. Nevertheless, in the next section we will show that it fulfills information monotonicity.
It is interesting to note that the two Euclidean distances (Hellinger and Aitchison) are each related to different isometries of the simplex. In the case of Hellinger distance, compositions are isometrically mapped into the positive orthant of the sphere⁶⁶6But note that this mapping does not obtain the spherical representative of the composition in the sense of the definition of an equivalence class.. This isometry also holds for the Fisher metric itself Nihat (which makes it possible to view the Fisher metric as a Euclidean metric on the $\mathbb{R}^{D}$ where the sphere is embedded). In the case of Aitchison distance, the isometry in question is of central interest in CoDA. It is the clr transformation, i.e., the map between $\mathcal{S}^{D}$ and $\mathcal{T}^{D}$. Here, however, there is no corresponding isometry of the Fisher metric. Although a Euclidean metric may appear convenient, it does not have the same desirable properties as the Fisher metric. In fact, a central result in information geometry states that the Fisher metric is the only metric Chentsov2 that stays invariant under coarse graining of information.

Let us denote by $\mathcal{A}$ a subset of $\mathcal{D}:=\{1,\dots,D\}$, and let $\boldsymbol{x}_{\mathcal{A}}$ denote the corresponding subcomposition, i.e., the vector where parts with indices not belonging to $\mathcal{A}$ have been removed. Let $H(\boldsymbol{x})$ denote the Shannon entropy of composition $\boldsymbol{x}$, i.e., the potential $-\phi(\boldsymbol{\eta}_{\boldsymbol{x}})$. Consider its decomposition

where $s(\boldsymbol{x}_{\mathcal{A}})=\sum_{i\in\mathcal{A}}x_{i}$. We see that this is a convex combination of the entropies of the two subcompositions plus a binary entropy, where all terms involve the amalgamation $s(\boldsymbol{x}_{\mathcal{A}})$. Probabilistically speaking, this particular amalgamation corresponds to the probability that an event occurs for any of the events $\mathcal{A}$ that are left out to obtain the subcomposition $\boldsymbol{x}_{\mathcal{D}\backslash\mathcal{A}}$. Generally, amalgamation is nothing else but a coarse graining of the events and their probabilities. The corresponding coarse graining of information is described by this alternative decomposition of Shannon entropy:

As the second summand is greater equal zero, this also shows that information cannot grow after coarse graining.

These considerations lead us to an important result that concerns the divergence associated with the potential $\phi(\boldsymbol{\eta}_{\boldsymbol{x}})$, i.e., the relative entropy. This result is the information monotonicity under coarse graining, where the notion of monotonicity is somewhat related to the notion of subcompositional dominance. The latter refers to the property that a measure of distance does not increase when evaluating it on a subset of parts only. This is often seen as a desirable property of distances in CoDA (and not fulfilled by distances like Hellinger and Batthacharyya, see clustering for a discussion of distance measures with respect to compositions). A similar—but perhaps more natural—requirement that has not received attention yet in the CoDA community is the one that a distance between compositions should not increase when comparing it with the one after amalgamating over a subset of parts.⁷⁷7Subcompositional coherence, i.e., the fundamental requirement that quantities remain identical on a renormalized subcomposition, is not an issue for amalgamation: after amalgamation there is no need for renormalization. As we have seen in the previous subsection, we cannot gain information when amalgamating parts, so we should lose resolution when comparing the amalgamated compositions. This is also related to the notion of sufficient statistic, see Amari . The information monotonicity property of relative entropy can be expressed as

This result can be shown for the more general case of $f$-divergences and continuous distributions using Jensen’s inequality AmariNagaoka .
Note that in balances , when discussing amalgamations of parts, the notion of “monotonicity” is used differently. There, the authors argue against amalgamations referring to the observation that Aitchison distances between amalgamated compositions and the amalgamated center of the simplex show a non-monotonic behaviour along an ilr-coordinate axis defined before amalgamation. We will show below that information monotonicity does hold for Aitchison distance. We see the lack of distance monotonicity as discussed in balances rather as an argument against the use of a Euclidean coordinate system here.

A symmetrized version of relative entropy has recently been used in the context of data-driven amalgamation amalgams , where it was shown to be better preserved between samples than Aitchison distance. While the information-theoretic meaning and mathematical properties reflected in the decompositions shown in section 4.1 make Shannon entropy an ideal measure of information, alternative indices that can sometimes be evaluated more easily on real-world data (e.g., making use of sums of squares) have also been considered. In our context, it is interesting that $A(\boldsymbol{z}_{\boldsymbol{x}})$, the potential of Eq. (42), has been proposed as an alternative measure of information within an attempt to reformulate information theory from a CoDA point-of-view E&PGinfo . More recently, it has also been proposed as an inequality index (when divided by the number of parts $D$) sampleSpace . Here, the following decomposition was shown:

where we denoted the set size of $\mathcal{A}$ by $a$. If we now want to decompose with respect to a composition that was partly amalgamated, we find a corresponding relationship that is more complicated⁸⁸8It is interesting to note that the two interaction terms have the form of squares of the balance and pivot coordinates mentioned in section 2.:

Clearly, if we replace the amalgamation $s(\boldsymbol{x}_{\mathcal{A}})$ by the geometric mean $g(\boldsymbol{x}_{\mathcal{A}})$, we get a simpler equality:

Aggregating by geometric means or by amalgamations has been a subject of debate in the CoDA community sampleSpace ; amalgGreen . As we can see, measures like the Aitchison norm lend themselves much better to taking geometric means rather than amalgamations. There is, however, no straight-forward probabilistic interpretation of geometric means⁹⁹9The product over parts specifies the probability that all events in the subset occur, but this is then re-scaled by the exponent to the probability of a single event., and the more elegant formal expressions that result often suffer from reduced interpretability.
To the best of our knowledge, the information monotonicity property in its general form has not been considered yet for Aitchison distance. We here exploit the various decompositions stated above for proving it. Results are summarized in the following propositions. All proofs can be found in the Appendix.

From this decomposition, we get the following monotonicity result:

As we can see, for geometric-mean summaries, the sum of the interaction terms (i.e., of the terms not involving norms) will remain greater equal zero. This is no longer true for amalgamation of parts, and it is less straightforward to show the corresponding inequality: