Rényi Cross-Entropy Measures for Common Distributions
and Processes with Memory

The Rényi entropy [2] of order $\alpha$ of a probability mass function $p$ with finite support $\mathbb{S}$ is defined as

$$H_{\alpha}(p)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}}p(x)^{\alpha}$$

for $\alpha>0,\alpha\neq 1$. The Rényi entropy generalizes the Shannon entropy,

$$H(p)=-\sum_{x\in\mathbb{S}}p(x)\ln p(x),$$

in the sense that as $\alpha\to 1$, $H_{\alpha}(p)\to H(p)$. Several other Rényi-type information measures have been put forward, each obeying the condition that their limit as $\alpha$ goes to one reduces to a Shannon-type information measure. This includes the Rényi divergence (of order $\alpha$) between two discrete distributions $p$ and $q$ with common finite support $\mathbb{S}$, given by

$$D_{\alpha}(p||q)=\frac{1}{\alpha-1}\ln\sum_{x\in\mathbb{S}}p(x)^{\alpha}q(x)^{1-\alpha}$$

which reduces to the familiar Kullback-Leibler divergence,

$$D_{\text{\tiny KL}}(p||q)=\sum_{x\in\mathbb{S}}p(x)\ln\frac{p(x)}{q(x)},$$

as $\alpha\to 1$. Note that in some cases [3], there may exist multiple Rényi-type generalisations for the same information measure (particularly for the mutual information).

Many of these definitions admit natural counterparts in the case when the involved distributions have a probability density function (pdf). This gives rise to information measures such as the Rényi differential entropy for pdf $p$ with support $\mathbb{S}$,

$$h_{\alpha}(p)=\frac{1}{1-\alpha}\ln\int_{\mathbb{S}}p(x)^{\alpha}\,dx,$$

and the Rényi (differential) divergence between pdfs $p$ and $q$ with common support $\mathbb{S}$,

$$D_{\alpha}(p\|q)=\frac{1}{\alpha-1}\ln\int_{\mathbb{S}}p(x)^{\alpha}q(x)^{1-\alpha}\,dx.$$

The Rényi cross-entropy between distributions $p$ and $q$ is an analogous generalization of the Shannon cross-entropy

$$H(p;q)=-\sum_{x\in\mathbb{S}}p(x)\ln q(x).$$

Two definitions for this measure have been recently suggested. In light of the fact that Shannon’s cross-entropy satisfies $H(p;q)=D(p\|q)+H(p)$, a natural definition of the Rényi cross-entropy is:

$$\tilde{H}_{\alpha}(p;q)\coloneqq D_{\alpha}(p||q)+H_{\alpha}(p).$$

(1)

This definition was indeed proposed in [4] in the continuous case, with the differential cross-entropy measure given by

$$\tilde{h}_{\alpha}(p;q)\coloneqq D_{\alpha}(p||q)+h_{\alpha}(p).$$

(2)

In contrast, prior to [4], the authors of [5] introduced the Rényi cross-entropy in their study of shifted Rényi measures expressed as the logarithm of weighted generalized power means. Specifically, upon simplifying Definition 6 in [5], their expression for the Rényi cross-entropy between distributions $p$ and $q$ is given by

$$H_{\alpha}\left(p;q\right)\coloneqq\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}}p\left(x\right)q\left(x\right)^{\alpha-1}.$$

(3)

For the continuous case, (3) can be readily converted to yield the Rényi differential cross-entropy between pdfs $p$ and $q$:

$$h_{\alpha}(p;q)\coloneqq\frac{1}{1-\alpha}\ln\int_{\mathbb{S}}p(x)q(x)^{\alpha-1}\,dx.$$

(4)

Note that both (1) and (3) reduce to the Shannon cross-entropy $H(p;q)$ as $\alpha\to 1$ [6]. A similar result holds for (2) and (4), where the Shannon differential cross-entropy, $h(p;q)=-\int_{\mathbb{S}}p(x)\ln q(x)\,dx$, is obtained. Also, the Rényi (differential) entropy is recovered in all equations when $p=q$ (almost everywhere). These properties alone make these definitions viable extensions of the Shannon (differential) cross-entropy.

Finding closed-form expressions for the cross-entropy measure in (2) for continuous distributions is direct, since the Rényi divergence and the Rényi differential entropy were already calculated for numerous distributions in [7] and [8], respectively. However, deriving the measure in (4) is more involved. We hereafter refer to the measures $\tilde{H}_{\alpha}(p;q)$ in (1) and $\tilde{h}_{\alpha}(p;q)$ (2), as the Natural Rényi cross-entropy and the Natural Rényi differential cross-entropy, respectively; while we plainly call the measures in $H_{\alpha}(p;q)$ in (3) and $h_{\alpha}(p;q)$ (4) as the Rényi cross-entropy and the Rényi differential cross-entropy, respectively.

In a recent conference paper [1], we showed how to calculate the Rényi differential cross-entropy $h_{\alpha}(p;q)$ between distributions of the same type from the exponential family. Building upon the results shown there, the purpose of this paper is to derive in closed form the expression of $h_{\alpha}(p;q)$ for thirteen commonly used univariate distributions from the exponential family, as well as for multivariate Gaussians, and tabulate the results for ease of reference. We also analytically derive the Natural Rényi differential cross-entropy $\tilde{h}_{\alpha}(p;q)$ for the same set of distributions. Finally, we present tables summarising the Rényi and Natural Rényi (differential) cross-entropy rate measures, along with their Shannon counterparts, for two important classes of sources with memory, namely stationary Gaussian sources and finite-state time-invariant Markov sources.

Motivation for determining formulae for the Rényi cross-entropy originates from the use of the Shannon differential cross-entropy as a loss function for the design of deep learning generative adversarial networks (GANs) in [9]. The parameter $\alpha$, ubiquitous to all Rényi information measures, allows one to fine-tune the loss function to improve the quality of the GAN-generated output. This can be seen in [10, 6] and [4], which used the Rényi differential cross-entropy, and the Natural Rényi differential cross-entropy measures, respectively, to generalize the original GAN loss function (which is recovered as $\alpha\to 1$), resulting in both improved GAN system stability and performance for multiple image datasets. It is also shown in [10, 6] that the introduced Rényi-centric generalized loss function preserves the equilibrium point satisfied by the original GAN via the so-called Jensen-Rényi divergence [11], a natural extension of the Jensen-Shannon divergence [12] upon which the equilibrium result of [9] is established. Other GAN systems that utilize different generalized loss functions were recently developed and analyzed in [13] and [14, 15] (see also the references therein for prior work).

The rest of this paper is organised as follows. In Section II, the formulae for the Rényi differential cross-entropy and Natural Rényi differential cross-entropy for distributions from the exponential family are given. In Section III, these calculations are systematically carried for fourteen pairs of distributions of the same type within the exponential family and the results are presented in two tables. The Rényi and Natural Rényi differential cross-entropy rates are presented in Section IV for stationary Gaussian sources; furthermore, the Rényi and Natural Rényi cross-entropy rates are provided in Section V for finite-state time-invariant Markov sources. Finally, the paper is concluded in Section VI.

An exponential family is a class of probability distributions over a support $\mathbb{S}\subseteq\mathbb{R}^{n}$ defined by a parameter space $\Theta\subseteq\mathbb{R}^{m}$ and functions $b:\mathbb{S}\mapsto\mathbb{R}$, $c:\Theta\mapsto\mathbb{R}$, $T:\mathbb{S}\mapsto\mathbb{R}^{m}$, and $\eta:\Theta\mapsto\mathbb{R}^{m}$ such that the pdf in this family have the form

$$f(x)=c(\theta)b(x)\exp\left(\langle\eta(\theta),T(x)\rangle\right),\qquad x\in\mathbb{S}$$

(5)

where $\langle\cdot,\cdot\rangle$ denotes the standard inner product in $\mathbb{R}^{m}$. Alternatively, by using the (natural) parameter $\eta=\eta(\theta)$, the pdf can also be written

$$\displaystyle{f(x)=b(x)\exp\left(\langle\eta,T(x)\rangle+A(\eta)\right)},$$

(6)

where $A(\eta):\eta(\Theta)\mapsto\mathbb{R}$ with $A(\eta)=\ln c(\theta)$. Examples of important pdfs we consider from the exponential family are included in Appendix A.

In [1] the cross-entropy between pdfs $f_{1}$ and $f_{2}$ of the same type from the exponential family was proven to be

$$h_{\alpha}\left(f_{1};f_{2}\right)=\frac{A\left(\eta_{1}\right)-A\left(\eta_{h}\right)+\ln E_{h}}{1-\alpha}-A\left(\eta_{2}\right),$$

(7)

where

$$E_{h}=\mathbb{E}_{f_{h}}\left[b(X)^{\alpha-1}\right]=\int b(x)^{\alpha-1}f_{h}(x)\,dx.$$

Here $f_{h}$ refers to a distribution of the same type as $f_{1}$ and $f_{2}$ within the exponential family with natural parameter

$$\eta_{h}\coloneqq\eta_{1}+(\alpha-1)\eta_{2}.$$

It can also be shown that the Natural Rényi differential cross-entropy between $f_{1}$ and $f_{2}$ is given by

$$\tilde{h}_{\alpha}\left(f_{1};f_{2}\right)=\frac{A\left(\eta_{\alpha}\right)-A\left(\alpha\eta_{1}\right)+\ln E_{\alpha}}{1-\alpha}-A\left(\eta_{2}\right),$$

(8)

where

$$\eta_{\alpha}=\alpha\eta_{1}+(1-\alpha)\eta_{2},$$

and

$$E_{\alpha}=\mathbb{E}_{f_{\alpha 1}}\left[b(X)^{\alpha-1}\right]=\int b(x)^{\alpha-1}f_{\alpha 1}(x)\,dx$$

where $f_{\alpha 1}$ refers to a distribution of the same type as $f_{1}$ and $f_{2}$ within the exponential family with natural parameter $\alpha\eta_{1}$.

Tables LABEL:RenDivTable and LABEL:RenNatTable list Rényi and Natural differential cross-entropy expressions, respectively, between common distributions of the same type from the exponential family (which we describe in Appendix A for convenience). The closed-form expressions were derived using (7) and (8), respectively. In the tables, the subscript of $i$ is used to denote that a parameter belongs to pdf $f_{i}$, $i=1,2$.

In [1] the Rényi differential cross-entropy rate for stationary zero-mean Gaussian processes was derived. This, alongside with the Shannon and Natural Rényi differential cross-entropy rates, are summarised in Table LABEL:RencrosenTable. Here, $f(\lambda)$ is the spectral density of the first zero-mean Gaussian process, $g(\lambda)$ is the spectral density of the second,

$$h(\lambda)=f(\lambda)+(\alpha-1)g(\lambda),$$

and

$$j(\lambda)=\alpha f(\lambda)+(1-\alpha)g(\lambda).$$

In [1], the Rényi cross-entropy rate between finite-state time-invariant Markov sources was established, using as in [16] tools from the theory of non-negative matrices and Perron-Frobenius theory (e.g., cf. [17, 18]). This, alongside the Shannon and Natural Rényi differential cross-entropy rates, are derived and summarised in Table LABEL:MarkovcrosenTable. Here, $P$ and $Q$ are the $m\times m$ (stochastic) transition matrices associated with the first and second Markov sources, respectively, where both sources have common alphabet of size $m$. To allow any value of the Rényi parameter $\alpha$ in $(0,1)\cup(1,\infty)$, we assume that the transition matrix $Q$ of the second Markov chain has positive entries ($Q>0$); however, the transition matrix $P$ of the first Markov chain is taken to be an arbitrary stochastic matrix. For simplicity, we assume that the initial distribution vectors, $p$ and $q$, of both Markov chains also have positive entries ($p>0$ and $q>0$).¹¹1This condition can be relaxed via the approach used to prove [16, Theorem 1]. Moreover, $\pi_{p}^{T}$ denotes the stationary probability row vector associated with the first Markov chain and $\mathbf{1}$ is an $m$-dimensional column vector where each element equals one. Furthermore, $\odot$ denotes element-wise multiplication (i.e., the Hadamard product operation) and $\dot{\ln}$ is the element-wise natural logarithm.

Finally, the definition of $\lambda(R):\mathbb{R}^{m\times m}\mapsto\mathbb{R}$ for a matrix $R$ is more involved. If $R$ is irreducible, $\lambda(R)$ is its largest positive eigenvalue. Otherwise, rewriting $R$ in its canonical form as detailed in [16, Proposition 1], we have that $\lambda(R)=\max(\lambda^{*},\lambda_{*})$, where $\lambda^{*}$ is the maximum of all largest positive eigenvalues of (irreducible) sub-matrices of $R$ corresponding to self-communicating classes, and $\lambda_{*}$ is the maximum of all largest positive eigenvalues of sub-matrices of $R$ corresponding to classes reachable from an inessential class.

We have derived closed-form formulae for the Rényi and Natural Rényi differential cross-entropies of commonly used distributions from the exponential family. This is of potential use to further studies in information theory and machine learning, particularly problems where deep neural networks, trained according to a Shannon cross-entropy loss function, can be improved via generalized Rényi-type loss functions in virtue of the extra degree degree of freedom provided by the Rényi ($\alpha$) parameter. In addition, we have provided formulae for the Rényi and Natural Rényi differential cross-entropy rates for stationary zero-mean Gaussian processes and expressions for the cross-entropy rates for Markov sources. Further work includes expanding the present collection by considering distributions such as Levy or Weibull and investigating cross-entropy measures based on the $f$-divergence [19, 20, 21], starting with Arimoto’s divergence [22].

This work was supported in part by Natural Sciences and Engineering Research Council (NSERC) of Canada.

• •

  ${\displaystyle\mathrm{B}(a,b)=\int_{0}^{1}t^{a-1}(1-t)^{b-1}\,dt}$ is the Beta function.

• •

  ${\displaystyle\Gamma(z)=\int_{0}^{\infty}x^{z-1}e^{-x}\,dx}$ is the Gamma function.