Tight Lower Bounds for $\alpha$ -Divergences Under Moment Constraints and Relations Between Different $\alpha$

Tomohiro Nishiyama
Email: [email protected]

Abstract

The $\alpha$ -divergences include the well-known Kullback-Leibler divergence, Hellinger distance and $\chi^{2}$ -divergence. In this paper, we derive differential and integral relations between the $\alpha$ -divergences that are generalizations of the relation between the Kullback-Leibler divergence and the $\chi^{2}$ -divergence. We also show tight lower bounds for the $\alpha$ -divergences under given means and variances. In particular, we show a necessary and sufficient condition such that the binary divergences, which are divergences between probability measures on the same $2$ -point set, always attain lower bounds. Kullback-Leibler divergence, Hellinger distance, and $\chi^{2}$ -divergence satisfy this condition.

Keywords: Alpha-divergence, Kullback-Leibler divergence, Hellinger distance, Chi-squared divergence, Relative entropy, Renyi divergence.

I Introduction

The Kullback–Leibler divergence [12] (also known as the relative entropy) and the Hellinger distance [10] are divergence measures which play a key role in information theory, statistics, machine learning, physics, signal processing, and other theoretical and applied branches of mathematics. They both belong to an important class of divergence measures, defined by means of convex functions $f$ , and named $f$ -divergences [5, 6, 7]. The most notable class of $f$ -divergence is the $\alpha$ -divergence [1, 4]. By choosing different $\alpha$ , we get a large number of well-known divergences as special cases, including Kullback-Leibler divergence, Hellinger distance, and $\chi^{2}$ -divergence [17].

In this paper, we study relations between the $\alpha$ -divergences for different $\alpha$ , and derive the tight lower bounds for the $\alpha$ -divergences under given means and variances. The relation between the Kullback-Leibler divergence and the $\chi^{2}$ -divergence was shown in [14, 2, 16], and we generalize this relation for general $\alpha$ and $\alpha+1$ . Regarding the lower bounds under given means and variances, there are some works for the $\chi^{2}$ -divergence and the Hellinger distance [3, 8, 11]. Recently, for the Kullback-Leibler divergence and the Hellinger distance, we showed that the tight lower bounds are all attained by their binary divergences that are divergences between probability measures on the same $2$ -point set [16, 15]. Our motivation is to study necessary and sufficient conditions for $\alpha$ such that the binary $\alpha$ -divergences always attain lower bounds. Furthermore, we show tight lower bounds under given means and variances for the Rényi divergences [18], which are closely related to the $\alpha$ -divergences.

In this work, Section 2 presents notation and definitions, Section 3 refers to the main results, Section 4 shows the proofs of the main results. Finally, Section 5 concludes this paper, and lemmas that are necessary for the proofs of the main results are proved in Appendices.

II Preliminaries

This section provides definitions of divergence measures which are used in this paper.

Definition 1.

[13, p. 4398] Let $P$ and $Q$ be probability measures, let $\mu$ be a dominating measure of $P$ and $Q$ (i.e., $P,Q\ll\mu$ ), and let $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu}$ and $q:=\frac{\mathrm{d}Q}{\mathrm{d}\mu}$ be the densities of $P$ and $Q$ with respect to $\mu$ . The $f$ -divergence from $P$ to $Q$ is given by

\displaystyle D_{f}(P\|Q):=\int q\,f\Bigl{(}\frac{p}{q}\Bigr{)}\,\mathrm{d}\mu,

(1)

where

	$\displaystyle f(0):=\underset{t\to 0^{+}}{\lim}\,f(t),\quad 0f\biggl{(}\frac{0}{0}\biggr{)}:=0,$
	$\displaystyle 0f\biggl{(}\frac{a}{0}\biggr{)}:=\lim_{t\to 0^{+}}\,tf\biggl{(}\frac{a}{t}\biggr{)}=a\lim_{u\to\infty}\frac{f(u)}{u},\quad a>0.$

It should be noted that the right side of (1) is invariant in the dominating measure $\mu$ .

Definition 2.

[4] The basic asymmetric alpha-divergence is the $f$ -divergence with

\displaystyle f(t):=\begin{dcases}\frac{t^{\alpha}-t}{\alpha(\alpha-1)},&\alpha\neq 0,1,\\ -\log t,&\alpha=0,\\ t\log t,&\alpha=1.\end{dcases}

for $t>0$ ,

\displaystyle D_{A}^{(\alpha)}(P\|Q):=\begin{dcases}\frac{1}{\alpha(\alpha-1)}\Bigl{(}\int p^{\alpha}q^{1-\alpha}\mathrm{d}\mu-1\Bigr{)},&\alpha\neq 0,1,\\ \int q\log\frac{q}{p}\mathrm{d}\mu=:D(Q\|P),&\alpha=0,\\ \int p\log\frac{p}{q}\mathrm{d}\mu=:D(P\|Q),&\alpha=1,\end{dcases}

(2)

where $D(P\|Q)$ denotes the Kullback-Leibler divergence (relative entropy).

In the special case for $\alpha=2,\;0.5,\;-1$ , we obtain from (2) the well known Pearson Chi-square, Hellinger and Neyman Chi-square distances, given respectively by

	$\displaystyle D_{A}^{(2)}(P\\|Q)=\frac{1}{2}\chi^{2}_{P}(P\\|Q)=\frac{1}{2}\int\frac{(p-q)^{2}}{q}\mathrm{d}\mu,$
	$\displaystyle D_{A}^{(1/2)}(P\\|Q)=4H^{2}(P,Q)=2\int(\sqrt{p}-\sqrt{q})^{2}\mathrm{d}\mu,$
	$\displaystyle D_{A}^{(-1)}(P\\|Q)=\frac{1}{2}\chi^{2}_{N}(P\\|Q)=\frac{1}{2}\int\frac{(p-q)^{2}}{p}\mathrm{d}\mu.$

The $\alpha$ -divergences have duality as follows.
Duality:

\displaystyle D_{A}^{(\alpha)}(P\|Q)=D_{A}^{(1-\alpha)}(Q\|P).

(3)

The Rényi divergences [18] are closely related to $\alpha$ -divergences.

Definition 3.

The Rényi divergence for the simple orders $\alpha\in(0,1)\cup(1,\infty)$ is defined as

\displaystyle D_{R}^{(\alpha)}(P\|Q):=\frac{1}{\alpha-1}\log\int p^{\alpha}q^{1-\alpha}\mathrm{d}\mu=\frac{1}{\alpha-1}\log\Bigl{(}1+\alpha(\alpha-1)D_{A}^{(\alpha)}(P\|Q)\Bigr{)}.

For the extended orders, the Rényi divergence is defined as

\displaystyle D_{R}^{(\alpha)}(P\|Q):=\begin{dcases}-\log Q(p>0),&\alpha=0,\\ D(P\|Q)=D_{A}^{(1)}(P\|Q),&\alpha=1,\\ \log\operatorname*{ess\,sup}\frac{p(Z)}{q(Z)},&\alpha=\infty,\end{dcases}

where $Z\sim\mu$ .

Definition 4.

Let us define a set of pairs of probability measures $(P,Q)$ defined on $n$ -point set $\{u_{1},u_{2},\cdots,u_{n}\}$ by $\mathcal{P}_{n}$ , where $\{u_{i}\}_{1\leq i\leq n}$ are arbitrary real numbers.

If $m<n$ , $\mathcal{P}_{m}$ is a subset of $\mathcal{P}_{n}$ .

Definition 5.

Let $P$ and $Q$ be probability measures defined on a measurable space $(\mathbb{R},\mathscr{B})$ , where $\mathbb{R}$ is the real line and $\mathscr{B}$ is the Borel $\sigma$ -algebra of subsets of $\mathbb{R}$ . Let $\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ be a set of pairs of probability measures $(P,Q)$ under given means and variances, i.e.,

\displaystyle\mathbb{E}[X]=:m_{P},\;\mathbb{E}[Y]=:m_{Q},\quad\mathrm{Var}(X)=:\sigma_{P}^{2},\;\mathrm{Var}(Y)=:\sigma_{Q}^{2},

(4)

where $X\sim P$ and $Y\sim Q$ .

Definition 6.

The binary $\alpha$ -divergence is defined for $(R,S)\in\mathcal{P}_{2}$ .

\displaystyle d_{A}^{(\alpha)}(r\|s):=\begin{dcases}\frac{1}{\alpha(\alpha-1)}\Bigl{(}r^{\alpha}s^{1-\alpha}+(1-r)^{\alpha}(1-s)^{1-\alpha}-1\Bigr{)},&\alpha\neq 0,1,\\ s\log\frac{s}{r}+(1-s)\log\frac{1-s}{1-r},&\alpha=0,\\ r\log\frac{r}{s}+(1-r)\log\frac{1-r}{1-s},&\alpha=1.\end{dcases}

(5)

where $R(u_{1})=r$ and $S(u_{1})=s$ .

Definition 7.

The binary Rényi divergence for orders $\alpha\in[0,\infty]$ is defined for $(R,S)\in\mathcal{P}_{2}$ .

\displaystyle d_{R}^{(\alpha)}(r\|s):=\begin{dcases}\frac{1}{\alpha-1}\log\Bigl{(}r^{\alpha}s^{1-\alpha}+(1-r)^{\alpha}(1-s)^{1-\alpha}\Bigr{)},&\alpha\neq 0,1,\\ -\log\Bigl{(}s1\{r>0\}+(1-s)1\{1-r>0\}\Bigr{)},&\alpha=0,\\ r\log\frac{r}{s}+(1-r)\log\frac{1-r}{1-s},&\alpha=1,\\ \log\max\Bigl{(}\frac{r}{s},\frac{1-r}{1-s}\Bigr{)},&\alpha=\infty.\end{dcases}

where $1\{\mbox{relation}\}$ denotes the indicator function.

III Main results

Theorem 1.

Let $Q_{t}:=(Q-P)t+P$ and $P_{t}:=(P-Q)t+Q=Q_{1-t}$ for $t\in[0,1]$ . Then, for all $t\in(0,1]$ ,

	$\displaystyle D_{A}^{(\alpha+1)}(P\\|Q_{t})=\frac{t^{2-\alpha}}{(\alpha+1)}\frac{\mathrm{d}}{\mathrm{d}t}\Bigl{(}t^{\alpha-1}D_{A}^{(\alpha)}(P\\|Q_{t})\Bigr{)},\quad\alpha\neq-1,$		(6)
	$\displaystyle D_{A}^{(\alpha)}(P_{t}\\|Q)=\frac{t^{2+\alpha}}{(-\alpha+1)}\frac{\mathrm{d}}{\mathrm{d}t}\Bigl{(}t^{-\alpha-1}D_{A}^{(\alpha+1)}(P_{t}\\|Q)\Bigr{)},\quad\alpha\neq 1.$		(7)

Proof.

See Section IV. ∎

Corollary 1.

For all $t\in(0,1]$ ,

	$\displaystyle D_{A}^{(\alpha)}(P\\|Q_{t})=(\alpha+1)t^{1-\alpha}\int_{0}^{t}s^{\alpha-2}D_{A}^{(\alpha+1)}(P\\|Q_{s})\mathrm{d}s,\quad\alpha>-1,$		(8)
	$\displaystyle D_{A}^{(\alpha+1)}(P_{t}\\|Q)=(-\alpha+1)t^{1+\alpha}\int_{0}^{t}s^{-\alpha-2}D_{A}^{(\alpha)}(P_{s}\\|Q)\mathrm{d}s,\quad\alpha<1.$		(9)

The relation (8) for $\alpha=1$ yields the relation between the Kullback-Leibler divergence and the $\chi^{2}$ -divergence.

\displaystyle D(P\|Q_{t})=\int_{0}^{t}s^{-1}\chi_{P}^{2}(P\|Q_{s})\mathrm{d}s.

(10)

Proof.

By the Taylor expansion for a differentiable function $f(\cdot)$ such that $f(1)=0$ , for efficiently small $s$ , we obtain

\displaystyle\int q_{s}f\Bigl{(}\frac{p}{q_{s}}\Bigr{)}\mathrm{d}\mu=\int q_{s}\Bigl{(}f^{\prime}(1)\Bigl{(}\frac{p}{q_{s}}-1\Bigr{)}+\frac{f^{\prime\prime}(1)}{2}{\Bigl{(}\frac{p}{q_{s}}-1\Bigr{)}}^{2}+O(s^{3})\Bigr{)}\mathrm{d}\mu=\frac{f^{\prime\prime}(1)s^{2}}{2}\chi^{2}_{P}(P\|Q)+O(s^{3}).

(11)

Since the $\alpha$ -divergences belong to $f$ -divergences, it follows that

\displaystyle t^{\alpha-1}D_{A}^{(\alpha)}(P\|Q_{t})=O(t^{\alpha+1}).

(12)

Replacing $t$ by $s$ , multiplying $(\alpha+1)s^{\alpha-2}$ and integrating both sides of (6), we obtain (8). The condition $\alpha>-1$ is due to (12). The equality (9) follows in a similar way. ∎

Theorem 2.

Let $(P,Q)\in\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ .

(a)

If $m_{P}\neq m_{Q}$ , the binary $\alpha$ -divergence always attains a lower bound under given means and variances if and only if $\alpha\in[-1,2]$ .

\displaystyle D_{A}^{(\alpha)}(P\|Q)\geq d_{A}^{(\alpha)}(r\|s),

(13)

where

		$\displaystyle r:=\frac{1}{2}+\frac{\sigma_{Q}^{2}-\sigma_{P}^{2}+a^{2}}{4av}\in[0,1],$		(14)
		$\displaystyle s:=\frac{1}{2}+\frac{\sigma_{Q}^{2}-\sigma_{P}^{2}-a^{2}}{4av}\in[0,1],$		(15)
		$\displaystyle a:=m_{P}-m_{Q},$		(16)
		$\displaystyle v:=\frac{1}{2\|a\|}\sqrt{(\sigma_{Q}^{2}-\sigma_{P}^{2})^{2}+2a^{2}(\sigma_{P}^{2}+\sigma_{Q}^{2})+a^{4}}.$		(17)

(b)

The lower bound in the right side of (13) is attained for $(R,S)\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ defined on $\{u_{1},u_{2}\}$ , and

$\displaystyle R(u_{1})=r,\quad S(u_{1})=s,$ (18)

with $r$ and $s$ in (14) and (15), respectively, and

$\displaystyle u_{1}:=m_{P}+\sqrt{\frac{(1-r)\sigma_{P}^{2}}{r}},\quad u_{2}:=m_{P}-\sqrt{\frac{r\sigma_{P}^{2}}{1-r}}.$ (19)
(c)

If $m_{P}=m_{Q}$ , then,

$\displaystyle\inf_{(P,Q)\in\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]}D_{A}^{(\alpha)}(P\|Q)=0.$ (20)

Proof.

See Section IV. ∎

For the Hellinger distance and the $\chi^{2}$ -divergence, their binary divergences are simplified as

	$\displaystyle d_{A}^{(2)}(r\\|s)$	$\displaystyle=\frac{a^{2}}{2\sigma_{Q}^{2}},$
	$\displaystyle d_{A}^{(1/2)}(r\\|s)$	$\displaystyle=4\Bigl{(}1-\sqrt{\frac{(\sigma_{P}+\sigma_{Q})^{2}}{a^{2}+(\sigma_{P}+\sigma_{Q})^{2}}}\Bigr{)}.$

See Subsection IV-B and [15][Lemma 2], respectively.

Corollary 2.

If $m_{P}\neq m_{Q}$ and $\alpha\in[0,2]$ , the binary Rényi divergence always attains a lower bound under given means and variances.

Proof.

For $\alpha\in(0,2]$ , the result follows by Definition 3 and Theorem 2. Since the binary Rényi divergence is equal to $0$ for $\alpha=0$ and $\sigma_{P}>0$ , we obtain the result. For $\alpha=\sigma_{P}=0$ , we have $P(u)=1$ at $u=m_{P}$ . Letting $q=Q(u)$ , we obtain

	$\displaystyle\sum_{u_{i}\neq m_{P}}Q(u_{i})u_{i}=m_{Q}-qm_{P},$		(21)
	$\displaystyle\sum_{u_{i}\neq m_{P}}Q(u_{i})u_{i}^{2}=\sigma_{Q}^{2}+m_{Q}^{2}-qm_{P}^{2}.$		(22)

By the Cauchy-Schwarz inequality, it follows that

\displaystyle\Bigl{(}\sum_{u_{i}\neq m_{P}}Q(u_{i})u_{i}\Bigr{)}^{2}=\Bigl{(}\sum_{u_{i}\neq m_{P}}\sqrt{Q(u_{i})}(\sqrt{Q(u_{i})}u_{i})\Bigr{)}^{2}\leq\Bigl{(}\sum_{u_{i}\neq m_{P}}Q(u_{i})\Bigr{)}\Bigl{(}\sum_{u_{i}\neq m_{P}}Q(u_{i})u_{i}^{2}\Bigr{)}=(1-q)(\sigma_{Q}^{2}+m_{Q}^{2}-qm_{P}^{2}).

(23)

By combining this inequality with (21), we obtain

\displaystyle q\leq\frac{\sigma_{Q}^{2}}{\sigma_{Q}^{2}+a^{2}}.

(24)

From (14) and (15), we have $r=1$ and $s=\frac{\sigma_{Q}^{2}}{\sigma_{Q}^{2}+a^{2}}$ for $a>0$ . By combining (24) with Definition (3), the result follows. The case $a<0$ can be justified in a similar way. ∎

IV Proofs of main results

IV-A Proof of Theorem 1

Proof.

Let $q_{t}:=(q-p)t+p$ for all $t\in(0,1]$ . For $\alpha\neq 0,\;\pm 1$ , we obtain

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\Bigl{(}\Bigl{(}\int p^{\alpha}q_{t}^{1-\alpha}\mathrm{d}\mu-1\Bigr{)}t^{\alpha-1}\Bigr{)}$	$\displaystyle=(\alpha-1)t^{\alpha-2}\Bigl{(}\int\Bigl{(}-p^{\alpha}q_{t}^{-\alpha}(q-p)t+p^{\alpha}q_{t}^{1-\alpha}\Bigr{)}\mathrm{d}\mu-1\Bigr{)}$	(25)
	$\displaystyle=(\alpha-1)t^{\alpha-2}\Bigl{(}\int\Bigl{(}p^{\alpha}q_{t}^{-\alpha}(-(q-p)t+q_{t})\Bigr{)}\mathrm{d}\mu-1\Bigr{)}$
	$\displaystyle=(\alpha-1)t^{\alpha-2}\Bigl{(}\int p^{1+\alpha}q_{t}^{-\alpha}\mathrm{d}\mu-1\Bigr{)}=(\alpha+1)\alpha(\alpha-1)t^{\alpha-2}D_{A}^{(\alpha+1)}(P\\|Q_{t}).$

Dividing $(\alpha+1)\alpha(\alpha-1)t^{\alpha-2}$ both sides of (25), we obtain (6). For $\alpha=1$ , we obtain

\displaystyle\frac{t}{2}\frac{\mathrm{d}}{\mathrm{d}t}\int p\log\frac{p}{q_{t}}\mathrm{d}\mu=\frac{1}{2}\int\frac{p(p-q_{t})}{q_{t}}\mathrm{d}\mu=\frac{1}{2}\int\frac{(q_{t}-p)^{2}}{q_{t}}\mathrm{d}\mu=D_{A}^{(2)}(P\|Q_{t}).

(26)

For $\alpha=0$ , we obtain

\displaystyle t^{2}\frac{\mathrm{d}}{\mathrm{d}t}\int t^{-1}q_{t}\log\frac{q_{t}}{p}\mathrm{d}\mu=t^{2}\int\Bigl{(}-t^{-2}p\log\frac{q_{t}}{p}+t^{-1}(q-p)\mathrm{d}\mu\Bigr{)}=\int p\log\frac{p}{q_{t}}\mathrm{d}\mu=D_{A}^{(1)}(P\|Q_{t}).

(27)

By the duality (3), and swapping $P$ and $Q$ for (6), it follows that

\displaystyle D_{A}^{(-\alpha)}(P_{t}\|Q)=\frac{t^{2-\alpha}}{(\alpha+1)}\frac{\mathrm{d}}{\mathrm{d}t}\Bigl{(}t^{\alpha-1}D_{A}^{(1-\alpha)}(P_{t}\|Q)\Bigr{)}.

(28)

Replacing $\alpha$ by $-\alpha$ yields (7). ∎

IV-B Proof of Theorem 2

Lemma 1.

For $R>0$ , let $\mathcal{P}_{n,R}\subseteq\mathcal{P}_{n}$ be a set of pairs of probability measures such that $|u_{i}|\leq R$ for all $i=1,2,\cdots,n$ . Let $(P,Q)\in\mathcal{P}_{n,R}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ . If $\alpha\in(0,1)$ and $m_{P}\neq m_{Q}$ , the global minimum points $(P^{*},Q^{*},\bm{u}^{*})=\mathrm{argmin}_{(P,Q)\in\mathcal{P}_{n,R}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]}D_{A}^{(\alpha)}(P\|Q)$ satisfy any of the following conditions.

(1)

$(P^{*},Q^{*})\in\mathcal{P}_{2}$ , and $\max_{i}|u^{*}_{i}|<R$ .
(2)

$\max_{i}|u^{*}_{i}|=R$ .

Proof.

See Appendix A. ∎

Lemma 2.

If $\alpha\in(0,1)$ and $m_{P}\neq m_{Q}$ , the binary $\alpha$ -divergence is monotonically decreasing with respect to both $\sigma_{P}$ and $\sigma_{Q}$ .

Proof.

See Appendix B. ∎

Lemma 3.

If $m_{P}\neq m_{Q}$ , a set $\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ has one component $(R,S)$ that is given by (18), and (19).

Proof.

See [15][Lemma 1]. ∎

Lemma 4.

Let $(R,S)\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ , and let $(R^{\prime},S^{\prime})\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},0]$ . If $\alpha<-1$ , there exixts $\sigma_{Q}$ such that $d_{A}^{(\alpha)}(r\|s)>d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime})$ .

Proof.

See Appendix C. ∎

We show Theorem 2 by 4-step.

Proof for $0<\alpha<1$ .

We first prove (13) for pairs of finite discrete probability measures.
Let ${D_{A}^{(\alpha)}}^{*}:=\inf_{(P,Q)\in\mathcal{P}_{n}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]}D_{A}^{(\alpha)}(P\|Q)$ , and suppose ${D_{A}^{(\alpha)}}^{*}<d_{A}^{(\alpha)}(r\|s)$ . By Lemma 1 as $R\rightarrow\infty$ and Lemma 3, there exist sequences of vectors $\{\bm{u}_{j}\}$ , and probability measures $\{P_{j}\}$ and $\{Q_{j}\}$ , which are defined on $\{\bm{u}_{j}\}$ , such that

\displaystyle D_{A}^{(\alpha)}(P_{\infty}\|Q_{\infty})={D_{A}^{(\alpha)}}^{*},

(29)

where $Z_{\infty}$ denotes $\lim_{j\rightarrow\infty}Z_{j}$ for variables $Z=\{P,Q,u_{i}\}$ . Without any loss of generality, one can assume that $|u_{i,\infty}|<\infty$ for $1\leq i\leq I$ and $|u_{i,\infty}|=\infty$ for $I<i\leq n$ . Let $\sum_{i>I}p_{i,\infty}u^{2}_{i,\infty}=C^{2}$ and $\sum_{i>I}q_{i,\infty}u^{2}_{i,\infty}=D^{2}$ , where $p_{i,j}=P_{j}(u_{i,j})$ and $q_{i,j}=Q_{j}(u_{i,j})$ . By the variance constraints, we have $C^{2}\leq m_{P}^{2}+\sigma_{P}^{2}$ and $D^{2}\leq m_{Q}^{2}+\sigma_{Q}^{2}$ . Hence, $p_{i,\infty}=O(u_{i,\infty}^{-2})$ and $q_{i,\infty}=O(u_{i,\infty}^{-2})$ hold for $i>I$ , and due to $0<\alpha<1$ , it follows that

	$\displaystyle\sum_{i>I}p_{i,\infty}=\sum_{i>I}p_{i,\infty}u_{i,\infty}=0,$		(30)
	$\displaystyle\sum_{i>I}q_{i,\infty}=\sum_{i>I}q_{i,\infty}u_{i,\infty}=0,$		(31)
	$\displaystyle\sum_{i>I}p_{i,\infty}^{\alpha}q_{i,\infty}^{1-\alpha}=0.$		(32)

Let $P^{\prime}$ and $Q^{\prime}$ be probability measures defined on $\{u_{1,\infty},u_{2,\infty},\cdots,u_{I,\infty}\}$ , and let $P^{\prime}(u_{i,\infty})=p_{i,\infty}$ , $Q^{\prime}(u_{i,\infty})=q_{i,\infty}$ for $1\leq i\leq I$ . From (30)-(32), it follows that

	$\displaystyle(P^{\prime},Q^{\prime})$	$\displaystyle\in\mathcal{P}_{I,R^{\prime}}\cap\mathcal{P}\Bigl{[}m_{P},\sqrt{\sigma_{P}^{2}-C^{2}},m_{Q},\sqrt{\sigma_{Q}^{2}-D^{2}}\Bigr{]},$		(33)
	$\displaystyle{D_{A}^{(\alpha)}}^{*}$	$\displaystyle=D_{A}^{(\alpha)}(P^{\prime},Q^{\prime}),$		(34)

where we set $R^{\prime}>\max_{i\leq I}|u_{i,\infty}|$ . Since the variances of $P^{\prime}$ and $Q^{\prime}$ are non-negative, we have $0\leq C^{2}\leq\sigma_{P}^{2}$ , and $0\leq D^{2}\leq\sigma_{Q}^{2}$ . If ${D_{A}^{(\alpha)}}^{*}$ is not a global minimum in $\mathcal{P}_{I,R^{\prime}}\cap\mathcal{P}\Bigl{[}m_{P},\sqrt{\sigma_{P}^{2}-C^{2}},m_{Q},\sqrt{\sigma_{Q}^{2}-D^{2}}\Bigr{]}$ , there exists a sequence in $\mathcal{P}_{n}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ such that $\sum_{i>I}p_{i,\infty}u^{2}_{i,\infty}=C^{2}$ , $\sum_{i>I}q_{i,\infty}u^{2}_{i,\infty}=D^{2}$ , which gives a smaller value than ${D_{A}^{(\alpha)}}^{*}$ . It contradicts that ${D_{A}^{(\alpha)}}^{*}$ is global minimum in $\mathcal{P}_{n}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ . Hence, by Lemma 1, it follows that $(P^{\prime},Q^{\prime})\in\mathcal{P}_{2}$ , and

\displaystyle{D_{A}^{(\alpha)}}^{*}=D_{A}^{(\alpha)}(P^{\prime}\|Q^{\prime})=d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime}),

(35)

where $(r^{\prime},s^{\prime})$ with means $(m_{P},m_{Q})$ and variances $(\sigma_{P}^{2}-C^{2},\sigma_{Q}^{2}-D^{2})$ . By Lemma 2, since $d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime})$ is monotonically decreasing with respect to $\sigma_{P}$ and $\sigma_{Q}$ , we have ${D_{A}^{(\alpha)}}^{*}=d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime})\geq d_{A}^{(\alpha)}(r\|s)$ . This contradicts the assumption of ${D_{A}^{(\alpha)}}^{*}<d_{A}^{(\alpha)}(r\|s)$ , thus we obtain ${D_{A}^{(\alpha)}}^{*}=d_{A}^{(\alpha)}(r\|s)$ .

Next, we prove (13) for pairs of probability measures in $\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ . For sufficiently small $\epsilon$ , there exists $R$ such that

\displaystyle|\int_{|x|>R}px^{k}\mathrm{d}\mu(x)|<\epsilon,\quad|\int_{|x|>R}qx^{k}\mathrm{d}\mu(x)|<\epsilon,\quad\mbox{for}\hskip 4.26773ptk=0,1,2.

(36)

Since $p^{\alpha}q^{1-\alpha}\leq p+q$ , we have

\displaystyle\int_{|x|>R}p^{\alpha}q^{1-\alpha}\mathrm{d}\mu(x)<2\epsilon.

(37)

In the interval $[-R,R]$ , one can approximate probability measures by finite discrete probability measures $(P_{d},Q_{d})\in\mathcal{P}_{n,R}$ as follows.

		$\displaystyle\|\int_{\|x\|\leq R}px^{k}\mathrm{d}\mu(x)-\sum_{i}p_{i}u_{i}^{k}\|<\epsilon,\quad\mbox{for}\hskip 4.26773ptk=0,1,2,$		(38)
		$\displaystyle\|\int_{\|x\|\leq R}qx^{k}\mathrm{d}\mu(x)-\sum_{i}q_{i}u_{i}^{k}\|<\epsilon,\quad\mbox{for}\hskip 4.26773ptk=0,1,2,$		(39)
		$\displaystyle\|\frac{1}{\alpha(\alpha-1)}\int_{\|x\|\leq R}p^{\alpha}q^{1-\alpha}\mathrm{d}\mu(x)-D_{A}^{(\alpha)}(P_{d}\\|Q_{d})\|<\epsilon.$		(40)

From (37) and (40), we have $D_{A}^{(\alpha)}(P\|Q)=D_{A}^{(\alpha)}(P_{d}\|Q_{d})+O(\epsilon)$ . From (36), (38), and (39), it follows that differences of means and variances between $(P,Q)$ and $(P_{d},Q_{d})$ are $O(\epsilon)$ . By applying $D_{A}^{(\alpha)}(P_{d},Q_{d})\geq d_{A}^{(\alpha)}(r_{d},s_{d})$ , it follows that

\displaystyle D_{A}^{(\alpha)}(P\|Q)=D_{A}^{(\alpha)}(P_{d},Q_{d})+O(\epsilon)\geq d_{A}^{(\alpha)}(r_{d},\|s_{d})+O(\epsilon)=d_{A}^{(\alpha)}(r\|s)+O(\epsilon),

(41)

where $(r_{d},s_{d})\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P_{d}},\sigma_{P_{d}},m_{Q_{d}},\sigma_{Q_{d}}]$ , and we use differentiability of $d_{A}^{(\alpha)}(r\|s)$ with respect to means and variances. Since one can choose arbitrary small $\epsilon$ , we obtain $D_{A}^{(\alpha)}(P\|Q)\geq d_{A}^{(\alpha)}(r\|s)$ . ∎

Proof for $-1\leq\alpha\leq 0$ and $1\leq\alpha\leq 2$ .

Let $(P,Q)\in\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ , and $(R,S)\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ . For all $t\in[0,1]$ , probability measures $Q_{t}=(Q-P)t+P$ and $S_{t}=(S-R)t+R$ have the same means and variances. Hence, by the tight lower bounds for $0<\alpha+1<1$ , the relation (8) for $-1<\alpha<0$ , and setting $t=1$ in (8), we obtain

\displaystyle D_{A}^{(\alpha)}(P\|Q)=(\alpha+1)\int_{0}^{1}t^{\alpha-2}D_{A}^{(\alpha+1)}(P\|Q_{t})\mathrm{d}t\geq(\alpha+1)\int_{0}^{1}t^{\alpha-2}d_{A}^{(\alpha+1)}(r\|s_{t})\mathrm{d}t=d_{A}^{(\alpha)}(r\|s).

The inequality for $1<\alpha<2$ follows due to the duality (3). Next, we prove for $\alpha=-1,2$ . By the Hammersley–Chapman–Robbins bound [3], we obtain

\displaystyle D_{A}^{(2)}(P\|Q)\geq\frac{a^{2}}{2\sigma_{Q}^{2}}.

(see [16][(180)]). Since $s(1-s)=\frac{\sigma_{Q}^{2}}{4v^{2}}$ and $r-s=\frac{a}{2v}$ hold due to (14) and (15), it follows that

\displaystyle d_{A}^{(2)}(r\|s)=\frac{1}{2}\Bigl{(}\frac{(r-s)^{2}}{s}+\frac{(1-r-(1-s))^{2}}{1-s}\Bigr{)}=\frac{(r-s)^{2}}{2s(1-s)}=\frac{a^{2}}{2\sigma_{Q}^{2}}.

By the duality (3), we obtain inequalities for $\alpha=-1$ . The relation (8) for $\alpha=1$ and the duality (3) yield inequalities for $\alpha=0,1$ (see [16][Theorem 2]). By combining these results, we obtain lower bounds for $-1\leq\alpha\leq 0$ and $1\leq\alpha\leq 2$ . ∎

Proof of the necessary condition.

Due to the duality $\eqref{eq-duality}$ , it is enough to show for $\alpha<-1$ . We show an example of $(P,Q)\in\mathcal{P}_{3}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ such that $d_{A}^{(\alpha)}(r\|s)>D_{A}^{(\alpha)}(P\|Q)$ . Let $(R^{\prime},S^{\prime})\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},0]$ , which is defined on $(u^{\prime}_{1},u^{\prime}_{2})$ . We obtain $s^{\prime}=1$ due to $\sigma_{Q}=0$ . Let $(P,Q)\in\mathcal{P}_{3}$ be

	$\displaystyle(P(u^{\prime}_{1}),P(u^{\prime}_{2}),P(u^{\prime}_{3}))$	$\displaystyle=\Bigl{(}r^{\prime}-\frac{1}{{u^{\prime}_{3}}^{2+\delta}},\;1-r^{\prime},\;\frac{1}{{u^{\prime}_{3}}^{2+\delta}}\Bigr{)},$
	$\displaystyle(Q(u^{\prime}_{1}),Q(u^{\prime}_{2}),Q(u^{\prime}_{3}))$	$\displaystyle=\Bigl{(}1-\frac{\sigma_{Q}^{2}}{{u^{\prime}_{3}}^{2}},\;0,\;\frac{\sigma_{Q}^{2}}{{u^{\prime}_{3}}^{2}}\Bigr{)},$

where $\delta$ is a small positive real number such that $2+\alpha\delta>0$ . As $u_{3}^{\prime}\rightarrow\infty$ , $P$ and $Q$ have means $(m_{P},m_{Q})$ and variances $(\sigma_{P}^{2},\sigma_{Q}^{2})$ . Since $P(u^{\prime}_{3})^{\alpha}Q(u^{\prime}_{3})^{1-\alpha}=O({u^{\prime}_{3}}^{-(2+\alpha\delta)})$ , it follows that $D_{A}^{(\alpha)}(P\|Q)\rightarrow d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime})$ . By Lemma 4, there exists $\sigma_{Q}$ such that $d_{A}^{(\alpha)}(r\|s)>D_{A}^{(\alpha)}(P\|Q)$ . ∎

Proof of Item (c).

Since the proof is similar to [16][Theorem 2], we outline of the proof. We construct sequence of probability measures $\{(P_{j},Q_{j})\}$ with zero mean and respective variances $(\sigma_{P}^{2},\sigma_{Q}^{2})$ for which $D_{A}^{(\alpha)}(P_{j}\|Q_{j})\rightarrow 0$ as $j\rightarrow\infty$ (without any loss of generality, one can assume that the equal means are equal to zero). We start by assuming $\min\{\sigma_{P}^{2},\sigma_{Q}^{2}\}\geq 1$ . Let

\displaystyle\mu_{j}:=\sqrt{1+j(\sigma_{Q}^{2}-1)},

and define a sequence of quaternary real-valued random variables with probability mass functions

\displaystyle Q_{j}(u):=\begin{dcases}\frac{1}{2}-\frac{1}{2j},&\quad u=\pm 1,\\ \frac{1}{2j},&\quad u=\pm\mu_{j}.\end{dcases}

It can be verified that, for all $j\in\mathbb{N}$ , $Q_{j}$ has zero mean and variance $\sigma_{Q}^{2}$ .

Furthermore, let

\displaystyle P_{j}(u):=\begin{dcases}\frac{1}{2}-\frac{\xi}{2j},&\quad u=\pm 1,\\ \frac{\xi}{2j},&\quad u=\pm\mu_{j},\end{dcases}

with

\displaystyle\xi:=\frac{\sigma_{P}^{2}-1}{\sigma_{Q}^{2}-1}.

If $\xi>1$ , for $j=1,\cdots,\lceil\xi\rceil$ , we choose $P_{j}$ arbitrary with mean $0$ and variance $\sigma_{P}^{2}$ . Then,

\displaystyle D_{A}^{(\alpha)}(P_{j}\|Q_{j})=d_{A}^{(\alpha)}\Bigl{(}\frac{\xi}{j}\|\frac{1}{j}\Bigr{)}\rightarrow 0.

Next, suppose $\min\{\sigma_{P}^{2},\sigma_{Q}^{2}\}:=\sigma^{2}<1$ , then construct $P^{\prime}_{j}$ and $Q^{\prime}_{j}$ as before with variances $\frac{2\sigma_{P}^{2}}{\sigma^{2}}>1$ and $\frac{2\sigma_{Q}^{2}}{\sigma^{2}}>1$ , respectively. If $P_{j}$ and $Q_{j}$ denote the random variables $P^{\prime}_{j}$ and $Q^{\prime}_{j}$ scaled by a factor of $\frac{\sigma}{\sqrt{2}}$ , then their variances are $\sigma_{P}^{2},\sigma_{Q}^{2}$ , respectively, and $D_{A}^{(\alpha)}(P_{j},Q_{j})=D_{A}^{(\alpha)}(P^{\prime}_{j},Q^{\prime}_{j})\rightarrow 0$ as we let $j\rightarrow\infty$ . ∎

V Conclusion

In this paper, we derived relations between $\alpha$ and $\alpha+1$ for the asymmetric $\alpha$ -divergences. These relations are generalizations of the integral relation between the Kullback-Leibler divergence and the $\chi^{2}$ -divergence. We showed that $\alpha\in[-1,2]$ is the necessary and sufficient condition that the binary $\alpha$ -divergences always attain lower bounds under given means and variances. Kullback-Leibler divergence, Hellinger distance, and $\chi^{2}$ -divergence satisfy this condition. It is intuitively natural that discrepancy between probability measures under moment constraints is smaller for localized measures than for broad probability measures. In this point of view, $\alpha$ -divergences for $\alpha\in[-1,2]$ have preferable properties. The tight lower bounds for the Kullback-Leibler divergence and the Hellinger distance recently began to be applied to physics [9, 19]. In the future, we hope that the range of applications of relations between the $\alpha$ -divergences and tight lower bounds will expand, and we hope that they will help to progress many fields and to deepen the understanding of properties of divergences.

References

[1] S.-i. Amari. Information geometry and its applications, volume 194. Springer, 2016.
[2] K. M. Audenaert. Quantum skew divergence. Journal of Mathematical Physics, 55(11):112202, 2014.
[3] D. G. Chapman, H. Robbins, et al. Minimum variance estimation without regularity assumptions. The Annals of Mathematical Statistics, 22(4):581–586, 1951.
[4] A. Cichocki and S.-i. Amari. Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities. Entropy, 12(6):1532–1568, 2010.
[5] I. Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229–318, 1967.
[6] I. Csiszár. On topological properties of f-divergences. Studia Math. Hungar., 2:329–339, 1967.
[7] I. Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1-4):191–213, 1972.
[8] M. Dashti and A. M. Stuart. The bayesian approach to inverse problems. arXiv preprint arXiv:1302.6989, 2013.
[9] Y. Hasegawa. Irreversibility, loschmidt echo, and thermodynamic uncertainty relation. Physical Review Letters, 127(24):240602, 2021.
[10] E. Hellinger. Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik (Crelles Journal), 1909(136):210–271, 1909.
[11] M. A. Katsoulakis, L. Rey-Bellet, and J. Wang. Scalable information inequalities for uncertainty quantification. Journal of Computational Physics, 336:513–545, 2017.
[12] S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951.
[13] F. Liese and I. Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):4394–4412, 2006.
[14] J. Melbourne, M. Madiman, and M. V. Salapaka. Relationships between certain f -divergences. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1068–1073, 2019.
[15] T. Nishiyama. A tight lower bound for the hellinger distance with given means and variances. arXiv preprint arXiv:2010.13548, 2020.
[16] T. Nishiyama and I. Sason. On relations between the relative entropy and $\chi$ 2-divergence, generalizations and applications. Entropy, 22(5):563, 2020.
[17] K. Pearson. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302):157–175, 1900.
[18] A. Rényi et al. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. The Regents of the University of California, 1961.
[19] T. Van Vu, Y. Hasegawa, et al. Unified approach to classical speed limit and thermodynamic uncertainty relation. Physical Review E, 102(6):062132, 2020.

Appendix A Proof of Lemma 1

Proof.

Let $n\geq 3$ ( $n=1,2$ are trivial), and let $p_{i}:=P(u_{i})$ , $q_{i}:=Q(u_{i})$ . Consider the following minimization problem.

	$\displaystyle\mbox{minimize}\quad-\sum_{i}p_{i}^{\alpha}q_{i}^{1-\alpha},$	(42)
subject to	$\displaystyle g_{k}(\bm{p},\bm{u}):=\sum_{i}p_{i}u_{i}^{k-1}-A_{k}=0,$	(43)
	$\displaystyle g_{k+3}(\bm{q},\bm{u}):=\sum_{i}q_{i}u_{i}^{k-1}-B_{k}=0,\quad\mbox{for}\hskip 4.26773ptk=1,2,3,$	(44)
	$\displaystyle 0\leq p_{i}\leq 1,\quad 0\leq q_{i}\leq 1,\quad\|u_{i}\|\leq R,\quad\mbox{for}\hskip 4.26773pt1\leq i\leq n,$	(45)

where $\bm{A}:=(1,m_{P},\sigma_{P}^{2}+m_{P}^{2})^{\mathrm{T}}$ , $\bm{B}:=(1,m_{Q},\sigma_{Q}^{2}+m_{Q}^{2})^{\mathrm{T}}$ , and (43) and (44) correspond to (4). Since the feasible set is compact, there exists a global minimum. We first consider $\sigma_{P}>0$ and $\sigma_{Q}>0$ , then we have $p_{i},q_{i}<1$ for $1\leq i\leq n$ . Hence, the global minimum point must be a stationary point, or be on the boundary at $\max_{i}|u^{*}_{i}|=R$ , $p^{*}_{i}=0$ , or $q^{*}_{i}=0$ . By rearranging the order of $\{p^{*}_{i}\},\{q^{*}_{i}\}$ appropriately, let

	$\displaystyle p^{}_{i}>0,\;q^{}_{i}$	$\displaystyle>0,\quad\mbox{for}\hskip 4.26773pti\leq I,$
	$\displaystyle p^{}_{j}>0,\;q^{}_{j}$	$\displaystyle=0,\quad\mbox{for}\hskip 4.26773ptI<j\leq I+J,$
	$\displaystyle p^{}_{k}=0,\;q^{}_{k}$	$\displaystyle>0,\quad\mbox{for}\hskip 4.26773ptI+J<k\leq I+J+K=n.$

Since $-\sum_{i\leq I}p_{i}^{\alpha}q_{i}^{1-\alpha}<0$ holds for the global minimum point, we have $I\geq 1$ . If $J>0$ , let $(P^{\prime},Q^{\prime},\bm{u^{\prime}})$ be

	$\displaystyle p^{\prime}_{i}=p^{}_{i}+\mathrm{d}p_{i},\quad q^{\prime}_{i}=q^{}_{i}+\mathrm{d}q_{i},\quad\mbox{for}\hskip 4.26773pt1\leq i\leq I,$
	$\displaystyle p^{\prime}_{I+1}=p^{*}_{I+1}+\mathrm{d}p_{I+1},\quad q^{\prime}_{I+1}=\epsilon>0,$
	$\displaystyle p^{\prime}_{j}=p^{*}_{j}+\mathrm{d}p_{j},\quad q^{\prime}_{j}=0,\quad\mbox{for}\hskip 4.26773ptI+1<j\leq I+J,$
	$\displaystyle p^{\prime}_{k}=0,\quad q^{\prime}_{k}=q^{*}_{k}+\mathrm{d}q_{k},\quad\mbox{for}\hskip 4.26773ptI+J<k\leq I+J+K=n,$
	$\displaystyle u^{\prime}_{i}=u^{*}_{i}+\mathrm{d}u_{i},\quad\mbox{for}\hskip 4.26773pti\leq n,$

where $\epsilon$ and $\mathrm{d}\cdot$ denote sufficiently small real numbers. The moment constraints for probability measures $P^{\prime}$ and $Q^{\prime}$ include $\{\mathrm{d}p_{1},\mathrm{d}p_{I+1},\mathrm{d}u_{I+1}\}$ and $\{\mathrm{d}q_{1},\mathrm{d}q_{l},\mathrm{d}u_{l}\}$ ( $l\neq I+1$ ), respectively. These variables are independent, and it can be easily verified that the determinants for each probability measure constraint are non-zero by $u^{*}_{i}\neq u^{*}_{j}$ for $i\neq j$ . Hence, one can choose $\mathrm{d}\cdot$ such that $(P^{\prime},Q^{\prime})\in\mathcal{P}[m_{P},\sigma_{P},m_{Q},\sigma_{Q}]$ and $\mathrm{d}\cdot=O(\epsilon)$ . By $0<\alpha<1$ and $p^{*}_{I+1}>0$ ,

\displaystyle-\sum_{i\leq I}{p^{*}}_{i}^{\alpha}{q^{*}}_{i}^{1-\alpha}-\Bigl{(}-\sum_{i\leq I}{p^{\prime}}_{i}^{\alpha}{q^{\prime}}_{i}^{1-\alpha}-{p^{\prime}}_{I+1}^{\alpha}{q^{\prime}}_{I+1}^{1-\alpha}\Bigr{)}=\epsilon^{1-\alpha}{p^{*}_{I+1}}^{\alpha}+O(\epsilon)>0.

(46)

This contradicts that $(\bm{p}^{*},\bm{q}^{*},\bm{u}^{*})$ is a global minimum, then we obtain $J=0$ . In a similar way, we also obtain $K=0$ and $I=n$ . Hence, the global minimum point is an interior point or on the the boundary at $\max_{i}|u^{*}_{i}|=R$ . Supposing $\max_{i}|u^{*}_{i}|<R$ , it must be a stationary point of the following Lagrangian.

\displaystyle L(\bm{p},\bm{q},\bm{u},\bm{\lambda}):=-\sum_{i\leq n}p_{i}^{\alpha}q_{i}^{1-\alpha}+\sum_{i\leq n}p_{i}\phi_{\bm{\lambda}}(u_{i})+\sum_{i\leq n}q_{i}\psi_{\bm{\lambda}}(u_{i})-\sum_{k=1}^{3}\lambda_{k}A_{k}-\sum_{k=1}^{3}\lambda_{k+3}B_{k},

(47)

where $\phi_{\bm{\lambda}}(u):=\sum_{k=1}^{3}\lambda_{k}u^{k-1}$ and $\psi_{\bm{\lambda}}(u):=\sum_{k=1}^{3}\lambda_{k+3}u^{k-1}$ . Since $u_{i}\neq u_{j}$ for $i\neq j$ , and $n\geq 3$ , it follows that $\{\nabla g_{k}\}_{k\leq 6}$ are linearly independent. The stationary conditions are

$\displaystyle\frac{\partial{L}}{\partial{p_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=-\alpha{\Bigl{(}\frac{p^{}_{i}}{q^{}_{i}}\Bigr{)}}^{\alpha-1}+\phi_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(48)
$\displaystyle\frac{\partial{L}}{\partial{q_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=-(1-\alpha){\Bigl{(}\frac{p^{}_{i}}{q^{}_{i}}\Bigr{)}}^{\alpha}+\psi_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(49)
$\displaystyle\frac{\partial{L}}{\partial{u_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=p^{}_{i}\phi^{\prime}_{\bm{\lambda}^{}}(u^{}_{i})+q^{}_{i}\psi^{\prime}_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(50)

where ^′ denotes the derivative with respect to $u$ .

From (48) and (49), it follows that

\displaystyle\psi_{\bm{\lambda}^{*}}(u^{*}_{i})=(1-\alpha){\Bigl{(}\frac{\phi_{\bm{\lambda}^{*}}(u^{*}_{i})}{\alpha}\Bigr{)}}^{\frac{\alpha}{\alpha-1}}.

(51)

Substituting (48) and (50) into (49), we have

\displaystyle\alpha\phi^{\prime}_{\bm{\lambda}^{*}}(u^{*}_{i})\psi_{\bm{\lambda}^{*}}(u^{*}_{i})+(1-\alpha)\phi_{\bm{\lambda}^{*}}(u^{*}_{i})\psi^{\prime}_{\bm{\lambda}^{*}}(u^{*}_{i})=0.

(52)

Since $\phi_{\bm{\lambda}^{*}}(u^{*})$ and $\psi_{\bm{\lambda}^{*}}(u^{*})$ are positive from (48) and (49), one can define the function $\beta(u):=\alpha\log\frac{\phi_{\bm{\lambda}^{*}}(u)}{\alpha}+(1-\alpha)\log\frac{\psi_{\bm{\lambda}^{*}}(u)}{1-\alpha}$ . From (51) and (52), we obtain

\displaystyle\beta(u^{*}_{i})=\beta^{\prime}(u^{*}_{i})=0.

(53)

Since $\phi_{\bm{\lambda}^{*}}(u)$ and $\psi_{\bm{\lambda}^{*}}(u)$ are at most quadratic functions with respect to $u$ , the equation (52) has at most a degree of $3$ . If (52) is not identically $0$ , we have $n\leq 3$ . If $n=3$ , by the relation (53) and the mean value theorem, there exists $u_{l}\neq u^{*}_{1},u^{*}_{2},u^{*}_{3}$ such that $\beta^{\prime}(u_{l})=0$ . By the definition of $\beta(\cdot)$ , it follows that $u_{l}$ is a solution of (52). This contradicts that the equation (52) has at most a degree of $3$ . If equation (52) is identically $0$ , the equality (51) is identity. Since the degree of $\phi_{\bm{\lambda}^{*}}(u)$ and $\psi_{\bm{\lambda}^{*}}(u)$ are any of $\{0,\;1,\;2\}$ , and due to $0<\alpha<1$ , it follows that they are constant. The equality (48) yields $q_{i}=Cp_{i}$ for all $i$ and a constant $C$ . This implies $p_{i}=q_{i}$ for all $i$ , and it contradicts $m_{P}\neq m_{Q}$ . Hence, we obtain $n\leq 2$ .

Next, we show the case for $\sigma_{P}>0$ and $\sigma_{Q}=0$ briefly. The notation is the same as above, and $I=1$ and $J=n-1$ hold due to $\sigma_{Q}=0$ . The Lagrangian is

\displaystyle L(\bm{p},\bm{u},\bm{\lambda}):=-p_{1}^{\alpha}+\sum_{i\leq n}p_{i}\phi_{\bm{\lambda}}(u_{i})-\sum_{k=1}^{3}\lambda_{k}A_{k}.

(54)

Supposing $\max_{i}|u^{*}_{i}|<R$ , the stationary conditions are

$\displaystyle\frac{\partial{L}}{\partial{p_{1}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=-\alpha{p^{}_{1}}^{\alpha-1}+\phi_{\bm{\lambda}^{}}(u^{*}_{1})=0,$	(55)
$\displaystyle\frac{\partial{L}}{\partial{p_{i}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=\phi_{\bm{\lambda}^{}}(u^{}_{i})=0,\quad\mbox{for}\hskip 4.26773pt2\leq i\leq n,$	(56)
$\displaystyle\frac{\partial{L}}{\partial{u_{i}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=p^{}_{i}\phi^{\prime}_{\bm{\lambda}^{}}(u^{*}_{i})=0,\quad\mbox{for}\hskip 4.26773pt1\leq i\leq n.$	(57)

From (55) and (56), it follows that $\phi_{\bm{\lambda}^{*}}(u^{*}_{i})$ is not a constant. Since $\phi^{\prime}_{\bm{\lambda}^{*}}(u)$ is at most a dgree of 1 from (57) and $p^{*}_{i}>0$ for $2\leq i\leq n$ , it follows that $n\leq 2$ . The result for $\sigma_{P}=0$ and $\sigma_{Q}>0$ also follows by swapping $P$ and $Q$ . ∎

Appendix B Proof of Lemma 2

We first prove the following lemma.

Lemma 5.

Let $0<\alpha<1$ . If $0<x\leq 1$ ,

\displaystyle\frac{(1-\alpha x)(1+x)^{\alpha}}{(1+\alpha x)(1-x)^{\alpha}}>1.

If $-1\leq x<0$ ,

\displaystyle\frac{(1-\alpha x)(1+x)^{\alpha}}{(1+\alpha x)(1-x)^{\alpha}}<1.

Proof.

Letting $F(x):=\log\frac{(1-\alpha x)(1+x)^{\alpha}}{(1+\alpha x)(1-x)^{\alpha}}$ , for $0<|x|\leq 1$ and $0<\alpha<1$ , we have

\displaystyle F^{\prime}(x)=\alpha\Bigl{(}\frac{1}{1+x}+\frac{1}{1-x}-\frac{1}{1-\alpha x}-\frac{1}{1+\alpha x}\Bigr{)}=\frac{2\alpha(1-\alpha^{2})x^{2}}{(1-x^{2})(1-\alpha^{2}x^{2})}>0.

By combining this relation with $F(0)=0$ , the results follow. ∎

Proof of Lemma 2.

Let $V_{P}:=\sigma_{P}^{2}$ and $V_{Q}:=\sigma_{Q}^{2}$ , and $f^{(\alpha)}(V_{P},V_{Q},a):=r^{\alpha}s^{1-\alpha}$ . From (14)-(17), by replacing $a$ by $-a$ , we obtain $(r,s)\rightarrow(1-r,1-s)$ . Hence, the binary $\alpha$ -divergences are written by $d_{A}^{(\alpha)}(r\|s)=\frac{1}{\alpha(\alpha-1)}(f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)-1)$ . Since $0<\alpha<1$ , we show that $f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)$ is monotonically increasing with respect to $V_{P}$ and $V_{Q}$ .

From (14)-(17), we have

$\displaystyle\frac{\partial v}{\partial V_{Q}}$	$\displaystyle=\frac{V_{Q}-V_{P}+a^{2}}{4a^{2}v},$	(58)
$\displaystyle\frac{\partial r}{\partial V_{Q}}$	$\displaystyle=-\frac{(V_{Q}-V_{P}+a^{2})^{2}}{16a^{3}v^{3}}+\frac{1}{4av}=\frac{V_{P}}{4av^{3}},$	(59)
$\displaystyle\frac{\partial s}{\partial V_{Q}}$	$\displaystyle=-\frac{(V_{Q}-V_{P}+a^{2})(V_{Q}-V_{P}-a^{2})}{16a^{3}v^{3}}+\frac{1}{4av}=\frac{V_{P}+V_{Q}+a^{2}}{8av^{3}}.$	(60)

By combining (59) and (60), it follows that

\displaystyle\frac{\partial f^{(\alpha)}}{\partial V_{Q}}=\frac{1}{8av^{3}}{\Bigl{(}\frac{r}{s}\Bigr{)}}^{\alpha}\Bigr{(}(1-\alpha)(V_{P}+V_{Q}+a^{2})+2\alpha V_{P}\frac{s}{r}\Bigr{)}.

(61)

By $r-s=\frac{a}{2v}$ and $s(1-s)=\frac{V_{Q}}{4v^{2}}$ , we obtain

\displaystyle\frac{r}{s}=1+\frac{a}{2vs}=1+\frac{2va(1-s)}{V_{Q}}=\frac{V_{P}+V_{Q}+a^{2}+2va}{2V_{Q}}.

(62)

By replacing $(V_{P},V_{Q},a)$ by $(V_{Q},V_{P},-a)$ , we obtain $(r,s)\rightarrow(s,r)$ . From (62), we have

\displaystyle\frac{s}{r}=\frac{V_{P}+V_{Q}+a^{2}-2va}{2V_{P}}.

(63)

Substituting (62) and (63) into (61), we obtain

	$\displaystyle\frac{\partial f^{(\alpha)}}{\partial V_{Q}}(V_{P},V_{Q},a)$	$\displaystyle=\frac{1}{8av^{3}}{\Bigl{(}\frac{V_{P}+V_{Q}+a^{2}+2va}{2V_{Q}}\Bigr{)}}^{\alpha}\Bigr{(}(V_{P}+V_{Q}+a^{2})-2\alpha va\Bigr{)}$		(64)
		$\displaystyle=\frac{1}{8av^{3}}{\Bigl{(}\frac{V_{P}+V_{Q}+a^{2}}{2V_{Q}}\Bigr{)}}^{\alpha}(V_{P}+V_{Q}+a^{2})(1+x)^{\alpha}(1-\alpha x),$		(65)

where $x:=\frac{2va}{V_{P}+V_{Q}+a^{2}}$ and $|x|\leq 1$ . Hence, we obtain

		$\displaystyle\frac{\partial\Bigl{(}f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)\Bigr{)}}{\partial V_{Q}}$
		$\displaystyle=\frac{1}{8av^{3}}{\Bigl{(}\frac{V_{P}+V_{Q}+a^{2}}{2V_{Q}}\Bigr{)}}^{\alpha}(V_{P}+V_{Q}+a^{2})\Bigl{(}(1+x)^{\alpha}(1-\alpha x)-(1-x)^{\alpha}(1+\alpha x)\Bigr{)}.$		(66)

If $a>0$ , by Lemma 5, (B) and $0<x\leq 1$ , it follows that

\displaystyle\frac{\partial\Bigl{(}f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)\Bigr{)}}{\partial V_{Q}}>0.

(67)

If $a<0$ , we obtain (67) due to $-1\leq x<0$ and Lemma 5. By (14), (15), and the definition of $f^{(\alpha)}(V_{P},V_{Q},a)$ , we have $f^{(\alpha)}(V_{P},V_{Q},a)=f^{(1-\alpha)}(V_{Q},V_{P},-a)$ . Hence, by combinig this relation with (67) for $1-\alpha$ , we have $\frac{\partial\Bigl{(}f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)\Bigr{)}}{\partial V_{P}}>0$ . ∎

Appendix C Proof of Lemma 4

Proof.

Let $\alpha<-1$ . In a similar way to the proof of Lemma 2, we obtain (B). For $a>0$ , let

\displaystyle x(V_{Q})=\frac{2va}{V_{P}+V_{Q}+a^{2}},

where we write $V_{Q}$ explicitly in $x$ in the proof of Lemma 2. By $x(0)=1$ , there exists $V_{Q}=\sigma_{Q}^{2}$ such that $x(z)>0$ and $1+\alpha x(z)<0$ for all $z\in[0,V_{Q}]$ . By combining these inequalities with $|x(V_{Q})|\leq 1$ , it follows that (B) is positive for all $z\in[0,V_{Q}]$ . By $d_{A}^{(\alpha)}(r\|s)=\frac{1}{\alpha(\alpha-1)}(f^{(\alpha)}(V_{P},V_{Q},a)+f^{(\alpha)}(V_{P},V_{Q},-a)-1)$ and $\alpha(\alpha-1)>0$ , it follows that

\displaystyle d_{A}^{(\alpha)}(r\|s)>d_{A}^{(\alpha)}(r^{\prime}\|s^{\prime}),

(68)

where $(R^{\prime},S^{\prime})\in\mathcal{P}_{2}\cap\mathcal{P}[m_{P},\sigma_{P},m_{Q},0]$ . The case for $a<0$ can be justified in a similar way.

∎

$\displaystyle\frac{\partial{L}}{\partial{p_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=-\alpha{\Bigl{(}\frac{p^{}_{i}}{q^{}_{i}}\Bigr{)}}^{\alpha-1}+\phi_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(48)
$\displaystyle\frac{\partial{L}}{\partial{q_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=-(1-\alpha){\Bigl{(}\frac{p^{}_{i}}{q^{}_{i}}\Bigr{)}}^{\alpha}+\psi_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(49)
$\displaystyle\frac{\partial{L}}{\partial{u_{i}}}(\bm{p}^{},\bm{q}^{},\bm{u}^{},\bm{\lambda}^{})$	$\displaystyle=p^{}_{i}\phi^{\prime}_{\bm{\lambda}^{}}(u^{}_{i})+q^{}_{i}\psi^{\prime}_{\bm{\lambda}^{}}(u^{}_{i})=0,$	(50)

$\displaystyle\frac{\partial{L}}{\partial{p_{1}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=-\alpha{p^{}_{1}}^{\alpha-1}+\phi_{\bm{\lambda}^{}}(u^{*}_{1})=0,$	(55)
$\displaystyle\frac{\partial{L}}{\partial{p_{i}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=\phi_{\bm{\lambda}^{}}(u^{}_{i})=0,\quad\mbox{for}\hskip 4.26773pt2\leq i\leq n,$	(56)
$\displaystyle\frac{\partial{L}}{\partial{u_{i}}}(\bm{p}^{},\bm{u}^{},\bm{\lambda}^{*})$	$\displaystyle=p^{}_{i}\phi^{\prime}_{\bm{\lambda}^{}}(u^{*}_{i})=0,\quad\mbox{for}\hskip 4.26773pt1\leq i\leq n.$	(57)

Tight Lower Bounds for α\alpha-Divergences Under Moment Constraints and Relations Between Different α\alpha

Abstract

I Introduction

II Preliminaries

Definition 1.

Definition 2.

Definition 3.

Definition 4.

Definition 5.

Definition 6.

Definition 7.

III Main results

Theorem 1.

Proof.

Corollary 1.

Proof.

Theorem 2.

Proof.

Corollary 2.

Proof.

IV Proofs of main results

IV-A Proof of Theorem 1

Proof.

IV-B Proof of Theorem 2

Lemma 1.

Proof.

Lemma 2.

Proof.

Lemma 3.

Proof.

Lemma 4.

Proof.

Proof for 0<α<10<\alpha<1.

Proof for −1≤α≤0-1\leq\alpha\leq 0 and 1≤α≤21\leq\alpha\leq 2.

Proof of the necessary condition.

Proof of Item (c).

V Conclusion

References

Appendix A Proof of Lemma 1

Proof.

Appendix B Proof of Lemma 2

Lemma 5.

Proof.

Proof of Lemma 2.

Appendix C Proof of Lemma 4

Proof.

Tight Lower Bounds for $\alpha$ -Divergences Under Moment Constraints and Relations Between Different $\alpha$

Proof for $0<\alpha<1$ .

Proof for $-1\leq\alpha\leq 0$ and $1\leq\alpha\leq 2$ .