ON LOG-SUM INEQUALITIES

In information theory, the so-called log-sum inequality is a generalization of Shannon inequality (often also called Klein inequality) which is used to prove the non-negativity of relative entropy. The essence of the non-negativity of the relative entropy is the simple inequality $\ln x\leq x-1$ for $x>0$. Therefore, log-sum inequality is important to study information theory. This is a variant of the Jensen inequality of convex functions, which plays a crucial role for proving the Gibbs’ inequality or the convexity of Kullback-Leibler divergence [1].

Recall that, a function $f:X\rightarrow\mathbb{R}$ defined on a convex set $X$ is said to be a convex function if for all $x_{1},x_{2}\in X$ and for all $t\in[0,1]$ we have $tf(x_{1})+(1-t)f(x_{2})\geq f(tx_{1}+(1-t)x_{2})$. When $X$ is the set of real numbers, this inequality is mentioned as the Jensen inequality. In general, we represent the Jensen inequality [2], [3] as

A twice-differentiable real valued function $f(x)$ on $\mathbb{R}$ is convex if and only if $f^{\prime\prime}(x)\geq 0$. Let $a_{1},a_{2},\cdots,a_{n}$ and $b_{1},b_{2},\cdots,b_{n}$ be non-negative numbers. As $f(x)=x\log(x)$ is a convex function, the Jensen inequality (1) suggests

where $t_{i}=\dfrac{b_{i}}{\sum\limits_{j=1}^{n}b_{j}}$ and $x_{i}=\dfrac{a_{i}}{b_{i}}$. Now, replacing $a=\sum\limits_{j=1}^{n}a_{j}$ and $b=\sum\limits_{j=1}^{n}b_{j}$ we obtain,

which is the standard log-sum inequality. The inequality is valid for $n=\infty$ provided $a<\infty$ as well as $b<\infty$. In [4], an analog of log-sum inequality is derived as

where $f$ is strictly convex at $\dfrac{a}{b}$. The equality holds if and only if $a_{i}=cb_{i}$, for $i=1,2,\cdots n$, and constant number $c$.

Although log sum inequality is important in classical information theory, a detailed study on this topic is not available in the literature, to the best of our knowledge. This article demonstrates a number of inequalities which are analogous to the log-sum inequality. First, we extend the log-sum inequality with two real-valued functions, which is illustrated in Theorem 1. This theorem expands the applicability of the log-sum inequality as it breaks the restriction that $a_{i}$ and $b_{i}$ are the real numbers. The convexity of a function is essential for proving log-sum inequality. Considering the concavity of a function, we also find analogous results. Investigating the variants of log-sum inequality for concave functions is crucial as there are classes of functions whose convexity depends on specific values of a parameter, for instance, the $q$-deformed logarithm. We also elaborate the matrix analogs of the log-sum inequality. Here we observe that all the inequalities derived for the real functions can be easily extended as trace-form-inequality for commuting self-adjoint matrices. A recent trend in matrix analysis is constructing matrix theoretic counterparts of the known inequalities for real functions, where the matrix inequality is driven by the Löwner partial order relation. Proving the log-sum inequality in this context is non-trivial and difficult. We are able to derive it under several conditions. To the best of our knowledge, this is the first attempt in literature to extend the log-sum inequality for the non-commutative positive semidefinite matrices.

This article is distributed into five sections. Preliminary concepts are discussed before applying them in a mathematical derivation. In section 2, we consider inequalities for real-valued functions. Here, we discuss the log-sum inequality for deformed logarithms, for instance the $q$-deformed logarithm. Section 3 is dedicated to the discussion of log-sum inequality as a trace-form-inequality for commuting self-adjoint matrices. It has a number of consequences in quantum information theory. In the next section, we attempt to derive the log-sum inequality for Löwner partial order relation. This section extensively uses the idea of operator monotone, operator convex functions, and the operator Jensen inequality, which we mention at the beginning of section 4. Then we conclude the article.

In this section, we extend the idea of the log-sum inequality for real-valued functions. Both concave and convex functions are considered for the investigation. Interestingly, the $q$-deformed logarithm is convex when $q<2$, which assists us to illustrate two sets of log-sum inequalities. We begin our discussion with the following theorem.

From now on, we denote $m_{g}:=\min\limits_{i}\left\{\dfrac{g(a_{i})}{g(b_{i})}\right\}$ and $M_{g}:=\max\limits_{i}\left\{\dfrac{g(a_{i})}{g(b_{i})}\right\}$ throughout the article, where $g$ is a real valued function whose domain of definition contains $a_{1},a_{2},\cdots,a_{n}$ and $b_{1},b_{2},\cdots,b_{n}$, such that $g(b_{i})>0$ for all $i=1,2,\cdots n$.

Defining $g(x)=x$ and $f(x)=\log(x)$ for $x\in\mathbb{R}_{>0}$, we observe that $h(x)=x\log(x)$ is a convex function. Then, Theorem 1 leads us to the log-sum inequality mentioned in equation (2).

Theorem 1 expands the scope of applications of the log-sum inequality. The domain of $g$ may not be a convex set. We can consider $a_{i}$ and $b_{i}$ arbitrarily such that $g(b_{i})>0$ for all $i=1,2,\cdots n$. In other words, the range of $g$ must have a non-empty intersection with the set of positive reals.

Let $f$ be a twice-differentiable function in $\left[m_{g},M_{g}\right]$. The function $xf(x)$ is convex if and only if

If $m_{g}\geq 0$, then any monotone increasing and convex function $f$ defined on $\left[m_{g},M_{g}\right]$ fulfill equation (5). Now, we are in a position to consider a few special cases of Theorem 1.

Recall that, a real-valued function $f$ defined on a convex set $X$ is said to be concave if $-f(x)$ is convex. Applying it in equation (1) observe that for a concave function $f$ we have

Considering $h(x)=xf(x)$ as a concave function in equation (4), we obtain

under the equivalent conditions on $a_{i}$ and $b_{i}$ as well as the real valued functions $f$ and $g$ as mentioned in Theorem 1.

When $q>2$ in equation (7) we observe that $h(x)=xf(x)$ is a concave function. Therefore the inequalities in equation (9) and (10) becomes

and

respectively, under the similar conditions on $a_{i}$ and $b_{i}$.

If $f(x)=\log(x)$ we find that $-xf(x)=xf\left(\dfrac{1}{x}\right)$ is a concave function. The equation (2) suggests that

which implies

In general, $-xf(x)=xf\left(\dfrac{1}{x}\right)$ does not hold. For instance, consider $f(x)=\ln_{q}(x)$, which refers $-x\ln_{q}(x)=x^{q}\ln_{q}\left(\dfrac{1}{x}\right)$. This fact leads us to a new inequality which we consider in the following theorem.

From now on, we use $\widetilde{m}_{g}=\min\limits_{1\leq i\leq n}\left\{\dfrac{g(b_{i})}{g(a_{i})}\right\}$, and $\widetilde{M}_{g}=\max\limits_{1\leq i\leq n}\left\{\dfrac{g(b_{i})}{g(a_{i})}\right\}$ throughout this article, where $g$ is a real valued function whose domain of definition contains $a_{1},a_{2},\cdots,a_{n}$ and $b_{1},b_{2},\cdots,b_{n}$, such that $g(a_{i})>0$ for all $i=1,2,\cdots n$.

We know that a twice-differentiable function $h(x)$ is concave if and only if $h^{\prime\prime}(x)\leq 0$, which indicates

Note that, equation (12) and (5) are not equivalent, in general. For example, consider the function $f(x)=\frac{x}{x^{2}+2}$, where $x>0$. We can calculate

Now, equation (12) suggests that $xf\left(\dfrac{1}{x}\right)$ is concave when

But, putting the values of $f^{\prime}$ and $f^{\prime\prime}$ in equation (5) we observe that $xf(x)$ is concave when

Now, replacing $f(x)=\dfrac{x}{x^{2}+2}$ and $g(x)=x$ in Theorem 2 we observe

where $a_{i}$ and $b_{i}$ are greater than $\dfrac{1}{\sqrt{6}}$.

We begin this section with a number of basic concepts of matrix analysis. Given any self-adjoint matrix $A$, there exists a unitary matrix $U$, such that, $A=U\Lambda(A)U^{\dagger}$, where $\Lambda(A)=\operatorname{diag}\{a_{1},a_{2},\cdots,a_{n}\}$ is a diagonal matrix whose diagonal entries, $a_{i}$ for $i=1,2,\cdots,n$, are the eigenvalues of $A$. Throughout this article $U^{\dagger}$ denotes the conjugate transpose of the matrix $U$. If the matrices $A$ and $B$ commute, that is $AB=BA$, then there exists a unitary matrix $U$, such that, $A=U\Lambda(A)U^{\dagger}$ and $B=U\Lambda(B)U^{\dagger}$, hold simultaneously. We also denote $a=\operatorname{trace}(A)=\sum\limits_{i=1}^{n}a_{i}$. Let $f$ be a continuous real valued function defined on an interval $J$ and $A$ be a self-adjoint matrix with eigenvalues in $J$, then

Now, we demonstrate the results derived in Section 2 for the self-adjoint matrices.

The following corollary holds trivially from the above theorem.

Theorem 3 is a matrix-theoretic counterpart of Theorem 1, which assists us to establish a number of matrix inequalities, which are derived in Section 2 for real functions. Some of these matrix inequalities have immediate consequences in quantum information theory. Consider the following example.

Recall that, in quantum information theory [8] a quantum state is represented by a density matrix $\rho$ which is a positive semidefinite, self-adjoint matrix with $\operatorname{trace}(\rho)=1$. The von-Neumann entropy is a well-known measure of quantum information which is given by $-\operatorname{trace}(\rho\log(\rho))$ for a density matrix $\rho$. Given two density matrices $\rho$ and $\sigma$ the quantum relative entropy is determined by $D(\rho||\sigma)=\operatorname{trace}\left[\rho(\log(\rho)-\log(\sigma))\right]$.

Considering $\operatorname{trace}(A)=\operatorname{trace}(B)=1$ in equation (15), we find that $A$ and $B$ are density matrices. Then, the quantum relative entropy given two density matrices $A$ and $B$,

Considering $B=I$, the identity matrix of order $n$, in equation (14) we observe that