Norms of structured random matrices

In what follows, $A=(a_{ij})_{i,j}$ is a real deterministic matrix and $G=(g_{ij})_{i,j}$ always stands for a random matrix with i.i.d. standard Gaussian entries (usually the matrices are of size $m\times n$ unless explicitly stated otherwise). We use $C(r)$, $C(r,K)$, etc. for positive constants which may depend only on the parameters given in brackets and write $C,C^{\prime},c,c^{\prime},\dots$ for positive absolute constants. The symbols $\lesssim$, $\lesssim_{r}$, $\lesssim_{r,K}$, etc. denote that the inequality holds up to multiplicative constants depending only on the parameters given in the subscripts; we write $a\asymp b$ if $a\lesssim b$ and $b\lesssim a$, and $\asymp_{r}$, $\asymp_{r,K}$, etc. if the constants may depend on the parameters given in the subscript.

In 1975, Bennett, Goodman, and Newman [9] proved that if $X$ is an $m\times n$ random matrix with independent, mean-zero entries taking values in $[-1,1]$, and $2\leq q<\infty$, then

In fact, up to constants, this estimate is best possible: for any $m\times n$ matrix $X^{\prime}$ with $\pm 1$ entries one readily sees that $\|X^{\prime}\colon\ell^{n}_{2}\to\ell^{m}_{q}\|\geq\max\{n^{1/2},m^{1/q}\}$; just use standard unit vectors and operator duality. Moreover, in this ‘unstructured’ case, where $a_{ij}=1$ for all $i,j$, it is easy to extend (1.2) to the whole range of $p,q\in[1,\infty]$ (see [8, 13] or Remark 4.2 below). Also, if all entries are i.i.d. Rademacher random variables, then the bounds are two-sided, i.e., the expected operator norm is, up to constants, the same as the minimal norm for all $p$, $q$ (see [8, Proposition 3.2] or [13, Satz 2]).

The case most studied in the literature is the one of the spectral norm, i.e., the $\ell_{2}^{n}\to\ell_{2}^{m}$ operator norm. Seginer [51] proved in 2000 that if $X=(X_{ij})_{i\leq m,j\leq n}$ is an $m\times n$ random matrix with i.i.d. mean-zero entries, then its operator norm is of the same order as the sum of expectations of the maximum Euclidean norm of rows and columns of $X$, i.e.,

Riemer and Schütt [49] proved that, up to a logarithmic factor $\ln(en)^{2}$, the same holds true for any random matrix with independent but not necessarily identically distributed mean-zero entries. Let us also mention that in the Gaussian setting one can use a non-commutative Khintchine bound (see, e.g., [59, Equation (4.9)]) to show that, up to a factor $\sqrt{\ln n}$, the expected spectral norm is of the order of the largest Euclidean norm of its rows and columns.

In the very same setting that was considered by Riemer and Schütt, Latała [37] had obtained a few years earlier the dimension-free estimate

This bound is superior to the Riemer–Schütt bound in the case of matrices with all entries equal to $1$ and is optimal for Wigner matrices. In other cases, like the one of diagonal matrices, the Riemer–Schütt bound is better.

In the case of structured Gaussian matrices, Latała, van Handel, and Youssef [40], building upon earlier work of Bandeira and van Handel [7] (which combined the moment method with combinatorial considerations) as well as results proved by van Handel in [61] (which used Slepian’s lemma), obtained the precise behavior without any logarithmic terms in the dimension, namely

Their proof is based on a clever block decomposition of the underlying matrix (see [40, Figure 3.1]). This result finally answered in the affirmative a conjecture made by Latała more than a decade before. We also refer the reader to the survey [62] discussing in quite some detail results prior to [40] and [61] — the latter work discusses the conjectures of Latała and van Handel and shows their equivalence.

Very recently, Latała and Świątkowski [39] investigated a similar problem when the underlying random matrix has Rademacher entries. They proved a lower bound which, up to a $\ln\ln n$ factor, can be reversed for randomized $n\times n$ circulant matrices.

In [23], Guédon, Hinrichs, Litvak, and Prochno studied our main and motivating question on the order of the expected operator norm of structured random matrices considered as operators between $\ell_{p}^{n}$ and $\ell_{q}^{m}$ in the special case where $p\leq 2\leq q$ and the random entries are Gaussian. In this situation, where we are not dealing with the spectral norm, the moment method cannot be employed. The approach in [23] was therefore different and based on a majorizing measure construction combining the works [24] and [25]. In [23, Theorem 1.1], the authors proved that if $1<p\leq 2\leq q<\infty$, then

where $\gamma_{r}\coloneqq(\mathbb{E}|g|^{r})^{1/r}$ for a standard Gaussian random variable $g$. Moreover, for $p=1$ and $q\geq 2$, it was noted in [23, Remark 1.4] (see also [45, Twierdzenie 2]) that

Later, an extension of (1.5) to the case of matrices with i.i.d. isotropic log-concave rows was obtained by Strzelecka in [55].

Trying to extend the upper bound for $\mathbb{E}\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|$ to the whole range $1\leq p,q\leq\infty$ one encounters two difficulties. First of all, the methods used in order to prove (1.5) fail if $q\leq 2$ or $p\geq 2$, because the majorizing measure construction used in [23] is restricted to the case $q\geq 2$ and the assumption $1<p\leq 2$ is required in a Hölder bound. Moreover, when $q\leq 2$ or $p\geq 2$ the result cannot hold with the right-hand side of the same form as in (1.5) (see Remark 4.2 below for counterexamples¹¹1By Jensen’s inequality, the expected norm of a matrix with i.i.d. Rademacher entries is less than or equal to $\sqrt{2/\pi}$ times the expected norm of the matrix with Gaussian entries, so (1.5) for $q\leq 2$ or $p\geq 2$ would imply the same (up to a constant) bound for $\pm 1$ matrices, which does not hold in this range of $(p,q)$ as we explain in Remark 4.2. to (1.5) in the cases $q\leq 2$ and $p\geq 2$). This explains the different form of expressions $D_{1}$ and $D_{2}$ in (1.1), which in the range $p\leq 2\leq q$ reduce to the maxima of norms on the right-hand side of (1.5) — see (1.9) below.

By arguments similar to the ones used in order to prove the lower bound in (1.4), one can check that in the range considered in [23, 45] (i.e., $1\leq p\leq 2\leq q\leq\infty$) one has

Note that for $p=1$,

which explains the simplified form of (1.6).

We remark that the proof of (1.7) is based merely on the observation that the operator norm is greater than the maximum entry of the matrix and the appropriate maximum norms of its rows and columns, combined with comparison of moments for Gaussian random vectors. Another but related way to proceed, valid for all $1\leq p,q\leq\infty$, is to exchange expectation and suprema over the $\ell_{p}^{n}$ and $\ell_{q^{\ast}}^{m}$ balls in the definition of the operator norm. We present the details in Subsection 5.1. In particular, Proposition 5.1 and Corollary 5.2 imply²²2We use here also a trivial observation that $\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|\geq\max_{i,j}|a_{ij}g_{ij}|$. that, for $1\leq p,q\leq\infty$,

It is an easy observation (see Lemma 2.1 below) that for $p\leq 2\leq q$,

Thus, in the range $1\leq p\leq 2\leq q<\infty$ considered in [23, 45], the lower bounds (1.7) and (1.8) coincide.

Although it would be natural to conjecture at this point that the bound (1.8) may be reversed up to a multiplicative constant depending only on $p,q$, such a reverse bound turns out not to be true in the case $p\leq q<2$ (and in the dual one $2<p\leq q$) as we shall show in Subsection 5.3.

In order to conjecture the right asymptotic behavior of $\mathbb{E}\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|$, one may take a look at the boundary values of $p$ and $q$, i.e., $p\in\{1,\infty\}$ or $q\in\{1,\infty\}$. Note that (1.6) provides an asymptotic behavior of $\mathbb{E}\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|$ on a part of this boundary (i.e., for $p=1$ and $2\leq q\leq\infty$ and in the dual case $q=\infty$ and $1\leq p\leq 2$). We provide sharp results on the remaining parts of the boundary of $[1,\infty]\times[1,\infty]$ (see dense lines on the boundary of Figure 1 below):

where

and with $(x_{1}^{\downarrow{}},\ldots,x_{n}^{\downarrow{}})$ denoting the non-increasing rearrangement of $(|x_{1}|,\ldots,|x_{n}|)$ for a given $(x_{j})_{j\leq n}\in\mathbb{R}^{n}$. (For the precise formulation see Propositions 1.8 and 1.10, and Corollary 1.11 below.)

Moreover, in Subsection 5.1 we generalize the lower bounds from the boundary into the whole range $(p,q)\in[1,\infty]\times[1,\infty]$ (see Figure 1 below), i.e., we prove

Let us now discuss the relation between the terms appearing above. We postpone the proofs of all the following claims to Section 5.

In the case $p\leq 2\leq q$, we have

where the matrices $(a_{ij}^{\prime})_{i,j}$ and $(a_{ij}^{\prime\prime})_{i,j}$ are obtained by permuting the columns and rows, respectively, of the matrix $(|a_{ij}|)_{i,j}$ in such a way that $\max_{i}a_{i1}^{\prime}\geq\dots\geq\max_{i}a_{in}^{\prime}$ and $\max_{j}a_{1j}^{\prime\prime}\geq\dots\geq\max_{j}a_{mj}^{\prime\prime}$. Therefore, in the range $1\leq p\leq q\leq\infty$ the right-hand side of (1.10) changes continuously with $p$ and $q$ (for a fixed matrix $A$).

Obviously, $\max_{j\leq n}\sqrt{\ln(j+1)}b_{j}^{\downarrow{}}\geq\max_{i\leq m,j\leq n}\sqrt{\ln(j+1)}a_{ij}^{\prime}$ and, in general, the former quantity may be of larger order than the latter one. In Subsection 5.3 we shall present a more subtle relation: for every $1\leq p\leq q<2$ we shall give an example showing that the right-hand side of (1.10) may be of larger order than $D_{1}+D_{2}+\mathbb{E}\max_{i\leq m,j\leq n}|a_{ij}g_{ij}|$. Note that by duality, i.e., the fact that

the same holds in the case $2<p\leq q$. This suggests that the behavior of $\mathbb{E}\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|$ is different in the regions with horizontal or vertical lines than in the region with northeast lines.

Moreover, we have

(see Subsection 5.2). Note that this is not the case for $p\leq q$, as one can easily see by considering, e.g., $A$ equal to the identity matrix. This suggests a different (than in other regions), simplified, behavior of $\mathbb{E}\|G_{A}\colon\ell_{p}^{n}\to\ell_{q}^{m}\|$ in the region with northwest lines.

Given the discussion above, the lower bounds presented in (1.10), and the fact that they can be reversed for all $p\in[1,\infty]$, $q\in\{1,\infty\}$ (and for all $q\in[1,\infty]$, $p\in\{1,\infty\}$), it is natural to conjecture the following.

Let us now present a conjecture concerning the boundedness of linear operators given by infinite dimensional matrices. In what follows, we say that a matrix $B=(b_{ij})_{i,j\in\mathbb{N}}$ defines a bounded operator from $\ell_{p}(\mathbb{N})$ to $\ell_{q}(\mathbb{N})$ if for all $x\in\ell_{p}(\mathbb{N})$ the product $Bx$ is well defined, belongs to $\ell_{q}(\mathbb{N})$ and the corresponding linear operator is bounded.

We remark that it suffices to prove Conjecture 1 in order to confirm Conjecture 2.

We postpone the proof of this proposition to Subsection 5.5.

In this article, in addition to the cases $p=q=2$ obtained in [40] and $p=1,q\geq 2$ proved in [23, 45], we confirm Conjecture 1 when $p\in\{1,\infty\}$, $q\in[1,\infty]$ and when $q\in\{1,\infty\}$, $p\in[1,\infty]$. In all the other cases, we are able to prove the upper bounds only up to logarithmic (in the dimensions $m,n$) multiplicative factors (see Corollary 1.4 below). In particular, Proposition 1.2 implies that Conjecture 2 holds for all $p\in\{1,\infty\}$, $q\in[1,\infty]$ and for all $q\in\{1,\infty\}$, $p\in[1,\infty]$.

Note that in the structured case we work with, interpolating the results obtained for the boundary cases $p\in\{1,\infty\}$ or $q\in\{1,\infty\}$ gives a bound with polynomial (in the dimensions) multiplicative constants which are much worse than logarithmic constants from Corollary 1.4 below. However, as we shall see in Remark 4.2 below, interpolation techniques work well in the non-structured case.

We start with general theorems valid for the whole range of $p$, $q$. Results which are based on methods working only for specific values of $p$, $q$, but yielding better logarithmic terms, are presented in the next subsection. A brief summary and comparison of all results can be found in Table LABEL:table:summary.

Before stating our main results, we need to introduce additional notation. For a non-empty set $J\subset\{1,\ldots,n\}$, and $1\leq p\leq\infty$, we define

By $\ell_{p}^{J}$ we denote the space $\mathbb{R}^{J}\coloneqq\bigl{\{}(x_{j})_{j\in J}:x_{j}\in\mathbb{R}\bigr{\}}$ equipped with the norm

whose unit ball is $B_{p}^{J}$. Obviously, the space $\ell_{p}^{J}$ can be identified with a subspace of $\ell_{p}^{n}$. If $A\colon\ell_{p}^{n}\to\ell_{q}^{m}$ is a linear operator, the notation $A\colon\ell_{p}^{J}\to\ell_{q}^{I}$ means that $A$ is restricted to the space $\ell_{p}^{J}$ and composed with a projection onto $\ell_{q}^{I}$. Moreover, for $x=(x_{1},\ldots,x_{n})\in\mathbb{R}^{n}$, $\sup_{J}\|x\|_{\ell_{p}^{J}}=\bigl{(}\sum_{j\leq k}|x_{j}^{\downarrow{}}|^{p}\bigr{)}^{1/p}$, where the supremum is taken over all $J\subset\{1,\ldots,n\}$ with $|J|=k$, and $(x_{1}^{\downarrow{}},\ldots,x_{n}^{\downarrow{}})$ is the non-increasing rearrangement of $(|x_{1}|,\ldots,|x_{n}|)$.

The above theorem gives an estimate on the largest operator norm among all submatrices of $G_{A}$ of fixed size. Let us remark that apart from being of intrinsic interest, quantities of this type (for $p=q=2$) have appeared in connection with the study of the restricted isometry property of random matrices with independent rows [2] or in the analysis of entropic uncertainty principles for random quantum measurements [3, 47].

Let us now give an outline of the proof of Theorem 1.3. Note that

In the first step of our proof, we find polytopes $L$ and $K$ approximating (with accuracy depending logarithmically on the dimension) the unit balls in $\ell_{p}^{J_{0}}$ and $\ell_{q^{\ast}}^{I_{0}}$, respectively. The extreme points of the sets $K$ and $L$ have a special and simple structure: absolute values of their non-zero coordinates are all equal to a constant depending only on the size of the support of a given point. Since $K$ is close to $B_{q^{*}}^{I_{0}}$ and $L$ is close to $B_{p}^{J_{0}}$, we may consider only $x\in\operatorname{Ext}(L),y\in\operatorname{Ext}(K)$ in (1.16). Since non-zero coordinates of $x\in\operatorname{Ext}(L)$ and $y\in\operatorname{Ext}(K)$, respectively, are all equal up to a sign we may use a symmetrization argument and the contraction principle to remove $x$ and $y$ in the sum on the right-hand side of (1.16). Thus, in the next step of the proof we only need to estimate the expected value of

where $I$ and $J$ represent the potential supports of points in $\operatorname{Ext}(K)$ and $\operatorname{Ext}(L)$. To deal with this quantity, we first consider the suprema over the subsets of fixed sizes and use Slepian’s lemma to compare the supremum of the Gaussian process above with the supremum of another Gaussian process, which may be bounded easily. Then we make use of the term $|I|^{-1/q^{*}}|J|^{-1/p}<1$, which allows us to go back to suprema over the sets $B_{p}^{J_{0}}$ and $B_{q^{*}}^{I_{0}}$. At the end, we use the Gaussian concentration inequality to unfix the sizes of sets $I$ and $J$ and complete the proof.

Applying Theorem 1.3 with $N=n$, $M=m$ immediately yields the following result, which confirms Conjecture 1 up to some logarithmic terms.

Moreover, we easily recover the same bound in the case of independent bounded entries. We state and prove a general version with sets $I_{0}$ and $J_{0}$ akin to Theorem 1.3 in Subsection 3.2.

We use the two results above to obtain their analogue in the case of $\psi_{r}$ entries for $r\leq 2$; these random variables are defined by (1.17). This class contains, among others,

• •

  log-concave random variables (which are $\psi_{1}$),

• •

  heavy tailed Weibull random variables (of shape parameter $r\in(0,1)$, i.e., $\mathbb{P}(|X_{ij}|\geq t)=e^{-t^{r}/L}$ for $t\geq 0$),

• •

  random variables satisfying the condition

  $$\|X_{ij}\|_{2\rho}\leq\alpha\|X_{ij}\|_{\rho}\qquad\text{for all }\rho\geq 1.$$

  These random variables are $\psi_{r}$ with $r=1/\log_{2}\alpha$. They were considered recently in [38].

A general version of the following Corollary 1.6 with sets $I_{0}$ and $J_{0}$ is stated and proved in Subsection 3.2.