Convergence of de Finetti’s mixing measure in
latent structure models for observed exchangeable sequences

Latent structure models with many observed variables are among the most powerful and widely used tools in statistics for learning about heterogeneity within data population(s). An important canonical example of such models is the mixture of product distributions, which may be motivated by de Finetti’s celebrated theorem for exchangeable sequences of random variables [1, 29]. The theorem of de Finetti states roughly that if $X_{1},X_{2},\ldots$ is an infinite exchangeable sequence of random variables defined in a measure space $(\mathfrak{X},\mathcal{A})$, then there exists a random variable $\theta$ in some space $\Theta$, where $\theta$ is distributed according to a probability measure $G$, such that $X_{1},X_{2},\ldots$ are conditionally i.i.d. given $\theta$. Denote by $P_{\theta}$ the conditional distribution of $X_{i}$ given $\theta$, we may express the joint distribution of a ${N}$-sequence $X_{[{N}]}:=(X_{1},\ldots,X_{N})$, for any ${N}\geq 1$, as a mixture of product distributions in the following sense: for any $A_{1},\ldots,A_{N}\subset\mathcal{A}$,

$$P(X_{1}\in A_{1},\ldots,X_{N}\in A_{N})=\int\prod_{n=1}^{{N}}P_{\theta}(X_{n}\in A_{n})G(d\theta).$$

The probability measure $G$ is also known as de Finetti mixing measure for the exchangeable sequence. It captures the uncertainty about the latent variable $\theta$, which describes the mechanism according to which the sequence $(X_{i})_{i}$ is generated via $P_{\theta}$. In other words, the de Finetti mixing measure $G$ can be seen as representing the heterogeneity within the data populations observed via sequences $X_{[{N}]}$. A statistician typically makes some assumption about the family $\{P_{\theta}\}_{\theta\in\Theta}$, and proceeds to draw inference about the nature of heterogeneity represented by $G$ based on data samples $X_{[{N}]}$.

In order to obtain an estimate of the mixing measure $G$, one needs multiple copies of the exchangeable sequences $X_{[{N}]}$. As mentioned, some assumption will be required of the probability distributions $P_{\theta}$, as well as the mixing measure $G$. Throughout this paper it is assumed that the map $\theta\mapsto P_{\theta}$ is injective. Moreover, we will confine ourselves to the setting of exact-fitted finite mixtures, i.e., $G$ is assumed to be an element of $\mathcal{E}_{k}(\Theta)$, the space of discrete measures with $k$ distinct supporting atoms on $\Theta$, where $\Theta$ is a subset of $\mathbb{R}^{q}$. Accordingly, we may express $G=\sum_{j=1}^{k}p_{j}\delta_{\theta_{j}}$. We may write the distribution for $X_{[{N}]}$ in the following form, where we include the subscripts $G$ and ${N}$ to signify their roles:

$$P_{G,{N}}(X_{1}\in A_{1},\ldots,X_{{N}}\in A_{{N}})=\sum_{j=1}^{k}p_{j}\biggr{\{}\prod_{n=1}^{{N}}P_{\theta_{j}}(X_{n}\in A_{n})\biggr{\}}.$$

(1)

Note that when ${N}=1$, we are reduced to a mixture distribution $P_{G}:=P_{G,1}=\sum_{j=1}^{k}p_{j}P_{\theta_{j}}$. Due to the role they play in the composition of the distribution $P_{G,{N}}$, we also refer to $\{P_{\theta}\}_{\theta\in\Theta}$ as a family of probability kernels on $\mathfrak{X}$. Given ${m}$ independent copies of exchangeable sequences $\{X_{[{N}_{i}]}^{i}\}_{i=1}^{{m}}$ each of which is respectively distributed according to $P_{G,{N}_{i}}$ given in (1), where ${N}_{i}$ denotes the possibly variable length of the $i$-th sequence. The primary question of interest in this paper is the efficiency of the estimation of the true mixing measure $G=G_{0}\in\mathcal{E}_{k}(\Theta)$, for some known $k=k_{0}$, as sample size $(m,{N}_{1},\ldots,{N}_{m})$ increases in a certain sense.

Models described by Eq. (1) are also known in the literature as mixtures of repeated measurements, or mixtures of grouped observations [23, 12, 10, 28, 44, 36], with applications to domains such as psychological analysis, educational assessment, and topic modeling in machine learning. The random effects model described in Section 1.3.3 of [30] in which the mixing measure is a discrete measure with finite number of atoms is also a special case of (1) with $P_{\theta}$ a normal distribution with mean $\theta$. While [23, 10] consider the case that the number of components $k$ is unknown, [12, 28, 44, 36] focus on the case that $k$ is known, the same as our set up. In many of the aforementioned works the models are nonparametric, i.e., no parametric forms for the probability kernels are assumed, and the focus is on the problem of density estimation due to the nonparametric setup. By contrast, in this paper we study mixture of product distributions (1) with the parametric form of component distribution imposed, since in practice prior knowledge on the component distribution $P_{\theta}$ might be available. Moreover, we investigate the behavior of parameter estimates — the convergence of the parameters $p_{j}$ and $\theta_{j}$, which are generally more challenging than density estimation in mixture models [32, 26, 25, 27].

Before the efficiency question can be addressed, one must consider the issue of identifiability: under what conditions does the data distribution $P_{G,{N}}$ uniquely identify the true mixing measure $G_{0}$? This question has occupied the interest of a number of authors  [43, 11, 20], with decisive results obtained recently by [3] on finite mixture models for conditionally independent observations and by [44] on finite mixtures for conditionally i.i.d. observations (given by Eq. (1)). Here, the condition is in the form of ${N}\geq n_{0}$, for some natural constant $n_{0}\geq 1$ possibly depending on $G_{0}$. We shall refer to $n_{0}$ as (minimal) zero-order identifiable length or 0-identifiable length for short (a formal definition will be given later). For the conditionally i.i.d. case as in model (1), [44] proves that as long as $N\geq 2k-1$, model (1) will be identifiable for any $G_{0}$. Note that $2k-1$ is only an upper bound of $n_{0}$. For a given parametric form of $\{P_{\theta}\}_{\theta\in\Theta}$ and a given truth $G_{0}$, the $0$-identifiable length might be smaller than $2k-1$.

Drawing from existing identifiability results, it is quite apparent that the observed sequence length ${N}$ (or more precisely, ${N}_{1},\ldots,{N}_{m}$, in case of variable length sequences) must play a crucial role in the estimation of mixing measure $G$, in addition to the number ${m}$ of sequences. Moreover, it is also quite clear that in order to have a consistent estimate of $G=G_{0}$, the number of sequences ${m}$ must tend to infinity, whereas ${N}$ may be allowed to be fixed. It remains an open question as to the precise roles ${m}$ and ${N}$ play in estimating $G$ and on the different types of mixing parameters: the component parameters (atoms $\theta_{j}$) and mixing proportions (probability mass $p_{j}$), and the rates of convergence of a given estimation procedure.

Partial answers to this question were obtained in several settings of mixtures of product distributions.  [23] proposed to discretize data so that the model in consideration becomes a finite mixture of product of identical binomial or multinomial distributions. Restricting to this class of models, a maximum likelihood estimator was applied, and a standard asymptotic analysis establishes root-${m}$ rate for mixing proportion estimates.  [21, 20] investigated a number of nonparametric estimators for $G$, and obtained the root-${m}$ convergence rate for both mixing proportion and component parameters in the setting of $k=2$ mixture components under suitable identifiability conditions. It seems challenging to extend their method and theory to a more general setting, e.g., $k>2$. Moreover, no result on the effect of ${N}$ on parameter estimation efficiency seems to be available. Recently,  [34, 33] studied the posterior contraction behavior of several classes of Bayesian hierarchical model where the sample is also specified by ${m}$ sequences of ${N}$ observations. His approach requires that both ${m}$ and ${N}$ tend to infinity and thus cannot be applied to our present setting where ${N}$ may be fixed.

In this paper we shall present a parameter estimation theory for general classes of finite mixtures of product distributions. An application of this theory will be posterior contraction theorems established for a standard Bayesian estimation procedure, according to which the de Finetti’s mixing measure $G$ tends toward the truth $G_{0}$, as ${m}$ tends to infinity, under suitable conditions. In a standard Bayesian procedure, the statistician endows the space of parameters $\mathcal{E}_{k_{0}}(\Theta)$ with a prior distribution $\Pi$, which is assumed to have compact support in these theorems, and applies Bayes’ rule to obtain the posterior distribution on $\mathcal{E}_{k_{0}}(\Theta)$, to be denoted by $\Pi(G|\{X_{[{N}_{i}]}^{i}\}_{i=1}^{{m}})$. To anticipate the distinct convergence behaviors for the atoms and probability mass parameters, for any $G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}},G^{\prime}=\sum_{i=1}^{k}p^{\prime}_{i}\delta_{\theta^{\prime}_{i}}\in\mathcal{E}_{k}(\Theta)$, define

$$D_{{N}}(G,G^{\prime})=\min_{\tau\in S_{k}}\sum_{i=1}^{k}(\sqrt{{N}}\|\theta_{\tau(i)}-\theta^{\prime}_{i}\|_{2}+|p_{\tau(i)}-p^{\prime}_{i}|),$$

where $S_{k}$ denotes all the permutations on the set $[k]:=\{1,2,\ldots,k\}$. (The suitability of $D_{N}$ over other choices of metric will be discussed in Section 3).

Given ${m}$ independent exchangeable sequences denoted by $\{X^{i}_{[{N}_{i}]}\}_{i=1}^{{m}}$. We naturally require that $\min_{i}{N}_{i}\geq n_{0}$, where $n_{0}$ is the zero-order identifiable length depending on $G_{0}$. Moreover, to obtain concrete rates of convergence, we need also $\min_{i}{N}_{i}\geq n_{1}$ for some minimal natural number $n_{1}:=n_{1}(G_{0})\geq 1$. We shall call $n_{1}$ the minimal first-order identifiable length depending on $G_{0}$, or 1-identifiable length for short (a formal definition will be given later). Assume that $\{{N}_{i}\}_{i=1}^{{m}}$ are uniformly bounded from above by an arbitrary unknown constant, in Theorem 6.2 it is established that under suitable regularity conditions on $P_{\theta}$, the posterior contraction rate for the mixing proportions is bounded above by ${m}^{-1/2}$, up to a logarithmic quantity. For mixture components’ supporting atoms, the contraction rate is

$$O_{P}\biggr{(}\sqrt{\frac{\ln(\sum_{i=1}^{{m}}{N}_{i})}{\sum_{i=1}^{{m}}{N}_{i}}}\biggr{)}.$$

Note that $\sum_{i=1}^{{m}}{N}_{i}$ represents the full volume of the observed data set. More precisely, for suitable kernel families $P_{\theta}$, as long as $\min_{i}{N}_{i}\geq\max\{n_{0},n_{1}\}$ and $\sup_{i}{N}_{i}<\infty$, there holds

$\displaystyle\Pi\biggr{(}G\in\mathcal{E}_{k_{0}}(\Theta):D_{\sum_{i=1}^{{m}}{N}_{i}/{m}}(G,G_{0})\leq C(G_{0})\bar{M}_{m}\sqrt{\frac{\ln(\sum_{i=1}^{{m}}{N}_{i})}{{m}}}\biggr{|}X_{[{N}_{1}]}^{1},\ldots,X_{[{N}_{{m}}]}^{{m}}\biggr{)}\to 1$

in $P_{G_{0},{N}_{1}}\otimes\cdots\otimes P_{G_{0},{N}_{{m}}}$-probability as ${m}\to\infty$ for any sequence $\bar{M}_{m}\to\infty$. The point here is that constant $C(G_{0})$ is independent of ${m}$, sequence lengths $\{{N}_{i}\}_{i=1}^{{m}}$ and their supremum. In plain terms, we may say that with finite mixtures of product distributions, the posterior inference of atoms of each individual mixture component receives the full benefit of "borrowing strength" across sampled sequences; while the mixing probabilities gain efficiency from only the number of such sequences. This appears to be the first work in which such a posterior contraction theorem is established for de Finetti mixing measure arising from finite mixtures of product distributions.

The Bayesian learning rates established appear intuitive, given the parameter space $\Theta\in\mathbb{R}^{q}$ is of finite dimension. On the role of ${m}$, they are somewhat compatible to the previous partial results [23, 21, 20]. However, we wish to make several brief remarks at this juncture.

• •

  First, even for exact-fitted parametric mixture models, "parametric-like" learning rates of the form root-${m}$ or root-$({m}{N})$ should not to be taken for granted, because they do not always hold [25, 27]. This is due to the fact that the kernel family $\{P_{\theta}\}_{\theta\in\Theta}$ may easily violate assumptions of strong identifiability often required for the root-${m}$ rate to take place. In other words, the kernel family $\{P_{\theta}\}$ may be only weakly identifiable, resulting in poor learning rates for a standard mixture, i.e., when ${N}=1$.

• •

  Second, the fact that by increasing the observed exchangeable sequence’s length ${N}$ so that ${N}\geq n_{1}\vee n_{0}$, one may obtain parametric-like learning rates in terms of both ${N}$ and ${m}$ is a remarkable testament of how repeated measurements can help to completely overcome a latent variable model’s potential pathologies: parameter non-identifiability is overcome by making ${N}\geq n_{0}$, while inefficiency of parameter estimation inherent in weakly identifiable mixture models is overcome by ${N}\geq n_{1}$. For a deeper appreciation of this issue, see Section 2 for a background on the role of identifiability notions in parameter estimation.

Although the posterior contraction theorems for finite mixtures of product distributions presented in this paper are new, such results do not adequately capture the rather complex behavior of the convergence of parameters for a finite mixture of ${N}$-product distributions. In fact, the heart of the matter lies in the establishment of a collection of general inverse bounds, i.e., inequalities of the form

$$D_{{N}}(G,G_{0})\leq C(G_{0})V(P_{G,{N}},P_{G_{0},{N}}),$$

(2)

where $V(\cdot,\cdot)$ is the variational distance. Note that (2) provides an upper bound on distance $D_{N}$ of mixing measures in terms of the variational distance between the corresponding mixture of ${N}$-product distributions. Inequalities of this type allow one to transfer the convergence (and learning rates) of a data population’s distribution into that of the corresponding distribution’s parameters (therefore the term "inverse bounds"). Several points to highlight are:

• •

  The local nature of (2), which may hold only for $G$ residing in a suitably small $D_{{N}}$-neighborhood of $G_{0}$ whose radius may also depend on $G_{0}$ and ${N}$, while constant $C(G_{0})>0$ depends on $G_{0}$ but is independent of ${N}$. In addition, the bound holds only when ${N}$ exceeds threshold $n_{1}\geq 1$, unless further assumptions are imposed. For instance, under a first-order identifiability condition of $P_{\theta}$, $n_{1}=1$, so this bound holds for all ${N}\geq 1$ while remaining local in nature. Moreover, inequality (2) is sharp: the quantity ${N}$ in $D_{N}$ cannot be improved by $D_{\psi({N})}$ for any sequence $\psi({N})$ such that $\psi({N})/{N}\rightarrow\infty$ (see Lemma 8.3).

• •

  The inverse bounds of the form (2) are established without any overt assumption of identifiability. However, they carry striking consequences on both first-order and classical identifiability, which can be deduced from (2) under a compactness condition (see Proposition 5.1): using the notation $n_{0}(G,\cup_{k\leq k_{0}}\mathcal{E}_{k}(\Theta_{1}))$ and $n_{1}(G,\mathcal{E}_{2k_{0}}(\Theta_{1}))$ to denote explicitly the dependence of 0- and 1-identifiable lengths on $G$ in the first argument and its ambient space in the second argument, respectively, we have

  $$\sup_{G\in\cup_{k\leq k_{0}}\mathcal{E}_{k}(\Theta_{1})}n_{0}(G,\cup_{k\leq k_{0}}\mathcal{E}_{k}(\Theta_{1}))\leq\sup_{G\in\mathcal{E}_{2k_{0}}(\Theta_{1})}n_{1}(G,\mathcal{E}_{2k_{0}}(\Theta_{1}))<\infty.$$

  Note that classical identifiability captured by $n_{0}(G,\cup_{k\leq k_{0}}\mathcal{E}_{k}(\Theta_{1}))$ describes a global property of the model family while first-order identifiability captured by $n_{1}(G,\mathcal{E}_{2k_{0}}(\Theta_{1}))$ is local in nature. The connection between these two concepts is possible because when the number of exchangeable variables ${N}$ gets large, the force of the central limit theorem for product distributions comes into effect to make the mixture model eventually become identifiable, either in the classical or the first-order sense, even if the model may be initially non-identifiable or weakly identifiable (when ${N}=1$).

• •

  These inverse bounds hold for very broad classes of probability kernels $\{P_{\theta}\}_{\theta\in\Theta}$. In particular, they are established under mild regularity assumptions on the family of probability kernel $P_{\theta}$ on $\mathfrak{X}$, when either $\mathfrak{X}=\mathbb{R}^{d}$, or $\mathfrak{X}$ is a finite set, or $\mathfrak{X}$ is an abstract space. A standard but non-trivial example of our theory is the case the kernels $P_{\theta}$ belong to the exponential families of distributions. A more unusual example is the case where $P_{\theta}$ is itself a mixture distribution on $\mathfrak{X}$. Kernels of this type are rarely examined in theory, partly because when we set ${N}=1$ a mixture model using such kernels typically would not be parameter-identifiable. However, such "mixture-distribution" kernels are frequently employed by practitioners of hierarchical models (i.e., mixtures of mixture distributions). As the inverse bounds entail, this makes sense since the parameters become more strongly identifiable and efficiently estimable with repeated exchangeable measurements.

• •

  More generally, inverse bounds hold when $P_{\theta}$ does not necessarily admit a density with respect to a dominating measure on $\mathfrak{X}$. An example considered in the paper is the case $P_{\theta}$ represents probability distribution on the space of probability distributions, namely, $P_{\theta}$ represents (mixtures of) Dirichlet processes. As such, the general inverse bounds are expected to be useful for models with nonparametric mixture components represented by $P_{\theta}$, the kind of models that have attracted much recent attention, e.g., [41, 37, 8, 7].

The above highlights should make clear the central roles of the inverse bounds obtained in Section 4 and Section 5, which deepen our understanding of the questions of parameter identifiability and provide detailed information about the convergence behavior of parameter estimation. In addition to an asymptotic analysis of Bayesian estimation for mixtures of product distributions that will be carried out in this paper, such inverse bounds may also be useful for deriving rates of convergence for non-Bayesian parameter estimation procedures, including maximum likelihood estimation and distance based estimation methods.

The rest of the paper will proceed as follows. Section 2 presents related work in the literature and a high-level overview of our approach and techniques. Section 3 prepares the reader with basic setups and several useful concepts of distances on space of mixing measures that arise in mixtures of product distributions. Section 4 is a self-contained treatment of first-order identifiability theory for finite mixture models, leading to several new results that are useful for subsequent developments. Section 5 presents inverse bounds for broad classes of finite mixtures of product distributions, along with specific examples. An immediate application of these bounds are posterior contraction theorems for de Finetti’s mixing measures, the main focus of Section 6. Particular examples of interest for the inverse bounds established in Section 5 include the case the probability kernel $P_{\theta}$ is itself a mixture distribution on $\mathfrak{X}=\mathbb{R}$, and the case $P_{\theta}$ is a mixture of Dirichlet processes. These examples require development of new tools and are deferred to Section 7. Section 8 gives several technical results demonstrating the sharpness of the established inverse bounds, which is then used to derive minimax lower bounds for estimation procedures of de Finetti’s mixing parameters. Section 9 discusses extensions and several future directions. Finally, (most) proofs of all theorems and lemmas will be provided in the Appendix.

We start by setting up basic notions required for the analysis of mixtures of product distributions. Given exchangeable data sequences denoted by $X^{i}_{[{N}_{i}]}:=(X_{1}^{i},\ldots,X_{{N}_{i}}^{i})$ for $i=1,\ldots,{m}$, while ${N}_{i}$ denotes the length of sequence $X_{[{N}_{i}]}^{i}$. For ease of presentation, for now, we shall assume that $N_{i}=N$ for all $i$. Later we will allow variable length sequences. These sequences are composed of elements in a measurable space $(\mathfrak{X},\mathcal{A})$. Examples include $\mathfrak{X}=\mathbb{R}^{d}$, $\mathfrak{X}$ is a discrete space, and $\mathfrak{X}$ is a space of measures. Regardless, parameters of interest are always encapsulated by discrete mixing measures $G\in\mathcal{E}_{k}(\Theta)$, the space of discrete measures with $k$ distinct support atoms residing in $\Theta\subset\mathbb{R}^{q}$.

The linkage between parameters of interest, i.e., the mixing measure $G$, and the observed data sequences is achieved via the mixture of product distributions that we now define. Consider a family of probability distributions $\{P_{\theta}\}_{\theta\in\Theta}$ on measurable space $(\mathfrak{X},\mathcal{A})$, where $\theta$ is the parameter of the family and $\Theta\subset\mathbb{R}^{q}$ is the parameter space. Throughout this paper it is assumed that the map $\theta\mapsto P_{\theta}$ is injective. For $N\in\mathbb{N}$, the ${N}$-product probability family is denoted by $\{P_{\theta,{N}}:=\bigotimes^{{N}}P_{\theta}\}_{\theta\in\Theta}$ on $(\mathfrak{X}^{{N}},\mathcal{A}^{{N}})$, where $\mathcal{A}^{{N}}$ is the product sigma-algebra. Given a mixing measure $G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}}\in\mathcal{E}_{k}(\Theta)$, the mixture of ${N}$-product distributions induced by $G$ is given by

$$P_{G,{N}}=\sum_{i=1}^{k}p_{i}P_{\theta_{i},{N}}.$$

Each exchangeable sequence $X^{i}_{[N]}=(X_{1}^{i},\ldots,X_{{N}}^{i})$, for $i=1,\ldots,{m}$, is an independent sample distributed according to $P_{G,{N}}$. Due to the role they play in the composition of distribution $P_{G,{N}}$, we also refer to $\{P_{\theta}\}_{\theta\in\Theta}$ as a family of probability kernels on $(\mathfrak{X},\mathcal{A})$.

In order to quantify the convergence of mixing measures arising in mixture models, an useful device is a suitably defined optimal transport distance [32, 31]. Consider the Wasserstein-$p$ distance w.r.t. distance $d_{\Theta}$ on $\Theta$: $\forall G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}},G^{\prime}=\sum_{i=1}^{k^{\prime}}p^{\prime}_{i}\delta_{\theta^{\prime}_{i}}$, define

$$W_{p}(G,G^{\prime};d_{\Theta})=\left(\min_{\bm{q}}\sum_{i=1}^{k}\sum_{j=1}^{k^{\prime}}q_{ij}d_{\Theta}^{p}(\theta_{i},\theta^{\prime}_{j})\right)^{1/p},$$

(7)

where the infimum is taken over all joint probability distributions $\bm{q}$ on $[k]\times[k^{\prime}]$ such that, when expressing $\bm{q}$ as a $k\times k^{\prime}$ matrix, the marginal constraints hold: $\sum_{j=1}^{k^{\prime}}q_{ij}=p_{i}$ and $\sum_{i=1}^{k}q_{ij}=p^{\prime}_{j}$. For the special case when $d_{\Theta}$ is the Euclidean distance, write simply $W_{p}(G,G^{\prime})$ instead of $W_{p}(G,G^{\prime};d_{\Theta})$. Write $G_{\ell}\overset{W_{p}}{\to}G$ if $G_{\ell}$ converges to $G$ under the $W_{p}$ distance w.r.t. the Euclidean distance on $\Theta$.

For mixing measures arising in mixtures of ${N}$-product distributions, a more useful notion is the following. For any $G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}}\in\mathcal{E}_{k}(\Theta)$ and $G^{\prime}=\sum_{i=1}^{k}p^{\prime}_{i}\delta_{\theta^{\prime}_{i}}\in\mathcal{E}_{k}(\Theta)$, define

$$D_{{N}}(G,G^{\prime})=\min_{\tau\in S_{k}}\sum_{i=1}^{k}(\sqrt{{N}}\|\theta_{\tau(i)}-\theta^{\prime}_{i}\|_{2}+|p_{\tau(i)}-p^{\prime}_{i}|)$$

where $S_{k}$ denote all the permutations on the set $[k]$. It is simple to verify that $D_{{N}}(\cdot,\cdot)$ is a valid metric on $\mathcal{E}_{k}(\Theta)$ for each ${N}$ and relate it to a suitable optimal transport distance metric. Indeed, $G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}}\in\mathcal{E}_{k}(\Theta)$, due to the permutations invariance of its atoms, can be identified as a set $\{(\theta_{i},p_{i}):1\leq i\leq k\}$, which can further be identified as $\tilde{G}=\sum_{i=1}^{k}\frac{1}{k}\delta_{(\theta_{i},p_{i})}\in\mathcal{E}_{k}(\Theta\times\mathbb{R})$. Formally, we define a map $\mathcal{E}_{k}(\Theta)\to\mathcal{E}_{k}(\Theta\times\mathbb{R})$ by

$$G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}}\mapsto\tilde{G}=\sum_{i=1}^{k}\frac{1}{k}\delta_{(\theta_{i},p_{i})}\in\mathcal{E}_{k}(\Theta\times\mathbb{R}).$$

(8)

Now, endow $\Theta\times\mathbb{R}$ with a metric $M_{{N}}$ defined by $M_{{N}}((\theta,p),(\theta^{\prime},p^{\prime}))=\sqrt{{N}}\|\theta-\theta^{\prime}\|_{2}+|p-p^{\prime}|$ and note the following fact.

A proof of the preceding lemma is available as Proposition 2 in [31]. By applying Lemma 3.1 with $\bar{\Theta}$, $d_{\bar{\Theta}}$ replaced respectively by $\Theta\times\mathbb{R}$ and $M_{N}$, then for any $G,G^{\prime}\in\mathcal{E}_{k}(\Theta)$, $W_{1}(\tilde{G},\tilde{G^{\prime}};M_{{N}})=\frac{1}{k}D_{{N}}(G,G^{\prime})$, which validates that $D_{{N}}$ is indeed a metric on $\mathcal{E}_{k}(\Theta)$, and moreover it does not depend on the specific representations of $G$ and $G^{\prime}$.

The next lemma establishes the relationship between $D_{{N}}$ and $W_{1}$ on $\mathcal{E}_{k}(\Theta)$.

We see that $W_{1}$ and $D_{1}$ generate the same topology on $\mathcal{E}_{k}(\Theta)$, and they are equivalent while fixing one argument. The benefit of $W_{p}$ is that it is defined on $\bigcup_{k=1}^{\infty}\mathcal{E}_{k}(\Theta)$ while $D_{N}$ is only defined on $\mathcal{E}_{k}(\Theta)$ for each $k$ since its definition requires the two arguments have the same number of atoms. $D_{N}$ allows us to quantify the distinct convergence behavior for atoms and probability mass, by placing different factors on the atoms and the probability mass parameters, while $W_{p}$ on $\bigcup_{k=1}^{\infty}\mathcal{E}_{k}(\Theta)$ would fail to do so, because $W_{p}$ couples the atoms and probability mass parameters (see Example A.1 for such an attempt).

The factor $\sqrt{N}$ present in the definition of $D_{N}$ arises from the anticipation that when we have independent exchangeable sequences of length $N$, the dependence of the standard estimation rate on $N$ for component parameters $\theta$ will be of order $1/\sqrt{N}$. Indeed, given one single exchangeable sequence from some component parameter $\theta_{i}$, as the coordinates in this sequence are conditionally independent and identically distributed, the standard rate for estimating $\theta_{i}$ is $1/\sqrt{N}$. On the other hand, the mixing proportions parameters $p_{i}$ cannot be estimated from a single such sequence (i.e., if $m=1$). One expects that for such parameters the number of sequences coming from the $\theta_{i}$ among all exchangeable sequences plays a more important role. In summary, the distance $D_{N}$ will be used to capture precisely the distinct convergence behavior due to the length $N$ of observed exchangeable sequences.

Let ${N}=1$, a finite mixture of ${N}$-product distributions is reduced to a standard finite mixture of distributions. Mixture components are modeled by a family of probability kernels $\{P_{\theta}\}_{\theta\in\Theta}$ on $\mathfrak{X}$, where $\theta$ is the parameter of the family and $\Theta\subset\mathbb{R}^{q}$ is the parameter space. As discussed in the introduction, throughout the paper we assume that the map $\theta\mapsto P_{\theta}$ is injective; it is the nature of the map $G\mapsto P_{G}$ that we are after. Within this section, we further assume that $\{P_{\theta}\}_{\theta\in\Theta}$ has density $\{f(x|\theta)\}_{\theta\in\Theta}$ w.r.t. a dominating measure $\mu$ on $(\mathfrak{X},\mathcal{A})$. Combining multiple mixture components using a mixing measure $G$ on $\Theta$ results in the finite mixture distribution, which admits the following density with respect to $\mu$: $p_{G}(x)=\int f(x|\theta)G(d\theta)$. The goal of this section is to provide a concise and self-contained treatment of identifiability of finite mixture models. We lay down basic foundations and present new results that will prove useful for the general theory of mixtures of product distributions to be developed in the subsequent sections.