A probabilistic view on predictive constructions for Bayesian learning

There is another approach to Bayesian prediction, usually called the predictive approach (P.A.), which is quite recurrent in the Bayesian literature and recently gained increasing attention. (Such an approach, incidentally, has been referred to as the “non-standard approach” in [8, 9]). According to P.A., the forecaster directly selects her strategy $\sigma$. Merely, for each $n\geq 0$, she selects the predictive $\sigma_{n}$ without passing through the prior/posterior scheme described above. Among others, P.A. is supported by de Finetti, Savage, Dubins [22, 23, 26] and more recently by Diaconis and Regazzini [4, 16, 24, 25, 31]. P.A. is also strictly connected to Dawid’s prequential approach [19, 20, 21] and to Pitman’s treatment of species sampling sequences [54, 55, 56]. In addition, several prediction procedures arising in non-necessarily Bayesian frameworks, such as Machine Learning and Data Mining, are consistent with P.A.; see e.g. [17, 18, 27, 43]. Some further related references are [8, 9, 29, 30, 32, 40, 41, 44].

The theoretical foundation of P.A. is the Ionescu-Tulcea theorem; see e.g. [46, p. 159]. Roughly speaking this theorem states that, to assign the joint distribution of $X$, it suffices to choose, in an arbitrary way, the marginal distribution of $X_{1}$, the conditional distribution of $X_{2}$ given $X_{1}$, the conditional distribution of $X_{3}$ given $(X_{1},X_{2})$, and so on. Note that this fact would be obvious if $X$ would be replaced by a finite dimensional random vector $(X_{1},\ldots,X_{m})$. So, in a sense, the Ionescu-Tulcea theorem extends to infinite sequences a straightforward property of finite dimensional vectors. In any case, a formal statement of the theorem is as follows.

Because of Theorem 1, to make predictions on the sequence $X$, the forecaster is free to select an arbitrary strategy $\sigma$. In fact, for any $\sigma$, there is a (unique) probability distribution for $X$, denoted above by $P_{\sigma}$, whose predictives $P_{\sigma}\bigl{(}X_{n+1}\in\cdot\mid X_{1},\ldots,X_{n}\bigr{)}$ agree with $\sigma$ in the sense of equation (1).

The strengths and weaknesses of I.A. versus P.A. are discussed in a number of papers; see e.g. [8, 18, 27, 36, 58] and references therein. Here, we summarize this issue (from our point of view) under the assumption that prediction is the main target.

I.A. is not motivated by prediction alone. The main goal of I.A. is to make inference on other features of the data distribution (typically some parameters) and in this case the prior $\pi$ is fundamental. It should be added that $\pi$ often provides various meaningful information on the data generating process. However, to assess $\pi$ is not an easy task. In addition, once $\pi$ is selected, to evaluate the posterior $\pi_{n}(\cdot\mid x)$ is quite difficult as well. Frequently, $\pi_{n}(\cdot\mid x)$ cannot be written in closed form but only approximated numerically. In short, I.A. is a cornerstone of Bayesian inference, but, when prediction is the main target, it is actually quite involved.

In turn, P.A. has essentially four merits. First, P.A. allows to avoid an explicit choice of the prior $\pi$. Indeed, when prediction is the main target, why select $\pi$ explicitly ? Rather than wondering about $\pi$, it seems reasonable to reflect on how the information in $(X_{1},\ldots,X_{n})$ is conveyed in the prediction of $X_{n+1}$. Second, the data sequence $X$ is not required any distributional assumption. This point is developed in Subsections 1.2 and 1.3. By now, we stress a consequence of such a point. The Bayesian nature of a prediction procedure does not depend on the data distribution. For instance, a forecaster applying P.A. is certainly Bayesian independently of the distribution attached to $X$. Third, P.A. requires the assignment of probabilities on observable facts only. The value of $X_{n+1}$ is actually observable, while $\pi$ and $\pi_{n}$ (being probabilities on $\Theta$) do not necessarily deal with observable facts. Fourth, the strategy $\sigma$ may be assigned stepwise. At each time $n$, the forecaster has observed $x=(x_{1},\ldots,x_{n})\in S^{n}$ and has already selected $\sigma_{0},\sigma_{1}(x_{1}),\ldots,\sigma_{n-1}(x_{1},\ldots,x_{n-1})$. Then, to predict $X_{n+1}$, she is still free to select $\sigma_{n}(x)$ as she wants. No choice of $\sigma_{n}(x)$ is precluded. This is consistent with the Bayesian view, where the observed data are fixed and one should condition on them. In spite of these advantages, P.A. has an obvious drawback. In fact, assigning a strategy $\sigma$ directly may be very difficult, in principle as difficult as selecting a prior $\pi$.

A last (basic) remark is that, if $X$ is exchangeable, both I.A. and P.A. completely determine the probability distribution of $X$. Selecting a prior $\pi$ or choosing a strategy $\sigma$ are just equivalent routes to fix the distribution of $X$. In particular, selecting $\sigma$ uniquely determines $\pi$. An intriguing line of research is in fact to identify the prior corresponding to a given $\sigma$; see e.g. [10, 24, 25, 31].

Recall that, for any strategy $\sigma$, there is a unique probability measure $P_{\sigma}$ on $(S^{\infty},\mathcal{B}^{\infty})$ satisfying condition (1).

In principle, when applying P.A., the data sequence $X$ is free to have any probability distribution. Nevertheless, in most applications, it is reasonable (if not mandatory) to impose some conditions on $X$. For instance, the forecaster may wish $X$ to be exchangeable, or stationary, or Markov, or a martingale, and so on. In these cases, $\sigma$ is subjected to some constraints. If $X$ is required to be exchangeable, for instance, $\sigma$ should be such that $P_{\sigma}$ is exchangeable. Hence, those strategies $\sigma$ which make $P_{\sigma}$ exchangeable should be characterized.

More generally, fix any collection $\mathcal{C}$ of probability measures on $(S^{\infty},\mathcal{B}^{\infty})$ and suppose the data distribution is required to belong to $\mathcal{C}$. Then, P.A. gives rise to the following problem:

• Problem (*): Characterize those strategies $\sigma$ such that $P_{\sigma}\in\mathcal{C}$.

Sometimes, Problem (*) is trivial (Markov, martingales) but sometimes it is not (stationarity, exchangeability). To illustrate, we mention three examples (which correspond to the three dependence forms examined in the sequel).

In the exchangeable case, Problem (*) admits a solution [31, Th. 3.1] but the conditions on $\sigma$ are quite hard to check in real problems. Hence, applying P.A. to exchangeable data is usually difficult (even if there are some exceptions; see Section 2).

A condition weaker than exchangeability is conditional identity in distribution. Say that $X$ is conditionally identically distributed (c.i.d.) if $X_{2}\overset{d}{=}X_{1}$ and, for each $n\geq 1$, the conditional distribution of $X_{k}$ given $(X_{1},\ldots,X_{n})$ is the same for all $k>n$; see Section 3. It can be shown that

see [5, 47]. Hence, conditional identity in distribution can be regarded as one of the two basic ingredients of exchangeability (the other being stationarity). Now, in the c.i.d. case, Problem (*) has been solved [6, Th. 3.1] and the conditions on $\sigma$ are quite simple. The class of admissible strategies includes several meaningful elements which cannot be used if $X$ is required to be exchangeable. As a consequence, P.A. works quite well for c.i.d. data; see [8, 9].

The stationary case is more involved. In fact, to our knowledge, there is no general characterization of the strategies $\sigma$ which make $P_{\sigma}$ stationary. However, such a characterization is available in some meaningful special cases (e.g. when $P_{\sigma}$ is also required to be Markov); see Section 4.

Finally, Problem (*) is usually easier in a few (meaningful) special cases. For instance, Problem (*) is simpler if $P_{\sigma}$ is also asked to be Markov; see e.g. [33] and Section 4. Or else, if the strategy $\sigma$ is required to be dominated.

• Dominated strategies: Let $\lambda$ be a $\sigma$-finite measure on $(S,\mathcal{B})$. Say that a strategy $\sigma$ is dominated by $\lambda$ if each $\sigma_{n}(x)$ admits a density $f_{n}(\cdot\mid x)$ with respect to $\lambda$, namely,

  	$$\displaystyle\sigma_{0}(dy)=f_{0}(y)\,\lambda(dy)\quad\quad\text{and}$$
  	$$\displaystyle\sigma_{n}(x,\,dy)=f_{n}(y\mid x)\,\lambda(dy)$$

  for all $n\geq 1$ and $x\in S^{n}$. Here, $f_{0}:S\rightarrow\mathbb{R}^{+}$ and $f_{n}:S\times S^{n}\rightarrow\mathbb{R}^{+}$ are non-negative measurable functions.

For instance, if $S=\mathbb{R}$ and $\sigma_{n}(x)$ is a non-degenerate normal distribution for all $n$ and $x$, then $\sigma$ is dominated by $\lambda=\,$Lebesgue measure. Or else, if $S$ is countable, any strategy is dominated by $\lambda=\,$counting measure. Instead, if $S$ is uncountable, a non-dominated strategy is $\sigma_{n}(x_{1},\ldots,x_{n})=\delta_{x_{n}}$ where $\delta_{x_{n}}$ denotes the unit mass at the point $x_{n}$. Another non-dominated strategy is the empirical measure $\sigma_{n}(x_{1},\ldots,x_{n})=(1/n)\,\sum_{i=1}^{n}\delta_{x_{i}}.$

In a sense, dominated strategies play an analogous role to the usual dominated models in parametric statistical inference. The main advantage is that one can use the conditional density $f_{n}(\cdot\mid x)$ instead of the conditional measure $\sigma_{n}(x)$. A related advantage is that, if one fixes $\lambda$ and restricts to strategies dominated by $\lambda$, Problem (*) becomes simpler. However, even in applied data analysis, various familiar strategies are not dominated. In the framework of species sampling sequences, for instance, most strategies are not dominated. Therefore, in this paper, we focus on general strategies while the dominated ones are regarded as an important special case.

This is a review paper on P.A. which also includes some (minor) new results. Our perspective is mainly on the probabilistic aspects of Bayesian predictive constructions. Moreover, we tacitly assume that the major target is to predict future observations (and not to make inference on other random elements, such as random parameters).

Essentially, we aim to achieve three goals. First, we try to put P.A. in the right framework, to provide a unifying view, and to make clear a few misunderstandings. This has been done in the Introduction. Second, in Section 2 and Subsection 3.1, we report some known results. Third, we provide some new strategies and we prove a few related results. The strategies, introduced by means of examples, deal with generalized Pólya urns, random change points, covariates and stationary sequences. The results consist in determining the distribution of the data sequence $X$ under such strategies. To our knowledge, Examples 7, 9, 12, 14 and Theorems 8, 11, 13 are actually new, while Theorem 6 makes precise a claim contained in [29]. Moreover, as far as we know, Section 4 is the first attempt to develop P.A. for stationary data. It provides a brief discussion of Problem (*) and introduces two large classes of stationary sequences.

As already noted, even if $X$ could be potentially given any distribution, in most applications $X$ is required some conditions. There is obviously a number of such conditions. Among them, we decided to focus on exchangeability, stationarity and conditional identity in distribution. This choice seems reasonable to keep the paper focused, but of course it leaves out various interesting conditions, such as partial exchangeability. To write a paper of reasonable length, however, some choice was necessary.

To defend our choice, we note that, in addition to be natural in various practical problems, exchangeability is the usual assumption in Bayesian prediction. Hence, taking exchangeability into account is more or less mandatory. Moreover, since $X$ is exchangeable if and only if it is stationary and c.i.d., the other two conditions can be motivated as the basic components of exchangeability. But there are also other reasons for dealing with them. Stationarity is in fact a routine assumption in the classical treatment of time series, and it is reasonable to consider it from the Bayesian point of view as well. Conditional identity in distribution, even if not that popular, seems to be quite suitable for P.A.; see Section 3.

The rest of the paper is organized in three sections, each concerned with a specific assumption on $X$, plus a final section of open problems. All the proofs are gathered in the Appendix.

We close this Introduction with some further notations.

As usual, $\delta_{u}$ is the unit mass at the point $u$. For each $x\in S^{n}$, where $n$ is a positive integer or $n=\infty$, we denote by $x_{i}$ the $i$-th coordinate of $x$. Moreover, we take $X$ to be the sequence of coordinate random variables on $S^{\infty}$, namely,

From now on, we fix a strategy $\sigma$ and we assume

We write $\nu$ instead of $\sigma_{0}$ (i.e., we let $\sigma_{0}=\nu$). Hence, $\nu$ is a probability measure on $\mathcal{B}$ to be regarded as the distribution of $X_{1}$ under the strategy $\sigma$. Finally, to avoid technicalities, $S$ is assumed to be a Borel subset of a Polish space and $\mathcal{B}$ the Borel $\sigma$-field on $S$.

For $n\geq 1$ and $x=(x_{1},\ldots,x_{n})\in S^{n}$, denote by $k_{n}=k_{n}(x)$ the number of distinct values in the vector $x$ and by $x_{1}^{*},\ldots,x_{k_{n}}^{*}$ such distinct values (in the order that they appear). Say that $X$ is a species sampling sequence if it is exchangeable, $\sigma_{0}=\nu$ is non-atomic, and

where the $p_{j,n}$ are non-negative measurable functions on $S^{n}$ and $q_{n}=1-\sum_{j=1}^{k_{n}}p_{j,n}$. Under this strategy, quoting from [42, p. 253], $X$ can be regarded as: “the sequence of species of individuals in a process of sequential random sampling from some hypothetical infinite population of individuals of various species. The species of the first individual to be observed is assigned a random tag $X_{1}=X_{1}^{*}$ distributed according to $\nu$. Given the tags $X_{1},\ldots,X_{n}$ of the first $n$ individuals observed, it is supposed that the next individual is one of the $j$-th species observed so far with probability $p_{j,n}$, and one of a new species with probability $q_{n}$”.

A nice consequence of the definition is that $p_{j,n}(x)$ depends on $x$ only through the vector $(N_{1,n},\ldots,N_{k_{n},n})$, where

is the number of times that $x_{j}^{*}$ appears in the vector $x$; see [42, 54].

The most popular example of species sampling sequence is probably the two-parameter Poisson-Dirichlet, introduced by Pitman in [53], which corresponds to the weights

where $b$ and $c$ are constants such that: either (i) $0\leq b<1$ and $c>-b$ or (ii) $b<0$ and $c=-m\,b$ for some integer $m\geq 2$. In this model, if $L$ denotes the number of distinct values appearing in the sequence $X$, one obtains $L\overset{a.s.}{=}\infty$ under (i) and $L\overset{a.s.}{=}m$ under (ii). Note also that $X$ reduces to a Dirichlet sequence in the special case $b=0$.

Another example, due to [39], is

where $b>0$ and $c$ is such that $k^{2}+bk+c>0$ for all integers $k>0$. This time, unlike the two-parameter Poisson-Dirichlet, $L$ is a finite but non-degenerate random variable.

In general, to obtain a species sampling sequence, the forecaster needs to select $\nu$ and the weights $p_{j,n}$. While the choice of $\nu$ is free (apart from non-atomicity) the $p_{j,n}$ are subjected to the constraint that $X$ should be exchangeable. (Incidentally, the choice of $p_{j,n}$ is a good example of the difficulty of applying P.A. when $X$ is required to be exchangeable). The usual method to select $p_{j,n}$ involves exchangeable random partitions. Let $\mathbb{N}=\bigl{\{}1,2,\ldots\bigr{\}}$ and let $\Pi$ be a random partition of $\mathbb{N}$. For each $n\geq 1$, call $\Pi_{n}$ the restriction of $\Pi$ to $\{1,\ldots,n\}$, namely, the random partition of $\{1,\ldots,n\}$ whose elements are of the form $\{1,\ldots,n\}\cap A$ for some $A\in\Pi$. Say that $\Pi$ is exchangeable if

for all $n\geq 1$ and all permutations $\varphi$ of $(1,\ldots,n)$, where $\varphi(\Pi_{n})$ denotes the random partition $\varphi(\Pi_{n})=\bigl{\{}\varphi(B):B\in\Pi_{n}\bigr{\}}$. For instance, given any sequence $Y=(Y_{1},Y_{2},\ldots)$ of random variables, define $\Pi$ to be the random partition of $\mathbb{N}$ induced by the equivalence relation $i\sim j$ $\Leftrightarrow$ $Y_{i}=Y_{j}$. Then, $\Pi$ is exchangeable provided $Y$ is exchangeable. Now, the weights $p_{j,n}$ of a species sampling sequence correspond, in a canonical way, to the probability law of an exchangeable partition; see [53, 54]. Hence, choosing the $p_{j,n}$ essentially amounts to choosing an exchangeable partition. We stop here since a detailed discussion of exchangeable partitions is bejond the scopes of this paper. The interested reader is referred to [38, 39, 48, 49, 53, 56] and references therein.

A last remark is that the definition of species sampling sequences can be generalized. In particular, non-atomicity of $\nu$ can be dropped (as in [3] and [13]) and exchangeability can be replaced by some weaker condition (as in [1] and [2]).

In [10], to generalize Dirichlet sequences while preserving their main properties, a class of strategies has been introduced. Among other things, such strategies make $X$ exchangeable.

A kernel $\alpha$ on $(S,\mathcal{B})$ is a collection

such that $\alpha(\cdot\mid x)$ is a probability measure on $\mathcal{B}$, for each $x\in S$, and the map $x\mapsto\alpha(A\mid x)$ is measurable for each $A\in\mathcal{B}$. Sometimes, to make the notation easier, we will write $\alpha_{x}$ instead of $\alpha(\cdot\mid x)$. A straightforward example of kernel is $\alpha_{x}=\delta_{x}$ for each $x\in S$.

Fix a probability measure $\nu$ on $\mathcal{B}$, a constant $c>0$, a kernel $\alpha$ on $(S,\mathcal{B})$, and define the strategy

for all $n\geq 0$ and $x\in S^{n}$. Clearly, $X$ reduces to a Dirichlet sequence if $\alpha=\delta$. In this case, we also say that $X$ is a classical Dirichlet sequence.

If $\alpha$ is an arbitrary kernel, $X$ may fail to be exchangeable. However, a useful sufficient condition for exchangeability is available. In fact, $X$ is exchangeable if $\alpha$ agrees with the conditional distribution for $\nu$ given some sub-$\sigma$-field $\mathcal{G}\subset\mathcal{B}$. For instance, if $\mathcal{G}=\mathcal{B}$, then $\alpha=\delta$ and $X$ is a classical Dirichlet sequence. At the opposite extreme, if $\mathcal{G}$ is the trivial $\sigma$-field, then $\alpha_{x}=\nu$ for all $x\in S$ and $X$ is i.i.d. with common distribution $\nu$. In general, for fixed $\nu$ and $c$, a strategy $\sigma$ which makes $X$ exchangeable can be associated with any sub-$\sigma$-field $\mathcal{G}\subset\mathcal{B}$. It suffices to take $\alpha$ as the conditional distribution for $\nu$ given $\mathcal{G}$.

If $\sigma$ is the strategy (2), in addition to exchangeability, $X$ satisfies various other properties of classical Dirichlet sequences. We refer to [10] for details. Here, we just note that the prior $\pi$ and the posterior $\pi_{n}$ can be explicitly determined. In particular, up to replacing $\delta$ with $\alpha$, the Sethuraman’s representation of $\pi$ (see [57]) is still true. Precisely, $\pi$ is the probability distribution of a random probability measure $\mu$ of the form

where:

• •

  $(Z_{j})$ and $(V_{j})$ are independent sequences of random variables;

• •

  $(Z_{j})$ is i.i.d. with common distribution $\nu$;

• •

  $V_{j}=U_{j}\,\prod_{i=1}^{j-1}(1-U_{i})$ for all $j\geq 1$, where $(U_{i})$ is i.i.d. with common distribution beta$(1,c)$. Namely, $(V_{j})$ has the stick breaking distribution with parameter $c$.

A possible condition for a strategy $\sigma$ is

for all $n\geq 0$, $x\in S^{n}$ and $y\in S$, where $y$ denotes the $(n+1)$-th observation and

Under (4), the predictive $\sigma_{n+1}(x,y)$ is just a recursive update of the previous predictive $\sigma_{n}(x)$ and the last observation $y$. Recursive properties of this type are useful in applications. They have a long history (see e.g. [51, 52, 59]) and have been recently investigated in [41].

For each $n\geq 0$, let $q_{n}:S^{n}\rightarrow[0,1]$ be a measurable function (with $q_{0}$ constant) and $\alpha_{n}$ a kernel on $(S,\mathcal{B})$. Define a strategy $\sigma$ through the recursive equations

for all $n\geq 0$, $x\in S^{n}$ and $y\in S$. Since $\sigma_{n+1}(x,y)$ is a convex combination of the previous predictive $\sigma_{n}(x)$ and the kernel $\alpha_{n}(\cdot\mid y)$, which depends only on $y$, the strategy $\sigma$ satisfies condition (4). The obvious interpretation is that, at time $n+1$, after observing $(x,y)$, the next observation is drawn from $\sigma_{n}(x)$ with probability $q_{n}(x)$ and from $\alpha_{n}(\cdot\mid y)$ with probability $1-q_{n}(x)$.

An example of strategy satisfying equation (5) is Newton’s algorithm [51, 52]. More precisely, Newton’s algorithm aims to estimate the latent distribution in a mixture model rather than to make predictions. However, if reinterpreted as a predictive rule, Newton’s algorithm corresponds to a strategy $\sigma$ and such a $\sigma$ meets equation (5) for a suitable choice of $q_{n}$ and $\alpha_{n}$; see e.g. [35, p. 1095]. Moreover, as shown in [35], $\sigma$ makes $X$ c.i.d.

The strategies satisfying equation (5) are investigated in [9]. Under such strategies, $X$ is usually not exchangeable but it is c.i.d. under some conditions on the kernels $\alpha_{n}$. Precisely, $X$ is c.i.d. if $\alpha_{n}$ is the conditional distribution for $\nu$ given $\mathcal{G}_{n}$ for each $n\geq 0$, where

is any filtration (i.e., any increasing sequence of sub-$\sigma$-fields of $\mathcal{B}$). This condition is trivially true if $\alpha_{n}(\cdot\mid y)=\delta_{y}$ for all $y\in S$ (just take $\mathcal{G}_{n}=\mathcal{B}$ for all $n\geq 0$).

We next turn to a strategy introduced in [41]. Once again, under this strategy, the data are c.i.d. but not necessarily exchangeable.