Similar to multi-task learning, negative transfer draws its roots from transfer learning (Pan & Yang, 2009). A widely accepted description of negative transfer is stated as “transferring knowledge from the source can have a negative impact on the target learner”. In an $\mathcal{MGP}$, negative transfer could be defined similarly: the integrative analysis of all outputs can have negative impact on the performance of the model compared with separate modeling of each output or a subset of them.

Definition 1 implies that negative transfer happens when using $D_{\mathcal{I}}$ leads to worse accuracy compared to using a subset of the data or just an individual $\mathcal{GP}$. Therefore, one can provide a model flexible enough to avoid negative transfer if there exists an $\mathcal{MGP}$ such that $\forall\>\bm{x}_{0}\in\mathcal{R}^{\mathcal{D}}$

One can think of $\mathcal{I}_{m}$ as the index for the subset of outputs that share commonalities with $y_{i}$ ($\mathcal{I}_{m}$ here includes $i$). For instance, if $y_{i}$ shares no commonalities with any other output ($\mathcal{I}_{m}=i$) then the $\mathcal{MGP}$ should be able to have $pr(y_{i}(\bm{x}_{0})|D_{\mathcal{I}})=pr(y_{i}(\bm{x}_{0})|D_{i})$ for all $\bm{x}_{0}$, i.e., the conditional predictive distribution in (2) for output $i$ is independent of all other outputs. In other words, we need an $\mathcal{MGP}$ that is able to collapse to independent $\mathcal{GP}s$ or an $\mathcal{MGP}$ with only related outputs.

Building a model that can achieve (4) is a challenging task since, following the conditional predictive distribution in (2), (4) occurs if and only if (Whittaker, 2009) $\forall\>\bm{x},\bm{x}^{\prime}\in\mathcal{R}^{\mathcal{D}}$

In the following section we study the necessary condition for the $\mathcal{MGP}$ to achieve (5). Specifically we show that if $M$ is known (i.e., we know how many distinct/unrelated subgroups of outputs exist) then to achieve (5) the necessary condition is that $Q\geq M$. However the fact that $M$ is not known before hand implies that we need $Q\geq N$.

We first provide a lemma based on the $\mathcal{CP}$ covariance in (3) needed to establish our result.

The technical details for Lemma 1 are given in the Appendix A. Clearly when using one latent function $X_{1}$ if one of the kernels is identically zero then the $\mathcal{MGP}$ is invalid ($\mbox{cov}^{f}_{uu}(\bm{x},{\bm{x}}^{\prime})=0\>\forall\>\bm{x},u\in\{1,2\}$). On the other hand, if we use $Q\geq 2$ latent functions, then the model has enough flexibility to construct $f_{i}$ from different latent functions, i.e. $f_{1}=K_{1}\star X_{1}$ and $f_{2}=K_{2}\star X_{2}$. In this case, $\mbox{cov}^{f}_{12}=0,\ \forall\>\bm{x},{\bm{x}}^{\prime}$. Hence, Lemma 1 implies that only if $Q\geq 2$ we can achieve $\mbox{cov}^{f}_{12}(\bm{x},{\bm{x}}^{\prime})=0\>\forall\>\bm{x},{\bm{x}}^{\prime}$.

In Lemma 1 the assumption that kernels belong to the $L^{1}$ space is also needed for a stable $\mathcal{CP}$ construction in (1). Here we note that despite the fact that the conditions presented satisfy most (if not all) of the kernels currently used in the $\mathcal{CP}$, in the appendix we also provide some simple means to check the conditions. For instance, for any even kernel $K(\bm{x})=K(-\bm{x})$, then $\mbox{cov}^{f}_{12}(\bm{x},{\bm{x}}^{\prime})=K_{11}\star K_{12}$. Hence, since the Fourier operator $\mathcal{F}$ is injective, it suffices to show that $\mathcal{F}\{K_{11}\}(\xi)\cdot\mathcal{F}\{K_{12}\}(\xi)=0$ if and only if $\mathcal{F}\{K_{11}\}$ or $\mathcal{F}\{K_{12}\}$ is identically zero.

We now give the main theorem for the necessary condition to avoid negative transfer.

The technical details for Theorem 2 are given in the Appendix B. It is crucial to note here that in reality we do not know $M$, i.e., we do not know how many distinct subgroups that are uncorrelated exist. Thus, in order to guarantee that negative transfer can be avoided we need $Q\geq M=N$. This also implies that the model is flexible enough to collapse to $N$ independent $\mathcal{GP}$s and hence predict each output independently. An illustrative example is also given in Appendix C.

Despite the many works in the previous decade on reducing the computational complexity of both the $\mathcal{GP}$ and $\mathcal{MGP}$, the results in Sec.3.2 induce another key challenge for $\mathcal{MGP}$: the high dimensional parameter space. This challenge is inherited from the $\mathcal{CP}$ construction which provides different covariance parameters (via the kernels) to different outputs levels. For instance, assume any kernel $K_{qi}(\bm{x})$ has $\omega$ parameters to be estimated then using the $\mathcal{CP}$, this implies estimating $QN\omega+N$ parameters, where the added $N$ parameters are for $\epsilon_{i}(\bm{x})$. Following our results, a model that can avoid negative transfer thus needs at least $N^{2}\omega+N$. Note here that $\omega$ also increases with $\mathcal{D}$, i.e., the dimension of $\bm{x}$. Obtaining good estimates in such a high dimensional space is an impractical task specifically under a non-convex and highly nonlinear objective, be it the exact Gaussian likelihood or its variational bound. Indeed, it is crucial to note that computational complexity and parameter space are two separate challenges and the many papers that tackle the former still suffer from the latter challenge. We conjecture that for this reason, most (if not all) $\mathcal{MGP}$ literature (including all aforementioned cited papers) have used $1\leq Q\leq 4$.

To address this challenge, in Sec. 4 we provide relaxation models that can significantly reduce the parameters space and scale to arbitrarily large datasets by parallelization. Further, our proposed relaxations allow any sparse approximation to be directly plugged in. This in turns allows utilization of the many advances in reducing the computational complexity (inducing point, state space approximation, etc..).

The idea of an arrowhead model is originated from the arrowhead matrix. While still using $N$ latent functions, we can assume that all outputs $y_{i}$, $i\in\mathcal{I}_{/1}$, are independent and only share information with $y_{1}$, the output of interest. This implies that $\mbox{cov}^{f}_{ij}(\bm{x},{\bm{x}}^{\prime})=0,\forall\>i,j\in\mathcal{I}_{/1}$. The structure and covariance matrix are highlighted in Fig. 1(a) and (6) respectively. As shown in the figure, $y_{1}$ possesses unique features encoded in $X_{1}$ and shared features with other outputs encoded in $X_{i},i\in\mathcal{I}_{/1}$.

The arrowhead structure in fact poses many unique advantages: (1) Linear increase of parameter space dimension with $N$: The number of parameters to be estimated is reduced to $(2N-1)\omega+N$. (2) provides enough flexibility to achieve (5) and hence avoid negative transfer. For instance, if $\mbox{cov}^{f}_{1i}(\bm{x},{\bm{x}}^{\prime})=0\>\forall\bm{x}$ and $i\in\mathcal{I}_{/1}$ then outputs $y_{1}$ and $y_{i}$ are predicted independently. (3) can be parallelized, where independent arrowhead models are build to predict each output. (4) One nice interpretation of the arrowhead structure is through a Gaussian directed acyclic graphical model (DAG) with vertices $V=\{\bm{y}_{i}:i\in I\}$. Unlike typical DAGs, each vertex in this graph is in itself a fully connected undirected Gaussian graphical model, i.e. a functional response. This is shown in Fig.1(b). Based on this, the full likelihood factorizes over parent nodes. To see this, let $\mathcal{L}(\bm{\theta};\bm{y})$ denote the likelihood of the dataset, where $\bm{\theta}=\{\bm{\theta}^{\top}_{f},\bm{\sigma}^{\top}\}^{\top}$, such that $\bm{\theta}_{f}$ and $\bm{\sigma}$ are kernel and noise parameters. Then, $\mathcal{L}(\bm{\theta};\bm{y})=\mathcal{L}^{1|i\in\mathcal{I}_{/1}}(\bm{\theta};\bm{y}_{1}|\bm{y}_{2},\cdots,\bm{y}_{N})\prod_{i\in\mathcal{I}_{/1}}\mathcal{L}^{i}(\bm{\theta};\bm{y}_{i})$. This reduces the complexity of exact inference to $O(Np^{3})$ assuming $p_{i}=p\>\forall i$, i.e, complexity of $N$ independent $\mathcal{GP}$s. This complexity is similar to the well known inducing point sparse approximation in Alvarez & Lawrence (2009), however without the assumption of conditional independence given discrete observations from the latent functions.

Despite reduced complexity, the main advantge is the reduction in the parameter space. Here it is crucial to note that any sparse approximation, be it an inducing point/variational approximation, a state space approximation, a matrix tapering approach or just a faster matrix inversion/determinant calculation scheme, can be plugged in into this structure.

Despite the linear increase in parameter space in the arrowhead model, when $N$ is extremely large model estimation can still be prohibitive. To this end, and inspired by the work of Fieuws & Verbeke (2006), we propose distributing the $\mathcal{MGP}$ into a group of bivariate $\mathcal{GP}$s which are independently built. Predictions are then obtained through combining predictions from each bivariate $\mathcal{GP}$. As previously mentioned we focus on predicting output 1 through borrowing strengths from the other $N-1$ outputs.

Fig. 2(a) illustrates the pairwise submodel between $y_{1}$ and $y_{i},i\in\mathcal{I}_{/1}$, where two latent functions are used to avoid negative transfer. Note here that the structure in Fig. 2(b) is proposed for efficient regularization and is discussed later in Sec. 4.2.1. The key advantage of the pairwise structure is that: (1) it can scale to an arbitrarily large $N$ by parallelization where each submodel is estimated with a limited number of parameters ($4\omega+2$) and with complexity of $O(2p^{3})$ (assuming exact inference with no approximations and that $p_{i}=p\>\forall i$) (2) It can avoid negative transfer without the need for $Q=N$ latent functions.

After building the $N-1$ sub-models, combining predictions boils down to combining $N-1$ predictive distributions $pr(y_{1}(\bm{x}_{0})|D_{1},D_{i})_{i\in\mathcal{I}_{/1}}$ in (2). This can be readily done using the rich literature on product of experts and Bayesian committee machines (see Deisenroth & Ng (2015), Moore & Russell (2015) and Tresp (2000) for an overview). The key idea across such approaches, in our context, is that sub-models that are uncertain about their predictions of $y_{1}(\bm{x}_{0})$ (i.e., have larger predictive variance) will get less weight.

Still under the setting of Sec 6.1.1, we try to verify theorem 3 and the impact of $\mathbb{P}_{\lambda}$ on automatic selection of related output. We use the pairwise model described in Fig. 2(b) where two bivariate submodels are used to predict $y_{1}$: $(y_{1},y_{2})$ and $(y_{1},y_{3})$. The covariance function of $y_{1}$ and $y_{i}$ ($i=2,3$) constructed using Fig. 2(b) are given as

We applied the pairwise model with a regularization term respectively to the data. For the penalty we use $\mathbb{P}_{\lambda}^{G}(\bm{\alpha}_{0})=\lambda\|\bm{\alpha_{0}}\|_{2}^{2}$ where $\bm{\alpha}_{0}=[\alpha_{01},\alpha_{0i}]^{T}$. Table 2 shows the estimated parameters.

One can directly observe from Table 2 that when adding regularization, $\alpha_{01}$ is shrunk to nearly zero in both submodels. This implies that $\operatorname{cov}_{1i}^{f}(d)\approx 0\>\forall x$ and $i\in\{2,3\}$ as $K_{01}$ is almost identically zero and hence $y_{1}$ is predicted independently (for exact sparse solutions one can use SCAD or the $L^{1}$ norm). This not only confirms that regularization can limit information sharing but also illustrates that in the proposed $\mathcal{MGP}$ models, one can automatically perform variable selection (cluster the outputs that ought to be predicted independently). A user might then choose to perform a separate $\mathcal{MGP}$ on the selected subsets. To the best of our knowledge this is the first model to achieve simultaneous estimation and functional selection using dependent $\mathcal{GP}$s.

In this setting, we use the real data set from the pacific exchange rate service (http://fx.sauder.ubc.ca/data.html). Our goal is to predict the foreign exchange rate compared to the United States dollar currency. We utilized the exchange rates of the top ten international currencies during the 157 weeks of from 2017 to 2020. Each output is adjusted to have zero mean and unit variance. We randomly choose 100 points as training points and the remaining 57 points as testing points. Table 3 shows the predictive arrow using different models while Fig. 7 provides illustartions. Once again the results confirm the need for the proposed relaxation models as parameter estimates tend to deteriorate as the parameter dimension increases.

Note that analysis on a Parkinson dataset to predict a disease symptom score is provided in Appendix G. The dataset is available on http://archive.ics.uci.edu/ml/datasets/Parkinsons+Telemonitoring.