Deep Latent-Variable Kernel Learning

In the machine learning community, Gaussian process (GP) [1] is a well-known Bayesian model to learn the underlying function $f:\mathbf{x}\mapsto\mathbf{y}$. In comparison to the deterministic, parametric machine learning models, e.g., neural networks (NN), the non-parametric GP could encode user’s prior knowledge, calibrate the model complexity automatically, and quantify the uncertainty of prediction, thus showing high flexibility and interpretability. Hence, it has been popularized within various scenarios like regression, classification [2], clustering [3], representation learning [4], sequence learning [5], multi-task learning [6], active learning and optimization [7, 8].

However, the two main weaknesses of GP are its poor scalability and the limited model capability on massive data. Firstly, as an representative of kernel method, the GP employs the kernel function $k(.,.)$ to encode the spatial correlations of $n$ training points into the stochastic process. Consequently, it performs operations on the full-rank kernel matrix $\mathbf{K}_{nn}\in R^{n\times n}$, thus raising a cubic time complexity $\mathcal{O}(n^{3})$ which prohibits the application in the era of big data. To improve the scalability, various scalable GPs have then been presented and studied. For example, the sparse approximations introduce $m$ ($m\ll n$) inducing variables $\mathbf{u}$ to distillate the latent function values $\mathbf{f}$ through prior or posterior approximation [9, 10], thus reducing the time complexity to $\mathcal{O}(nm^{2})$. The variational inference with reorganized evidence lower bound (ELBO) could further make the stochastic gradient descent optimizer, e.g., Adam [11], available for training with a greatly reduce complexity of $\mathcal{O}(m^{3})$ [12]. Moreover, further complexity reduction can be achieved by exploiting the structured inducing points and the iterative methods through matrix-vector multiplies, see for example [13, 14]. In contrast to the global sparse approximation, the complexity of GP can also be reduced through distributed computation [15, 16] and local approximation [17, 18]. The idea of divide-and-conquer splits the data for subspace learning, which alleviates the computational burden and helps capturing local patterns. The readers can refer to a recent review [19] of scalable GPs for further information.

Secondly, the GP usually uses (i) the Gaussian assumption to have closed-form inference, and (ii) the stationary and smoothing kernels to simply quantify how quickly the correlations vary along dimensions, which thus raise urgent demand for developing new GP paradigms to learn rich statistical representations under massive data. Hence, the interpretation of NN from kernel learning [20] inspires the construction of deep kernels for GP to mimic the nonlinearity and recurrency behaviors of NN [21, 22]. But the representation learning of deep kernels in comparison to deep models reviewed below is limited unless they are richly parameterized [23].

Considering the theoretical connection between GP and wide deep neural networks [24, 25], a hybrid, end-to-end model, called deep kernel learning (DKL) [26, 27, 28], has been proposed to combine the non-parametric property of GP and the inductive biases of NN. In this framework, the NN plays as a deterministic encoder for representation learning, and the sparse GP is built on the latent inputs for providing Bayesian estimations. The NN+GP structure thereafter has been extended for handling semi-supervised learning [29] and time-series forecasting [30]. The automatic regularization through the marginal likelihood of the last GP layer is expected to improve the performance of DKL and reduces the requirement of fine-tuning and regularization. But we find that the deterministic encoder may deteriorate the regularization of DKL, especially on small datasets, which will be elaborated in the following sections. Alternatively, we could stack the sparse GPs together to build the deep GP (DGP) [31, 32], which admits the layer-by-layer GP transformation that yields non-Gaussian distributions. Hence, the DGPs usually resort to the variational inference for model training. Different from DKL, the DGPs employ the full GP paradigm to arrive at automatic regularization. But the representation learning through layer-by-layer sparse GPs suffers from (i) high time complexity, (ii) complicated approximate inference, and (iii) limited capability due to the finite global inducing variables in each layer.

From the foregoing review and discussion, it is observed that the simple and scalable DKL enjoys great representation power of NN but suffers from the mismatch between the deterministic representation and the stochastic inference, which may risk over-fitting, especially on small datasets. While the sparse DGP enjoys the well calibrated GP paradigm but suffers from high complexity and limited representation capability.

Therefore, this article presents a complete Bayesian version of DKL which inherits (i) the scalability and representation of DKL and (ii) the regularization of DGP. The main contributions of this article are three-fold:

• •

  We propose an end-to-end latent-variable framework called deep latent-variable kernel learning (DLVKL). It incorporates a stochastic encoding process for regularized representation learning and a sparse GP part for guarding against over-fitting. The whole stochastic framework ensures that it can fully benefit from the automatic regularization of GP;

• •

  We further improve the DLVKL by constructing (i) the informative variational posterior rather than the simple Gaussian through neural stochastic differential equations (NSDE) to reduce the gap to exact posterior, and (ii) the flexible prior incorporating the knowledge from both the SDE prior and the variational posterior to arrive at a trade-off. The NSDE transformation improves the representation learning and the trade-off provides an adjustable regularization in various scenarios;

• •

  We showcase the superiority of DLVKL-NSDE against existing deep GPs through intensive (un)supervised learning tasks. The tensorflow implementations are available at https://github.com/LiuHaiTao01/DLVKL.

The remainder of the article is organized as follows. Section II briefly introduces the sparse GPs. Section III then presents the framework of DLVKL. Thereafter, Section IV proposes an enhanced version, named DLVKL-NSDE, through informative posterior and flexible prior. Then extensive numerical experiments are conducted in Section V to verify the superiority of DLVKL-NSDE on (un)supervised learning tasks. Finally, Section VI offers the concluding remarks.

Let $\mathbf{x}=\{\mathbf{x}_{i}\in R^{d_{\mathbf{x}}}\}_{i=1}^{n}=\{\mathbf{x}_{d}\in R^{n}\}_{d=1}^{d_{\mathbf{x}}}$ be the collection of $n$ points in the input space $\mathcal{X}\in R^{d_{\mathbf{x}}}$, and $\mathbf{y}=\{\mathbf{y}_{i}\in R^{d_{\mathbf{y}}}\}_{i=1}^{n}=\{\mathbf{y}_{d}\in R^{n}\}_{d=1}^{d_{\mathbf{y}}}$ the observations in the output space $\mathcal{Y}\in R^{d_{\mathbf{y}}}$, we seek to infer the latent mappings $\{f_{d}:R^{d_{\mathbf{x}}}\mapsto R\}_{d=1}^{d_{\mathbf{y}}}$ from data $\mathcal{D}=\{\mathbf{x},\mathbf{y}\}$. To this end, the GP characterizes the distributions of latent functions by placing independent zero-mean GP priors as $f_{d}(\mathbf{x})\sim\mathcal{GP}(0,k_{d}(\mathbf{x},\mathbf{x}^{\prime}))$, $0\leq d\leq d_{\mathbf{y}}$. For regression and binary classification, we usually have $d_{\mathbf{y}}=1$; while for multi-class classification and unsupervised learning, we are often facing $d_{\mathbf{y}}>1$.

We are interested in two statistics in the GP paradigm. The first is the marginal likelihood $p(\mathbf{y}_{d}|\mathbf{x})=\int p(\mathbf{y}_{d}|\mathbf{f}_{d})p(\mathbf{f}_{d}|\mathbf{x})d\mathbf{f}_{d}$, where $p(\mathbf{f}_{d}|\mathbf{x})=\mathcal{N}(\mathbf{f}_{d}|\mathbf{0},\mathbf{K}_{nn})$ with $\mathbf{K}_{nn}=k(\mathbf{x},\mathbf{x})\in R^{n\times n}$.¹¹1For the sake of simplicity, we here use the same kernel for $d_{\mathbf{y}}$ outputs. The marginal likelihood $p(\mathbf{y}|\mathbf{x})=\prod_{d=1}^{d_{\mathbf{y}}}p(\mathbf{y}_{d}|\mathbf{x})$, the maximization of which optimizes the hyperparameters, automatically achieves a trade-off between model fit and model complexity [1]. As for $p(\mathbf{y}_{d}|\mathbf{f}_{d})$, we adopt the Gaussian likelihood $p(\mathbf{y}_{d}|\mathbf{f}_{d})=\mathcal{N}(\mathbf{y}_{d}|\mathbf{f}_{d},\nu_{d}^{\epsilon}\mathbf{I})$ for regression and unsupervised learning with continuous outputs given the i.i.d noise $\epsilon_{d}\sim\mathcal{N}(0,\nu_{d}^{\epsilon})$ [1, 4]. While for binary classification with discrete outputs $y\in\{0,1\}$ and multi-class classification with $y\in\{1,\cdots,d_{\mathbf{y}}\}$, we have $p(\mathbf{y}_{d}|\mathbf{f}_{d})=\mathrm{Benoulli}(\pi(\mathbf{f}_{d}))$ and $p(\mathbf{y}_{d}|\mathbf{f}_{d})=\mathrm{Categorial}(\pi(\mathbf{f}_{d}))$, respectively, wherein $\pi(.)\in[0,1]$ is an inverse link function that squashes $f$ into the class probability space [33].

The second interested statistic is the posterior $p(\mathbf{f}_{d}|\mathbf{y}_{d},\mathbf{x})\propto p(\mathbf{y}_{d}|\mathbf{f}_{d})p(\mathbf{f}_{d}|\mathbf{x})$ used to perform prediction at a test point $\mathbf{x}_{*}$ as $p(\mathbf{f}_{*}|\mathbf{y},\mathbf{x},\mathbf{x}_{*})=\int p(\mathbf{f}_{*}|\mathbf{f},\mathbf{x},\mathbf{x}_{*})p(\mathbf{f}|\mathbf{y},\mathbf{x})d\mathbf{f}$. Note that for non-Gaussian likelihoods, the posterior is intractable and we resort to approximate inference algorithms [34].

The scalability of GPs however is severely limited on massive data, since the inversion and determinant of $\mathbf{K}_{nn}$ incur $\mathcal{O}(n^{3})$ operations. Hence, the sparse approximation [19] employs a set of inducing variables $\mathbf{u}_{d}\in R^{m}$ with $m\ll n$ at $\widetilde{\mathbf{x}}\in R^{m\times d_{\mathbf{x}}}$ to distillate the latent function values $\mathbf{f}_{d}$. Then, we have

where $\mathbf{K}_{mm}=k(\widetilde{\mathbf{x}},\widetilde{\mathbf{x}})$ and $\mathbf{K}_{nm}=k(\mathbf{x},\widetilde{\mathbf{x}})$. Thereafter, variational inference could help handle the intractable $\log p(\mathbf{y}|\mathbf{x})$. This is conducted by using a tractable variational posterior $q(\mathbf{u}_{d})=\mathcal{N}(\mathbf{u}_{d}|\mathbf{m}_{d},\mathbf{S}_{d})$, and maximizing the KL divergence

where $p(\mathbf{f}_{d},\mathbf{u}_{d}|\mathbf{x},\mathbf{y}_{d})=p(\mathbf{f}_{d}|\mathbf{u}_{d},\mathbf{x})p(\mathbf{u}_{d}|\mathbf{y}_{d})$ and $q(\mathbf{f}_{d},\mathbf{u}_{d}|\mathbf{x})=p(\mathbf{f}_{d}|\mathbf{u}_{d},\mathbf{x})q(\mathbf{u}_{d})$. It is equivalent to maximizing the following ELBO

The likelihood term of $\mathcal{L}_{\mathrm{gp}}$ represents the fitting error, and it factorizes over both data points and dimensions

thus having a remarkably reduced time complexity of $\mathcal{O}(m^{3})$ when performing stochastic optimization. Note that $q(\mathbf{f}_{d}|\mathbf{x})=\int p(\mathbf{f}_{d}|\mathbf{u}_{d},\mathbf{x})q(\mathbf{u}_{d})d\mathbf{u}_{d}=\mathcal{N}(\mathbf{f}_{d}|\bm{\mu}^{f}_{d},\bm{\Sigma}_{d}^{f})$ is conditioned on the whole training points, where $\bm{\mu}^{f}_{d}=\mathbf{K}_{nm}\mathbf{K}_{mm}^{-1}\mathbf{m}_{d}$ and $\bm{\Sigma}_{d}^{f}=\mathbf{K}_{nn}-\mathbf{K}_{nm}\mathbf{K}_{mm}^{-1}[\mathbf{I}-\mathbf{S}_{d}\mathbf{K}_{mm}^{-1}]\mathbf{K}_{nm}^{\mathsf{T}}$. While the posterior $q(\mathbf{f}_{i}|\mathbf{x}_{i})=\prod_{d=1}^{d_{\mathbf{y}}}\int p(f_{id}|\mathbf{u}_{d},\mathbf{x}_{i})q(\mathbf{u}_{d})d\mathbf{u}_{d}=\mathcal{N}(\mathbf{f}_{i}|\bm{\mu}^{f}_{i},\bm{\nu}^{f}_{i})$ only depends on the related point $\mathbf{x}_{i}$, where $\bm{\mu}^{f}_{i}\in R^{d_{\mathbf{y}}}$ collects the $i$-th element from each in the set $\{\bm{\mu}^{f}_{d}\}_{d=1}^{d_{\mathbf{y}}}$; and $\bm{\nu}^{f}_{i}\in R^{d_{\mathbf{y}}}$ collects the $i$-th diagonal element from each in the set $\{\bm{\Sigma}^{f}_{d}\}_{d=1}^{d_{\mathbf{y}}}$.²²2This viewpoint inspires the doubly stochastic variational inference [32]. Besides, the covariance here is summarized by the inducing variables. Besides, the analytical KL term in (1) guards against over-fitting and seeks to deliver a good inducing set. Note that the likelihood $p(\mathbf{y}|\mathbf{f})$ in (1) is not limited to Gaussian. By carefully reorganizing the formulations, we could derive analytical expressions for $\mathcal{L}_{\mathrm{gp}}$ for (un)supervised learning tasks [12, 35, 36].

Finally, when the inputs $\mathbf{x}$ are unobservable, we use the unsupervised ELBO for $\log p(\mathbf{y})$ as

to infer the latent variables $\mathbf{x}$ under the GP latent variable (GPLVM) framework [37]. This unsupervised model can be used for dimensionality reduction, data imputation and density estimation [35].

We consider a GP with additional $d_{\mathbf{z}}$-dimensional latent variables $\mathbf{z}$ as³³3We could describe most of deep GPs by the model (3), see Appendix A.

where the conditional distribution $p(\mathbf{z}|\mathbf{x})$ indicates a stochastic encoding of the original input $\mathbf{x}$, which could ease the inference of the following generative model $p(\mathbf{y}|\mathbf{z})$. As a result, the log marginal likelihood satisfies

In (4),

• •

  as for $p(\mathbf{y}|\mathbf{z})=\int p(\mathbf{y}|\mathbf{f})p(\mathbf{f}|\mathbf{z})d\mathbf{f}$, we use independent GPs $f_{d}\sim\mathcal{GP}(0,k_{d}(.,.))$, $0\leq d\leq d_{\mathbf{y}}$, to fit the mappings between $\mathbf{y}$ and $\mathbf{z}$. Consequently, we have the following lower bound by resorting to the sparse GP as

  $\displaystyle\log p(\mathbf{y}|\mathbf{z})\geq\mathbb{E}_{q(\mathbf{f}|\mathbf{u},\mathbf{z})q(\mathbf{u})}[\log p(\mathbf{y}|\mathbf{f})]-\mathrm{KL}[q(\mathbf{u})||p(\mathbf{u})].$

  (5)

  Note that the GP mapping employed here learns a joint distribution $p(\mathbf{y}_{d}|\mathbf{z})=\mathcal{N}(\mathbf{y}_{d}|\mathbf{0},\mathbf{K}_{nn}+\nu_{d}^{\epsilon}\mathbf{I})$ by considering the correlations over the entire space in order to achieve automatic regularization;

• •

  as for the prior $p(\mathbf{z}|\mathbf{x})$, we usually employ the fully independent form $p(\mathbf{z}|\mathbf{x})=\prod_{i=1}^{n}\mathcal{N}(\mathbf{z}_{i}|\mathbf{x}_{i})=\prod_{i=1}^{n}\prod_{d=1}^{d_{\mathbf{z}}}\mathcal{N}(z_{i,d}|\mathbf{x}_{i})$ factorized over both data index $i$ and dimension index $d$;

• •

  as for the decoder $q(\mathbf{z}|\mathbf{x})$, since we aim to utilize the great representational power of NN, it often takes the independent Gaussians $q(\mathbf{z}|\mathbf{x})=\prod_{i=1}^{n}\mathcal{N}(\mathbf{z}_{i}|\bm{\mu}_{i},\mathrm{diag}[\bm{\nu}_{i}])$. Instead of directly treating the means $\{\bm{\mu}_{i}\}_{i=1}^{n}$ and variances $\{\bm{\nu}_{i}\}_{i=1}^{n}$ as hyperparameters, they are made of parameterized function of the input $\mathbf{x}_{i}$ through multi-layer perception (MLP), for example, as

  $\displaystyle\bm{\mu}_{i}=\mathtt{Linear}(\mathtt{MLP}(\mathbf{x}_{i})),\quad\bm{\nu}_{i}=\mathtt{Softplus}(\mathtt{MLP}(\mathbf{x}_{i})).$

  This strategy is called amortized variational inference that shares the parameters over all training points, thus allowing for efficient training.

Combing (4) and (5), the final ELBO for the proposed deep latent-variable kernel learning (DLVKL) writes as

which can be optimized by the reparameterization trick [38].

It is found that in comparison to the ELBO of DKL in Appendix A-A, the additional $\mathrm{KL}[q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}|\mathbf{x})]$ in (6) regularizes the representation learning of encoder, which distinguishes our DLVKL from the DKL proposed in [26]. The DKL adopts a deterministic representation learning $\mathbf{z}=\mathtt{MLP}(\mathbf{x})$, which is equivalent to the variational distribution $q(\mathbf{z}_{i}|\mathbf{x}_{i})=\mathcal{N}(\mathbf{z}_{i}|\bm{\mu}_{i},\bm{\nu}_{i}\rightarrow\mathbf{0}^{+})=\delta(\mathbf{z}_{i}-\bm{\mu}_{i})$. Intuitively, in order to maximize $\mathcal{L}$, it is encouraged to learn a $p(\mathbf{y}|\mathbf{z})$ which maps $\mathbf{z}$ to a distribution concentrated on $\mathbf{y}$. Particularly, pushing all the mass of the distribution $p(\mathbf{y}|\mathbf{z})$ on $\mathbf{y}$ results in $\mathcal{L}\rightarrow\infty$, which however risks severe over-fitting [39]. The last GP layer employed in DKL is expected to alleviate this issue by considering the joint correlations over the output space. But the deterministic encoder, which is inconsistent to the following stochastic GP part, will incur free latent representation to weaken the model regularization, which results in over-fitting and under-estimated prediction variance in scenarios with finite data points. Contrarily, the proposed DLVKL builds a complete statistical learning framework wherein the additional KL regularization could avoid the issue by pushing $q(\mathbf{z}|\mathbf{x})$ to match the prior $p(\mathbf{z}|\mathbf{x})$.

Note that when $\mathbf{x}$ is unobservable, i.e., $\mathbf{x}\triangleq\mathbf{y}$, the proposed DLVKL recovers the GPLVM using back constraints (recognition model) [40]. Also, the bound (6) becomes the VAE-type ELBO

for unsupervised learning [38], wherein the main difference is that a GP decoder is employed.

Though the complete stochastic framework makes DLVKL be more attractive than DKL, it has two challenges to be addressed:

• •

  firstly, the assumed variational Gaussian posterior $q(\mathbf{z}|\mathbf{x})$ is often significantly different from the exact posterior $p(\mathbf{z}|\mathbf{x},\mathbf{y})$. This gap may deteriorate the performance and thus raises the demand of expressive $q(\mathbf{z}|\mathbf{x})$ for better approximation, which will be addressed in Section IV-A;

• •

  secondly, the choice of prior $p(\mathbf{z}|\mathbf{x})$ affects the regularization $\mathrm{KL}[q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}|\mathbf{x})]$ on latent representation. The flexibility and expressivity of the prior will be improved in Section IV-B.

The improvements of DLVKL come from two aspects: (i) the more expressive variational posterior transformed through neural stochastic differential equation (NSDE) for better approximating the exact posterior; and (ii) the flexible, hybrid prior to introduce adjustable regularization on latent representation. The two improvements are elaborated respectively in the following subsections.

This section first uses two toy cases to investigate the characteristics of the proposed models, followed by intensive evaluation on nine regression and classification datasets. Finally, we also simply showcase the capability of unsupervised DLVKL on the $\mathtt{mnist}$ dataset. The model configurations in the experimental study are detailed in Appendix D. All the experiments are performed on a windows workstation with twelve 3.50 GHz core and 64 GB memory.

This paper proposes the DLVKL which inherits the advantages of DKL but provides better calibration through regularized latent representation. We further improve the DLVKL through (i) the NSDE transformation to yield expressive variational posterior, and (ii) the hybrid prior to achieve adjustable regularization on latent representation. We investigate the algorithmic characteristics of DLVKL-NSDE and compare it against existing deep GPs. The comparative results imply that the DLVKL-NSDE performs similarly to the well calibrated GP on small datasets, and shows superiority on large datasets.

This work was supported by the Fundamental Research Funds for the Central Universities (DUT19RC(3)070) at Dalian University of Technology, and it was partially supported by the Research and Innovation in Science and Technology Major Project of Liaoning Province (2019JH1-10100024), the National Key Research and Development Project (2016YFB0600104), and the MIIT Marine Welfare Project (Z135060009002).