Uniform Interpolation Constrained Geodesic Learning on Data Manifold

With the success of deep learning on approximating complex non-linear functions, representing data manifolds using neural networks has received considerable interest and been studied from multiple perspectives. Autoencoders (AE) [14], generative adversarial network (GAN) [13] and their combinations [16, 22] are notable generative models for learning the geometry structure or the distribution of data on the manifold. However, they usually fail to generate a convex latent representation when confronting with even simple curved manifolds such as swiss-roll and S-curve. For geodesic learning, Arvanitidis et al. [3, 24] and Chen et al. [10, 9] both present a magnification factor [7] to help find the shortest path that follows the regions of high data density in the latent space as the learned geodesic. But these methods explore geodesics on latent space rather than on data space and lack strict guarantees for geodesics in mathematics.

In this paper, we propose a method to capture the geometric structure of a data manifold and to find smooth geodesics between data samples on the manifold. First, motivated by DIMAL [17], we introduce a framework combining an autoencoder with traditional manifold learning methods to explore the geometric structure of a data manifold. In contrast to existing combinations of autoencoders and GANs, we resort to the classical manifold learning algorithms to obtain an approximation of geodesic distance. Then the encoder and decoder are optimized with this approximation being the constraint. Using this method, we alleviate the problem that the latent distribution is non-convex on curved manifolds, which means our encoder can unfold the data manifold to flattened latent representations.

Second, we propose a geodesic learning method to establish smooth geodesics between data samples on a data manifold. The Riemannian metric on the data manifold can be induced by the canonical metric in the Euclidean space in which the manifold is isometrically immersed. Existing methods cannot guarantee a uniform interpolation along the geodesic. We parameterize the generated curve by a parameter $t$ and propose a constant-speed loss to achieve a uniform interpolation by making $t\in[0,1]$ be an arc-length parameter of the interpolation curve. According to Riemannian geometry [8], we propose a geodesic loss to force the interpolation network to generate points along a geodesic. Another key factor is a minimizing loss to make the geodesic be the minimizing one. Our contributions are summarized as follows:

• •

  We propose a framework in which an autoencoder and an interpolation network are introduced to explore the manifold structure. The autoencoder network promotes convex latent representations achieved by adding some prior geometric information to it, so that the generated samples are able to remain within the distribution of the data manifold.

• •

  We propose a constant-speed loss and a minimizing geodesic loss for the interpolation network to generate the minimizing geodesic on the manifold given two endpoints. We parameterize each geodesic by an arc-length parameter $t$, such that we can generate points moving along the minimizing geodesic with constant speed and thus fulfill a uniform interpolation.

Traditionally, manifold learning methods aim to infer latent representations that can capture the intrinsic geometric structure of data. They are generally applied in dimensionality reduction and data visualization. Classical nonlinear manifold learning algorithms such as Isomap [21], locally linear embedding [18] and local tangent space alignment (LTSA) [25] preserve local structures in small neighborhoods and derive the intrinsic features of data manifolds.

The interpolation between data samples has attracted wide attention because it can achieve a smooth transition from one sample to another, such as intermediate face generation and path planning. There are several works of literature focusing on the interpolation of data manifold. Agustsson et al. [2] propose to use distribution matching transport maps to obtain analogous latent space interpolation which preserves the prior distribution of the latent space, while minimally changing the original operation. It can generate interpolated samples with higher quality but it does not have an encoder to map the given data samples to the latent space. Several other works [5, 19, 12] that combine autoencoders with GANs are proposed to generate interpolated points on data manifold. But limited by the ability of GAN, they usually fail to generate a convex latent representation when confronting with curved manifolds such as swiss-roll and S-curve. To avoid this problem, Arvanitidis et al. [3, 24] and Chen et al. [10] both present that the magnification factor [7] can be seen as a measure of the local distortion of the latent space. This helps the generated geodesics to follow the regions of high data density in the latent space. Chen et al. [9] also find the shortest path in a finite graph of samples as the geodesics on the data manifold. This finite graph is built in the latent space using a binary tree data structure with edge weights based on Riemannian distances.

We briefly present the essentials of Riemannian geometry. For a thorough presentation, see [8]. Suppose $M$ is a Riemannian differentiable manifold. $\alpha:(-\epsilon,\epsilon)\to M$ is a differentiable curve in $M$. Suppose that $\alpha(0)=p\in M$, the tangent vector to the curve $\alpha$ at $t=0$ is a function $\alpha^{\prime}(0)$ defined on the arbitrary function $f$ on $M$ given by $\alpha^{\prime}(0)f=\frac{d(f\circ\alpha)}{dt}\bigg{|}_{t=0}$. The set of all tangent vectors to $M$ at $p$ will be indicated by tangent space $T_{p}M$. Let $(U,x)$ be a parametrization of $M$ at $p$ and for $\forall q\in U$, suppose $x^{-1}(q)=(z^{1}(q),z^{2}(q),\cdots,z^{n}(q))$ is the local coordinate of $q$. Then the tangent vector at $p$ of the coordinate curve $\frac{\partial}{\partial z^{i}}\big{|}_{p}$ is defined as$\frac{\partial}{\partial z^{i}}\big{|}_{p}(f)=\frac{\partial(f\circ x)}{\partial z^{i}}\big{|}_{p}$. $\{\frac{\partial}{\partial z^{i}}\arrowvert_{p},1\leqslant i\leqslant n\}$ spans a tangent space $T_{p}M$ at $p$. If the tangent vector is orthonormal with each other, $\{\frac{\partial}{\partial z^{i}}\arrowvert_{p},1\leqslant i\leqslant n\}$ is the orthonormal basis of tangent space.

Let $M^{m}$ and $N^{n}$ be two differentiable manifolds. Given a differentiable mapping $\varphi:M\to N$, we can define a differential of $\varphi$ at $p$ as linear mapping $d\varphi_{p}:T_{p}M\to T_{\varphi(p)}N$ given by $d\varphi(p)(v)(f)=v(f\circ\varphi)$, for $\forall f\in C_{\varphi(p)}^{\infty}$. If $d\varphi(p):T_{p}M\to T_{\varphi(p)}N$ is injective for all $p\in M$, then the mapping $\varphi$ is said to be an immersion.

A Riemannian manifold is a differentiable manifold equipped with a given Riemannian metric which assigns on each point $p$ of $M$ an inner product $\langle,\rangle_{p}$ (i.e. a symmetric, bilinear and positive-definite form) on the tangent space $T_{p}M$. Furthermore, an isometric immersion $\varphi:M\to N$ is an immersion satisfying $\big{\langle}u,v\big{\rangle}_{p}=\big{\langle}df_{p}(u),df_{p}(v)\big{\rangle}_{f(p)}$ for $\forall p\in M$ and $\forall u,v\in T_{p}M$.

A geodesic is a parametrized curve $\gamma:I\to M$ satisfying $\frac{D}{dt}(\frac{d\gamma}{dt})=0$ for all $t\in I$. $\frac{D}{dt}$ is the covariant derivative of a vector field along $\gamma$ [8]. $\frac{d\gamma}{dt}$ is the tangent vector at $t$.

Original AE and some combinations between AE and GANs can generate high-quality samples from specific latent distributions on high-dimensional data manifold. But they fail to get convex embeddings on some curved surfaces with changing curvature such as swiss-roll or S-curve. Some other works [5, 19, 12] were proposed to generate interpolations within the distribution of real data by distinguishing interpolations from real samples. But they generate unsatisfying samples on the above-mentioned low-dimensional manifolds. The encoding results can be seen in Fig. 1. The reason for this problem may be the insufficient ability of GANs and the decoder of AE. The discriminator of GANs can only distinguish the similarities of distributions between generated samples and real ones and the autoencoders are prone to put the curvature information of data manifold in latent representations to make the network simple as possible. So the latent embeddings also have a changing curvature to save the curvature information. It may result in the non-convex representations.

In our method, we add some prior geometric information obtained by traditional manifold learning approaches to encoding a convex latent representation. Traditional nonlinear manifold learning approaches such as Isomap [21], LLE [18] and LTSA [25] are classical algorithms to get a convex embedding by unfolding some curved surfaces. We apply them to our method by adding regularization to the autoencoder. The loss function of the autoencoder can be written as:

where $x$ is the input sampled on the data manifold. $M$ denotes the number of input $x$. $DKS_{ij}$ represents the approximated geodesic distance between $E(x_{i})$ and $E(x_{j})$. Encoder $E$ and Decoder $D$ of an autoencoder are trained to minimize the above loss $L_{AE}$. $DKS$ can be obtained by building a graph from the data with k-nearest neighbors for each node and computing the corresponding pairwise geodesic distances using traditional manifold learning algorithms. For more details we refer the interested reader to [21, 18, 25]. With $DKS_{ij}$ as an expected approximation, the encoder is forced to train towards obtaining a convex latent representation while the decoder is forced to learn the lost curvature information on latent embeddings. Behaviors induced by the $L_{AE}$ loss can be observed in Fig. 1: the swiss-roll can be flattened on 2-dimensional latent space. For our experiments, we choose LTSA to compute the approximated geodesic distance for regularizing the autoencoder because of its learned local geometry which views the neighborhood of a data point as a tangent space to flat the manifold.

In our model, we denote $\mathcal{X}$ as a data manifold. The Riemannian metric on $\mathcal{X}$ can be induced from the canonical metric on the Euclidean space $\mathbb{R}^{N}$ to guarantee the immersion is an isometric immersion. Thus to obtain a geodesic on manifold $\mathcal{X}$, we can use the Riemannian geometry on $\mathbb{R}^{N}$ and the characteristics of isometric immersion.

In this section, we present experiments on the geodesic generation and image interpolation to demonstrate the effectiveness of our method.

We explore the geometric structure of the data manifold by proposing a uniform interpolation constrained geodesic learning algorithm. We add prior geometric information to regularize our autoencoder to generate interpolants within the data manifold. We also propose a constant-speed loss and a minimizing geodesic loss to generate geodesic on the underlying manifold given two endpoints. Different from existing methods in which geodesic is defined as the shortest path on the graph connecting data points, our model defines geodesic consistent with the definition of a geodesic in Riemannian geometry. Experiments demonstrate our model can fulfill a uniform interpolation along the minimizing geodesics both on 3-D curved manifolds and high-dimensional image space.