Convolutional Autoencoders for
Lossy Light Field Compression

Multiview displays along with Volumetric Geometry Models are currently the clashing viewpoints for creating functional immersive content. Volumetric methods are based on building polygons in a closed environment and applying textures and shading tools to them before rendering them to the user, often used in video games, animated movies and medical imaging [1]. The common critiques of Volumetric based techniques are their inability to correctly account for viewer-position-dependent effects such as opacity and occlusion [2], as well as the large bandwidth requirements due to the density of volumetric pixels (known as voxels) [3, 4, 5]. Some estimates have been made as high as 135GB/s for smooth video playback for such a system [6].

Multiview displays on the other hand rely on synthesizing different views needed by the user, based on properties of light and the plenoptic function.

The plenoptic function is a 5-D field representing the radiance of a ray of light, parameterized by the coordinates $(x,y,z)$, and the spherical angles $(\theta,\phi)$. However, the system happens to be overdetermined in this formulation, as radiance along a given ray of light remains constant. Using a carefully constructed parametrization, the system can be reduced down to 4 dimensions. This is known as a Light Field. There are many choices for the parametrization of a Light Field, (such as through 2 points on a unit sphere described entirely though $(\theta_{1},\phi_{1},\theta_{2},\phi_{2})$), but the most common is the pair of points on separate planes formulation. This is known as the $uv$, $st$ plane parametrization, where points on the $uv$ plane define the perspective (or viewpoint) of an image, and the $st$ the actual pixels in the image (or view). A visual explanation of this can be seen in Figure 1.

Light Field Rendering became popularized at ACM SIGGRAPH 96’ by Mark Levoy based on a talk and paper by the same name [7]. Though computational means and displays have come a long way since, the overall idea is still relatively the same. Light Fields encompass all the information needed to synthetically reproduce a scene from all perspectives in a nearby neighbourhood of the observed ground truth. This can include viewpoints in-between those already measured on the $uv$ plane using techniques such a cubic interpolation, as well as viewpoints off the $uv$ plane, simulating viewer depth changes relative to the image.

Compression generally splits into 2 categories: lossy and lossless. As the names imply, lossy compression allows for the lost of information, whereas lossless does not, at the cost of compression size. Lossless compression techniques are based on the idea that the data contains redundant information which can be packed more efficiently, while lossy relies on the removal of less important information present in the data. In this fashion, lossless compression allows for perfect reconstruction of the original data, while lossy has an error associated with it, commonly measured by the Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR) [8] or the Structural Similarity Index (SSIM) [9] for images.

With the rise of deep learning in recent years, a reasonable attempt at compression would see the utilization of an neural network. More precisely, the goal is to reduce the size of the initial data during the transfer/streaming and storage phases, while also limiting our data quality loss to a minimum, during the consumption phase. All this of course ideally, should be done fast enough to run in real-time. Hence, what’s required is an efficient encoder and decoder.

Generally, neural network designs that fan inwards layer to layer, tend to reduce down the amount of information required to represent the data. Alternatively, designs that fan out layer to layer tend to increase the information. The earlier structure is known as an encoder network, whereas the latter as a decoder network. Stacking these two one after the other, and training them together as one, gives rise to what is known as an autoencoder [10]. A visual representation of this architecture can be seen in 2.

An autoencoder network is a common tool for compression of data. They have achieved great results in the encoding of images [12, 13, 14], videos [15] and synthesization [16] and style transfer of audio [17].

The input and output of the network are the same data sample and ideally the network will figure out the optimal encoding in the hidden layers, by varying the weights of the model. The entire network gets trained together, but the encoder and decoder can be separated post-training, as to allow for separate encoding and decoding functions. The middle layer (output to encoder sub-network and input to decoder sub-network) contains a latent representation of the input/output data in the compression. This layer does not necessarily have an observable meaning associated with each neuron, but this is not a problem as applying the decoder network to it can map it back to a meaningful result.

The size of this middle layer relative to the input layer defines the compression size. This can be fine tuned as desired, based on the architecture design of the network. Of course there is a trade-off between compression size and compression loss, strongly dependant on this bottleneck layer. Too large of a layer keeps the compression low, but too small of a choice can lead to artifacts such as ringing, blocking, colour distortion, image blur [18], and checkerboard patterns [19].

When it comes to network structures for visual data such as images and video, it has been shown that convolutional networks [20] are best for extracting local spatial information in frames such as edge and corner detection [21, 22]. The breakthrough itself is one of the major reasons deeplearning has rose to claim recently, through the success of networks such as LeNet, GoogleNet, VGGNet, ResNet and most notably AlexNet [23, 24, 25, 26, 27]. The convolution architecture also allows for naturally arising reduction in the size of layer to layer, with or without the use of pooling layers. This is useful in building the encoder subnetwork. For the decoder part however, an inverse type of convolution operation is required, known as deconvolution [28]. Deconvolutional layers are not as common as convolutional ones, but their use is well understood as an ”unpooling” agent in the network, well suited for a decoder.

With the relative youth of the technology, there still is little known about effective compression of light fields. The JPEG group has proposed standards when it comes to light representation imagery files, in the form of what they call JPEG Pleno [29], similar to their other JPEG codecs for imagery. Little is still agreed upon, and much is still in debate in the community.

From a lossless perspective, Santos et al. [30] look at alternative data arrangements and colour transformations to achieve a more compact representation of the data, with respectable results. Like any lossless approach however, they are still orders of magnitude off from large scale adoption.

When perfect data retention can be sacrificed for significant size reduction, a few reasonable attempts have shown promise. Chen et al. [31] were able to reduce bit-rates close to 50% while maintaining reasonably high PSNR values, by carefully encoding disparity information with some optimized key views of the light field. Zhang et al. [32] proposed an alternative method encoding a simulated point cloud into a B-Spline wavelet, achieving PSNR values near 30 dB depending on compression parameters, but their computation times vary between 5 and 24 seconds.

More similar to our inspiration, Barik et al. [33] utilize the use of convolutional neural networks to achieve bitrate gains of 30%. Barik proposes a standard video encoder to transform the light fields into sparse representation of the data, and a neural network to decode it back to the original light field. Computation times were not outlined. Gupta et al. [34] go one step further and rely only on a neural network, splitting the spatial and angular encoding tasks in separate branches of their network, until combining the results in the final layers. Their model achieved respectable PSNR scores between 26-32 dB, attaining a compression ratio of 16:1, but coming at the cost of computation time, logging processing times on the order of minutes.

There are far less open source light field datasets available, than for more mainstream media content such as images and videos. The Stanford Light Field Archive is the most well known, with the datasets commonly showing up in research papers [35]. One common issue with all the Stanford datasets for our purpose was the different sizes of supplied images. Images are often stretched in convolutional networks so as to have 1:1 ratios along their spatial dimensions, but this would provide problems in light filed images as the disparity maps would change along these stretched dimensions. The Old Archives resolutions are too low, where as the New Archives only provide 13 reasonable light fields, though they are made up of 289 viewpoints each. The Multiview and Three-View datasets each offered too large of baselines between viewpoints.

MIT have their own achieve [36], with applications to other commonly cited papers [37], but the data was entirely synthetic, lacking variations representative of true world structures such as materials, diffusion and lighting.

Lytro was a pioneer in the light field imaging sector, and strategically offer some publicly available data from captured on their own cameras [38]. They offer 25 high quality light fields, all in 1:1 image ratios, and great variety of viewpoints at close baselines, but the formatting and preprocessing needed to access the data, was far too convoluted. Furthermore, with the company closing down, and the format not being standardized we chose to go in a different direction.

We decide on the HCI Light Field Dataset as collected by groups from Heidelberg University and the University of Konstanz [39]. The dataset consists of 24 light fields, each made up of 81 viewpoints. All the image slices are of equal size and aspect ratios. This is ideal for an input layer of a neural network, as the size would have to be fixed, while the aspect ratio assures our images would not suffer from any resizing artifacts. Furthermore, the data already comes segmented into a test/train split ideal for research purposes, and the true disparity maps are also provided if we choose to utilize them in the future. Each light field is made up of 9x9 densely packed viewpoints, each of 512x512 pixels along 3 color channels, in standard .png format.

Training neural networks requires vast amounts of data, yet we are limited by just 24 light fields. In order to provide our network a large enough sample size to train on we utilize a few data augmentation techniques commonly used for image datasets.

We first split our 24 light fields into testing and training partitions. Honauer et al. [39] already proposed a testing standard of the first 4 light fields, so we use the subsequent 20 as our training set. We can double our sample size to 40 by allowing for horizontal flips of the light fields. Note however, that we abstain from flipping them vertically, as we would prefer our network to learn features in their correct rotation as oppose to generalized for different angles.

Next, we proceed by applying random brightness and saturation changes to all viewpoints in a given light field to simulate 1000s of ”new” light fields. We chose uniform random brightness adjustments $\in(-0.2,0.2)$, while our saturation factors stay $\in(0.6,1.6)$ (Note all our images are processed in float32 dtype, with pixel values ranging from 0 to 1). Lastly, we subsample viewpoints at random locations at random sized windows, keeping aspect ratios at 1:1 and resizing subimages back to the initial 512x512 size. The minimum size window we use for such subsampling is 256x256. This allows us to turn different parts of a light field into new light fields, at new zoom depths with new scales to their disparities maps, thus allowing for better generalization. Figures 4 and 3 illustrate both of these techniques in more detail.

After all our augmentations, we have enough data to never see the exact same light field twice in a training period. Thus, the exact definition of a training epoch will be adjusted in section 4.