DeepMI: A Mutual Information Based Framework For Unsupervised Deep Learning of Tasks

$\mathcal{MI}$ is a convex function and attains global minima when any two random variables under consideration are independent. Mathematically, ${\mathcal{MI}}\to 0$ when the variables are independent, whereas, ${\mathcal{MI}\to\{\mathcal{H}(X)=\mathcal{H}(Y)\}}$ when both the variables are identical statistically. This property of $\mathcal{MI}$ can readily be employed to quantify similarity between two signals. However, while doing so, the definition of $\mathcal{MI}$ has to be interpreted in a quite different manner.

To better understand, let us consider an example of image matching, provided two images $X$ and $Y$. In order to measure the similarity between the images using $\mathcal{MI}$, the image itself can not be considered as a random variable. Because in that case, $p_{X},p_{Y}$ and $p_{XY}$ shall be meaningless. In other words, per sample $\mathcal{MI}$ is not defined. Hence, instead of an image, its pixel values are considered as a random variable over which the relevant distributions can be defined. The pixel values may refer to intensity, color, gradients etc. In order to compute the similarity score, first the marginals and joint pdfs over the selected variable has to be computed, and the similarity can be obtained by using the Eq. 1. While doing so, Eq. 2 needs to be rewritten as given below.

	$\displaystyle\mathcal{H}(X)$	$\displaystyle=-\sum_{i=1}^{N}p_{X}^{i}log(p_{X}^{i}),$		(3)
	$\displaystyle\mathcal{H}(Y)$	$\displaystyle=-\sum_{i=1}^{N}p_{Y}^{i}log(p_{Y}^{i}),$
	$\displaystyle\mathcal{H}(X,Y)$	$\displaystyle=-\sum_{i=1}^{N}\sum_{j=1}^{N}p_{XY}^{ij}log(p_{XY}^{ij})$

Where $N$ is the number of bins in the pdf.

As an another example, we can consider matching of two time series signals by using $\mathcal{MI}$. Following the above discussion, the two signal instances under consideration can not be considered as random variables, instead their instantaneous values are considered as random variable. It must be noticed that the choice of random variable depends on the application.

Consider two images $X$ and $Y$, with $p_{X}\in\mathbb{R}^{N},p_{Y}\in\mathbb{R}^{N}$ and $p_{XY}\in\mathbb{R}^{N\times N}$ as their marginals and joint pdfs respectively. The dimensions of $p_{X},p_{Y}$ is $N\times 1$, whereas it is $N\times N$ for $p_{XY}$. From, Eq. 1, 2, we can immediately say that $\mathcal{MI}\to 0$ when two signals are dissimilar while ${\mathcal{MI}\to\{\mathcal{H}(X)=\mathcal{H}(Y)\}}$ when the signals are exactly the same. Hence, for the images $X$ and $Y$ to be identical, the necessary but not sufficient condition is that $p_{X}$ and $p_{Y}$ should be same. While, in order to guarantee, the following has to be satisfied.

$$p_{X}^{i}=p_{Y}^{i}=p_{XY}^{ii},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ p_{XY_{|i\neq j}}^{ij}=0,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ i,j=1,2,...,N$$

(4)

In other words, when $X\equiv Y$, the off-diagonal elements of $p_{XY}$ are zero while all the diagonal elements are non-zero (depending on a distribution) and equals to $p_{X}$ and $p_{Y}$ simultaneously. The above insight leads us to derive an expression for the $\mathcal{LMI}$ function to train deep networks.

We know that for any probability density function Eq. 5 and 6 holds. These equations represent a $1$D and a $2$D probability density function respectively.

	$\displaystyle\footnotesize\sum_{i=1}^{N}p^{i}_{X}=\sum_{i=1}^{N}p^{i}_{Y}=1,$		(5)
	$\displaystyle\sum_{i=1}^{N}\sum_{j=1}^{N}p^{ij}_{XY}=1$		(6)

Rewriting Eq. 6 as combination of its diagonal $(i=j)$ and off-diagonal elements $(i\neq j)$, we get

	$\displaystyle\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+\sum_{i=1}^{N}p_{XY}^{ii}=1$		(7)
	$\displaystyle\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+\sum_{i=1}^{N}p_{XY}^{ii}\leavevmode\nobreak\ +\sum_{i=1}^{N}p_{XY}^{ii}-\sum_{i=1}^{N}p_{XY}^{ii}=1$		(8)

		$\displaystyle\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+\sum_{i=1}^{N}p_{XY}^{ii}\leavevmode\nobreak\ +\sum_{i=1}^{N}p_{XY}^{ii}-\sum_{i=1}^{N}p_{XY}^{ii}+$		(9)
		$\displaystyle\sum_{i=1}^{N}p_{X}^{i}-\sum_{i=1}^{N}p_{X}^{i}+\sum_{i=1}^{N}p_{Y}^{i}-\sum_{i=1}^{N}p_{Y}^{i}=1$

		$\displaystyle\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+\sum_{i=1}^{N}|p_{XY}^{ii}-p_{X}^{i}|\leavevmode\nobreak\ +$		(10)
		$\displaystyle\sum_{i=1}^{N}|p_{XY}^{ii}-p_{Y}^{i}|-\sum_{i=1}^{N}p_{XY}^{ii}+1+1\geq 1$

		$\displaystyle\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+\sum_{i=1}^{N}|p_{XY}^{ii}-p_{X}^{i}|\leavevmode\nobreak\ +$		(11)
		$\displaystyle\sum_{i=1}^{N}|p_{XY}^{ii}-p_{Y}^{i}|-\sum_{i=1}^{N}p_{XY}^{ii}+1\geq 0$

Now, reffering to Eq. 7, we can write

$$\sum_{i=1}^{N}p_{XY}^{ii}\leq 1\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \Rightarrow\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ 1-\sum_{i=1}^{N}p_{XY}^{ii}\geq 0$$

(12)

Using the above into Eq. 11, we get

		$\displaystyle\mathcal{L}_{LMI}=\frac{1}{3}\Big{(}\sum_{i=1}^{N}\sum_{\begin{subarray}{c}j=1\\ i\neq j\end{subarray}}^{N}p_{XY}^{ij}+$			(13)
		$\displaystyle\sum_{i=1}^{N}|p_{XY}^{ii}-p_{X}^{i}|+\sum_{i=1}^{N}|p_{XY}^{ii}-p_{Y}^{i}|\Big{)}$	$\displaystyle\geq 0$

Where the factor $\nicefrac{{1}}{{3}}$ is included to ensure $\mathcal{LMI}\leq 1$. It is obtained by replacing each of the three terms to their maximum. The equality of the equation to $0$ will hold iff $p_{X}^{i}=p_{Y}^{i}=p_{XY}^{ii},\leavevmode\nobreak\ \leavevmode\nobreak\ p_{XY_{|i\neq j}}^{ij}=0\leavevmode\nobreak\ \leavevmode\nobreak\ \forall\leavevmode\nobreak\ i,j\in 1,2,..,N$. i.e. two images match perfectly. Hence, the L.H.S. of the equation Eq. 13 is treated as the objective function which we call the $\mathcal{LMI}$ function. The “$\mathcal{L}$” stands for “linearized” which arises because the $\mathcal{LMI}$ formulation is linear in the elements of the pdfs, whereas the “$\mathcal{MI}$” term arises because at the equality, the regular $\mathcal{MI}$ expression will also be maximized. The $\mathcal{LMI}$ formulation is quite interesting because it is essentially a combination of three different losses weighted equally. The $\mathcal{L}_{LMI}$ formulation is quite intuitive and the gradients are straightforward to compute.

The $\mathcal{LMI}$ function utilizes the pdfs $p_{X},p_{Y}$ and $p_{XY}$ which are discrete in nature. These are typically obtained by computing an $N$ bin histogram followed by a normalization step with $||.||_{1}=1$. As per the standard procedure to compute a regular histogram, first, a bin-id corresponding to an observation of the random variable is computed and later, the count of the respective bin is incremented by unity. The computation of the bin-id is carried out by a $ceil$ or $floor$ operation which is not differentiable. While performing the rounding step, the actual contribution of the observation is lost. Thus, both the incremental and rounding procedure prevent the gradient flow which is needed during the training process.

To better understand the above, let $h_{X}$ be an $N$ bin histogram of the random variate $X$ and $x\in X$ be an observation. In the case of a regular histogram, the bin-id $x_{b}$ corresponding to an observation $x$ is computed as follows:

	$\displaystyle x_{b}=\frac{\hat{x}}{bin\_res},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak$	$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \hat{x}=\frac{x-min_{X}}{max_{X}-min_{X}},$		(14)
	$\displaystyle bin\_res$	$\displaystyle=\frac{max_{X}-min_{X}}{N}$

where $max_{X},min_{X}$ are the maximum and minimum values which the variable $X$ can attain at any instant. Typically for $8$-bit images, $[min_{X},max_{X}]=[0,255]$. From the Eq. 14, it can be noticed that the value of $x_{b}$ is not necessarily an integer. In this case, $x_{b}$ is rounded to the nearest integer by using $ceil\equiv\lceil x_{b}\rceil$ or $floor\equiv\lfloor x_{b}\rfloor$, depending upon one’s convention. Now, the count of $x_{b}$ is incremented by one. Therefore, it becomes evident that the rounding procedure and the unit incremental procedure do not allow the gradient computation w.r.t. the observations. To cope up with this, we employ fuzzification strategy in order to ensure valid gradients during back-propagation.

From the Eq. 16, it can be inferred that, gradient of $x$ w.r.t. $\mathcal{L}_{LMI}$ depends on $p_{X}^{x_{0}},p_{X}^{x_{0}},p_{XY}^{x_{0}y_{0}},p_{XY}^{x_{0}y_{1}},p_{XY}^{x_{1}y_{0}},p_{XY}^{x_{1}y_{1}}$. Therefore, we can write:

$$\frac{\partial\mathcal{L}}{\partial x}=\sum_{i=0}^{1}\frac{\partial\mathcal{L}}{\partial p_{X}^{x_{i}}}\leavevmode\nobreak\ \frac{\partial p_{X}^{x_{i}}}{\partial x}+\sum_{i=0}^{1}\sum_{j=0}^{1}\frac{\partial\mathcal{L}}{\partial p_{XY}^{x_{i}y_{i}}}\leavevmode\nobreak\ \frac{\partial p_{XY}^{x_{i}y_{i}}}{\partial x}$$

(17)

Using chain rule:

	$\displaystyle\frac{\partial p_{X}^{x_{i}}}{\partial x}=\frac{\partial p_{X}^{x_{i}}}{\partial\hat{p}_{X}^{x_{i}}}\times\frac{\partial\hat{p}_{X}^{x_{i}}}{\partial{m_{x_{i}}}}\times\frac{\partial m_{x_{i}}}{\partial x_{b}}\times$		(18)
	$\displaystyle\frac{\partial x_{b}}{\partial\hat{x}}\times\frac{\partial\hat{x}}{\partial x},\leavevmode\nobreak\ \leavevmode\nobreak\ p_{X}^{x_{i}}=\frac{\hat{p}_{X}^{x_{i}}}{\sum_{i=1}^{N}\hat{p}_{X}^{x_{i}}}$

and similarly,

	$\displaystyle\frac{\partial p_{XY}^{x_{i}y_{j}}}{\partial x}=\frac{\partial p_{XY}^{x_{i}y_{j}}}{\partial\hat{p}_{XY}^{x_{i}y_{j}}}$	$\displaystyle\times\frac{\partial\hat{p}_{XY}^{x_{i}y_{j}}}{\partial m_{x_{i}y_{j}}}\times\frac{\partial m_{x_{i}y_{j}}}{\partial m_{x_{i}}}\times\frac{\partial m_{x_{i}}}{\partial x_{b}}$		(19)
		$\displaystyle\times\frac{\partial x_{b}}{\partial\hat{x}}\times\frac{\partial\hat{x}}{\partial x},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ p_{XY}^{x_{i}y_{j}}=\frac{\hat{p}_{XY}^{x_{i}y_{j}}}{\sum_{i=1}^{N}\hat{p}_{XY}^{x_{i}y_{j}}}$

Each of the above partial derivatives can be easily computed using the Eq. 13 - 16 via the chain rule. It must be noticed that in the gradient calculations, $x_{0},x_{1}$ does not occur which would have been there in case of regular histograms, leading to undefined derivatives. Similarly, gradients can be also be computed for an observation $y$.

The formulation of $\mathcal{L}_{LMI}$ seems quite intuitive. There is however a consideration which must be accounted while using it for signal matching, especially in the scenarios where one of the signal is the groundtruth and the another is the estimated version of the first by a neural network.

To understand this, consider two images $X$ and $Y$, where $X$ is the image to be reconstructed and $Y$ is the reconstruction or simply the output of a neural network. The marginals $p_{X},p_{Y}$ and the joint $p_{XY}$ of $X$ and $Y$ are computed using the fuzzification procedure as described previously. As we know that these distributions are the approximated version of the underlying distribution, therefore, undersampling or oversampling of the underlying distribution is possible. By nature the distributions $p_{X}$, $p_{Y}$ and $p_{XY}$ do not have a well defined mathematical expressions in such scenarios. Moreover, both the $p_{X}$ and $p_{Y}$ can be obtained using $p_{XY}$, therfore, while backpropagation, the gradients only w.r.t $p_{XY}^{ij}$ are backpropagated. Mathematically, gradients w.r.t. $p_{Y}$ should also be backpropagated because calculations of $p_{Y}$ is dependent on $Y$. However doing so, disturbs the training process and affects the neural network performance. After our experimental study, we mark this observation as an outcome of the distributions approximation procedure.

The DeepMI framework has $N$ as the only hyperparameter. The number of bins $N$ mainly depicts that how precisely the $\mathcal{L}_{LMI}$ should penalize the network while matching. For example, consider an $8$-bit image with dynamic range $[0,255]$. Now, we set $N=255$, the network will be penalized very strongly while matching. While if the $N$ is set to a small number, the network will be forced to focus only on the important details. This property can be quite useful in cases where two images from different cameras need to be matched while both the images have different brightness levels.

This experiment consists of two binary images where the second image is a spatially transformed version of the first i.e. a $2$D rigid body transform is defined between the two images. Both the images consist of a black background and a white rectangular bar. The bar sized of $50\times 125$ in an image ($192\times 640$) has two degrees-of-freedom (DoF): $tx$, and $\theta$ which correspond to horizontal motion and in-plane rotation. The goal of the experiment is to learn the $2$DoF parameters in an unsupervised manner. For the training purpose, we generate a dataset of $1500$ images with $1000+500$ train-test split. The dataset is generated by transforming the bar in the image by randomly generating $tx\in[-100,100]$ pixels and $\theta\in[-40,40]$ degrees. The training of this experiment is performed using SGD with Nesterov momentum $=0.95$ for $5$ epochs.

The unsupervised training pipeline for the task is depicted in Fig. 1. While training, the neural network takes two images, source ($I_{s}$) and target ($I_{t}$) as input and predicts $tx,\theta$. The image $I_{s}$ is then warped ($\hat{I}_{s}$) using fully differentiable spatial-transformer-networks (STN) [36] and $\hat{I}_{s}$ is obtained. Now, the neural network is penalized using back-propagation to force $\hat{I}_{s}\to I_{t}$. In the testing phase, the neural network predicts $tx,\theta$ on the test data and Mean-Absolute-Error (MAE) is reported between the predictions and the groundtruth $tx,\theta$, already stored during the data generation process. From the Table-I under the column Exp₁, it can be noticed that the network trained using $\mathcal{L}_{LMI}$ formulation exhibits better performance as compared to the rest of the loss functions. Fig. 2 shows a few qualitative results of this experiment. In this task, almost all of the losses perform equally well. It can be verified visually as well as from the quantitative results provided in the Table I.

This experiment consists of two ordinary grayscale images: source ($I_{s}$) and target ($I_{t}$). From the the image $I_{t}$, a significantly large rectangular region is cropped out and filled with zeros. The end goal of the experiment is to predict a mask $M_{s}$ which depicts similar and dissimilar regions between $I_{s}$ and $I_{t}$. To achieve this, a neural network is trained in an unsupervised manner. For the learning process, we generate a dataset of $1500$ images with $1000+500$ train-test split. The dataset is generated by randomly cropping a rectangular region from $I_{s}$ and filling th cropped region with zeros. The obtained image is referred as $I_{t}$. The cropping region has a size of $40\times 200$.

Fig. 4 shows the the unsupervised training framework for this task. We use UNet [35] network architecture. The network is trained to predict $M_{s}$ such that $I_{s}*M_{s}\to I_{t}$. This experiment is essentially a binary segmentation task, therefore, we adopt intersection-over-union metric (IoU) which is widely used to quantify the segmentation performance. In order to best evaluate different training losses, we provide IoU scores by thresholding the predicted mask at different confidence levels. It is done in order to examine the network’s capability to push the feature embeddings of similar and dissimilar regions significantly apart. In other words, a perfectly trained network will exhibit same IoU scores at various threshold levels. From the Table-I under Exp₂, it is clear that the proposed $\mathcal{L}_{LMI}$ formulation shows consistent and better performance over the other loss functions across various thresholding levels.

Fig. 3 shows a few qualitative results corresponding to this experiment. The results visualized are thresholded at $0.10$ (IoU_.10). It can be seen that $\mathcal{L}_{1}$ and $\mathcal{L}_{SSIM}$ performs worst whereas $\mathcal{L}_{2}$ and $\mathcal{L}_{LMI}$ performs marginally equally. The visual observation can also be verified quantitatively from the Table I, under Exp₂. In the quantitative results, $\mathcal{L}_{1}$ and $\mathcal{L}_{SSIM}$ performs worst in increasing order, as verified visually. On the other hand, $\mathcal{L}_{2}$ and $\mathcal{L}_{LMI}$ have only marginal difference, which can be also be verified visually by zooming in the results and examining the boundary of white region.

Depth estimation has been a long standing task in front of the computer vision community. This task has worldwide industrial importance because depth perception is a must for autonomous robotics and vehicles. Due to the advancements in the supervised learning techniques, researchers have developed several methods to predict depth using neural networks by employing supervised learning techniques. However, obtaining accurate groundtruth for supervised learning of this task is extremely challenging because it requires very expensive measurement instruments such as LIDARs. Hence, this task has gained considerable amount of attention in the recent years in order to develop unsupervised learning frameworks for this task. The existing methods make use of several loss functions in order to learn the depth effectively.

In this experiment, We demonstrate the learning of a neural network for the task of depth estimation using stereo images in an unsupervised manner. We form the seminal work for this task [9] as our basis. Our aim is not to show improvements in the datasets, instead we emphasis that how the DeepMI framework can be easily integrated for these real world applications. Hence, instead of a bigger dataset, we use $87$ grayscale rectified stereo images from KITTI [37] sequence-$113$. We select this sequence because it is quite difficult sequence from the perspective of this task.

The framework to carryout the experiment is shown in Fig. 6. It must be noticed that instead of the three different losses as in [9], we only use one loss for the evaluation. This is done in order to demonstrate that $\mathcal{L}_{LMI}$ provides stronger gradients and alone can lead to improved results. We report MAE between the predicted depth and the groundtruth measurements. From the Table-I under Exp₃, one can notice that the neural network trained using $\mathcal{L}_{LMI}$ outperforms other five variants by large margin. For the case of $\mathcal{L}_{1},\mathcal{L}_{2}$, the base_lr is lowered to $0.00001$ to prevent gradient explosion.

Fig. 5 shows a few qualitative results for this experiment. From the figure, it can be noticed that, both the $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ perform poorly, whereas $\mathcal{L}_{SSIM}$ performs quite better than them. This indicates the reason behind recent adaptation of $\mathcal{L}_{SSIM}$ over $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ for image matching purpose.

Further, it can be noticed that, visually the results of $\mathcal{L}_{LMI}$ are the most pleasing as well as consistent among all. As a comparison between the $\mathcal{L}_{SSIM}$ and $\mathcal{L}_{LMI}$, we can see that the depth estimations of the former contains texture copy [9] artifacts, whereas these are absent in latter. It can also be verified by taking a closer look at the car in the bottom right of the image. It can be seen that for the case of $\mathcal{L}_{SSIM}$, the depth map of the car has holes near edges and also contains severe texture copy artifacts near the number plate and other body area of the car. These artifacts, on the other hand are not present in the case of $\mathcal{L}_{LMI}$. This shows the clear effectiveness of $\mathcal{L}_{LMI}$ loss and the DeepMI framework.

Table-II shows the effect of the hyperparameter $N$ on each of the three tasks. For Exp₁, it can be noticed that the performance for all the four values of $N$ is same. It has to be the case because the images are binary in this task. For Exp₂, there is considerable drop in the performance for $N=3$ for IoU_0.05 (highlighted in red). It is because, the images in this case have multiple grayscale levels and details about which can not be efficiently captured. The same is the case for Exp₃ when $N=3$. Overall, we can see that $\mathcal{L}_{LMI}$ has significant advantages over other loss functions. Through our experimentations, it sufficient to keep the value of $N\leq 25$ for a signal having dynamic range $\in[0,255]$.

From the Table-I, it can be noticed that the information theory based measures are much more consistent as compared to the other losses. Also, it can be noticed that for the Exp₃, the $\mathcal{L}_{LMI}$ proves to be better over the case when $\mathcal{L}_{SSIM}$ and $\mathcal{L}_{2}$ are used together. Also, it is noticeable that $\mathcal{L}_{LMI}$ shows consistent and best scores amongst all variants of losses. In the experiments, our intention has not been to outweigh the existing losses, instead to mark the potential of information theory based methods in deep learning for real world applications.