UAN: Unsupervised Adaptive Normalization

Deep neural networks have shown significant advancements in various domains. However, they still face specific challenges, particularly in terms of accelerating learning and improving performance. The diverse nature of data often requires pre-training methods to normalize them to the same scale and eliminate biases. In a neural network with a single layer, normalizing input data accelerates convergence, thus enhancing performance [1]. This approach, however, is limited as contemporary neural networks consist of multiple layers. Input data is connected only to the first layer of the neural network, which does not guarantee that subsequent layers will benefit from input data normalization due to weight updates during the backward pass. To address this, normalization approaches have been proposed for layers not directly connected to input data, including activation normalization, weight normalization, and gradient normalization. Batch Normalization (BN) introduced by Ioffe and Szegedy [2] normalizes the output of each neural network layer to have a zero mean and unit variance property. This approach stabilizes gradient variation, accelerating convergence and improving deep neural network performance. Despite its effectiveness, BN has limitations such as mini-batch dependency, restricting its utility for small batches, and the weak assumption that samples within the mini-batch are from the same distribution, which is not always true, especially with real-world data.

Layer Normalization (LN) [3], Instance Normalization (IN) [4], and Group Normalization (GN) [5] were proposed to address mini-batch dependency issues. Mixture Normalization (MN) [6] presents an approach based on the assumption of Gaussian mixture models (GMM) on input data to overcome the distribution assumption made by BN. MN employs a dual-stage process for normalizing neuron activations: (i)Expectation-Maximization (EM) algorithm [7] is initially used to deduce parameters for each mixture component; (ii) in the training of deep neural networks, samples linked to the same component are normalized utilizing the parameters specific to their component. This approach has demonstrated quicker convergence and enhanced performance in neural networks, particularly in tasks of supervised classification. However, MN’s dependence on the EM algorithm as a preprocessing step could be a limiting factor due to its computational intensity. Additionally, MN’s static parameters might not be sufficiently adaptive to the shifting distributions of neuron activations during the learning phase.

To tackle this challenge, we introduce Unsupervised Adaptive Normalization (UAN), a streamlined normalization approach. UAN is grounded in the assumption that data are governed by a Gaussian mixture model in a latent space. The objective is to discover this space during the deep neural network learning process by normalizing neuron activations. The parameters of each mixture component called ”cluster”, are estimated as weights of the deep neural network during the backpropagation phase, enabling the construction of mixture components tailored to the target task addressed by the deep neural network. This one-pass approach provides an adaptive data representation based on the clustering methodology, enhancing separability and facilitating deep neural network learning. Notably, UAN demonstrates superior results in terms of convergence and deep neural network performance when compared to BN and MN.

In summary, the main contributions of this work are as follows:

• •

  Simultaneous clustering for normalization and deep Learning: Unsupervised Adaptive Normalization(UAN) integrates clustering and deep neural network learning in a single step. By normalizing neuron activations based on the Gaussian mixture model assumption, UAN estimates parameters for each mixture component, improving data representation for the target task and boosting gradient stability.

• •

  Efficiency and complexity reduction: UAN stands out by simplifying the architecture and eliminating the preprocessing clustering phase using EM used in Mixture Normalization. UAN adapts normalization to the target task, determining mixture component parameters during backpropagation as model weights. This streamlined approach demonstrates superior convergence and performance compared to BN and MN.

• •

  Enhanced convergence and performance: UAN surpasses Batch Normalization and Mixture Normalization regarding convergence and deep neural network performance. Its adaptive data representation, achieved through simultaneous clustering and learning, enhances separability and accelerates the learning process.

To ensure clear understanding and consistency in our model, and to facilitate comparison with existing work, we adopt the same notations as those used in [6]. Let $x\in\mathbb{R}^{N\times C\times H\times W}$ neurons activations in a convolutional neural network layer, where $N$, $C$, $H$, and $W$ represent the batch, channel, height, and width axes, respectively. Batch Normalization (BN) [2] standardizes a mini-batch $B=\{x_{1:m}:m\in[1,N]\times[1,H]\times[1,W]\}$ of $m$ samples with $x$ flattened across all axes except the channel. The standardization is expressed as:

where $\mu_{B}=\frac{1}{m}\sum_{i=1}^{m}x_{i}$, $\sigma^{2}_{B}=\frac{1}{m}\sum_{i=1}^{m}(x_{i}-\mu_{B})^{2}$ are respectively the mean and variance. $\epsilon>0$ is a small number to prevent numerical instability. To mitigate the constraints imposed by standardization, learnable parameters $\gamma$ and $\beta$ are introduced, transforming activations as follows:

During inference, population statistics are essential for deterministic inference. They are computed by averaging over training iterations:

If the mini-batch samples come from the same distribution, the transformation in equation (1) yields a distribution with zero mean and unit variance. This constraint stabilizes activation distribution, aiding training.
The mini-batch-wise approach allows for a more suitable data representation, enabling faster processing and enhancing deep neural network performance in terms of convergence. Despite its efficacy, BN’s performance is contingent on mini-batch size, and the disparity between training and inference hinders its application in intricate networks (e.g., recurrent neural networks). To address these challenges and handle parameter estimation on unrelated samples, numerous variants and alternative methods have been proposed.
Various extensions to batch normalization (BN) have emerged, including Layer Normalization (LN) [3], Instance Normalization (IN) [4], Group Normalization (GN) [5], and Mixture Normalization (MN) [6]. These alternatives share a common transformation $x\rightarrow\hat{x}$, applied to the flattened $x$ across the spatial dimension $L=H\times W$, expressed as:

where $B_{i}=\{j:j_{N}\in[1,N],j_{C}\in[i_{C}],j_{L}\in[1,L]\}$, and $i=(i_{N},i_{C},i_{L})$ indexing the activations $x\in\mathbb{R}^{N\times C\times L}$.

Layer Normalization (LN): Alleviating inter-dependency among batch-normalized activations, LN computes mean and variance using input from individual layer neurons. LN benefits recurrent networks but may face challenges with convolutional layers due to spatial variations in visual data. Formulated as Equation 4 with $B_{i}=\{j:j_{N}\in[i_{N}],j_{C}\in[1,C],j_{L}\in[1,L]\}$.

Group Normalization (GN): GN partitions neurons into groups and autonomously standardizes layer inputs for each sample within these groups. In visual tasks with small batch sizes, GN is useful for applications like object detection and segmentation. GN calculates statistics along the $L$ axis, restricting computation to subgroups of channels. Equivalent to IN when $G=C$ and identical to LN when $G=1$.

Mixture Normalization (MN): In deep neural networks (DNNs), neuron activations exhibit multiple modes of variation due to non-linearities, challenging the Gaussian distribution assumption of batch normalization (BN) [2]. Mixture Normalization (MN) [6] approaches BN from a Fisher kernels perspective, using a Gaussian Mixture Model (GMM) to normalize based on multiple means and standard deviations associated with the clusters. In convolutional neural networks, MN outperforms BN in terms of convergence and accuracy in supervised learning tasks. Implemented in two stages: estimation of GMM parameters using the Expectation-Maximization (EM) algorithm [7], followed by neuron activations normalization and aggregation using posterior probabilities during deep neural network training.
The distribution $p_{\theta}$ characterizing the data is represented as a parameterized Gaussian Mixture Model (GMM). Let $x\in\mathbb{R}^{D}$, and $\theta=\{\lambda_{k},\mu_{k},\Sigma_{k}:k=1,...,K\}$,

where

represents the $k^{th}$ Gaussian in the mixture model $p(x)$, with $\mu_{k}$ as the mean vector and $\Sigma_{k}$ as the covariance matrix.
The posterior probability that $x$ is generated by the $k^{th}$ component in the mixture model is defined as:

Based on these assumptions and the general transform in Equation (4), the Mixture Normalizing Transform for a given $x_{i}$ is defined as

given

where

and

represents the normalized contribution of $x_{i}$ with respect to the mini-batch $B_{i}=\{j:j_{N}\in[i_{N}],j_{C}\in[i_{C}],j_{L}\in[1,L]\}$ in estimating the statistical measures of the $k^{th}$ Gaussian component.

Mixture Normalization (MN) effectively stabilizes deep neural network training, but it encounters limitations in its two-stage process, which involves the Expectation-Maximization (EM) algorithm, introducing complexities. The static parameters of MN do not adapt to the changing distributions of neuron activations during learning and weight updates. Recognizing the need for a more adaptive and streamlined approach, we propose Unsupervised Adaptive Normalization (UAN). In response to these challenges, UAN operates as a one-stage algorithm, embodying an ”unsupervised” approach where prior knowledge of mixture components is absent. In this context, ”clusters” refers to a mixture component within the latent space modeled as a Gaussian mixture. UAN estimates mixture component parameters as network weights, dynamically adjusting to the target task and evolving alongside deep neural network weights. This unique one-stage algorithm achieves mixture parameter estimation and target task resolution during a unified training process. Unlike MN, where normalization parameters remain constant, UAN updates these parameters throughout learning, ensuring the formation of adaptive clusters in line with the evolving distribution of neuron activations.

In implementing this approach, we assume the presence of a latent space where clusters are modeled as a mixture model. Specifically, we consider a Gaussian mixture model with parameters $\theta=\{\lambda_{k},\mu_{k},\Sigma_{k}:k=1,...,K\}$, where $K$ denotes the total number of defined clusters. It is important to note that the assumption of a Gaussian distribution is not mandatory; alternative distributions can be explored. In such cases, the corresponding parameters are estimated accordingly. The normalization of each neuron activation takes place using Equation 8, where all clusters contribute to the normalization process for each activation.

Throughout the training of the deep neural network, cluster parameters are set at the beginning to fulfill the conditions $\sum_{i=1}^{K}\lambda_{k}=1$ and $\sigma_{k}>0$, where these parameters act as weights. To adapt these cluster parameters, we use backpropagation and a moving average technique within mini-batches (Figure 1 and Algorithm 1). The former aligns neuron activations with the target task, dynamically evolving parameters alongside deep neural network weights. Simultaneously, the latter serves as an approximation of the EM approach tailored to neuron activations. This one-stage process in UAN allows the simultaneous estimation of mixture component parameters and resolution of the target task in deep neural networks. Unlike methods with static parameters that struggle to adapt during learning, UAN demonstrates adaptability and efficiency in overcoming evolving neuron activation distributions and weight updates.

In the upcoming experiments, our goal is to evaluate the performance of the Unsupervised Adaptive Normalization (UAN) method and compare it with Batch Normalization (BN) and Mixture Normalization (MN). We suggest applying UAN to tasks involving classification (Section IV-B), and domain adaptation (Section IV-C).
UAN, a deep neural network layer equipped with Gaussian mixture parameters, undergoes a distinctive initialization process during training. The mean ($\mu$) parameters are set with random values uniformly distributed between $-1.0$ and $1.0$, establishing a varied starting point conducive to capturing central tendencies. The standard deviation ($\sigma$) parameters receive initialization with strictly positive random values within the range of 0.001 to 0.01, ensuring numerical stability and facilitating the exploration of diverse data patterns. Additionally, the mixing coefficients ($\lambda$) parameters are initialized with values spanning $0.01$ to $0.99$. These values are normalized to ensure their collective sum equals $1$, thereby establishing a well-balanced and probabilistically sound foundation for incorporating prior knowledge within the Gaussian mixture model.

Compared to Batch Normalization [2] and Mixture Normalization [6], the computational overhead of adaptative normalization arises from estimating cluster parameters. Let $B$ represent a batch of activations in $\mathbb{R}^{d}$, where $d$ is the dimensionality of activations.
For Batch Normalization, mean ($\mu_{B}$) and standard deviation ($\sigma_{B}$) are computed considering the entire batch $B$, leading to a computational complexity of $\mathcal{O}(d)$. Learning parameters $\gamma$ and $\beta$ for each batch dimension adds a parameter learning complexity of $\mathcal{O}(d)$. Thus, the overall complexity, accounting for both computational and parameter aspects, is $\mathcal{O}(d)$ for the entire batch $B$.
Mixture normalization faces computational challenges in estimating Gaussian mixture model (GMM) parameters accurately. Using K-means++ for seeding, complexity for batch $B$ with dimension $d$ is $\mathcal{O}(d\cdot K\cdot\log(K))$. In the EM algorithm, per-iteration complexity is $\mathcal{O}(d\cdot n\cdot K)$, mainly determined by initialization, also $\mathcal{O}(d\cdot K\cdot\log(K))$. Like Batch Normalization, mean ($\mu_{k}$) and standard deviation ($\sigma_{k}$) are computed per cluster $k$, $\mathcal{O}(K\cdot d)$. Parameter learning complexity ($\gamma$, $\beta$) per batch dimension is $\mathcal{O}(d)$. Overall complexity, considering both computational and parameter aspects, is $\mathcal{O}(d\cdot n\cdot K)$ for batch $B$.

In this paper, we introduced Unsupervised Adaptive Normalization (UAN), a novel one-stage unsupervised normalization technique for deep neural networks. UAN provides mixture components during the training of deep neural networks, tailoring them specifically for the target task. This method is efficient as it computes cluster parameters simultaneously with the neural network’s weight adjustments, thus minimizing computational overhead compared to multi-stage normalization approaches or classical ones. UAN stabilizes neural network training across a range of applications, including supervised classification, and domain adaptation, proving its versatility and effectiveness in various contexts. In future work, we plan to further validate the efficacy of our approach on different context as Generative Adversarial Networks (GANs) and different types of data, particularly focusing on its adaptability to architectures that handle multimodal data.