Consensus Function from an $L_{p}^{q}-$norm Regularization Term for its Use as Adaptive Activation Functions in Neural Networks

Neural Networks are widely used for different applications, such as identification, security, defense, face and speech recognition, healthcare, or weather and stock market prediction, among others, [1, 2, 3, 4, 5, 6, 7, 8]. Although the architectural design of the neural networks may amply vary depending on the particular application, they all share the basic intrinsic foundation of mimicking the behavior of a biological brain by connecting several computational nodes, the so-called neurons, that perform simple non-linear operations, with the aim of developing complex practices. Mathematically, neural networks are a powerful tool that can basically describe any relationship between any given input-output set of data, potentially modeling any imaginable non-linear mapping. Despite its questionless capabilities, neural networks rely on the simplicity of connecting several layers of neurons that enchains two straightforward operations: a linear combination of the results provided from the neurons of a previous layer, and a non-linear function, which outcome is passed to the next layer. The number of layers, the number of neurons per layer, and the type of non-linear functions implemented to the neurons of each layer, the so-called activation or transfer function, is part of the architectural design of the neural network. The larger the network, the more complex capabilities and highly non-linearities could be modeled, [9].

The fitting of the data to the designed network is done by adapting the weights and biases that define the linear combination on the first operation done in each neuron during the training process. However, the selection of the activation functions generally is reduced to a limited set of non-linear functions, such as the sigmoid, rectified linear units, max-pooling, etc. This selection has, a priori, little or no influence from the dataset that is used for training the network. The exploration of adaptive activation functions to increase the flexibility of the neural networks during the training process is an active topic of research. [10] points out the importance that the activation functions have on the neural network performance and review some initial attempts to find an optimal distribution of activation functions from a reduced set of options, by using evolutionary algorithms. [11] defines the maxout activation function, which extracts the maximum of a set of linear functions, arbitrarily approximating any convex function. [12] proposes adaptive piecewise linear units that are learned using gradient descend during the training process, enabling the approximation of both convex and non-convex functions; and [13] introduces an $L_{p}$ unit, which performs the $p-$norm of the normalized input data, adapting the values of $p$ during the training process and generalizing the pooling operator, for which the maxpooling and maxout units are particular cases of it. These methods progress towards neural network architectures whose training process learns, not only the relationships among the neurons, but also the neurons themselves. However, they do not totally generalize the activation functions and require software applications to approximate the gradients during the training process, thus, a better understanding of the underlying functional analysis of generalized activation functions is still required.

In this paper, we present a parametric activation function whose parameters are learned during the training process and that can adapt to mimic most of the common activation functions regularly used. On one hand, this imply to increase the space of training parameters, but on the other hand, this generalizes the global structure of the neurons, since their activation functions do not need to be predefined. The definition of the proposed activation function comes from the functional analysis of the consensus variable when performing a linear optimization problem with a general $L_{p}^{q}-$norm regularization term, via the Alternating Direction Method of Multipliers (ADMM), as it is described in Sect. II. The mathematical development for the use of the proposed activation function, and the analysis for the computation of the gradients for the optimization of the implied parameters is detailed in Sect. III. Section IV presents some preliminary results, showing that the use of these adaptive activation functions reduces both the training and testing errors when they are applied on shallow neural networks, compared to equivalent regular feedforward networks in which the activation functions are predetermined. Finally, Sect. V concludes the paper.

In this work, we propose an implicit, parametric, non-linear activation function that can adapt its shape during the training process to create a flexible neural network. This function appears as the result of the research carried out by the same authors on the field of optimization of linear under-determined problems, via the ADMM, [14, 15, 16, 17, 18, 19, 20, 21, 22]. Specifically, the linear problem $\textbf{Hu}=\textbf{g}$ can be optimized with an $L_{p}^{q}-$norm regularization term as follows:

On one hand, $\textbf{H}_{i}$ is each of the $M$ submatrices in which $\textbf{H}\in\mathbb{C}^{N_{m}\times N_{p}}$ is divided by rows, where $N_{m}$ is the number of data and $N_{p}$ the number of unknowns, usually having $N_{p}\gg N_{m}$. $\textbf{g}_{i}$ is each of the $M$ subvectors in which $\textbf{g}\in\mathbb{C}^{N_{m}}$ is divided; and $\textbf{u}^{i}\in\mathbb{C}^{N_{p}}$ is each of the $M$ independent solutions that can be generated for each division. The optimization process using the ADMM algorithm imposes the agreement of all solutions via the consensus variable $\textbf{v}\in\mathbb{C}^{N_{p}}$, as depicted in Fig. 1.

On the other hand, the regularization term $\lambda\left\|\textbf{v}\right\|_{p}^{q}$ imposes a certain structure in the particular solution sought for the problem, where $\lambda$ is just a design hyperparameter to weight the importance of this type of structure in the optimization process. The parameter $p$ induces a preferred direction of searching for the solution. In this sense, a norm-1 regularization (lasso) imposes a sparse solution, [22, 23, 24], a norm-2 regularization (ridge regression) seeks for the solution with minimum energy [25, 26, 27], meanwhile a norm-infinity minimizes the magnitude of the greatest component [28, 29, 30]. As a more general expression, the bridge regression regularization term, of the form $\|\textbf{v}\|_{p}^{p}$, accounts for the previously mentioned lasso for $p=1$ and ridge regression for $p=2$, and performs a smooth version of the elastic net regularization for $1<p<2$, [31, 32, 33, 18, 34, 35]. Meanwhile, the parameter $q$ defines the way the distances are measured, distorting the metric space. Particularly, for large values of $p$, the distance between two different points tends to be very similar, and specifically, $p=\infty$, leads to the discrete distance.

The optimization problem on (1) is solved by introducing the Lagrangian multipliers or dual variables $\textbf{s}^{i}$, one for each constrain, and sequentially optimizing $\textbf{u}^{i}$, v, and $\textbf{s}^{i}$, in this order. See Ref. [21] for a detailed explanation. The optimization of the primal and dual variables $\textbf{u}^{i}$ and $\textbf{s}^{i}$ are always the same regardless the type of regularization, since this one only affects the definition of the consensus variable v. Its optimization, therefore, comes from the computation of the gradient of the Lagrangian with respect to each component $v_{l}$:

where $\rho$ is the augmented parameter. Notice that the input variable in the expression (2) is $\left(\bar{u}_{l}+\bar{s}_{l}\right)$, namely, the sum of the average of the $l-th$ components of the variables $\textbf{u}^{i}$ and $\textbf{s}^{i}$, while the output is, indeed, $v_{l}$. Consequently, this is, in general, a non explicit expression except for some particular values of $p$ and $q$, and it has to be solved by iteratively reweighting the expression with the previous iterations of $v_{l}$.

Definitely, the selection of the values of $p$ and $q$ for the regularization term will determine the shape of this implicit function. Interestingly, the behavior of this function resemble the definition of the common activation functions, such as the ReLU ($p=1$), PReLU ($1<p<2$), linear ($p=2$), sigmoid ($2<p=q<\infty$), or soft clipping ($p=q=\infty$), as depicted in Figure 2, [36, 37]. By redefining $a_{l}=\bar{u}_{l}+\bar{s}_{l}$ as the input in Eqn. (2), dropping the component dependence $l$, grouping and redefining the constant $\lambda:=\frac{\lambda}{\rho M}>0$ as a tuning hyperparameter, and for the particular case of $1<p=q<\infty$, we define the implicit, non-linear, parametric activation function as follows:

which defines the non-linear mapping

where $a$ is the input and so-called activation variable, $p$ is the learning parameter of the function, and $v$ is the output of the activation function $f$. We call the $\bm{p}$-network to the neural networks designed with this type of activation function. The general case for $p\neq q$ would lead to the denominated $\bm{pq}$-network, which will extend this work in the future.

The top part of Fig. 3 depicts an example of a fully connected $p-$network of $m$ layers with $r^{k}$ neurons per layer, for $k=1,\dots,m$ ($k$ indicates a super index, not a power). Additionally, the layer $0$ is considered the input layer, and does not have neurons properly, but is it just the input data; the layer $m$ is the output layer and all the others are the intermediate or hidden layers. The parameters that define the network include the weights of the synapses that connect the different layers and the internal parameter of each neuron for defining the activation function. The benefit of using a parametric activation function is that the non-linear general expression is the same in all neurons. In this way, as represented at the bottom part of Fig. 3, the activation variable $a_{j}^{k}$, namely, the input to the activation function of the $j-th$ neuron of layer $k$, can be expressed as follows:

where $w_{j0}^{k}=b_{j}^{k}$ is the bias of the $j-th$ neuron of layer $k$, thus considering $v_{0}^{k-1}=f\left(a_{0}^{k-1};p_{0}^{k-1}\right)=1$. The output of the neuron is $v_{j}^{k}=f(a_{j}^{k};p_{j}^{k})$, and notice that $v_{s}^{0}=f\left(a_{s}^{0};p_{s}^{0}\right)=x_{s}$ corresponds to the input data to the network, where $r^{0}$ is its dimensionality, and $v_{s}^{m}=f\left(a_{s}^{m};p_{s}^{m}\right)=\hat{y}_{s}$ is the output data, with dimensionality $r^{m}$.

The network has been implemented in Matlab and tested to validate its correct performance and to see how the activation functions adapt to the given data. The training is performed until reaching a maximum of number of iterations or when the training error goes below maximum error threshold.

This paper has proposed a new structure for the design of neural networks by defining a parametric non-linear activation function that can adapt its shape during the training process. This allows to configure the whole network with one single activation function definition, not having to predefine its expression for each layer, and also allowing different shapes for the neurons within the same layer after training. This increases the space of parameters for optimization, but provides more flexibility and generalization on the architecture of neural networks.

The proposed activation functions comes as the expression of the consensus variable when optimizing a linear problem with an $L_{p}^{q}-$norm regularization term. This expression is, in general, implicit, meaning that there is no an explicit expression of the output variable given the input value. Although this is a challenge for its use, a detailed analysis is provided for its evaluation by the use of a reweighted iteration process. In this paper, the particular case of $1<p=q<\infty$ is implemented. The evaluation of the feedforward pass of the network with this activation function, as well as a detailed explanation of the training process via the backpropagation method for the optimization of the parameter $p$ and the weigths and biases is also provided for completeness.

Preliminary results for both regression and classification show that the proposed $p-$network reduces the error of the testing sets when comparing with an equivalent regular neural network with the same number of layers and neuron, and with predefined activation functions that are as similar as possible as the $p-$network activation function with the initial values of $p$ at the beginning of the training process.

Future work will implement the $pq-$networks, that is, allowing the distinction of the parameters $p$ and $q$ during the training process to account for a larger set of possible shapes of the activation function. The application of this methodology to more complex networks, such as Deep, Convolutional, or Recurrent Neural Networks will open the way to the definition of more general and flexible neural networks. It is expected, consequently, a boost in the performance of these neural networks in terms of error reduction when using the proposed adaptive activation functions.

This work has been funded by the Department of Energy (Award # DE-SC0017614).