Regularized Flexible Activation Function Combinations for Deep Neural Networks

In this study, we implement a general form of parameterized activation functions linearly combined by different activation functions as components[23, 24]. We further extend the existing works by introducing trainable parameters in each component, which is essential for building trainable activations with bounded domain. We assume that the parameters in the combined activation functions can be different for each neuron, which can be trained during the main training process of the model parameters with back propagation.

$$\begin{split}o_{i}(z,\boldsymbol{\alpha}^{i},\boldsymbol{\beta}^{i})&=\sum_{k=1}^{K}\alpha_{ik}f_{k}(z,\boldsymbol{\beta}_{ik}),\\ \quad\sum^{K}_{k=1}\alpha_{i,k}&=1,\quad 0\leq\alpha_{i,k}\leq 1\quad\forall k,i\end{split}$$

(2)

where $i$ indexes the neuron, and $z=z_{l}=\boldsymbol{W_{l}}X_{l-1}+\boldsymbol{b_{l}}$ is the input of the activation layer indexed by $l$. This means that, at each neuron $i$, it is possible to have its own set of parameters $\boldsymbol{\alpha}^{i}=[\alpha_{i1},...,\alpha_{iK}]^{T}$ and $\boldsymbol{\beta}^{i}=[\boldsymbol{\beta}_{i1},...,\boldsymbol{\beta}_{iK}]$ where $\alpha_{ik}$ is the combination weights and $\boldsymbol{\beta}_{ik}$ is the activation parameter vector for the $k$-th component activation $f_{k}$, respectively. Thus Eq. (2) defines a form of activation function as a linear combination of a set of basic parameterized non-linear activation functions $f_{k}(z,\boldsymbol{\beta}_{k})$ with the same input $x$ to the neuron. Normally, we require $0\leq\alpha_{i,k}\leq 1$ for all $k$ and $i$ to ensure that the output is strictly bounded between 0 and 1. This setting will take advantage of the low computational costs of existing activation functions, while it will be much easier to implement weights normalization when we need a bounded activation function.

Since the specific activation function corresponding to each neuron only depends on its own activation parameters, the back propagation of these activation parameters by stochastic gradient descent can be done as follows:

$$\begin{split}&\alpha_{ik}\rightarrow\alpha_{ik}-\gamma\frac{\partial L}{\partial\alpha_{ik}}=\alpha_{ik}-\gamma\frac{\partial L}{\partial o_{i}}\cdot f_{ik}(z,\boldsymbol{\beta}_{ik})\\ &\boldsymbol{\beta}_{ik}\rightarrow\boldsymbol{\beta}_{ik}-\gamma\frac{\partial L}{\partial\boldsymbol{\beta}_{ik}}=\boldsymbol{\beta}_{ik}-\gamma\frac{\partial L}{\partial o_{i}}\cdot\alpha_{ik}\frac{\partial f_{ik}(z,\boldsymbol{\beta}_{ik})}{\partial\boldsymbol{\beta}_{ik}}\end{split}$$

(3)

where $L$ is the loss function of all parameters of the model, $i$ is the index of the hidden neuron with output $o_{i}$ and $k$ is the index of combined flexible activation functions. Here we use a simplified expression that does not include the indices of layer and training examples in each mini-batch. $\gamma$ is the learning rate of gradient descent for all the parameters in activation functions. With the gradients given by $\partial L/\partial\alpha_{ik}$, adaptive optimizers such as AdaGrad, Adam and RMSProp can also be applied. In general, gradient descent approach and its derivatives can push the activation parameters toward the direction that minimizes the empirical risk of the model on training data.

To build effective combinations with the general form given by Eq. (2), we introduce the following three principles for selecting the components:

• •

  Principle 1: Each component should have the same domain as the baseline activation function.

• •

  Principle 2: Each component should have an equal range as the baseline activation function.

• •

  Principle 3: Each component activation functions should be expressively independent of other component functions with the following definition.

Definition 1: If a component activation function $f_{k}$ is expressively independent of a set of other component functions: $f_{1},...,f_{n}$, there does not exist a set of combination coefficients $\alpha_{1}$,…,$\alpha_{n}$, inner activation parameters $\beta_{1}$,…$\beta_{n}$, parameters of the previous linear layers $\boldsymbol{W^{\prime}}$, $\boldsymbol{b^{\prime}}$ such that for any input $X$, activation parameters $\boldsymbol{\beta_{k}}$, and parameters of the previous linear layer $\boldsymbol{W_{k}}$, $\boldsymbol{b_{k}}$, the following equation holds:

$$\begin{split}&f_{k}(z_{k},\boldsymbol{\beta_{k}})=f_{k}(\boldsymbol{W_{k}}X+\boldsymbol{b}_{k},\boldsymbol{\beta_{k}})\\ &=\sum_{i=1}^{n}\alpha_{i}f_{i}(\boldsymbol{W^{\prime}}X+\boldsymbol{b^{\prime}},\boldsymbol{\beta_{i}})=\sum_{i=1}^{n}\alpha_{i}f_{i}(z^{\prime},\boldsymbol{\beta_{i}})\end{split}$$

(4)

Proposition 1: For a single-layer network with $m$ neurons, if a component activation function $f_{k}$, which is not expressively independent of other components, is excluded, we need at most $2m$ neurons to express the same mapping exactly.
Proof: For any original combined activation function: $F=\sum_{i=1}^{n}\alpha_{i}f_{i}$. Assume that $f_{k}$ is not expressively independent. If we exclude $f_{k}$ from the combination and get $F^{\prime}=\sum_{i=1,i\neq k}^{n}\alpha_{i}f_{i}$, then for any input $X$, parameters of the previous linear layer: $W$, $b$, and activation parameters: $\{\beta\}_{i=1}^{n}$ of a specific neuron, we have:

$$F(WX+b,\beta)=\sum_{i=1,i\neq k}^{n}{\alpha}_{i}f_{i}(WX+b,\beta_{i})+f_{k}(WX+b,\beta_{k})$$

(5)

and there exist $W^{\prime}$, $b^{\prime}$ and $\{\beta\}_{i=1,i\neq k}^{n}$ such that:

$$\begin{split}F(WX+b,\beta)&=\sum_{i=1,i\neq k}^{n}{\alpha}_{i}f_{i}(WX+b,\beta_{i})\\ &+\sum_{i=1,i\neq k}^{n}{\alpha}^{{}^{\prime}}_{i}f_{i}(W^{\prime}X+b^{\prime},\beta_{i}^{{}^{\prime}})\end{split}$$

(6)

which can be expressed by two neurons with function $\{f\}_{i=1,i\neq k}^{n}$ with corresponding weights and bias in the previous linear layer. Therefore, for a single-layer network with $m$ neurons, we need at most $2m$ neurons without $f_{k}$ to express the any mappings by the original function.

The first two principles are aiming at keeping the same ranges and domains of the information flow with the mapping in each layer. The third principle is aiming at reducing the redundant parameters that do not contribute to the model expressiveness even with limited number of units. For example, $\sigma_{1}(z)=1/(1+e^{-\beta z})$ is not expressively independent with $\sigma(z)=1/(1+e^{-z})$ since when $\boldsymbol{W^{\prime}}=\beta\boldsymbol{W}$, we have $\sigma(\boldsymbol{W^{\prime}}X)=\sigma_{1}(\boldsymbol{W}X)$. Therefore, the combined activation $a(z,\beta)=\alpha_{1}\sigma(z)+(1-\alpha_{1})\sigma(\beta z)$ will not be a good choice. Based on this, we can then design the combined trainable activation functions for both bounded or unbounded domains.

Sigmoid and Tanh activation functions are widely used in recurrent neural networks, including basic recurrent nets and recurrent nets with cell structure such as LSTMs and GRUs. For the sigmoid function, the output should be in the domain of $[0,1]$, while for tanh the output should be in $[-1,1]$. In the first case, one basic choice is:

$$o(z;\alpha,\beta)=\alpha\cdot\sigma(z)+(1-\alpha)\cdot f(z;\boldsymbol{\beta})$$

(7)

where $0\leq\alpha\leq 1$ and

$$f(z;\boldsymbol{\beta})=\begin{cases}0&\quad\text{if }z<-\frac{1}{2\beta}\\ \beta z+\frac{1}{2}&\quad\text{if }-\frac{1}{2\beta}\leq z\leq\frac{1}{2\beta}\\ 1&\quad\text{if }z>\frac{1}{2\beta}\end{cases}$$

(8)

In Eq. (7), $f(z;\beta)$ can be considered as a combination of two Ramp functions bounded between 0 and 1 with parameter $b$. The shapes of a sample of combined activation with Eq. (7) are shown in Fig. 1. Similarly, we can build a function with the same boundary as $\tanh$ function, and use the corresponding combination to replace $\tanh$ in the LSTM cell. Consequently, for each combined flexible activation function, there are two parameters to be optimized during the training process. By combining the original activation function $\sigma$ with another function $f(z;\beta)$ with the same boundary using normalized weights, the model can be trained with flexible gates and have the potential to achieve better generalization performance.

The outputs of ReLU function is unbounded on both sides, while the derivative with respect to the inputs is a step function. To build more flexible activation in the condition when ReLU function is used, we can make a weighted combination between ReLU and other non-linear functions with unbounded ends. Considering the goodness of candidate components as well as the principles proposed in Section  II-A, we introduce the following combined form:

$$\begin{split}o(z;\alpha,\beta)&=\alpha\text{ReLU}(z)+\beta\text{Elu}(z)\\ &+(1-\alpha-\beta)(-\text{Elu}(-z))\end{split}$$

(9)

This is actually a weighted combination of ReLU function and two ELU-family functions, which introduces non-linear functions with different shapes in both positive and negative region. If we bound all the wights between 0 and 1, each of the three components $\text{ReLU}(z)$, $\text{Elu}(z)$ or $-\text{Elu}(-z)$ can not be fully expressed by the combination of the others with certain model parameters. Figrue  7 demonstrates the examples of P-E2-ReLU given by Equation  (9) with different activation parameters.

In addition, if we set $\beta=1-\alpha-\beta=\frac{1-\alpha}{2}$, the combined two ELU-family terms will be symmetric with respect to point $(0,0)$, which is similar to the combination of $\tanh$ and ReLU proposed in [24] but can be more naturally implemented for unbounded domains. Also, we can consider other combinations such as $\{\text{ReLU},\text{Elu}(z)-\text{Elu}(-z)\}$, $\{\text{Id},\text{Elu(z)}-\text{Elu(-z)}\}$ and $\{\text{ReLU},\text{Elu}(z;\beta)-\text{Elu}(-z;\beta)\}$, where “Id” represent identity functions, and $\beta$ is the trainable parameter of ELU (default is 1.0). We categorize them all as in “P-E2-X” family. In this study we will focus on the form provided by Equation  (9), while the others could achieve even better performances in some tasks.

Similar to the weights decay regularisation for model weights in NN models, we introduce regularisation terms for parameters in activation functions to avoid over-parameterization during learning process. When we set the summation of each component’s weights in each flexible activation function to 1, it is not suitable to implement a L1 or L2 norm on the absolute value of activation weights. Instead, we use the L2 norm for the absolute difference between each specific activation parameter and the mean of corresponding parameters in the same layer. In addition, we introduce another L2 regularisation term controlling the difference between the trained parameters and the initial parameters of benchmark activation function. This can make sure that the benchmark is actually a specific case of flexible activation, while the variations can be learned to adapt to the training dataset and controlled by these regularisation effects. Thus, the cost function can be given by Equation  (10), where $L_{0}$ is the predictive loss, $\alpha_{ijk}$ refers to the $k$th activation parameter $\alpha_{k}$ for $i$th element in $j$th layer, $\bar{\alpha}_{jk}$ is the average value of $\alpha_{k}$ in $j$th layer, $\alpha_{k0}$ is the combination coefficient of $k$th component in basic or standard activation functions (e.g. Sigmoid).

$$\begin{split}&L=L_{0}+\delta_{1}\sum_{j}\frac{\lambda_{j}}{m_{j}}\sum_{i}\sum_{k}||\alpha_{ijk}-\bar{\alpha}_{jk}||^{2}\\ &+\frac{\delta_{2}}{n}\sum_{i}\sum_{j}\sum_{k}||\alpha_{k0}-\alpha_{ijk}||^{2}\\ &+\frac{\delta_{3}}{n}\sum_{i}\sum_{j}\sum_{k}(||\text{ReLU}(\alpha_{ijk^{*}}-(1-\Delta))||^{2}\\ &+||\text{ReLU}(-\Delta-\alpha_{ijk^{*}})||^{2})+\text{other terms}\end{split}$$

(10)

In the first regularization term, $m_{j}$ is the number of neurons in $j$th layer, while $\lambda_{j}$ is the layer-wise regularisation coefficients, and $\delta_{1}$ is mutual regularization coefficient. $\delta_{2}$ and $\delta_{3}$ are the regularization coefficients for the second and third regularization terms, and $n$ is the number of units in the whole network architecture. We can consider the first two regularization terms as prioris. For the first one, we consider that in the layer structure of deep neural networks, usually different layer is learning different level of patterns, which could be in favor of using similar activation functions in each layer. we call it as “towards-mean regularization”. Meanwhile, the second regularization terms can be considered as another priori in assuming that the initial activation functions are good enough and the learned activation parameters should not differ too much from the initial values. We call this as “towards-default regularization”. In addition, the third regularization term is introduced to ensure the combination parameters are bounded between 0 and 1, where $\Delta$ is a small number. We set $\delta_{3}$ to be a relatively large value (e.g. $\delta_{3}=1$) to make this as a strong constraint. The regularization coefficients $\delta_{1}$ and $\delta_{2}$ can be implemented for controlling the flexibility in activation parameter space. The effects of these two regularization terms can be summarized as follows:

• •

  Small $\delta_{1}$, small $\delta_{2}$: The activation functions can be trained independently for different neurons in different layers with high flexibility.

• •

  Large $\delta_{1}$, small $\delta_{2}$: The activation functions in the same layer will be constraint to be close to each other, while the activation in different layers can be different.

• •

  Large $\delta_{2}$: The activation function will be constraint to be close to the default fixed ones.

Therefore, with optimized activation coefficients, the models can achieve performances at least no worse than models with fixed default activation. Moreover, gradient-based methods may also be applied in optimizing these regularization coefficients [27, 28].

Since we introduced a set of new trainable parameters, the way to initialize them could make a difference. The default choice is to set the coefficients of benchmark activation functions to 1 and train the other terms from 0. However, this setting could suffer from the risk of being trapped in local optimal. Other solutions include the following:

• •

  Treat the initialization values as hyper-parameters, and use hyper-parameter searching algorithms to find the optimal setting.

• •

  Use random initialization following certain types of distributions with mode close 1 for the default activation.

The advantage of treating initialization values of activation parameters as hyper-parameters is that we can search for the optimal solution based on the final validation performance of model, while the drawback is higher computational cost for extra hyper-parameter optimizations. Alternatively, by using random initialization for each training, an ensemble of activation functions with different shapes are used at the beginning of model training, which can be more flexible in explore larger region in search space. In this study, to avoid tunning too many hyper-parameters, we initialized the parameters of activation function as standard activation functions such as sigmoid and ReLu, which corresponds to $\alpha=1$ in Eq. (7) and Eq. (9).

For testing the performance of the model with flexible activation in recurrent neural networks, we build a multiple-layer LSTM model. We change the three sigmoid functions in Eq. (1) to the parameterized combined function as shown in Eq. (7), then compare the model performances in the cases with or without flexible activations.

The datasets being experimented on is a combination of daily stock returns of G7 countries, which is a multi-variate time series [29, 30]. The returns of each day can be considered as an input vector to the corresponding hidden layer, while the output is one-step ahead forecast given a sequence of historical data. Instead of using random sampling, we directly split the set of sequences with 10 lagging vectors into training set (64%), validation set (16%) and test set (20%), while the learning curves on training and validation sets can be obtained. The loss is selected as the average of mean squared errors of all forecasted values with respect to the true values for each example. For the hyper-parameter setting, the batch size was set to be 50, the window size is 10 time steps as is recommended by related papers[31], while the optimizer implemented in training is Adam optimizer with the same learning rate of 0.001 on both weights, bias and activation parameters. The initialization of the flexible activation parameters in replacing sigmoid function is $\alpha=1$ and $\beta=0.1$, which means that we train them from baseline settings. On the other hand, four stacked LSTM models with different layer configurations are implemented, then we compare the validation and test performances of these models with (a) fixed activations, (b) flexible activations without regularization, (c) flexible activation with a toward-mean regularization using a same default regularization coefficient of 0.025. We perform 50 trials for each setting with different random initialization for each of them.

For BRICS indices forecasting, we test the model performances with 4 stacked LSTM configurations, including models with layer structures: $[5,8]$, $[5,8,8]$, $[5,8,8,8]$ and $[5,16,8]$. For example, in the first layer structure, we have 5 input units for each time step, and a LSTM layer with a hidden state of length 8 for each time step. The results are provided in Fig. 3, where we use error bars to show the sample standard errors of each method in each time step.

As we can see, for all the four cases, generally the learning curves of models with flexible activation and regularized flexible activation (with fixed towards-mean regularization) lie below the learning curves of models with fixed sigmoid activations. For G7 indices forecasting, we exam the performances on stacked-LSTM models with layer structures: $[7,10]$, $[7,20]$, $[7,10,10,10]$ and $[7,20,10]$. The results on training set are provided in Fig. 4. The curves are plotted from the 5th epoch to focus on the comparison in later stage.

We can see that in Fig. 4, for all the four cases, the learning curves of model with flexible activation functions (denoted as “flexible models”) on training set lie below the corresponding curves of models with fixed activation functions (denoted as “fixed models”) during most of training time. In addition, towards-mean regularization on activation parameters can further improve the convergence performance. Meanwhile, the error bars of population means demonstrate that the improvement of flexible models in terms of validation performance is significant in general.

Similarly, we can see that in Fig. 5, for all the four cases, the learning curves of flexible models on validation set lie below the corresponding curves of fixed models during most of training time. Moreover, the error bars of population means demonstrate that the improvement of flexible models in terms of validation performance is significant in general. To make a further investigation, we use optimized learning rates instead of the default ones in each case. First, we perform grid searches for both models without flexible activation functions and model with flexible activation functions in the range of $[0.001,0.1]$ on logarithm scale. Also, for models with flexible activation, we perform another grid search in the range of $[0.01,0.2]$ on logarithm scale for the optimal activation regularization coefficients defined in Section  II-D, while the searching is based on the optimal learning rate in each case. Then for each architecture, we compare the models with four settings: (a) Models without flexible activation on the corresponding optimal learning rate; (b) Models with flexible activation function on the corresponding optimal learning rate, without regularization on the activation function; (c) Models with flexible activation function on the corresponding optimal learning rate, with regularization on the activation function on the default value of 0.025; (d) Models with flexible activation function on the corresponding optimal learning rate, with optimal regularization on the activation function in each case. The results of 50 runs with corresponding error bars are shown in Fig. 6.

We can learn from Fig. 6 that with optimized learning rates and Adam optimizer, LSTM models with flexible activation functions perform better in terms of convergence compared with fixed models for all the four architectures. By introducing regularization on flexible activation towards the mean, the convergence can be further improved. To investigate the effect of two types of regularization, we conduct a simulation and plot the relationship between the minimum validation model performance and the values of regularization coefficients. We use a default learning rate of 0.001. For regularization coefficients, we take 20 discrete grid points in the logarithm scale of 0.001 to 0.2. At each point, we plot the average of 30 trials with corresponding error bars for the standard error of sample means.

It is shown in Fig. 7, the minimum validation model performance is not constant under different regularization coefficients for both the two types of regularization of flexible activation functions. By referring to the error bars for 30 trials, we can learn that the change is not random, but with statistical significance. Generally, the minimum validation performances are obtained by regularization towards the mean with a intermediate regularization coefficient. Also, we have had a similar investigation based on the minimum validation loss in each setting. We use the test set of G7 for testing and all training and validation samples for training, with a default learning rate of 0.001. At each point, we plot the average of 20 trials with corresponding error bars for the standard error of sample means.

As is shown in Fig. 8, the test model performance is not constant under different regularization coefficients for both the two types of regularization of flexible activation functions. By referring to the error bars for 20 trials, we can learn that the change of test performance is not random, but with statistical significance. Generally, the best performances are obtained with a intermediate regularization coefficient. This demonstrates that a better test performance can be achieved by optimizing the towards-mean or towards default regularization of flexible activation function given the architectures unchanged.

The second experiment is conducted with autoencoder models for lossy image compression on several benchmark datasets including MNIST and FMNIST [4]. The architecture used in this experiment can be shown as follows:
Convolutional Autoencoder 1 (CAE 1):

$$\begin{split}&\text{Input(28*28*1)}\rightarrow\text{Conv2d(16, 3, 3)}\xrightarrow{\text{ReLU}}\text{MP(2, 2)}\rightarrow\\ &\text{Coding}\rightarrow\text{Conv2d(8, 5, 3)}\xrightarrow{\text{ReLU}}\text{Conv2d(1, 2, 2 )}\\ &\xrightarrow{\text{Tanh}}\text{Output}\end{split}$$

Convolutional Autoencoder 2 (CAE 2):

$$\begin{split}&\text{Input(28*28*1)}\rightarrow\text{Conv2d(16, 3, 3)}\xrightarrow{\text{ReLU}}\text{MP(2, 2)}\rightarrow\\ &\text{Conv2d(8, 3, 2)}\xrightarrow{\text{ReLU}}\text{MP(2, 1)}\rightarrow\text{Coding}\\ &\rightarrow\text{Conv2d(16, 3, 2)}\xrightarrow{\text{ReLU}}\text{Conv2d(8, 5, 3)}\\ &\xrightarrow{\text{ReLU}}\text{Conv2d(1, 2, 2 )}\xrightarrow{\text{Tanh}}\text{Output}\end{split}$$

Convolutional Autoencoder 3 (CAE 3):

$$\begin{split}&\text{Input(32*32*3)}\rightarrow\text{Conv2d(12, 4, 2)}\xrightarrow{\text{ReLU}}\text{Conv(24, 4, 2)}\\ &\xrightarrow{\text{ReLU}}\text{Conv(48, 4, 2)}\xrightarrow{\text{ReLU}}\text{Coding}\rightarrow\text{Conv2d(24, 4, 2)}\\ &\xrightarrow{\text{ReLU}}\text{Conv2d(12, 4, 2)}\xrightarrow{\text{ReLU}}\text{Conv2d(3, 4, 2)}\xrightarrow{\text{ReLU}}\text{Output}\end{split}$$

CAE 1 and CAE 2 compress images of size 28*28 to 16 filters with size 2*2 for each. For each trial, we randomly sample 5,000 examples from the original training datasets of MNIST and FMNIST as the training data, another randomly sampled example from the remaining of the training set as the validation set, and use the original test sets as the test sets in our experiment. We then compare the performances of following activation functions in replacing the ReLU activation functions in the original architectures of CAE 1 and CAE 2: (a) ReLU function; (b) PReLU function; (c) ELU function; (d) Gelu function; (e) Newly proposed P-E2-ReLU function, which is initialized as $o(z)=0.4\text{ReLU}+0.3\text{Elu}(z)+0.3(-\text{Elu}(-z))$. The learning rate of each model with different activation function is optimized with a grid-search in the range between 0.001 and 0.1 on logarithmic scale. In addition, we run four more models in each trials with P-E2-ReLU on the optimized learning rates for other four candidates activation functions. After 50 trials, we can compare the learning curves of these five activation functions. The validation curves are given in Fig. 9, while the training curves are given in Fig. 10.

We noticed that models with the newly proposed P-E2-Relu activation function outperform all the other activation function in terms of the both validation and training performance with statistical significance during the whole learning process after a small amount of initial steps. Meanwhile, the standard ReLU activation function perform the worst, and the other three activation functions including PReLU, ELU and Gelu get similar performances in general. Moreover, even with the learning rates optimized under models with other activation functions, the proposed P-E2-ReLU activation function still outperform other activation functions in this task. To make a further investigation, we perform experiments on CIFAR10 and SVHN with another convolutional auto-encoder architecture given by CAE 3, where the encoder compress each image of size 32*32*3 to 48 filters with size 4*4. In addition, we introduced other two flexible activation functions proposed in Section  II-C, which are combinations of $\{\text{ReLU},\text{Elu}(z)-\text{Elu}(-z)\}$ and $\{\text{Id},\text{Elu(z)}-\text{Elu(-z)}\}$. The initialization of the two combination weights for these two activation functions are both 0.5. The learning rate is set to be the default value of 0.001 for all activation functions.

As is shown in Fig. 11, the experiment with CAE 3 architecture show the advantage of bounded P-E2-ReLU and P-E2-Id activation functions over other state-of-the-art activation functions in the image compression tasks on CIFAR10 and SVHN. For both two datasets, the model with the proposed P-E2-Id activation function perform the best, although the performance is close to ELU and PReLU. Further, we made one-tail two sample t-test for the difference of population means on test set. The results are given in Table  I, where the null hypothesis $\text{H0: }m_{4}\geq m_{2}$ means the population mean test loss of bounded P-E2-ReLU ($m_{4}$) is worse or equal to that of ELU ($m_{2})$. The test statistics demonstrate that P-E2-ReLU significantly outperforms PReLU and ELU in terms of test reconstruction MSE in both cases, with very small p-values, while P-E2-Id performs even better.

Again, to check the effect of regularization on flexible activations, we train and evaluate flexible models under different regularization coefficients. The learning rate is optimized for P-E2-ReLU. We take 20 discrete grid points from 0.001 to 0.2 in logarithm scale. The result is shown in Fig. 12.

As we can see in Fig. 12, for towards-default regularization, the models achieve the best performances under the smallest regularization coefficients. This indicates that model performance can be significantly improved by introducing the proposed flexible activating function. On the other hand, for towards-mean regularization, in half of the cases the minimum validation errors are obtained when the regularization coefficients are 0, while in the other two cases, the minimum validation errors are obtained at intermediate points. In addition, we notice that the curves representing towards-mean regularization are below the curves of towards-default regularization in most of the time. In (c) and (d), a strong towards-mean regularization does not harm the model performance but a strong towards-default regularization does, which indicates that the models can be improved by sharing different activations functions in different layers without variation across each same layer, and we may only need two more parameters in each layer to achieve this improvement with layer-wise learned activation functions.

When batch-normalization is not implemented, the number of parameters in each deep neural network is equal to the number of weighs plus the number of bias. For an FFNN, $N=N_{W}+N_{b}=\Sigma_{i=1}^{L-1}n_{i}n_{i+1}+\Sigma_{i=2}^{L}n_{i}$, where $L$ is the number of layers including input layer, hidden layer and output layer, while $n_{i}$ is the number of units in $i$th layer. If flexible activations are introduced in all hidden and output units, with two independent extra parameters in each unit, the total number of model parameters will increase by $N_{a}=2\sum_{i=2}^{L}n_{i}$. The number of parameters in basic RNN models is depend on the type of hidden cells used, in general it is $N=g\sum_{i=1}^{L-1}(n_{i+1}(n_{i+1}+n_{i})+n_{i+1})$, where $g$ is the number of weight matrices in each cell. For basic RNN, $g=1$, for GRU, $g=3$, and for LSTM, $g=4$. Meanwhile, the extra number of parameters needed for RNNs with flexible activations is $N_{a}=2\sum_{i=2}^{K}s_{i}n_{i}$, where $s_{i}$ is the number of activation functions replaced by flexible ones in $i$th layer.

Following this derivation, in the LSTM models used in Section III-A, we can make a summary table for the increasing ratio of model parameters in each case.

As shown in Table II and Table III, we can see that the models with flexible activations have about 5% to 10% increase in the number of parameters. However, it is shown in Section  III-A that model 1, model 3 and model 4 have very similar validation performances during training even though their number of parameters varies in much higher (even more than 100%) proportions, while the corresponding validation performances for models with flexible activations are better than all the models with fixed activations.

In Table II, the best performed model with layer size $[5,8]$ has a single hidden layer and the number of parameters is smaller than the other models with layer sizes $[5,8,8]$, $[5,16,8]$ and $[5,8,8,8]$. Similarly, in Table III, the best performed model with layer size $[7,10]$ has a single hidden layer and the number of parameters is smaller than the other models with layer sizes $[7,10,10]$, $[7,20,10]$ and $[7,10,10,10]$. Therefore, the number of parameters does not really matters in this range when we consider the minimum validation performance in a long training time, and the improvement of model performance can be explained by the advantage of introducing flexible activation with trainable parameters.

For the three Convolutional auto-encoder models in Section III-B, the basic model parameters include those in convolutional layers. The number of parameters of each convolutional neural layer is: $N_{c}=(n*n*l+1)*k$, where $n$ is the filter size, while $l$ and $k$ represent the numbers of input and output channels. For filter-shared flexible activation functions with $p$ activation parameters, the extra number of parameters is $N_{a}=p*k$, where $p=1$ for P-ReLU-Id and P-ReLU-E2-1, and $p=2$ for P-ReLU-E2. The summary table showing the numbers of basic model parameters as well as extra activation parameters for three CAE models are given in Table  IV and  V. We can see that although the experimental results shown in Fig. 10 and Fig. 11 demonstrate that the proposed P-E2-ReLU achieves a significant improvement compared with fixed activation such as ReLU and GeLU, the extra number of parameter introduced only a quite small proportion of the total number of parameters in each architecture. Moreover, as is indicated in Fig. 12, when layer-wise flexible activation are applied, the model performance can be improved with even smaller number of extra activation parameters.

To sum up, the results at least show that we can achieve a quite significant performance improvement with only a small proportion of extra parameters in the flexible activation functions. Although the same mapping may be learned without flexible activation functions, it may need much more number of parameters or much larger effort of hyper-parameter searching with a bunch of different model architectures.

LSTM is local in space and time [32]. Its computational complexity per time step and weight is $O(1)$, while the overall complexity of an LSTM per time step is equal to $O(w)$, where $w$ is the number of weights [33]. Therefore, theoretically the computational complexity is proportion to the number of parameters in LSTM models. For activation functions, each of them will include 2 parameters, and will process one input for either forward or backward path. Therefore, the extra computational complexity will still be proportional to the number of activation parameters, which corresponds to the ratios shown in Table  II and  III.

For convolutional neural networks, we have learned that the total time complexity of all convolutional layers is [34]:

$$T(n)=O(\sum_{l=1}^{d}n_{l-1}\cdot s_{l}^{2}\cdot n_{l}\cdot m_{l}^{2})$$

(11)

where $l$ is the index of a convolutional layer, and $d$ is the depth (number of convolutional layers). $n_{l}$ is the number of filters (also known as “width”) in the $l$-th layer, while $n_{l-1}$ is known as the number of input channels of the $l$-th layer. $s_{l}$ is the spatial size (length) of the filter. $m_{l}$ is the spatial size of the output feature map. For the whole training process we have:

$$O(m\cdot n_{iter}\cdot\sum^{d}_{l=1}(n_{l-1}\cdot s^{2}_{l}\cdot n_{l}\cdot m^{2}_{l}))$$

(12)

where $m$ is the input length and $n_{iter}$ the number of iterations. When flexible activation function is implemented, there will be extra computational cost in both forward and backward propagations: In forward propagation, when flexible activation is added after a convolutional layer, the number of different activations in this layer equals to the number of channels in that convolutional layer. Therefore, we need to do more element-wise operations that is proportional to the number of the input elements to each flexible activation function. It depends on the number of input channels and input filter size to the activation layer, but does not depend on the number of output filters and corresponding filter size. This is because after the input mapping and activation function, each element will become a single value before multiplying with filters in the next layer. The extra computational cost can be written as:

$$O(m\cdot n_{iter}\cdot\sum^{d}_{l=2}(n_{l-1}\cdot s^{2}_{l}))$$

(13)

where we start from $l=2$ since the first activation layer comes after the first convolutional layer rather than the input layer of data. On the other hand, in backward propagation, the updating in Eq. (3) will require extra computation that is proportional to the number of input element as well as the number of activation parameters, also we need to pass the gradients through flexible activation functions, which requires extra element-wise operations that proportional to the number of elements of the input to each flexible activation function. Notice that the gradients of the loss with respect to the output of activation are stored in cache and shared for calculating other gradients. Therefore, the theoretical extra computational cost is at least one-order smaller than that of the baseline model with fixed activation functions. This will not add a heavy burden for model training from the theoretical prospective.