Weights Augmentation: it has never ever ever ever let her model down

Some weights are considered to be "by-products" or "useless", but they can potentially help networks from different perspectives. On specific data, they exhibit good performance but could be more optimal when dealing with broader data. It is discarded during the model selection process. However, with the deepening of deep network research, researchers have begun to recognize the potential value of these "by-products". For example, Ensemble Learning dietterich2002ensemble ; krawczyk2017ensemble ; huang2009research can integrate these seemingly useless weights to form a more powerful and robust model. This method takes advantage of different weights on different data and ultimately improves the model’s overall performance by combining high-quality weights. This approach is merely the stitching of weights and has yet to reach the level of weight augmentation. A diverse weight space is crucial, and weights learned from weight distributions perform better. The formula for increasing the diversity of the weight space is:

$$h\left(x\right)=ReLU\left(TW^{{}^{\prime}}x\right)=ReLU\left(\sum_{i=1}^{k}T_{i}W_{i}x_{i}+b\right)$$

(3)

where $W$ is the model’s weight vector. $W^{{}^{\prime}}$ is its transpose. ReLU is a popular activation function. $T_{i}$ represents an augmentation strategy with transformations like rotation, translation, and scaling.

Reorganizing weights may cause bias and hinder generalization due to incomplete weight distribution capture. To solve this, we introduce randomness, the formula is as follows:

$$h\left(x\right)=ReLU\left(\sum_{i=1}^{j}\gamma\left(T_{i}\right)W_{i}x_{i}+b\right)$$

(4)

$\gamma$ is significant as it’s not defined by a simple function. We enhance randomness through Whether to use WAS or select WAS, etc.

As shown in Figure. 3, each execution of mini-batch gradient descent WAS will lead to a weight change, thus producing a variant of SW. As training continues, the distribution of SW gradually improves. The model parameter update formula based on weight update is as follows:

$$\theta_{j}=\theta_{j}-\alpha\cdot\frac{\partial}{\partial\theta_{j}}J(\theta)$$

(5)

$$\theta_{j}^{pw}=\theta_{j}^{pw}-\alpha\cdot\frac{\partial}{\partial\theta_{j}^{pw}}J(\theta_{j}^{sw})$$

(6)

Where $\theta_{j}^{sw}$ and $\theta_{j}^{pw}$ represent the jth parameter of SW and the jth parameter of PW, respectively. $\alpha$ denotes the learning rate. $J(\theta)$ is the loss function. $\frac{\partial}{\partial\theta_{j}^{sw}}$ indicate the partial derivative of the loss function $J(\theta_{j}^{sw})$ with respect to the parameter $\theta_{j}^{pw}$. Equation(6) is the formula of the original network Gradient Descent with SGD, and Equation(6) is the SGD formula using WAS.

$$\frac{\partial J}{\partial\theta_{j}}=\sum_{x,y}\frac{\partial J}{\partial Z_{xy}}\cdots\frac{\partial Z_{xy}}{\partial\theta_{j}}$$

(7)

where $Z_{xy}$ represent the feature map at position (x, y), $\frac{\partial J}{\partial Z_{xy}}$ is the partial derivative of the loss function relative to $Z_{xy}$, $\frac{\partial Z_{xy}}{\partial\theta_{j}}$ denote the partial derivative of a relative to $\theta_{j}$. We use SW to calculate the loss function to evaluate the difference between output data and the actual results. This evaluation result will directly affect the update of PW. Additionally, SW is used to learn the distribution of data, and we use it as the input of the loss function to approximate the data distribution. PW mainly learns the distribution of SW, so only PW needs to be updated. The above is the biggest difference between this method and the traditional method. The relationship between PW, SW, and loss function is shown in Figure 4.

In their research, Ding et al. ding2021repvgg proposed the Repvgg, which reduces time complexity by adjusting weights during inference while maintaining model performance. This method exploits the inherent potential relationship between the model and the weights but does not associate the model with the weights. In this field, Zheng et al. zheng2023learn indicate the "Learn From Model" concept in their paper, emphasizing the importance of research, modification, and design of Foundation models based on model interfaces. Due to researchers’ superficial exploration of this field, more research is needed to uncover the intrinsic interaction between weights and model structure. Therefore, although the concept has revolutionary potential, direct manipulation of weights is undoubtedly challenging without such understanding. At this point, if the model is consciously guided during training. Doing so allows human intervention during inference, and direct adjustments to the weights do not cause significant degradation in model performance. To achieve this goal, we will use weight-based training instead of data-based training, allowing the final model to learn from the weight distribution instead of extracting features directly from the data. We can generate a lot of flawed weight through data-driven training, but it can still capture subtle features and deep relationships of data specific to particular datasets. If the model training based on data distribution is regarded as one stage, then the model training based on the distribution of defect weights is called two stages. These two stages alternate with each other in the entire training. The final weights depend on the model’s autonomous learning without human intervention. Such a design allows the model to be trained directly to optimize its structure and weights.

WAS encourages the model to explore a broader area of the weight space, promoting high-performance weight configurations while avoiding over-reliance on specific features. It also improves the model’s computational efficiency, reducing resource consumption training and inference.

The core of WAS is to encourage the generation of a large number of weights, which makes the weight distribution more complete, enhancing the generalization and robustness of the model. The model obtained always has two modes in training: AOM and DOM. AOM is akin to conventional deep learning models, as it is directly applied to inference. However, the key innovation of AOM lies in its dynamic weight formation mechanism. This is achieved through a competitive process that unfolds during training. The competition continuously drives the model to optimize its performance. Specifically, SW demonstrates superior performance and will effectively reduce the loss function. As a result, it significantly impacts PW determination. Conversely, those with poorer performance will have less influence on AW determination.

The second mode focuses on changing WAS, which has solved specific needs such as improving the model’s specificity, sparsity, and computational efficiency. In DOM, we can devise WAS tailored to the particular demands of the task and implement it within training. Subsequently, this strategy can be used to tune PW for inference. Although the model’s prediction accuracy may be worse than AOM in this mode, it has significant advantages in specific tasks. For instance, random weight cropping can increase the matrix’s sparsity of weight. This approach can reduce computational load, potentially by several orders or even tens of times, which is particularly advantageous for applications involving large-scale data processing or model deployment within environments with constrained resources. In summary, the DOM approach can reduce computational, potentially by several or even tens of times, which is particularly advantageous for applications involving large-scale data processing or model deployment within Special needs.

Through WAS, we have implemented a flexible model management strategy. In inference, only one weight needs to be saved to realize two different weight functions simultaneously, and the working mode can be flexibly switched according to the application scenario. This dual-mode simplifies the management of model storage and maintenance. It also strengthens the algorithm’s resilience and adaptability.

To explore the effectiveness of WAS, we have chosen six class deep learning architectures as our experimental models: VGG-16 simonyan2014very , ResNet18 he2016deep , ResNet34 he2016deep , GoogLeNet szegedy2015going , EfficientNet-Lite tan2019efficientnet , and MobileNetV2 sandler2018mobilenetv2 . As illustrated in Table LABEL:Table1, we have chanced a series of WAS for comparative experiments against the baseline.

We deliberately eschewed the incorporation of additional techniques, primarily to mitigate the impact of extraneous variables. On a solitary GPU, we established a global batch size of 128. We employed the conventional SGD, initializing the learning rate at 0.01. Furthermore, we fine-tuned the SGD optimizer, assigning a momentum coefficient of 0.9, thereby augmenting the model’s stability and hastening convergence throughout the training regimen.

On the CIFAR-10 dataset, AOG for the models demonstrated discernible enhancements. Specifically, the accuracy of VGG16-C and VGG16-CT saw an increase of 2.13% and 4.27% relative to the baseline VGG16, respectively. For ResNet18, ResNet18-C and ResNet18-CT attained an accuracy improvement of 3.06% and 5.03%. Similarly, the ResNet34-C and ResNet34-CT models realized respective accuracy improvements of 3.05% and 5.97%. The GoogleNet-C and GoogleNet-CT models recorded accuracy enhancements of 1.59% and 2.35%. Within the MobileNetV2 series, the accuracy for MobileNetV2-C and MobileNetV2-CT marked a significant rise of 12.76% and 13.42% over MobileNetV2. Lastly, the EfficientNetLite-C and EfficientNetLite-CT models secured accuracy improvements of 9.52% and 13.23%.

On the CIFAR-100 dataset, WAT also demonstrated enhanced performance. The accuracy of AOG for Models VGG16-C and VGG16-CT exhibited respective increases of 5.39% and 7.66% over the base VGG16. Similarly, the accuracy of AOG for Models ResNet18-C and ResNet18-CT recorded increases of 6.23% and 6.19% relative to the original ResNet18. Models ResNet34-C and ResNet34-CT demonstrated accuracy of AOG enhancements of 6.05% and 3.41% respectively, in comparison to ResNet34. The GoogleNet-C and GoogleNet-CT models noted slight improvements in the accuracy of AOG, with gains of 1.40% and 1.44% over GoogleNet. The MobileNetV2-C and MobileNetV2-CT models marked a significant advancement in the accuracy of AOG, showing increases of 17.89% and 18.93% over MobileNetV2. Finally, Models EfficientNetLite-C and EfficientNetLite-CT displayed significant accuracy of AOG improvements, with increases of 13.78% and 18.93% over EfficientNetLite.

The performance of Model-C and Model-CT under DOM is lower than under AOM. Their accuracy difference remains at about 1% to 2%, but DOM’s accuracy is still about 1% to 2% higher than the baseline model. This shows that both modes can improve the performance of the model. Particularly, in the context of lightweight models, these adventures are particularly prominent. Although computational speed is not our main consideration, it is worth noting that the floating point operations (FLOPs) of the GOMs of Model-C and Model-CT are both lower than their corresponding baseline models. The FLOPs of Model-C decreased by about 5% to 20%, while the reduction in FLOPs for Model-CT is more significant, with the maximum reduction reaching up to 47%.

WAS gives the model unique capabilities that can be customized to specific needs inference. These capabilities can include many aspects, such as reducing computational complexity, decreasing sensitivity to specific data, and so on. This functionality be used by switching the weights to SW during inference. It can customizable adjustments based on specific requirements, enabling the model to accommodate unique environmental conditions.

Table 2 presents three representative WAS: rotation (the random rotation angle of weights is set between 0° and 90°), translation (the random translation range of weights in horizontal and vertical directions is 0% to 30%), and cropping (the random cropping ratio of weights is between 0.8 and 1.0), following the official PyTorch example imambi2021pytorch . To verify the efficacy of WAS for processing particular data, we implemented random data augmentation on the test set to assess WAS strategies. After using WAS, the network model will have two working modes: AOM, a weight that does not apply WAS inference; and DOM, a weight that applies WAS inference. Shown in Table 2, when the strategy for WAT training is a random rotation of 0° 90°, AOM outperforms DOM in Top-1 accuracy. Especially when the rotation of data is limited to 0° 45° and 0° 90°, the accuracy loss of DOM compared to AOM is 13.84% and 3.55% lower. For other rotations, the accuracy loss of DOM is about 1% lower than that of AOM. When randomly cropping is the WAS strategy, as the weight cropping ratio increases, the accuracy loss of DOM compared to AOM decreases by 0.58%, 0.22%, 0.92%, and 1.77%Ẇe noticed that with the increase of the random cropping ratio, the disparity in accuracy loss between DOM Two and AOM initially contracts and subsequently expands. This fluctuation is mainly due to the increase in the random crop ratio during training, which causes the model performance to decrease when the parameters exceed a certain threshold. In summary, WAS helps to promote the ability to process specific data. We found that when the WAS is randomly translated, the accuracy loss of DOM is higher than that of AOM, which is inconsistent with our preliminary conclusion. After combining random crops, we think the reduction of model parameters leads to the fitting ability of decline.

As depicted in Table 3, we have implemented AD pipeline that random translations within the test dataset. Moreover, the WAT strategy deviates from conventional AD. As the parameter for random translation from 10% to 40%, the AOM’s Top-1 drop rate surged from 2.56% to 32.45%, while the DOM’s Top-1 drop rate saw a similar rise, climbing from 3.27% to 33.70%. Furthermore, if the test set is replaced with randomly cropped data, as the cropping ratio is reduced from 1.0 to 0.2, the Top-1 accuracy drop rate for both modes increases. For AOM, the drop rate is from 2.69% to 21.53%, whereas for DOM, it ascends from 2.61% to 25.10%. DOM experiences a more rapid performance degradation rate than AOM, especially at higher cropping ratios. Based on the preliminary analysis of the above data, we can conclude that AOM has stronger generalization when the WAS strategy is inconsistent with the data augmentation strategy.

Similarly, as the rotation angle of the test set increases from 0° to 180°. the WAS strategy is random translation. The drop rate for AOM increases from 5.09% to 58.96%. DOM soared from 5.27% to a remarkable 59.67%. However, when WAS uses random cropping, the drop rate for AOM increases from 4.80% to 58.98%, and for DOM, it increases from 4.85% to 56.51%. This is the same conclusion we reached above. when WAS uses random cropping, the drop rate for AOM increases from 4.80% to 58.98%; for DOM, it increases from 4.85% to 56.51%. Contrary to the above conclusion, the drop rate of AOM under extreme conditions is higher than that of DOM. Contrary to the above conclusion, the drop rate of AOM under extreme conditions is higher than that of DOM.

In order to explore the above phenomenon, Table 4 illustrates the models trained with different random cropping ratios as WAS. DA uses random rotation. On the test set, how the training parameters of WAS change, the drop rate of AOM remains relatively stable. In contrast, as the random cropping ratio of WAS increases, the Top-1 accuracy drop rates of DOM are 2.47%, 3.82%, and 4.02% lower than AOM respectively. This trend shows that DOM is more effective in processing rotated data. As the cropping ratio ranges from 0.2 to 0.4, the Top-1 accuracy drop rates between AOM and DOM narrow. This is attributed to the number of decreases in parameters, which impacts model fitting. In summary, data rotation can lead to information loss, akin to the effects of cropping. Therefore, DOM (weight-based random cropping) has a lower drop rate than AOM.

WAS is not only reflected in its accuracy but also in the sparsity of the model. As shown in Table 5, we take the Crop as an example, using different cropping ratios. When the Cropping parameters are (0.4,0.6), the proportion of 0 elements in the entire model accounts for 33.01%, and the model’s accuracy is 87.99%, which is close to the base model. When the Cropping parameters are reduced to (0.2,0.6), the accuracy decreases by 1.41%, but the proportion of 0 elements increases by 3.32%, and the FLOPs are reduced by 11.06M. When introducing translation parameters (30%,30%), even without cropping, the sparsity rate is increased to 30.74%, floating-point operations(FLOPs) are reduced to 230.63M, and the model’s accuracy is improved by 3.26% compared to the base model. Combining cropping and translation weight pruning can further improve the model’s accuracy and reduce computational costs.

With cropping parameters set at (0.4,0.6), the model’s sparsity reaches 33.01% with zero elements, and its accuracy is 87.99%, nearly closing the base model. When the Cropping parameters are reduced to (0.2,0.6), the accuracy decreases by 1.41%, but the proportion of 0 elements increases by 3.32%, and the FLOPs are reduced by 11.06M. Introducing translation parameters of (30%,30%) boosts sparsity to 30.74%, cuts FLOPs to 230.63M, and improve accuracy by 3.26% over the base model. Combining cropping and translation, WAS improves model accuracy and simultaneously decreases computational costs.

WAS as an effective approach for convolutional networks. It is capable of enhancing accuracy without increasing computational complexity. WAS shines in the customization of model training for specific tasks, offering tailored optimization strategies. Yet, despite its considerable benefits, WAS has not gained the same level of popularity as data augmentation practices.