Efficient Context Integration through Factorized Pyramidal Learning for Ultra-Lightweight Semantic Segmentation

In real-time semantic segmentation networks, for reducing the number of parameters and computational cost, usually, the spatial resolution is hastily downsampled to quarter resolution by using two consecutive downsampling blocks [16, 28]. This strategy undoubtedly saves some parameters and convolution operations. However, this aggressive reduction in spatial resolution so early has a significant negative effect on the fine delineation of the object boundaries that ultimately leads to poor accuracy performance [15]. This is because much of the finer spatial information, that could have been harvested in the first stage, is lost due to early downsampling. So, to avoid the loss of finer details, we use two successive convolutional layers (excluding downsampler) in stage-1. In this way, we are able to harvest better spatial information at half spatial resolution in the first stage, that yields better feature representation at the input of the second stage.

In order to correctly recognize a small region of an image, we need the information of surrounding regions as well [11]. So, context plays a very important role in vision problems and especially in semantic segmentation. The simplest way to increase the context integration is to enlarge the receptive fields of kernals by increasing their sizes. So, researchers have used kernals of different sizes ranging from $3\times 3$ to $11\times 11$ [7]. However, this strategy is very inefficient as it leads to increase in the parameters exponentially. For example, a $3\times 3$ conventional kernal has 9 parameters, whereas an $11\times 11$ kernal has 121 parameters per channel. One simple solution to this problem is to use dilated convolutions [34]. Dilation allows us to increase the receptive field without increasing the number of parameters. For example, a $3\times 3$ dilated convolution with a dilation factor 16 spans a region of $33\times 33$ of input feature map, whereas to span the same region, a conventional kernal would require 1089 parameters which is $121\times$ more than its dilated counterpart. So, with this strategy one can save a huge number of parameters. However, in complex pixel-level dense prediction tasks, such as semantic segmentation, single-scale context is not sufficient and we need multi-scale context.
In literature, the most popular way to harvest multi-scale context is to employ spatial pyramid [16]. Spatial pyramid is nothing but multiscale context aggregation. This multiscale context can be intergrated either by pooling features with different window sizes [24, 23, 9] or by employing a bank of convolutional filters with different dilation rates [16]. The former is used once in the network usually before the final prediction stage [24] whereas the latter is usually distributed across the network and is built into the basic blocks themselves [16]. This eliminates the need to use a seperate pyramid module for context aggregation as the basic blocks themselves are well equipped to carry out the required task. Therefore, we adopted the latter approach, i.e., context aggregation as an in-built functionality of the basic blocks. More specifically, we adapted the ESP module of the classical ESPNet and reformulated it. The ESP module, in order to save parameters and computations, drastically squeezes the channel depth of the input feature; 1/5 to be more specific. This saves the parameters at the cost of information loss along the channel dimension.

The proposed approach, however, is distinct from [16] in two important ways. Firstly, we decomposed the spatial pyramid into two consecutive stage. We employed a conventional convolution in the first stage to gather local information whereas a bank of dilated convolutions is employed in the second stage to gather multi-scale context. Secondly, we factorized the spatial pyramid (i.e., bank of dilated convolutions) in the second stage which leads to significant parameter saving while incurring a slight toll on the accuracy. To be more specific, each $3\times 3$ dilated kernal is factorized into a pair of $3\times 1$ and $1\times 3$ kernal. In this way, we are able to save three parameters in each channel of the kernal. The factorization also allows us to put an extra layer of non-linearity in between the two asymmetric kernals. This adds an additional linear region in the manifold space which leads to better accuracy with a negligible increment in computation cost [28]. Being on top of the factorized spatial pyramid of kernals, the conventional convolution in the first stage adds an additional layer of transformation which increases the effective depth of the network without increasing the number of basic modules. Moreover, being a symmetric convolution, it helps in balancing the accuracy loss incurred by the bank of factorized asymmetric 1D convolutional filters. This two-stage pyramid also enhances the effective receptive field with respect to ESP module [35]. More importantly, the pyramid decomposition enables us to solve the problem of gridding artifact in a quite simple way. To be more specific, we applied a simple pairwise feature fusion (PFF) that adds only two features as opposed to HFF of ESP module which engages all the five features of the module.

Table I shows the comparison of different modules. $C_{i}$ is the number of channels in the incoming feature map and $C_{o}$ is the number of channels in the outgoing one. $k\times k$ is the size of the symmetric convolutional kernal. $b$ is the number of branches the incoming feature tensor is split into. The exact number of parameters per module, shown in Table I, is computed for $C_{i}=C_{o}=60$, $k=3$ and $b=5$ for ESP-original and $b=4$ for both FPL-decomp. and FPL. Please note that the decomposition of pyramid both in FPL-decomp. and FPL results in enhanced receptive field (hence more wider context) and allows the depth of each branch to increase from 1/5 to 1/4. It is clear from the Table I that the FPL-decomp. module requires 53.125 % more parameters in comparison to ESP-original module. Our proposed final version of FPL module, on the other hand, achieves the same objectives as that of FPL-decomp. but with only 15.6 % more parameters while having only a slight effect on the final accuracy.

Intra-stage skip connections are used to enhance the information flow of the network [20, 16]. Image insertion in also done to improve feature representation at different stages of the encoder [18]. To accomplish this, additive or concatenative fusion is usually used. Concatenation increases the parameters but performs slightly better than additive fusion. In this work, to improve it further, we design a dedicated FIR unit to jointly combine the deep and shallow features with the downsampled images. The proposed FIR unit employs a hierarchical combination of additive and concatenative fusion. This gives significantly better performance compared to addition or concatenation alone with the same number of trainable parameters. The FIR unit is shown in the Fig. 3. Features from deep and shallow layers are first added. The resultant feature tensor is then concatenated with the feature from the deep layer follwed by image insertion.

The comparison of FPLNet with other state-of-the-art networks has been summerized in Table IV. To measure accuracy, mIoU (%) has been used and for efficiency, we use parameters (M) and speed (FPS). Apart from that, other comparative indicators such as type of backbone network, pretraining information, etc., are also included to facilitate a multifaceted perspective. To keep the discussion tractable, we divide the sementic segmentation models into 4 broad categories based on number of parameters as shown in Table V.

Recent large-scale networks such as BiSeNetV1 [10], HyperSeg [17] and HSBNet [38] achieve excellent accuracies. However, to achieve such high accuracies, they had to design networks with huge number of parameters. More specifically, BiSeNetV1, HyperSeg and HSBNet achieve 68.4, 78.1 and 73.1% mIoU at 5.8, 10.2 and 12.1 million parameters, respectively. It is quite interesting to note here that HyperSeg [17] is 5% more accurate than HSBNet despite having almost 2 millions less parameters. This shows that a straight-forward increase in parameters does not guarantee proportional accuracy boost. This can be further observed, when we compare mid-scale networks with the large-scale ones. When we Compare CABiNet [22] with BiSeNetV1, we find that CABiNet has 7.5% more accuracy than BiSeNetV1 while being $2.2\times$ smaller. Similarly, RegSeg [31] and BiAttenNet [30] are 0.2 and 1.6% more accurate than HyperSeg [17] and HSBNet [38], while being $3\times$ and $5.5\times$ smaller, respectively. A similar effect can be seen when we compare BiSeNetV1 and BiSeNetV2 [38]. Despite having 1.21 million less parameters, BiSeNetV2 is almost 7% more accurate than BiSeNetV1. These observations demonstrate the possibility for smaller networks of achieving accuracies similar to multiple-times bigger networks through smart architecture design. Exploiting this possibility is extremely important because model sizes of the large-scale and mid-scale networks become bottleneck in achieving real-time processing in resource-constrained devices [22]. To address this issue, many networks have been developed with less than 1 million parameters.

To further scale-down the network size while still maintaining a decent accuracy, we propose an ultra-lightweight real-time FPLNet. Experimental results show that the proposed network achieves 66.93% and 66.28% mIoU on validation and test set, respectively. It achieves similar accuracy compared to lightweight models with much less number of parameters such as DABNet [18] and EDANet [20]. When compared to other state-of-the-art ultra-lightweight (less than 0.5 million parameters) networks [16, 15, 19, 11], the proposed FPLNet outperforms them in terms of accuracy (almost 1-10% accuracy improvement) while having similar number of parameters; 0.49 million to be more specific. This shows the effectiveness of our network in achieving a decent accuracy-efficiency tradeoff in Ultra-Lightweight range. More importantly, it achieves accuracy close to large-scale BiSeNetV1 [21] and mid-scale ERFNet [28] while being 11.8$\times$ and 4.5$\times$ smaller, respectively. To put everything into perspective, a comparison of the proposed FPLNet with other lightweight and ultra-lightweight networks have been shown in Fig. 5. The inherent contention between accuracy and efficiency can also be clearly observed in Fig. 5. For calculating the trainable parameters of the network, pytorch-OpCounter library [41] has been used [17]. To give a visual perspective, the qualitative results are shown in Fig. 6.

In this section, a detailed ablation study of different design choices is presented. It should be noted that all the experiments in this study are conducted considering only the encoder of the FPLNet unless stated otherwise. Once the required efficient encoder was found, a lightweight decoder is attached to it to complete the proposed network, i.e., FPLNet.

ESP vs FPL module: To show the effectiveness of the proposed FPL module and the corresponding FPLNet, we conducted a series of experiments as shown in Table VI. By applying the ESP module in FPLNet and FPL module in ESPNet, we are able to show the effectiveness of both the FPL module and the architecture of our network. It is clear from Table VI that by merely replacing ESP module with FPL module in the ESPNet (while keeping other variables constant) we are able to improve the mIoU of ESPNet by 2.43 %; reflecting the effectiveness of FPL module. Similarly, applying ESP module in our FPLNet (while keeping other variables constant) gives a 2.99 % accuracy boost over ESPNet; showing the effectiveness of the proposed architecture.

Hasty downsampling vs Delayed downsampling: In order to reduce parameter contributed by the first stage, many works adopt a hasty downsampling strategy [16, 28]. This does not allow any feature extraction at stage-1 (i.e., half of input resolution) and perform two back-to-back downsampling operations to downsample the feature maps to 1/4 of input resolution. However, it has an adverse effect on the accuracy of the network as the low-level finer details are lost. Hence, we employ two conventional layers in this stage to extract the finer details which is crucial in fine delineation of the object boundaries in the final segmented map. Table VII shows the advantage of delayed downsampling over hasty one.

Different fusion strategies within the encoder: Table VIII presents a comparison between different variants of intra-stage feature fusion and image fusion. It is clear from the Table VIII that concatenation works better for intra-stage feature fusion as opposed to addition, with a slight increment in the network parameters. The proposed FIR unit

Number of FLP modules in different stages: A series of experiments have been conducted to find the optimal number of FPL modules to be used in different stages. Table IX presents the performance of the FPLNet encoder against different number of FPL modules in stage-2 and stage-3. After performing extensive experiments, it is found that the optimal configuration is 4-8, i.e., 4 and 8 FPL modules in stage-2 and stage-3, respectively.

Extent of factorization: Factorization of convolutional kernals saves parameters but affects the accuracy. Hence, a trade-off is required. To find the required trade-off, a set of experiments have been conducted to reveal how much factorization within the FPL module should be done. Table X presents the results of the corresponding experiments.

In the proposed composite scheme, first stage of pyramid, i.e., $3\times 3$ convolution is kept symmetric while the second stage, i.e., bank of dilated convolutions are factorized. It is clear from Table X that the proposed setting of factorization in the FPL module results in a balanced trade-off. With no factorization at all, we observe only 0.38 % mIoU increment at the expense of 0.13 million more parameters compared to the composite scheme. Hence, we can easily conclude that the proposed configuration of factorization offers the optimal balance between mIoU and number of parameters.

Activation function: Using PReLU [42] instead of ReLU [43] results in 0.45 % increment in accuracy, while having a negligible effect on the model complexity. So, we adopted PReLU in our network.