Lacunarity Pooling Layers for Plant Image Classification using Texture Analysis

We introduce a new model, designed for integrating pre-trained CNN architectures with additional lacunarity pooling layers for improved feature extraction and classification tasks. As seen in Figure 2, the model takes an input image and extracts features using any pre-trained backbone (i.e., ConvNexT, ResNet18). For our work, the pre-trained backbone is fixed and only the final output classification layer is tuned. The backbone is fixed 1) to reduce computational costs of fine-tuning the model end-to-end and 2) to evaluate if the lacunarity pooling layer can capture features from the backbone network that are effective for classification.

The pooling layer, determined by the specified configuration, is used to spatially aggregate the feature maps. Following inspiration from [10], a global average pooling (GAP) layer is employed to capture global context from the feature maps. The fusion process combines the lacunarity feature maps with the GAP feature maps, enhancing the model’s ability to capture both local and global spatial information. Finally, the fused features are fed into a fully connected layer for making predictions. This flexible model architecture provides a robust framework for leveraging both pre-trained CNNs and lacunarity-based features. We investigated three approaches to compute lacunarity and each calculation is discussed in the next three subsections.

Lacunarity is based on the pixel distribution for an image, which is obtained from scans for different box sizes at various grid orientations. Lacunarity [28] can be defined through Equation 1:

By examining the distribution of gaps or holes at various scales, lacunarity provides valuable insights into the structural complexity and heterogeneity of the image. The process of computing lacunarity involves applying normalization to the input through the hyperbolic tangent (tanh) function to ensure that the feature values are transformed to a range between -1 and 1. To adapt the feature maps to typical pixel values ranging from 0 to 255, an additional transformation is applied as shown in Equation 3:

The differential box counting (DBC) method is another approach for lacunarity computation [8]. The box counting algorithm is only valid for binary images, and therefore, it is not applicable to grayscale images, as required by DBC algorithm. Briefly, DBC algorithm determines the fractal dimension value of a grayscale image by performing a least square linear fit of $\log(N_{r})$ with respect to $\log(1/r)$, where $N_{r}$ represents the box-counting for a grid of size $r$ pixels. The box-counting process for a grid of size $r$ involves computing the difference between the boxes that denote the maximum and minimum gray levels of the pixels within the grid.

The DBC module is a novel approach to estimate lacunarity. Given input image(s) or feature map(s), the module normalizes pixel intensities and performs dilated box counting through max and min pooling operations. The lacunarity at different dilation factors $r$ is computed using the formula shown in Equation 4:

In the context of analyzing mass distribution in sets, Allain and Cloitre [1] introduced a gliding-box algorithm to define lacunarity. This method involves a box of radius $r$ sliding across a lattice superimposed on the set. Here, $n(M,r)$ represents the count of gliding-boxes with radius $r$ and mass $M$. The probability function $Q(M,r)$ is derived by normalizing $n(M,r)$ by the total box count.

The mass $M$ in Equation 4 can be computed using a DBC method introduced by Sarkar and Chaudhuri [26]. As shown in Figure 3, a cubic of size $r\times r\times r$ is positioned over the upper-left corner of an image window with dimensions $W\times W$, where $W$ is an odd number, and $r<W$. Based on pixel values, a column with more than one cubic may be required to cover the image intensity surface. Each $r$-th gliding-box is assigned numbers (e.g., 1, 2, 3) from bottom to top. For a given $r$-th gliding-box, with minimum and maximum pixel values falling in boxes $u$ and $v$ respectively, the relative height of the column is given by Equation 5:

where $i$ and $j$ denote spatial coordinates. $v$ and $u$ can be computed using max and min pooling. As the $r$-th gliding-box traverses the $W\times W$ image window, the total mass is calculated through Equation 6:

The multi-scale method involves computing lacunarity from different resolution levels in a Gaussian pyramid [3] as shown in Figure 4. The image undergoes recursive down-sampling and convolution with a Gaussian filter. The Kornia library [25] is used to construct the gaussian pyramid. At each level of the pyramid, lacunarity is calculated using Equation 2, capturing the spatial distribution of feature variations. Subsequently, these lacunarity values are upsampled using the bilinear interpolation to the original resolution, providing a comprehensive representation of lacunarity at different scales. The concatenated lacunarity values are then processed through a convolutional layer.

The lacunarity values are calculated at multiple scales for each channel resulting in $CS$ features maps where $C$ is the number of input channels and $S$ is the number of scales. We use a $1\times 1$ convolution operation to perform to learn a linear combination of the different scales for each channel to ensure the number of input and output channels are the same. This will allow the layer to be be easily integrated into any existing network to replace other pooling layers. This approach allows for a multi-scale analysis, enabling the model to capture intricate features at various levels of granularity within the image. The multi-scale lacunarity pooling layer is summarized in Algorithm 1.

The following experimental setup was used for training: a batch size of 128, cross-entropy loss function, Adam optimization with an initial learning rate of $0.001$, a total of 100 epochs, and early stopping implemented after 10 epochs if the validation loss did not improve. The experiments were conducted on two NVIDIA A100 GPUs. The datasets were split into 70$\%$ training, 10$\%$ validation, and 20$\%$ testing sets. We evaluated the performance of three CNN models: ConvNeXt [17], ResNet18 [12], and DenseNet161 [13], using various pooling strategies for image classification. A similar data augmentation procedure to [21, 29, 24] was added to the training data: random cropping and horizontal flipping. In the implementation of the multi-scale lacunarity pooling layer, we use two scales ($S=2$). The scale was set to this value as the spatial size of the feature maps into the pooling layer were $7\times 7$.

PlantVillage The PlantVillage dataset [14] is for multi-class image classification with 55,448 images divided into 39 classes representing background-only (out of domain images such as animals and buildings), healthy, and diseased plants. Images span 14 plant species: Apple, Blueberry, Cherry, Corn, Grape, Orange, Peach, Bell Pepper, Potato, Raspberry, Soybean, Squash, Strawberry, and Tomato and contains images of 17 fungal diseases, four bacterial diseases, two mold (Oomycete) diseases, two viral diseases, and one disease caused by a mite.

As seen in Table 1, the DenseNet161 model consistently outperformed the other two models, achieving the highest classification accuracy across all pooling methods. Specifically, DenseNet161 with GAP yielded an average accuracy of 97.49$\%$, while our pooling layer yields an improvement in accuracy to 98.07$\%$. Max pooling selects the highest values within the local region. This can be problematic when dealing with images that have varying lighting conditions or shadows such as leaf images since max pooling is sensitive to outliers [24]. In such cases, these extreme values from max pooling might not represent the overall characteristics of the leaf accurately. On the other hand, average pooling or the other pooling layers used calculate “smoothed” values giving a more balanced representation across the local regions and being less influenced by extreme values. The recently proposed fractal pooling technique [10], which aims to capture the fractal nature of textures, has a comparable classification accuracy in our experiments for DenseNet161 and ConvNeXt in comparison to the baseline GAP models, but this comes at a computational cost as discussed in Section 4.6. The DBC method shows an improvement in the accuracy for the ResNet18 and ConvNeXt models without adding additional learnable parameters.

LeavesTex1200 The real-world application, named LeavesTex1200 [4], employs commercial scanner-captured leaf surface images to classify Brazilian plant species. The dataset consists of 20 species, each represented by 60 images, totaling 1200 samples. Specimens were collected in vivo, cleaned, and aligned along the basal/apical axis. From each specimen, three non-overlapping 128 × 128 pixel windows were extracted to create texture images for classification purposes. As seen in Table 2 the multi-scale lacunarity performs the best when compared to our base lacunarity method. LeavesTex1200 images are distinctly focused on leaves without major variations in texture across a single scale. The leaves have a homogeneous vein structure in the leaves. This lack of textural heterogeneity makes the base lacunarity less effective since the base lacunarity is optimized for capturing textural variations within a single scale. However, the multi-scale lacunarity pooling layer becomes more adept at capturing subtle texture changes across different scales. When zoomed out, leaves exhibit greater textural heterogeneity, which the multi-scale method effectively leverages, resulting in improved performance compared to both the base lacunarity and baseline average pooling layers.

DeepWeeds DeepWeeds [23] comprises a total of 17,509 RGB images. These images were captured by a specialized ground weed control robot under natural field conditions, without controlled lighting. The dataset covers eight distinct weed species along with various non-weed plants for a total of nine classes. Each weed species is represented by more than 1000 images. DeepWeeds demonstrates better accuracy with the lacunarity pooling layer across ConvNeXt and ResNet18 models as shown in Table 3. However, notable accuracy improvements are particularly evident with the ConvNeXt model. Analysis of the results shows that fractal pooling exhibits better performance, especially with deeper models such as DenseNet161 [10]. Fractal pooling also uses residual block to improve the feature representation captured by this layer. DenseNet161 uses feature concatenation across multiple layers that enables the integration of features representing various levels of abstraction. In texture images, these features correspond to patterns observed at different scales. This multi-scale analysis has long been recognized as crucial for texture analysis [6], so it is expected that DenseNet models achieves the best accuracy in this context.

Comparing results across all the pooling layers in Tables 1, 2, and 3, it is evident that the lacunarity pooling layers consistently outperform the baseline average pooling layers on average. Specifically, the multi-scale approach demonstrates better performance compared to the base method described in Section 3.2. The DBC method, as discussed in Section 3.3, performs comparably to the average pooling layer. The operations involved in average pooling is similar with L₂ pooling, which is a form of power-average pooling. Given that we focus on feature extraction in our experiments, it is observed that the results for the L₂ pooling closely resemble those obtained through average pooling.

The reason why our lacunarity pooling method, especially the multi-scale variant, outperforms others lies in its capability for capturing the intricate details and varying complexities present in our image data. Unlike typical average pooling techniques that might overlook subtle nuances or textural variations, our multi-scale lacunarity pooling delves into different levels of detail, thereby increasing the model’s ability to discern patterns. Moreover, our pooling approach is finely tuned to use the self-similarity and spatial irregularities inherent in natural textures, which are prevalent in the wide array of plant images we dealt with across different datasets.

We used an Explainable AI technique, EigenCAM, [22] to visualize the salient regions identified in an input image by the lacunarity pooling layer and other pooling approaches with the ResNet18 model. EigenCAM showed more pronounced areas of significance in the input image, characterized by wider regions of activation for our proposed lacunarity pooling layer fusion model. The higher regions of interest, represented by the brighter highlighted areas, signifies stronger and more focused attention on crucial features within the images. This emphasis on specific regions is advantageous as this aids in capturing intricate details across varying spatial levels of the image. Compared to the other pooling layers such as max, average, and L₂, we observed improved performance with our model in terms of capturing more information across the input image. This suggests that our proposed lacunarity pooling strategy captures more extensive spatial information, thereby potentially enhancing the discriminative power of the model’s predictions.

The t-Distributed Stochastic Neighbor Embedding (t-SNE) [27] was used to visualize the high-dimensional feature representations learned by our fusion model and other pooling layers. The same random seed was used to fairly compare each t-SNE projection of the features extracted from each pooling approach. For max pooling as shown in Figure 6(b), there is not much separation or compactness in the projected space (similar to L₂ in Figure 6(c)). However, the lacunarity pooling layers appear to form more compact and separable clusters, especially the multi-scale lacunarity approach.

Moreover, to quantitatively assess the discriminative power of features extracted by our fusion model, we computed the log Fisher Discriminant Ratio (FDR) [24] scores for each class in the original feature space. The FDR score measures the ratio of between-class variation to within-class variation, with higher values indicating greater discriminatory ability. In our experiments, a higher FDR score signifies that the features are more effective at differentiating between classes, thus contributing to enhanced classification performance. When examining the FDR scores across different pooling layers in Figure 6, we observed notable differences. Particularly, the multi-scale lacunarity pooling layer consistently yielded higher FDR scores compared to the DBC lacunarity pooling layer. This suggests that the multi-scale variant of the lacunarity pooling layer captures more discriminative information from the input data, leading to better separability between classes.

When comparing the confusion matrices generated by different pooling methods for PlantVillage, we observed improvements in the classification performance of our lacunarity pooling multi-scale approach across a majority of classes. Specifically, out of the total 39 classes in our dataset, our method demonstrated better classification accuracy in 34 classes compared to the fractal pooling layer [10] as seen in Figure 7. For classes like Apple Scab the accuracy increased from 79.84$\%$ to 93.65$\%$ and Tomato early blight went up from 70.85$\%$ to 80.61$\%$. Classes which were misclassified as Corn Cercospora leaf spot instead of Corn Northern leaf blight, where the false positive percentage at 21.50$\%$ was reduced to 19.63$\%$. This indicates an enhancement in the model’s ability to correctly classify a wide array of plant diseases and species. The confusion matrices vividly illustrate how our lacunarity pooling method effectively captures the intricate features and subtle variations present in the images, leading to more accurate predictions across diverse classes.

Table 4 outlines the number of learnable parameters for each model, highlighting the parameter increase introduced by the fractal method and multi-scale lacunarity across all models. Despite the additional parameters introduced by the fractal method, there is no statistically significant improvement in accuracy compared to the baseline layers except for the DeepWeeds dataset. The multi-scale lacunarity pooling layer introduces extra parameters due to the $1\times 1$ convolution; however, there are improvements in the model’s performance. In the case of DBC for PlantVillage, the DBC lacunarity was a viable alternative to the GAP baseline models to achieve a slight increase in average overall performance without additional learnable parameters.