Label-free Anomaly Detection in Aerial Agricultural Images with Masked Image Modeling

In precision agriculture, Unmanned Aerial Vehicles (UAVs) have emerged as a pivotal tool for monitoring agricultural landscapes efficiently. UAVs provide much higher resolution images compared to satellite images, thus capturing fine grained details on the agricultural fields. Accurate anomaly detection in UAV images is crucial for the early identification of potential issues such as pest infestations, diseases, and nutrient deficiencies. The dynamic and diverse nature of agricultural fields further compounds the challenge, as anomalies can vary greatly in appearance due to factors such as crop type, growth stage, and environmental conditions as compared to other anomaly detection settings, compared in Figure 1. Thus, there is a need for a completely label free approach to training anomaly detection models so that it can be applied across different crops for different kinds of anomalies. Traditionally, supervised learning methods have been used for anomaly detection systems [21, 11, 4, 2]. These methods are inherently limited by their dependence on large sets of annotated data, which are labor-intensive to create and may not capture the full spectrum of possible anomalies. Even in case of unsupervised and self supervised methods [6, 30, 29, 15, 12] where explicit anomaly labels are not used, there is a dependence on using only “normal” data for training thus making it necessary for a user to curate normal data that does not contain any types of anomalies.

To leverage self-supervised learning through masked image modeling, we utilize Masked Auto-encoders (MAE) [9] to effectively learn normal features from unlabeled image samples. This “normality” then facilitates the detection of anomalies through higher reconstruction errors for patches containing anomalies. Incorporating a Swin Transformer-based Masked Autoencoder [7] enables our model to learn both local and global features, ensuring robust detection across a wide range of anomaly types. Our work introduces an approach that also learns to detect anomalies with abnormal samples within the training data. This inclusion allows users to simplify their data collection pipeline by removing the need to curate “normal” data. This enables the identification of a wide array of anomalies without training multiple models for detecting different anomalies (by removing anomalies of a particular type from the data). The effectiveness of our approach is demonstrated through extensive evaluation on the Agriculture-Vision Challenge dataset [5], showcasing significant improvement of 6.3% in mean Intersection over Union (mIOU) score and generalization across all the given anomaly classes.

The Agriculture-Vision dataset[5] provides multispectral aerial images for fields at 10cm/pix resolution along with annotations for anomalies of 9 types - drydown, planet skip, water, weed cluster, nutrient deficiency, endrow, double plant, waterway, and, storm damage. Unlike other datasets that may include hyperspectral and multispectral data for general land cover classification [10] or crop type identification [23], Agriculture-Vision specifically targets the semantic segmentation of agricultural patterns for recognizing various field anomaly patterns crucial to farmers.

Supervised image segmentation approaches like FusePN [11] and AAFormer [21] have demonstrated competitive performance on detecting anomalies in UAV images. AAFormer uses a transformer based architecture. FusePN[11] uses a multimodal fusion approach fusing RGB and NIR bands of the image along in an encoder-decoder style architecture with additional modifications for inference efficiency. The limitations of supervised methods have steered research towards unsupervised and self-supervised learning approaches, where the focus shifts to learning from unlabeled data.

One-class classification (OCC) [22, 30] models provide another approach for anomaly detection utilizing high-level semantic information for anomaly identification in the feature space. OCC based anomaly detection uses high-level semantic information and distance metrics for anomaly scoring. However, they encounter challenges such as i) mode collapse and ii) overlook low-level structural features due to their focus on compact feature representation [18]. Anomaly Segmentation based on pixel Descriptors (ASD) [13] addresses anomaly segmentation in high spatial resolution (HSR) imagery by using deep one-class classification with discriminative pixel descriptors through abnormal sample generation, promoting descriptor compactness for normal data and diversity to prevent model collapse. ASD employs a multi-level, multi-scale feature extraction approach to capture low-level and semantic information.

Reconstruction-based anomaly detection methods like Attribute Restornation Network (ARnet) [28] and Deep Feature Reconstruction (DFR) [26] utilize autoencoders (AE) with an encoder-decoder architecture to capture the manifold of defect-free images to differentiate between normal and anomalous data based on reconstruction fidelity. These models, trained solely on normal imagery, are expected to yield higher reconstruction errors for anomalous inputs, using metrics such as mean square error (MSE) for anomaly quantification. Techniques to enhance anomaly detection capabilities include image degradation and subsequent restoration, notably through inpainting methods like RIAD [29], which mask parts of the image to challenge the model’s reconstruction abilities. Despite their success, these methods struggle, as with progression in training it inevitably involves the anomalies in the reconstructed image. This is because models favor learning all the information from input, including both background and anomalies simultaneously. In terms of architectural elements, convolutions are prone to learn identity mapping (from input image to output image) as their receptive fields are biased towards learning local spatial features [14]. To address these limitations, recent approaches propose integrating Transformer based reconstruction method like IntRA (Inpainting Transformer) [17] which pose anomaly detection as a patch-inpainting problem and propose to solve it with a purely self-attention based approach discarding convolutions.Other transformer based approaches like MAE which mask $\sim$75% of the images patches and use the remaining to reconstruct the complete image. MAEDAY [20] leverages MAE for image-reconstruction-based anomaly detection method that utilizes a pre-trained model, enabling its use for Few-Shot Anomaly Detection (FSAD). We provide a class-wise comparison of these methods in Table 1.

Objective - The anomaly segmentation of a high spatial resolution (HSR) image X with dimensions H×W×B (height, width, and Number of multi-spectral bands) is defined by a mapping function f, transforming X into an anomaly map A with dimensions H×W .

We use the SwinMAE architecture (Section 3.3) along with the anomaly suppression loss (Section 3.4) to learn the “normal” feature embeddings from the farm images. The architecture of SwinMAE is divided into encoder and decoder as given in Figure 4. The encoder of the Swin MAE network begins by partitioning the input image into non-overlapping patches and mapping these patches into a high-dimensional embedding space through a linear transformation, which allows for more complex feature extraction.The patches then go through widow masking strategy. The Swin Transformer blocks, which form the core of the encoder effectively captures hierarchical features by using window based multi-head self-attention (W-MSA) followed by shifted (SW-MSA) window mutli-head attention. After each Swin Transformer block the embeddings are merged to reduce the number of patches by half while doubling the dimensionality of the embeddings. This operation aggregates information from adjacent patches and reduces the spatial resolution, while increasing the feature dimension. The encoder consists of 4 Swin transformer blocks as given in the original work [7]. The decoder in the Masked Autoencoder (MAE) network undertakes the task of reconstructing the input image from its condensed and partially masked representation produced by the encoder. It begins by processing the mixed embeddings through Swin Transformer blocks. Following this, the decoder employs a series of expanding operations that gradually restore the spatial resolution of the image. This is essentially the reverse of the encoder’s patch merging process. As the resolution is increased, the complexity of the feature representation is reduced, aligning it closer to the original input space. Layer normalization steps interspersed within these operations ensure stable learning by maintaining a consistent scale of the features. The final stage involves a projection of the embeddings back to a space that mirrors the original image’s patches, which are then reassembled to predict the full image, effectively filling in the masked regions with the learned information. This reconstructed output aims to be as close as possible to the original unmasked image.

Given an input image $X$, a subset of patches $P$ representing 25% of $X$ are fed into the SwinMAE model as given in Figure 5. This process is repeated $K$ times with different random subsets, where $K=32$ so that each patch is likely to be masked once, allowing for accurate reconstruction assessment for each pixel.For each iteration $i$, a reconstruction $R_{i}$ is obtained and compared with $X$ to compute a reconstruction error map $E_{i}$. The error for each pixel $j$ in the error maps is averaged across all $N$ reconstructions to obtain an averaged error map $\bar{E}$:

The final anomaly map $A$ is produced by applying a threshold $\theta$, determined by identifying the knee point [19] in the distribution of $\bar{E}$, to binarize $\bar{E}$ into anomalous ($A_{j}=1$) and non-anomalous ($A_{j}=0$) pixels:

We compare SwinMAE with several state of the art anomaly detection algorithms ranging from convolutional, GAN-based, One-class classification (OCC), Transformers and MAE based models. To be specific quantitative analysis of Swin MAE in comparison to several models such as DSVDD [30], RIAD [29], ARNet [28], GANomaly [1], and the Anomaly Segmentation model based on Pixel Descriptors (ASD) [13], Inpainting Transformer [17] and MAE. Our quantitative experiments in Table 1 reveals that Swin MAE outperforms previous unsupervised and self supervised approaches across several anomaly categories. Supervised models like AAFormer [21] achieves an mIOU of 41.2 and Fuse-PN [11] achieves a dice score of 82.71 acting as strong baselines. For instance, in the Double plant and Endrow category, Swin MAE achieved an mIOU of 46.6 and 43.8 which is significantly better than other models. Swin MAE performs on par if not better across all the categories of anomaly. Moreover, when augmented with an anomaly suppression loss, the model’s proficiency further increases, where it recorded a jump in all categories showing the efficacy of the anomaly suppression mechanism during training. A Masked autoencoder also performs well compared to the other models , which enforces the fact that masked image modelling is very effective in detecting anomalies across multiple categories. However as Swin MAE learns both local and global features it outperforms MAE in all the categories. It is interesting to note that anomalies which spread out in the images, e.g., weeds, water, etc. are not caught well with reconstruction based methods, perhaps, owing to the fact that despite 75% masking, the pattern is caught by the encoder and successfully reproduced, thus acting as the normal distribution.

Label-free training We also analyze training SwinMAE with and without anomaly samples in the training set for a given class, as detailed in Table 2. Note that all methods in Table 1 are run without anomalies in the training data, i.e., in a Leave-one-out fashion. In general, it is observed that with our method, accuracies remain almost the same (or increase slightly by a few points in some cases) for all classes of anomalies. The median IOU only reduces by 3.5 points. For two classes - drydown and water, the IoU increases significantly when these classes are omitted during training. This effect can be attributed to the distinctive nature of these two classes, i.e., water and drydown not being present at all in the underlying data, which are less likely to be reconstructed if they are absent from the training set due to their minimal distribution overlap with other types of anomaly classes which are present in the training set.

These results indicate that Swin MAE is robust across most of the anomaly categories while it is trained with anomalous samples included in the training set and significantly outperforms other models which are trained on anomaly free samples which enables us to have to a single model better generalizing to all categories. Swin MAE ability to model both local and global dependencies across patches enables it to detect anomalies better than other transformer based techniques like a simple MAE and InTra.

Qualitatively we can observe in Figure 6(a), that SwinMAE is able to segment irregular anomaly patterns across multiple classes. Due to large intra-class and inter-class variance within every class in the Agriculture Vision Dataset [5], we observe that in a few classes like Nutrient Deficiency as shown in Figure 6(a) there are other miscellaneous anomaly patterns apart from nutrient deficient areas that get segmented as anomalies, similarly for classes like water and double plant. In Figure 6(b) there are objects such as Trees in (Endrow), Roads in (Drydown, Double Plant) and cars in (Weed) that are detected as anomalies because they deviate from normal agricultural patterns and global pattern of the image, however these are not agricultural anomalies. As our model is trained in a self-supervised setting and without agricultural anomaly specific supervision, any pattern that deviates from the global setting of the query image might be labeled as anomaly. This also solidifies the fact that Masked image modelling based transformer models are efficient global feature learners.

We propose a masked image modelling based self supervision methodology to detect anomalies in agricultural fields using UAV images. We demonstrate the effectiveness of masked image modelling through SwinMAE[7] and MAE[9] methods by benchmarking it on the Agriculture Vision dataset and improving mIOU by a margin of 6.3% compared to other unsupervised and self supervised methods. We also show that using an anomaly suppression loss[25] adds robustness even when trained with training data containing anomalous samples. Typically, the anomaly detection methods train only on normal samples (excluding the anomalous classes’ patches from the input). This improvement allows a simplification of the pipeline so that anomaly detection pipelines can be trained without filtering out “normal” samples. With this improved methodology, it is also possible to have a single model that can generalize across all the anomaly classes. Our work should provide a definitive direction in creating an anomaly detection system that generalizes to various anomalies and relies less on human interventions. As future work, we will investigate if the single model is able to find “new” anomalies beyond the one described in the dataset and if the method is useful in domains other than UAV images as well. We will also investigate if a generic model will be able to work across different types of crops.