Intelligent Monitoring of Stress Induced by Water Deficiency in Plants using Deep Learning

This network performs a time-invariant classification analysis to identify water stress in chickpea plant shoot images. For this purpose, we use the convolutional base of the VGG16 and Inception-V3 network and remove the corresponding dense layers. Then, we perform Global Average Pooling [35] after the last Max Pooling layer. Global Average Pooling is preferred over fully connected layers for flattening the feature maps to a linear vector because it is more native to the convolution structure and enforces correspondences between feature maps and categories. Further, this layer has no parameters to optimize, which reduces the chances of over-fitting and is also more robust to the input’s spatial translation. Finally, we add two dense layers after global average pooling, the first one has 512 dimensions, and the following is the output layer with three dimensions, equal to the number of classes, as shown in Fig. 3. We initialize each dense layer using the Glorot uniform initializer [36] and use Softmax activation in the final output Dense layer (Equation 9).

Our CNN-LSTM architecture consists of two main parts: CNN image feature extractor and LSTM to predict water stress category from the extracted features. The architecture is shown in Fig. 4.

CNN feature extractor: We use VGG16 and Inception-V3 models pre-trained on the ImageNet dataset to extract visual features from chickpea plant images. We employ two different feature pre-trained extractors to determine if our approach is CNN architecture-dependent. In both the models, we remove the dense layers and apply Global Average Pooling after the final Max-Pooling layer to obtain 1D vectors of size 512 and 2048 for VGG16 and Inception-V3, respectively. We use the CNN network in time distributed form, that is, the same network is shared across all time steps of subsequent LSTM network. The unrolled version is shown in Fig. 4.

LSTM predictor: In our LSTM network, the number of sequentially connected cells is equal to the number of session data used for prediction, as shown in Fig. 4. This variable length of the LSTM network helps us analyze the effect of the number of data sessions on the prediction performance metrics - Accuracy, Macro Sensitivity, Macro Specificity, and Macro Precision. An ablation study on the same is reported in section III-D. The LSTM network output is fed into a Dense Layer of size 512 dimension, which is connected to the Dense output layer of size 3, equal to the number of water stress categories. We initialize each dense layer using the Glorot uniform initializer and use Softmax activation in the final output Dense layer (Equation 9).

Let us mathematically trace how our proposed network processes an input image sequence of plant shoot images. An image of an input sequence can be written as $i_{t}$ defined as an image at timestep t, $i_{t}\in R^{m\times m}$, where the image is of dimension $m\times m,\ m=224$. We define one entire image input sequence as

where $I\in R^{T\times m\times m}$, T is the number of time steps. Then, we chose either VGG16 or Inception-V3 CNN to extract features from images. We apply the chosen CNN feature extractor to each image of a sequence in a time-distributed manner, such that its weights $W_{c}$ remain the same for all LSTM timesteps and obtain corresponding features. One feature of the output feature sequence can be written as $x_{t}$ feature at timestep t, $x_{t}\in R^{d}$, where d is the size of the feature vector. We define one entire feature output sequence as

, where $F\in R^{T\times d}$, T is the number of timesteps. Therefore, the convolutional feature extractor simulates a function

Then, we feed the feature sequence to a sequence of LSTM units. An LSTM unit comprises a cell, an input gate, an output gate, and a forget gate, as shown in Fig. 5. The cell remembers values over arbitrary time intervals, and the three gates regulate the flow of information into and out of the cell. The equations for an LSTM are defined as:

$\!x_{t}\in\mathbb{R}^{d}:Input\ feature\ vector\ to\ the\ LSTM\ unit\\ f_{t}\in\mathbb{R}^{h}:Forget\ gate^{\prime}s\ activation\ vector,\\ i_{t}\in\mathbb{R}^{h}:Input\ gate^{\prime}s\ activation\ vector,\\ o_{t}\in\mathbb{R}^{h}:Output\ gate^{\prime}s\ activation\ vector,\\ h_{t}\in\mathbb{R}^{h}:Hidden\ state\ output\ vector\ of\ the\ LSTM\ unit,\\ \tilde{c}_{t}\in\mathbb{R}^{h}:Cell\ input\ activation\ vector,\\ c_{t}\in\mathbb{R}^{h}:Cell\ state\ vector,\\ W\in\mathbb{R}^{h\times d}:Weight\ Matrix\\ U\in\mathbb{R}^{h\times h}Weight\ Matrix\\ \ b\in\mathbb{R}^{h}:Bias\ Matrix\\ \sigma_{g}:Sigmoid\ function\\ \sigma_{h}:Hyperbolic\ tangent\ function\\$

Here, d and h denote input feature and hidden state dimensions, respectively. In addition, $\odot$ denotes element-wise multiplication. Weight and bias matrices are learned during training.

Then, we take the hidden vector, also known as output vector $h_{t}$ of the final LSTM unit $h_{t=T}$ and feed it as input to the classification block consisting of two dense layers.

For the classification block, the Input is $H=h_{t=T}$, and output is $P\in R^{C}$, where C is the number of classes, here $C=3$, $Classes=\{Before\ Flowering,Control,Young\ Seedling\}$. Then, the dense layer simulates a function

The proposed network’s output is equal to the classification block’s output, that is, P. As it is a case of multi-class classification, softmax activation is applied to the output of the final fully connected layer, also called the classification layer. It helps convert the output score corresponding to each class into a probability value between 0 and 1.

where $p_{i}$ is the predicted probability of a class represented as an element of the 3 dimensional output vector P.

CNN-LSTM Network: This section describes input dataset preparation for the CNN-LSTM Network. Each input data sequence consists of 32 images of a plant pot, one from every photo session. To ensure the robustness of classification, we consider photographs at all angles, such that all images of one data sequence have been taken from the same angle. Thus, for both JG and Pusa, we have 120 data samples each, in which there are 40 samples for each water stress category. We use RGB input images of size (224,224,3) and perform CNN network-specific image preprocessing on them before feeding them to our CNN-LSTM. While training the models, we perform data augmentation like horizontal flipping, rotation, shear, and translation to increase the training data’s size on the fly. We ensure that a linear transformation is performed equivalently for each image in the image sequence. Besides linear transformations, we randomly introduce Gaussian noise perturbations to a few training samples. A typical image noise model is Gaussian, additive, independent at each pixel, and independent of the signal intensity. Further, Gaussian noise in digital images, usually a consequence of sensor noise, arises during acquisition. As data acquisition in real-world settings will often be accompanied by noise, we perform noise data augmentation to train a robust model. We perform this augmentation by sampling noise intensities from a Gaussian noise $N$ distribution, which has a mean $\mu=0$ and a standard deviation $\sigma=15\%$ of maximum pixel intensity of any image in our dataset.

The value of $\sigma$ has been empirically chosen to provide the best robustness capability for noisy shoot images without making noise, a relevant feature for the model to learn. JG and Pusa noisy input samples are shown in Fig. 7 and 7.

CNN Network: This section describes input data preparation for fine-tuning pre-trained VGG16 and Inception-V3 CNNs. In this case, we equivalently treat all images of a given plant taken at different points in time. Thus, we have 1280 images per category and 3840 in total for each species. We use RGB input images of size (224,224,3) and perform network-specific image preprocessing before feeding them to the corresponding CNN for fine-tuning. Similar to the CNN-LSTM, we perform data augmentation like horizontal flipping, rotation, shear, and translation to increase the training data’s size.

We optimize the CNN-LSTM network (temporal analysis) and CNN networks (time-invariant analysis) by minimizing the categorical cross-entropy loss for water stress classification.

Where C is the number of classes, $Classes=\{Before\ Flowering,Control,Young\ Seedling\}$, $y_{i}$ is the true class, and $\hat{y_{i}}$ is predicted class, which is obtained after softmax activation, refer to Equation (9).

For our CNN-LSTM, we freeze the weights of the CNN and train the LSTM and the dense layer. To train this network, we backpropagate the loss and update the weights of the network using the Backpropagation Through Time [37] algorithm. The LSTM is trained on 32 sessions data and has about 2.4M and 35M trainable parameters with VGG16 and Inception-V3 feature extractors, respectively. On the other hand, to simulate time-invariant classification, we fine-tune the pre-trained VGG16 and Inception-V3 networks on our dataset using the Backpropagation technique [38].

The training is performed using a mini-batch size of 32 images and neural network weights are optimized using Adam optimizer [39] with learning rate $\alpha=0.0001$ and the other optimizer parameters being $\{\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-7}\}$. Further, we train each model for 200 epochs and use them for metric evaluation.

Training Environment We use Tensorflow and Keras DL framework to train our models and train them on a single Nvidia Tesla K80 GPU.

In this paper, we perform 5-fold stratified cross-validation for each model type (plant-variety and CNN pair). In other words, we divide the entire dataset into 5 equivalent subsets and train a model on 4 out of 5 of them. Then, we test on the remaining subset such that each subset acts as a test set once. Finally, we report the average scores across all 5 models for each performance metric - Accuracy, Macro-Sensitivity, Macro-Specificity, and Macro-Precision. We also repeat the cross-validation process 10 times to ensure robustness of the reported scores.

Firstly, we observe that VGG16 and Inception-V3 models’ performance is similar to the performance of our previous techniques, that is, a custom-CNN architecture and ResNet-18 architecture (as shown in Table II). ResNet-18, with fewer parameters than Inception-V3 and VGG16 models, has better water stress classification performance due to a higher degree of overfitting in the latter two larger models. Secondly, we infer that VGG16 and Inception-V3 models’ performance on JG images is better than Pusa images, which is consistent with our previous results, as shown in Table II. This can be attributed to the water-resistant nature of Pusa species and the water-sensitive nature of JG species. In other words, the visual changes introduced due to water stress are more prominent in JG than Pusa, thus making it easier to classify JG images into the three water stress categories. Lastly, we observe that Inception-V3 models produced better results than VGG16 models across both chickpea species. This can be explained by relating these results with both these architectures’ classification results on the ImageNet dataset. Inception-V3 outperforms VGG16 in both the top 1 and top 5 error rates (%) because it can extract better visual features [32, 31]. This suggests that image-based classification by transfer learning from Inception-V3 should be better than VGG16, consistent with the results shown in Table II. Therefore, time-invariant water-stress classification is network-dependent.

On comparing the results of temporal analysis of visual changes induced in the chickpea plant shoots due to water stress (shown in Table III) with the time-invariant analysis (shown in Table II), we observe that temporal analysis outperforms the best reported time-invariant scores by at least 14% for both chickpea varieties. Additionally, each chickpea variety’s results are consistent for both feature extractors, showing that the proposed CNN-LSTM technique is independent of the feature extractor network. We also observe that the CNN-LSTM performs better on JG than Pusa. This observation corresponds to the inherent water stress sensitivity characteristics of these two chickpea varieties. JG is water stress-sensitive, thus producing more noticeable visual changes over time than Pusa, which is water stress-tolerant. This can also be seen from the time-invariant analysis results in this paper, as shown in Table II.

This section examines the feasibility of the CNN feature extractor considered in this paper. We employ Gradient-weighted Class Activation Mapping (Grad-CAM) [40]. Grad-CAM uses the gradient information flowing into the last convolutional layer of CNN to understand each neuron for a class label of interest. As we use the same CNN feature extractors for time-invariant and our proposed temporal analysis, Grad-CAM visualization of the CNN network used for time-invariant analysis can be used to approximately extrapolate the behavior of these extractors in the temporal context. Further, we use the CNN network with Inception-V3 feature extractor, the best-reported network by this work. While applying Grad-CAM, we obtain the class discriminative localization map of width u and height v for a water stress class c by first computing the gradient of the score for that class, that is, ${y}^{c}$ (before the softmax), for feature maps $\alpha_{k}$ of a convolutional layer. These gradients flowing back are global average-pooled over the width and height dimensions (indexed by i and j respectively) to obtain the neuron importance weights $\alpha_{k}^{c}$.

After calculating $\alpha_{k}^{c}$, we perform a weighted combination of the activation maps and follow it by a ReLU. Without it, the class activation map highlights more than required and achieves low localization performance.

Subsequently, we superimpose the activation map (heatmap) with the original image to coarsely visualize which region it focuses on to classify water stress. In the images shown in Fig. 11 and 11, the intensity of yellow is directly proportional to the intensity of neural activation with respect to the predicted class. In other words, CNN focuses on the yellow highlighted regions in the image to make its prediction. From the figures, we observe that the CNN focuses on the shoot of the chickpea plant to predict water stress for both varieties. Further, this area of focus varies with the size and shape of the shoot. These visualizations explain and validate the use of chickpea plant shoot images to detect water stress.

On comparing the results in Table III and Table IV, we observe that the mean accuracy of our model on noisy test data is less than the accuracy on noise-free test data, by atmost 2.5%. This decrease is consistent for JG and Pusa varieties and VGG16 and Inception-V3 feature extractors. Even in the presence of noise, each chickpea variety’s results are consistent for both the feature extractors, thereby highlighting that the temporal technique is independent of the feature extractor. The model accuracy on the JG variety is greater than that of Pusa, which further validates the water-sensitive nature of JG over the Pusa variety. An interesting observation is the small standard deviation from the mean accuracy for all the CNN-LSTM models. A small standard deviation demonstrates that the model will not be adversely affected by noise, and its accuracy will remain reasonably consistent. Thus, the small decrease in classification accuracy and a fairly consistent average accuracy in noisy conditions makes this technique suitable for real-time deployment.

We draw the following inferences from the result. Firstly, the graphs in Fig. 9 and Table V demonstrate that by decreasing the number of session data for training the model, we decrease its ability to discern water stress conditions. This observation is reasonable because a longer image sequence will learn better differentiating features, especially since water stress on the shoot is prominent in the later stages of growth. Secondly, the performance metric curves (as shown in Fig 9) for a given plant species are similar for both the feature extractors. This emphasizes that temporal analysis using CNN-LSTM models has negligible dependence on the CNN feature extractor used. On the contrary, the time-invariant classification of water stress depends on the CNN architectures employed, as shown in Table II. This observation further reinforces the merit of temporal analysis. Lastly, we also observe that these curves and final scores are similar across both species, thereby demonstrating that temporal analysis performs well across different chickpea plant species. Species invariance is another beneficial characteristic for the real-time deployment of this technique.