Importance of realism in procedurally-generated synthetic images for deep learning: case studies in maize and canola

The quality and quantity of agricultural outputs depends on many factors such as crop genotype, disease, growth patterns, nutritional deficiency, and environmental conditions. The assessment and determination of plant growth, morphology, function, composition, disease detection, or any phenotype, is collectively called plant phenotyping. Many of these factors are evident from consistent monitoring, which has been found to be crucial data towards helping with the successful cultivation of new crops. This has traditionally been done for massive numbers of plants in a manual fashion by plant breeders, however this is both time-consuming and often overly reliant on intuition [13]. However, the potential of using automated high-throughput phenotyping has long been recognised as an important step forward [35], and is now in the initial stages of being used to help cultivate new crop varieties [33, 7]. Furthermore, this will become even more important with increasingly extreme climate condition.

Image-based plant phenotyping is important and is gaining in popularity as a method to automatically extract useful information from plant images [34] and to identify phenotypic traits throughout a plant’s life [10]. One possible method to identify many features including plant architecture is to use image processing techniques to segment the plant from the image, and then to use skeletonization techniques to identify different plant components. However, accurate skeleton extraction is challenging due to the complex geometry of plants and self-occlusion in the images [6]. Moreover, the identification and removal of extra branches that can arise from noise in images [5] is also difficult. Another increasing useful approach to analyze the large volumes of data generated by phenotyping platforms is to use machine learning techniques. Artificial neural networks (ANNs) and deep learning are providing superlative results on many data and image analysis tasks [11], and have successfully been used in image classification, multi-instance detection, and segmentation [16]. ANNs have shown better accuracy than image processing algorithms in certain complex image-based plant phenotyping tasks, such as leaf counting, age estimation, mutant classification [38], plant disease detection and diagnosis from images [23], classification of fruits and other organs [29], and pixel-wise localization of root and shoot tips [30]. These are classified as regression (counting) or identification (detection or classification) problems, and various types of ANNs with complex architectures have been implemented to solve them.

One of the most popular and powerful networks for deep learning is the convolutional neural network (CNN), which have layers of neurons representing image pixels, and connections between layers that perform linear filtering. They are capable of learning highly discriminative features during the training phase, and can classify plants without needing segmentation, or feature extraction. Many works in the literature provide examples of different plants where important phenotypic traits have been extracted from plant images with them; e.g. automatic joint feature and classifier learning for temporal phenotype/genotype classification [24], crop lodging detection [20], and leaf counting [2]. Deep Plant [17] uses CNNs to learn feature representations from leaves using 44 plant species. Deep Plant Phenomics (DPP) [38] has pre-trained CNNs built using TensorFlow for common phenotyping tasks involving object detection, object counting, and semantic segmentation.

Supervised machine learning methods require a training set where correct labels for the phenotype of interest are known, and the model is trained on the labelled dataset. A major challenge for deep learning applications for plant phenotyping is the availability of a large quantity of annotated data for training. Image datasets of the desired species, environment, scale, and size, labelled with the phenotypic properties of interest, are likely not available, and might be difficult to obtain due to the large cost of collecting and annotating data [37]. Hence, a motivation arises to use an alternative source of training data.

Computational systems for modelling and simulating plants have been an important area of research. Such simulations can incorporate different plant geometries [19, 26, 14], environmental factors [3, 40], and mechanistic controls [26, 31]. One of the most common systems used for this purpose are Lindenmayer systems (L-systems), introduced by Lindenmayer in 1968 [18] as a formalism for simulating the development of multicellular organisms in terms of division, growth, and death of individual cells. They can be used as a mathematical theory of plant development [19] and for simulating plants using a simulator such as [27].

The idea of using synthetic images from these simulations to train ANNs for the purpose of identifying phenotypic properties was initially explored by Ubbens et al.  [37]. This idea overcomes many of the obstacles of manual annotations. If the phenotype of interest is built into the L-system model, then the correct annotations are automatically known. For the purposes of identifying plant organs and architecture, these can be precisely incorporated into an L-system model [19, 21]. In comparison to the cost of real data collection, and the precise labelling of them, it is easier to obtain synthetic data from these procedural models. Once a model has been created for a species, it is possible to generate arbitrarily large synthetic images datasets at no extra expense. They showed [37] that synthetic images from an L-system can be used to augment datasets of real plant images or can even be used alone as a large source of training data. Using both together led to better prediction of leaf count in Arabidopsis thaliana. A similar approach was also used with maize to predict leaf count [22] but using a maize procedural model within Plant Factory Exporter [1] instead of L-systems to create synthetic images. Here, results were mixed and there were cases (depending on the number of synthetic images used) where adding synthetic images to a real dataset for training improved prediction versus only training on real images, but often adding synthetic images did not help.

Cieslak et al. [9] demonstrated an approach for creating L-system models using real images for guidance, for maize and canola as a case study. Primarily, the modelling process has two phases: the first is the qualitative or topological specification of the developing plant architecture, and the second is the calibration, so they can accurately capture the growth and visual characteristics in order to generate realistic synthetic images. Indeed, one might expect that the more realistic the synthetic images look, the more replaceable they are for real images for training. But the degree to which this is true is not yet understood. Their approach also demonstrates that it is particularly easy to take an existing L-system model and adapt it to a new environment, which is another big potential advantage of using images from L-systems over manually labelling in each new environment.

In this paper, we continue studying situations where it may or may not be helpful to use synthetic images for training ANNs. We use two complex plants, maize and canola, and start with the same L-systems created in [9].

Maize is an important cereal crop, with thin leaves, leaf occlusion, and large leaf curvatures, that make them difficult to analyze. We systematically vary different amounts of real images together with synthetic images from L-system simulations, and measure success at leaf counting on real images using both mean absolute loss and Pearson’s correlation. We find that using synthetic images in addition to real images always improves success, with a large benefit when the number of real images is small, and the benefit decreasing as the number of real images increases.

The next experiment involves canola, which is even more complex in our dataset with larger numbers of occluded leaves. We try to predict the inflorescence branch numbers. Here, ANNs trained with only synthetic images and no real images work reasonably well at predicting inflorescence branch numbers in real images. However, in most situations real annotated images work better for training than a combination of real and synthetic images. While visually these synthetic images look similar to real plant images, refining the L-system models for the purposes of creating even more realistic synthetic images has a considerable benefit. The accuracy of the predictions was quite sensitive to the calibration of the L-systems, and improving the realism of the synthetic images had a substantial effect on the predictive accuracy. The ANN prediction results were used to help improve the L-system calibration process, which were in-turn used to create even more accurate L-systems and synthetic images. This process can improve both the phenotype predictions, but also the L-systems themselves.

In maize, by comparing the model’s mean absolute loss for real and synthetic datasets (blue and green lines) in Figure 3 (as with the red and pink lines), it is evident that for a fixed number of real plants, augmenting with synthetic images always improved results. This was especially true when there were fewer real plants, such as 2 (51 images) where adding synthetic images cut the mean absolute loss almost in half from $1.01$ to $0.55$. Training with only synthetic images requires no groundtruth labelling, and provided a better mean absolute loss than training with 51 images. Also, training with only one real plant (26 images) and synthetic images had a mean absolute loss of $0.51$ which was comparable to training with eight real plants (201 images) with a loss of $0.45$. Hence, augmenting with synthetic images can significantly reduce the amount of labelled real images needed, and they can be outright eliminated in some cases without a large cost in accuracy. The results also present an overview of the minimum number of real plant images needed to get a desired result.

It should be noted that it was relatively easy to produce visually realistic synthetic images of maize due to its simpler architecture, and indeed using these synthetic images was quite helpful. We achieved a RMSE of $1.16$. In contrast, in [22] the RMSE obtained using their synthetic maize images was between $1.50$ to $1.75$ with almost equal amounts of real data augmented with synthetic images. Their $r^{2}$ ranged from between 0.5 and 0.75, whereas ours mostly ranged from 0.62 and 0.89 after augmentation. The predictions in [22] were sometimes worse when augmenting with synthetic images, and our results were always improved, and our RMSE and $r^{2}$ were better. While their real images of maize were taken at the same imaging facility, we caution against any firm conclusions as their real images were different and include up until 66 days after planting, versus 29 days in our dataset which could affect predictions. However, their often worse prediction results when augmenting with synthetic images were likely partially caused by less realistic synthetic images. Indeed, they indicated that despite maize following alternating phyllotaxy, their procedurally generated images had successive leaves that emerged from the stalk at random angles [22]. Our L-system was built to properly follow alternating phyllotaxy. This shows the importance of biological realism in creating synthetic images. Figure 6 plots real leaf count versus predicted count, and shows the frequency of the data points.

For canola, results were mixed. There were later stage images with more overlapping and occluded components with flowering images. In addition, canola has a more a complex branching pattern, so the CNN has to distinguish different orders of branches. The fourth L-system variant showed by far the most promising results. While there were some cases where adding synthetic data led to a slightly better prediction, it mostly was not helping for all five model variants constructed. There are however several points of interest. Figure 7 presents the distribution of relative count difference for inflorescence branch count in canola, while testing on 100 real canola images and trained on all the remaining real images augmented with $C^{S}_{4}$.

The columns of Table 2 demonstrates the improvements made to the L-systems, thereby leading to improved deep learning results. Therefore, this exercise of obtaining deep learning results followed by a manual visual inspection and calibration of the model had a great role in improving the L-system itself and further results. Pearson’s $r^{2}$ values show that the synthetic images from the fourth canola L-system were quite similar to the real canola images as the Pearson’s correlation coefficients indicates the strength of associations between variables. The $r^{2}=0.86$ when trained on 32 real images and tested on 100 real images, and $r^{2}=0.81$ when trained on the $1200$ synthetic images generated from $C^{S}_{4}$ and tested on the same 100 real images.

It is evident that these results were quite sensitive to the precise L-system construction. For example, when trained on only $C^{S}_{2}$, the mean absolute loss was exceptionally large at $16.6$, but it was quite good when trained only on $C^{S}_{4}$ at $1.45$. This again shows that producing realistic synthetic images is extremely important. It also shows the simulated images alone even in canola can perform quite well, while requiring no groundtruth labelling.

One interesting contrast between the maize and canola experiments is that when both training and testing on only synthetic images, the mean absolute loss in maize was extremely low at $0.08$, whereas in canola with $C^{S}_{4}$, it was much higher at $1.19$. This seems to indicate that the canola L-system is considerably more complicated and more difficult for the ANN to make accurate predictions. This was one of the important points of discussion in [37] regarding the A. thaliana model.

Finally, we chose to always train with all the available synthetic images when training rather than smaller numbers (such as the same number of real images). This was done because it is easy to generate arbitrarily large sets of synthetic images from the same L-system. Also, by contrasting Table 2 and Supp Table 1, the latter being a test to see if the amount of synthetic images played a role in prediction accuracy, showed that it was better to always augment by all the synthetic images rather than gradually adding them in.

Machine learning and L-systems plant modelling are two broad and impactful fields in plant phenotyping research. Here, we showed how each can help the other. This study conducted computational experiments in maize and canola regarding the use of synthetic images of plants for training artificial neural networks (ANNs). For the task of leaf counting in maize, we calculated the mean absolute value of the difference between the predicted and correct number of leaves (mean absolute loss). We found that augmenting with synthetic images was always beneficial, and it was especially so if there were fewer real images, with the benefit gradually shrinking as the number of real images increased. This confirms similar findings from [25] but in the plant phenotyping domain. The L-system used to create synthetic images was quite realistic, following the real architecture of maize, and this realism may have been a key contributor to its improved success versus others in the literature. This is similar to van Breugel et al. who showed [39] how imperfection in synthetic data affects downstream machine language tasks. Furthermore, it was more accurate to only use synthetic images with no groundtruth labelling than using 51 real images. This confirms the potential benefits for using synthetic images for training ANNs.

The task of inflorescence branch counting in canola was conducted and results were evaluated using a similar procedure. This study again showed the importance of realism. In some cases, the L-system models require calibration, and results can be very sensitive to changes in the models. Small changes in L-systems can improve the synthetic images remarkably. Here, we also saw that the deep learning results served to quantify how replaceable the synthetic images were for real images rather than using only visual inspections. For example, synthetic images alone from an early L-system variant when tested on real images resulted in a high mean absolute loss of $16.61$. After refinement of the L-system to make the model calibration more accurate, and the images more realistic, this number was reduced to $1.45$. While augmenting with synthetic data only sometimes improved results, this result also shows that using only synthetic images worked reasonably well at predicting on real image, while requiring no groundtruth labelling.

This study opens the door for many potential future research directions. Directly, this includes more refinement to the canola L-systems. Secondly, we would like to further test the level of realism required for synthetic images to become useful for training purposes in additional dimensions beyond those explored between the five L-systems variants. Another interesting objective would be to study advantages of it being comparatively easy to adjust an L-system to a new environment [9]. One could compare an ANN trained on real images from a different environment from the test set to those trained with synthetic images from the same environment as the test set. If it is better to use synthetic images from the correct environment, then this provides an opportunity to only use one L-system-based model while only needing to adjust the environment for every new use of phenotyping, instead of manually labelling a new large set of images with the new environment.