Dataset and Performance Comparison of Deep Learning Architectures for Plum Detection and Robotic Harvesting

Automated tree crop harvesting is a well known focus in agricultural robotics and has the potential to reduce the high labour costs associated with fruit picking. Accurate target detection is the first step for most harvesting systems and inaccuracies at this early stage can disproportionately affect system performance. Optimising detector accuracy should therefore be a priority in agricultural robotics and automation. As deep learning methods from computer vision have advanced rapidly, the application of these in agriculture has become common place. However, these algorithms require a large and representative corpus of training data to perform well, something not currently available for the full range of image conditions seen in harvesting.

Eye-in-hand sensing provides distinct advantages for fruit harvesting by allowing for continuous feedback control right up to the point of picking. This decreases positioning error upon gripper final approach. Off-hand sensors provide a much more regular view of target fruit, but are difficult to re-position for better viewing angles. Unfortunately, the added sensor movement of eye-in-hand systems results in much more variable imaging conditions. Easily accessing the tree canopy also dictates the use of a compact, and preferably low cost, sensor. All of these present additional challenges for object detectors deployed in harvesting systems.

To address these challenges, new datasets representative of eye-in-hand harvesting conditions are required. For this, images were gathered during a field evaluation of a prototype system that uses an eye-in-hand configuration for harvesting plums on a 2D trellis [4]. A small and cheap Intel Realsense D435 RGB-Depth (RGBD) camera is used. Many image artefacts caused by the eye-in-hand setup can be observed in the dataset, see Figure 1. A total of 700 images are extracted and annotated, these are split evenly between day and night datasets. Two previous generation object detection deep learning networks, Faster-RCNN and YoloV3, are benchmarked against this dataset, along with two current networks; RetinaNet and CenterNet.

Depth sensing is a required modality for harvesting, so the fusion of depth information for detection is also tested. Both early and late data fusion is trialled using the RetinaNet architecture. Early fusion treats the network input as a 4 dimensional RGBD image, while in late fusion the image and depth features are concatenated after being extracted by parallel network backbones.

Key contributions of this work are the presentation of two realistic datasets from an eye-in-hand harvesting operation, the benchmarking of multiple current deep learning architectures against these and the assessment of RGBD data fusion for detection. All data and models are available publicly at the link in Section 3.1.

The primary limitation of this work is the relatively small dataset size, which makes it impossible to test which benchmark networks perform best with very large datasets. Numerous works have shown a clear correlation between training set size and network performance for fruit detection, though the exact performance-by-size function is not linear and varies between architectures.

Fruit detection is a key problem that is common in the literature for goals such as yield estimation, crop health assessment and harvesting [21, 22, 1, 9, 32, 2, 3, 34]. The most basic form consists of placing bounding boxes around each of the fruit in an image. Traditional computer vision methods have been extensively employed, while many recent works make use of deep learning tools. These require large amounts of training data, though this requirement can be relaxed using data augmentation and simulation methods, or transfer learning techniques.

Specular reflections from round fruit, combined with local image gradients are used in [37] for object detection under controlled lighting. A Support Vector Machine (SVM) is trained to detect apples using thermal imagery in [8]. The accuracy performance of SVM object detectors has now been surpassed by deep learning approaches, though they remain competitive for classification when using low dimensional hand-engineered image features [17, 15]. Edge features are used with Hough voting and an SVM classifier by [33] to identify green citrus fruit under varying illumination. Similarly, [25] make use of a bas-relief representation with edges, Hough voting and an SVM to also count green citrus fruit. [27] perform RGB and depth channel thresholding to identify point cloud blobs corresponding to apples. These are separated using a Euclidean distance metric which is tested in the field under semi-controlled lighting conditions.

Avocados, apples and lemons are counted using a hand held camera in [36]. Both Faster-RCNN and the Single Shot multibox Detector (SSD) network are tested with a video based object tracker to prevent multiple-counting. Avocados were the most accurately detected fruit for both networks, followed by lemons and apples. This is counter-intuitive given the strong colour features present for the latter two fruit, but may be explained by a larger set of training images for avocados.

Harvesting-oriented apple detection data is gathered in [19]. The YoloV3, Mask-RCNN and Faster-RCNN architectures are thoroughly tested, along with their own deep learning model, described in [18]. This implements the idea of focal loss, similar to RetinaNet. Unfortunately the dataset and trained models are not released for comparison and, unlike the present work, the harvesting system does not use an eye-in-hand camera. [11] train a multi class Faster-RCNN detector to distinguish between different obscuration cases for apples, including those behind branches or wires, so that they can be harvested appropriately.

Camera movement is exploited to build 3D maps using Structure From Motion (SFM) in [13]. SFM is effective where camera motion is smooth and unobscured, but comes at a computational cost and struggles with burring or poor exposure, both common during harvesting. Previous work within the same group examined apple detection using image, depth and radiometric data with Faster-RCNN [14]. This multi-modal data was gathered using a robotic platform at a fixed distance from the trellis and results indicated that early depth fusion alone was not effective, but combining image, intensity and range produced the best detector.

Bounding boxes can be trivially extracted from image instance semantic segmentation masks, so pixel wise classification is one approach to fruit detection. This technique is considered by [26] who trial various local pixel features to perform classification on multi-spectral images, and find local binary features to be most effective. Numerous pixel wise classifiers are applied to specifically detect plums in [28] using a variety of manually specified colour space features.

Various sensing modalities for fruit detection, such as hyper-spectral, thermal and stereo vision are explored in [20]. Thermal imagery is found to only be useful at certain times of day, while geometry and colour are identified as the strongest features for distinguishing fruit. Similar observations regarding the limitations of thermal imagery use are made in [5], and by [10] who also present a novel algorithm for fusing thermal and RGB imagery.

Depth data is frequently leveraged for object detection, as in [35] where a multiscale implementation of Faster-RCNN is improved with the late fusion of RGBD imagery. A Microsoft Kinect V2 camera is used, which requires avoiding direct sunlight and is kept a fixed distance from the fruit trellis. This provides detailed and consistent depth data, intended to be used for fruit counting rather than harvesting. LiDAR sensors have large depth ranges and very high accuracy, but low resolution. [12] leverage the lighting invariance of LiDAR to detect fruit in point clouds, captured with and without a commercial air blower being applied to the crop. Combinations of data both with, and without, the blower active led to improved single frame detector accuracy, though it was not beneficial for yield prediction. Late fusion of near infra red and RGB imagery is used to detect a range of fruit using Faster-RCNN in [31] and dataset size is found to have a critical impact on detector performance.

To the best of the authors’ knowledge, this work presents the first example of an eye-in-hand RGBD dataset for object detection gathered during actual tree crop harvesting. Unlike much existing literature, multiple current generation object detector architectures are benchmarked and publicly released, along with day and night datasets.

To test the performance of current generation object detector architectures on a fruit picking task, images are gathered during the trial of a robotic harvesting system, shown in Figure 2. This consists of an eye-in-hand RGBD camera mounted to a gripper on the end of a 6 Degree of Freedom (DoF) robotic arm. Several hundred images of red plums are gathered during both day and night operations, these are manually labelled to create two datasets. Four deep-learning detectors are trained and evaluated on this data. Each network architecture is trained and tested on the day and night datasets separately. Both pretrained weights for transfer learning from a non-agricultural task, and randomly initialised weights are used with each architecture. Separately to the benchmark tests, RGB and depth fusion is tested with RetinaNet on the day and night datasets.

Each RGB network is tested against the day and night dataset separately, both with and without transfer learning. The impact of depth fusion is assessed using the RetinaNet architecture. All tested configurations are summarised in Table 2 with PR curves for each dataset shown in Figure 7.

Some training runs were unstable, resulting in no validation set AP increase during training. Each unstable training configuration was tested three times and in all cases the three runs failed. No training runs failed where an AP value is reported.

Differences between performance reported on the COCO dataset, and the two datasets tested, were surprising. CenterNet performed poorly on both plums tasks, despite having the highest stated COCO accuracy, it was also the slowest. RetinaNet was effective for day time detection, with augmentation, transfer learning and backbone size all playing a role in overall performance.

Applying transfer learning had an overall larger impact than architecture selection and is essential when using small datasets, as in many agricultural applications. Conversely, using the ResNet-101 backbone significantly increased RetinaNet processing time, for only a small benefit in precision. Design decisions such as these can play a more essential role than architecture choice, and should be carefully considered. Additional factors, such as dataset and batch size, are not investigated in this work but typically also have an impact on accuracy.

Faster-RCNN outperformed both more modern and slower networks on the night dataset. The reason for this is unknown, and is in contrast to the daytime performance. Fortuitous hyperparameter defaults may be a contributing factor, though properly exploring these for all 10 RGB configurations is not feasible. This result highlights the importance of testing a range of network types on application specific data, such as harvesting under controlled lighting.

Early data fusion was counterproductive for these datasets. Although late data fusion was effective, the gains from this method were less than that provided by data augmentation, and expanding the RGB dataset size would likely be significantly more useful than incorporating depth information. Additionally, longer ResNet backbones exist and typically show a small AP improvement over ResNet101. So the additional network capacity introduced by the late fusion approach may be better used as a longer RGB only backbone.

Predicting the full extent of partially obscured fruit is a requirement of the harvesting system and is well met by all of the tested networks. No accuracy metric is ideal for all use cases, and the 0.5 IOU threshold is an arbitrary assessment point commonly used in computer vision. Other metrics may be more suitable for tasks such as harvesting, where the IOU threshold required for a successful pick can often be estimated.

The overall AP values for both datasets may be sufficient for commercial fruit harvesting, but definitely show room for improvement. Lack of standardised datasets for testing has been an issue when trying to measure improvement in both robotic harvesting and fruit detection tasks. We hope that releasing this dataset and suite of trained models goes some way towards addressing this for plums.

In this work two datasets gathered during a robotic harvesting trial on 2D trellis plums are presented, and four deep learning object detection architectures are benchmarked on these. The fusion of depth information was trialled and found to be marginally effective for late fusion, though data augmentation provides a larger performance boost. On the day time dataset RetinaNet was the most accurate, while Faster-RCNN showed the best average precision for the night time data.

Relative network performance differed significantly to that published for the COCO dataset, which is commonly used when making design decisions for applications in agriculture. So the public availability of a wide variety of application specific datasets, such as tree fruit harvesting, is important to future progress.

During the next harvesting season the limited dataset size can be addressed by expanding the amount of plum picking data and testing on additional fruit types. Future investigations into accuracy metrics specific to tree crop harvesting, multi-view detection and multi-spectral imaging are also planned.