Central object segmentation by deep learning for fruits and other roundish objects

Techniques of computer vision have various applications in fruit production, for example making yield estimation by processing images of farms before the harvest. Such pre-harvest estimate is crucial for efficient resource allocations and these computer vision techniques certainly would constitute a part of smart agriculture [GAK⁺15].

To this end, initially, algorithms of human-made feature extraction engineering have been developed to locate fruits in camera images [DM04, NAB⁺11, MSD⁺16, PWSJ13, DLY17], and interestingly some used controlled illumination at night [NWA⁺14]. Also, non-destructive size and volume measurement of fruits is useful in production and distribution of fruits [MOCGRRA09]. This is in fact one of our motivations in this paper. Naturally computer vision techniques have been developed, for example using color analysis to conduct image segmentation [BSSR18]. Some used proper backgrounds for image segmentation [OKT10, MBMRMV⁺12], and even to develop a smart phone application [WKW⁺18].

Inevitably, machine learning techniques came in to play, for example for harvest estimation of tomatoes; image segmentation based on pixels and blobs and then X-means clustering were applied [YGYN14]. Even multi-spectral data was processed with conditional random field for image segmentation for various crops [HNT⁺13]. In addition, for robotic harvesting, examples include the uses of Viola-Jones object detection framework [PVTG16], and image segmentation with K-means clustering [WTZ⁺17]. For accurate size measurement, support vector machine performed image segmentation for apples with black backgrounds [ML13] for example.

Deep learning is one of machine learning techniques. There are more and more applications of deep learning not only in fruit production but also in agriculture as a whole [KPB18].

Human-made algorithms require humans to set features to be extracted from raw data. In this case, the number of such features must be limited if one wants to establish algorithms within a reasonable time frame. However, DNNs (Deep Neural Networks) extract important features automatically, which is one of key properties of deep learning. Of course, DNNs process data in a black box, but such data-driven methods could handle complex data, for example camera images in farms, which are affected heavily by time of day, weather conditions, seasons, and other illumination conditions such as shade and light reflection. Therefore, huge amount of features would be necessary in processing such images, and it would be simply too much for humans to find and list all important features in order to write algorithms by hand.

By contrast, things are completely different in processing data with DNNs, which consist of many layers. Roughly writing, as data flows in a DNN, each layer makes the data little more abstract, so that the DNN yields abstract understanding deep within itself. This way, humans do not have to pick important features by themselves. Interested readers can consult [GBC16] for more detailed theories of deep learning.

Recently one can find more and more applications of deep learning, for example counting apples [BU17b], apples and oranges [CSD⁺17], and tomatoes [RS17]. Also, in [WLS⁺20] deep learning was applied to image segmentation in order to obtain horizontal diameters of apples, by processing images taken on cloudy days or at dusk and pre-processed with Laplace transform. Moreover in [MPU17], light-weighted neural networks were developed for image segmentation for agricultural robotics.

Faster R-CNN [RHGS15] is one of famous DNNs for object detection, and yielded successful applications for example in detecting apples [BU17a], mangoes with spacial registration[LCL⁺19], and sweet papers by using multi-modal data (color and Near-infrared) [SGD⁺16]. Despite its popularity, Faster R-CNN has disadvantages; it gives rectangular bounding boxes for fruits in images but they do not give accurate size measurement. By contrast, human-made algorithms for object detection have aspects of image (semantic) segmentation, because they usually start with processing pixels to extract information such as colors and textures before finally detecting fruits. This way, object detection and image segmentation are often interwoven with each other in conventional techniques. In [SdSdSA20], however, instance segmentation via another DNN called Mask-RCNN [HHG⁺19] was used to overcome this disadvantage, together with space registration. One can find detailed explanations on application of Faster R-CNN and Mask-RCNN (and YOLOv3 [RF19]) to images of grapes in [SdSdSA20].

Our research put more weight on accurate measurement. Our neural network, which we call CROP (Central Roundish Object Painter), identifies and paints the fruit at the center of an image. It works for roundish fruits in various illumination conditions. This technique could provide us with a means of automatically collecting statistical data of fruit growth in farms, and perhaps might allow automated robots to make better decisions. We also developed programs to run CROP to automatically process time series photos by a fixed camera to collect the data on the size and the position of the target fruit; see Section 3.3. Our trained neural network CROP and the above automatic programs are available on GitHub (https://github.com/MotohisaFukuda/CROP), with user-friendly interface programs.

In this research project, we used several groups of images from the internet and farms in Kaminoyama, Yamagata, Japan, which are listed below with sample images.

Data_Fruits:

172 images of a variety of fruits downloaded from (https://pixabay.com).

Data_Pears1:

26 images of pears in the farm in 2018 with Brinno BCC100 (time-lapse mode).

Data_Pears2:

86 images of pears in the farm in 2019 with various cameras.

Brinno BCC100 is a time-lapse camera used by one of the authors to keep track on growth of pears, which give blurred images to be processed.

These images were all annotated by labelme [Wad16] to train and evaluate CROP. In Section 4.3 quantitative analysis was conducted; we trained CROP with Data_Fruits being training data (80%) and validation data (20%), and afterwards fine-tuned it with Data_Pears2 being training data (80%) and validation data (20%). On the other hand, CROP in Section 2.2 was trained by all the images of Data_Fruits to make qualitative analysis.

In this section, we show how CROP works by using sample images. We used all of Data_Fruits for training because we have a limited number of annotated images; we decided when to stop the training by processing random images on the internet. The sample images in this section do not belong to Data_Fruits, so that they can be thought of as test data, i.e. they are new to CROP. Predictions by CROP are represented by mask images pasted onto the original (cropped) images; red pixels for the central object and yellow for the rest.

In this section and the next, some of the original photos were provided by Hideki Murayama, or came from some institutions. In the latter case, they are credited in the captions, and the explanations of the acronyms of the institutions and the disclaimers are found in the section titled “About photos in this paper”.

First, the name of CROP does not come from the network architecture but from its function of identifying and painting the object at the center of an image. Examples in Figure 2(a) and Figure 2(b) indicate how CROP identifies individual grapes at the center of images. By contrast, CROP got confused in Figure 2(c) because the central grape was behind others. On the other hand, it even could handle wide-angle images like in Figure 2(d).

Let us go over some examples. Firstly, in Figure 3, one can see that CROP handled images of fruits of various shapes and colors.

Secondly, we classified common mistakes by CROP, and included representative examples in Figure 4. Here are our guesses why: in Figure 4(a), the object is not round enough, in Figure 4(b) the angle of two meeting boundaries is not acute enough, in Figure 4(c) there is a disruptive object, and in Figure 4(d) “ there is no boundary”. Thirdly, however, CROP can ignore those unimportant parts such as peduncle and calyx, like in Figure 5.

Although CROP has been trained solely by 172 fruit images (no transfer learning or fine-tuning), it started to understand the meanings of boundaries of central roundish objects; some other foods than fruits in Figure 6, various materials some of which were through microscopes in Figure 7 , and photos in the space in Figure 8.

“La France” is one of the most popular cultivars of European pear (Pyrus communis) in Japan, its yield is about 70% of European pears in Japan. La France pomes are usually harvested at the mature-green stage and then chilled for stimulation of ethylene biosynthesis prior to being ripened at the room temperature. If the harvest is delayed or too early, the fruit does not ripe properly, and the texture, taste and flavor will be poor. In commercial pear cultivation, harvest time of La France greatly influences both the amount and quality of the harvest. Therefore, it is important to measure the maturity of the fruit precisely to optimize the time to harvest, but criteria to estimate the fruit maturity are limited such as fruit firmness and blooming date. The fruit growth in terms of fruit size is described by an asymmetric sigmoid curve. The growth rate of the fruit on the tree is significantly affected by environmental factors and the physiologically active state. Precise time-lapse measurement of fruit growth should be useful for estimation of the fruit maturation status. In order to measure the fruit size change as it grows, the size of the same fruit must be measured repeatedly (daily) with a caliper.

Research on sizes of agricultural products, without computer vision, was previously conducted in predicting optimum harvest time of carrots [HS00], estimating yield of pears [Mit86] or anticipating cracking of bell peppers [NKG03]. Also devices for continuously measuring fruit size change have been developed, for example the kiwifruit volume meter (KVM) [GMA90], stainless frames with potentiometers to be put on fruits [MMZ⁺07], and flexible tapes around fruits to be read by infrared reflex sensors [Tha16]. However, one of the authors plans to estimate fruit size based on images, possibly by using CROP. We hope that our research will improve the accuracy of fruit size measurement and reduce the labor required in the data collection.

Since CROP was initially trained by clear images of various fruits of Data_Fruits, we used a technique called fine-tuning to re-train CROP by rather unclear images of pears of Data_Pears2, so that it can process similar images of Data_Pears1. One can consult Section 2.1 for these datasets, and Section 4.3 for quantitative analysis, where the effect of fine-tuning is investigated. In the rest of the section, however, let us make qualitative analysis of fine-tuning through examples.

In Figure 9, each triple consists of the original image, and images processed before and after the fine-tuning, placed from left to right. Some images were processed well even before the fine-tuning, like in Figure 9(a). There are some small improvements in Figures 9(b), 9(c), 9(d). Some showed dramatical improvement, like in Figure 9(e), and some on the contrary, like in Figure 9(f).

In this section, we show our way of using CROP in the local pear farms. All the implementations presented here are available on GitHub (https://github.com/MotohisaFukuda/CROP). We will focus on our method of collecting data and have no intention to draw any scientific conclusions, although we make some remarks based on the data. Indeed, eight photos were taken a day, at 8:00, 9:49, 11:49, 13:49, 15:49, 17:49, 19:49, 21:49, but the graphs and plots would look different if photos had been taken isochronously, for example once every three hours. In the following example, we processed 510 time series photos by a fixed camera to get the data on the size and the position of the chosen pear. All the process is automatic once we choose a target fruit in the first photo, and it took less than 14 minutes in this example. Note that these photos were taken in 2020 and are new to the CROP used in this section; see Section 2.1.

Image segmentation can be seen as classification of image pixels. For example, if you want to cut out a fruit in an image, all you have to do is classify each pixel into one of two classes: fruit and background. The number of classes can be more than one, and in this case we want to classify each pixel into one of these classes. This kind of image process is called (image) semantic segmentation. In this sense, image semantic segmentation is much more difficult than object detection, where one classifies each image, and not each pixel. In the rest of this section we explain about our neural network and compare it with some other neural networks.

Strictly speaking, CROP is the name for our trained neural network for the specific purpose, however, we call our neural network CROP even before the training to avoid confusion. As in Figure 18(a), inputs of CROP are RGB $512\times 512$-pixel images and outputs $512\times 512$-pixel mask images. The downward red arrows represent convolutions with kernel $2\times 2$ and stride $2$, which double the number of channels. Here, channel corresponds to another dimension than height and width, where the number of channels of RGB images is 3, and mask images 1. Similarly, the upward green arrows represent convolutions with kernel $2\times 2$ and stride $2$, which however make the number of channels half. The red rectangles are concatenations of two convolutions with kernel $3\times 3$ and stride $1$, which keep the number of channels unchanged. The green rectangles are again concatenations of convolutions, where the second convolutions are the same as the ones from the red rectangles, but the first ones are a bit different. Their inputs are direct sums (in the channel space) of the outputs of the layers below and the copies from the left, the latter of which are indicated by horizontal arrows. With these inputs then the first convolutions in the green rectangles make the number of channels half.

To make our explanation complete, we explain the first and last boxes. Pink and light green boxes represent again concatenations of convolutions with kernel $3\times 3$ and stride $1$. Through these layers the number of channels change as $3\rightarrow 16\rightarrow 16$ and $16+16\rightarrow 16\rightarrow 16\rightarrow 1$, respectively. Note that ReLU and batch normalization are applied adequately, which are not explicit in the figure.

The architecture of CROP is based on U-Net [RFB15]. U-Net was developed for medical image segmentation, and the architecture is depicted in Figure 18(b), which was taken from the website (https://lmb.informatik.uni-freiburg.de). After the emergence of U-Net, many related research projects were conducted, including V-Net for 3-dimensional medical images [MNA16], from which we took the loss function for our training.

U-Net (and V-net as well) belongs to the family of Fully Convolutional Networks (FCN), which is a subset of Convolutional Neural Networks (CNN). A CNN treats 2-dimensional data as it is, so that it gives better performance in image processing, and mathematically it is literally comparable to convolution operation. The idea of CNN is rather old [LHBB99] (some argue it comes from [Fuk80]). However, it proved useful in deep learning when AlexNet [KSH12] won ImageNet Large Scale Visual Recognition Challenge in 2012, which is an image classification contest. The key idea behind was that they combined the two ideas of CNN and Deep Neural Network (DNN). This new architecture, realized by computational power of GPUs, enabled the neural network to improve the error rate dramatically.

Naturally this architecture of (deep) FCN was then applied to the task of semantic segmentation in [LSD15], and with encoder-decoder structure in [NHH15] and [BKC17]. The encoder-decoder structure consists of two parts; encoding with convolutions or max pools and decoding with transposed convolutions or up samples. One can see this structure in the u-shaped part of U-Net. In Figure 18(b), an input RGB image of $572\times 572$ pixels becomes smaller by iterative application of max pools down to as small as $28\times 28$, although the number of channels becomes $1024$, i.e. $3\times 527\times 527$ pixels are transformed into $1024\times 28\times 28$ at the bottom, where U-Net processes the data “abstractly”. This down-sizing process corresponds to the $encoder$. By contrast, through the decoder with up convolutions, U-Net yields an image of $388\times 388$ with channel size one, i.e. a mask image. Importantly, there are four skip connections, which are represented by horizontal gray arrows in Figure 18(b). They are supposed to transmit location information from the encoder to the decoder, and this architecture characterizes U-Net.

Now, CROP has a similar structure as in Figure 18(a). The biggest difference is that the decoder and encoder are much deeper than those of U-Net. CROP makes the size of an input image $2^{7}$ times smaller at the bottom, while U-Net nearly $2^{4}$ times smaller. Those numbers $7$ and $4$ correspond to the number of red down arrows in Figure 18(a) and Figure 18(b). We believe that this difference in depth enables CROP to have more global and abstract understanding of images; see Section 4.5 for quantitative experiments on this matter. Another difference is that we adopted convolutions with kernel $2\times 2$ and stride $2$ instead of max pools with kernel $2$ in the decoder, because convolutions learn but max pools do not. Note that “up convolution” in Figure 18(b) is same as “transposed convolution” in Figure 18(a).

Before concluding this section, we need to write about two more famous neural networks. The first one is an implementation of instance segmentation and is called Mask R-CNN [HHG⁺19]. Instance segmentation can distinguish neighboring objects of the same class, while semantic segmentation would mix them up because it would just classify the pixels of these objects to one class. Mask R-CNN was applied in [SdSdSA20], to detect grapes bunch-wise. However, since we choose a fruit and fix a camera in the first place (Section 3.1), we do not have to detect fruits and rather can focus on image segmentation. The second one, for semantic segmentation, is called DeepLabv3+ [CZP⁺18]. This neural network has atrous convolutions to capture contextual information, and works very well for general purposes. However, we do not need contextual information and would rather go for preciser segmentation ability of U-Net, which has been yielding many applications in medical image segmentation, where accuracy matters.

Loss functions tell neural networks how to improve themselves. In this sense, choice of loss function is very important in training neural networks. In this project, we picked soft dice loss [MNA16]:

where $k$ runs over all the pixels, $512\times 512$ pixels in our case. Here, $\{x_{k}\}_{k}$ are outputs of CROP and $\{t_{p}\}_{p}$ are the target (“the right answers”). As usual $\{x_{k}\}_{k}$ take values between $0$ and $1$ after going through the sigmoid function: $1/(1+e^{-x})$, and $\{t_{k}\}_{k}$ exactly $0$ or $1$. As for the latter, $0$ and $1$ correspond to pixels of the background and the object, respectively, and they are set based on the ground truth, i.e. annotated data. Note that the loss vanishes if $x_{k}=t_{k}$ for all $k$’s.

Among loss functions, pixel-wise cross entropy:

could seem to be a good choice if we consider image segmentation as classification of each pixel. In fact, the cross entropy loss is commonly used for image classification tasks. However, we did not use it because otherwise each pixel would carry the same share in (2). That is, in every batch, each mis-classified pixel makes the same amount of contribution to back-propagations (training process) regardless of the size of objects, and as a consequence, the neural network could learn to rather ignore small objects. By contrast, with soft dice loss as in (1), such imbalance would be compensated by regularization, which appears as the denominator. For similar reasons, we did not adopt $l_{p}$ loss ($p\geq 1$):

Note that the above soft dice loss is a variant of dice loss, This category of loss functions treat small and large objects relatively equally, and moreover if there are more than one classes they also modify imbalance among different classes. In [RFB15], they applied more penalty to boundaries to segment images of cells, but we did not follow their path to avoid complication.

Finally, IoU (Intersection over Union), or Jaccard index:

measures how two sets are close to each other. It takes the value between 0 and 1, corresponding to $A\cap B=\emptyset$ and $A=B$, respectively. This is not a loss function but an evaluation criteria, but shares the same spirit.

In this subsection, we give data-scientific evaluation of the training process of CROP, by using soft dice loss and IoU. All the evaluations were made with data augmentation unless otherwise stated.

To make the predictions stable, we feed CROP eight different images made by applying actions of the dihedral group $D_{4}$ to the original input image. They are eight different combinations of flips and rotations which keep a square unchanged in the two-dimensional space. The final decision was made by averaging the corresponding eight predictions by CROP. This is how we applied CROP to the examples in this paper, and it is implemented on GitHub, but the functionality can be switched off for quicker and possibly less accurate predictions.

In this section, we report on experiments on the shallow version of CROP, which is comparable to the original U-Net. Provisionally we call it CROP-Shallow (see Figure 21). To this end, we trained CROP-Shallow and CROP in the same condition: loss function, optimization method, learning rate, but random were partitions of training and validation sets, initialization of these neural networks, choice of batches and application of data augmentation. Also, the batch size was fixed to 6 for CROP-Shallow, which consumes a lot of GPU memory probably because of the way how the channel size (or feature map size) changes. However, the number of parameters itself is much smaller than that of CROP; 40,103,873 for CROP-Shallow and 160,829,681 for CROP.

Despite these potential problems, however, we still believe that we captured significant difference between the two architectures. See Figure 22, which shows over-fitting. The difference can be read also from Table 2 of IoUs for the validation datasets.