Cluttered Food Grasping with Adaptive Fingers
and Synthetic-Data Trained Object Detection

There exist several labelled datasets for foods, often under the field of food computing [5]. Existing food image segmentation datasets, such as Foood201-Segmented [6], UECFoodPix [7], and others [8, 9, 10, 11, 12, 13] provide instance-level labels but are not at the granularity required for our application (a plate full of pieces of chicken would be identified as one instance).

The gathering of datasets of the size required for deep learning models are unavoidably labor-intensive tasks. Prior works have explored the use of simulation or synthetic data for training semantic or instance segmentation models in numerous application such as crop seed detection [14] and even food [12]. However, they use simple shapes and focus on detecting food areas like the datasets above. Advances in photogrammetry and techniques such as differentiable rendering [15] allow for easy capture and generation of near photo-realistic models of objects from a handful of images, eliminating the need for specialist knowledge to create them. In our work, we use these techniques for quick and easy capture of novel foods that can be used in simulation to generate synthetic data to identify individual pieces of food.

Rapid R&D and adoption of soft grippers that reduce damage to objects or its surroundings is ongoing. The gripper fingers are soft and do not require precise control, but adjust with the food passively [16, 17, 18, 19]. However, the fingers tend to be thick, which makes it difficult to grasp food that is tightly packed in a tray. Grasping by suction can be done even in a cramped setting, but is challenging to use due to concerns of hygiene and contamination of the vacuum hose [20, 21, 22]. Prior work has also looked at grasping a single object with a needle gripper, but the needle is inserted into the food, raising two concerns- damage to the food and hygiene [23]. In our work, we develop a parallel gripper with an adaptive finger mechanism that passively retracts when it encounters the surface of the food to prevent damage.

Grasping in clutter makes the task of collision avoidance significantly harder. This is of particular importance to food since collisions could cause undue damage to foodstuff. There are three main approaches: 1) heuristic-based, 2) sampling-based, and 3) data-driven. 1) Heuristic-based approaches such as finding the highest grasp points [24] are simple but powerful if the appropriate one is selected for the task. 2) Sampling-based approaches evaluate grasp candidates for collision by using analytical methods [25, 26, 27, 28, 29] or by using task-and-motion planning libraries like OpenRAVE for collision checking [30, 31, 32, 33, 34, 35]. These approaches are computationally expensive to do at the resolution needed to grasp small pieces of food. 3) Data-driven approaches are popular in robotics with deep learning [36, 37]. These methods require a large amount of data or learn in simulation [31, 38, 39, 40, 41, 42, 43]. However, collection of a large manipulation dataset of food is not possible and realistic simulation of soft-bodied objects like food is still challenging to apply.

Our approach belongs to 1) heuristic-based. The method closest to our own is that of [44] where they represent a gripper model by two mask images representing a contact region and a collision region (which should be free of objects). However, they focus on rigid objects with regular shapes and are able to enforce collision regions to be completely empty. However, in dealing with soft, irregular foodstuffs, collisions to gently push food are unavoidable for a successful grasp.

In this section, we describe how we create a training dataset to train an instance segmentation model capable of detecting individual pieces of food in a cluttered tray. We first create a high-quality 3d model from several photographs of each piece of food using Preferred Networks, Inc.’s technology [45]. The details of the 3d model generation are outside the scope of this work. Fig. 2 shows the real foods and their corresponding 3d models. Highly uniform foods, such as sausage, can easily be represented by a single model, while we prepare the capture of multiple pieces for foods that show variance in shapes and sizes, such as fried chicken. We model a tray in simulation of the same size as the one shown in Fig. 1, and drop a number of these captured 3D models into the tray to generate training data for instance segmentation. We vary the number of pieces generated and initialize them at random locations over the tray. In doing this, we depend on the physics engine to prevent the models from conjoining and resolving collisions. Detailed parameters are described in Section IV-A

Although we can capture high-quality models of the foods and tune the simulation to be as realistic as possible, photorealism to the level needed to transfer to the real world is unattainable. To improve the model’s generalization, we adopt concepts of domain randomization from sim2real literature to randomly vary simulation parameters: the tray color, scale of the model, lighting parameters (direction, ambient light, shadows), and direction of the camera. Using domain randomization allows the model to traverse the reality gap and be used directly for real-world predictions without the need to additionally fine-tune with actual food data.

In this section, we explain the method used to find a grasp point and pose for grasping a piece of food while avoiding damage to it or its neighbours when the gripper is inserted. The method consists of three steps; 1) Instance segmentation, 2) grasp detection, and 3) grasp filtering (Fig. 3).

This section describes the mechanism of the proposed adaptive finger to grasp fragile foods in a cluttered tray. Foods may vary in shape and size, and their surfaces are often uneven. This surface irregularity is further exaggerated when a large volume of them is placed in a tray. Many foods are fragile and may be easily scratched or damaged during a grasp. Foods such as fried chicken with an outer coating are especially prone to having their surfaces broken. When grasping such foods with a typical gripper with solid tongs-inspired fingers (which we call fixed fingers in Fig. 1), a slight deviation in localization, sensor, or robot control may cause the grasp to fail or the food to be damaged.

In order to address this issue, we have developed a parallel gripper type with an adaptive function to conform to the shape of the food for the fingers (which we call adaptive fingers in Fig. 1 & Fig. 4 (a)). This adaptive function allows the gripper’s fingers to passively retract when they make contact with the surface of the food thereby avoiding damage caused by the insertion into the food surface. The fingers are constructed to retract a total of 22.5mm with the spring providing a certain restoring force to the finger. The contact force of fingertip at maximum shrink is 4.1N (Fig. 4 (b)) and chosen such that is unable to penetrate the surface of foods like fried chicken but still be inserted into grooves between pieces. As a result, even if the foods to be grasped are densely packed, the fingertips enter the gap while sliding over the surface of the foods to be grasped, so that it is possible to grasp fragile foods without damaging them (Fig. 4 (c)).

To create an instance segmentation model for grasping each kind of food, we first collect 3d models of the food (Fig. 2). We do this by capturing several images and combine them into a high-quality 3d model of the food. For some foods like broccoli and fried chicken with remarkable diversity amongst pieces, we capture more than one model in an attempt to improve generalization. To further decrease the reality gap, we create a simplified tray modelled after the industrial food tray shown in Fig. 1 and position the synthetic camera in a realistic way. We use the Python bindings of the Bullet physics simulator [47, 48] and employ ideas of domain randomization [4] to generate images of size $600{\times}600$ to use as training data for the segmentation model. Pieces of food are dropped from some height over the tray and allowed to fall in order to generate a number of unique, but reasonably realistic images. We vary a number of simulation parameters to improve generalization, including the number of pieces dropped (10-60), their sizes (0.7x-1.1x), the color of the tray, camera directions and lighting conditions (speculative coefficients, light directions, shadows).

Depending on the complexity of the 3d model, the time required for the physics engine used to generate plausible synthetic data may vary significantly as collision detection and handling becomes more expensive. The average time to generate a single image (using the same machines used to train models specified below) varies from 10 sec for simple models like sausage, meatball and taro, 14-18 sec for more complex models like fried chicken and mushroom, to 93-459 sec for complex models like gyoza and broccoli. By breaking generation up into small batches of 10 and deploying them in parallel, we effectively generate the 1200 samples required for each kind of food in around 1 hour.

We train the instance segmentation model using the 1200 samples generated (1000 to train and 200 as the validation set). Additionally, during the training of the instance segmentation model, we apply a number of data augmentations including randomly flipping the images (and their masks) and apply Median Blur to the input images to make the model less reliant on perfectly visible borders.

For the instance segmentation model, we use the highly popular and versatile Mask-RCNN [49] with a resnet50 backbone initialized with weights pretrained on the Microsoft COCO dataset [50]. Therefore, the instance segmentation model classifies each piece as either background or belonging to an instance of food. We trained the networks on a machine equipped with 384 GB RAM, an Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz, and NVIDIA V100 with 32GB of memory. Training for each model (initialized randomly for each food) took roughly 24 hours. Our experiments for inference with trained models were performed on a machine equipped with 31.3 GB RAM, an Intel Core i7-7700K CPU, and GeForce GTX 1080 Ti. Inference with the trained model during experiments takes about 0.3 sec.

Our robotic system, shown in Fig. 1, consists of a Sawyer 7-DOF robotic arm equipped with a self-designed two-fingered parallel gripper driven by a servomotor (Dynamixel XM430-W350-R). We use two custom designed end-effectors whose general shape was modelled after tongs commonly used to serve food. As seen in Fig. 1 we call the two tongs-inspired end effectors the “Fixed Finger” and a custom designed passive retraction mechanism that allows the fingers to retract when it meets resistance the “Adaptive Finger”. The shape of the fixed finger’s contact area is the same as the adaptive finger, and both are made of ABS. Furthermore, we use an Ensenso N35 stereo camera in combination with an IDS uEye RGB camera to overlook the workspace of the robot arm and use them to retrieve registered point clouds of the scene. We use two industry-standard food trays ($424mm{\times}308mm{\times}160mm$) and designate one to pick from and another to place in. The Sawyer, gripper and RGB-D sensor are connected to a PC running Ubuntu 16.04 with ROS Kinetic. We experiment with 7 foods; fried chicken, broccoli, meatball, sausage, mushroom, gyoza (steamed meat dumplings) and taro (colocasia) (Fig. 2).

In this section, we motivate the use of synthetic data over human annotations by describing difficulties in collection and their propensity for errors. Due to limitations in the image quality, a high degree of clutter and the visual similarity of each piece, it is sometimes difficult to identify the boundaries between the pieces. This would, understandably, lead to noise in the labeling process, which may confuse the model. To verify this hypothesis, we ask 5 unique individuals to label the same image using Amazon Mechanical Turk (AMT) and their provided region annotation web tool. By treating each person’s labels as ground truth and comparing them against the others, we can measure the agreement between them as the Precision (IoU between 0.5 to 0.95) of segmentation labels provided. Precision measures the percentage of masks output by someone that overlaps at least 50% with the ground truth. As seen in Table I, precision values between human annotators H1-H5 tend to be quite low ($<0.1$). We take this as an indication of noise, or disagreement between individual annotators. By visualizing the annotations of the bottom half of the tray in Fig. 5, we see a marked difference in how individuals handled areas where boundaries between the pieces are unclear. We posit that this disagreement may hinder the learning process further indicating a need for the perfect labels we can obtain with the use of synthetic data.

When manual annotations were sourced from AMT, we created a small dataset to be labeled. However, the collection of labels proved harder than anticipated due to many factors. A significant number of annotators submitted zero annotations, despite at least 10-20 pieces being visible at all times. Others labeled all the pieces with the same instance despite providing instructions and a sample image- possibly due to forgetting to switch the ID. A smaller fraction provided inconsistent labels where they started labeling them as multiple instances but then stopped and used the same one for a large number of pieces. Since hundreds to thousands of images are typically needed for deep learning (especially so to overcome label noise), having these images collected, labeled, verified, and cleaned for every new kind of food would require an enormous investment of time, money and labor. If the use of synthetic data is a feasible alternative, the cost required to train these models can be greatly reduced.

The purpose of this section is to show that training on synthetic data provides a practical alternative to training on real images, provides results agreeable with human annotators and is accurate enough to use for grasping.

A direct comparison between training with synthetic data and human-annotated images is difficult due to the reasons discussed in the previous section. To indirectly evaluate whether our model produces reasonable predictions that a human might make, we adopt the label agreement metric used in the previous section to the model’s predictions (taken as ground truth) and the 5 human annotators. From the precision values in bottom line of Table I, we see that the model outputs predictions that 3 out of 5 humans most agree with, and the other 2 with reasonable agreement.

We find that despite being trained purely with synthetic data, the model is able to generalize to predictions in the real world quite well as seen in Fig. 6. We find that it is also robust to changes in lighting conditions, scales and angles and is competent enough to depend on for choosing and executing grasps with high accuracy using the parametrization strategy described in Section III-B. We note that our end goal is not to train an ideal instance segmentation but one that can be used to grasp food well. Even if some pieces of food are not detected, as long as we can find a single reasonable target to grasp, it is acceptable. Therefore, by setting a high detection threshold and only retaining predictions that it is confident in, we should be able to achieve good grasp results even if the model is unable to transfer fully to the real-world data, and the results are a little worse.

In this section, we study the effects of using the adaptive finger. The success rates of 50 grasps with and without grasp filtering for the adaptive finger and fixed finger are shown in Table II. We define success as when each food item is grasped and put into another food tray. We also show the incidence where multiple pieces are grasped in a single grasp. Multiple grasps will be discussed in Section V-D.

The experimental results in Table II show that using the adaptive finger (with and without the grasp filtering) tends to either match or exceed the performance of the fixed finger with the exception of sausage (which we discuss later). Since the adaptive finger is able to adapt to the surface of the food, it is additionally able to compensate for inaccuracies in instance segmentation, sensing and actuation as visualized in Fig 4 (c). This improvement in success rate appears most significant in the case of fried chicken $(+16\%)$, broccoli $(+6\%)$ and mushroom $(+20\%)$, which we attribute to the variety in size and surface profiles as well as some compressibility. On the other hand, we see a drop with sausage $(-12\%)$, which we attribute to the hardness of the food. While a fixed finger is able to forcefully push neighbouring or blocking pieces away, the adaptive finger can retract prematurely and simply slip over the target piece due to its smooth, taut skin and may be more suited to be grasped with fixed fingers. However, we note that this forced grasping may come at the cost of increase in damage done to food via piercing by downward motion or scratching the surface of the food during gripper closure(See “by fixed finger” Fig. 7).

While it is difficult to come up with a generic subjective evaluation criterion of damage caused by the grasping process on each foodstuff, we instead choose to measure it specifically for a very delicate food- gyoza, which are meat-filled dumplings. We do this by counting the number of pieces in both trays with fully pierced skins and whose stuffing is visible after the 50 grasp attempts made in Table II for each experiment. We note these numbers for each method evaluated in Table III. We see the stark contrast in the number of damaged pieces when the adaptive finger is used instead of the fixed one. We also note that not just the number but also the amount of damage inflicted on the gyoza is also more significant when a fixed finger is used, with some pieces being practically ripped apart, as seen in Fig. 7. We see that even with the use of adaptive finger, some pieces do get damaged due to a mixture of gripper closure and the spring potentially being too strong for something as delicate as gyoza and could be adjusted to further reduce it. The use of the adaptive finger presents a trade-off between performance and damage for some more rigid foodstuff. We believe it is a good default choice based on the foods tested.

In this section, we aim to study the effect of using the grasp filtering. The success rates of 50 grasps with and without grasp filtering for the adaptive finger and fixed finger are shown in Table II as explained in section V-C.

From the results in Table II, we see that the use of grasp filtering tends to increase the success rate for most foods for both the adaptive fingers and the fixed fingers. The use of grasp filtering shows improved performance most prominently for fried chicken $(+14\%)$ and meatball $(+8\%)$. The fixed fingers also shows improvement to a smaller extent for fried chicken $(+8\%)$ and mushroom $(+16\%)$. Since the grasp filtering criterion picks grasps where the fingers are less likely to make contact with neighbouring pieces, we see that it is also able to reduce the incidence of grasping multiple pieces of food with a single grasp. This can be seen from looking at the differences in success rates when multiple pieces are included, which are reported in brackets in Table II. This reduction in the percentage of multiple grasps is observed in grasping of foods which are small such as mushroom ($18\%\rightarrow 12\%$) or meatball ($6\%\rightarrow 2\%$) or easily entangled like broccoli ($6\%\rightarrow 2\%$).

The grasp filtering criterion not only improves grasp performance, but in filtering out candidates where collision between the fingers and neighbouring pieces is likely, the amount of damage caused is also reduced during the grasp process. This is demonstrated in the gyoza experiments in Table III. We see that using grasp filtering decreases the number of pieces damaged for both the adaptive finger $(9\rightarrow 6)$ and the fixed finger $(18\rightarrow 14)$.