Object Occlusion of Adding New Categories in Objection Detection

For real objects occlusion relationship, small objects can occluded small partial of large objects, while large objects can occluded large partial of small objects. In real scenarios, objects occlusion relationship has relative fixed distribution, which give us a chance to imitate distribution important sample points. If we can place objects to make occlusion which imitates important sample pointa as real scenarios’ objects occlusion distribution. And we find that according to a target object A of a category X, if in one point of occlusion distribution, A’s bottom 50% is occluded by a object B of a category Y, but now we do not have any object of category Y at hand, then we can use a object C of a category Z to occlude A’s bottom 50%, which gives a same effect as use a object B of a category Y. Therefore, the category of frontal objects used to occlude target object is not important, but whether it imitates a occlusion relationship fitting the real occlusion relationship is important.

We can see general placement of goods of one layer in shelf in Fig. 1 and Fig. 2. Because we use fisheye to take pictures in our scenarios, we should follow fisheye characteristics, which is an ultra wide-angle lens that produces strong visual distortion intending to create a hemispherical image. Therefore, for the purpose of make all goods have distinctive regions that can be visible in camera view, the tall beverages should be placed near shelf wall, and dwarf goods will be placed in the center area of one layer of our shelf. As the size of goods increase, goods will be placed more peripheral. We should guarantee all goods can be seen properly by fisheye camera, so they all can potentially be detected by neural network model.

Preparing more different sizes of objects can be helpful for imitate more occlusion relationships. For instance, one real occlusion relationship is that a target object is occluded a small part by small objects. And another real occlusion relationship is that a target object is occluded a large part by large objects.

We can generate occlusion in data collection stage or data augmentation stage. we should be aware of importance that the object size of each category in constructing occlusion in data collection stage, because different sizes of categories can generate different occlusion relationship by nature. But we do not care object size in constructing occlusion in data augmentation stage, we can use any category to occlude a target object, using techniques including copy-paste, cut-paste, image scale and image translation.

Correctly finding the real object occlusion distribution is the precondition of imitation. In particular scenarios, object occlusion is determined by many factors, such as camera viewpoint, object size, object location, etc. Objects relative positions should be reasonable. For example, in indoor scenarios, a cup on the table along with a TV controller. Paper picker may partial occlude cup, even total occlude cup in some viewpoint, but the TV controller is rarely placed on top of the cup, almost all cases are alongside with the cup. So we give major cases high priority in data collection. Each occlusion relationship only collecting one or two cases is enough for hit high accuracy in real test cases.

Monte-Carlo method can be used to pick data sample points from occlusion distribution. Firstly, we should find data occlusion distribution of our new category in real scenarios. we deal each new category one by one. Then we generate sythetic images by copy-paste bounding boxes to occlude objects of new category following occlusion distribution of our new category. And we associate these sythetic images with a few real images to train our detection network.

In data-collection stage, we do not have previous data occlusion distribution of a new category. But we possibly have at least one similar size category before. For occlusion generation, we do not care the surface material or texture detail of our new category. the location this new category should be placed may be affected by new category’s volume and intrinsic characteristics. In dataset FVSS, small box package snacks or canned drinks usually are placed in center of a layer of the shelf, while big bag snacks are placed in the periphery of the layer. For most effecient utilization of one layer space, we fill a layer with goods as possible as we can, which under the conditions that each good placed in this layer can be viewed by top-centered fisheye camera clearly. At least each good visible region is necessary and distinct that human can recognize the good belong to which category from the fisheye camera captured images. Therefore, in a fisheye view, we should guarantee the top part or top lateral part of every good is visible and no good stacks. In a replete layer, below part of each good usually is occluded by nearby goods. We follow a rule that small size objects are placed at center of a layer, while large size objects are placed at periphery of the layer. Therefore, if we add a new category which has big size, so it is usually placed in periphery of one layer, or near the walls of the shelf. This kind of placement make objects of this category less possible to be occluded by nearby objects of same category or other categories. Below part of objects are usually occluded with different occlusion ratios by different categories. For example, we add a bottle water. If goods of same category occlude this good, maybe only top bottle cap of this good is visible. If a smaller milk box is next to this bottle, maybe below two third part of bottle is occluded. If a lie-down snack bag is next to this bottle, maybe below one third part of bottle is occluded. And we should not add a bigger object next to the bottle in the side near fisheye camera center, because we do not want our bottle to be totally occluded.

In data-occlusion stage, we use already annotated images to generate new occluded images following the data occlusion distribution we found. Firstly, we should find correct data occlusion distribution of our target new category. The most reasonable method is that we analyze the new category attributes and infer the occlusion distribution by an experienced researcher. But this method is hard to generalization, because we always need some experienced experts. So an automatic method is needed.

In dataset COCO, the category named ’person’ usually stands in street or sit near a table, so one person may be occluded by other people at the outdoor, or the table at the indoor. Interestingly, one person that is annotated almost show his head in most cases even in crowd scene or remote scene. If no head appear of one person, the dataset organizer may be not collect that kind of images, because our human being usually identify another human being by its head. And annotators may be feel strange to annotate a person only show below half part of body while no head can be seen.

Some features: (1) Notice that a person head maybe occluded by umbrella. (2) Head maybe show in a lateral view or show the back side of head. These view should imitate in data collection stage, while data augmentation stage only imitate the real occlusion relationship rather than different viewpoints. (3) In rare cases, in a image, a region inlcudes only visible a small part of human that is still annotated as a person, such as only close-up hands or a foot can be seen.

These features can generalize to all kinds of animals, which possibly show their heads in pictures as photographers can make the objects in their picture easy to be understood.

COCO is a general dataset which contains large diversity of each category. In COCO, each category may appear in abundant environments and light conditions, with diversity of gestures and viewpoints. For example, a animal category named ’bear’ has many sub-categories, such as polar bear, black bear, brown bear and raccoon. We can add a new category with little number of images like dozens of images, then train a detection network well, which includes all categories besides our new category.

For stuffs have small size, like toothbrush, remote controller, may have near view or remote view, so the object sizes in picture vary in large range. There is a question: is it useful that use small size object occlusion distribution in large size objects? we find it is useful, and can be even better with image scale changing as data augmentation.

We also analyze Open Images Dataset [9]. Each category has more samples as well as more diversity. It show 4 types of annotation: Detection, Segmentation, Relationships and Localized Narratives. Detection is annotated with bounding boxes. Segmentation is annotated with polygons. Relationships show many kind of relationships between human and objects or between different objects. Relationships use dotted-line bounding boxes that show one object is in another object. These relationships also strong relate to data occlusion distribution of categories and give us inspiration. We show one Relationships image in Fig. 3, which describes a contain relationship between a piatter and a tomato.

It is hard to imitate all viewpoints, especially in large outdoor scenarios. But we find a simple way which only use several camera viewpoints and reach high accuracy in real scenarios. We refer the method proposed in NERFIES [14], using main camera viewpoint and several nearly main camera viewpoints to take objects pictures, ignoring those rare seen viewpoints.

After collecting tens of images for our new skus, we choose copy-paste [12] data augmentation to cover more sample points representing the data placement and data occlusion distribution, which can improve detection performance. Copy-paste strategy we used is that random copy-paste many image bounding box regions to another image following our data occlusion distribution, and following some boundary conditions like not overlap too much over exist objects.

We mainly use bounding boxes as annotation for each objects. In situation that data collection is controllable, we can move objects slow in each frames. Thus we can use tracking models to annotate each object in continuous slow movement if the first frame is annotated, reducing human annotators workload.

We try single object tracking at first. If we move one object slowly each time, we can annotate a target object in first frame of a clip which only capture the movement of a target object, and the other objects keep static. We assume camera also stay in same position to simplify the issue, but we also can move camera slowly to achieve excellent single object tracking performance. But there are two main issues using SOT.

(1) If there are several objects of smae category placed near each other, when we slowly move one of them, the tracker may be jump to capture an another static but near object. It requires a more accuracy SOT model, which is necessary for scalable use of SOT for helping annotation.

(2) Because we only support single object tracking, we need annotate many clips, each of which moves a new object.

Multi-objects tracking stratey can tracking many objects, so we can move different objects simultaneously. So we combine FairMOT to boost our annatation process.

We do experiments in shelf environment and we use fisheye cameras to take pictures, which follow our FVSS dataset construction style. We use 10 thousand images containing 457 categories as our base training dataset, and we add one new category with only 10 images into base training dataset, then we test the performance in an validation dataset with 1000 images, in which every image contains the new category at least one bounding box. we show two categories heatmap in our dataset in Fig. 6, which is relavant to the occlusion distribution of these categories.

For example, we use ”coke can” as our new category. Adding 10 images of ”coke can” to 10000 images training dataset , which contain 457 categories but do nothing contains category ”coke can”. These 10 images of ”coke can” is taken pictures in important sample points in data occlusion distribution of ”coke can” in shelf environment. Therefore, we have total 58 bounding boxes of ”coke can”. We also construction validation dataset with 1000 images, which contains 179 categories in total. And each image in validation dataset contains at least one ”coke can” bounding box. There are total 3939 ”coke can” bounding boxes. We use three metrics, first is the [email protected] and [email protected]:0.95 of ”coke can” in validation dataset. Second is pass rate of ”coke can”, which means if ”coke can” bounding boxes are all found in correct positions, this image is pass, otherwise is not pass. Third is if a ”coke can” bounding box is not detectd as ”coke can”, it is possible that the bounding box is detected as one other category or just ignore by neural network model. Therefore, we design a rate to describe whether this bounding box is detected as a other category with a confidence lower than 95%. Results show in Table I.

We test 16 categories as one new category each time. Our categories all are retail field goods such as snacks, milks and beverages, etc. And we conduct a conclusion that we can use few images for training a new category and keep accuracy above 80%, and mis-detected rate above 85% in average, which mean only 3% images in validation dataset is mis-detected with high confidence or ignore by detect model. We show some categories results in Table II. we use mis-detected rate that confidence below 90% is counted. And we add a new accuracy called final undistinguishable rate, which particular indicates bounding boxes are mis-detected as another category and have high confidence above 90% or cannot be detected by model. Results show in Table II and Table III.

We also find a way to better imitate important data occlusion distribution using many different size categories to generate diverse data occlusion relationship. And this method can get near double accuracy in average.

We try add images of a new category from another domain, like using phone take pictures, and add to training dataset including shelf fisheye images. We do not apply any domain adaptation methods for enhancement. We get 0% pass rate of the new category in 1000 images validation dataset. and mis-detection of the new category is almost same as adding no new category data. Therefore, domain adaptation still a issue in training a new category. Results show in Table III.

We test whether add a new category or not in training dataset. Validation dataset has new category data, evaluating in validation dataset. we discover if no new category data in training dataset, the new category in validation dataset cannot be detected correctly even one image, so pass rate is always 0.0%. And there is 21% bounding box will miss or be error detected as others with high confidence. Meanwhile, average pass rate is 72.0% of the new category if we add only 10 new category images to training dataset, and mis-detect [email protected] will increase to 90.0%. And we found that some high aspect ratio categories can get higher mis-detect [email protected] increasement. Maybe if we add images of this new high aspect ratio category, there are less chance that the objects of this category could be error detected as other categories. But a high aspect ratio category often gets a lower pass rate than other categories using few images. Therefore, we think adding new category images is important even few images is added. And if these few images can fit well in data occlusion distribution of the new category, most necessary and important information will be learned by neural network. Results show in Table IV.

We do a further experiment of few images of adding a new category in large dataset. We use a large shelf fisheye view dataset with more than 360 thousand images, and add a new category called ”sizhoushaokaoweixiatiao” with only 15 images containing 60 bounding boxes to this large dataset. we train 1.5 epochs, with common data augmentation methods like image flipping, hue tuning and normalization, etc. We evaluate on test dataset including 500 real images. Only about 10 images is incorrect. This result show the huge potential of use copy-paste with data occlusion distribution in detection using bounding boxes annotation format. Results show in Fig. 7 and Fig. 8.

We do experiments that compare normal data training with empirical enough images of a new category which have more than 3000 bounding boxes to only 20 new category training images. These 20 category training images is a subset of the before large dataset. And we conduct two kind of experiments of 20 new category images. one we use only these 20 images with some data augmentation such as image flipping, hsv transformation, hue tuning. Another experiment we use copy-paste data augmentation following data occlusion distribution to create more 100 images from these 20 images, Then we get 120 new category images in total, 100 of them are copy-pasted. We test 5 new categories one by one and show results in Table V. The results is amazing, which show that little amount of images with copy-paste generated images is better than original large images training results. Fig. 9 shows a image augmented with copy-paste by our strategy. Fig. 10 shows a failure case of detection by a network trained by our strategy.

There is a situation that we use already collected dataset to training, and we cannot control the data collection stage, but we also want to add few images of a new category to exist dataset. We design experiments for demonstrating that few images of a new category can give a relative high level accuracy of this new category if we pick these images from important sample points of data occlusion distribution of this new category in test dataset. Our implementation refers to [12]. We think [12] use simple copy-paste data augmentation strategy can get noticeable improvement of accuracy. This conclusion is based on these copy-paste operations create many new occlusion relationship which contains the important sample points of data occlusion distribution. Fig. 11 and Fig. 12 give us two categories test results, which show the condence distribution of target categories in test dataset.