Saliency-Aware Class-Agnostic Food Image Segmentation

It is well-known that dietary habits have profound impacts on the quality of one’s health and well-being (Nordström et al., 2013; Mesas et al., 2012). While a nutritionally sound diet is essential to good health (Organization, 2009), it has been established through various studies that poor dietary habits can lead to many diseases and health complications. For example, studies from the World Health Organization (WHO) (Organization, 2009) have shown that poor diet is a key modifiable risk factor for the development of various non-communicable diseases such as heart disease, diabetes and cancers, which are the leading causes of death globally (Organization, 2009). In addition, studies have shown that poor dietary habits such as frequent consumption of fast food (Ginny et al., 2012), diets containing large portion size of energy-dense foods (Piernas and Popkin, 2011), absence of home food (Hammons et al., 2011) and skipping breakfast (PR1 et al., 2010) all contribute to the increasing risk of overweight and obesity. Because of the many popular diseases affecting humans are related to dietary habits, there is a need to study the relationship between our dietary habits and their effect on our health.

Understanding the complex relationship between dietary habits and human health is extremely important as it can help us mount intervention programs to prevent these diet related diseases (Daugherty et al., 2012). To better understand the relationship between our dietary habits and human health, nutrition practitioners and researchers often conduct dietary studies in which participants are asked to subjectively assess their dietary intake. In these studies, participants are asked to report foods and drinks they consumed on a daily basis over a period of time. Traditionally, self-reporting methods such as 24-hr recall, dietary records and food frequency questionnaire (FFQ) are popular for conducting dietary assessment studies (Shim et al., 2014). However, these methods have several drawbacks. For example, both the 24-hr recall and FFQ rely on the participants’ ability to recall foods they have consumed in the past. In addition, they are also very time-consuming. For dietary records, participants are asked to record details of the meals they consumed. Although this approach is less reliant on the participants’ memory, it requires motivated and trained participants to accurately report their diet (Shim et al., 2014). Another issue that affects the accuracy of these methods is that of under-reporting due to incorrect estimation of food portion sizes. Under-reporting has also been associated with factors such as obesity, gender, social desirability, restrained eating and hunger, education, literacy, perceived health status, age, and race/ethnicity (Zhu et al., 2010). Therefore, there is an urgent need to develop new dietary assessment methods that can overcome these limitations.

In the past decade, experts from the nutrition and engineering field have combined forces to develop new dietary assessment methods by leveraging technologies such as the Internet and mobile phones. Among the various new approaches, some of them use images captured at the eating scene to extract dietary information. These are called image-based dietary assessment methods. Examples of such methods include TADA™ (Zhu et al., 2010), FoodLog (Aizawa and Ogawa, 2015) , FoodCam (Kawano and Yanai, 2015b), Snap-n-Eat (Zhang et al., 2015), GoCARB (Vasiloglou et al., 2018), DietCam (Kong and Tan, 2012) and  (McCrory et al., 2019), to name a few. In these methods, participants are asked to capture images of foods and drinks consumed via a mobile phone. These images are then analyzed to estimate the nutrient content. Estimating the nutrient content of foods in an image is commonly performed by trained dietitians, which can be time consuming, costly and laborious. More recently, automated methods have been developed to extract nutrient information of the foods from images (Fang et al., 2017, 2015; Fang et al., 2018). The process of extracting nutrient information from images generally involves three sub-tasks, food segmentation, food classification and portion size estimation (Zhu et al., 2010). Food image segmentation is the task of grouping pixels in an image representing foods. Food classification can then identify the food types. Portion size estimation (Fang et al., 2015) is the task of estimating the volume/energy of the foods in the image. Each of these tasks is essential for building an automated system to accurately extract nutrient information from food in images. In this paper, we focus on the task of food segmentation. In particular, we propose a food segmentation method that does not require information of the food types.

Food segmentation plays a crucial role in estimating nutrient information as the image segmentation masks are often used to estimate food portion sizes (Kong and Tan, 2012; Fang et al., 2017, 2015; Okamoto and Yanai, 2016; Pouladzadeh et al., 2014). Food segmentation from a single image is a challenging problem as there is a large inter- and intra-class variance among different food types. Because of this variation, techniques developed for segmenting a particular class of foods will not be effective on other food classes. Despite these drawbacks, several learning based food segmentation methods (Zhu et al., 2015; Wang et al., 2017; Shimoda and Yanai, 2015; Dehais et al., 2016) have been proposed in recent years. One of the constraints of learning based methods is data dependency. They are only effective on the food categories they trained on. For instance in (Wang et al., 2017), class activation maps are used to segment food images. The Food-101 dataset (Bossard et al., 2014) is used to train the model and the method is tested on a subset of another dataset that have common food categories with Food-101. This is a clear indication that their method (Wang et al., 2017) is only effective on food classes that have been trained on. Similarly, the learning based method proposed in (Shimoda and Yanai, 2015) is trained and tested only on UEC-FOOD100 (Yuji et al., 2012). The UEC-FOOD100 dataset has a total of 12,740 images with 100 different food categories, out of which 1,174 have multiple foods in a single image. In their method, the dataset is partitioned into training and testing subsets, each contains all the food categories. The authors of (Shimoda and Yanai, 2015) split this dataset into training and testing in the following way. All the images containing a single food category were used for training and images containing multiple food categories were used for testing. This way the training set contained 11,566 images and the testing set contains 1,174 images. Splitting the dataset in this fashion does not guarantee that the training and testing subsets contain images belonging to different food categories. In fact this would mean they contain common food categories. Furthermore, the authors in (Shimoda and Yanai, 2015) did not conduct any cross dataset evaluation. Thus the learning based method in (Shimoda and Yanai, 2015) is also only effective on food categories it has been trained on. In (Dehais et al., 2016), a semi automatic method is proposed to segment foods. The authors of (Dehais et al., 2016) assume that foods are always present in a circular region. In addition, they assume information about the number of different food categories is known. The experiments are conducted on a dataset of 821 images. While they achieved promising results, the proposed approach is not designed for real world scenario as their assumptions may not hold. In (Chen et al., 2015), a food segmentation technique is proposed that exploits saliency information. However, this approach relies on successfully detecting the food container. In (Chen et al., 2015), the food container is assumed to be a circular plate. Experimental results were reported using a dataset consisting of only 60 images. While the assumptions in (Chen et al., 2015) are valid in some cases, it may not be true in many real life scenarios.

In addition, there are also constraints imposed by the available datasets. Publicly available food image datasets such as UECFOOD-100 (Yuji et al., 2012), Food-101 (Bossard et al., 2014) and UECFOOD-256 (Kawano and Yanai, 2015a) are biased towards a particular cuisine and also do not provide pixel level labelling. Pixel level labelling is crucial because it forms the necessary ground truth for training and evaluating learning based food segmentation methods. To overcome the limitations posed by learning based methods and the availability of public datasets with ground truth information, we proposed to develop a food segmentation method that is class-agnostic. In particular, our class-agnostic food segmentation method uses information from two images, the before eating and after eating image to segment the foods consumed during the meal.

Our data is collected from a community dwelling dietary study (Kerr et al., 2016) using the TADA™ platform. In this study, participants were asked to take two pictures of their eating scene, one before they start eating which we call the before eating image and one immediately after they finished eating which we call the after eating image. The before eating and after eating image represent the same eating scene, however for the purpose of this work, we only select image pairs where the after eating image does not contain any food. Our goal is to segment the foods in the before eating image using information from both before and after eating images. To illustrated this problem in a more general scenario, lets consider an experimental setup in which a person is given a pair of images shown in Fig.1 and is asked the following question, “Can you spot the salient objects in Fig. 1a that are missing in Fig. 1b?”. We refer to these as the salient missing objects. To find salient missing objects, the Human Vision System (HVS) compares regions that are salient in both images. In this example, the food, container and color checkerboard in Fig. 1a are the salient objects and in Fig. 1b, the color checkerboard, spoon and container are the salient objects. Comparing the salient objects in both of these images, HVS can identify the food as the salient missing object. In this paper, our goal is to build a model to answer this question. By looking for salient missing objects in the before eating image using the after eating image as the reference we can then segment the foods without additional information such as the food classes. As the above approach does not require information about the food class, we are able to build a class-agnostic food segmentation method by segmenting only the salient missing objects.

The above question does not bear significance for just any pair of random images. It only becomes relevant when the image pairs are related. For example, in Fig. 1, both images have many regions/objects with same semantic labels such as color checkerboard, container and the black background. However, the relative positions of these regions/objects are different in both images due to camera pose and different time of capturing the images. Because of similarity at the level of semantics between both images, it is plausible to define the notion of salient missing objects. Notice that we are not interested in pixel-level differences due to changes in illumination, poses and angles.

In this experimental scenario, the visual attention of HVS is guided via a task, hence it falls under the category of top down saliency. Visual attention  (Borji et al., 2013; Borji and Itti, 2013) is defined as the process that capacitates a biological or artificial vision system to identify relevant regions in a scene (Borji et al., 2013). Relevance of every region in a scene is attributed through two different mechanisms, namely top down saliency and bottom up saliency. In top down saliency, attention is directed by a task. An example of this mechanism in action is how a human driver’s HVS identifies relevant regions on the road for a safe journey. Other examples where top down saliency have been studied are sandwich making (Ballard et al., 1995) and interactive game playing (Peters et al., 2007). In bottom up saliency, attention is directed towards those regions that are the most conspicuous. Bottom up saliency is also known as visual saliency. In the real world, visual attention of HVS is guided by a combination of top down saliency and bottom up saliency. In the above question of finding salient missing objects, visual attention is guided by a task and hence it falls under the category of top down saliency. Top down saliency has not been studied as extensively as visual saliency because of its complexity (Borji et al., 2013).

In this paper, we propose an unsupervised method to find the salient missing objects between a pair of images for the purpose of designing a class agnostic food segmentation method. We use the after eating image as the background to find the contrast of every pixel in the before eating image. We then fuse the contrast map along with saliency maps to obtain the final segmentation mask of the salient missing objects in the before eating image. We also compare our method to other class-agnostic methods. Since food is a salient object in the before eating image, by detecting salient objects in the before eating image we are able to segment the food. We compared our method to four state-of-the-art salient object detection methods, namely R3NET (Deng et al., 2018), NLDF (Zhiming et al., 2017), UCF (Pingping et al., 2017) and Amulet (Zhang et al., 2017).

The paper is organized as follows. In Section 2, we formulate our problem and discuss related work. We describe our proposed method in detail in Section 3. In Section 4, we discuss dataset and experiment design. In Section 5, we discuss experimental results and compare our method with other salient object detection methods. Conclusions are provided in Section 6.

In this section, we first introduce common notations used throughout the paper. We then discuss related works on modeling top down saliency and change detection.

In this section, we describe the details of our proposed method to segment salient missing objects in $I^{b}$. Our method consists of three parts, segmentation and feature extraction, contrast map generation and saliency fusion. An overview of our proposed method is described in Fig. 2.

The goal of our method is to segment the salient missing objects in $I^{b}$ using information from a pair of images $I^{a}$ and $I^{b}$. In the contrast map generation step as described in Section 3.2, we provide an estimate of the probability of pixels belonging to objects/regions in $I^{b}$ but missing in $I^{a}$. In the saliency fusion step as described in Section 3.3, saliency information of pixels in $I^{b}$ is fused into the contrast map $C^{b}_{a}$ so as to emphasize that we are looking for salient missing objects. In order to show that the various steps of our proposed method achieve their individual objectives, we plotted the PR and ROC curves of the contrast map $C^{b}_{a}$, the visual saliency map $S^{b}$ from R3NET (Deng et al., 2018) and the estimated salient missing objects probability map $\hat{M}^{b}_{a}$ in Fig. 5c and Fig. 5a. In addition, we also plot PR and ROC curves for the 3 other salient object detection methods. From these plots, we can see that combining $S^{b}$ and $C^{b}_{a}$ as described in Section 3.3 improves the overall performance. This is also illustrated in Table 4.3.2, where both AUC and $F_{\beta}^{\text{max}}$ of $\hat{M}^{b}_{a}$ are higher than $C^{b}_{a}$. This is because the contrast map $C^{b}_{a}$ by itself models all the missing objects/regions while the probability map $\hat{M}^{b}_{a}$ also takes into account the visual saliency map $S^{b}$, which can more accurately model the salient missing objects. We can also observe from the PR and ROC curves in Fig. 5 and values in Table4.3.2 that our method achieved better performance than the state-of-the-art salient object detection methods such as R3NET (Deng et al., 2018), NLDF (Zhiming et al., 2017), Amulet (Zhang et al., 2017) and UCF (Pingping et al., 2017). We also visually verify the performance of our method as illustrated in Fig. 6. The salient object detection methods Amulet (Zhang et al., 2017), UCF (Pingping et al., 2017) and NLDF (Zhiming et al., 2017) failed to detect only foods in these images, while R3NET (Deng et al., 2018) succeeded in detecting the foods but also placed equal importance to other salient objects such as the color checkerboard. Our method gave higher probability to the foods which are the salient missing objects compared to other salient objects in the scene. It must also be noted that our method did not have access to information about food classes in $\mathcal{D}_{test}$. This is because $\mathcal{D}_{val}$ and $\mathcal{D}_{test}$ have very few food classes in common. By tuning the parameters on $\mathcal{D}_{val}$, our method will not have access to information about the food classes in $\mathcal{D}_{test}$. Hence the performance of our method on $\mathcal{D}_{test}$ is indicative of its effectiveness of segmenting foods in a class-agnostic manner. These unique characteristics of $\mathcal{D}$ are also explained in section 4.1. Hence, by modeling the foods as salient missing objects, we are able to build a better class-agnostic food segmentation method compared to existing methods.

In this paper, we propose a class-agnostic food segmentation method by segmenting the salient missing objects in a before eating image $I^{b}$ using information from a pair of before and after eating images, $I^{b}$ and $I^{a}$. We treat this problem as a paradigm of top down saliency detection where visual attention of HVS is guided by a task. Our proposed method uses $I^{a}$ as background to obtain a contrast map that is an estimate of the probability of pixels of $I^{b}$ belonging to objects/regions missing in $I^{a}$. The contrast map is then fused with the saliency information of $I^{b}$ to obtain a probability map $\hat{M}^{b}_{a}$ for salient missing objects. Our experimental results validated that our approach achieves better performance both quantitatively and visually when compared to state-of-the-art salient object detection methods such as R3NET(Deng et al., 2018), NLDF(Zhiming et al., 2017), Amulet(Zhang et al., 2017) and UCF(Pingping et al., 2017). As discussed in Section 1, we have only considered the case where there is no food in the after eating image. In the future, we will extend our model to consider more general scenarios.