Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective

Cross-modal recipe retrieval aims at mutual retrieval between recipes and corresponding food images. Earlier works [4, 26, 46, 41, 14, 45, 23] use a pre-trained textual representation (e.g., word2vec [18] for word embedding, skip-thought [13] for sentence embedding), followed by LSTM to obtain the final recipe embeddings. [36] introduces StyleGAN2 [11] to generate images and aligns them in latent space with text for data enhancement. The recipe retrieval performance is further advanced by transformer-based works [7, 25, 28, 37]. [28, 27, 34] have introduced CLIP-based models and achieved promising performance, but they all fully fine-tuned the image encoder of the pre-trained CLIP to encode food images with extensive computation cost. Different from existing methods, we propose a new data augmentation paradigm to enhance the cross-modal recipe retrieval performance.

Large language models (LLMs) [2, 1, 31, 3, 32] have demonstrated strong language abilities and achieved great success by training with large-scale data. In computer vision, the recently proposed Segment Anything Model (SAM) [12], a visual foundation model can segment the specified content of an image for a given prompt (e.g., box, point, or mask), thus providing semantic visual information for downstream tasks. These foundation models have been attempted for data augmentation. For example, SAM is used in [44] to obtain priority maps for medical image segmentation enhancement. [40] explores data augmentation on multilingual commonsense datasets with powerful instruction-tuned LLMs, and diffusion model is used to enhance image diversity in [33]. In this paper, we employ Llama2 [32] and SAM to generate visual imagination description from recipe to capture visual cues of food images and image segments corresponding to ingredients in recipe respectively.

Given an image as a query, image-to-recipe retrieval aims to find the most relevant recipe from a recipe corpus, and vice versa. Assume a set of $N$ image-recipe pairs $\{(i_{t},r_{t})\}_{t}^{N}$ are given, where $i_{t}$ is a food image and $r_{t}$ is the corresponding recipe. $i_{t}\in V$ and $r_{t}\in R$, where $V$, $R$ represent the visual and recipe modal spaces, respectively. As the image and recipe belong to different modalities, they cannot be directly compared. As a result, the key to cross-modal recipe retrieval is to learn a mapping function $\Psi(V,R)\rightarrow(E_{V},E_{R})$ that maps recipes and images into a common embedding space for similarity measurement, where $E_{V}$ and $E_{R}$ are image and recipe embeddings, respectively.

This paper aims to explore a new data augmentation paradigm using foundation models to enhance cross-modal recipe retrieval performance. The overview of our proposed model is depicted in Fig. 2. Firstly, we propose to employ foundation models to produce augmented data from recipe and image respectively. Specifically, on the one hand, image segments are extracted from the original food image by the visual foundation model SAM, which corresponds to the key ingredients in the recipe at the semantic level. On the other hand, for the recipe, Llama2 is instructed to imagine the visual description of the food based on the recipe, the result of which we refer to “visual imagination description”, denoted as $d$. Furthermore, inspired by [30], we propose to adopt adapters based on pre-trained CLIP model to encode both original and augmented data, then we further introduce a multi-level circle loss function to align original and augmented recipe and image data in the common embedding space.

By leveraging the powerful language generation capabilities of the LLM, we aim to produce a description that is visually aligned with a food image, using a recipe as input. As shown in Fig. 2 (c), we carefully design a “visual imagination” prompt to instruct LLM to play the role of a helpful assistant who can imagine what the food will look like based on a recipe. The generated visual imagination description is limited to approximately 30 words to fit the maximum number of input tokens of CLIP. Furthermore, to mitigate the hallucination, we instruct the LLM to generate “objective” and “informative” responses “according to this recipe” in the prompt.

We use the “everything” ^{†\dagger†\dagger}$\dagger$https://segment-anything.com/demo mode to obtain multiple segments from each food image from SAM. i.e., Zero-Shot object proposal generation. SAM samples a large number of points on the image as the prompt, then filters and de-duplicates the generated prediction masks, and finally generates all the prediction masks for the whole image. Note that not all the segments are useful. For instance, there are a large number of background and decorative noises that are not related to food, and there are also many segments that are too small with less semantic information. As a result, we filter out valuable segments by setting an area threshold and semantically consistent matching of segments with text embeddings of {a picture of food} using the CLIP model. As shown in Fig. 2, the image segments extracted from the food image are the key ingredients to cook the dish. We use the top-$n$ segments as the image augmented data based on the filtering scores.

To fully make use of augmented data, i.e., visual imagination description and image segments, we propose Data Augmented Retrieval (DAR) framework to enhance cross-modal recipe retrieval performance. We aim to make the model efficient without significantly increasing computational cost and introduce a flexible cross-modal alignment learning strategy for the rich set of data.

The CLIP image encoder can be directly applied to encode the food image. In contrast, a recipe is a long document with three parts: title, ingredients and instructions, denoted as $r=(r_{\text{tit}},r_{\text{ing}},r_{\text{ins}})$, CLIP text encoder is incapable to model a recipe in one shot. To address this issue, we propose to use three independent CLIP text encoders to encode the three parts separately. As title is usually one sentence, we can directly use title encoder $\phi_{\text{tit\&A}}$ to obtain title features $E_{R_{\text{tit}}}$. Different from title, the ingredients and instructions sections generally consist of multiple sentences, thus we encode each sentence independently with the CLIP text encoder with adapter layers to get the sentence-level features. Moreover, to learn the compact features of ingredients and instructions, we propose to use a two-layer transformer encoder to learn the interactions among the sentences in ingredient and instruction as follows:

where $M_{\text{ing}}$ and $M_{\text{ins}}$ are the maximum number of sentences that can be acceptable for ingredients and instructions, $[r_{*}^{1},\cdots,r_{*}^{M_{*}}]$ form the list of sentences for ingredients or instructions, and Trans represents the transformer structure.

The features of the three parts of the recipe are then concatenated together, which is subsequently fed into a fully-connected (FC) layer to get the final recipe embedding $E_{R}$, which can be formalized as follows:

Similar to the operation for the recipe, we employ CLIP text encoder with the adapter to compute the output of the visual imagination description $d$ as follows:

Considering that there are several image segments and to prevent the effect of partially noisy samples from increasing, we use the fully frozen CLIP image encoder $\phi_{\text{seg}}$ to encode each image segment first. The final segment embedding $E_{S}$ is obtained by averaging the embeddings of the $n$ segments. The formula can be written as:

where $s_{i}$ is the image segment and $n$ is the number of segments. $\phi_{\text{seg}}^{*}$ refers to frozen segment encoder.

Existing works typically employ triplet loss and its variants for cross-modal embedding space learning [47, 27, 25, 38, 6], Given a query, triplet loss aims to push distance between the query and positive samples larger than that of negative pairs with a pre-defined margin, but this means that increasing the cosine similarity score $c_{p}$ between a query and its positive sample is equivalent to decreasing the cosine similarity score $c_{n}$ between the query and negative sample, e.g. when $c_{p}$ is small and $c_{n}$ approaches 0, it keeps on penalizing $c_{n}$ with a large gradient, thus the optimization lacks flexibility.

To address this issue, we first introduce circle loss [29] for cross-modal recipe retrieval. Specifically, the key idea is to employ different penalties for $c_{p}$ and $c_{n}$ as follows:

where $m\in[0,1]$ represents the relaxation factor, and $\gamma$ is the scale factor used to rescale the cosine similarity score. For the set of data pairs $(A,B)$, $K$ and $L$ denote the number of positive and negative pairs for all queries. In addition, to improve the robustness of the model, we use the symmetric bidirectional circle loss, with the following equation:

We then build the cross-modal embedding space with multiple alignment objectives, including recipe and image $L(E_{V},E_{R})$, image segment and recipe $L(E_{S},E_{R})$, and image and visual imagination $L(E_{V},E_{D})$. Furthermore, as there are three sections in a recipe, i.e., title, ingredients and instructions, we are inspired by  [25] to align any two sections of a recipe in a self-supervised manner to encourage semantic consistency within the recipe. We define the recipe loss $L_{\text{rec}}$ using circle loss as follows:

where $a,b\in[tit,ing,ins]$, $\delta(a,b)=1$ if $a\neq b$ otherwise $0$ and $LN(\cdot)$ is a linear projection. $E_{R_{a}}$ and $E_{R_{b}}$ are two different content embeddings in a recipe.

Finally, we can obtain the overall multi-level circle loss by combining the above losses as follows:

where $\alpha$, $\beta$ and $\sigma$ are hyperparameters to balance the losses.

Dataset. In line with previous works, Recipe1M [26] dataset is used to train and evaluate our method. The recipes with corresponding images are split into 238,408, 51,119 and 51,303 for training, validation and testing respectively. In addition to utilizing these pairs of data, following [25], we also utilize unpaired 482,231 training recipes from the rest of the dataset for the recipe loss $L_{\text{rec}}$.

For evaluation protocols, the augmented data can be not only used for training but also pre-computed for evaluation. Existing works [34, 28] generally compute the cosine similarity between recipe and image as a measure to evaluate the retrieval performance. In contrast, we introduce two extra evaluation protocols based on the augmented visual imagination description and image segments as follows:

• •

  DAR. Similar to previous works, DAR only utilizes the distance between image and recipe for evaluation, i.e., $dist=dist_{i-r}$.

• •

  DAR+ adds visual imagination description into the evaluation, which calculates the distance between image and visual imagination description $dist_{i-d}$, and multiplies by $dist_{i-r}$ to get the distance, i.e., $dist=dist_{i-d}\cdot dist_{i-r}$.

• •

  DAR++ adds both visual imagination and image segments for evaluation, which computes the distance $dist_{s-r}$ between segments embeddings and recipe embeddings, then multiplies by the distance metric of DAR+, i.e., $dist=dist_{s-r}\cdot dist_{i-d}\cdot dist_{i-r}$.

In the training phase, we set the relaxation factor $m=0.25$ and scale factor $\gamma=32$ for circle loss. The weight factors for multi-level circle loss are set to be $\alpha=1$, $\beta=1$, $\sigma=1$. Following the baseline settings, we train the model with batch size =128 and optimize the parameters using Adam optimizer. The initial learning rate $lr$ is set to $10^{-4}$ and every 30 epochs step decays to 0.1 of the previous $lr$. The model is trained for a total of 100 epochs, and the model parameter with the highest R@1 in validation is selected for testing.

As shown in Table 1, we compare the cross-modal recipe retrieval performance of our DAR with state-of-the-art methods. Our DAR outperforms all existing methods for the majority of the metrics in both 1k and 10k setups, including CLIP-based methods [27, 34, 28, 35, 9]. All of these methods simply fine-tune the entire image encoder of the CLIP to improve the performance. In contrast, by inserting lightweight adapter layers, the number of trainable parameters in our image encoder is only 8% of theirs. Compared to MALM [34], the image-to-recipe retrieval performance in 1k testing is boosted by 0.9%, 3.4% and 3.2% in terms of R@1, R@5 and R@10 respectively. The results show that our DAR learns more discriminative recipe and image embeddings, enhancing cross-modal recipe retrieval through data augmentation.

In addition, we also report the results with data augmentation in test time. By adding visual imagination description, the retrieval performance of DAR+ can be further improved with noticeable margins in terms of all the metrics than DAR. Furthermore, by adding both augmented image segments and visual imagination description during testing, i.e., “DAR++”, image-to-recipe performance is slightly boosted than DAR+, but the recipe-to-image retrieval performance is inferior to DAR+. We believe the reason is the limitation of image segments, which is analyzed in Sec.4.4. Regarding CLP [9], the re-ranking step during inference is essential for achieving higher R@1 than our DAR and DAR+, yet it remains inferior to our DAR++. Moreover, our DAR, DAR+ and DAR++ consistently outperform CLP.

We conduct the ablation study to validate the key components of our proposed method DAR, including the adapters in CLIP encoders, augmented data from foundation models and training losses. Unless otherwise specified, all the results are reported in 10K testing for evaluation with original image-recipe pairs.

In addition, we conduct experiments by instructing LLM to generate a summary of recipe for data augmentation as well. The results show that visual imagination description manages to outperform summary by 0.7% and 0.9% in image-to-recipe and recipe-to-image retrieval respectively. Finally, as listed in Table 4, we examine the performance with different numbers of segments from SAM. When segments are too few (n=1 or 2), key ingredients may be excluded. Conversely, a large number of segments (n=6) not only raises computational costs but also increases the likelihood of introducing noise. Based on the results, our DAR sets segment numbers to 4.

Fig. 3 and Fig. 4 show the qualitative examples of recipe-to-image retrieval and image-to-recipe retrieval respectively. The results are reported based on DAR++, which includes extra augmented image segments and visual imagination description for testing.

First, we show the result of retrieving the corresponding image using recipe as query in Fig. 3. During inference, we can generate visual imagination description for each recipe query, which is combined as query to retrieve food images. In the first example, the query recipe is “feta chicken bake”. We can see that all the Top-5 retrieval images are chicken, which are semantically and visually similar. Though our model DAR ranks the corresponding image in third position, our model DAR++ manages to improve the rank to first place. The reason is that our visual imagination description is capable of providing auxiliary visual-related information “topped with fresh parsley” and “red pepper flakes …” to further distinguish the similar image returned by DAR but where the chicken is topped with scallions. The same phenomenon can be observed in the second example as well where “rollups” is interpreted as “…wrapped up in corn tortilla” to correct the result that retrieves a picture of the rollups made of meat by DAR. As the visual imagination description contains visual cues that do not exist in recipes, the recipe-to-image retrieval could be more interpretable by incorporating visual imagination description in the retrieval system. In the third example “boulettes de pomme de terre”, which means “potato dumplings”, compared to DAR, our DAR++ is not able to rank higher than DAR, it is because the visual imagination provides misleading information “coated in a light layer of golden brown breadcrumbs” whereas in the recipe the color is described as “brown a bit” and there are no ingredients for “breadcrumbs”.

As shown in Fig. 4, we present examples of Top-5 retrieved recipes using food image as query of DAR++. Following [25], the recipes are shown with word clouds for better visualization. Given a query image, our DAR++ employs SAM to obtain image segments, which are subsequently combined as query as well. Multiple segments are combined as one image for visualization purpose. It can be observed that there is a correspondence between the segments and key ingredients in the corresponding recipe. For instance, as the most notable ingredients, “ribs” in the first example and “beef”, “carrots” in the second example can be found in the segment picture and the first ranked retrieved recipes. As a result, our method potentially provides better interpretability for cross-modal recipe retrieval with the image segments.

The adoption of adapter’s structure allowed us to leverage CLIP’s cross-modal retrieval capabilities with few parameters. In Table 6, we provide a comparison of the number of trainable parameters with existing CLIP-based methods [27, 28]. In contrast to the full fine-tuning of CLIP, our DAR manages to achieve superior performance with much fewer parameters, showcasing the efficiency of our approach.

In the paper, we propose multi-level circle loss [29] to regularize the common semantic space. Meanwhile, in order to balance the weights between each loss, we set the weight factors $\alpha$, $\beta$ and $\sigma$ for $L(E_{S},E_{R})$, $L(E_{V},E_{D})$ and $L_{\text{rec}}$ respectively. $\sigma$ is set to 1 following [25]. Table 7 shows the performance of the model with different settings of $\alpha$ and $\beta$. The performance of assigning $\alpha=1,\beta=1$ is slightly better than that of $\alpha=0.1,\beta=0.1$, in particular in terms of R@1. In addition, we show more selection results on the hyperparameters $\gamma,m$ of circle loss in Table 8, which are searched on the validation set.

We further examine the performance of H-T(ViT) [25] with our proposed test-time augmentation by directly introducing the distance matrices of augmented data in DAR+ and DAR++ into its evaluation. As shown in the Table 9, similar to DAR+ and DAR++, H-T(ViT)+ and H-T(ViT)++ exhibit consistent improvements over H-T(ViT), which further validate the effectiveness of test-time augmentation.

In Fig 5, we show more samples of visual imagination descriptions and corresponding image, notably “goat cheese oozing out of the edges” of example (b) and “each with several vertical slits cutting into the skin without cutting through the potato” of example (e) and other fine-grained alignment information.

In addition to Llama2 adopted in the main paper, we also examine GPT-3.5 to generate visual imagination description. We show the visual imagination description obtained by Llama2 and GPT-3.5 for the same recipe with the same prompt in Fig. 6. Here we adopt gpt-3.5-turbo model for GPT-3.5 and Llama2-13b-chat model for Llama2. In the example in Fig. 6(a), both models capture the “golden-brown” color and the “filled with smoked ham and Gruyere cheese” feature. In (b), our Llama2 also focuses on the detail that “sausages are cut into bite-sized pieces”. In contrast to the GPT-3.5 which only notes “Topped with a lid” of the pumpkin risotto in (c), our model describes it as “pumpkins filled with a rich, golden rice”. As a result, both LLMs capture key visual features in the recipe.

Fig. 7 shows more qualitative examples of image segments and the corresponding word clouds of top-5 retrieved recipes. It can be seen that SAM accurately segments the food parts from the original image, e.g., in example (b), the pasty is accurately distinguished from the similarly colored table, and in example (d), the three shrimps clearly segmented correspond to the “shrimp” ingredient in the recipe. These results show that segments effectively help DAR++ to align the image and the recipe through the ingredients.

Nevertheless, previous experimental results in the main paper both in terms of the degradation of model performance after adding adapter layers into the segment encoder, and the fact that DAR++ cannot bring stable improvement compared to the DAR+ during test-time augmentation, we believe that these problems are due to the quality of some segments obtained by SAM. The inherent shortcomings of SAM make some segments mixed with noise [10, 43]. We show some representative noise samples (examples with purple box) selected from the recipe1M in Fig. 7 to illustrate the problems we observed in SAM segmentation for data augmentation. There are two imperfections of SAM, the first one is that it sometimes misses small, irregular structures, such as the omission of lemon in example (f) and the discarding of irregular parts of the salad in example (j). Another imperfection is that the “everything” mode over-segments the large objects that include many details and ignores overall semantics at times. In example (g), SAM ignores the whole cream at the bottom, while in (h) it ignores the whole pork chop. In addition, the filtering of segments is not successful for all samples, which results in background information remaining in examples (f) and (i).

To address the problems of segments, we designed sample analysis experiments to illustrate the future implications of solving these problems. Specifically, for a portion of the test samples that were retrieved incorrectly by DAR++, we would like to eliminate the noise in the segments by manually utilizing SAM (with points prompt) to obtain image segments instead of the original segments. The experimental results also show that the bad segment samples limit the further improvement of model performance. In Fig. 8, for the original segmentation, the “Sprite” in the first row is plagued by the complex background and only the lid of the cup is segmented. Due to SAM’s over-segmentation of large objects, the cake in the second row and the cheese in the third row only retain the decoration on top of the cake and a very small portion of the cheese, and also contain irrelevant information such as the background and the plate. With our manual segments improving these problems, DAR++ successfully retrieved the correct recipes with blue boxes. We will explore how to further improve the quality of the segments in the future work.