Assistive Recipe Editing through Critiquing

We build a powerful latent representation that captures the different elements of a recipe via the mean-pooled output of the representation of each sentence using a Transformer encoder Vaswani et al. (2017). We provide the title $\mathbf{x}^{\mathit{ttl}}$, ingredients $\mathbf{X}^{\mathit{ing}}$, and instructions $\mathbf{X}^{inst}$ as raw text input. While the title $\mathbf{x}^{\mathit{ttl}}$ comprises a single sentence, the ingredients $\mathbf{X}^{\mathit{ing}}$ and instructions $\mathbf{X}^{inst}$ are provided as lists of sentences; we use raw recipe texts directly as input, removing the need of a pre-processing step. We encode the ingredients and instructions in a hierarchical manner using another Transformer to create fixed-length representations. We compute the latent representation $\mathbf{z}$ by concatenating the representations of each component and applying a projection followed by a tanh function: $\mathbf{z}=\textrm{tanh}(\mathbf{W}[\textrm{TRF}(\mathbf{x}^{\mathit{ttl}})\mathbin{\|}\textrm{HTRF}(\mathbf{X}^{\mathit{ing}})\mathbin{\|}\textrm{HTRF}(\mathbf{X}^{\mathit{ins}})]),$ where $\mathbin{\|}$ is the concatenation, and $\mathbf{W}$, $\mathbf{b}$ the projection parameters.

We treat ingredient prediction as multi-label binary classification over an ingredient vocabulary $I$, with a boolean target vector $\mathbf{y}^{\mathit{ing}}$ representing ingredients in a ground truth recipe. We use a Set Transformer Salvador et al. (2019) to decode ingredients, pooling ingredient logits over time-steps to compute binary cross-entropy loss against the target; we also employ an EOS token to predict the ingredient set cardinality:

where $\lambda$ controls the impact of the EOS loss. At inference, we return the top-k ingredients, where k is the first position with a positive EOS prediction.

The last component generates cooking instructions using a Transformer decoder. We condition the decoder on $\mathbf{z}$, previously generated outputs $\mathbf{\hat{y}}^{\mathit{ins}}_{1:t-1}$, and ingredients $\mathbf{\hat{y}}^{\mathit{ing}}$. Specifically, we encode the ingredients using an embedding layer $A(\cdot)$ and concatenate their representations with the recipe representation $\mathbf{z}$. We train using teacher-forcing and cross-entropy:

Taking inspiration from masked language and span modeling Devlin et al. (2019); Joshi et al. (2020), we train RecipeCrit as a de-noising recipe auto-encoder via the task of recipe completion: We mask random ingredients and instruction sentences in model input, and task our model to generate the full recipe. We train our model in two stages: first minimizing ingredient prediction loss $\mathcal{L}_{\mathit{ing}}$; then freezing the encoder and optimizing for instruction decoding loss $\mathcal{L}_{\mathit{ins}}$ using ground truth ingredients.

We aim to refine a recipe based on the user’s feedback and the predicted ingredients $\mathbf{\hat{y}}^{\mathit{ing}}$. We denote $\mathbf{\tilde{y}}^{\mathit{ing}}$ the vector of desired ingredients. Simply incorporating user feedback by explicitly including/removing ingredients before generating instructions often cannot satisfy user preferences due to weak conditioning between predicted ingredients and generated instructions. In RecipeCrit we turn to a critiquing method that modifies the recipe representation $\mathbf{z}$ before using the updated representation to jointly generate the edited ingredients and instructions. Specifically, users add a new ingredient $c$ by setting $\mathbf{\hat{y}}^{\mathit{ing}}_{c}=1$ or remove some using $\mathbf{\hat{y}}^{\mathit{ing}}_{c}=0$.

Inspired by success in editing the latent space in text style transfer and recommendation Antognini et al. (2021a); Wang et al. (2019), we first compute the gradient with respect to $\mathbf{z}$:

Then ,we use the gradient to modulate $\mathbf{z}$ such that the new predicted ingredients $\mathbf{\hat{y}}^{\mathit{ing}}$ are close to the desired ingredients $\mathbf{\tilde{y}}^{\mathit{ing}}$:

Prior work stopped updating $\mathbf{z}$ when $\parallel\mathbf{\tilde{y}}^{\mathit{ing}}-\mathbf{\hat{y}}^{\mathit{ing}}\parallel_{1}<\epsilon$ for some threshold $\epsilon$. We instead propose to compute the absolute difference $|\mathbf{\tilde{y}}^{\mathit{ing}}_{c}$ - $\mathbf{\hat{y}}^{\mathit{ing}}_{c}|$. Since the optimization is nonconvex, we improve convergence by using an early stopping mechanism. Our approach is unsupervised and can update the full recipe latent representation, reflecting how adding or removing an ingredient can necessitate adjustments to other ingredients and cooking steps. Pseudo-code is available in the App.

Another advantage of our approach is the possibility to update multiple ingredients simultaneously: adding or removing an ingredient might affect other ones as well and thus, a local-based stopping criteria allows such a change.

We assess our model on the Recipe1M Salvador et al. (2017) dataset of 1M recipe texts. Each recipe contains a title, a list of ingredients, and a list of cooking instructions. We filter out recipes with more than $20$ ingredients or steps, creating train, val, and test splits with 635K, 136K, and 136K recipes, respectively. The average recipe comprises 9 ingredients and 166 words. We follow Salvador et al. (2019) and build a set of $1,488$ ingredients. For critiquing, we select $20$ ingredients to be critiqued among the most and the least popular ingredients across the train set. For each critique, we randomly sample $50$ recipes that contain the critiqued ingredient and $50$ that do not.

We compare our proposed RecipeCrit architecture against large language models trained using our denoising objective. We fine-tune BART Lewis et al. (2020), an encoder-decoder language model trained to denoise documents, as well as RecipeGPT Lee et al. (2020), a decoder-only language model pre-trained on Recipe1M to predict ingredients and cooking steps. To demonstrate the necessity of our denoising approach, we also compare against PPLM Dathathri et al. (2020), a recent method for controllable generation from language models that leverages sets of desired and undesired sequences (for ingredient addition and substitution, respectively). All models use greedy decoding.

We evaluate edited recipes via metrics that reflect user preferences. First, a user wants a recipe similar to the base recipe—we measure ingredient fidelity via IoU (Jaccard distance) and F1 scores between the edited and base recipe ingredients list. Next, the recipe must satisfy the user’s specific ingredient feedback—we report the percentage of edited recipes that properly include/exclude the target ingredient (Success Rate). Finally, the recipe must be coherent: able to be followed and internally consistent. As an ingredient constraint can be satisfied in many ways, we follow Kiddon et al. (2016) and measure coherence via precision, recall, and F1-score of ingredients mentioned in the generated steps compared to the predicted ingredients. This verifies that the recipe itself relies on the listed ingredients.

For fair comparison, we compare similar-sized models. RecipeCrit uses an encoder and decoder with 4 Transformer layers, 4 attention heads, and hidden size of 512. We randomly mask $50\%$ of the ingredients and instructions during training, and tune them on the validation set using random search. We give more details in App. B.

We evaluate whether our models can edit recipes by creating new ingredient sets and corresponding recipe instructions when faced with positive and negative feedback: an ingredient that must be added or removed (substituted) from the recipe to create a new version. For ingredient substitution, we mask the critiqued ingredient and all steps that reference it as denoising inputs; for addition, we use the full base recipe. For RecipeGPT and BART, we filter the predicted ingredients lists to exclude/include the target ingredient. For PPLM, we provide the target ingredient as a bag of words to steer generation, using RecipeGPT as the base generative model. RecipeCrit uses our iterative critiquing framework (Section 2) to accommodate user feedback.

We show results for constraint satisfaction (success rate), ingredient fidelity, and recipe coherence (predicted instructions) in Table 1. RecipeCrit out-performs baselines across all metrics for ingredient addition and removal. While our baselines take advantage of pre-trained language models, they cannot successfully incorporate user feedback during editing. PPLM-guided constrained decoding is not only two orders of magnitude slower than our denoising models (3min vs. 1s per recipe), but we observe poor fidelity and frequent incoherent instructions (e.g., repetition). Meanwhile, forcing ingredient lists to omit or contain specific ingredients has little impact on the generated recipe instructions—even when the desired ingredient is manually inserted into the ingredients list, RecipeGPT and BART mention using the ingredient only in 33% and 41% of generated instructions.

Our model and gradient-based critiquing method leads to a stronger influence of the edited ingredients on recipe instructions. By directly modifying the recipe latent representation that is then attended over during step generation, RecipeCrit achieves 30-50% relative improvement in success rate for adding ingredients and 20-65% relative improvements in coherence (F1 score between predicted ingredients and those mentioned in the instructions) for both addition and removal. Meanwhile, baselines tend to ignore many ingredients in the ingredient list when generating new recipe directions.

Human Evaluation We have established that RecipeCrit creates edits that better satisfy user constraints (as expressed via critiques), more closely resemble the user’s original preferences (base recipe), and make better use of the predicted ingredients (ingredient coherence). We next perform a qualitative human evaluation of our edited recipes via Mechanical Turk, asking the user: how pleasantly surprised they were (Serendipity); whether the recipe respected their feedback (Correctness); how easy the recipe was to follow (Coherence); and whether the recipe resembled the original recipe (Relevance). We uniformly sampled 800 edited recipes (400 for adding and 400 for removing) across the ingredients to critique and showed them in random order. The annotators judged the edited recipes using best-worst scaling Louviere et al. (2015) with scores normalized to $[-1,+1]$. Table 2 shows that our edited recipes are largely preferred on all criteria. Our results highlight that critiquing improves the coherence of generated recipes and their resemblance to the original ones.

Case Study Table 3 shows a sample of our best-performing baseline (BART) and RecipeCrit editing the “cherry tomato confit” recipe to include “kale”. While both edited recipes include kale, RecipeCrit stays faithful to the user’s preference for “tomato confit” while incorporating the new feedback: it makes a slightly different tomato confit but uses kale as the “fresh” or salad part of the dish. However, BART generates a cocktail recipe instead that ignores the base recipe: it’s a drink rather than food, sweet rather than savory, and ignores tomatoes altogether. This aligns with the results of the human evaluation. Complementary results are shown in Table 5 and 6.

Now we show the significance of the early stopping mechanism in our particular critiquing module compared to previous thresholding methods Antognini et al. (2021a); Wang et al. (2019). To demonstrate why, we re-run experiments from RQ1 and compare our early stopping against two baseline thresholding criteria using 1) the absolute difference (i.e., $|C(\mathbf{z}^{*}_{t})_{c}-\mathbf{\tilde{y}}^{\mathit{ing}}_{c}|<\tau$) and 2) the L1 norm (i.e., $||C(\mathbf{z}^{*}_{t})-\mathbf{\tilde{y}}^{\mathit{ing}}||_{1}<\tau$). We find that an L1-based stopping criterion is suboptimal due to the high dimensionality of the ingredients. Using the absolute difference considerably improves the success rate ($+25$% for add and $+12$% for remove). Finally, our early stopping further increases the success rate ($+10$%) for both adding and removing an ingredient (see App. for exact numbers).