PDPP: Projected Diffusion for Procedure Planning in Instructional Videos

With an aim to learn the inter-relationship between different events or steps in videos, the problem of procedural video understanding, which is important for the training of AI models as human assistants, has gained more and more attention recently. Zhao et al.[56] investigated the problem of abductive visual reasoning, which requires vision systems to infer the most plausible visual explanation for the given visual observations. Furthermore, Liang et al.[28] proposed a new task: given an incomplete set of visual events, AI agents are asked to generate descriptions not only for the visible events and but also for the lost events by logical inferring. Unlike these works trying to learn the abductive information of intermediate events, Chang et al.[6] introduced procedure planning in instructional videos which requires AI systems to plan an action sequence that can transform the given start observation to the goal state in order to assist humans in daily scenarios. Different with the long-term action anticipation (LTA) task[29, 14, 31], procedure planning requires a visual observation as goal to make goal-directed planning and covers a wide range of goal-oriented activities in everyday life. Patel et al.[35] further point out the unavailability of the goal visual state in procedure planning and propose VPA task to use a language described goal instead. In this paper, we study the procedural video understanding problem by learning the goal-directed procedure planning task.

Multiple methods have been proposed to solve the procedure planning. Chang et al.[6] conduct a dual dynamic network to model the transformation relationship between intermediate states and actions. When inference, the actions and states are generated step by step. Bi et al.[4] follow this idea and use model-based reinforcement learning to predict the state-action pairs in an autoregressive way. They also learn the time-invariant context information from the given observations for better planning. Instead of using the intermediate states supervision, Zhao et al.[55] apply a single branch non-autoregressive transformer decoder to predict all intermediate actions in parallel with language annotations. We here propose our PDPP model, which can generate both accurate and diverse plans without any states or language supervision, to solve this problem.

Diffusion probabilistic models[44] are generative models which have been gaining significant popularity nowadays and have achieved great success in many areas. Ho et al.[20] used a reweighted objective to train diffusion model and achieved great synthesis quality for image synthesis problem. Janner et al.[23] studied the trajectory planning problem with diffusion model and get remarkable results. Besides, diffusion models are also used in video generation[22, 19], density estimation[25], human motion[49], sound generation[54], text generation[27] and other domains, all achieved competitive results. There are various architectures for implementing diffusion models, such as Unet[41], Unet with attention layers[20] and Transformer[36].

Diffusion Models work by destroying training data through the successive addition of Gaussian noise. Then these models learn to reverse this noising process by removing the added noises step by step to recover the initial data. Under such a design, diffusion models require no adversarial training and have the added benefits of learning scalability and parallelizability with a simple learning scheme. Besides, the generated results can vary with different noises, which is helpful to the diverse generation. With the above advantages of diffusion models, we in this work treat the whole action sequence in procedure planning as a distribution and apply diffusion process to this problem to model the uncertainty. Our proposed projected diffusion model PDPP achieves state-of-the-art performance only with a simple learning scheme.

Projected gradient descent is an optimal solution suitable for constrained optimization problems, which is proven to be effective in optimization with rank constraints[7], online power system optimization problems[16] and adversarial attack[8], etc. The core idea of projected gradient descent is adding a projection operation to the normal gradient descent method to project the output onto the feasible set, so that a function subject can be optimized to a constraint and the result is ensured to be in the feasible region. In our PDPP, we apply diffusion to procedure planning to fit the goal distribution, which contains both the action sequence and condition-guided information. These conditions can be changed during the noise-adding and denoising stages. Inspired by projected gradient descent, we add a similar projection operation to our diffusion process, which keeps the conditional information for diffusion unchangeable and thus provides accurate guides for learning. The process of “projecting back to a bounded normal range from an abnormal state” is analogous to how we rectify a condition altered by noise to its accurate state, thereby remapping the overall distribution to a precisely guided range.

As an ensemble learning technology, MoEs[43] divides a problem space into homogeneous regions by learning multiple expert modules to process the input signal, together with a gating module called as router to help combine the outputs of different experts. When training, the router is learned to assign the combination weight across different experts, thus each expert is expected to focus on learning a certain sub-region of the input space. MoEs has shown its remarkable ability to scale neural networks such as LSTM[43], CNN[1, 53] and Transformer[38, 42, 26]. By activating dynamic sub-networks for multiple inputs with routing strategy, MoEs provides separate parameters for different tasks and can thus reduce the bad influence of sharing parameters for conflicting inputs[57]. We in this paper apply MoEs to our Unet-attention based model on large scale dataset COIN[48] for better joint training. Specifically, we try two routing strategies: direct routing and learned routing for MoEs and apply this method to different parts of our model when planning with varied horizons.

For procedure planning, we follow the problem set-up of Chang et al.[6]: given two video clips, start visual observation $o_{s}$ and visual goal $o_{g}$, a model is required to plan a sequence of actions $a_{1:T}$ so that the environment state can be transformed from $o_{s}$ to $o_{g}$. Here $T$ is the horizon of planning, which denotes the number of action steps for the model to take and {$o_{s}$, $o_{g}$} indicates two different environment states appeared in an instructional video. We decompose the procedure planning problem into two sub-problems, as shown in Eq. 1. The first problem is to learn the task-related information $c$ with the given {$o_{s}$, $o_{g}$} pair. This can be seen as a preliminary inference for procedure planning. Then the second problem is to generate action sequences with the task-related information and given observations. Note that Bi et al.[4] also decompose the procedure planning problem into two sub-problems, but their purpose of the first sub-problem is to provide long-horizon information for the second stage since Bi et al.[4] plans actions step by step, while our purpose is to get condition for sampling to achieve an easier learning.

One of our motivations is to use one model for multiple planning horizons with joint training. That is, we sample one batch of action sequences for every prediction horizon and conducting gradient descent on all these data to complete one training step. Thus horizon information $h$ for every action sequence is also provided to our model as condition.

At training time, we first train a simple model (implemented as multi-layer perceptrons (MLPs)) with the given observations {$o_{s},o_{g}$} to predict which the task category is. We use the task labels in instructional videos $\overline{c}$ to supervise the output $c$. After that, we evaluate $p(a_{1:T}|o_{s},o_{g},c,h)$ in parallel with our model and leverage the ground truth (GT) intermediate action labels as supervision for training. Compared with the visual and language supervision in previous works, task label supervision is easier to get and brings simpler learning schemes. At inference phase, we just use the start and goal observations to predict the task class information $c$ and then samples action sequences $a_{1:T}$ from the learned distribution with the given observations, horizon and predicted $c$, where $T$ is the planning horizon.

As explained in Sec. 3.1, the main part of our model is how to generate $a_{1:T}$ with given conditions to solve the procedure planning problem. Bi et al.[4] assume procedure planning as a Goal-conditioned Markov Decision Process and use a policy $p(a_{t}|o_{t})$ along with a transition model $\tau_{\mu}(o_{t}|c,o_{t-1},a_{t-1})$ to perform the planning step by step, which is complex to train and slow for inference. We instead solve this as a direct distribution fitting problem with a diffusion model. In this section, we will first introduce the standard diffusion model. Then show our conditional sampling ways and our learning objective.

Diffusion model. A diffusion model[20, 34] solves the data generation problem by modeling the data distribution $p(x_{0})$ as a denoising Markov chain over variables {$x_{N}...x_{0}$} and assume $x_{N}$ to be a random Gaussian distribution. The forward process of a diffusion model is incrementally adding Gaussian noise to the initial data $x_{0}$ and can be represented as $q(x_{n}|x_{n-1})$, by which we can get all intermediate noisy latent variables $x_{1:N}$ with a diffusion step $N$. In the sampling stage, the diffusion model conducts iterative denoising procedure $p(x_{n-1}|x_{n})$ for $N$ times to approximate samples from the target data distribution. The forward and reverse diffusion processes are shown in Fig. 3.

In a standard diffusion model, the ratio of Gaussian noise added to the data at diffusion step $n$ is pre-defined as $\{\beta_{n}\in(0,1)\}_{n=1}^{N}$. Each adding noise step can be parametrized as

Since hyper-parameters $\{\beta_{n}\}_{n=1}^{N}$ are pre-defined, there is no training in the noise-adding process. As discussed in[20], re-parameterize Eq. 2 we can get:

where $\overline{\alpha}_{n}=\prod_{s=1}^{n}(1-\beta_{s})$ and $\epsilon\sim\mathcal{N}(0,I)$.

In the denoising process, each step is parametrized as:

where $\mu_{\theta}$ is produced by a learnable model and $\Sigma_{\theta}$ can be directly calculated with $\{\beta_{n}\}_{n=1}^{N}$[20]. Then Ho et al.[20] conduct a further derivation and set the learning objective for their diffusion model as the noise added to the uncorrupted data $x_{0}$ at each step. When training, the diffusion model first selects a diffusion step $n\in[1,N]$ and calculates $x_{n}$ as shown in Eq. 3. Then the learnable model will compute $\epsilon_{\theta}(x_{n},n)$ and calculate loss with the true noise add to the distribution at step $n$. After training, the diffusion model can simply generate data like $x_{0}$ by iteratively processing the denoising step starting from a random Gaussian noise.

However, such a diffusion model can not be applied to procedure planning problem since the sampling process in this task is condition-guided while no condition is applied in the standard diffusion model. There are multiple methods for implementing conditional-guided diffusion model, which could be divided into two categories: adding condition to diffusion model or conducting sampling guidance. The former adds conditional inputs to the model through cross-attention[39], AdaGN[9] or token concatenation along the sequence dimension[30], while the later conducts conditional sampling by providing explicit[9] or implicit[21] guidance during the generation phase. Inspired by Janner et al.[23], we here choose to add condition to our diffusion model via designing conditional action sequence input and applying condition projection. Besides, we also tried to restrict predicted actions by task mask and apply MoEs to our model for task and horizon conditions.

Another noteworthy point is that the distribution we want to fit is the whole action sequence, which has a strong semantic information. We notice that in experiments which take noise as the predicting objective like[20], our model just fails to be trained, indicating that directly predicting the semantically meaningless noise sampled from random Gaussian distribution can be hard. To address this problem, we change the learning objective of our model to provide a clear and strong learning distribution.

Conditional action sequence input. The input of a standard diffusion model is the data distribution it needs to fit and no guided information is required. For the procedure planning problem, the distribution we aim to fit is the intermediate action sequences $[a_{1},a_{2}...a_{T}]$, which depends on the given observations. Thus we need to find how to add these guided conditions into the diffusion process. For a concise learning strategy, we here apply a simple way to achieve our goal: just treat these conditions as additional information and concatenate them along the action feature dimension. Specifically, our conditions include observation(o), task label(c) and horizon(h) for joint learning. For observation, since the start/end observations are more related to the first/last actions, we concatenate $o_{s}$ and $a_{1}$ together, same for $o_{g}$ and $a_{T}$. Task label and horizon features are useful to the whole predicted sequence, thus we duplicate them for every action. Thus our model input for training now can be represented as a multi-dimension array:

Each column in our model input represents a certain action and the corresponding condition information. Note that we do not need the intermediate visual observations as supervision, so all observation dimensions are set to zero except for the start and end observations. For VPA, $o_{g}$ is not provided and thus set as zero.

Adding condition by task mask and MoEs. Note that the conditional action sequence input can change with different ways to add task and horizon conditions. That is, task label dimension is removed when task mask condition is applied and horizon dimension will be discarded when using MoEs.

Task mask can be seen as a more adequate application of task information. Once the task category is determined, the planning sequence can only consist of actions belonging to this task. For example, actions in task ”Build Simple Floating Shelves” are ”cut shelve”, ”assemble shelve”, ”sand shelve”, ”paint shelve” and ”attach shelve”. Only these five actions can be involved in any action sequence describing how to build simple floating shelves. So with the predicted or given task condition, we can get a task mask which blocks out all actions that are not part of this task and thus reduce the searching space for action planning. In this sense, we can introduce task condition by multiplying the action distribution by the task mask.

As for horizon condition, in addition to concatenating horizon information and action together, we also apply MoEs[43]. The main idea of MoEs is to replicate certain parameters within a model and use routing algorithms to process different inputs with different sets of parameters. Thus, we replicate the parts of our model where MoEs is used and provide condition information to the routing gate module added in PDPP. We implement both direct routing and learned routing strategies. Direct routing just distributes input to specified sub-networks according to the horizon condition while learned routing learns a weight to integrate the outputs from all sub-networks together with the given conditions. Specifically, the learned routing module is implemented as MLP. Since our main goal is to save model parameters by joint training, we here only consider applying MoEs to the convolution part or attention part of Unet with attention layers model. By introducing MoEs, we hope PDPP can learn how to handle inputs with different planning horizons to minimize the negative impact brought by joint training.

Learning objective of our diffusion model. As mentioned above, the learning objective of a standard diffusion model is the random noise added to the distribution at each diffusion step. This learning scheme has demonstrated great success in visual data synthesis area. For procedure planning, however, the distribution we need to fit contains high-level features rather than raw pixels. Predicting random noise for procedure planning results in fault training, which is similar to text generation[27]. One possible reason is that the action sequence distribution we need to predict has strong semantics. When the noise predicted in the early stage of sampling is not very accurate, the feature distribution after initial denoising steps can not have the required strong semantics, which leads to the following denoising operation deviates from correct direction and get meaningless sampling result. So we modify the learning objective to the initial input $x_{0}$, which is described in Sec. 4.1.

Our model transforms a random Gaussian noise to the final result, which has the same structure with our model input by conducting $N$ denoising steps. Since we combine the conditional information with action sequences as the data distribution, these conditional guides can be changed during the denoising process. However, the change of these conditions will bring wrong guidance for the learning process and make the conditions useless. To address this problem, we add a condition projection operation into the learning process. That is, we force the condition dimensions not changed during training and inference by assigning the initial value. The input $x$ of condition projection is either a noise-add data (Alg.1 L5) or the predicted result of model (Alg.1 L7). We use $\{\hat{c},\hat{a},\hat{o},\hat{h}\}$ to represent different dimensions in $x$, then our projection operation $\mathrm{Proj}()$ can be denoted as:

where $\hat{h}_{i}$, $\hat{c}_{i}$, $\hat{o}_{i}$ and $\hat{a}_{i}$ denote the $i^{th}$ horizon, task, observation dimensions and predicted action logits in $x$, respectively. $h$, $c$, $o_{s},o_{g}$ are the conditions. When task mask is applied, condition projection for task information is replaced with multiplying the action distribution by the task mask. For joint training with MoEs, horizon condition is used in routing gate module and there is no condition projection for horizon dimensions.

On the basis of discussions in Sec. 3.1, we further split $p(a_{1:T}|o_{s},o_{g},c,h)$ modeling process into two stages for procedure planning. Considering that the given observations are visual states around the start and end actions, we propose to predict $a_{1},a_{T}$ first, then the predicted two actions can be regarded as action conditions for planning and stay unchanged with condition projection. The two stage prediction process makes full use of the given observations and provides more guidance to the sampling process. One problem with this scheme is that the action condition used in training is the ground truth, but $a_{1},a_{T}$ predicted by model cannot be completely correct, thus the distribution difference between the training set and test set may be further amplified, resulting in overfitting problem. Experiments show that this strategy can benefit our model on large scale dataset, while performs bad for datasets with limited size, which confirms our thought.

Our training scheme contains two stages: a) training a task-classifier model $\mathcal{T}_{\phi}(c|o_{s},o_{g})$ that extracts conditional guidance from the given start and goal observations; b) leveraging the projected diffusion model to fit the target action sequence distribution.

For the first stage, we apply MLP models to predict the task class $c$ with the given $o_{s},o_{g}$. Ground truth task class labels $\overline{c}$ are used as supervision. In the second learning stage, we follow the basic training scheme for diffusion model, but change the learning objective as the initial input $x_{0}$. We denote our learnable model as $f_{\theta}(x_{n},n)$ and our training loss is:

We believe that predicting actions corresponding to the given observations are more important because they can be inferred directly from observation conditions. Thus we rewrite Eq. 7 as a weighted loss by assigning a greater weight to {$a_{1},a_{T}$} in procedure planning with a weight matrix $w$.

Besides, we add a condition projection step to our diffusion process. So given the initial input $x_{0}$ which contains action sequences, task conditions and observations, we first add noise to the input to get $x_{n}$, and then apply condition projection to ensure the guidance not changed. With $x_{n}$ and the corresponding diffusion step $n$, we calculate the denosing output $f_{\theta}(x_{n},n)$, followed by condition projection again. Finally, we compute the weighted $\mathcal{L}_{\mathrm{diff}}$ and update model, as shown in Algorithm 1. Note that the planning horizon of $x_{0}$ can vary for joint training.

At inference time, only the start observation $o_{s}$ and goal observation $o_{g}$ are provided for procedure planning. We thus first predict the task class by choosing the maximum probability value in the output of task-classifier model $\mathcal{T}_{\phi}$. Then the predicted task class $c$ is used as the task condition. To sample from the learned action sequence distribution, we start with a Gaussian noise, and iteratively conduct denoise and condition projection for $N$ times. The detailed inference process is shown in Algorithm 2.

To further accelerate the sampling process, we apply DDIM[45] to our model and thus generate output with $N^{\prime}(<N)$ sampling steps. The sampling process with DDIM is presented in Algorithm 3.

Once we get the predicted output $\hat{x}_{0}$, we take out the action sequence dimensions $[\hat{a}_{1},...,\hat{a}_{T}]$ and select the index of every maximum value in $\hat{a}_{i}(i=1,...,T)$ as the action sequence plan. Note that in the training stage of procedure planning, the class condition dimensions of $x_{0}$ are the ground truth task labels, not the output of our task-classifier as in inference.

For procedure planning, We randomly select 70% data for training and 30% for testing as previous work[6, 4, 55]. Following previous work[6, 4, 55], we extract all action sequences $\{[a_{i},...,a_{i+T-1}]\}_{i=1}^{n-T+1}$ with predicting horizon $T$ from the given video which contains $n$ actions by sliding a window of size $T$. Then for each action sequence $[a_{i},...,a_{i+T-1}]$, we choose the video clip feature at the beginning time of action $a_{i}$ and clip feature around the end time of $a_{i+T-1}$ as the start observation $o_{s}$ and goal state $o_{g}$, respectively. Both clips are 3 seconds long. For experiments conduct on CrossTask, we use two kinds of pre-extracted video features as the start and goal observations. One are the features provided in CrossTask dataset: each second of video content is encoded into a 3200-dimensional feature vector as a concatenation of the I3D, ResNet-152 and audio VGG features[17, 18, 5], which are also applied in[4, 6]. The other kind of features are generated by the encoder trained with the HowTo100M[32] dataset, as in[55]. For experiments on the other two datasets, we follow[55] to use the HowTo100M features for a fair comparison.

For VPA, we follow Patel et al.[35] to conduct our experiments on CrossTask and COIN. Ratios of training set and test set are 7/3 and 9/1 for these two datasets, respectively. Action sequences are extracted as in procedure planning, and the start observation video clip is 4 seconds long. All video features are get with the encoder trained on HowTo100M dataset. Note that our PDPP do not require video history as additional input.

Architectures. We use three backbones to implement our learnable model for projection diffusion.

$\bullet$ Unet[41]. We implement our model as a basic 3-layer Unet. As in[23], each layer in our model consists of two residual blocks[17] and one downsample or upsample operation. One residual block consists of two convolutions, each followed by a group norm[52] and Mish activation function[33]. Time embedding is produced by a fully-connected layer and added to the output of first convolution. We apply a 1d-convolution along the planning horizon dimension as the downsample/upsample operation. We set the kernel size of 1d-convolution as $2$, stride as $1$, padding as $0$ so the length change of planning horizon dimension keeps $1$ after each downsample or upsample. The hidden dimensions for each layer are 256, 512, 1024, respectively.  
$\bullet$ Unet with attention layers[20]: We implement our Unet-attention architecture by adding self-attention[50] operation to a 2-layer Unet. The hidden dimensions are 512, 1024, respectively. Follow[20], we use SiLU as activation function and set the number of attention heads as 32.  
$\bullet$ Transformer[36]: Since there is no downsample operation for transformer based model, we here stack more layers in our model to study whether attention with large number parameters can bring better results. We instantiate our transformer-based diffusion model with 12 layers and set the hidden dimension as 1024. The number of attention heads is 32. Layer norm[3] is applied to this backbone, time embedding is produced by a fully-connected layer and added with AdaLN, as in[36].

Diffusion process. For diffusion, we use the cosine noise schedule to produce the hyper-parameters $\{\beta_{n}\}_{n=1}^{N}$, which denote the ratio of Gaussian noise added to the data at each diffusion step. We set diffusion step to 200 for CrossTask and COIN dataset and 50 for NIV due to their different scales. When DDIM sampling is applied, we set the sampling step number to 10 for faster inference.

Training. We use the linear warm-up training scheme to optimize our model and train different steps for the three datasets, corresponding to their scales. In CrossTask_Base, we set the diffusion step as $200$ and train a Unet based model for $12,000$ steps with learning rate increasing linearly to $8\times 10^{-4}$ in the first $4,000$ steps. Then the learning rate decays to $4\times 10^{-4}$ at step $10,000$. In CrossTask_How, we keep diffusion step as 200 and train our Unet based model for $24,000$ steps with learning rate increasing linearly to $5\times 10^{-4}$ in the first $4,000$ steps and decays by 0.5 at step $10,000,16,000$ and $22,000$. When joint training is applied to CrossTask_How, only $12,000$ training steps are required since gradient descent is conducted on multiple batches with different horizons in one training step. In NIV, the diffusion step is $50$ and we train a Unet based model for $6,500$ steps due to the small size of this dataset. The learning rate increases linearly to $3\times 10^{-4}$ in the first $4,500$ steps and decays by $0.5$ at step $6,000$. We train a Unet-attention model on the COIN dataset with task mask and two stage prediction. We set diffusion step as $200$ and train our model for $14,000$ steps.The learning rate increases linearly to $1\times 10^{-4}$ in the first $4,000$ steps, then we keep learning rate unchanged for the remaining training steps. The training batch size for all experiments is $256$. We set $w=10$ for the weighted loss. All our experiments are conducted with ADAM[24] on 8 NVIDIA TITAN XP GPUs.

The first stage of our learning is to predict the task class with the given start and goal observations. We implement this with MLP models. The classification results for different planning horizons on three datasets are shown in Tab. I. We can see that our classifier can perfectly figure out the task class in the NIV dataset since only 5 tasks are involved. For larger datasets CrossTask and COIN, our model can make right predictions most of the time.

To further study the effectiveness and performance of our PDPP, we in this section conduct detailed ablation studies. We start with the basic setting that we use Unet as backbone and train our model separately. Task condition is added by concatenating task feature in the conditional action sequence input. We mainly focus on the ablation study on CrossTask with better HowTo100M features.

In this section, We follow previous work[55] and compare our approach with other alternative methods for procedure planning on the three datasets, across multiple prediction horizons.

All the results suggest that our model performs well across datasets with different scales.

We further analyze failure cases where PDPP makes wrong predictions on the largest and most comprehensive COIN dataset. Specifically, we calculate the proportions of the following three types of errors: task prediction error (task error), start-end actions prediction error (action error), and intermediate action prediction error (intermediate error). The results are presented in Tab. XV. Visualization results are shown in Fig. 5. It can be seen that the primary cause of failure cases is start-end actions prediction error, indicating that inferring semantic action information from given observations is the key bottleneck to be improved. Model needs to incorporate finer details observed, such as the lower hammer and seasoning bottle in Fig. 5 “Action error”, to make more accurate predictions. For task predicting error, an imbalance in the distribution of task categories within the training set makes certain tasks difficult to recognize accurately (e.g. very few training samples of task “Resize Watch Band”). Besides, similarities between certain tasks give rise to errors in prediction due to their overlapping nature (e.g. “Make Cookie” and “Make Chocolate” share similar steps, “Change Battery Of Watch” and “Resize Watch Band” share similar appearances). Intermediate errors are caused by two reasons: the uncertainty of procedure planning and the gap between training and inference. In the example of Fig. 5, all the ingredients in the test video have already been cut, but the model has encountered many examples during training where ingredients are chopped and then added into the bowl, so it predicts the action of “cut ingredients”.

The computational efficiency of PDPP is provided in Tab. XVI, including the number of training parameters, FLOPs and time required for a single sampling of PDPP across different datasets. We have also studied the effect of joint training on model parameter savings and training cost. Experimental results demonstrate that joint training effectively saves model training parameters and training steps when planning step $T>1$, which also enables flexible multi-step procedure predicting. All the experimental results are obtained on a single NVIDIA TITAN XP GPU.

As discussed in Sec. 1, we introduce diffusion model to procedure planning to model the uncertainty in this problem. Here we follow[55] to evaluate our probabilistic modeling.

Our model is probabilistic by starting from random noises and denoising step by step. We here introduce two baselines to compare with our diffusion based approach. We first remove the diffusion process in our method to establish the Noise baseline, which just samples from a random noise with the given observations and task class condition in one shot. Then we further establish the Deterministic baseline by setting the start distribution $\hat{x}_{N}=0$, thus the model directly predicts a certain result with the given conditions. For the Deterministic baseline, we just sample once to get the plan since the result for Deterministic is certain when observations and task class conditions are given. For the Noise baseline and our diffusion based model, we sample 1500 action sequences as our probabilistic result to calculate the uncertain metrics. Note that the multiple sampling process is only required while evaluating probabilistic modeling and our model can generate a good plan just by sampling once.

We reproduce the KL divergence, NLL, ModeRec and ModePrec in[55] and use these metrics along with SR to evaluate our probabilistic model. Specifically, the SR and ModePrec can reflect the accuracy of planning results, ModeRec reflects the diversity of plans, and KL divergence, NLL indicate the overall agreement between predictions and the ground truth distribution.

We in this section apply our PDPP to the Visual Planners for human Assistance (VPA) problem by removing the goal observation and using the ground truth task label as condition instead. The input of VPA task includes the current observation, the goal description, and the video history. The video history is an additional input (not required by procedure planning) that records the entire process from the beginning to the current state. Although it can provide some contextual information, it is computationally expensive and difficult to achieve real-time planning in the existing frameworks. Therefore, we choose to only use the current observation as starting point to predict action sequence to reach the goal, which is more convenient and efficient. We follow Patel et al.[35] to evaluate our model on CrossTask and COIN datasets. For CrossTask, we use the Unet based model and add both the task and horizon conditions by concatenation. For COIN, we apply the Unet-attention model with task mask and horizon concatenation.

Take the start time of the first action to be predicted as $t_{start}$, Patel et al.[35] gets their start observation $o_{s}$ by capturing the video clip from time $t_{start}-2$ to $t_{start}+2$, which we denote as ”protocol 1”. However, we here argue that since the aim for VPA problem is to assist human in everyday lives, taking video clip from time $t_{start}$ to $t_{start}+2$ is not suitable because actions after $t_{start}$ are inaccessible and have not been performed. Thus we select 4 seconds-long video content before $t_{start}$ as $o_{s}$, which we denote as ”protocol 2”.

We compare our model with all approaches in[35]:

Tab. XX presents the experiment results for VPA. Our PDPP achieves the state-of-the-art performance, even without the video history. This again shows the great generalization ability of our PDPP on goal-directed planning in instructional videos and the effectiveness of modeling action sequence as a whole.