Scene-aware Human Pose Generation using Transformer

The concept of affordance (Gibson, 1979) is first proposed by ecological psychologist James Gibson, which describes “opportunities for interactions” of environment. Recently, affordance learning has included extensive applications, including contextual affordance learning (Lopes et al., 2007; Ugur et al., 2011), functionality understanding (Zhu et al., 2015; Shiraki et al., 2014), affordance classification (Varadarajan and Vincze, 2013; Ugur et al., 2014), affordance detection (Grabner et al., 2011; Moldovan and De Raedt, 2014), affordance segmentation (Eigen and Fergus, 2015; Roy and Todorovic, 2016), etc. Our work is more related to contextual affordance learning, which employs affordance relationships as context for associated tasks. For examples, Lopes et al. (Lopes and Santos-Victor, 2005) proposed to use contextual affordance as prior information to enhance gesture recognition and decrease ambiguities according to motor terms. Castellini et al. (Castellini et al., 2011) employed affordance as visual features and motor features to benefit the object recognition task. Gupta et al. (Gupta et al., 2011) learned affordance in indoor images to detect the workspace according to human poses. Recently, Wang et al.(Wang et al., 2017) adopted the scene affordance as context for human pose generation. In this paper, we also focus on generating a scene-aware human pose as in (Wang et al., 2017).

Human could be considered as a specific object, and thus our task is also related to the field of object placement, which is an important subtask of image composition (Niu et al., 2021). Early methods (Tan et al., 2018; Tripathi et al., 2019; Li et al., 2019; Zhang et al., 2020a, b) mainly depended on explicit rules for placing foreground objects in background images, while recent methods (Liu et al., 2021; Zhang et al., 2020a; Tripathi et al., 2019; Zhou et al., 2022) employed deep learning for object placement. To name a few, Lin et al. (Lin et al., 2018b) proposed to use spatial transformer networks to learn geometric corrections to warp composite images to appropriate layouts. PlaceNet (Zhang et al., 2020a) predicted a diverse distribution for common sense locations to place the given foreground object on the background scene. TERSE (Tripathi et al., 2019) proposed to train synthesizer and target networks in adversarial manner for plausible object placement. Lee et al. (Lee et al., 2018) employed a two-stage pipeline to determine where to place objects and what classes to place. Azadi et al. (Azadi et al., 2020) proposed a self-consistent composition-by-decomposition network to learn to place objects, based on the insight that a successful composite image could be decomposed back into individual objects. Most recently, Zhou et al. (Zhou et al., 2022) treated object placement as a graph completion problem and proposed a graph completion module to solve the task. The work (Niu et al., 2022) proposed to identify the reasonable placements using a discriminative approach, by predicting the rationality scores of all scales and locations. In our task, the model needs to understand the scene and the human-scene interaction, and generate reasonable human poses from a large searching space, which is a very challenging task.

Human pose estimation and generation has extensive applications and thus has attracted wide attention in decades. As for human pose estimation, conventional methods (Dantone et al., 2013; Sapp et al., 2010; Sun et al., 2012; Wang and Li, 2013; Wang and Mori, 2008) employed probabilistic graphical models or the pictorial structure models to represent the relations between human body parts. Recent human pose estimation methods (Artacho and Savakis, 2020; Cheng et al., 2020; Lin et al., 2018a; Su et al., 2019; Tang et al., 2019; Li et al., 2021) have achieved significant progress by using deep learning models. Technically, there are roughly two groups of pipelines for pose estimation: regression-based (Toshev and Szegedy, 2014; Carreira et al., 2016; Sun et al., 2017; Luvizon et al., 2019) and heatmap-based (Wei et al., 2016; Chu et al., 2017; Newell et al., 2016; Martinez et al., 2017). As for human pose generation, recent works generate human pose conditioned on past poses in video (Walker et al., 2017), textual description (Hong et al., 2022), or video motion (Rempe et al., 2021). Except for (Wang et al., 2017), there are few works generating human poses skeletons conditioned on 2D scene images.

Considering that the definition of affordance learning in our task is how a hypothetical human could interact with the scene, we construct a number of representative pose templates in training set to facilitate the interaction modeling between the hypothetical human and scene image. Specifically, we first normalize all ground-truth (GT) training poses. Then, we apply K-means clustering and further select $K$ representative cluster centers as $K$ pose templates $\{\textbf{T}_{i}\}_{i=1}^{K}$ by considering poses as $2M$-dim vectors and using Euclidean distance as distance metric ($K=14$ by default).

For pose normalization, we define the box centered at $(0,0)$ with side length $1$ as the unit box, denoted as $\textbf{B}_{u}$. We also define a function $\textbf{B}_{e}(\cdot)$ to output the enclosing box of its input pose, i.e., by fetching the maximum and minimum x,y-coordinates of keypoints in the input pose. For each GT training pose $\textbf{P}^{*}$, we compute its normalized pose by

where $\mathcal{D}_{\textbf{B}_{x}\rightarrow\textbf{B}_{y}}(\cdot)$ is a linear deformation function defined by mapping box $\textbf{B}_{x}$ to box $\textbf{B}_{y}$ and could be used to deform the keypoints of input pose. In this way, each normalized pose will have the unit box $\textbf{B}_{u}$ as its enclosing box, i.e., $\textbf{T}_{i}\in[-0.5,0.5]^{2M}$. In our method, these pose templates will be refined to affordance pose according to the interaction with scene image.

When the scene image $\textbf{I}_{full}$ and target point o are given, we prepare three forms of input images to leverage multi-scale information following (Wang et al., 2017). As shown in Figure 2, the first input image $\textbf{I}_{in_{1}}$ is the whole image offering global context of scene image. The second input image $\textbf{I}_{in_{2}}$ is a square patch centered at o and the side length is the height of the scene image, which captures the scene context around the target location. The final input image $\textbf{I}_{in_{3}}$ is also a square patch but in half height to provide high-resolution information of scene context. We re-scale three images to the same size $H_{in}\times W_{in}$, which is set as $H_{in}=W_{in}=224$ in implementation. For each sample, the ground-truth label consists of three parts: class $i^{*}$, scale $\textbf{s}^{*}$, and normalized pose $\mathcal{N}(\textbf{P}^{*})$. Class $i^{*}$ is the index of nearest pose template. We calculate the scale $\textbf{s}^{*}$ as the actual height and width divided by the actual height of $\textbf{I}_{full}$.

As shown in Figure 2, our network consists of four modules: 1) a foundation network module, which generates feature map of the input scene image and predicts the compatibility score of each pose template; 2) a scale transformer module, which contains several transformer decoder layers as in (Carion et al., 2020; Cheng et al., 2021) and an MLP prediction head; 3) an offset transformer module, which is similar to scale transformer module in architecture; 4) a knowledge distillation module, which is a pre-trained offset regression network.

Foundation network module. Foundation network module consists of a backbone network, an upsampling network, and a classifier. We employ ResNet-$18$ (He et al., 2016) as our backbone network. Because the spatial resolution of ResNet-$18$ output ($7\times 7$) is not enough for fine-grained context, we append it with an upsampling module to yield relatively high-resolution feature map. Specifically, similar to the pixel decoder in MaskFormer (Cheng et al., 2021), we gradually upsample the feature map and sum it with the intermediate feature map of the corresponding resolution from the backbone network. As aforementioned, for each full scene image, we have three forms of input image $\textbf{I}_{in_{1}}$, $\textbf{I}_{in_{2}}$, and $\textbf{I}_{in_{3}}$, leading to three feature maps $\textbf{F}_{in_{1}}$, $\textbf{F}_{in_{2}}$, and $\textbf{F}_{in_{3}}$ after the foundation network module. Finally, three feature maps are concatenated and squeezed via $1\times 1$ convolutional layer to produce the scene feature map $\textbf{F}\in\mathbb{R}^{C\times H_{f}\times W_{f}}$, in which $H_{f}=W_{f}=28$ in our implementation. In this way, the scene feature map contains coarse context, fine context, and global context, which provide complementary information for later scene-aware human pose generation. The feature map will pass a global pooling layer and then pass a full-connected layer to get the binary compatibility score for each of $K$ pose templates.

Scale transformer module. Given the feature map of scene image F and each pose template $\textbf{T}_{i}$, we use transformer module to model the interaction between pose templates and scene, which can be formulated as

where $\mathcal{T}_{s}(query,key,value)$ is a transformer module as in (Carion et al., 2020; Cheng et al., 2021) and $\textbf{u}_{i}\in\mathbb{R}^{d}$ is the interacted embedding decoded from the $i$-th template embedding $\textbf{q}_{i}$. Specifically, the transformer module consists of $3$ transformer decoder layers. In the transformer module, the embedding of each pose template interacts with each pixel of scene feature map, which facilitates human pose scale prediction through the global context of the whole image. We then use an MLP to predict the scale $\textbf{s}_{i}\in\mathbb{R}^{2}$ (two scale values along the x-axis and y-axis respectively) of each pose template based on the interacted embedding $\textbf{u}_{i}$.

Offset transformer module. After getting the predicted scale, we compute the enclosing human bounding box and pay more attention to the features within the human bounding box. Specifically, we crop the feature map of the enclosing box region and resize it to the size of original feature map ($28\times 28$) using a ROIAlign layer. The resized feature map is then used as the key and value of the transformer module, which can be formulated as

where $\textbf{F}^{\prime}$ is the resized feature map within human bounding box. The transformer module $\mathcal{T}_{p}$ consists of $1$ transformer decoder layer. Then we use a MLP to predict the coordinate offsets $\bm{\Delta}_{i}\in\mathbb{R}^{2M}$.

Next we will show how to get the actual output pose prediction given a scene image. As shown in Figure 3, the relative coordinate $-0.5$ (resp., $0.5$) on $x$-axis corresponds to the left (resp., right) boundary of input image $\textbf{I}_{in_{2}}$. Similarly, the relative coordinate $-0.5$ (resp., $0.5$) on $y$-axis corresponds to the bottom (resp., top) boundary of input image $\textbf{I}_{in_{2}}$. We first add the offsets to the pose template (i.e., $\textbf{T}_{i}+\bm{\Delta}_{i}$) to obtain the raw affordance pose. Then, we normalize this pose by $\mathcal{N}(\textbf{T}_{i}+\bm{\Delta}_{i})$ (see Eq  1) to ensure that the center of predicted affordance pose is at the target point o. Finally, we scale the pose with $\mathcal{N}(\textbf{T}_{i}+\bm{\Delta}_{i})\cdot\textbf{S}_{i}$, where $\textbf{S}_{i}\in\mathbb{R}^{2M}$ denotes the scales of all $M$ keypoints and all $M$ keypoints share the same $x$,$y$-scale value $\mathbf{s}_{i}\in\mathbb{R}^{2}$. As for the range of offset prediction, we assert that any offset for normalized pose should not exceed a certain range, which is set to the size of normalized box, so we define the offset $\bm{\Delta}_{i}\in[-0.5,0.5]^{2M}$ for normalized pose. We also observe that the max height of ground-truth pose does not exceed twice the image height, so we set the scale $\textbf{s}_{i}\in[0,2]^{2}$ to ensure that the final predicted pose is in range of $[-1,1]^{2M}$. Specifically, we refine each pose template to get the final pose by

Distillation module We introduce a distillation module to improve the performance of offset prediction. Because our model can be seen as a multi-task model, introducing a teacher model $M$ that focuses on predicting offsets given the scale and pose template can effectively help our model search for optimal solutions in a smaller range and converge faster. Specifically, we employ ResNet-$18$ (He et al., 2016) as our teacher network. During the pre-training phase, we first use ground-truth scale $\textbf{s}^{*}$ to derive the human bounding box, then we can fit the ground-truth pose template $\textbf{T}_{i^{*}}$ into the bounding box and get the heatmap of each keypoint. Finally, we concatenate the heatmaps and the scene image $\textbf{I}_{in_{2}}$ as the model input. The teacher model predicts an offset vector, based on which we can get the corresponding predicted pose and use $L_{2}$ loss to supervise the model during pre-training.

During the training phase of our model, the keypoint heatmaps are generated by the ground-truth pose template $\textbf{T}_{i^{*}}$ and predicted scale from scale transformer module $\textbf{s}_{i^{*}}$. Provided with keypoint heatmaps, we use fixed pre-trained teacher model to extract the feature vector $\textbf{v}_{i^{*}}$. Recall that in our main network, offset transformer embedding $\textbf{u}^{\prime}_{i^{*}}$ accounts for offset prediction. Therefore, we realize knowledge distillation by narrowing the $L_{2}$ distance between offset transformer embedding $\textbf{u}^{\prime}_{i^{*}}$ and $\textbf{v}_{i^{*}}$.

In existing dataset Sitcom, each training sample is only annotated with one ground-truth pose, corresponding to one pose template. However, other pose templates might also be plausible for this scene image. Therefore, we employ self-training to mine more reasonable human poses in a scene. Specifically, we design a multi-stage training strategy. In the first stage, we assign each sample its nearest pose template as its ground-truth class and train our model. After that, we change the label to positive if the trained model of the previous stage predicts a compatibility score higher than a threshold (i.e., $0.7$). The training moves on to the next stage when the test classification accuracy begins to decrease, and stops when the labels no longer change. We apply binary cross-entropy loss to supervise the predicted compatibility scores:

where $c_{i}$ is the compatibility score for the $i$-th pose template, $l_{i}$ is the label of the $i$-th pose template ($l_{i}=1$ if the $i$-th pose template is positive), $w_{i}$ is the weight to balance the quantity differences between positive and negative categories of the $i$-th pose template in dataset, and $BCE$ is the binary cross-entropy loss.

Though multiple pose templates could be classified as positive, the ground-truth poses and pose templates are unique for each sample. Therefore, we only apply the refinement loss to the ground-truth pose template. The loss functions for the two refinement (scale and offsets) predictions could be formulated as

We apply the distillation loss to the output embedding of ground-truth pose template in offset transformer module, which can be formulated as

where $\textbf{u}^{\prime}_{i^{*}}$ is the output embedding of offset transformer decoder, $\textbf{v}_{i^{*}}$ is the output embedding of teacher model.

Besides the single ground-truth pose template, there could be other positive pose templates. We apply an adversarial loss to these templates to ensure the quality of output poses. We use $Q_{i}=[\mathcal{N}(\textbf{T}_{i}+\bm{\Delta}_{i}),\textbf{s}_{i},\textbf{f})]$ to denote the concatenation of three vectors, in which f is the feature vector after average pooling layer in foundation network module. $Q_{i}$ is fed into the discriminator $D$ as input. The adversarial loss can be formulated as

where the discriminator $D$ is an MLP and predicts whether the input is from ground-truth template, $\tilde{i}$ is the index $i$ which satisfies $l_{i}=1$ and $i\neq i^{*}$.

Therefore, the total training objective of our model is

where $\theta_{D}$ includes the model parameters of discriminator $D$ and $\theta_{G}$ includes the other model parameters. $\lambda_{\text{o}}$, $\lambda_{\text{s}}$, $\lambda_{\text{adv}}$, and $\lambda_{\text{dis}}$ are hyper-parameters for balancing loss items. In our experiments, we set $\lambda_{\text{o}}=10$, $\lambda_{\text{s}}=10$, $\lambda_{\text{adv}}=10$, and $\lambda_{\text{dis}}=1$ via cross-validation.

In the inference stage, we sort the predicted poses $\{(c_{i},\bm{\Delta}_{i},\textbf{s}_{i})\}_{i}^{K}$ according to their compatibility scores $c_{i}$, and select the poses with top $k$ compatibility scores as our final prediction.

We conduct experiments on Sitcom affordance dataset (Wang et al., 2017). The training samples are collected from the TV series of “How I Met Your Mother”, “The Big Bang Theory”, “Two and A Half Man”, “Everyone Loves Raymond”, “Frasier” and “Seinfeld”, consisting of $25$k accurate poses over $10$k different scenes. The test samples are collected from “Friends”, consisting of $3$k accurate poses over $1$k different scenes. Each scene image is annotated with one or more target location points and each target point has one annotated ground-truth pose with $16$ keypoints.

We use three evaluation metrics in our experiments, i.e., PCK (Percentage of Correct Keypoints), MSE (mean-square error), and user study.

PCK is a common metric used in pose estimation (Artacho and Savakis, 2020; Li et al., 2021; Cheng et al., 2020), which indicates the percentage of correctly predicted keypoints that fall within a certain distance threshold of the ground-truth. In our task, we choose [email protected] as in (Artacho and Savakis, 2020), which refers to a threshold of $20\%$ of the torso diameter (the distance between left hip and right shoulder, corresponding to the $3$-rd point and the $12$-th point in Sitcom dataset).

MSE measures the mean-square error between the predicted pose and GT pose. Specifically, we compute the mean-square error using the normalized coordinates which are divided by the heights of corresponding images, considering that the sizes of testing images may be quite different.

User study is conducted with $50$ voluntary participants by comparing the poses generated by different methods. For each test sample, every participant chooses the method producing the most reasonable pose. The score of each method is defined as the frequency that this method is chosen as the best one.

For the network architecture, we use ResNet-$18$ (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) as our backbone network and follow the setting of transformer module in (Cheng et al., 2021) for the two transformer modules. For the MLP settings in the prediction head of two transformer modules, the first $2$ hidden layers in pose offset and scale prediction head have hidden dimension of $256$ with ReLU activate function. The last layer of offset prediction is followed by a tanh function, while the last layer of scale prediction is followed by a sigmoid function. During the training stage, we apply SGD optimizer with learning rate $10^{-4}$ and weight decay $10^{-4}$. A learning rate multiplier of $0.1$ is applied to backbone network and $1.0$ is applied to other modules of model. The batch size is set to $8$. For the system environment, we use Python $3.7$ and Pytorch $1.8.0$ (Paszke et al., 2017). We conduct experiments on Ubuntu $18.04$ with Intel(R) Xeon(R) CPU $E5-2678$ v$3$ @ $2.50$GHz CPU and four NVIDIA GeForce GTX $1080$ Ti GPUs. The random seed is set as $0$ for all experiments unless stated otherwise.

To provide in-depth analyses on how our model performs, we visualize the top-$3$ confident pose templates and the corresponding generated poses in Figure 4. From the visualizations, we could see that our model could not only choose pose templates suitable for the input scene image but also refine them to reasonable affordance poses. Taking the first row as example, our model produces high compatibility scores for all sitting pose templates, and then our model refines it to an affordance pose naturally sitting on the sofa. More qualitative analyses are left to Supplementary.

Baseline Setting. We compare with the following representative baselines: two simple baselines, Wang et al. (Wang et al., 2017), pose estimation methods, and object placement methods.

1) Heatmap is a basic method producing keypoint heatmap. Specifically, we concatenate three feature maps $\textbf{F}_{in_{1}}$, $\textbf{F}_{in_{2}}$, and $\textbf{F}_{in_{3}}$ after the ResNet-$18$ (He et al., 2016) backbone network and upsampling module with upsampled resolution of $56\times 56$. Then, the concatenated feature maps are fed into a $1\times 1$ convolution layer to produce an $M$-channel keypoint heatmap.

2) Regression is also a intuitive method directly predicting the coordinates of human pose joints. We perform spatial average pooling on the concatenated feature maps which are the same as that in Heatmap baseline and send the pooled feature to a fully-connected layer to produce a $2M$-dim vector.

3) Wang et al. (Wang et al., 2017) is a two-stage approach for scene-aware human pose generation based on pose templates. In the first stage, it classifies the scene into one of $14$ pose templates which are the same as ours by sending the concatenated feature maps to a spatial average pooling layer and a linear classifier. In the second stage, it employs VAE (Kingma and Welling, 2014) to produce the deformations based on latent code, one-hot pose label vector, and the concatenated feature maps.

4) Unipose (Artacho and Savakis, 2020) and PRTR (Li et al., 2021) belong to pose estimation methods. Unipose (Artacho and Savakis, 2020) is a unified framework based on ”Waterfall” Atrous Spatial Pooling architecture. PRTR (Li et al., 2021) is the first transformer-based model used in pose estimation tasks, which regards query embedding as keypoint hypothesis and generates the final pose by finding a match.

5) PlaceNet (Zhang et al., 2020a) and GracoNet (Zhou et al., 2022) are two state-of-the-art methods in object placement task. To adapt the model to our task, we remove the encoder of foreground in these models and replace foreground feature vector with a learnable embedding.

In these methods, we use the same backbone as ours to extract the image feature map.

Quantitative Comparison. We first conduct quantitative comparison to precisely measure the discrepancies to GT poses in test set. Specifically, in our method and Wang et al. (Wang et al., 2017), we obtain top-$k$ confident predicted poses with $k$ highest compatibility scores and calculate their PCK/MSE, after which the predicted pose achieving the optimal PCK/MSE is used for evaluation. in PlaceNet(Zhang et al., 2020a) and GracoNet (Zhou et al., 2022), there are only random vectors for generate multiple results, we sample $k$ times to get $k$ results and choose the predicted pose achieving the optimal PCK/MSE for evaluation. Considering that the other baselines can not produce multiple confident predicted poses, we only report their top-$1$ metrics. We summarize all the results in Table 1. Firstly, directly applying the simple methods (i.e., Heatmap and Regression) could only achieve barely satisfactory performances. PlaceNet (Zhang et al., 2020a) gets an inferior result because there is no reconstruction loss between generated pose and ground-truth pose during training. The performances of GracoNet (Zhou et al., 2022) and Unipose (Artacho and Savakis, 2020) are hardly improved compared to Regression, indicating that these methods may need further improvement to fit this task. Wang et al. (Wang et al., 2017) and PRTR (Li et al., 2021) could produce better results than baselines above, which proves the feasibility of using transformer and template-based method in this task respectively. Finally, our method outperforms all the baseline methods especially in the top-$3$ and top-$5$ prediction.

Qualitative Comparison. Secondly, we conduct qualitative comparison to compare the visual rationality of various methods in different scenes. We present representative baselines, including Heatmap, Regression, PRTR (Li et al., 2021), Wang et al. (Wang et al., 2017), and GracoNet (Zhou et al., 2022). We show top-$3$ poses for the methods which can produce multiple results. As shown in Figure 5, our method could produce overall better affordance poses in various scene images. Specifically, Heatmap, Regression baselines, and GracoNet(Zhou et al., 2022) are prone to produce a standing human pose. One possible reason is that standing pose is the most common case, so they lack explicit knowledge about various human poses and tend to produce the average pose. PRTR (Li et al., 2021) is better to some extent, but the predicted poses often lack integrity and rationality, probably focusing more attention on the location of single keypoint. Wang et al. (Wang et al., 2017) outperforms PRTR (Li et al., 2021) and GracoNet (Zhou et al., 2022), producing more reasonable human poses due to representative pose templates. Our model could further generate more diverse and plausible results. More qualitative comparisons are left to Supplementary.

The pose templates play a pivotal role in our method. In order to investigate its impacts and characteristics, we conduct a series of experiments. We first group all annotated training poses into $10$, $20$, $30$, $40$ clusters using K-means algorithm respectively. Then, based on $30$ clusters, we further select $20$, $14$, $10$, $7$ representative clusters. We use $K^{\prime}$ to denote the number of templates derived by K-means and $K$ to denote the number of templates derived by selection. We summarize the results of various values of $K^{\prime}$ in Table 2, from which we could observe that as $K^{\prime}$ increases, the performance first increases and then decreases. The cause of this may be that too few pose templates increases the refinement work of the model, while an excessive number of pose templates renders the classification process more challenging. In manual selection cases, the performance first increases as $K$ decreases from $20$ to $14$, the decrease of $K$ after $14$ will lead to a decrease in model performance. It is probably because that the templates are redundant when $K$ is greater than $14$, while decreasing $K$ after $14$ will lead to the loss of semantic categories and the rest categories could not cover all data. With an equivalent number of templates, manual selection yields better performance. We infer that it is probably because that the results from K-means algorithm lack semantic information, in which some pose templates share highly similar semantics.

Utility of Different Modules. As introduced in Section 3.3, we use two cascaded transformer modules to predict scale and offsets in sequence and predict the compatibility score based on global feature vector. Additionally, we incorporate knowledge distillation and self-training to facilitate learning. To validate the effectiveness of these modules, we conduct the following experiments using different degraded versions of our model. Firstly, we use a single transformer to perform all the classification, scale and offset prediction tasks (row 1). Then, we decouple the classification task and predict the compatibility score based on the feature vector (row 2). We further employ self-training for the classification task (row 3). Next, we introduce a new transformer with local feature map of human bounding box as input to predict offsets (row 4). Finally, we add knowledge distillation to help predict the offsets (row 5). The results are shown in Table 3, from which we could observe that each change results in a performance improvement in the model.

Utility of Different Loss Functions. We conduct experiments on different loss functions. The results are shown in Table 4. We observe that each loss item can improve the performance respectively. Generally, each loss makes up an important part, and they work together to guarantee the effectiveness of our method.