Unposed: Unsupervised Pose Estimation based Product Image Recommendations

Clothing product pages tend to show images of human models / mannequins exhibiting the clothing / accessories from different poses / directions. However, many products on the webstores may suffer from poor cataloguing practices like duplicated images, bad image lighting, poor background, insufficient number of angles of clothing, etc. leading to poor customer experience. A few A/B experiments done at Amazon have confirmed this and have shown a non-trivial positive impact of having larger number and varied type of images.

The problem of inadequately imaged product, thus has a user experience improvement incentive. Any automation that can meaningfully nudge the selling partners with right suggestions for improving their catalogue images will help them better their listing. This in turn can improve the end user experience, there by driving the virtuous cycle.

The missing images can be indicated to the sellers in text hints, or they can be provided reference image set of popular products in the Category/ Subcategory corresponding to their underrepresented product in the clothing category. For example, Figure 2 shows the reference imageset with 6 images, covering front, farview, neckview, back, fabric pattern, side view etc. whereas, the subject product image are missing a few from the reference set. In this work, we discuss ways to detect, and indicate the missing images to improve the product listing by detecting missing poses in the image set.

The product listing image selection is largely done manually by professionals specializing in product listing adhering to Standard Operating Procdeures based on listing guidelines. Some prior published work (Chaudhuri et al., 2018) suggests solutions to build systems to solve image listing problems like de-duplication, image cropping etc.

In our scheme we suggest what images can be added to the Product imageset to enhance the user experience, and no prior work in our knowledge discusses pose detection (discussed next $\S$2.2) as a method to generate candidates for image addition.

Human pose detection has been a widely studied computer vision application. The core problem of human pose detection is to identify the body landmarks or regions/points of interests (PoI) like nose, shoulder, knee etc. and predict coordinates of these landmarks w.r.t. the given image. A few prominent models use HourGlass(Newell et al., 2016) and U-NET(Tang et al., 2018) type of Encoder-Decoder NN architecture to identify Points of Interest.

Then there are multistage approaches where in the Bottoms-Up approach the model first detects body joints location on the image and then links them to guess complete human instance. In Top-down approach models (Li et al., 2019), in the first stage the model identifies the human body in the image, and in subsequent stages identifies landmarks. The current state-of-the-art model is based on Fully Convolutional Networks(Bulat et al., 2020), with PCKh (87%) on the MPII dataset (Andriluka et al., 2014). We use pretrained MMPose (Contributors, 2020) toolbox library which implements HRNet full body keypoints detection model (Sun et al., 2019) and Blazepose(Bazarevsky et al., 2020), which is a lightweight implementation of HourGlass style network architecture to generate position embeddings in our pipeline.

We built the reference imageset by selecting top-K product listings by popularity belonging to Clothing sorted by number of customer average review ratings and number of reviews. This sorting criteria is easily configurable based on available signals.

The reference set images selected like described in the previous section are then passed through a data pipeline (Figure 3) which

1. (1)

   Reads the image.

2. (2)

   Estimates the coordinates of various human body landmarks like nose, shoulders, knees, wrists, elbows etc. w.r.t. the image.

3. (3)

   Normalizes each of these coordinates w.r.t. to image size.

4. (4)

   Creates a vector representing one image from the imageset using the normalized coordinates.

5. (5)

   Creates clusters of the vector representation of all the images using K-Means algorithm to detect K centroids, where each centroid corresponds to cluster of similar poses detected in all the images in the train imageset.

6. (6)

   Ranks all the centroids for each cohort grouped by attributes like category, subcategory, product type based on occurrence in most popular / best rated products to determine importance of each centroid within the cohort.

For a given product, all the images are collected as a subject product imageset. To detect the missing images from this subject imageset w.r.t. reference imageset:

1. (1)

   The model first generates the landmarks / pose embeddings coordinates for all images using the same estimation method as reference dataset.

2. (2)

   Then after normalizing these coordinates w.r.t. to image size, the vector embeddings are created.

3. (3)

   For each image, the model then finds the nearest centroid corresponding to approximate closest pose w.r.t. reference poses detected in the previous section.

4. (4)

   Further each centroid for which there were no images in the subject imageset, is ranked as per popularity criteria defined while training the reference set. Then these missing poses are flagged in the decreasing order of popularity or usefulness as illustrated in Figure 6.

After finding the indices / labels corresponding to missing centroid in the inference flow mentioned in previous section, the user can refer the reference imageset to find out the reference images corresponding to these centroids for a given Product.

Given a set $\mathbf{Q_{D}}=\{q_{1}^{D},q_{2}^{D}...,q_{n}^{D}\}$ of $n$ images for a subject product $\mathbf{D}$ in imageset and a universal set $\mathbf{U}=\{u_{1},u_{2},..u_{m}\}$ of all the $m$ images of Top-K products in the training product imagesets, the problem is to determine a set $\mathbf{T}$ of tags, $1\leq i\leq o$, corresponding to $\mathbf{o}$ optimal clusters $C_{1},C_{2},...,C_{o}$ for images from $\mathbf{U}$ and further find a set $t_{j}\subseteq\mathbf{T}$, $1\leq j\leq o$ of tags, corresponding to images in subject imageset $q_{t_{j}}^{D}\mid q_{t_{j}}^{D}:q_{t_{j}}^{D}\not\in\mathbf{Q_{D}},1\leq j\leq o$.

To calculate these we first determine the ideal $\mathbf{o}$-clusters from the images in $\mathbf{U}$ representing all images of all Product in reference training sets. Let {$x_{1},x_{2},...,x_{G}\}\in\mathbf{X_{v}}$ be the vector representation of any image in $\mathbf{U}$, then we determine $\mu_{o}$ vectors representations of these $\mathbf{o}$-clusters such that we minimize:

Here the vector $\mathbf{X_{v}}$ is created by using normalized position coordinates of Point of Interests (PoI) from Pose Detection.

Later we determine all the tags $l_{j}$ so as to maximize the probability $p(q_{i}\mid\mathbf{X_{v}})$ of image $I$ to belong to the cluster given by tag $l_{j}\in\mathbf{T}$ such that

Further we determine all tags $t_{j}$’s such that, all tags that are in T but not are detected in the previous step.

These $t_{j}$’s will be the missing poses / centroid in the subject imageset w.r.t. $\mathbf{o}$ centroids defined from reference imageset.

We aggregated public data from popular e-commerce website’s clothing products metadata and listing images, and selected top 3000 most popular products with available image count per product ¿ 9 based on customer ratings, etc. This collection provided us with an imageset of ~20K for training. For evaluation purpose, we selected unseen products from the similar selection criteria collection of top products, but such that images per product imageset were ¡ 5.

For experimentation, the Pose Landmark coordinates are estimated using Blazepose pretrained implementation of HourGlass style Region detection NN library as well as MMPose toolkit pretrained implementation of HRNet. The implementation Blazepose (Bazarevsky et al., 2020) (MMPose (Contributors, 2020)) generates 3D coordinates for 33 (HRNet 2D: 17) landmarks points as depicted in Figure 4. These coordinates are normalized w.r.t. images dimensions. The pose landmarks are cartesian pixel coordinates w.r.t. to image top left.

The embedding vectors are compiled using landmark coordinates w.r.t. to image size. To add more pose related information, the embeddings are extended by taking various ratios of x, y coordinates of landmarks to detect the front, side, backview, e.g. $\tfrac{\mathbf{x}_{left\ shoulder}}{\mathbf{x}_{right\ shoulder}}$. Further the z components (available only for 3D-model) help identify if the image is close-up, distant pose etc. We generate a 77 (MMpose2D: 61) dimensional vector to store the features thus engineered. These embeddings are calculated for all training as well as test set images.

After generating the embeddings, the pipeline further uses clustering using K-Means implementation of faiss (Johnson et al., 2017) library to identify K-centroids which represents mean value of K vectors over the training image set embeddings. The centroid and clusters thus generated signify the embeddings corresponding to the most similar and recurring poses in reference image dataset. The clustering algorithm is fit over flattened representations of all images in the training imageset.

After calculating the centroids for the clusters, we determine the suitability order of each of the centroids, which in turn correspond to the poses. To quantify this suitability, for each image in the reference dataset, we estimate the closest pose (represented by the index value of cluster centroid, a value $[0,Num_{poses})$), and then rank these centroids against the target variable which is weighted mean of image’s Product’s customer average review rating, number of ratings etc. To account for the effect of category, subcategory, and other categorical attributes, we train a GBDT based regressor with these features along with centroids to predict the target variable. This gives us a relative order of a centroid for a given category, subcategory, etc. The Algorithm 1 summarizes the scheme.

The inference pipeline (Figure 5) follows the same process of PoI detection, embedding generation. However, inference is calculated at per Product listing - imageset. The pipeline calculates the index of nearest centroid corresponding to each image in the imageset. The inference flow is outlined in the Algorithm 2.

The MissingCentroids thus obtained gives the indices of the Centroids corresponding to the pose w.r.t. AllCentroids representing the reference set centroids and are determined during the training phase. In Figure 6 we observe that $\{D,E,F\}$ are the detected as the missing Centroids in decreasing order of importance, and user can refer the corresponding images from the reference set.