Fashion Recommendation: Outfit Compatibility using GNN

We conducted experiments on two tasks: fill-in-the-blank fashion (FITB) recommendation and compatibility prediction.

Due to space constraints, we worked with a subset of 1600 outfits which we split using a 70-30% split into test and training data.
We replicated the two methods using TensorFlow v1.15. We also used Google Inception and Hugging Face’s Visual-Transformer (ViT) models for creating image embeddings. All our experiments were run on the COC ICE server using RTX6000 GPUs. For optimization, we adopt the RMSProp and Adam optimizers. We early stop the training process when the loss stabilizes, usually around 15 epochs.

The Node-wise Graph Neural NetworksCui et al. (2019) approach was the first paper to utilise graphs for solving the outfit compatibility problem. By representing categories as nodes in a fashion graph, researchers were able to tackle many challenges that only graphs could address. First, they were able to circumvent the problem of variable length input by representing outfits as a subgraph. Though this problem was addressed using Bi-LSTMs Han et al. (2017), which represent an outfit as a sequence, Bi-LSTMs introduced a bias because outfits do not inherently possess a sequential structure. Additionally, by representing an outfit as a subgraph, the NGNN approach allowed for the modelling of more complex interactions between clothing/items and outfits. Finally, it made the category of an item a mainstay of the model in a number of ways, like a) using category-dependent weighted message passing aggregation b) using category-specific feature mapping for item embeddings.

The code repository for the paper was available to us via GitHub. However, the code needed to be adapted to the latest Tensorflow version as it was written in Tensorflow 1.15. We also implemented code to extract embeddings of the images of items using a pre-trained Vision Transformer to compare results.

The paper uses the 120 clothing/accessory cate gories as nodes with an edge existing between them if they co-occur in an outfit. The training strategy discussed below helps to bring the process of representing an outfit as a subgraph to life. It also illustrates how this subgraph can be distilled to a single score using an attention mechanism.

The following describes one iteration of a forward pass by passing a single outfit as input. As a reminder, an outfit consists of a list of 7-8 items for each of which we have a text description and an image.

The embeddings of the text description are obtained using the one-hot encoding(2757-dimensional Boolean vector) after the pre-processing text. The embeddings of the image are created using a pre-trained InceptionV3 model(2048- dimensional vector).

• •

  Step 1: For an outfit, we create a corresponding ”negative” outfit. This is done by randomly selecting an existing item in the outfit and replacing it with a randomly sampled item. Then, the next steps are performed for both the positive and negative outfits.

• •

  Step 2: We then fetch the embeddings of each of the items(text/visual depends on the mode, for now, assume visual embedding only).

• •

  Step 3: Pick the category nodes that the items belong to and initialize the feature vector of these nodes with the embeddings of the items to each of the respective category nodes.

• •

  Step 4: Then, we start the message passing and update the embeddings of each of the items.

• •

  Step 5: Next, we obtain the updated item embeddings.

• •

  Step 6: Finally we use these embeddings of an item and feed it to an attention layer to compute attention scores. These are the outfit compatibility scores for this outfit.

• •

  Step 7: Now we obtain the difference of 2 scores of the positive and negative outfit

• •

  Step 8: Finally we get the loss and backpropagate through the network to update the attention layer and GNN weights.

It’s important to note that the rest of the category nodes of the graph are completely ignored and all further message passing is done only between the existing nodes.

The hyper-parameters were determined by the grid search strategy. These include learning rate, batch size, visual-text trade-off parameter $\beta$, the parameter for L2 regularization $\lambda$, hidden size d and the number of message propagation steps T. The model achieved optimal performance with the learning rate as 0.001, batch size as 16, $\beta$ as 0.2, $\lambda$ as 0.001, d as 12 and T as 3. After subsetting the input dataset, we were able to run around 12 epochs for the training process for HGNN and around 4 epochs for NGNN. NGNN was significantly slower to train than HGNN even after tweaking the learning rate. All the experiments were conducted using GPU RTX-6000.

Another method that was explored to solve the problem of outfit compatibility involves leveraging hypergraphs Li et al. (2022) to represent outfits and outfit categories. A hypergraph is similar to any other graph, but has a key difference that an edge can connect multiple nodes – these special edges are called ”hyperedges.” The code repository for the paper was available to us via GitHub.

A hypergraph can encode high-order data correlation (beyond paired connections) utilizing its degree-free hyperedges, in contrast to a simple graph, in which all edges must have a degree of 2, as shown in Figure 2. The adjacency matrix in Figure 2 is used to represent a generic graph, where each edge only connects two vertices. A hypergraph, on the other hand, can be easily enlarged for multi-modal and heterogeneous data.

.

As shown in Figure 2, we created a fashion hypergraph H = (V, E) based on the training dataset where the category information of the items acts as the prior knowledge of the items. The hyperedge (i.e., the linkages of the same colour) represents the interaction between several categories when they occur in the same outfit, whereas each node is specifically linked to only one unique category. Therefore, each outfit may be thought of as a hyperedge of a hypergraph if each component is filled into its respective nodes.

In this framework, each clothing/accessory item category (e.g. black pants) is represented as a node, and several nodes are connected by a hyperedge if the categories represented by the nodes constitute an outfit.

.

Only two nodes are chosen for each hyperedge during the transformation process to represent the full hyperedge. The remaining nodes, referred to as mediator nodes, are linked with the key nodes to create a new graph in order to fully utilize all of the nodes’ information. Each mediator node connects two essential nodes, as seen in Figure 4.

This is similar to the previously discussed NGNN approach in that it utilizes item categories as nodes, but instead of outfits being represented as sub-graphs, outfits are simply connected by hyperedges.

As outlined in the steps for the NGNN framework, the node embeddings in step 3 are generated from the hypergraph structure.

The hyper-parameters were determined by the grid search strategy. These include learning rate, batch size, visual-text trade-off parameter $\beta$, the parameter for L2 regularization $\lambda$, hidden size d and the number of message propagation steps T. The model achieved optimal performance with the learning rate as 0.001, batch size as 16, $\beta$ as 0.2, $\lambda$ as 0.001, d as 12 and T as 3. After subsetting the input dataset, we were able to run around 12 epochs. NGNN was significantly slower to train than HGNN even after tweaking the learning rate. All the experiments were conducted using GPU RTX-6000.

As mentioned in Dataset Section, we use a subset of the dataset (both for training and validation and testing). This leads to our accuracy scores being lower than what was reported in the original papers Cui et al. (2019)

As we have both textual descriptions of our items as well as images, we can compare the performance of models built using different modalities. This is summarized in the table below:

The text-Only model uses features created using detailed textual descriptions of each item. Embeddings created using that were fed into the NGNN model. Similarly, the visual-only model uses embeddings from the images of items only. The muli-modal model uses both of those embeddings to train the NGNN model.
As expected the multi-modal performs best (as seen in Table 2). Though the text-only model performs slightly better than the visual-only model. This could be because the detailed descriptions of each item in an outfit and the embeddings created from them are concise enough to capture all key features of the item.