Language Guided Local Infiltration for Interactive Image Retrieval

In IIR task, we are given a reference image and a modification text that describes what content should be modified in the image. The objective of our primary task is to learn an image-text compositional embedding that encodes the information required to retrieve the target image of interest, which should reflect the changes specified by the modification text. Let $I_{r}$ denote the reference image, $T$ denote the modification text, and $I_{t}$ denote the target image. Given an input pair $(I_{r},T)$, we aim to learn image-text compositional features $f=\mathcal{C}(I_{r},T;\Theta)$ that are well-aligned with the target image feature $f_{t}=\mathcal{F}(I_{t};\Theta)$ by maximizing their similarity as,

where $\mathcal{C}(\cdot)$ and $\mathcal{F}(\cdot)$ denote the feature composer and image feature extractor, respectively. $\kappa(\cdot,\cdot)$ denotes the similarity kernel, which is implemented as a dot product. $\Theta$ denotes the learnable parameter of the whole model.

To achieve this goal, we propose a Language Guided Local Infiltration (LGLI) system. Fig. 2 illustrates the overall pipeline to obtain the image-text infiltrated feature. Our proposed method contains four components: 1) an image encoder to learn visual representation, 2) a text encoder to learn text representation, 3) a Language Prompt Visual Localization (LPVL) module to generate localization masks of reference images, and 4) an image-text compositor with Text Infiltration with Local Awareness (TILA) to generate the image-text compositional representation (infiltrated feature). Model parameters are optimized by multi-layer matching loss.

Without loss of generality, to encapsulate the visual contents into discriminative representations, we employ an image encoder, i.e., a standard CNN, for image representation learning. In this work, we extract the feature maps from multiple convolution layers to capture the visual information of different granularities. Concretely, the image features are obtained from three different levels inside the CNN, i.e., low, mid, and high layers. To represent the semantics of modification texts, we first utilize the LSTM as a text encoder to extract text representations. Then we construct MLPs to represent the text features of low, mid, and high layers.

After obtaining the feature representations of texts and images, our primary goal is to learn the image-text compositional presentation. In contrast to previous approaches, we argue that the local region in the image wanted to modify should be paid extra attention, and other regions should be as unchanged as possible. Therefore, we design a LPVL module detailed in Sec. 3.2 to generate localization masks of reference images and an image-text compositor via TILA module detailed in Sec. 3.3 to learn the infiltrated representation of the reference image and modification text.

In this paper, we propose to pre-locate the region to be modified in the reference image. We expect to use the location information as the input of the image-text compositor to achieve the purpose of modifying only the desired location content and preserving other information.

Specifically, we propose a Language Prompt Visual Localization (LPVL) module to generate localization masks of reference images. Fig. 3 illustrates the pipeline of our proposed LPVL. The role of the LPVL module is to get the desired localization mask while inputting the reference image and modification text. For example, when the purpose is “make blue object purple”, we can locate the position of the “blue object” in advance through the LPVL module and generate the localization mask. In this paper, we define “blue object” as the localization text $T_{L}$ and all localization texts comes from the corresponding modification text $T$. In our LPVL module, we use Faster-RCNN [25] model pre-trained on the COCO dataset to detect the regions of interest (i.e., object). CLIP [24] model pre-trained on a dataset of 400 millions image-text pairs collected from the internet is used to extract language and visual representations. Specifically, Given a reference image $I_{r}$, we first apply a pre-trained Faster-RCNN to obtain $K$ regions of interest. The formulas are written as:

where $FasterRCNN(\cdot)$ denotes the pre-trained Faster RCNN model and $O$ is the set of detected regions of interest. Then each region of interest is represented as a $512$-dimensional feature vector through the pre-trained image encoder $CLIP_{img}$. The localization text $T_{L}$ is also represented as a $512$-dimensional feature vector through the pre-trained text encoder $CLIP_{txt}$. The features of the regions of interest and the modification text are represented as:

In order to capture the exact local region to be modified, we calculate the similarity between each region and the modification text and select the most similar region to generate a localization mask, i.e., the localization object $LO$ is calculated as:

where $\kappa(\cdot,\cdot)$ present the similarity function, e.g., cosine similarity.

In order to accurately change the local semantics of the image to be specified and preserve other unrelated local semantics, we propose a Text Infiltration with Local Awareness (TILA) module for the image-text compositor. Fig. 4 illustrates the overall pipeline of TILA. The key insight is to capture the local semantics and achieve specified change via the localization mask and modification text. Meanwhile, we expect that other regions cannot be inadvertently changed.

After generating the localization mask $I_{m}$ from the LPVL module, we leverage the modification text and localization mask to jointly change the local semantic of the reference image. Specifically, we feed the reference image into the image encoder to extract the low-, mid-, and high-level image representations, i.e., $f_{r}^{L}$, $f_{r}^{M}$, and $f_{r}^{H}$. Similarly, the mask representations can also be extracted via the image encoder. The text representations are extracted by the text encoder. For brevity, we use $f_{r}^{i}$, $f_{m}^{i}$, and $f_{t}^{i}$ to represent the image, mask, and text features, respectively, which are formulated as:

where $E(\cdot),Linear(\cdot),Reshape(\cdot)$ are text encoder, linearization and reshape operators, respectively. $f_{r}^{i},f_{m}^{i},f_{t}^{i}\in\mathbb{R}^{c_{i}\times h_{i}\times w_{i}}$. $c_{i}$, $h_{i}$, and $w_{i}$ refer to the number of the channel, height, and width in the $i$-layer of the image encoder, respectively. To achieve text guided local-aware image feature, we first compose the features of the reference image and localization mask and obtain visual features enhanced by local information. This operation is equivalent to increasing the weight of the specified region so that the specified region is easier to modify than others. Besides, considering that the localization mask may be inaccurate, we propose a cross-modal attention mechanism to gradually adjust the modified regions. As shown in Fig. 4 (b), the cross-modal attention contains a channel attention module and a spatial attention module, which is similar as CBAM [38] but the difference is that we consider the information of two modalities (image vs. text). In the channel attention module, we first aggregate spatial information of an image feature map by using both average-pooling and max-pooling operations, generating two different spatial context descriptors, i.e., $f_{Avg}^{Ci}=AvgPool(f_{r}^{i})$ and $f_{Max}^{Ci}=MaxPool(f_{r}^{i})$, where $f_{Avg}^{Ci},f_{Max}^{Ci}\in\mathbb{R}^{C\times 1\times 1}$ . Both descriptors are then forwarded to a shared MLP network. Then the image features and text features are combined to produce our channel attention map $A_{C}^{i}$.

where $\sigma$ denotes the sigmoid function. Similarly, we generate a spatial attention map $A_{S}$ by utilizing the inter-spatial relationship of features. We aggregate channel information of a feature map by using two pooling operations, generating two 2D maps, i.e., $f_{Avg}^{Si}=AvgPool(A_{C}^{i}\odot f_{r}^{i})$ and $f_{Max}^{Si}=MaxPool(A_{C}^{i}\odot f_{r}^{i})$, where $f_{Avg}^{Si},f_{Max}^{Si}\in\mathbb{R}^{1\times H\times W}$. These two 2D maps are then concatenated and convolved by a standard convolution layer to produce our spatial attention map $A_{S}^{i}$.

where $[\cdot,\cdot]$ denotes concatenation. Different from the channel attention, the spatial attention focuses on where is an informative part, which is complementary to the channel attention. Finally, the weighted feature map by the cross-modal attention mechanism is expressed as

Then, the enhanced visual features by localization mask and the weighted features by the cross-modal attention mechanism are processed by an Instance Normalization (IN). We then use a residual-like structure to infiltrate text features into image features as much as possible. The infiltrated features generated by TILA in layer $i$ are defined as:

where $\alpha$ is a trade-off parameter used to control the effect of the localization mask.

Note that our TILA module can be plugged and played in each layer of the CNN to precisely modify the reference image and generate image-text infiltrated representation. In this work, the image-text compositor is formulated by deploying the TILA module in the low, middle, and high layers of CNN. Finally, we get the concatenation of multi-level outputs after average-pooling and reduce the dimension by an FC-layer.

where [$\cdot$] is concatenation operator and $avg(\cdot)$ is the average pooling operator. The final infiltrated feature $f$ used for image retrieval is a $512$-dimensional feature vector.

In training phase, given an input pair ($I_{r}$, T), we aim to learn image-text compositional infiltrated features $f=\mathcal{C}(I_{r},T;\Theta)$ that are well-aligned with the target image feature $f_{t}$. Following TIRG [35], we then adopt the multi-layer softmax cross-entropy loss as the standard loss:

where $B$ is the batch size and $l$ is the layer number. ${f_{i}^{l}}^{+}$ is the target image feature of the $i^{th}$ input pair. The difference from TIRG is that we not only supervise the final output features but also supervise features in each layer by the softmax cross-entropy loss, such that the local discrimination is guaranteed. The details of the proposed algorithm are described in Algorithm 1.

CSS [35] consists of 38K synthesized images with different colors, shapes and sizes. It contains about 19K training images and 19K testing images, respectively. Modification text for this dataset is fallen into three categories: adding new objects to the scene, removing objects from the image, and changing the attributes of the current objects in the image. Following the same evaluation protocol of TIRG [35], we use the same training split and evaluate on the test set. Certain object shape and color combinations only appear in training but not in testing, and vice versa.

Fashion200k [39] includes about 200K clothing images of 5 different fashion categories, namely: pants, skirts, dresses, tops, and jackets. Each sample is an image of a piece of a costume with accompanying attributes as the description (e.g., “black lace scallop skater dress”). We use the same training split (around 172K images) as TIRG [35]. During training, because there is no already matched reference image, modification text, and target image, pairwise images with attribute-like modifications are generated by comparing their product descriptions, i.e., the reference image and the target image have different attributes.

MIT-States [17] contains about 60K images. Each image is annotated with a noun and an adjective (such as “sunny sky” or “diced fruit”). In total, the images are annotated using 245 unique nouns and 115 unique adjectives. Following the standard training and test splits provided by TIRG [35], we use 49 of the nouns for testing, and the rest is for training. This allows the model to learn about state/adjective (modification text) during training and has to deal with unseen objects presented in the test query. Pairs of images with the same object labels and different state labels are sampled to create the query image and target image. Therefore the system is to retrieve images that possess the same object but a new state compared with the query image.

To validate the efficacy of our approach in the interactive image retrieval task, we compare with many representative multi-modal learning methods and interactive image retrieval methods, such as Concatenation, Show and Tell [34], Parameter Hashing [22], MRN [18], Relationship [26], FiLM [23], TIRG [35], LBF [11], JVSM [3], MAAF [6], JAMMAL [42], JPM [41], ComposeAE [1], CoSMo [19], TIRG-DIM [9], CLVC-Net [37], and FashionVLP [8], where the last 11 of these methods are developed specifically for interactive image retrieval task over the last three years. Concatenation is a simple feature fusion method composing visual and textual features of the final layer by concatenation.

We implement our method using PyTorch. Without loss of generality, we use the same backbone as most previous methods. Concretely, we adopt ResNet-18 as the image encoder to extract image features from and the LSTM as the text encoder to learn text representations. We use an SGD optimizer with a learning rate set to 0.01 and train the model for a maximum of 50 epochs. We empirically set $\alpha=10^{-4}$. The batch size $B=200$ for the CSS dataset, and $B=32$ for other datasets. We use retrieval accuracy $R@N$ as our evaluation metric, which computes the percentage of test queries where at least one target or correctly searched image is within the top $N$ retrieved images. We also present the mean precision when $N$ is set as different values. Besides quantitative evaluation, we also provide qualitative results.

Results on CSS: Table 1 summarizes the retrieval performance of our method and existing methods on the CSS dataset. As can be seen, our model demonstrates compelling results compared to all other alternatives. Specifically, the performance improvement of our method over the second-best method is 5.5%.

Fig. 5 shows our qualitative results on the CSS dataset. The figure indicates our model can capture visual cues and selectively preserve and transform the reference image features according to language semantics. We observe that our model can retrieve new images that resemble the reference image, while changing certain content conditioned on text feedback, e.g., adding new objects to the scene, removing objects from the image, and changing the attributes of the current objects in the image.

Results on Fashion200k: Table 2 shows our comparison with existing methods on the Fashion200k dataset. As can be seen, our model demonstrates competitive results compared to all other alternatives. We can observe that the performance improvement of our method over the second-best method is 4.1%, 3.6%, 2.9%, and 3.9% on R@1, R@10, R@50, and average, respectively.

Fig. 6 shows our qualitative results on the Fashion200k dataset. We notice our model can retrieve new images that resemble the reference image while changing certain content conditioned on text feedback, e.g., color, material, pattern, style, and trim.

Results on MIT-States: Table 3 shows our comparison with existing methods on the MIT-States dataset. As can be seen, our model demonstrates competitive results compared to all other alternatives. We can observe that the performance improvement of our method over the second-best method is 0.2%, 1.1%, and 0.6% on R@1, R@5, and average, respectively.

Fig. 7 shows our qualitative results on the MIT-States dataset. We notice our model can retrieve new images when the modification is easy, e.g., “sunny sky”. However, Our model retrieval failed when the modification is abstract, e.g., “windblown horse”.

Interestingly, the proposed method is usually better than others in most cases but the R@10 is lower than ComposeAE on the MIT-States dataset. This is because there are more target images in the database, i.e., the average number of target images per query on the CSS, Fashion200k and MIT-States datasets are 1, 3, and 26.7, respectively. Since there are multiple target images in the database, the correct target image can be easy to retrieved without any operation when the number of images returned by the retrieval system is sufficient.

Besides, we present the results of localization visualization through the LPVL module. As shown in Fig. 8, the top 3 localization boxes of LPVL are green, blue, and red, respectively. We can see the localization boxes is usually accurate on the CSS and Fashion200k datasets, but sometimes inaccurate on the MIT-States dataset. Because MIT-States contains so many more complex categories that it is difficult to locate the region of modification. It is the reason why the improvements of our methods is not high on MIT-States.

• •

  w/o LPVL: To investigate the effect of the LPVL module, we remove the LPVL module, i.e., not paying attention to extra location information.

• •

  w/o CA: To explore the effect of the cross-modal attention mechanism, we remove the cross-modal attention mechanism, i.e., $f_{A}^{i}$ is replaced by $f_{r}^{i}$ in Eq. (9).

• •

  w/o TILA: To verify the role of the TILA module in the interactive image retrieval task, we replace the TILA module with concatenation to conduct a variant of our method.

Table 4 shows the ablation results of our LGLI, which proves the necessity of incorporating the LPVL module, cross-modal attention mechanism, and TILA module.

We perform experiments on the CSS dataset to verify the parameter sensitivity of $\alpha$ in Eq. (9). Specifically, we tune the value of ${\alpha}$ from $10^{-1}$ to $10^{-9}$. The interactive image retrieval performance variation is presented in Fig. 9. We can observe that the performance of our proposed method is slightly sensitive to the settings of ${\alpha}$. The recommended parameter range is between $10^{-3}$ and $10^{-5}$.