How well does CLIP understand texture?

We report the average per-image accuracy on all four datasets. For DTD we evaluate on the first split (“test1.txt”) with 1180 images across 47 classes. For FMD we evaluated on the entire dataset (10 material classes with 100 images per class). For KTH-TIPS we evaluate on 810 images equally split across 10 classes with categories such as “brown bread”, “sandpaper”, “cotton”, etc. While for KTH-TIPS2a we use 4608 images across 11 classes, with roughly the same number of images per class. The datasets are publicly available and linked via the project’s github repository https://github.com/ChenyunWu/CLIP_Texture.

Table 1 shows the zero-shot classification accuracy of CLIP on DTD, FMD and KTH datasets. For this task we use the prompt “a photo of a [c] pattern” for each category “c” in the dataset. We observe that transformer variants (e.g., ViT-B/32 and ViT-L/14) are generally better than the ResNet (e.g., RN50 and RN101) counterparts. The ViT-L/14@336 transformer trained on larger images (336$\times$336 vs. 224$\times$224) performs the best. Table 2 shows how the accuracy varies across prompts for two different image encoders. The best prompt varies across datasets but on average “a photo of a [c] object” and “a photo of a [c] pattern” performs best. Prompts have a larger impact on the performance of the smaller ViT-B/32 model compared to ViT-L/14 model indicated by the larger variance in performance across prompts. Table 3 shows some additional results using the ViT-L/14@336 image encoder. The accuracy on FMD reaches 93.6% using this encoder for the prompt “a photo of [c] object”, surprisingly outperforming current state-of-the-art ($\approx$ 85%) based on bilinear representations and their variants [3, 13, 12, 6]. This could indicate potential overlap between FMD and the training set of CLIP. We are unable to verify this as the training dataset of CLIP is not publicly available, nor described in detail in the original paper.

DTD² contains multiple attributes for each image which could be incorporated into the prompt design for each category. We include the 20 most frequent attributes for each category in the prompt as “an image of [$p_{1}$, $p_{2}$,…, $p_{20}$] texture”, where $p_{i}$ is the $i^{th}$ most-frequent phrase. For example, the “gauzy” category is described as “an image of gauzy, sheer, transparent, light, thin, white, translucent, soft, see through, delicate, netted, meshy, airy, silky, fabric, see-through, folded, wavy, curtains, cloth texture.” This improves the accuracy to 54.8% from 41.1% when only including the category as prompts with the ViT-B/32 encoder on DTD.

We conduct the compositionality modeling analysis on synthetic texture images the same as described in Section 5.3 of [22]. Given a query phrase such as “blue and red”, the task is rank the positive and hard negative images (which are “blue” or “red” but not both). The R-precision for the retrieval task is listed in Table 5. CLIP achieves a significant improvement on “Two-colors” and a slight improvement on “Color+Pattern” over the DTML model. The larger training set of CLIP allows better generalization to rare or novel combinations in DTD². We also see a slight improvement for CLIP on “Background” compared against DTML but it performs lower than random guesses on “Foreground”. This suggests that CLIP likely has a foreground bias. We investigate this aspect further on CUB Dataset.

To construct a list of attributes for each category, for each attribute we estimate the ratio of images that contain the attribute within a category to the images that contain that attribute across all categories. Top $k$ such attributes are added to prompts for the category in the format “an image of a [$P$] [$C$] with [$P_{1}$] [$N_{1}$], [$P_{2}$] [$N_{2}$], …” where $C$ is the category label, $P$ are attributes of the entire bird, $N_{i}$ are body parts (such as “belly”, “tail”) and $P_{i}$ are attributes for the part $N_{i}$. Figure 2 shows three examples of constructed descriptions. The attributes reflect subtle differences among the three similar species, e.g., “Rhinoceros Auklet” is more “duck-like” with “buff leg”, “Parakeet Auklet” has “white eye” and “white belly”, “Crested Auklet” has “crested head” and “black nape”. We also use different choices for the category label, varying them from the common name to scientific names at different levels of the biological taxonomy. The common name is often used to describe the category and be associated with the corresponding images on the Internet. Scientific names at various levels in the taxonomy are less likely to have been observed by the language models.

Classification results are shown in Table 7. We report the top-k accuracy for the 200-way classification. When no attribute is added, we construct the description simply as “an image of a [$C$]” where [$C$] can be the common name, species, genus, family, or order. In such cases species from the same genus, family or order have the same category embedding resulting in ties. There is a significant performance drop when using scientific names compared to the common names. CLIP has learned plenty about the common names during training as it exploits images available on the web but fails when using species names which are relatively rare. Adding attributes to the descriptions improves performance by a large margin in this case, especially on the top 5/10 accuracy. With the help of attributes, we achieve similar performance when we replace the scientific names with the generic category description “bird”.