RetailKLIP : Finetuning OpenCLIP backbone using metric learning on a single GPU for Zero-shot retail product image classification

Recognizing Retail Product or packaged grocery goods in images or videos can help to solve multiple problems in supermarket floors and supply chain hubs. Enabling self checkout stores, measuring retail execution to evaluate merchandising activities and automatic slotting of products during fulfillment are some of the applications. Retail Product Image Classification has generally been treated as one-shot or few-shot classification problem, because, unlike say a class like dog or cat, the variance between different images of a retail product is much lesser and just involves different occlusions, blurring and lighting changes. Finetuning or Transfer learning in Deep Neural Networks, often Convolutional Neural Networks, has been used in existing literature for this purpose. However, even in such cases, previous classification methods require gradient descent to be run to train classifiers for the full neural network backbone or the last linear layer to get best results. Because grocery products have very frequent new launches, package redesigns and offer tags, this involves finetuning quite often. This process requires usage of computing resources to finetune models being used to classify 100s or 1000s of products, again, even when say 2 images of a newly launched product are discovered. This process not just makes maintaining real world Retail Product Classification models costlier, it also creates a ”blind spot” time, where for the time being a new model is being finetuned to add new products as classes, the model cannot recognize these new products.

CLIP is a way to train an image encoder and a text encoder in parallel such that the embedding of an image and its text description are close in a common space [Radford et al., 2021] . OpenCLIP is an implementation of CLIP available under a permissive license [Ilharco et al., 2021] . Because of large number of training examples and text descriptions providing descriptive annotations for objects, the CLIP image encoder has been used for many classification problems with a linear layer or even nearest neighbors. While embeddings from CLIP encoder are much better than any other pretrained weights to use in zero shot setting and can be used as good baselines, finetuning them for a specific domain is always desirable to make the zero shot classification more accurate. However, CLIP has generally proven to be hard to finetune. We propose an end to end pipeline to finetune a CLIP backbone for zero shot Retail Product image classification, identifying and addressing different concerns.

Our work has two components : 1. Creating a large dataset which can be used for finetuning a CLIP backbone for retail product images. 2. Proposing a learning rate strategy, class balancing approach and other finetuning components which solve for erratic finetuning of CLIP backbones and address imbalanced large retail datasets.

Our constraint in this work is that we have to limit our work on one GPU and so cannot train any models which go beyond a single GPU both for Zero Shot classification or any baselines.

Retail Product Image Recognition has been studied using multiple techniques. Older works use feature descriptors like SIFT and BRISK [Leutenegger et al., 2011, Lowe, 2004] to recognize retail products. More recently, tricks to finetune Convolutional Neural Networks [Srivastava, 2020, Peng et al., 2020] to perform well in the task were proposed. However, given finetuning the entire backbone is costly, recent works propose training just a linear layer to learn classify representations given by a backbone trained by contrastive supervised/semi-supervised methodology on relevant datasets [Srivastava, 2022] . There have also been works where GAN-like backbones have been used to recognize retail products using information retrieval techniques [Tonioni and Stefano, 2019].

More recently, there has been a trend to use LVMs (Large Vision Models) as a backbone instead of traditional ImageNet pretrained backbones for transfer learning in Computer Vision. CLIP is one method of training these LVMs where Billion plus sized datasets of images and their text descriptions are trained to learn image embedding and text embedding of description of the image such that they lie close on a common space [Radford et al., 2021] . Because these datasets are huge and text descriptions have more details than just a classification dataset, this training mechanism yields excellent image encoders. CLIP Image encoders have shown great results on zero shot and linear layer only training tasks on internet images. A CLIP image encoder for example can be used to get excellent results on ImageNet, by just simply comparing the embeddings or training a linear layer on top of embeddings generated. OpenCLIP is an implementation of CLIP with permissive license, that we use in our work [Ilharco et al., 2021].

CLIP image encoder backbone however is considered hard to finetune for related but different domains. There have been some publications which give some hints about finetuning [Dong et al., 2022, Kumar et al., 2022]. Their work inspires our finetuning technique. The image encoder finetuned has a Vision Transformer architecture (specifically, a large variant of Vision transformer or ViT-L) [Dosovitskiy et al., 2020].

Deep Metric Learning [Musgrave et al., 2020] techniques can be used to learn encoders which generate discriminative embeddings. Unlike softmax based classification, metric learning approaches generalize better to openset recognition [Deng et al., 2019]. ArcFace loss, a metric learning loss function, used generally in long tailed facial recognition tasks is used to finetune CLIP [Deng et al., 2019]. To our knowledge, this is the first time metric learning has been used to finetune a LVM to a new domain.

Given that retail datasets are very imabalanced is another reason we use ArcFace loss to finetune the CLIP model to the retail domain. Previous works have shown that plain ArcFace handles imbalanced datasets better than softmax loss even when aided with data balancing techniques [Zhang and Deng, 2020] . We use ArcFace loss with a custom data balancing technique for our final results.

We would like to list the datasets used for the experiments in our work. To finetune CLIP image encoder to retail product image recognition, we use an inhouse (not publicly available) dataset called RP6K with 6500 retail products. This has over 1 Million images of retail products with their class tags. Just like real world, the number of images of these products in RP6K is long-tailed with some products having upto 4000 images, while many having lesser than 10 images. We finetune CLIP image encoder on this dataset using a novel mixture of techniques which yields a model that can be used for Zero shot classification on other datasets.

Grozi-120 is the first dataset used for evaluation [Merler et al., 2007]. It is a one shot classification dataset with one train image per product, which is packshot like, while test set images are from real world. It has 120 products.

CAPG-GP [Geng et al., 2018] is another one shot dataset, but with fine grained products. However, the train and test images both appear to be from similar domain here.

We have explicitly made efforts to make sure none of the products in CAPG-GP and Grozi-120 are present in RP6K.

RP2K [Peng et al., 2020] is a dataset with 2388 retail products, with substantial number of both train and test images for each product. However, we still use the dataset in a few shot setting like Grozi120 and CAPG-GP. That is, while the entire test set will be used to test the performance of Zero Shot classification, only one image per class from train set will be used to calculate class representations for classification.

Our work involves finetuning OpenCLIP image encoder on a retail product image dataset RP6K such that resultant image encoder can yield embeddings of product images useful for zero shot retail product image classification on other datasets. The image encoder after finetuning with RP6K is referred to as RetailKLIP. For zero shot classification, one image per product is taken and augmented and then passed to RetailKLIP to get embeddings for that product. This process is repeated for all products to create a set of products’ embeddings. To test an unseen image, its image embedding created using RetailKLIP is compared to product embeddings created earlier and the product with closest embedding is considered to be the output of RetailKLIP.

For Grozi-120 [Merler et al., 2007] and CAPG-GP [Geng et al., 2018], we compare the accuracy of RetailKLIP Zero Shot classification with full finetuning of a ResNext-WSL model [Mahajan et al., 2018]. We also compare it with tricks like using LCA layer and maxent loss while full finetuning ResNext-WSL to increase accuracy from [Srivastava, 2020]. These methods involve full finetuning of the Convnet backbone making the process slow and compute intensive. We also compare it with accuracy of using a pretrained semi supervised backbone with trainable linear layers [Srivastava, 2022]. While training just a linear layer is also quite fast, zero shot classification with RetailKLIP needs no training at all. The basis for comparing ResNext-WSL with ViT-L despite difference in number of parameters is because these are the largest models than can fit on a single Nvidia 1080Ti GPU for training.

As we can see, RetailKLIP gives competitive results to full finetuning or linear layers trained on semi supervised trained backbone.

Our work proposes a method to create a Zero shot classifier for Retail Product images on a single GPU by finetuning OpenCLIP. The accuracy is competitive or even sometimes better than full finetuning large Convnet backbones on the same GPU. This enables real world retail computer vision systems to quickly integrate new products into their range and avoid resource intensive trainings multiple times.