Beyond RNNs: Benchmarking Attention-Based Image Captioning Models

Image captioning is the automatic generation of textual descriptions for images, a fundamental challenge in artificial intelligence at the intersection of computer vision and natural language processing. This task involves identifying key objects, their attributes, and their relationships within an image while generating syntactically and semantically coherent sentences that accurately describe the scene. Deep learning-based techniques, known for their effectiveness in various complex tasks, have also been applied to image captioning.

In this study, we conduct a comparative analysis of two deep learning-based image captioning models using the MS-COCO benchmark dataset. Our goal is to evaluate their performance using natural language processing metrics.

In this section, we provide relevant background on previous work in image caption generation and attention mechanisms. Extensive research has been conducted on automatic image captioning, which can broadly be categorized into three approaches: template-based captioning, retrieval-based captioning, and novel image caption generation [5].

Template-based image captioning first detects objects, attributes, and actions in an image before filling predefined slots in a fixed template [1]. Retrieval-based approaches, on the other hand, identify visually similar images from the training dataset and select a caption from the retrieved images [6]. While these methods can generate syntactically correct captions, they often fail to produce semantically meaningful and image-specific descriptions. Both approaches typically involve an intermediate “generalization” step to remove details that are only relevant to a specific retrieved image, such as the name of a city. However, these methods have largely been replaced by neural network-based approaches, which have become the dominant paradigm in image captioning.

In contrast, novel approaches to image caption generation first analyze the visual content of an image before generating captions using a language model [7]. The visual representation is typically extracted using a convolutional neural network (CNN), which is often pre-trained on large-scale image classification datasets [8]. The caption is then generated using a recurrent neural network (RNN)-based language model. The key advantage of this approach is that the entire system can be trained end-to-end, allowing all parameters to be learned directly from the data.

The first approach to using neural networks for caption generation was proposed by Kiros et al. (2014a), who introduced a multimodal log-bilinear model biased by image features. This work was later extended by Kiros et al. (2014b). Mao et al. (2014) adopted a similar approach but replaced the feedforward neural language model with a recurrent one. Both Vinyals et al. (2014) and Donahue et al. (2014) employed recurrent neural networks (RNNs) based on long short-term memory (LSTM) units (Hochreiter & Schmidhuber, 1997) for their models.

The problem with these methods is that when the model attempts to generate the next word in the caption, it usually describes only a specific part of the image. As a result, it fails to capture the essence of the entire input image. Conditioning the generation of each word on the whole image representation does not effectively produce distinct words for different parts of the image. This is precisely where an attention mechanism proves useful.

The MS-COCO (Common Objects in Context) dataset is used in this project. It is widely utilized for training and benchmarking object detection, segmentation, and captioning algorithms. Some key features of the dataset include 80 object categories, 91 stuff categories, 330K images, and five captions per image. Since the dataset has fewer categories but a higher number of instances per category, we have chosen it for training our model.

We tokenized the captions and performed basic cleaning, such as lowercasing all words and removing special tokens. Furthermore, we created a vocabulary of all unique words present across the image captions, resulting in a total of 40,000 data points (8,000 × 5). After filtering, 8,267 unique words remained.

We then selected the top 5,000 or 7,000 words from the vocabulary to improve the model’s robustness to outliers and reduce errors. According to PyTorch and Keras documentation, pre-trained Inception V3 and ResNet50 models require input as 3-channel RGB images of shape (3 × H × W), where H and W must be at least 224. Therefore, data preprocessing involves loading and resizing images. The tokens and images are then converted into embeddings of size 256, which are fed to the decoder.

In this technique, the image is first divided into $n$ parts, and we compute Convolutional Neural Network (CNN) representations for each part, $h_{1},\dots,h_{n}$. When the RNN generates a new word, the attention mechanism allows it to focus on the part of the image most relevant to the word it is about to generate. The model learns where to look, and as we generate a caption word by word, we can observe the model’s focus shifting across different regions of the image.

BLEU-1, BLEU-2, BLEU-3, and BLEU-4 (Bilingual Evaluation Understudy) are among the most commonly reported metrics for evaluating the quality of text generation in natural language processing tasks. Here, we chose the 4-gram BLEU score (BLEU-4) as our primary evaluation metric. Furthermore, we also use the METEOR score, which is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision.

Additionally, we use GLEU (for sentence-level fluency) and WER (Word Error Rate), which are common metrics for evaluating the performance of speech recognition and machine translation systems.

GitHub Repository: Open Repository

The complete source code, dataset preprocessing scripts, model architectures, and training details for this study are available in our GitHub repository. This repository provides a structured implementation of both Vanilla RNN-based and Attention-based image captioning models, along with scripts for evaluation metrics such as BLEU, METEOR, and CIDEr. Additionally, sample outputs, visualization scripts for attention heatmaps, and detailed documentation are included to facilitate reproducibility.

Researchers and developers interested in extending this work or applying it to other datasets can explore the repository for pre-trained models and training logs. Contributions, suggestions, and discussions are welcome through GitHub issues and pull requests.

Having implemented and studied the performance of both Vanilla RNN and Attention-based Image Captioning techniques on the MS COCO dataset, Natural Images, and Abstract Images, we can conclude that, in terms of performance on both human judgment and automatic metrics, the Attention Mechanism is superior to the Vanilla RNN Image Captioning on the COCO dataset and Natural Images with fewer subjects. However, both models perform similarly on abstract paintings and Natural Images with a large number of subjects.

Design considerations, such as the choice of CNN-Encoder, also affect the performance of the model. Moreover, we have observed that the currently used automatic metrics for image captioning judge caption quality by determining similarity between candidate and reference captions. As a result, these metrics fail to achieve adequate levels of correlation with human judgments at the sentence level, reflecting the fact that they do not fully capture the set of criteria humans use when evaluating caption quality.

We have accomplished our goal of comparing and understanding the architecture and performance aspects of deep learning-based image captioning models. In the process, we have learned about the interplay between Computer Vision and NLP, encoder-decoder architectures, loss functions in complex architectures, and the crucial ability to comprehend academic research papers in deep learning.