Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning ^††thanks: This work has received funding from the European Union’s Horizon 2020 programme dAIEDGE (G.A. No 101120726).

In both Static and Dynamic Expansion, the expansion coefficient $N_{E}$ defines a collection of learnable parameters $E_{Q},E_{B}\in\mathbb{R}^{N_{E}\times d_{m}}$. However, in the Static Expansion, $N_{E}$ defines exactly the size of the expanded sequences regardless of the input length $L$. In particular, the expansion queries $Q_{E}$ and biases $B_{E}$ equal to $E_{Q}$ and $E_{B}$ respectively. In contrast, in the Dynamic Expansion, the expanded sequence is of size $N_{E}\cdot L$, and the expansion queries $Q^{E}$ and biases $B_{E}$ are calculated with the BroadSum operator, defined in the two cases as:

where $C\in\mathbb{R}^{L\times d_{m}}$ denotes a linear projection of the input and $\mathbb{H}_{E}\in\mathbb{R}^{L\times(L\cdot N_{E})}$ is defined as:

whereas $\mathbb{I}_{E}\in\mathbb{R}^{N_{E}\times(L\cdot N_{E})}$ is defined by the column-wise concatenation of $L$ identity matrices of size $N_{E}\times N_{E}$:

. An example of the input and output of the BroadSum operation is depicted in the bottom left of Figure 2, where the bias vectors are omitted for simplicity.

The forward expansion generates the expanded sequences and involves three linear projections of the input, denoted as $K,V_{1},V_{2}\in\mathbb{R}^{L\times d_{m}}$. First of all, the “Length Transformation Matrix”, denoted as $M$, is computed as the dot-product similarity between $K$ and the expansion queries $Q_{E}$:

The result is fed into the following operations:

where $\Psi:(X,\epsilon)\rightarrow Y$, $X,Y\in\mathbb{R}^{N_{1}\times N_{2}},\epsilon\in\mathbb{R}^{+}/\{0\}$ is the row-wise normalization function defined as:

the coefficient $\epsilon$ ensures the feasibility of the operation. Then, the expanded sequences are calculated as follows:

In the backward step, the original sequence length is retrieved by transposing the length transformation matrix in Equation 2 and applying the same operations of Equation 3:

This time, the matrices $R^{bw}_{i}$ are multiplied with the expanded sequences of Equation 5:

Finally, the final results $B^{bw}_{1}$ and $B^{bw}_{2}$ are combined by means of a sigmoid gate:

where $S\in\mathbb{R}^{L}$ is a linear projection of the input.

The backward operation completes the operations performed in the Static and Dynamic Expansion. It can be noted that all operations in the forward (3)(5) and backward expansion (7) (6) are duplicated in two operations streams for $i=1$ and $i=2$, differing mainly in the sign of the computation of the Length Transformation Matrix in (2). The decision was made to mitigate the remote possibility of the matrix being populated only by zeros. This does not affect the results compared to the single path but slightly increases the computational cost.

In the case of Dynamic Expansion, masking is applied when calculating the results in (5) and (7) to preserve the auto-regressive property. The operation principle of Static and Dynamic Expansions are illustrated in Fig. 2, which, for simplicity, depicts only a single operation stream and omits biases and the output sigmoid gates.

To increase the effectiveness of the Static Expansion, we perform the Forward and Backward operations on a collection of target lengths instead of one. We call the operation Block Static Expansion. From a formulation perspective, all operations are repeated over a group of expansion coefficients $G=\{N^{1}_{E},N^{2}_{E},\ldots,N^{N_{G}}_{E}\}$ and can be implemented in a way such that both forward and backward steps are performed over all targets at the same time. All expansion group queries and biases can be combined into a single one:

and the computational efficiency of the previous formulation can be preserved. During the backward stage, the length transformation matrix is scaled by the inverse number of elements in the group $G$.

The training dataset consists of the popular MS-COCO benchmark [41] split according to [45], resulting in 113287 image-description pairs for the training, 5000 in the validation set, and in the 5000 test set. Each reference caption is pre-processed by a simple pipeline consisting of lowering casing, removing punctuation, and filtering out words that do not occur at least 5 times (vocabulary of size 10000). Additionally, the final model is evaluated over the Novel Object Captioning at Scale (nocaps) dataset validation set [46], which consists of three classes of images called in-domain, near-domain, and out-domain, according to the familiarity of the classes with respect to those contained in the training set. This dataset is subject to the same pre-processing of MS-COCO and serves the purpose of further challenging the model in unfavourable conditions.

Two models are implemented for the experimental setup. The baseline, which is the Base Transformer and our main model, referred to as “ExpansionNet v2”, is implemented with the following configurations $d_{m}$=$512$, $d_{ff}$=$2048$, $N_{enc}$=$N_{dec}$=$3$. In the latter, the Dynamic expansion coefficient is set to 16, and the Static expansion coefficients consist of $G$=$\{32,64,128,256,512\}$ (more details in Section IV-B). Each one relies on top of the same backbone, the Swin-Transformer in the Large configuration [13] pre-trained on ImageNet [47]. All images are subject to a minimal pre-processing: first, they are resized into a $3{\times}384{\times}384$ tensor, then RGB values are converted into a $[0,1]$ range and further normalized using $mean$=$(0.485,0.456,0.406)$ and $std$=$(0.229,0.224,0.225)$. The source code of the experiments is available¹¹1Code available at: https://github.com/jchenghu/ExpansionNet_v2.

It can be observed that the Swin-Transformer backbone is the most computationally expensive part of the system. For this reason, inspired by [48], to enable the End to End training step to a broader number of computational architectures, our training is divided into four steps in particular, each phase (in both the cross-entropy training and the reinforcement stage) consists of initial training in which the backbone’s weights are frozen and a fine-tuning step during which gradients flow throughout the whole system:

Step A) Cross-Entropy – Freezed backbone. The model is trained using batch size 48, an initial learning rate of 2e-4, a warmup of 10000, and is annealed by 0.8 every 2 epochs for 8 epochs;

Step B) Cross Entropy – End to End. The whole system is trained for 2 additional epochs, using batch size 48 and an initial learning rate of 3e-5 annealed by 0.55 every epoch;

Step C) CIDEr-D optimization – Freezed backbone. Reinforcement phase adopts a batch size of 48, an initial learning rate of 1e-4, no warmup, annealed by 0.8 every epoch for 9 epochs;

Step D) CIDEr-D optimization – End to End. The whole system is fine-tuned for a few more iterations up to an additional epoch using a batch size of 20 and fixed learning rate 2e-6. This step is optional since it only slightly contributes to the final performances and can be skipped if no improvements are observed. All CIDEr-D optimization steps are implemented according to the Standard configuration²²2SacreEOS signature [49]:
STANDARD_wInit+Cider-D[n4,s6.0]+average[nspi5]+1.0.0.

Despite its apparent complexity, it is much more computationally friendly than the standard method consisting of a small batch size of 10 for 30 epochs for both optimization steps. As a matter of fact, only a much smaller number of training epochs are dedicated to fine-tuning the whole system. Thus, the time required for the calculation of the backbone’s gradient is often avoided and the time required for forward operations can be drastically reduced as well. In particular, in our implementation, during steps 1 and 3 the backbone’s forward pass is performed only once for each image in the data set. Therefore, its cost is replaced by a memory read and copy. All steps are trained using the RAdam optimizer [53] ($\beta_{1}=0.9,\ \beta_{2}=0.98$).

Table II and Table III report the score comparison between ExpansionNet v2 and the best-performing models in recent years. Up-Down [5] introduced the idea of extracting a collection of features from the images using an object detector like Faster-RCNN [6] in contrast to the classification backbone [4]. The idea was adopted in most of the following architectures as well, for instance, in the case of GCN-LSTM [50] and SGAE [51], which additionally implemented a convolutional graph network on top of it to exploit the information provided by a scene graph. AoANet [31] adopted the Transformer and improved the attentive components with two gates serving the purpose of simulating an additional level of attention over the inputs and augmented the language modelling part with an LSTM. On the other hand, X-Transformer [14] adopted a fully attentive architecture and further refined the attentive blocks by means of bilinear pooling techniques. The most recent and performing architectures focused on increasing the more effective ways to feed visual information into the sequence modelling network. For instance, RSTNet [52] showcased the effectiveness of grid features over regions, GET [35] processed the images using a global representation in conjunction with the local ones, DLCT [18] instead exploited the advantages of both regions and grid visual features. Finally, PureT [11] implemented the first end-to-end Transformer architecture applying the Window / Shifted-Window MHA [13] in both the encoder and decoder. ExpansionNet v2 outperforms PureT by a margin of 0.7 BLEU1, 0.6 BLEU4, 0.1 METEOR, 0.4 ROUGE, 2.2 CIDEr-D and 0.3 SPICE in the single model case and by 0.1 BLEU1, 0.6 BLEU4, 0.3 ROUGE, 2.7 CIDEr-D and 0.4 SPICE in the ensemble configuration.

We evaluate ExpansionNet v2 using the ensemble configuration and adopting the standard Beam Search (beam size 5) over the official testing set of 40775 images, submitting the predictions to the online testing server. Results are reported in Table IV. c5 and c40 represent the scores of 5 and 40 reference captions (unknown to the user), respectively.

Our model achieves State-of-the-Art performance (as of 2 July 2022) among non-generative models trained on MS-COCO 2014, outperforming the previous one [11] by a margin of 1.2 BLEU4 (c40), 0.2 METEOR (c40), 0.5 ROUGE-L (c40) and 2.5 CIDEr-D in both c5 and c40 instances. However, it is ultimately outperformed, by a significant margin, by generative models [15, 16] which we consider orthogonal to our work since they focus more on training method and data quality rather than architecture design.

We evaluate ExpansionNet v2 over the nocaps validation set. In particular, we adopt a single model trained exclusively on Cross-Entropy Loss, using no additional pre-training data sets. The predictions are generated by the standard Beam Search algorithm (beam size 3) in contrast to the CBS [54]. A limited comparison is reported in Table V, which showcases that our model achieves very competitive results among the architectures trained in similar configurations, with an overall lead of 17.6 CIDEr and 1.4 SPICE over the Up-Down model [5]. It is still ultimately outperformed by recent V+L pre-training-based works

such as [39, 56, 40].