NITS-VC System for VATEX Video Captioning Challenge 2020

Visual Feature Encoding: For this task, a traditional encoder-decoder based approach is used. For encoding, the visual feature vector of the input video, firstly, the video is evenly segmented into $n$ segments in the interval of $16$ then, using a pre-trained 3D Convolutional Neural Network (C3D), a visual feature vector $f=\{s_{1},s_{2}\dots s_{n}\}$ for video is extracted. The dimension of feature vector for each video is $f\in\mathbb{R}^{n\times m_{x}}$ where $m_{x}$ is dimension feature vector for each segment. Since, the high dimensional feature vector is prone to carry redundant information and affect the quality of features [20], for reducing the dimension of features an average pooling is performed with filter size $5$. All the averaged pooled features($f_{a}$) of each segment are concatenated in a sequence to preserve the temporal relation between them and passed to a decoder for caption generation.

Caption generator: For the decoding, two separate Long Short Term Memory (LSTM) recurrent networks are used as a decoder. In this stage, firstly, all the input captions are passed to a embedding layer to get a dense representation for each word in the input caption. The embedding representation is then passed to both LSTM separately. The first LSTM takes the encoded visual feature vector as an initial stage and for the second LSTM, the visual feature vector concatenated with the output of embedding layer and finally the element wise product is preformed between the output from both LSTM. The unrolling procedure of system is given below: ^††[ ; ] represent the concatenation

$$\tilde{y}=W_{e}X+b_{e}$$

(1)

$$\tilde{z_{1}}=LSTM_{1}(\tilde{y},h_{i})$$

(2)

$$\tilde{z_{2}}=LSTM_{2}([\tilde{y};f_{a}])$$

(3)

$$y_{t}=softmax(\tilde{z_{1}}\odot\tilde{z_{2}})$$

(4)

The Equation 1 represents the embedding of input captions ($X$) where $W_{e}$ and $b_{e}$ are learnable parameter weight and bias respectively. The $\tilde{z_{1}}$ in Equation 2 and $\tilde{z_{2}}$ in Equation 3 are the output of LSTM layers where, $h_{i}$ ($h_{i}=f_{a}$) is the average pooled feature vector which is passed as a hidden state for $LSTM_{1}$ and for $LSTM_{2}$ it concatenated with embedding vector. $\odot$ represent the product of $\tilde{z_{1}}$ and $\tilde{z_{2}}$ which is finally passed to softmax layer.

For the evaluation of the system performance, VATEX video captioning dataset is used. This dataset is proposed for motivating multilingual video captioning, in which each video is associated with corresponding $10$ captions in both English and Chinese. This dataset consist of two test sets namely, public test set for which reference captions are publicly available and private test set for which reference captions are not publicly available which are on hold for the evaluation in VATEX video captioning challenge 2020. The detailed statistics of the dataset is given in Table 1. The system employed in the paper is implemented using QuADro P2000 GPU and tested on both the test sets.

For the evaluation of generated captions, various evaluation metrics have been proposed. In this paper, for the quantitative evaluation of the generated output, the following evaluation metrics are used: BLEU[13]²²2https://github.com/tylin/coco-caption, METEOR [6], CIDEr [16] and ROUGE-L. BLEU evaluates the part of n-grams (up to four) that are similar in both reference (or a set of references) and hypothesis. CIDEr also measure the n-grams that are common in both hypothesis and the references, but in CIDEr, term frequency-inverse document is used for weighting the n-grams. In METEOR, once the common unigrams are found, then it calculates a score for this matching using a unigram-precision, unigram-recall and measure of fragmentation which is used for evaluating how well-ordered the matched word in generated caption against the reference caption.

Following the system architecture discussed in Section 3.1, after extracting the spatio-temporal features using C3D model pre-trained on Sports-1M dataset [9, 15], they are passed to caption generator module.

In the training process, each caption is concatenated by two special marker $<BOS>$ and $<EOS>$ for informing the model about the beginning and ending of caption generation process. We restricted the maximum number of words in a caption to $30$, and if the length of the caption is less than desired length the caption is padded with $0$. The captions are tokenized using Stanford tokenizer [11] and only to $15K$ words with most occurrence are retained. For out-of-vocabulary words, a special tag $UKN$ is used. In the testing phase, the generation of caption starts after watching start marker and the input visual feature vector, in each iteration, the most probable word is sampled out and passed to next iteration with previous generated words and visual feature vector until a special end marker is generated. For loss minimization, cross-entropy loss function is used with $ADAM$ optimizer and learning rate is set to $2\times 10^{-4}$. For reducing the overfitting situation, a dropout of $0.5$ is used and the hidden units of both LSTMs are set to $512$. The system is trained with different batch size $64$ and $256$ for 50 epochs each. On analysing the scores of evaluation metrics and the quality of generated captions in terms of coherence and relevancy, smaller batch size performs better.

The performance of the system on private and public test set is given in Table 3.

In the Table 2, sample output captions generated by the system is given along with videoId.