Reinforcement Learning Friendly Vision-Language Model for Minecraft

Downloading YouTube’s automatic transcripts directly can lead to several issues. Firstly, transcript blocks may overlap, resulting in overlapping timestamps in the transcript. Additionally, the caption quality is typically mediocre, with a higher occurrence of misidentifications, especially in non-English videos. Moreover, the transcript lacks punctuation, making it less friendly for understanding semantics. Based on the aforementioned issues, we implement a pipeline to construct high-quality transcripts as follows: (1) Extract audio from the videos and use Whisper [26] to obtain high-quality, temporally non-overlapping transcripts. (2) Employ FullStop [9] to generate punctuation, resulting in complete sentences of 10-35 words in length.

Following MineDojo [8], we also implement keyword-based filtering to ensure that the content in our dataset is pertinent to the key entities in Minecraft, thereby facilitating the learning of basic game concepts. As essential components of Minecraft, the key entities, such as stones, trees, and sheep, are common across multiple tasks in MineDojo and videos from YouTube. Specifically, we identify entity keywords in the transcripts using a keyword list from MineDojo and extract transcript clips formed into sentences to encompass as many keywords as possible. These extracted transcript clips then serve as the textual component of our dataset, determining the location of corresponding video clips.

After completing the previous two steps, we obtain transcript clips relevant to the keywords. For each transcript clip, we calculate the central timestamp that corresponds to the clip based on the transcript timestamps and then use this central timestamp to extract a video clip with a duration of $D$ seconds from the video. This process allows us to obtain temporally-aligned video clips. However, the video clips obtained in this manner still exhibit some issues, including scene transitions and discrepancies between video content and transcripts. We handle the former problem through video partition and filtering, while the latter problem is addressed in Section 4.4.

Owing to the informal nature of YouTube content, there is often a lack of semantic congruence between the video clips and their corresponding transcriptions, as noted in VideoCLIP [33]. Moreover, video clips frequently contain a few different behaviors which cause scene transitions, e.g. Chopping down the tree first, then suddenly switching the inventory bar. Since some scene transitions lead to irrelevant information, we partition the video content into several semantically coherent segments based on the semantic structure of the video. Then we select the segment that aligns best with the transcript.

To achieve semantic partition of video content, we employ the Bellman $k$-segmentation algorithm [13]. This algorithm divides a sequence of data points into $k$ distinct and constant-line segments, providing a piece-wise constant approximation of the data. To process the video, we first use the officially released MineCLIP [8] video encoder to obtain the embedding of each frame, since MineCLIP can capture video semantics to some extent. Subsequently, we partition these embeddings into $k$ segments and select the segment with the highest similarity score, as calculated in MineCLIP.

We employ correlation filtering techniques to address disparities that persist between video content and transcripts. The correlation filtering is done at two levels, the global and the local levels. From the global level, we calculate the cosine similarities between video embeddings and text embeddings via the original MineCLIP and then select clips based on these similarities.

Recent research [35, 17] has demonstrated that CLIP [25], though trained on whole images, can generate meaningful local features. Inspired by this and following MaskCLIP [35], we make modifications to the original MineCLIP visual encoder, empowering it with the ability to estimate and label the size of the key entity, which is mentioned in the corresponding transcript, in each frame of a video clip without fine-tuning. Then the correlations between video clips and transcripts are calculated as the summation of the sizes across all frames in each video clip. We consider this correlation at the local level.

Based on these two criteria, we select the top $k\%$ of clips, resulting in the final training set. We provide an elaboration on our implementation in Appendix 0.A and illustrate some examples in Figure 2.

We follow the same architecture design as MineCLIP [8], including a video encoder, a text encoder, and a similarity calculator. All the video frames first go through the spatial transformer to obtain a sequence of frame features. The temporal transformer is then utilized to summarize the sequence of frame features into a single video embedding. An adapter further processes the video embedding for better features. Refer to Appendix 0.B for more details about the architecture.

Empirically, we found that the video encoder (essentially MineCLIP) can provide a bond between the entities and the language prompts and give a similar high reward as long as the target entities are present in the video frames, similar to observations in [4]. However, such a reward is not instructive enough for RL tasks since it does not reflect the behavioral trends of agents.

We aim to minimize the sum of the multi-modal contrastive losses [23], including video-to-text, and text-to-video:

where $\mathcal{B}$ is the batch containing sampled video-text pairs, and ${\rm NCE}(\cdot,\cdot)$ is the contrastive loss that calculates the similarity of two inputs. To illustrate, the video-to-text contrastive loss is given by

where $\lambda$ is a temperature hyperparameter, $z_{t}^{+}$ is the positive text embedding matching with the video embedding $z_{v}$, and $\{z_{t}^{-}\}$ are negative text embeddings that are implicitly formed by other text clips in the training batch. Contrastive loss of text-to-video is defined in the same way.

In addition, we also need to incorporate the degree of task completion into the training objective. Specifically, we hope the similarity score between video clip and text embeddings could reflect the task completion degree, i.e., higher completion leads to higher similarity. The sizes of key entities provided by the local correlation filtering serve as a surrogate for the task completion degree. Initially, we try to dynamically adjust the weight of the positive pairs based on the size of the target object. However, it did not work. Essentially, positive and negative pairs will still be separated, even with smaller sample weights.

Therefore, we instead forcibly change the labels of the positive and negative video samples, as illustrated in Figure 3, thereby reducing the model’s confidence in certain positive pairs. Specifically, during training, a positive video sample is swapped with a random negative video sample from the $\mathcal{B}$ based on probability $P_{\rm swap}$. This swap only occurs when the labeled target size $H_{b}W_{b}$ in the positive video sample is below a certain threshold $\tau$, and the smaller the size, the greater the probability,

where $HW$ represents the size of the image.

Since contrastive learning brings positive samples closer and pushes negative samples away, converting a positive pair to a negative one with a certain probability, i.e., the aforementioned random swap, during training will lower the similarity score. As the decrease in similarity is directly proportional to the swapping probability, we set $P_{\rm swap}$ to be inversely related to task completion, ensuing similarity score increases as completion increases.

In addition, the upper limit $P_{\rm max}$ in Equation 3 is set to 0.5, as the swap should not disrupt severely the normal distinction between positive and negative pairs in most cases. This constraint enables CLIP4MC to preserve the understanding capability of MineCLIP for general behaviors that may lack explicit target entities in Minecraft, as discussed in Appendix 0.F.

For RL training, the first step is reward generation. At timestep $t$, we concatenate the agent’s latest 16 egocentric RGB frames in a temporal window to form a video snippet, $V_{t}$. CLIP4MC outputs the probability $P_{G,t}$ that calculates the similarity of $V_{t}$ to the task prompt, $G$, against all other negative prompts. To compute the reward, we further process the raw probability as previous work [8] $r_{t}=\max(P_{G,t}-1/N_{t},0)$, where $N_{t}$ is the number of prompts passed to CLIP4MC. Note that CLIP4MC can handle unseen language prompts without any further fine-tuning due to the open-vocabulary ability of CLIP [25].

The ultimate goal is to train a policy network that takes as input raw pixels and other structural data and outputs discrete actions to accomplish the task that is described by the language prompt, $G$. We use PPO [28] as our RL training backbone and the policy is trained on the CLIP4MC reward together with the sparse task reward if any. The policy input contains several modality-specific components and more details can be found in Appendix 0.D.

We conduct experiments on eight Programmatic tasks, comprising two harvest tasks: milk a cow and shear wool, two combat tasks: combat a spider and combat a zombie, and four hunt tasks: hunt a cow, hunt a sheep, hunt a pig, and hunt a chicken. These tasks are all built-in tasks in the MineDojo benchmark.

More elaborated introduction of the Minecraft environment and settings of these tasks are available in Appendix 0.D. To guarantee a fair comparison, we adopt the same RL hyperparameters for all models and tasks. These hyperparameters are listed in Appendix 0.E.

Through our evaluation of CLIP4MC, MineCLIP [ours], and MineCLIP [official] across eight Minecraft tasks, we want to answer two key questions:

It is noticeable that, in four hunt tasks, three models demonstrate varying performance on RL. Firstly, MineCLIP [ours] consistently achieves better results compared to MineCLIP [official] across all hunt tasks. The superior performance of MineCLIP [ours] provides a positive answer to our first question, suggesting that the YouTube dataset we construct indeed enhances the effectiveness of the reward signal in MineCLIP for task learning. Note that this is the dataset we plan to release for better training the VLM model for RL tasks on Minecraft. Secondly, CLIP4MC shows significantly higher success rates on hunt tasks compared to both MineCLIP [ours] and MineCLIP [official], meaning that the answer to the second question is also positive. As CLIP4MC provides a reward signal taking into account a surrogate for the degree of task completion, i.e., the size of the target object in our implementation, it becomes more RL-friendly in these challenging tasks.

In addition, we notice a practical example of misalignment in the officially released MineCLIP model. Specifically, we observe that in hunt a chicken, the agent trained with MineCLIP [official] tends to keep looking at the sky, indicating such behavior can provide a high intrinsic reward. This phenomenon suggests that the officially released MineCLIP indeed suffers from the misalignment problem in the YouTube dataset. In contrast, our MineCLIP [official] shows promising behaviors on this task, demonstrating that our dataset processing improves the alignment between the transcripts and videos.

In contrast to hunt tasks, these methods perform comparably on harvest tasks and combat tasks, while CLIP4MC still shows marginal advantage over the other two methods. This finding aligns with the results reported in [8], where the MineCLIP reward already achieves saturated performance on these easy tasks. Therefore, further enhancements in the reward signal, like MineCLIP [ours] and CLIP4MC, do not significantly improve performance. Unlike hunt tasks, neither harvest nor combat tasks require the agent to take multiple rounds of chasing. This result does not detract from the superiority of CLIP4MC, evidenced by its performance on hunt tasks, which are considered more difficult [2].

To quantitatively verify that CLIP4MC captures the size of the target entity specified in the language prompt, we collect 5000 steps in task hunt a cow and apply the method described in Appendix 0.A to estimate the maximal size of the cow in consecutive 16 frames. Then we use CLIP4MC, MineCLIP [ours], and MineCLIP [official] to calculate intrinsic rewards respectively. Before calculating the correlation between the size and intrinsic rewards, we transform the size value using $f(x)=\ln{(x+e^{-2})}$, focusing on smaller values. The relationship between the transformed size and intrinsic rewards is visualized in Figure 4. The corresponding Pearson correlation coefficients from left to right are 0.81, 0.66, and 0.62, indicating that CLIP4MC reward has a higher correlation with the size, especially when it is relatively small. This is crucial in RL, as the agent needs dense and distinguishing reward signals to guide the learning process, particularly when the target is distant. Such characteristic is the essential benefit of CLIP4MC in RL.

To evaluate the influence of the dataset quality on RL training and verify the effectiveness of our proposed correlation filter at the local level, we compare MineCLIP [ours] with two ablations: (1) MineCLIP [w/o LCF] is trained on the dataset processed with only global correlation filtering, omitting the local correlation filtering. (2) MineCLIP [RS] is trained on 640K video clips randomly sampled from the MineDojo released database, consistent with their stated dataset construction method [8]. Note that methods evaluated here do not apply random swap introduced in Section 5.2 since these datasets cannot provide entity size. The results are presented in Table 2. Given the overall performance across all three presented tasks, firstly, [RS] does not perfectly reproduce the performance of [official], suggesting that misalignment in the original database hinders the reproduction of [official]. Secondly, part of our proposed data filtering makes the model, [w/o LCF], comparable to [official]. In addition, with all of our proposed data filtering techniques, [ours] outperforms [official].

Table 3 shows the results of video-to-text retrieval and text-to-video on the test set with the same MineCLIP model for a fair comparison. We train these models for 20 epochs and select the models with the highest R@1 value on the test set, respectively. From the results, the model trained on the randomly sampled dataset gets the worst performance than those trained on our YouTube dataset. The results demonstrate that our neat dataset indeed can facilitate the learning of basic game concepts. A notable observation is that [ours] achieves higher success rates on RL tasks but lower performance on retrieval tasks compared to [w/o LCF], suggesting that the video-text alignment objective does not fully align with RL requirements. [w/o LCF] is trained on the dataset filtered only by global-level correlation, which is directly conducted at the video level. In contrast, [ours] is trained on the dataset further filtered by local-level correlation, which is object-centric designing for RL training. Note that [w/o LCF] is the dataset we plan to release for better training the VLM model for video-text retrieval tasks on Minecraft.