Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning ^††thanks: This research is partially supported by the JSPS KAKENHI Grant Number 19H04179, and based on a project, JPNP18002, commissioned by NEDO.

We defined a vision-based robotics task in the real world; that is, the real-world environment is a target MDP: $\mathcal{M}^{\tau}=(\mathcal{S}^{\tau},\mathcal{A},p^{\tau},r^{\tau},\gamma)$, where $\mathcal{S}^{\tau}$ is a state space, $\mathcal{A}$ is an action space, $p^{\tau}:\mathcal{S}^{\tau}\times\mathcal{A}\times\mathcal{S}^{\tau}\to\mathbb{R}$ is a transition probability density, $r^{\tau}:\mathcal{S}^{\tau}\times\mathcal{A}\times\mathcal{S}^{\tau}\to\mathbb{R}$ is a reward function, and $\gamma\in[0,1]$ is a discount factor. Because we assumed that the target MDP is a vision-based task, $\mathcal{S}^{\tau}$ consists of images, and each $s\in\mathcal{S}^{\tau}$ contains single or multiple image frames. In standard model-free reinforcement learning (RL) settings, agents can interact with the environment: they observe $s_{t+1}\sim p^{\tau}(\cdot\mid a_{t},s_{t})$ and reward $r_{t}=r^{\tau}(s_{t+1},a_{t},s_{t})$ by performing action $a_{t}$ at state $s_{t}$, which is internally preserved in the environment at timestep $t$; after the transition, $s_{t+1}$ is stored in the environment. However, there are concerns in terms of the risk and cost associated with learning a policy through extensive interaction with $\mathcal{M}^{\tau}$.

To reduce the risk and cost of training a policy in the target MDP, we pre-trained a policy on a simulator environment, called the source MDP: $\mathcal{M}^{\sigma}=(\mathcal{S}^{\sigma},\mathcal{A},p^{\sigma},r^{\sigma},\gamma)$. Note that the action space $\mathcal{A}$ is the same between the two MDPs. In contrast, the state space $\mathcal{S}^{\sigma}$, the transition probability density $p^{\sigma}:\mathcal{S}^{\sigma}\times\mathcal{A}\times\mathcal{S}^{\sigma}\to\mathbb{R}$, and the reward function $r^{\sigma}:\mathcal{S}^{\sigma}\times\mathcal{A}\times\mathcal{S}^{\sigma}\to\mathbb{R}$ are different from those of the target MDP. We assumed that because we considered robotics tasks, the deterministic transition function $Tr^{\sigma}(s,a)=s^{\prime}\sim p^{\sigma}(\cdot\mid a,s)$ could be defined in the simulator environment and $p^{\sigma}$ resembled a Dirac delta distribution.

The source state space $\mathcal{S}^{\sigma}$ corresponded to a semantic space, that is, each $s\in\mathcal{S}^{\sigma}$ was semantic information. For example, consider a robot-arm grasp task; each $s\in\mathcal{S}^{\tau}$ is a single or multiple image frame showing a robot arm and objects to be grasped. Each $s\in\mathcal{S}^{\sigma}$ consists of semantics such as $xyz$-coordinates of the end-effector and target objects and angles of joints.

The source MDP and target MDP are expected to have some structural correspondence. Here, we describe our assumptions regarding the relations of the two MDPs. We assumed the existence of a function $F:\mathcal{S}^{\tau}\to\mathcal{S}^{\sigma}$ satisfying the following conditions:

Transition Condition: For all $(s^{\prime},a,s)\in\mathcal{S}^{\tau}\times\mathcal{A}\times\mathcal{S}^{\tau}$, $p^{\sigma}(F(s^{\prime})\mid a,F(s))=\int_{\bar{s}\in\bar{\mathcal{S}}}p^{\tau}(\bar{s}\mid a,s)\mathrm{d}\bar{s}$, where $\bar{\mathcal{S}}=\{\bar{s}\in\mathcal{S}^{\tau}\mid F(\bar{s})=F(s^{\prime})\}$.

Reward Condition: For all $(s^{\prime},a,s)\in\mathcal{S}^{\tau}\times\mathcal{A}\times\mathcal{S}^{\tau}$, $r^{\sigma}(F(s^{\prime}),a,F(s))=r^{\tau}(s^{\prime},a,s)$.

In the above conditions, $F$ is considered an oracle that takes an image and outputs corresponding semantics; that is, $F$ is the true image-to-semantics translation mapping. In the transition condition, $\bar{\mathcal{S}}$ is a set of images that has common semantics $F(s^{\prime})$. Imagine the transition from $s\in\mathcal{S}^{\tau}$ to $s^{\prime}\in\mathcal{S}^{\tau}$ with action $a\in\mathcal{A}$ in the target MDP, the transition condition holds $F(s^{\prime})=Tr^{\sigma}(F(s),a)$. The reward condition indicates that a reward for this transition $r^{\tau}(s^{\prime},a,s)$ equals the one for a transition from $F(s)\in\mathcal{S}^{\sigma}$ to $F(s^{\prime})\in\mathcal{S}^{\sigma}$ with the action $a$ in the source MDP.

In this section, in addition to the two MDP environments, we define resources that can be used to approximate $F$.

In contrast to the above methods, [12, 13, 14, 15, 17, 16, 18, 19] aimed to perform style translation mapping among specific styles. To accomplish this, these methods required an offline dataset of the target MDP. Because these methods followed the principle of collection without execution of on-policy interaction, the offline dataset could be collected by a safety-guaranteed policy. Unsupervised style translation, such as domain adaptation [26] and CycleGAN [27], are often used to change the styles for state-of-the-art methods [13, 14, 15, 17, 18, 28, 24]. Using this translation mapping as a pre-processing function of the target agent, the pre-trained policy can determine actions in the same image style as the source MDP in the target MDP.

However, domain adaptation and cycle-consistency [27] only have a weak alignment ability [18], and some existing methods use paired datasets to properly transfer styles [19, 17, 16]. Therefore, these two datasets have been widely employed in previous studies and can be assumed to be a common setting.

The similarity of transfer via image-to-semantics and image-to-image is that they train style translation mapping $\hat{F}$ among the source and target state spaces that preserves essential information; furthermore, the agent is the composite $\pi\circ\hat{F}$, where $\pi$ is a policy.

Again, the above methods use a non-photorealistic renderer on the simulator. Thus, these methods cannot be compared with transfer via image-to-semantics, as explained in Section II-B2.

Previous studies have used semantics in the source MDP [18, 10, 17, 16, 12]. An important perspective on the applicability of these methods to image-to-semantics is whether they use a renderer on the simulator, as shown in Table I and as discussed in Section II-B2. Because methods using a renderer assume that the source state space is an image space, image-to-semantics is beyond their scope, and it is not certain that their mechanism will be successful in image-to-semantics. For example, CycleGAN, which has been successfully used for image-to-image learning, failed in image-to-semantics [18]. In this regard, we refer to [18], an unpaired method that applies the findings from image-to-image to image-to-semantics. In addition, [19] is compared as a representative method that uses a paired dataset as in this study.

The objective of pair augmentation is to construct artificial paired data $\mathcal{P}^{\prime}$ such that $s^{\sigma}\approx F(s^{\tau})$ for $(s^{\sigma},s^{\tau})\in\mathcal{P}^{\prime}$ and $s^{\tau}\in\mathcal{T}^{\tau}$. Using an augmented paired dataset, we aimed to obtain $\hat{F}$ that approximates $F$ by minimizing the loss

Note that CRAR [19] adopts $\mathcal{L}(\hat{F},\mathcal{P})$ instead of $\mathcal{L}(\hat{F},\mathcal{P}\cup\mathcal{P}^{\prime})$.

Our principle is as follows. Let $\mathcal{I}\subseteq\mathcal{O}$ be a subset of indices corresponding to the beginning of the episodes in $\mathcal{T}^{\tau}$. Suppose we have a paired dataset $\mathcal{P}$ constructed by querying semantics $s_{i}^{\sigma}=F(s_{i}^{\tau})$ corresponding to images $s_{i}^{\tau}$ in $\mathcal{T}^{\tau}$ for time index $i\in\mathcal{I}$. Although semantics $s_{i+1}^{\sigma}$ representing an image of the next timestep $s^{\tau}_{i+1}$ in $\mathcal{T}^{\tau}$ is unknown, because of the transition condition given in Section II-A and deterministic transition, it equals $s_{i+1}^{\sigma}=Tr^{\sigma}(s_{i}^{\sigma},a_{i})$, where $a_{i}$ is the action taken at timestep $i$ when collecting the offline dataset $\mathcal{T}^{\tau}$ and is included in $\mathcal{T}^{\tau}$. In reality, because human annotations and state transition contain errors as compared to the truth, the generated semantics $s^{\sigma}_{i+1}$ do not exactly represent the image $s^{\tau}_{i+1}$. However, even with errors in $F$ and $Tr^{\sigma}$, it is expected that the generation of the above semantics is a valuable approximation. By recursively applying the above generation, we obtained the augmented paired dataset $\mathcal{P}^{\prime}$.

Formally, $\mathcal{P}^{\prime}$ was constructed as follows: For each index $i\in\mathcal{I}$, we defined a sequence $\{\widehat{s}_{i+t}^{\sigma}\}_{t\in\mathcal{E}_{i}}$ as $\widehat{s}_{i}^{\sigma}=s_{i}^{\sigma}$ (contained in $\mathcal{P}$) and $\widehat{s}_{i+t}^{\sigma}=Tr^{\sigma}(\widehat{s}_{i+t-1}^{\sigma},a_{i+t-1})$ for $t\in\mathcal{E}_{i}$, where $a_{i+t-1}$ are contained in $\mathcal{T}^{\tau}$. The augmented paired dataset is then $\mathcal{P}^{\prime}=\{(\widehat{s}^{\sigma}_{i+t},s^{\tau}_{i+t})\}_{i\in\mathcal{I},t\in\mathcal{E}_{i}}$, where $s^{\tau}_{i+t}$ is contained in $\mathcal{T}^{\tau}$. Thus, we could construct an augmented paired dataset $\mathcal{P}^{\prime}$ of size $\lvert\mathcal{P}^{\prime}\rvert=\sum_{i\in\mathcal{I}}\lvert\mathcal{E}_{i}\rvert$ from the paired dataset $\mathcal{P}$ of size $\lvert\mathcal{P}\rvert=\lvert\mathcal{I}\rvert$.

Figure 2 illustrates the pair augmentation scheme.

The reason why $\mathcal{I}$ was a subset of episode start indices $\mathcal{O}$ rather than $\mathcal{I}\subseteq\{j\mid 0\leq j<\lvert\mathcal{T}^{\tau}\rvert,j\in\mathbb{Z}\}$ was to maximize the size of augmented pairs $\lvert\mathcal{E}_{i}\rvert$. In other words, because we could augment $s^{\sigma}_{i}=F(s^{\tau}_{i})$ until the end of the episode including $s^{\tau}_{i}$, to maximize $\lvert\mathcal{P}\cup\mathcal{P}^{\prime}\rvert$, human annotations should be conducted at the beginning of an episode of $\mathcal{T}^{\tau}$.

To select episodes for annotation, that is, decide $\mathcal{I}$, we incorporated the idea of diversity-based active learning (AL) [29, 30, 31]. Their motivation was to select dissimilar samples to effectively reduce the approximation error. Intuitively, if $\mathcal{P}\cup\mathcal{P}^{\prime}$ has many similar pairs, they might have a similar effect on training $\hat{F}$; this may lead to a waste in annotation cost. Therefore, we attempted to select episodes (indexed by $\mathcal{I}\subset\mathcal{O}$) to be annotated to ensure the inclusion of diverse pairs.

We successively selected the episode to annotate, and we called each selection step the $n$-th round. For $i\in\mathcal{O}$, let $B_{i}=\{s_{i+t}^{\tau}\}_{t\in\{0\}\cup\mathcal{E}_{i}}$ be a set of target state observations present in the episode starting at timestep $i\in\mathcal{O}$. We referred to it as batch. Let $\mathcal{I}_{n-1}$ be the set of selected indices before the $n$-th round, and let $S_{n-1}=\bigcup_{k\in\mathcal{I}_{n-1}}B_{k}$ be a set of all the state vectors in the episodes selected before the $n$-th round. Let $d:\mathcal{S}^{\tau}\times\mathcal{S}^{\tau}\to\mathbb{R}$ be some appropriate distance measure. In the $n$-th round, a batch was selected based on the following two diversity measures: The inter batch diversity

can evaluate the dissimilarity of $B_{i}$ and $S_{n-1}$. The batch with the greatest $f_{\mathrm{inter}}$ was considered to be the most dissimilar batch against the pre-selected batches. The intra batch diversity

can evaluate the dissimilarity of the states inside $B_{i}$. The batch with the greatest $f_{\mathrm{intra}}$ was considered to contain the most diverse states.

We selected a batch that maximizes the above two diversity measures; we performed a bi-objective optimization for selection. To avoid overemphasizing one measure over the other, we employed two separate single-objective optimizations for each measure. In each round, we picked up indices of batches with $f_{\mathrm{inter}}$ in the top $b\%$ ($b=10$ in our experiments) from unselected episodes as $\mathcal{Q}$, and subsequently, selected the batch with the greatest $f_{\mathrm{intra}}$ from $\mathcal{Q}$. $\mathcal{I}_{0}$ was initialized with the episode sampled from $\mathcal{O}$ uniformly at random.

For $d:\mathcal{S}^{\tau}\times\mathcal{S}^{\tau}\to\mathbb{R}$ to be a reasonable distance measure in the image space, we employed a VAE encoder [32]: $E^{\tau}:\mathcal{S}^{\tau}\to\mathcal{Z}$. It stochastically outputs a latent vector $z\in\mathcal{Z}$ for $s^{\tau}\in\mathcal{S}^{\tau}$. The distance between two states $s_{p}^{\tau}\in\mathcal{S}^{\tau}$ and $s_{q}^{\tau}\in\mathcal{S}^{\tau}$ was given by the Euclidean distance between the mean vectors for their latent representations, that is, $d(s_{p}^{\tau},s_{q}^{\tau})=\lVert\mathbb{E}[E^{\tau}(s_{p}^{\tau})]-\mathbb{E}[E^{\tau}(s_{q}^{\tau})]\rVert_{2}$. We trained $E^{\tau}$ using all states in the offline dataset $\mathcal{T}^{\tau}$ before performing the active learning procedure.

We used the states contained in $\bigcup_{i\in\mathcal{I}}B_{i}$ in training $\hat{F}$ by Equation 2; however, the remaining $\bigcup_{i\in\mathcal{O}\setminus\mathcal{I}}B_{i}$ were not used. In order to use it, we included $E^{\tau}$ as a feature extractor for $\hat{F}$ by receiving the benefit of representation learning for downstream tasks. We modeled $\hat{F}=\phi\circ E^{\tau}$, and we trained $\phi$ by Equation 2, whereas $E^{\tau}$ was fixed.

We evaluated the proposed approach on three environments.

We used a 7-layer convolutional neural network and a 4-layer fully connected neural network for the VAE encoders $E^{\tau}$ and $\phi$, respectively, for both the proposed and existing methods. We trained them in gradients using Adam [35]. The dimensions of the latent space of VAE $\mathcal{Z}$ were set to 32, 96, and 192 for Shooting, KUKA, and HalfCheetah, respectively. For CRAR [19], we uniformly selected indices $\mathcal{I}$ from $\{i\mid 0\leq i<\lvert\mathcal{T}^{\tau}\rvert,i\in\mathbb{Z}\}$. For our method, without an AL setting, $\mathcal{I}$ was selected uniformly and randomly from $\mathcal{O}$. For Shooting and KUKA, we used a handcrafted policy instead of one trained by RL as $\pi^{\sigma}$. In HalfCheetah, we trained $\pi^{\sigma}$ using PPO [36].

Tables II, III and IV show the results of the image-to-semantics learning in the three environments. These tables show the results on average$\pm$std over five trials. $\lvert\mathcal{I}\rvert$ denotes the number of paired data, which is the annotation cost. Because of the transition and reward conditions, the PP of $\pi^{\sigma}\circ F$ on $\mathcal{M}^{\tau}$ assimilate to that of $\pi^{\sigma}$ on $\mathcal{M}^{\sigma}$.

Note that most image-to-image methods shown in Table I cannot be compared with image-to-semantics methods because some assumptions cannot be satisfied under image-to-semantics settings. One way to speculate on the performance of the image-to-image techniques in an image-to-semantics setting is to see Zhang et al. [18]. Zhang et al. used domain adaptation [26], which is commonly used in image-to-image learning; thus, their method can be interpreted as a representative example in which the techniques cultivated in image-to-image are imported to image-to-semantics. Although CycleGAN [27] is also widely employed in image-to-image learning, along with domain adaptation, they confirmed in their experiments that this method did not outperform their method in the image-to-semantics setting [18].

In all cases, compared with the approach of Zhang et al. [18], our approaches with and without AL achieved a smaller MD and a greater PP. Zhang et al.’s approach is designed to learn $\hat{F}$ without a paired dataset to eliminate the annotation cost. However, learning without pairs does not necessarily lead to the true image-to-semantics translation mapping, as observed in the high MD and low PP in our results. This result shows the effectiveness of the paradigm using paired data when aiming for higher performance policy transfer while compromising the annotation cost to prepare a small number of paired data.

By comparing the results of our approaches with and without AL and those of CRAR, we confirmed the efficacy of pair augmentation in achieving a smaller MD and higher PP. We achieved PP=$44.73\pm 1.07$ in Shooting with 10 pairs using pair augmentation and AL, which is more than the PP=$42.29\pm 2.12$ achieved by CRAR with 50 pairs. In addition, in KUKA, we achieved PP=$0.65\pm 0.07$ in 10 pairs, which exceeds PP=$0.52\pm 0.15$ achieved by CRAR with 100 pairs. This means that the annotation cost was reduced by more than $\times$5 and $\times$10, respectively. This difference is even more pronounced in HalfCheetah. This may be because the ratio $\lvert\mathcal{P}\cup\mathcal{P}^{\prime}\rvert/\lvert\mathcal{P}\rvert$ is the greatest in this environment: CRAR uses $\lvert\mathcal{P}\rvert=\lvert\mathcal{I}\rvert$ paired data, whereas our proposed approach used an additional $\lvert\mathcal{P}^{\prime}\rvert=999\lvert\mathcal{I}\rvert$ augmented paired data because the number of timesteps per episode was 1000 in this environment.

A tendency of reduced MD and increased PP was observed in the proposed approach with AL compared to that without AL. Specifically, AL reduced MD except in KUKA, and clearly improved PP in KUKA, while achieving competitive PP in the other two environments.

In Figure 3, we present the effectiveness of our approach in Shooting. The proposed AL maximized the diversity in the latent space that represents the image space, but the diversity was also maximized when this result was visualized in the semantic space.

In this section, we verify the robustness of the proposed method against errors in annotation and state transitions.

To further reveal the behavior of image-to-semantics methods, we evaluated them on HalfCheetah by adopting a low performance policy, rather than the random policy, as the behavior policy. We pre-trained the low performance policy with a small number of iterations using PPO.

We observed that, compared with Table IV and Table VII, the performance of the behavior policy affected the PP of the resulting target agent. Here, the PP of the random policy was $-1230.01$ and that of the low performance policy was $822.39$. Therefore, the PP of our proposed method was improved from $968.49\pm 99.53$ to $1527.37\pm 133.19$ when $\lvert\mathcal{I}\rvert=50$.

These results indicate that owing to the low performance of the random policy, faster-running states, that is, states with high velocity, cannot be observed; in other words, the random policy can only observe a limited state. This limitation could lead to an increase in the approximation error of $\hat{F}$. This implied that image-to-semantics is affected by the performance of the behavior policy in some tasks.

A promising result for the image-to-semantics framework is that the target agents obtained by our approach outperform the behavior policy. In particular, in Table VII, the PP of the behavior policy is $822.39$; furthermore, when image-to-semantics was performed with 50 annotations ($\lvert\mathcal{I}\rvert=50$), we obtained a PP of $1527.37\pm 133.19$. In other words, we could achieve a higher performance compared with that of the behavior policy using a small number of annotations and the image-to-semantics protocol.

In the previous discussion, we found that the PP achieved by the image-to-semantics framework is affected by the quantity and quality of paired data, and the region of state space comprising the dataset for training $\hat{F}$. In fact, as an extreme example, $\hat{F}$ trained using $\lvert\mathcal{P}\rvert$=100k with the trajectories collected by the optimal policy, achieved PP=$2624.65\pm 29.09$, which is almost identical to PP=$2735.73$, the performance of the optimal source policy. Note that such a near-complete policy transfer is already achieved in Shooting and KUKA, as shown in Tables II and III.