Active Perception and Representation for Robotic Manipulation

Our task is adapted from the simulated setup used in [44] and [40]. [40] showed that intelligent manipulation behaviors can emerge through large-scale Q-learning in simulation and on real world robots. The robots were only given RGB inputs from an uncalibrated camera along with proprioceptive inputs. A comparative study of several Q-learning algorithms in simulation was performed in [44] using the same task setup. Achieving success rate over 80% required over 100K grasp attempts. Performance of 85% or over is reported with 1M grasp attempts. Furthermore, [40] and [44] restricted the action space to 4-DoF (top-down gripper orientations). We remove such restrictions, allowing the gripper to control all 6-DoF as this is important for general object manipulation.

Reinforcement learning with high-dimensional inputs and sparse rewards is data intensive ([42, 32]), posing a problem for real world robots where collecting large amounts of data is costly. Goal-conditioned policies have been used to mitigate the sparse reward problem in previous work ([33, 30]). In addition to optimizing the sparse rewards available from the environments, policies are also optimized to reach different goals (or states), providing a dense learning signal. We adopt a similar approach by using the 3D points produced by a fixation policy as reaching targets for the grasping policy. This ensures that the grasping policy always has a dense reward signal. We use the Soft Actor-Critic algorithm [28] for policy learning, which was shown to improve both sample-efficiency and performance on real world vision-based robotic tasks [29].

Classical computer vision algorithms infer geometric structure from multiple RGB or RGBD images. For example, structure from motion [20] algorithms use multiple views of a scene across time to produce an explicit representation of it in the form of voxels or point sets. Multiple, RGBD images across space can also be integrated to produce such explicit representations [31]. The latter approach is often used to obtain a 3D scene representation in grasping tasks ([25, 24]). In contrast to these methods, neural-based algorithms learn implicit representations of a scene. This is typically structured as a self-supervised learning task, where the neural network is given observations from some viewpoints and is tasked with predicting observations from unseen viewpoints. The predictions can take the form of RGB images, foreground masks, depth maps, or voxels ([16, 39, 22, 18, 17, 27]). The essential idea is to infer low-dimensional representations by exploiting the relations between the 3D world and its projections onto 2D images. A related approach is described in [38] where the network learns object descriptors using a pixelwise contrastive loss. However, data collection required a complex pre-processing procedure (including a 3D reconstruction) in order to train the network in the first place. Instead of predicting observations from a different views, Time Contrastive Networks (TCNs) [21] use a metric learning loss to embed different viewpoints closer to each other than to their temporal neighbors, learning a low-dimensional image embedding in the process.

Multiple view representation learning has proven useful for robotic manipulation. TCNs [21] enabled reinforcement learning of manipulation tasks and imitation learning from humans. Perspective transformer networks [18] were applied to a 6-DoF grasping task in [23], showing improvements over a baseline network. [38] used object descriptors to manipulate similar objects in specific ways. GQNs [27] were shown to improve data-efficiency for RL on a simple reaching task. In this work we chose to use GQNs for several reasons: a) they require minimal assumptions, namely, the availability of RGB images only and b) they can handle unstructured scenes, representing both multiple objects and background, contextual information. We adapted GQNs to our framework in three ways. First, viewpoints are not arbitrarily distributed across the scene, rather they maintain the line of sight directed at the 3D point chosen by the fixation policy. Second, we apply the log-polar like transform to all the images, such that the central region of the image is disproportionately represented. These two properties allow the representation to be largely focused on the central object, with contextual information receiving less attention according to its distance from the image center. Third, instead of learning the representation prior to the RL task as done in [36], we structure the representation learning as an auxiliary task that it is jointly trained along with the RL policies. This approach has been previously used in [15] for example, resulting in 10x better data-efficiency on Atari games. Thus APR jointly optimizes two RL losses and a representation loss from a single stream of experience.

Attention mechanisms are found in two forms in deep learning literature [12]. “Soft” attention is usually applied as a weighting on the input, such that more relevant parts receive heavier weighting. “Hard” attention can be viewed as a specific form of soft attention, where only a subset of the attention weights are non-zero. When applied to images, this usually takes the form of an image crop. Hard attention architectures are not the norm, but they have been used in several prior works, where a recurrent network is often used to iteratively attend to (or “glimpse”) different parts of an image. In [36], this architecture was used for scene decomposition and understanding using variational inference. In [10], it was used to generate parts of an image one at a time. In [41], it was applied to image classification tasks and dynamic visual control for object tracking. More recently in [35], hard attention models have been significantly improved to perform image classification on ImageNet. Our work can be seen as an extension of these architectures from 2D to 3D. Instead of a 2D crop, we have a 3D shift in position and orientation of the camera that changes the viewpoint. We found a single glimpse was sufficient to reorient the camera so we did not use a recurrent network for our fixation policy.

We based our task on the published grasping environment [44]. A robotic arm with an antipodal gripper must grasp procedurally generated objects from a tray (Figure 1). We modify the environment in two ways: a) the end-effector is allowed to move in full 6-DoF (as opposed to 4-DoF), and b) a second manipulator (the head) is added with a camera frame fixed onto its wrist. This second manipulator is used to change the viewpoint of the attached camera. The agent therefore is equipped with two action spaces: a viewpoint control action space and a grasp action space. Since the camera acts as the end-effector on the head, its position and orientation in space are specified by the joint configuration of that manipulator: $v=(j_{1},j_{2},j_{3},j_{4},j_{5},j_{6})$. The viewpoint action space is three-dimensional, defining the point of fixation $(x,y,z)$ in 3D space. Given a point of fixation, we sample a viewpoint from a sphere centered on it. The yaw, pitch and distance of the camera relative to the fixation point are allowed to vary randomly within a fixed range. We then use inverse kinematics to move the head to the desired camera pose. Finally, the second action space is 6-dimensional $(dx,dy,dz,da,db,dc)$, indicating the desired change in gripper position and orientation (Euler angles) at the next timestep.

Episodes are structured as follows. The agent is presented with an initial view (fixation point at the center of the bin) and then executes a glimpse by moving its head to fixate a different location in space. This forms a single-step episode from the point of view of the glimpse policy (which reduces the glimpse task to the contextual bandits formulation). The fixation location is taken as the reaching target; this defines the auxiliary reward for the grasping policy. The grasping policy is then executed for a fixed number of timesteps (maximum 15) or until a grasp is initiated (when the tool tip drops below a certain height). This defines an episode from the point of view of the grasping policy. The agent receives a final sparse reward if an object is lifted and the tool position at grasp initiation was within 10cm of the fixation target. The latter condition encourages the agent to look more precisely at objects, as it is only rewarded for grasping objects it was looking at. The objective of the task is to maximize the sparse grasp success reward. The grasping policy is optimized using the sparse grasp reward and the auxiliary reach reward, and the fixation policy is optimized using the grasp reward only.

Note that all views sampled during the grasping episode are aligned with the fixation point. In this manner, the grasping episode is implicitly conditioned by the line of sight. Essentially, this encourages the robot to achieve a form of eye-hand coordination where reaching a point in space is learnt as a reusable skill. The manipulation task is thus decomposed into two problems: localize and fixate a relevant object, then reach for and manipulate said object.

An overview of APR is given in Figure 2. Multimodal input from one view, consisting of the view parameterization (six joint angles of the head $v=(j_{1},j_{2},j_{3},j_{4},j_{5},j_{6})$), image ($64\times 64\times 3$) and gripper pose $g=(x,y,z,\sin(a),\cos(a),\sin(b),\cos(b),\sin(c),\cos(c))$, is encoded into a scene representation, $r1$, using a seven layer convolutional network with skip connections. $(a,b,c)$ are the Euler angles defining the orientation of the gripper. The scene representation $r1$ is of size $16\times 16\times 256$. The proprioceptive input vectors $g$ and $v$ are given spatial extent and tiled across the spatial dimension ($16\times 16$) before being concatenated to an intermediate layer of the encoder. The input from a second view (Figure 2C, D) is similarly used to obtain $r2$, which is then summed to obtain $r$, the combined scene representation.

The fixation policy and grasping policies operate on top of $r$. Their related outputs (action $a$, state-value $v$ and action-value functions $q$) are each given by a convolutional block followed by a fully-connected layer. The convolutional blocks each consist of three layers of $3\times 3$ kernels with number of channels 128, 64, and 32 respectively. The generator is a conditional, autoregressive latent variable model that uses a convolutional LSTM layer. Conditioned on the representation $r$, it performs 12 generation steps to produce a probability distribution over the query image. The encoder and generator architecture are unmodified from the original work, for complete network details we refer the reader to [27].

The log-polar like sampling we use is defined as follows. Let $(u,v)\in[-1,1]\times[-1,1]$ be a coordinate in a regularly spaced image sampling grid. We warp $(u,v)$ to obtain the log-polar sampling coordinate $(u^{\prime},v^{\prime})$ using the following equation:

We learn both policies using the Soft Actor-Critic algorithm [28], which optimizes the maximum-entropy RL objective. For detailed derivations of the loss functions for policy and value learning we refer the reader to [28]. In conjunction with the policy learning, the multimodal encoder and generator are trained using the generative loss (evidence lower bound) of GQN. This loss consists of KL divergence terms and a reconstruction error term obtained from the variational approximation [27]. Note that the encoder does not accumulate gradients from the reinforcement learning losses and is only trained with the generative loss. To obtain multiple views for training, we sample three viewpoints centered on the given fixation point at every timestep during a grasping episode. Two of those are randomly sampled and used as context views to obtain $r$, the third acts as the ground truth for prediction. We did not perform any hyperparameter tuning for the RL or GQN losses and used the same settings found in [28] and [27].

We evaluate active looking vs a passive (fixed) gaze at targeted grasping. This setting is designed to test goal-oriented manipulation, rather than grasping any object arbitrarily. For this experiment, we do not learn a fixation policy, but instead use goals, or target objects, that are selected by the environment. The policies are only rewarded for picking up the randomly chosen target object in the bin. The active model and the passive model receive the same inputs (visual and proprioceptive) along with a foreground object mask that indicates the target object. The only difference between the two models is the nature of the visual input: the active model observes log-polar images that are centered on the target object, while the passive model observes images of the bin from three static cameras (Figure 3). The static cameras are placed such that each can clearly view the contents of the bin and the gripper. This mimics a typical setup where cameras are positioned to view the robot workspace, with no special attention to any particular object or location in space. Using an instance mask to define the target object was previously done in [37] for example. Note that the generator is trained to also reconstruct the mask in this case, forcing the representation $r$ to preserve the target information.

Table 1 (Active-Target, Passive-Target) shows the evaluation performance of the active vs passive model with environment selected targets. We observe that the active model achieves 8% better performance. Figure 4 (yellow vs blue curves) shows that the active model is more sample-efficient as well.

The performance of the passive model (at 76%) is in line with the experiments in [44] on targeted grasping. All algorithms tested did not surpass an 80% success rate, even with 1M grasp attempts. The experiment above suggests that, had the robot been able to observe the environment in a more “human-like” manner, targeted grasping performance could approach performance on arbitrary object grasping.

The experiment above shows that active perception outperforms passive models in goal-oriented grasping. But can a robot learn where to look? Here we use the full version of the model with the learned fixation policy. Grasp rewards are given for picking up any object, as long as the object was within 10cm of the fixation point. This ensures that the model is only rewarded for goal-oriented behavior. In this setting, the model learns faster in the initial stages than in the targeted grasping case (Figure 4) and is slightly better in final performance (Table 1). This does not necessarily imply that this model is better at grasping, it could be due to the model choosing easier grasping targets. The latter may nevertheless be a virtue depending on the context (e.g., a bin-emptying application). This result indicates that active perception policies can be learnt in conjunction with manipulation policies.

Note that the full version of APR does not use additional information from the simulator beyond the visual and proprioceptive inputs. In contrast to Section A, the fixation point (and therefore the auxiliary reaching reward), is entirely self-generated. This makes APR directly comparable to the vanilla deep Q-learning learning algorithms studied in [44]. With 100K grasp attempts, the algorithms in [44] achieve approximately 80% success rate. We tested the model in the 4-DoF case, where it achieves an 85% success rate with 25K grasps (Table 1). Therefore, APR outperforms these previous baselines with four times fewer samples. [33] reported improved success rates of 89-91% with vanilla deep Q-learning after 1M grasps (though it was not reported what levels of performance were attained between 100K and 1M grasps). On the more challenging 6-DoF version, we achieve an 85% success rate with 70K grasps, but we have not yet extended the simulations to 1M grasps to allow a direct comparison with these results.

To examine effects of the log-polar image sampling and the representation learning, we ran two ablation experiments in the environment selected target setting (as in Section A). Figure 4 (red curve) shows that APR without representation learning achieves negligible improvement within the given amount of environment interaction. (Without the representation learning loss, we allow the the RL loss gradients to backpropagate to the encoder, otherwise it would not receive any gradient at all). The pink curve shows APR without log-polar images. The absence of the space-variant sampling impacts both the speed of learning and final performance (Table 1).