Training Robots without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfer

This framework uses a dual-arm robot system with two UR5 robots. In the authors’ previous study (e.g., [4]), this system was operated in teleoperation mode. By contrast, in the present study, demonstration data are generated in master-only mode. In this mode, the human demonstrator uses the fingertip of the master controller (Fig. 2(a)) to execute tasks. The controller collects joint angles from its encoders mounted on a structure that has DH parameters identical to those of the UR5 robot. The shared DH parameters between the master and robot facilitate the transfer of motor skills from humans to robots. The idea behind this is that the physicality of the robot has a significant influence on motor skills, and constraining the physicality of the human operators through the use of the master controller allows them to develop motor skills suitable for the robot. This helps to ensure that the kinematics of the system can be solved. Therefore, the robot can easily reproduce human behavior by following the recorded encoder values. The end effectors on the robot and master are identical: one degree-of-freedom grippers (Fig. 2(b)). Additionally, an F/T sensor (Leptrino, PFS 030YA301) is placed between the fingertip and rod so that the F/T of the fingertip can be measured.

The human operator observes a stereo image from a stereo camera (ZED-Mini, StereoLabs) using a head-mounted display (HMD). The stereo cameras are mounted in identical relational positions on both the robot and master sides (Fig. 2(c)). For the master, this location is just above the operator’s head. This shared camera mount position minimizes the visual gap between the master and robot caused by different viewpoints. The developed pan-tilt camera system can control its direction, which can be used in future studies.

The proposed controller and UR5 robot have identical DH parameters. However, the authors found that there is still an error because of deflection caused by gravity. Therefore, calibration is required to reduce the error.

Motion capture is used for calibration. The accurate end effector position and rotation are measured using the Optitrack motion capture device. The master controller is moved randomly by the human operator while it records the joint angle, and the motion capture device measures the exact position simultaneously, which is then played back in the robot. A transformation from the motion-captured master position/rotation $M^{mocap}$ to the robot position/rotation $R^{mocap}$ transforms the position/rotation $M^{enc}$ calculated from the joint angle into the calibrated robot state $R^{enc}$.

The position $p$ and orientation $o$ are separated, and the homogeneous transformation matrix $A_{p}$ and rotation matrix $A_{o}$ are calculated by learning the least-squares solution:

Therefore, the calibration of robot state $R^{enc}$ is calculated using the following equations:

and the calibrated robot state $R^{enc}$ is used for neural network model training.

Gaze-based visual attention was proposed in [4]. This method reproduces the human’s gaze, which can be acquired naturally during demonstration using an eye tracker on the HMD (Fig. 1). The gaze-based visual attention is used to minimize the visual gap between the demonstration data from the master and the test on the robot. Because the human gaze is correlated with the target object position [13, 15], this method can eliminate distractions from the human hand. Therefore, the visual appearance of both the robot and master are similar in foveated vision (Fig. 1 (c)).

This method first collects the two-dimensional gaze coordinates using the eye tracker during the demonstration. Then, mixture density network (MDN)-based [16] gaze prediction architecture [17, 4] is trained to reproduce the probability of the gaze position based on the global vision. The trained MDN predicts the gaze position in the test environment using robots. Foveated vision is an area around the gaze position cropped from the global vision. This visual attention process can suppress visual distractions from task-unrelated objects or background while fully providing information about the task-related object or area.

The details of MDN architecture are as follows: First, a series of five convolutional neural networks (CNNs) and rectified linear units (ReLUs) followed by SpatialSoftmax [18] extract visual features, which are then processed by the MDN, which is composed of a Gaussian mixture model (GMM) with eight Gaussians. Finally, this GMM is fit to the coordinates of the gaze using [2]

where $p^{i},\mu^{i},$ and $\mathbf{\Sigma}^{i}$ represent the weight, mean, and covariance matrix of the $i$th Gaussian, respectively, and $g$ denotes a target gaze coordinate. The gaze position is used to crop $128\times 128$ foveated vision from $256\times 256$ global vision.

It is important to note that kinesthetic teaching is unable to utilize the advantage of gaze-based attention as the human operator must control the dual-arm robot from the opposite side of the table, which creates confusion when attempting to observe the view from the robot’s camera through the HMD.

Because cap-reaching behavior requires precise manipulation, the dual-action architecture proposed in [1] is used. Based on a physiological study [19] in which the human’s reaching movement was divided into two distinctive systems, the dual-action architecture divides the entire robot trajectory into the global action and local action. This architecture first detects whether the end effector is in foveated vision [20]: if not, the global action, which is the fast action that delivers the hand to the vicinity of the target, is used; and if so, the local action, which is a precise and slow action when the hand is near the target, is activated (Fig. 1b). This enables high-precision manipulation, such as needle threading [1] or banana peeling [20].

In the dual-action architecture, each step is labeled as global-action, local-action from the demonstration data and used to train the global-action network or local-action network, respectively. A human manually annotated some demonstrations to train a CNN-based classifier to classify the remaining demonstrations. The classifier’s output was verified by a human (The details can be found at III-C1). The global and local-action networks are distinctively trained using the annotated data. During the test, a binary classifier, which inputs foveated vision to classify the global and local actions, is used to choose which action network to use.

The global-action network inputs stereo foveated images, the robot state, and the stereo gaze position to output the action (Fig. 4(a)). The local-action network inputs foveated vision to output the precise local action. Because the robot’s kinematic state is not used for input, this network can accurately predict the action regardless of the kinematic gap between the demonstration data from the master and the test environment of the robot (Fig. 4(b)).

The action is defined as the difference between the seven-dimensional robot state (three-dimensional position, three-dimensional orientation, and a gripper angle) at the next step and the current robot state. The orientations of the robot state input into the neural network models are decomposed using cosine and sine to prevent the drastic change of the angle representation; therefore, it is ten-dimensional. The global-action network uses a series of CNN, ReLU, and max-pooling layers with the stride equal to 2. The local-action network uses one CNN layer with an ReLU followed by nine residual blocks [21], and a max-pooling layer is inserted in every two residual blocks with the stride equal to 2. The CNN layer uses a $3\times 3$ filter with 32 channels. The multilayer perceptron (MLP) module is a series of fully connected layers (FCs) with a node size of 200 and an ReLU between the FCs. The action is optimized using $\ell_{2}$ loss.

The force feedback network uses the dual-arm F/T sensory data and foveated image to compute the action command (Fig. 4). This architecture processes F/T sensory data with the Transformer-based self-attention architecture [14], inspired by a study [2] in which researchers implemented the self-attention architecture to reduce distractions in dual-arm somatosensory inputs. In the present study, the Transformer is used to attend to useful F/T sensory information from both arms and reduce distractions from other F/T sensory information. The feed-forward network also processes the foveated image using the same CNN architecture as the global-action network. The output of the CNN and Transformer is concatenated and then processed with the MLP to predict the action command.

Fig. 5(a) illustrates part of the calibrated trajectory of the left arm. After calibration, the master controller’s position $M_{p}^{mocap}$ is translated into $A_{p}M_{p}^{mocap}$, which is much closer to the actual robot position $R_{p}^{mocap}$. Fig. 5(b) compares the mean Euclidean distance between the master and robot, $||M_{p}^{mocap}-R_{p}^{mocap}||$ (blue), and after calibration, $||A_{p}M_{p}^{mocap}-R_{p}^{mocap}||$ (orange). The result indicates that the overall error decreased after calibration. Particularly, the error on the $z$-axis, which was mainly caused by the deflection of the upper arm because of gravity, occupied the most significant error between the master and robot, but decreased after calibration. This calibration was robust because the main error on the $z$-axis was caused by gravity, which is almost constant on the surface of the earth.

Our previous robot system successfully solved various dexterous manipulation tasks (e.g., [1, 2, 20]) by demonstration with teleoperation. However, we encountered limitations in achieving contact-rich tasks with strict geometrical constraints, which even human teleoperators could not accomplish. The proposed M2R framework targets bottle-opening, a representative example of such tasks that requires precise gripper rotation with exact ($x$, $y$) translation to compensate for the offset between the cap and gripper. The M2R framework enables the demonstrator to directly feel force-feedback from the exerted action, facilitating the discovery of motor skills suitable for the robot in contact-rich tasks.

We trained and tested our model with three different plastic bottles (bottle-A,B, and C, Fig. 6) We used bottle-A for ablation studies and other analysis. Relatively small number of demonstration sets of bottle-B and C (Table III ) are added to and re-trained to evaluate the adaptation ability of the proposed model on new bottle with less dataset. Bottle-B has the same size and shape but different texture with the bottle-A. Bottle-C has the different size, shape and texture, thus require more adaptation ability than bottle-B.

The bottle-cap-opening task was segmented into the three subtasks GraspBottle, GraspCap, and Rotate described in Table II. GraspBottle $\rightarrow$ GraspCap was segmented because the robot changes its manipulating arm from right to left, and then GraspCap $\rightarrow$ Rotate because rotation requires force feedback, whereas GraspCap does not. Each subtask used a different neural-network architecture. First, GraspBottle is a simple grasping movement that the global-action network can accomplish. GraspCap is high-precision grasping, which requires a dual-action network that combines global and local-action networks. Finally, Rotate requires force feedback; therefore, a force feedback network was used.

The objective of the test was to evaluate whether the robot could grasp the bottle with its right hand, grasp the bottle cap with its left hand, and rotate the cap once without the cap slipping or bottle tilting. The test was repeated 18 times for different initial bottle positions, which were reproduced from the initial positions recorded in the randomly sampled validation set. The bottle cap was manually rotated approximately 90 degrees from the completed closed state because opening the completely locked cap required too much torque for the robot.

This study proposed the gaze-based dual-action approach with force feedback. Four types of neural network architecture were compared to evaluate the validity of each component:

• •

  DA-force: DA-force uses both F/T sensory feedback and gaze-based dual-action (DA) architecture.

• •

  No-force: to validate the necessity of an F/T sensor, No-force does not input F/T sensor information while rotating the cap.

• •

  No-DA: this architecture validates whether dual action is effective in the M2R framework. In this architecture, global and local actions are not distinguished. Therefore, the policy in each subtask is trained and inferred only from the global-action network.

• •

  No-gaze: DA-force inputs foveated images to minimize the visual gap. No-gaze inputs global vision to investigate whether the visual gap is indeed reduced by the gaze. As proposed in [3], the global image was processed using SpatialSoftmax after five layers of CNNs and ReLU.

Fig. 3 illustrates that the proposed DA-force successfully executed the bottle-cap-opening task on bottle-A. Among all the tested model architectures, DA-force had the best final cumulative success rate of $83.3\%$ by combining DA, F/T sensor input, and gaze (Fig. 7).

Second, No-force, which did not use F/T sensory information, executed accurate reaching and grasping; however, poor manipulation during Rotate caused a slip during cap opening (Fig. 8(a)), which resulted in a lower final success rate than DA-force.

Third, No-DA failed at accurately reaching the cap because it lacked the dual-action system (Fig. 8(c)). Some may argue that the gripper touched the bottle cap; therefore, simply closing it may have led to successful grasping. However, the bottle collapsed when the gripper was closed manually.

Finally, No-gaze demonstrated low accuracy for both GraspBottle and GraspCap (Fig. 8(b)). This low accuracy was because the global vision-based policy was weak against distractions caused by the visual gap. The visualized example of two-dimensional visual features extracted by SpatialSoftmax on the global vision processing network demonstrated the difference in the feature coordinates between the demonstration data collected by the master (validation set) and the robot data collected by DA-force during the test (Fig. 9). Some features focused on similar areas (Fig. 9(a), 9(b)), whereas others focused on different areas (Fig. 9(c), 9(d)). This difference resulted in errors in the action output computed from the MLP layer. Therefore the architecture using global vision failed to predict the accurate action.

Fig. 10 shows examples of F/T sensory values and the attention. First, sensory values of successful and failure trial were separately gathered from trials of DA-force and No-force. Then, the attention rollout [23] of the Transformer from the force feedback network was computed. Then, the two-dimensional attention-rollout was averaged on each channel. The result demonstrates that if the robot failed to Rotate (Fig. 10(a)), the force of the $z$-axis on the right arm (R:fz) increased during rotation (Fig. 10(b), time step $5-11$), and the Transformer captured such an abnormality (Fig. 10(c)). By contrast, in the successful trial, the force of R:fz did not increase (Fig. 10(d)); therefore, the Transformer did not give any alert (Fig. 10(e)). Therefore, the F/T sensor provides prediction whether the robot would fail, and the proposed Transformer-based method can capture such alert.

Table IV represents the results comparison of all bottles. Even though the bottle-B and C are trained with smaller number of dataset, the neural network well adapted to those new bottles.