CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation

There have been various studies on image segmentation applied in various sub-fields, like salient object segmentation, affordance segmentation, etc. Models like BASNet [8], $U^{2}$Net [9] and DISNet [10] learn to encode multi-scale information. While the models achieve high spatial accuracy in boundary and structure, their predictions are fixed classes, and therefore restricted by categorical annotations. With the availability of pretrained models like vision transformers [11] and CLIP [6], modern segmentation methods tend to use those large pretrained models as foundation models and finetune customized decoders on downstream tasks. Models like CLIPSeg [7], CRIS [12] and LSeg [13] leveraged image-text representations from CLIP for language-driven segmentation tasks, while models like SegViT [14] and UNETR [15] inferred attention masks from ViT with transformer decoders to generate segmentation results. However, evaluation only attends to regional accuracy (e.g. IoU), while spatial accuracy of boundary (e.g. S-measure [16]) is ignored.

One traditional way to construct robot perception is to train vision models and perform image segmentation to predict affordances for robot control. In Do et al [3] and Chu et al [17], correct robot commands were inferred from affordances predicted by semantic models. K-VIL [18] processed keypoints from demonstrations to form geometric constraints usable by imitation learning agents. Other methods [19, 5, 20] combined affordance and keypoint predictions to guide robot control. With the introduction of language as control commands, methods like Nguyen et al [21] and Yang et al [22, 23] combined affordance prediction and vision-language models to construct modulated controllers that interface with language commands. However, training of the affordance segmentation models is expensive, and usually requires a large amount of annotated affordance data. Recently, foundation models like CLIP are utilized to train language-conditioned visuo-motor control policies. In this case, models are usually constructed as keypoint prediction modules like transporter [24], and affordances are learned unsupervised from demonstrations. Typical models include CLIPort [25] and Perceiver-actor [26], where CLIP is used to encode actionable language commands, and affordances are represented as keypoints to guide the policy learning. However, the predicted affordances are less defined and structured versus the traditional approach.

Given a referring expression as a prompt, and an image of a manipulation workspace with one or more objects, the goal is to segment the pixels that best match the expression. To refer to objects in the manipulation workspace in a contextual manner, the prompt describes important semantics like object naming, shape, functionality or affordances. Sample templates of prompts are visualized in Figure 2, and categorized as follows:

• •

  Affordance-enriched Prompts The prompt describes the functionality of an object, or parts of an object, in diverse, descriptive expressions.

• •

  Object-oriented Prompts The prompt refers to an object explicitly by its common naming, or implicitly by describing shapes, colors, or intentional usages.

The proposed CLIPUNetr, visualized in Figure 3, is a U-Net-like Encoder-Decoder network, which encodes the input image-prompt pairs, and decodes the segmentation probability maps.

Visual Encoder First, CLIPUNetr infers CLIP’s pretrained visual encoder. The encoder is a vision transformer (ViT) which takes an image input $I$ to capture semantic information. The image is first divided into $N$ patch embeddings $E_{I}$, where $N=HW/r^{2}$, $r$ is the patch resolution, $H$ and $W$ are the image height and width. A learnable class embedding $I_{cls}$ is concatenated with the patch embeddings, acquiring the token as $\{I_{cls};E_{I}\}$, and the learnable positional embedding $E_{pos}$ is added to the token. Then, the token is processed by $L$ Transformer layers, composed by alternating multi-headed self-attention (MSA) and MLP blocks. The token from the final Transformer layer is then processed by an embedding head, outputting the image embedding $y_{I}$:

Then, similar to UNETR [15], additional encoders are attached, which take the same image input $I$, and tokens from the CLIP visual encoder. For image $I$, one block of $3\times 3$ convolution, leaky ReLU and normalization layers are applied. For the token $\{I^{i}_{cls};E^{i}_{I}\}$, where $i\in\{3,6,9,12\}$, the class embedding $I^{i}_{cls}$ is removed, and $E^{i}_{I}$ is reshaped from $N\times D$ to $r\times r\times D$. The reshaped embedding feature is then upsampled by a scale factor of 2, followed by consecutive $3\times 3$ convolution, leaky ReLU and normalization layers for a number of times.

Feature-wise Linear Modulation To inform the decoder with information from the prompt, the last encoder feature is conditioned with text embeddings through Feature-wise Linear Modulation (FiLM) [27]. Given a prompt $p$, CLIP text encoder encodes the prompt with transformer blocks, outputting a category [CLS] token $T_{cls}$. The token is projected by an embedding head to generate the text embedding $y_{p}$. For the token $E^{L}_{I}$ from $L^{th}$ layer, FiLM learns an affine transformation to perform conditional scaling:

Segmentation Decoder To generate the resulting probability maps with high resolution, we enable feature scaling with side outputs in the construction of CLIPUNetr’s U-shaped segmentation decoder, inspired by BASNet [8]. First, the language-conditioned token $FiLM(E^{L}_{I})$ is upsampled with a scale factor of 2, followed by consecutive 3 $\times$ 3 convolution, leaky ReLU and normalization layers. Then, the output is upsampled and merged with the feature of the previous transformer output via skip connections. The concatenated feature is again processed by another consecutive 3 $\times$ 3 convolution, leaky ReLU and normalization layers. The process is repeated till the output features are upsampled to the original input resolution, acquiring $5$ decoding features. Then, we produce side output probability maps, and refine the side outputs to generate the final probability map. Each decoder feature will be fed into a 3 $\times$ 3 convolution, followed with upsampling and a sigmoid function. The process generates 5 side output probability maps $S_{5}$, $S_{4}$, $S_{3}$, $S_{2}$, $S_{1}$. The side outputs are concatenated and refined by a 1$\times$1 convolution and a sigmoid function, generating the final probability map $S_{fuse}$.

Hybrid Loss Similar to BASNet [8], we define loss as the summation of all outputs:

To align the robot end effector position to a target position in the manipulation workspace, we construct geometric constraints [1] from image geometry, which specify the alignment context. In detail, given a list of ordered keypoints $\textbf{f}=\{f_{1},f_{2},...,f_{n}\}$ in homogeneous coordinates, a geometric constraint $T$ is a task function that maps the keypoints into $\{0,1\}$, $T(f)=0$ when the robot configuration is aligned with the task description, $T(f)=1$ when the alignment is violated (e.g. unparalleled lines). In the scope of this paper, four basic types of task descriptions are considered:

Given the output probability map $M$ from CLIPUNetr and the prompt $p$, the process to compose geometric constraint in an eye-in-hand camera configuration is expanded as follows: First, the probability map is used as an image descriptor function $M(F)=(M(f_{1}),M(f_{2}),...,M(f_{k}),...,M(f_{K}))$, where $M(f_{k})$ denotes the probability score of $p$ at a keypoint location $f_{k}$, $F=\{f_{1},f_{2},...,f_{k},...,f_{K}\}$ defines the full image grid, where $K=HW$. Next, by filtering the probability score with a threshold of 0.5, we obtain a set of candidate keypoints $F_{candidates}$, where $f_{k}\in F_{candidates}$ and $M(f_{k})>0.5$. Then, PCA is used to analyze the variance of $F_{candidates}$, obtaining the principal point $f_{point}$ and principal lines $f_{line}$. Last, simple heuristics are used to compose the constraints, visualized in Figure 4. For a p2p task, the target position $f_{2}=f_{point}$ aligns to the end effector position $f_{1}$, heuristically set as $f_{1}=(W/2,4H/5,1)$. Similarly for a p2l task, the end effector position $f_{1}$ is set as $(W/2,4H/5,1)$, while the target line $l_{23}$ is set as $f_{line}$. For l2l and par task, the end effector orientation $l_{12}$ is defined as the vertical line passing through mid image center, while the target line $l_{34}$ is set as $f_{line}$.

UIBVS [28] defines the visual-motor control law, where a robot can be controlled to reach a desired joint configuration from image geometric inputs. The equation is denoted as:

Following the above definitions, Algorithm 1 shows our pipeline to interface UIBVS with CLIPUNetr. Given an image observation $I_{t}$ at time $t$, the human operator first decides the geometric constraints to compose, and then specifies a prompt $p$ to initiate the robot perception. For each $I_{t}$, CLIPUNetr will be inferred, generating the probability map $M_{t}$. $m$ geometric constraints are composed following the discussed strategy, and are inputted into the UIBVS controller, generating the joint commands to move the robot.

Datasets We use the publicly available PhraseCut and UMD+GT datasets for experiments. PhraseCut dataset [29] contains 340,000 phrases with associating regional referring expression prompts. UMD+GT dataset [5] contains 30,000 RGBD images of 104 objects, originally annotated with 6 fix-class affordance labels. To generate referring expressions for UMD+GT dataset, we manually re-annotate the images based on 39 diverse prompt templates. In training, we randomly sample one of the prompts, and in testing, we replace the affordance label with one object-oriented prompt, and one affordance-enriched prompt. Prediction is done twice and we take the average metric scores of the two predictions. To evaluate object boundary and structure on both datasets, we use four metrics studied in salient object segmentation: Mean Absolute Error (MAE), Structure measure, weighted F-measure, and max F-measure.

Robot Control The robot control is evaluated offline and online. In offline setup, 19 robot manipulation tasks of moving and grasping, controlled by the classical manual clicking interface [1] and performed by a Kinova Gen3, are recorded using an eye-in-hand Intel Realsense D405 camera. Masks are annotated and evaluated every ${10}^{th}$ frame.

In online setup, CLIPUNetr is integrated as a perception module to replace visual tracking with manual clicking. We compare the modified interface against the classical interface [1] by completing 12 robot manipulation tasks of moving and grasping with 5 food objects (apple, red pepper, lemon, carrot, banana), 2 utility objects (tennis ball, umbrella) and 4 marker pens. 3 attempts are allowed for the user to complete a task. The task is successful if the robot approaches and grasps the object without falling. We report the success rate for the three categories of tasks.

Implementation Details The model is implemented using PyTorch, using the same hyperparameters from CLIPSeg [7]. A batch size of 32 is used to train the models on a single Nvidia V100 GPU. For UIBVS control, joint 1, 2, 6 and 7 are used for table-top manipulation, and the implementation is done in ROS. A $\lambda$ of 0.05 is set for Broyden’s update. Joint angle commands are published in 1 Hz to move the robot end effector in small amounts, until convergence.

Qualitative Evaluation Table I shows the qualitative results of referring expression segmentation on PhraseCut and UMD+GT datasets. For fair comparison, an reimplementation of CLIPSeg is trained following our training regime (CLIPSeg*). In summary, CLIPUNetr is able to outperform CLIPSeg on both PhraseCut dataset and UMD+GT dataset, reflecting CLIPUNetr’s capability of generating good quality predictions. For UMD+GT dataset, CLIPUNetr surpasses the performance of AffKp, a supervised network trained only with fix-class labels. This proves that leveraging image-text representations from CLIP is beneficial for learning.

Quantitative Evaluation To further illustrate the superior performance of CLIPUNetr, Figure 5 visualizes the prediction results from CLIPUNetr vs. CLIPSeg. As shown, predictions from CLIPSeg contain checkerboard artifacts, caused by abruptly upsampling low-resolution decoding features in the decoder. This validates the effectiveness of CLIPUNetr to learn multi-scale information, thus acquiring predictions with more accurate object boundaries and finer structures.

We validate the effectiveness of each key component used in CLIPUNetr in two parts: architecture ablation and loss ablation. Table II presents the results.

Architecture Ablation We take CLIPSeg as the base network, and first extend its decoder to use the same features from layer 3, 6, 9 and 12, denoted as CLIPSeg-NoSides. Then, we extend the decoder with side outputs, denoted as CLIPSeg-Sides. As shown, the model with side outputs achieves superior results.

Loss Ablation We take CLIP-Deconv as the base network, where simple MLP and deconvolution layers are inferred to decode segmentation outputs. CLIP-Deconv is first trained with only binary cross entropy, denoted as CLIP-Deconv-BCE. Then, the training is extended with the hybrid loss, denoted as CLIP-Deconv. In summary, CLIP-Deconv achieves superior results, validating the fact that using hybrid loss is beneficial in learning structure information.

Offline Evaluation Table III presents the qualitative results of CLIPUNetr with robot data. Our CLIPUNetr achieves superior performance versus CLIPSeg, being able to handle complex motions in unseen robot demonstration videos. Additionally, quantitative comparisons are presented in Figure 6. Versus CLIPSeg, where predictions are plagued by checkerboard artifacts, CLIPUNetr is able to generate predictions with good quality object boundary and fine structure. This ensures the viability to use CLIPUNetr for robot perception. Additionally, we attach the augmented prompts used to generate the segmentation results. Thanks to the prompt templates used in training, CLIPUNetr is able to better handle diverse prompts containing both affordance and object information. This allows the users to specify prompts in diverse ways, increasing the flexibility of the interface.

Online Evaluation The success rate of robot control over three categories of objects are presented in Table IV. In summary, perception with CLIPUNetr shows comparable performance versus the classical visual interface, being able to successfully segment the unseen daily living objects and perform moving and grasping actions over them. For classical interface, 1 out of 12 tasks fails as a result of poor visual tracking performance. With CLIPUNetr in robot perception, manual clicking can be successfully avoided. And the robot can successfully utilize results of referring expression segmentation to compose geometric constraints and complete the designated manipulation tasks.