Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

Language-conditioned robot manipulation refers to the use of natural language instructions to guide robotic actions. Natural language instructions allow non-experts to interact with robots through intuitive commands and enable robots to generalize to various tasks based on natural language input [10]. Recent advancements in language-conditioned embodied agents have leveraged Transformers [11] to enhance performance on multi-task settings. One category of recent approaches is language-conditioned behavior cloning (BC), where models learn to mimic demonstrated language-conditioned actions and output dense action sequences directly. 3D BC methods [9, 12] trained from scratch perform well on specific environment, while lacking of generalization ability across environments. Vision-language-action (VLA) models [3, 5, 13] built on pre-trained large language models (LLMs) show capacity to transfer on novel objects and task settings, but need additional fine-tuning when being deployed on new environments and robots. Another line is to create high-level planners based on LLMs [14, 15, 16], which output step-by-step natural language plans according to human instructions and environmental information. These methods show better generalization ability across environments, leveraging the reasoning and generalization ability of LLMs on language instructions and environments. However, there is still a gap between generated natural language plans and low-level robotic execution, requiring an extra step to score potential actions or decompose plans into relevant policies [17].

Code-as-Policies [18] proposes that executable code can serve as a more expressive way to bridge high-level task descriptions and low-level execution. Atomic skills to perceive 3D environments and plan primitive tasks are provided in predefined API libraries. LLMs process textual inputs and generate executable policy code conditioned on the API libraries [17, 18, 19, 20]. However, these methods rely solely on linguistic inputs, requiring detailed descriptions of environments and instructions as textual inputs, which limits their generalization and visual reasoning ability across environments. RoboCodeX [6] utilizes large vision-language model (VLM) to decompose multimodal information into object-centric units in a tree-of-thought format. Nevertheless, it relies on manually-built simulation environments and human-annotated data, which lacks environmental richness and is expensive for scaling up. Different from previous works using language-only LLMs, RoboPro enables visual reasoning ability and follows free-form instructions in a zero-shot manner. Furthermore, an automatic and scalable data curation pipeline Video2Code is developed to synthesize runtime code data from extensive videos in-the-wild in a quite efficient and low-cost fashion.

We consider language-guided robotic manipulation where each task is described with a free-form language instruction $I$. Given RGBD data from the wrist camera as the observation space $O_{t}$ and robot low-dimension state $s_{t}$ (e.g., gripper pose at current time $t$), the central problem investigated in this work is how to generate motion trajectories $T$, where $T$ denotes a sequence of end-effector waypoints to be executed by an Operational Space Controller [21]. However, generating dense motion trajectories at once according to the free-form instruction $I$ is quite challenging, as $I$ can be arbitrarily long-horizon and would require comprehensive contextual understanding. Policy code generation methods map long-horizon instructions to a diverse set of atomic skills, leading to rapid adaptation capabilities across various robotic platforms. With comprehensive contextual understanding and advanced visual grounding capabilities, large vision-language models can function as intelligent planners, translating the task execution process into generated programs due to their robust emergent capabilities.

To prompt vision-language models (VLMs) to generate policy code, we assume a set of parameterized skills with unified interface, which is defined as the API library $L_{API}$. $L_{API}$ can be categorized into perception module $L_{per}$ and control module $L_{con}$ based on the API’s role in task execution process. $L_{per}$ is tasked with segmenting the task-relevant part point cloud $\Pi_{I}$ and predicting the physical property $\phi_{I}$ of relevant objects, while $L_{con}$ predicts the contact pose of the gripper and generates the motion trajectory $T$ based on the output of $L_{per}$ and the current robot state $s_{t}$:

With the visual observation and the language instruction, VLMs generate executable policy code $\{\pi_{i},p_{i}\}_{i=1}^{N}$ conditioned on the API library $L_{API}$, where $\pi_{i}$ denotes the $i$-th $L_{per}$ or $L_{con}$ calls and $p_{i}$ represents corresponding parameters for API calls. Each API call generates a sub-trajectory sequence $\tau_{i}$ of arbitrary length (the length is $\geq$ 0). All sub-trajectory sequences $\{\tau_{i}\}_{i=1}^{N}$ are then concatenated to form the final complete motion trajectory $T$. The whole generation process is formulated as:

Explainable API calls generated by VLMs connect the observation and high-level instructions to low-level execution, enabling the capacity of zero-shot generalization in free-form language instructions and across different environments. Obviously, training such VLMs to perceive environments, follow instructions and generate executable code will inevitably require a vast amount of diverse and well-aligned robot-centric multimodal runtime code data, which poses a significant challenge.

Videos are widely available raw data sources for runtime code data synthesis. Extensive operational videos naturally provide low-level details of performing tasks such as ”how to pour tea into a cup”, which inherently contain necessary procedural knowledge for runtime code data. Despite their favorable diversity and considerable quantity, it is still an under-explored and challenging problem how to collect executable policy code from demonstration videos efficiently. To this end, we devise Video2Code, a low-cost and automatic data curation pipeline to synthesize high-quality runtime code data from videos in an efficient way. Although open-source or lightweight vision-language models exhibit promising performance on video understanding tasks, a performance gap remains when compared to code-domain large language models in handling complex code generation tasks. As depicted in Fig. 2, to combine the visual reasoning ability of VLM and coding proficiency of code-domain LLM, Video2Code adopts a two-stage strategy.

Plan extraction. The first stage is to extract robot-centric plans in natural language from instructional videos. These instructional videos are filtered from DROID [8], a large-scale robot manipulation dataset with 350 hours of interaction data across 564 scenes, 86 tasks, and 52 buildings. We extract 50k independent instructional videos with at least one free-form human instruction and further clip each video into 16 key frames. After that, we use Gemini-1.5-Flash [22] as the Draft VLM to generate a brief list of actions for human instruction with these key frames as reference. As shown in Fig. 2, the Draft VLM generates a step-by-step robot-centric plan from an instructional video to ”stack the cups together”. The generated natural language plans contain knowledge and habit of human to follow free-form embodied instructions, and key visual information is extracted automatically from the instructional video.

Policy code generation. After plan extraction, we use Code LLM DeepSeek-Coder-V2 [23] to ”translate” these natural language plans into executable code. A complete prompt fed into the Code LLM includes API definitions, the natural language plan, and auxiliary part containing rules to follow. In the API definitions part, parameterized API functions are classified into two categories as formulated in Sec. III-A: perception module, and control module. For each of these API functions, we provide API definitions and descriptions to demonstrate their usage. Auxiliary part contains prefix, third party tools, and rules to follow, similar to previous practices in RoboCodeX [6]. Natural language plans accompanied with original human instructions are attached at the end of the prompt. As shown in Fig. 2, step-by-step decomposed natural language plan guides the Code LLM to generate high-quality policy code in a Chain-of-Thought format. As for API implementation, we use GroundingDINO [24] and AnyGrasp [25] to get the bounding boxes and grasp preferences, respectively. Besides, we provide heuristic implementation for compositional skills. We finally collect 115k runtime code data with task descriptions and environmental observations using Video2Code for supervised fine-tuning.

RoboPro introduces a unified architecture that seamlessly integrates visual perception, instruction following, and code generation by leveraging end-to-end vision-language models (VLMs). The unified pipeline eliminates the potential loss of critical information during intermediate steps and enhances computational efficiency during inference. Powered by well-aligned image-instruction-code pairs from Video2Code,  RoboPro demonstrates strong capabilities in executing free-form instructions grounded in visual observations.

Model architecture. As shown in Fig. 3, RoboPro has a vision encoder and a pre-trained LLM. They are connected with a lightweight adaptor layer consisting of a two-layer MLP. Specifically, the vision backbone first encodes the image into a sequence of visual tokens. After that, the lightweight adaptor is designed to project visual tokens onto embedding space of the LLM. In addition, we provide the API definitions and the user instruction as the text inputs. The visual and text tokens are directly concatenated and then fed into the LLM. The LLM are trained to generate the runtime code based on the visual inputs and task description.

RoboPro is designed to reason on multimodal inputs and generate executable policy code for robotic manipulation. Thus, two key factors for the choice of its components are the ability of visual reasoning and the quality of policy code generation. RoboPro adopts SigLIP-L [26] as the vision encoder, which yields favorable performance on general visual reasoning tasks. For the base LLM, a code-domain LLM, CodeQwen-1.5 [27], is utilized, which shows state-of-the-art performance among open-source code models. The model architecture and working process of RoboPro are illustrated in Fig. 3.

Training. The training procedure of RoboPro consists of three stages: visual alignment, pre-training, and supervised fine-tuning (SFT). We first train a lightweight adaptor layer while freezing the vision encoder and LLM with LLaVA-Pretrain [28]. Then we pre-train the lightweight adaptor and the LLM on a corpus of high-quality image-text pairs [29]. For supervised fine-tuning, the 115k runtime code data generated by Video2Code (as noted in Sec. III-B) are used. To avoid overfitting and enhance visual reasoning ability, a general vision language fine-tuning dataset (LLaVA-1.5 [28]) is also involved during the SFT process. Thus, RoboPro is trained to follow free-form language instructions and perceive visual information to generate executable policy code for robotic manipulation.

Zero-shot generalization across instructions, tasks and environments is a significant challenge for robotic learning. Policy code generation methods leverage adaptability of large language models to generate code plans across tasks and scenarios in a zero-shot manner. RoboPro, a multimodal policy code generation model, is trained on real-world data. To validate zero-shot generalization of RoboPro across environments, we evaluate the performance on two distinct simulation environments (RLBench, LIBERO), and carefully verified that scenes, tasks and instructions during testing were entirely unseen during the training phase.

The baselines can be categorized into two groups. Our primary comparison tagets are other code generation methods. They first output robot-centric policy code, then execute it with provided APIs. We evaluate their zero-shot performance on RLBench and LIBERO. CaP [18] equips large language model with the ground-truth textual scene descriptions, containing object names, attributes, and instructions, to generate executable code. Following their paper, we implement CaP with GPT-3.5-Turbo (gpt-3.5-turbo-0125). GPT-4o ([30], gpt-4o-2024-05-13) is the state-of-the-art multimodal model for various vision-language tasks. For RoboPro and GPT-4o, we require the model to directly generate the executable code given the image from the wrist camera, user instructions and API definitions. For a fair comparison, we adopt the same API library for these methods (i.e., CaP, GPT-4o, and RoboPro). Our API library shares similar design formulation as RoboCodeX [6], with detailed implementation in Appendix V-E. The methods from another group directly output actions while require supervised training when implement on different tasks and environments, e.g., behavior cloning methods, including PerAct [9] and OpenVLA [5]. We use popular training-based methods primarily as an upper bound for reference.

Another challenge for code generation methods lies in generalization across different robotic configurations, which often manifests as variations in the format and implementation of API libraries. Additionally, real-world tasks may require robots to perform unseen skills with user-specific preferences. Despite this issue has been largely under-explored in prior works, enhancing adaptability to diverse API implementations and novel skills is crucial for making code generation methods more scalable in real-world applications. We take an early step toward examining the robustness of code generation methods across varying API formats and user-specified skills.

To evaluate the performance of RoboPro in real-world scenarios, we conduct realistic experiments on a Franka Emika robot arm equipped with an Intel RealSense D435i wrist camera. As emphasized in Sec. III-A, long-horizon task decomposition and visual understanding capabilities are crucial for zero-shot generalization in language-guided robotic manipulation. To assess RoboPro’s performance in these aspects, we carefully designed 8 tasks, ranging from short-horizon to long-horizon tasks, as well as tasks that require visual comprehension. For instance, RoboPro is required to select the object with ”wipe” affordance from the scene given instruction ”wipe the desk”. Additionally, to rigorously validate RoboPro’s generalization capability across different real-world scenarios, we ensure that each task consists of at least two variations in terms of object categories and physical properties (10 tests are run for each task). We select GPT-4o as an extra baseline on real-world environments. We also provide detailed real-world setup in Appendix V-C.

As shown in Table IV, RoboPro is able to achieve 72.5% success rate on average among all 8 tasks, which verifies RoboPro’s strong generalization ability in real-world scenarios without any specific fine-tuning. We also observe RoboPro exhibits impressive emergent ability in visual reasoning. For example, when asked to wipe the desk, RoboPro will choose the appropriate tool (the sponge) among irrelevant objects, and grasp it to wipe water on the desk. On directional moving and one-turn tasks, GPT-4o shows comparable performance with RoboPro (Move in Direction, Setup Food), while RoboPro shows better performance on tasks requiring visual understanding or target identification (Stack on Color, Tidy Table), which yields conclusions consistent with the experimental results on simulation platforms.

We conduct extensive ablation studies to analyze how Video2Code and the scaling of training data impact performance on manipulation tasks. Additionally, we explore the effect of the choice of base LLM within RoboPro. We provide detailed ablation results in Appendix V-D.

Effectiveness of Video2Code. We compare our model trained with and without Video2Code on manipulation tasks. For a fair comparison, we only remove Video2Code from the fine-tuning stage for the baseline, that is, the 115k runtime code data are excluded and only the general vision language fine-tuning dataset is used during the SFT process, as described in Sec. III-C. The first two rows of Table V show the comparison of the two settings. It is found that the Video2Code generated data have significantly improved the performance on both RLBench and LIBERO by a gain of 42.3% and 45.4%, respectively, which indicate Video2Code’s efficacy in enhancing the ability of skills utility and instruction following.

Scaling of training data. We further conducted an ablation study on the dataset size. Specifically, we trained RoboPro using 115k runtime code data collected from DROID, varying the dataset proportion of SFT stage to 25%, 50%, 75%, and 100%. We evaluated the models trained with different sizes of dataset on RLBench and LIBERO. As shown in Fig. 5, results indicate that RoboPro adheres to the scaling law: training with just 25% of the data already yields a well-performing model, while its performance continues to improve as the dataset size increases. Video2Code is efficient for scaling up of runtime code data, which deserves further exploration to involve in more robotic demonstrations.

Choice of base LLM. For the components of the RoboPro framework, we evaluate its performance using different code-domain base LLMs, specifically DeepSeek-Coder-6.7B-Instruct [33] and CodeQwen-1.5-7B-Chat [27]. As shown in Table V, the version of RoboPro trained with CodeQwen-1.5-7B-Chat consistently outperforms the one trained with DeepSeek-Coder-6.7B-Instruct across both manipulation tasks. These results demonstrate that employing a more powerful base LLM for code generation task can consequently enhance performance in both tasks.

RLBench is a simulation platform set in CoppelaSim [34] and interfaced through PyRep [35]. Robotic models control a 7-dof Franka Panda robot with a parallel gripper to complete language-conditioned tasks. RoboPro is evaluated on 9 tasks from RLBench [31]. Modification on these tasks is consistent with PerAct [9]. Each task in RLBench is provided with several variations on language instructions describing the goal. In order to validate RoboPro’s adaptation ability across various and vague instructions, we pop out an instruction from the language template list for each episode during evaluation instead of just using the first language template. Detailed descriptions and modification for each task in RLBench are provided below.

In this section, we provide a detailed description of 8 tasks selected from the LIBERO-100 dataset. Each task is associated with a specific language instruction, with the task ID and corresponding instruction shown in Table VI. The tasks ”Turn on Stove” and ”Close Cabinet” are taken from LIBERO-90, which focuses on testing atomic skills and environmental understanding. The remaining tasks are more complex, requiring multi-step execution, and are selected from LIBERO-10. These 8 tasks challenge RoboPro to comprehend diverse visual environments and follow extended language instructions. As illustrated in Fig. 6, the tasks encompass a wide range of robotic capabilities, including object selection, spatial reasoning, scene comprehension, and long-term execution.

The real-world experiments are implemented on a Franka Emika Panda robotic arm with a parallel jaw gripper, as shown in Figure 7. We use an Intel RealSense D435i camera to provide RGB-D input signals under the camera-in-hand setting. Easy-handeye ROS package is used to calibrate the extrinsics of the camera frame with respect to the robot base frame. For robot control, we use the open-source frankapy package to send real-time position-control commands to robot after receiving the control signals from RoboPro. During test time for each task, natural language instructions, extrinsic matrix, intrinsic matrix, current environment observation in the form of RGB-D image, and the low dimensional state of the robot are prepared for RoboPro to generate corresponding 6-DOF action trajectories. Examples of all 8 real-world tasks with natural language instructions are illustrated in Fig. 8, ranging from short-horizon to long-horizon tasks, as well as tasks that require visual comprehension.

We provide detailed results of our ablation study on two simulation platforms in Table VIII and Table VII. As shown in the first two rows, the results on tow simulation platforms improved significantly after trained with Video2Code runtime data, which indicates the effectiveness of 115k visual-aligned code data collected from video demonstrations. As for the choice of base LLM, the version of RoboPro trained with CodeQwen-1.5-7B-Chat consistently outperforms the one trained with DeepSeek-Coder-6.7B-Instruct across both manipulation tasks, which demonstrate that a better choice of code-domain LLM can consequently improve performance on robotic tasks.

The API libraries provide interface for code generation, enabling low-level execution on different embodiments and tasks. As shown in Listing V-E, the provided APIs can be divided into perception modules and action modules. The perception modules are primarily implemented for grounding of object positions and physical properties. For operations involving objects with similar properties, such as ”stacking identical blocks”, we provide the output in the form of a list of candidate objects. The action modules include direct motor execution via ROS, such as open_gripper(), rotate(), and move_to_pose() with 7-DoF action inputs. Additionally, they provide motion planning APIs for generating trajectories around joint axes or predefined sequential paths to support skill implementations. For each of the APIs, detailed explanations and definitions of functions and skills are provided in the prompt, which has proven to be a more effective approach for in-context learning of policy code generation. We also make an effort to avoid hard-coding in the prompts, and use parameterized preferences and output of other APIs as input for grounding and motion generation, which makes the system flexible and adaptive to different scenes and objects. As mentioned in Sec. IV-B1 , we discussed the influence of API Renaming and API Refactoring to the performance of RoboPro, where variations of API implementations and prompts are depicted in Listing V-E and Listing V-E .