Learning Predictive Visuomotor Coordination

Human visuomotor coordination is fundamental to action planning and execution, integrating first-person perspective visual perception, head orientation, and proprioception to enable fluid, goal-directed movements. Unlike purely reactive control, it operates predictively, allowing for smooth and adaptive actions despite sensory and neural delays [41, 48, 45, 50, 3]. Research in neuroscience and motor control highlights that head movement, gaze, and body posture work together to anticipate motion trajectories [31, 30, 5]. Head orientation provides a stable spatial reference, aligning sensory input with motor execution, especially in dynamic environments[16]. Proprioception further refines this process, allowing the brain to track limb positions and adjust movements accordingly[44]. These multimodal cues are tightly coupled—humans naturally orient their head and upper body before executing reaching, stepping, or object manipulation tasks [24]. This predictive integration is essential for both action preparation and real-time adaptation[30, 44].

Inspired by these biological mechanisms, our work explicitly integrates head orientation, gaze, and egocentric perception into motion forecasting, not as independent signals but bridging neuroscience insights with real-world human movement prediction as a structured visuomotor coordination representation.

Existing human motion forecasting methods often focus on isolated sub-components rather than their integration. Gaze forecasting predicts future fixation points from past gaze trajectories, often using egocentric video [52, 28, 29], but it does not model how gaze directs motor actions. Hand trajectory forecasting predicts hand motion in first-person views, typically for object interactions [35, 22, 14], yet it largely ignores head movement and body posture, which naturally influence hand coordination. Full-body motion forecasting predicts skeletal motion based on past joint positions [13, 40, 42, 39, 19, 17, 8], but lower-body motion is often dictated by external terrain constraints rather than internal visuomotor coordination. In contrast, upper-body motion is directly linked to fine motor tasks, object interactions, and skill-based activities, where visuomotor coordination plays a critical role. Thus, we focus on upper-body dynamics, ensuring our approach remains independent of environmental priors while maintaining relevance across diverse skilled activities.

Egocentric vision has emerged as a key modality for studying human visuomotor coordination, as it provides a first-person perspective of perception and action. Unlike third-person views, egocentric video directly captures the relationship between visual attention, head movement, and motor execution, making it well-suited for modeling predictive visuomotor behaviors [52, 22, 32, 46]. This perspective is particularly valuable for understanding how humans coordinate gaze, head, and body movements in dynamic environments.

Another line of research incorporates environmental cues for motion prediction. Gaze-informed full-body forecasting has been explored in navigation tasks, using gaze to anticipate walking trajectories [27, 53, 49, 21]. However, these tasks are simpler, as locomotion follows scene constraints rather than requiring complex visuomotor coordination. Full-body forecasting in interactive activities integrates environmental affordances to predict human motion [53, 2], but such methods heavily rely on external cues like object locations or scene semantics, making them less generalizable to unstructured or unseen environments.

In contrast, our work addresses a more complex and generalizable problem by unifying head orientation, gaze, and upper-body motion as predictive signals for motion forecasting. By focusing primarily on internal visuomotor cues, our model learns a biologically grounded representation applicable across diverse activities, without explicitly relying on structured environmental priors.

Recent advances in visuomotor imitation learning have enabled robots to acquire complex skills from human demonstrations [12, 51, 47, 36, 38, 54, 26, 6, 1, 23]. Works like EgoMimic [25] explore direct learning from human video data, removing the need for explicit robot data collection. However, modeling human visuomotor coordination remains a challenge, as it involves intricate couplings between gaze, head, and body movements that reflect hidden decision-making processes [34].

To simplify learning, many existing methods instruct demonstrators to behave “robot-like” by minimizing natural head and body movements [4, 33, 55, 43]. While effective for policy learning, this approach loses the richness of natural human behavior, making it difficult to generalize beyond constrained demonstrations.

Our work contributes to this direction by providing a predictive model of human visuomotor coordination, capturing the structured dependencies between head pose, gaze, and upper-body motion. Unlike imitation learning approaches that focus on end-to-end policy learning, our model predicts natural visuomotor behavior from in-the-wild datasets [15]. By explicitly modeling visuomotor coordination, our framework can serve as a foundation for future imitation learning research, providing data-driven insights into human movement dynamics that can improve robot learning from human demonstrations.

To effectively capture human visuomotor coordination, we consider three essential components: the head pose, which establishes a spatial reference frame for movement; the gaze point, which serves as an indicator of visual attention and intent; and the upper-body joints, which provide motion cues relevant to interaction and task execution. These components collectively define the Visuomotor Coordination Representation (VCR) by encapsulating the key factors that drive human visuomotor behavior.

We formally define a visuomotor state: $S=\{H,G,U\}$, where $H=(\mathbf{p}_{head},\mathbf{R}_{head})$ represents the head pose with position $\mathbf{p}_{head}\in\mathbb{R}^{3}$ and orientation $\mathbf{R}_{head}\in SO(3)$. The gaze component $G$ is represented by the gaze endpoint $\mathbf{g}$, defined as $\mathbf{g}=\mathbf{p}_{head}+\lambda\mathbf{d}_{gaze}$, where $\mathbf{d}_{gaze}\in\mathbb{R}^{3}$ is the unit gaze direction derived from $\mathbf{R}_{head}$, and $\lambda$ is a predefined scalar controlling the gaze ray length. Finally, $U=\{\mathbf{j}_{i}\in\mathbb{R}^{3}\mid i=1,\dots,6\}$ represents the upper-body joint positions, including shoulders, elbows, and wrists. Fig. 2 visualizes an example of VCR from Nymeria [37].

Given a Visuomotor Coordination Representation (S) sequence $S_{t-\tau:t}=\{S_{t},S_{t-1},\dots,S_{t-\tau}\}$ and its corresponding egocentric video clip $E\in\mathbb{R}^{C\times T\times H\times W}$ , our goal is to predict the future visuomotor states $\hat{S}_{t+1:t+\Delta}$ over a prediction horizon of $\Delta$ steps. This task requires joint modeling of the temporal evolution of human motion, including head pose, gaze, and upper-body joint dynamics, based on past visual and kinematic observations.

Formally, we define the Visuomotor Forecasting Task as learning a function:

$$\hat{S}_{t+1:t+\Delta}=f(S_{t-\Delta:t},E_{t-\Delta:t}),$$

where $f(\cdot)$ models the transition dynamics of visuomotor states given historical motion and egocentric perception, from which enables anticipation of human visuomotor coordination in dynamic environments.

We canonicalize kinematic data to ensure consistency across time and spatial frames. To achieve this, we standardize all motion data by defining a stable reference frame based on the final observation. Specifically, we set the head pose $H_{t}=(\mathbf{p}_{\text{head},t},\mathbf{R}_{\text{head},t})$ in the last observed step to a canonical state, applying the transformation $H^{c}_{t}=\Phi(H_{t})$, where $\Phi$ normalizes the head pose by aligning its orientation to identity rotation $\mathbf{I}$ and positioning it at the origin $\mathbf{0}$. This transformation ensures consistency across time, allowing the model to learn visuomotor coordination independently of absolute head motion.

To preserve intra-state spatial consistency, the same transformation $\Phi$ is applied to the gaze endpoint $\mathbf{g}_{t}$ and upper-body joint positions $U_{t}$ through $\mathbf{g}^{c}_{t}=\Phi(\mathbf{g}_{t})$ and $U^{c}_{t}=\Phi(U_{t})$. This results in $S^{c}_{t}=\Phi(S_{t})$, where all visuomotor elements remain consistent relative to the canonical head frame, ensuring the model captures meaningful coordination patterns without being affected by head motion variations.

To preserve inter-state temporal consistency, we extend this operation to all preceding and future kinematic states. For kinematic states $S_{i}$ with $i\in[t-\tau,t+\Delta]$, each state is first transformed relative to $S_{t}$ before being mapped to the canonical frame. As a result, we have $S^{c}_{i}=T_{i\to t}(S_{i})\circ S^{c}_{t},$ where $S^{c}_{t}=\Phi(S_{t})$ is the canonicalized reference state. This ensures that all frames remain correctly aligned relative to each other, preserving both temporal and spatial consistency.

This process removes absolute head motion while preserving the relative relationships between the head, gaze, and upper-body joints. It enables the model to learn visuomotor coordination independently of viewpoint variations, improving robustness and generalization. An example of this processing pipeline is shown in Fig. 3. Note that $S^{c}_{t}$ is used in the experiments; however, for simplicity, we continue using the notation $S_{t}$ throughout this paper.

We first describe how we extract and fuse multimodal information to construct the conditioning feature for diffusion (Sec. 3.4.1), followed by the diffusion-based visuomotor prediction process (Sec. 3.4.2). Figure 4 provides an overview of our model.

To comprehensively evaluate our predictive visuomotor learning task, we adopt a diverse set of metrics that assess structural consistency, positional accuracy, and orientation alignment. These metrics are computed over the visuomotor representation $S=\{H,G,U\}$, as defined in Sec. 3.1.

To evaluate the effectiveness of our method, we compare it against two naïve interpolation baselines and two learning-based methods, including a Diffusion Policy model.¹¹1Our work is concurrent with EgoCast [11] and EgoAgent [7], but a direct comparison is not feasible due to differences in task definitions and input modalities, as well as the lack of publicly available implementations at the time of writing.

• •

  Constant Pose: assumes no future motion, directly copying the last observed visuomotor state for all future time steps. It serves as a lower bound, representing a scenario where no predictive modeling is performed.

• •

  Constant Velocity: estimates future motion using first-order linear extrapolation, assuming a constant velocity based on the last observed state transition. It provides a simple motion forecasting heuristic.

• •

  Transformer Model: is a standard Transformer-based sequence-to-sequence predictor that models motion evolution through self-attention. We concatenate pose sequences with visual embeddings extracted from ResNet18 and feed them into a 3-layer Transformer encoder, followed by a regression head to predict future visuomotor states.

• •

  Diffusion Policy (CNN-Based) [9]: formulates motion prediction as a denoising diffusion process, where a neural network iteratively refines a noisy motion sequence into a plausible future trajectory. We apply this framework to our setting by extracting egocentric frame features using a pre-trained ResNet18, concatenating them with raw visuomotor observations, and flattening the resulting feature representation to condition a U-Net denoiser.

To assess the contribution of different components in our visuomotor prediction model, we conduct two types of ablations: one focusing on specific input and output signals, and another analyzing the impact of different modalities.

• •

  w/o Head Rotation: Removes head rotation from the output and optionally from the input to test its necessity.

• •

  w/o Head Rotation & Gaze: Removes both head rotation and gaze from the output and optionally from the input to evaluate their impact.

• •

  w/o Head: Removes all head-related information (position and rotation) from the output and optionally from the input.

• •

  w/o Gaze: Removes gaze from the output and optionally from the input to test its role.

• •

  w Last Step Arm: Uses only the last observed arm configuration instead of the full motion history to evaluate the importance of temporal context.

• •

  w/o Egocentric Frame: Removes egocentric visual input, leaving only kinematic information, to assess the role of first-person vision in visuomotor prediction.

Tables 1 and 2 summarize our results: baseline comparisons, ablation studies on different input modalities, and per-class prediction performance, respectively. The best results are highlighted in bold, and lower values across all metrics indicate better performance (Sec. 4.1).

Learning-based approaches significantly outperform interpolation baselines, demonstrating the need for temporal modeling and multimodal integration. Among prior works, Diffusion Policy-CNN and Transformer Encoder + MLP serve as strong baselines. Our model achieves a PA-MPJPE of 59, improving upon Diffusion Policy-CNN by 8.6% and the Transformer-based baseline by 10.7%. Similarly, all of our position errors outperform them, and our model improves the head and gaze error of Diffusion Policy by 5.7% and 6.5%, respectively. These improvements indicate that our method generates more precise motion predictions while maintaining realistic visuomotor coordination.

Notably, our model achieves the largest improvement in predicting hand position, which is the most challenging sub-task. This suggests that our approach effectively captures head-eye-hand coordination, leveraging structured visuomotor representations for fine-grained motion prediction.

When evaluating the orientations of head and gaze, our model achieves 13.2 degree as the average error across all steps, improving upon Diffusion Policy-CNN by 4.5% and the Transformer method by 6%. The fact that our model performs well across both head rotation (HRE) and translation suggests that it effectively learns a unified visuomotor representation, handling different motion modalities within the same framework.

Removing head rotation (row 2) increases head position error from 108 to 111, while additionally removing gaze input (row 4) further degrades performance to 112. Gaze position error rises from 126 to 130, and hand position error increases from 188 to 196, highlighting the importance of head and gaze signals for accurate motion prediction. When all head-related inputs are removed (row 6), gaze error increases from 127 to 132, and hand error from 190 to 194, indicating that head motion affects overall visuomotor coordination. The absence of gaze input (row 8) also increases head rotation error from 13.4 to 13.9, confirming its role in stabilizing head orientation.

The lower section of Table 2 examines high-level signal ablations. Preserving only the last-step arm pose (w/ Last Step Arm) increases head, gaze, and hand errors by 6.6%, 5.2%, and 5.9%, respectively, suggesting that a single arm pose lacks sufficient temporal context for predicting upper-body motion. Removing egocentric vision (w/o Egocentric Frame) leads to 4.7% higher head position error and a 6% increase in head rotation error, reinforcing the role of visual context in stabilizing head and gaze coordination. Interestingly, gaze error decreases slightly (124 to 130), suggesting a shift toward kinematic reliance for gaze estimation.

These results confirm that multimodal integration enhances predictive visuomotor coordination: temporal kinematic history improves hand predictions, while egocentric vision stabilizes head and gaze alignment.

Figure 5 illustrates the predicted visuomotor coordination across diverse real-world scenes. The model effectively captures head, gaze, and upper-body dynamics, maintaining smooth and temporally consistent motion. As shown in Rows (a)-(d), it successfully anticipates head and gaze shifts, even when adapting to scene constraints, demonstrating its ability to infer visuomotor intent from prior observations.

However, failure cases occur in scenarios with rapid, unexpected movements or occlusions, where subtle cues are insufficient for precise coordination. While our model still produces reasonable predictions, it may misalign with the ground truth. Row (e) presents a typical failure case: as the basketball bounces quickly—a nuance observable only in the last frame of the egocentric input—the person rapidly shifts their body rightward to catch it. In contrast, our prediction assumes a standard catching motion, failing to adapt to the sudden trajectory change. Despite these challenges, overall prediction quality remains strong, reinforcing the effectiveness of multimodal integration in learning visuomotor coordination. Future improvements could involve incorporating explicit contact modeling or leveraging environment-aware reasoning to enhance robustness in highly dynamic tasks. For a more comprehensive demonstration, we provide video examples in the supplementary materials.