MoMa-Pos: An Efficient Object-Kinematic-Aware Base Placement
Optimization Framework for Mobile Manipulation

The objective of object importance prediction is to select a subset of objects, $S$, from the object set $O$ for kinematic modeling, enabling efficient computation without processing every object. Rather than directly select the objects in an environment, we evaluate each object’s importance and select those with an importance score exceeding a predefined threshold $\alpha$. This approach allows for flexibility; if the initial selection is insufficient, reducing $\alpha$ can include additional objects [21]. We develop a prediction model, $f:O\times O\rightarrow(0,1]$, where the output, $f(o_{i},o_{t})$, represents the likelihood of including object $o_{i}$ in $S$ based on its importance relative to the target object $o_{t}$. Due to the challenges of obtaining precise labels for all objects in diverse environments[21], we adopt an unsupervised learning approach, guided by the observation that an object’s importance is often correlated with its spatial proximity to $o_{t}$ and its size, $o_{i}^{b}$. This correlation enables us to infer importance without exact labels.

Our method structures within a graph-based architecture to calculate spatial proximity between objects $o_{i}$ and $o_{t}$ while considering object size $o_{i}^{b}$. In $Env^{O}$, each object $o_{i}$ is represented as a node $v_{i}\in V$, with attributes such as size $o_{i}^{b}$. The spatial relationships between objects are captured by edges $E\subseteq V\times V$. An edge $(v_{i},v_{j})$ exists if object $o_{i}$ is spatially related to $o_{j}$ (e.g., on, in, or inside). The direction of the edge indicates spatial hierarchy, and the weight is defined as ${1}/{\textit{dist}_{xy}(o_{i},o_{j})}$, where $\textit{dist}_{xy}$ is the horizontal distance between $o_{i}$ and $o_{j}$. To quantify the spatial proximity between nodes $v_{i}$ and $v_{t}$, we explored multiple algorithms, such as word2vec[22], which were found inadequate in capturing the spatial information. After a thorough evaluation, we selected the DeepWalk algorithm [23] here. Node sequences are generated through biased random walks, with the transition probability $P(v_{i},v_{j})$ weighted by the function $w(v_{i},v_{j})=k_{0}/\textit{dist}_{xy}(o_{i},o_{j})+(1-k_{0})\times size(o_{j})$, where $k_{0}$ is a tuning parameter. This bias ensures the walks reflect both spatial distance and object size, generating sequences that better capture object importance.

$$P(v_{i},v_{j})=\begin{cases}\frac{w(v_{i},v_{j})}{\sum_{u\in N(v_{i})}w(v_{i},u)},&(v_{i},v_{j})\in E\\ 0,&otherwise\end{cases}$$

where $N(v_{i})$ represents the set of nodes adjacent to $v_{i}$. After obtaining node embeddings, we compute the importance score of each object $o_{i}$ by calculating the cosine similarity [24] between its embedding and that of the target object $v_{t}$. Objects with a score exceeding a threshold $\alpha$, where $\alpha\in[0.0,1.0]$, are included in the set $S$. The threshold $\alpha$ is adjustable, starting at a higher value and decreasing if the system requires more objects to achieve optimal performance.

The purpose of this subsection is to conduct kinematic modeling of key objects within simulation environments. These models will be utilized in the base placement computation for navigation planning. The models will also facilitate the prediction of manipulation waypoints [25] for articulated objects, enabling effective execution of real-world manipulation tasks. To achieve this, we employ the state-of-the-art model, URDFormer, an end-to-end model based on Vision Transformers (ViT), which excels in capturing an object’s kinematic structure through URDF models transformed from a single input image. However, URDFormer cannot directly ground the model to real-world objects, often leading to distortions in size and pose when rescaling the detected objects.

One of the reasons, discovered through our exploration, is that the input image’s viewpoint critically affects the modeling accuracy, with frontal views producing the best results. To address it, we design an algorithm to select the optimal viewpoint image of the target object from video streams, ensuring the input to URDFormer is optimal for more accurate kinematic modeling. This process involves using Detectron2 [26] to detect the target object in each video frame, followed by sorting the frames based on the object’s central position in the image. We then fine-tuned a ResNet50 model [27], trained on a dataset we created, which includes 1,000 labeled images (frontal and non-frontal) of over 20 everyday objects such as refrigerators and microwaves, to classify whether the object in each frame is facing forward. The first best image, which is classified as “frontal,” is selected as the input for URDFormer to generate an initial URDF model with interactive structures.

Furthermore, to address the inherent limitations of 2D single-image input which can often lead the generated URDF model to misalign the simulation with the real-world 3D environment, we correct the model using precise 3D information from point clouds. Specifically, we use 3D bounding boxes generated by Mask3D [20] to align the object’s size and pose, ensuring the URDF model accurately simulates an interactive environment that mirrors the real world. Finally, after inputting a recorded environment video, our algorithm selects the best frontal image of the target object and uses 3D bounding boxes to align the object’s size and pose, generating an accurate URDF model. This allows us to precisely locate the actionable parts on articulated objects at any opening angle, ensuring alignment with real-world conditions. The continuous location information at each angle forms the object’s manipulation waypoints. Note that rigid objects do not require modeling with URDFormer. Instead, they can be approximated as bounding boxes, where the box dimensions correspond to the object’s bounding size. This strategy significantly improves efficiency.

To optimize the base placement, we first calculate the potential area for base positions (denoted as $A$), where there are no collisions with surrounding objects. We then refine this area by considering both the robot’s constraints as well as the objects’ kinematics in determining base placements.

To get an initial robot’s potential base position area $A$, we consider both the robot’s model $r^{p}$ and the target object’s position $o_{t}^{p}$ to avoid any collisions. The potential area $A$ is calculated according to the following equation:

$$\begin{split}A=\left\{r^{p}\mid\textit{dist}_{xy}(r^{p},o_{t}^{p})\leq\Delta r,\,\forall i,\,\neg overlap\left(r^{p},r^{b},o_{i}^{p},o_{i}^{b}\right)\right\}\end{split}$$

Here, $\Delta r$ represents the allowable horizontal distance between a base position $r^{p}$ and the target $o_{t}^{p}$, illustrated in an orange line in Fig. 2. The overlap function checks for collisions between the robot’s base and surrounding objects $o_{i}$ by comparing their positions and boundaries. More specifically, $\Delta r$ is calculated as $\sqrt{(r^{l})^{2}-(o_{t}^{p.z}-r^{z}-r^{h})^{2}}$, where $r^{l}$ is the arm extension range (shown in green in the figure), derived from the robot’s DH matrix. The z-axis distance between $r^{p}$ and $o_{t}^{p.z}$ is indicated by $|o_{t}^{p.z}-r^{z}-r^{h}|$, marked as a blue line in the figure.

To further enhance the base placement optimization process, we integrate an inverse reachability matrix to account for robot-specific constraints and combine the potential field method to avoid potential interference with furniture and obstacles due to their possible kinematic trajectories. Utilizing the inverse reachability matrix, we can determine which points in $A$ a robot can reach given its joint configurations. Calculated offline, the matrix allows us to precompute feasible base positions within area $A$. By selecting base positions directly from the matrix rather than through random sampling, we ensure that each position satisfies the robot’s kinematic constraints. Inspired by the potential field method [28], we then We then assess the feasibility of each candidate position. The target object $o_{t}$ exerts an attractive force. For each candidate base position $r^{p}$, we sample a 3D position $(r^{p},z)$ and connect it to $o_{t}^{p}$ with a line. If this line collides with any furniture’s bounding box, the potential value from the potential field $F(r^{p},z)$ is set to zero, indicating the robot cannot reach the target from that position. Otherwise, we calculate $F(r^{p},z)=1/\textit{dist}_{xy}(r^{p},o_{t}^{p})$. The final potential value is the sum of the value from the inverse reachability matrix and the value from the potential field. Positions with higher final potential values are more feasible. Additionally, we incorporate trajectory planning by discretizing it into multiple waypoints and calculating the potential for each, resulting in a more comprehensive potential field. The sum of the potentials along the trajectory produces the final potential map, which guides the selection of the optimal base placement. Fig. 3 illustrates the potential values for sampled positions in a kitchen environment.

Within area $A$, we use the Latin Hypercube sampling[29] method to sample $M$ 2D positions $r^{p}_{i}$, with higher potential values suggesting better feasibility. Instead of naively selecting the position with the lowest potential and checking feasibility via a motion planner like RRT*[30], we aim to balance potential values with navigation costs. This leads to a problem resembling the Open Traveling Salesman Problem (TSP)[31], where the robot does not return to its starting position.

Given the large number of sampled positions, we group them into smaller sets, each containing $T$ candidates. Within each set, the optimal path is determined by solving the Open TSP, using edge weights that combine both distance $\textit{dist}_{xy}(r^{p}_{i},r^{p}_{j})$ and potential values. The edge weight is computed as $k_{1}\times\textit{dist}(r^{p}_{i},r^{p}_{j})+k_{1}^{\prime}\times\left(F(r^{p}_{j})-F(r^{p}_{i})\right)$, where $k_{1}$ and $k_{1}^{\prime}$ are user-defined coefficients. Algorithms ranging from exact methods [32] to heuristic approaches [33] can be used, depending on the size of $T$. The search continues until a collision-free trajectory is found, and the corresponding base position $r^{p*}$ is selected.

Lemma 1 (MoMa-Pos is complete). Given any object importance scorer $f:O\times O\rightarrow(0,1]$, threshold $\alpha\in[0.0,1.0]$, and $M$ sampled positions, $M\in[1,\infty)$, if the motion planner PLAN is complete, then MoMa-Pos is complete.