Relevance-driven Decision Making for Safer and More Efficient Human Robot Collaboration

Relevance fundamentally differs from previous works that aim to identify and underline the important regions in the scene, such as attention and saliency. Saliency focuses on identifying the visually prominent and conspicuous features in the image [10, 11, 12], while attention focuses on current short-term, reactive, and simple tasks without considering the future projection [13, 14]. Relevance fundamentally differs from those works in that relevance represents the inherent relationship between objects because of the human objective and the applicability of objects to achive the objective. Relevance considers the interconnection among environmental understanding, human models, task models, etc.

Large Language Models (LLMs) are revolutionizing robotics by enabling multi-modality reasoning [15], general prediction without transfer learning [16], and flexible connections between modules in robotic frameworks [17]. In this paper, we utilized LLMs for a dramatically different and novel application, i.e., relevance determination. Moreover, the current inference time of LLMs is longer than that of real-time computation requirements. Existing works with LLMs in robotics mainly apply to non-real-time policy generation in a static or pre-defined environment. Our unique two-loop design enables knowledge retrieval of LLMs in real-time applications by leveraging asynchronous computation.

APF is an intuitive and efficient real-time robot motion planning method by simulating attractive forces toward goals and repulsive forces away from obstacles, enabling smooth and collision-free navigation [18]. Song et al. proposed a methodology called Predictive APF to anticipate obstacles based on the robot’s velocity and the relative positions of obstacles to adjust the path before potential collisions [19]. However, their goal of incorporating prediction into APF is to smooth the path considering the dynamic constraints of the agent, which is dramatically different from ours. Moreover, their methodology is limited to environments with static obstacles. Our Relevance-based APF (RAPF) is to improve HRC safety by predicting the motion of the human and dynamically updating the path proactively in a highly dynamic environment.

Let $S=\{C_{1},C_{2},...,C_{n}\}$ be a set of class of objects in a visual scene, where each $C_{i}$ represents a class of objects and $n$ is the number of classes of objects in the scene. The set of relevant elements $\mathcal{E}_{r}$, which contains all the relevant elements to the human’s objective and associated tasks, is predicted and determined with the available information. Based on $\mathcal{E}_{r}$, the robot generates actions to assist the human to achieve the human’s objective.

An overview of the framework is shown in Fig. 2. Our framework for relevance quantification and safer HRC consists of two loops. The first loop fulfills the function of perceiving the world, understanding the scene, and decision making based on relevance for safer HRC. The second loop is an asynchronous loop that leverages LLMs and quantifies relevance. By optimally coordinating the two loops, the world knowledge of the LLM can be incorporated into real-time operation of the robots.

The input to the algorithm is the stream of observations from the robot sensors. The images are processed with scene understanding algorithms, such as YOLO [20] for object detection, MMPose [21] for human pose estimation, CLIP [22] for scene grounding, etc.

The module of human intention prediction determines current human actions and aggregates them into a history record. Here, we implement an algorithm that leverages Bayesian networks to fuse three modalities of human information: head orientation, hand orientation, and hand movement. By fusing three modalities for human intention recognition, the accuracy and precision of the prediction are dramatically increased because different modality provides valuable information in different stages of the actions.

In the original relevance formulation and quantification methodology, the scene is represented as classes and elements. The relevance of each class and element is determined in a sequential and hierarchical manner. In this paper, we adopted a direct prediction based on LLMs on the element level for simplicity. The goal of the LLMs in this paper is to predict the human objective and the set of relevant elements $\mathcal{E}_{r}$, reflecting potential future object interactions and human motion.

The first step in quantifying relevance with LLM is contextualizing the scene. Currently, this is achieved with a fixed prompt $\mathcal{P}$ designed and optimized specifically for the setup and the problem of HRC. However, in the future, automatic scene contextualization and prompt construction can be considered and developed. In the prompt $\mathcal{P}$, the environment, the objects in the scene, and the action history of the human are automatically generated from the scene understanding and human intention prediction modules.

The outputs of the LLM module are the prediction of the human objective and the set of relevant elements $\mathcal{E}_{r}$. The relevance of each element is quantified to be

$$R(e_{i})=\begin{cases}1&\text{if }e_{i}\in\mathcal{E}_{r}\\ 0&\text{otherwise}\end{cases}$$

(1)

With this method, world knowledge of an LLM can be extracted for a general environment and setup without any training and transfer learning.

Our framework’s decision-making can be decomposed into two components: human task allocation and real-time motion generation with collision avoidance.

For the sake of simplicity, we assume that humans need all the relevant elements predicted for the objective. A more detailed inquiry method to derive the necessary and preferred elements can be found in [8]. The human-robot task allocation is formulated as an optimization problem and solved with an optimization solver.

In the problem definition, we define the subscript $j$ representing the index of elements such that $e_{j}\in\mathcal{E}_{r}$. We define the position vector of element $e_{j}$ in the world coordiante as $\mathbf{p}_{e_{j}}$, the position of the common destination of relevant elements as $\mathbf{p}_{d}$, the initial location of the robot as $\mathbf{p}_{r_{0}}$, the velocity of the robot as $v_{r}$, the velocity of human as $v_{h}$, the start-up delay for the human to finish the current task as $d_{h}$. We consider $d_{h}$ because there is a temporal delay between the time the robot and the human start to fulfill the allocated tasks. We further define $T_{r}$ and $T_{h}$ as the time durations for the robot or the human to finish the assigned tasks, which can be computed as:

	$\displaystyle T_{r}$	$\displaystyle=\sum_{j}y_{j}\left(\frac{\|\mathbf{p}_{r_{0}}-\mathbf{p}_{e_{j}}\|+\|\mathbf{p}_{e_{j}}-\mathbf{p}_{d}\|}{v_{r}}\right)$		(2)
		$\displaystyle\quad+\sum_{j}x_{j}(1-y_{j})\left(\frac{2\|\mathbf{p}_{d}-\mathbf{p}_{e_{j}}\|}{v_{r}}\right)$

and

$$T_{h}=d_{h}+\sum_{j}(1-x_{j})\left(\frac{2\|\mathbf{p}_{d}-\mathbf{p}_{e_{j}}\|}{v_{h}}\right)$$

(3)

We define the decision variable $Z$ as the maximum time taken to complete all tasks, and the optimization problem is formulated as:

	Minimize		$\displaystyle Z$		(4)
	Subject to:				(5)
		$\displaystyle\sum_{j}y_{j}$	$\displaystyle=1,$
		$\displaystyle y_{j}$	$\displaystyle\leq x_{j}\quad\forall j,$		(6)
		$\displaystyle Z$	$\displaystyle\geq T_{r},$		(7)
		$\displaystyle Z$	$\displaystyle\geq T_{h},$		(8)
		$\displaystyle x_{j},y_{j}$	$\displaystyle\in\{0,1\}\quad\forall j.$		(9)

where $x_{j}$ is a binary decision variable where $x_{j}=1$ if the robot moves $e_{j}$, and $0$ otherwise, and $y_{j}$ is a binary variable where $y_{j}=1$ if $e_{j}$ is the first task done by the robot, and $0$ otherwise. (5) ensures that exactly one object is designated as the initial task performed by the robot. (6) specifies that an object can only be the first task executed by the robot if that object is assigned to the robot, as indicated by $x_{j}$. After the relevant objects are allocated to the robot or the human, the robot will execute the assigned tasks, generate the motion, and avoid obstacles dynamically in real time.

For dynamic motion planning and obstacle avoidance, our methodology is based on an APF formulation in [23]. The major contribution of our work, as shown in Fig. 3, is that we build a virtual obstacle from the current human hand based on the predicted human motion and update the repulsive force based on the virtual obstacle for proactive and safer motion generation.

In our APF, the attractive force $\mathbf{F}_{a}$ is modeled as:

$$\mathbf{F}_{a}=A\left(1-\exp\left(-\frac{d_{g}}{\alpha_{a}}\right)\right)\mathbf{\hat{u}_{a}}$$

(10)

where $A$ is the magnitude of the attractive force, $d_{g}$ is the distance between the robot and its goal, $\alpha_{a}$ is the constant that controls how the attractive force decreases with distance, and $\mathbf{\hat{u}_{a}}$ is the unit vector from the robot to the goal.

The repulsive force by obstacles other than the human’s hand $\mathbf{F}_{r_{o}}$ is constructed as:

$$\mathbf{F}_{r_{o}}=\sum_{i=1}^{m}\frac{R}{1+\exp\left(\left(\frac{2d_{e_{i}}}{\rho_{r}}-1\right)\alpha_{r}\right)}\mathbf{\hat{u}_{r_{i}}}$$

(11)

where $m$ is the number of elements in the scene, $R$ is the magnitude of the repulsive force, $d_{e_{i}}$ is the distance between the robot and the obstacle $e_{i}$, and $\mathbf{\hat{u}_{r_{i}}}$ is the unit vector from the obstacle $e_{i}$ to the robot location. $\rho_{r}$ and $\alpha_{r}$ are two factors defining the shape and decreasing rate of the repulsive force as the distance increases, respectively.

To consider the future motion of the human hand, we developed a novel methodology to construct an ellipsoid virtual obstacle as shown in Fig. 3. The two endpoints of the major axis of the ellipsoid are the current position of the human hand $\mathbf{p}_{h}$ and the position of the predicted hand destination $\mathbf{p}_{t}$. The semi-axes $a$, $b$, and $c$ of the ellipsoid are given by:

$$a=\frac{\|\mathbf{p}_{h}-\mathbf{p}_{t}\|}{2}$$

(12)

$$b=k_{b}\cdot a$$

(13)

$$c=k_{c}\cdot a$$

(14)

where $k_{b}$ and $k_{c}$ are two factors controlling the shape of the ellipsoid. The repulsive force from this virtual obstacle is updated to

$$\mathbf{F}_{r_{v}}=\frac{k_{r}\cdot R}{1+\exp\left(\left(\frac{2d_{c}}{\rho_{r}}-1\right)\alpha_{r}\right)}\mathbf{\hat{u}_{r_{v}}}$$

(15)

Where $d_{c}$ is the distance from the robot to the closest point on the ellipsoid $\mathbf{p}_{c}$, $\mathbf{\hat{u}_{r_{v}}}$ represents the unit normal vector at the closest point on the ellipsoid, and $k_{r}$ is a new factor we proposed representing a scale factor on the force magnitude based on the proximity to the human. In this paper, $k_{r}$ is computed as:

$$k_{r}=1-\frac{(\mathbf{p}_{c}-\mathbf{p}_{h})\cdot\mathbf{\hat{u}_{h\rightarrow t}}}{(v_{h}+v_{r})\cdot t_{s}}$$

(16)

where $\mathbf{\hat{u}_{h\rightarrow t}}$ represents the unit vector from $\mathbf{p}_{h}$ to $\mathbf{p}_{t}$, and $t_{s}$ is a time factor we proposed in the unit of $s$ reflecting the available safety buffer in terms of time. $k_{r}$ decreases as $\mathbf{p}_{c}$ becomes farther away from $\mathbf{p}_{h}$. Without $k_{r}$, $\mathbf{F}_{r_{v}}$ will consider the virtual obstacle too early and conservatively, which will result in unnecessary detour and a longer robot trajectory.

The total force acting on the robot $\mathbf{F}_{t}$ is the sum of the attractive force and repulsive forces from both the physical obstacles and the virtual obstacle:

$$\mathbf{F}_{t}=\mathbf{F}_{a}+\mathbf{F_{r_{o}}}+\mathbf{F}_{r_{v}}$$

(17)

This total force determines the direction and magnitude of the robot’s motion, allowing it to avoid both physical and virtual obstacles while moving towards its goal.

The performance of human objective prediction and relevance prediction is assessed via the Breakfast Actions Dataset [24], which comprises a variety of typical human activities performed during breakfast time (e.g., preparing coffee, cooking pancakes, and making hot chocolate, etc.). For each test, ground truth data provides the ground truth objective (GTO), and the ground truth plan (GTP), which describes the sequence of actions performed by an individual in the execution of the objective. In our evaluation, a step ratio of segments from the GTP is contextualized and input into LLM to predict the human objective and the set of relevant elements. Three distinct ratio step values: 0.25, 0.5, and 0.75 are employed. The objective prediction is evaluated through a manual assessment process. The predicted relevant objects are evaluated automatically with word matching. The metrics for the evaluation are percentages of tests with correct prediction of the objective and the relevant elements.

To verify the effectiveness of relevance for safer HRC, we developed a simulator as shown in Fig. 4. In the simulator, we constructed the environment with a table (the cyan cuboid) and objects on top of the table. The height of the table is set to be 73 cm. The table size is 180 cm in the $x$ direction, 6 cm in the $z$ direction, and 76 cm in the $y$ direction. The objects on the table consist of two parts: the necessary objects for the objective and randomly added objects from a list of kitchen objects. The locations for each objects on the tabletop are randomly selected from a collection of locations uniformly distributed on a half circle with a radius of 60 cm. The diameter of each object is 8 cm.

A human’s hand (the red dot) is added to the simulator to simulate the human’s motion. A UR5 robot is mounted on the other end of the table, reasoning about the human objective and the relevant objects and generating safe actions to best assist the human by accomplishing the human’s objective with minimum time. The initial locations of the human and the robot are shown in Fig. 4. To take the inference time of LLM into consideration, the simulation runs and progresses in real time with a frequency of 30 Hz. The velocities of the human’s hand and the end gripper are set to be 0.4 $m/s$.

To test the effectiveness of relevance and our decision-making methodology, we focus on comparing two methods. The first method uses relevance and our decision-making with virtual obstacles, as described in Section III. For the comparison, we built a baseline test without virtual obstacles but with relevance. It’s worth noticing that, without relevance, the robot has no idea about its tasks to assist the human. Thus, we assign the same tasks as the relevance test cases to the robot. And the robot starts to move with exactly the same frame index as the relevance cases. Without loss of generality and for simplicity, we test on the objective of making cereals, and the relevant objects are cereal, bowl, milk, and spoon.

Our evaluation results are depicted in Table. I, confirming the effectiveness of our methodology for objective and relevance prediction across various scenarios. At a step ratio of 0.75 with more human actions contextualized and fed into LLM, our methodology achieves a high objective prediction accuracy of 0.90 and a relevance prediction accuracy of 0.96.

We first analyze the performance of objective prediction using our methodology, upon which relevance determination depends. Upon further examination of individual objectives, certain objectives, such as the preparation of cereal, coffee, hot chocolate, juice, salad, and tea, exhibit consistently high predictive accuracy across all evaluated ratio steps. The prediction accuracy for those objectives is high even at a low step ratio of 0.25, demonstrating our methodology can accurately identify the human’s objective at an early stage of the human action and motion. Consistent with expectations, an increase in the “ratio steps” parameter correlates positively with enhanced prediction accuracy. The contextualization of the scene with a higher step ratio will contain more indicative cues about the objectives and thus improve the objective prediction accuracy. The inaccuracy for fried eggs is attributed to the inherent similarity between the objectives involving eggs, such as omelettes and scrambled eggs. However, the relevance for those objectives are very similar.

Next, we analyze the performance of relevance prediction using our methodology based on the predicted objective. The relevance prediction performs extraordinarily for specific objectives, including hot chocolate, juice, and salad, which achieves a 100% accuracy at the step ratio of 0.25. The prediction accuracy of all other objectives achieves at least 0.85 at the step ratio of 0.75. For the relevance prediction, the accuracy also increases with the step ratio. It’s worth noting that the relevance prediction for the objectives predicted with low accuracy can still be accurate. This is attributed to the fact that similar objectives, such as omelettes and scrambled eggs, share the same relevance, which bolster the relevance prediction despite inaccuracies in objective identification.

Through all the test cases, our methodology of relevance determination robustly and correctly predicts the human’s objective and the set of relevant elements for making cereals as bowl, spoon, milk, and cereal. Our human robot task allocation module robustly solves the optimization problem in real-time to generate the tasks for the robot. Thus, we focus on the evaluation of the virtual obstacle and collision avoidance. The test results of our real-time and safe decision making methodology are shown in Table II.

With our RAPF, the robot actions generated are much safer, with the rate of collided cases decreasing by 63.76% and the rate of collided frames decreasing by 44.74%. A visual comparison between the two methods is shown in Fig. 5. The frames at $t=3.37s$ are the frames before the collision. With the relevance and the virtual obstacle, the repulsive force (shown with a gold arrow) is more perpendicular to the hand trajectory and thus pushes the gripper further from the hand smoothly. At $t=3.67s$, the gripper and the human collide in the baseline test because there is not enough duration of repulsive force to push the gripper farther away from the hand trajectory. However, with the virtual obstacle, no collision happens because of the proactive repulsive force generated by the virtual obstacle. Those results demonstrate the effectiveness of relevance and RAPF in informing the robot how to best assist humans and generate safer and more efficient actions.