Learning to Rearrange with Physics-Inspired Risk Awareness

There has been significant recent advances in applying reinforcement learning algorithms to a variety of domains ranging from playing games [1] [2] to solving navigation and manipulation tasks in simulation environments [3, 4, 5, 6, 7, 8]. However, in many real-world robotic problems, avoiding risky behaviors in deployment is as crucial as optimizing the task objective. There is a rich body of prior work on safe RL [9] defining risk and enforcing safety from various perspectives, such as imposing constraints on state-action space [10] and value functions [11, 12], restricting the policies to be close to safe demonstrations [13, 14], etc. Motivated by the fact that many damage outcomes (e.g. falling down, dropping an object) are induced by the physical process of robot interacting with environments, we consider to model the risk in terms of physical cost over the trajectories. Besides collisions with the obstacles, physical properties such as the frictions of terrains and weights of the objects are also informative learning signals for the agent to assess the risk of its behaviors. However, these factors have not been sufficiently captured by the goal states defining the success of the classical navigation and manipulation tasks [15, 5, 6, 16, 17].

To this end, we study how to incorporate physical cost into the reward function to guide the policy learning in indoor rearrangement tasks. We build two novel and challenging tasks featuring a wheeled robot pushing boxes to goal regions with physically variant properties (Fig. 1). Accomplishing these tasks requires the agent to minimize its physical cost besides reaching the successful states. Our experiments have shown that the learned policies under this new reward function are able to stably distinguishing between different masses and frictions.

To estimate the physical cost in an arrangement task, we compute the amount of virtual physical work $E(t)$ the agent spends on pushing an object to move on a floor/table at each time step $t$ as

$$\begin{split}&E(t)=E_{trans}+E_{rot}=\mu mg\Delta x+\mu\frac{mg}{XY}\theta\Bigg{[}\frac{X^{3}}{12}\\ &\left(\frac{\sin\phi}{\cos^{2}\phi}+\ln\frac{1+\sin(\phi)}{\cos(\phi)}\right)-\frac{Y^{3}}{12}\left(\frac{-\cos\phi}{\sin^{2}\phi}+\ln\left|\tan\frac{\phi}{2}\right|\right)\Bigg{]}\end{split}$$

where $m$ and $\mu$ are the object mass and the friction coefficient between the box and the floor/table. The contact surface has width $X$, length $Y$, with $\phi=\arctan\frac{Y}{X}$. The object displacement of time step $t$ is $\Delta x$ and the object rotates around its z-axis by an angle of $\theta$. All physical quantities involved here can be either obtained from the simulator or measured in the real-world irregardless of the robot platforms.

We then normalize $E(t)$ by its running bounds to get step physical cost relative to all history steps the agent has taken so far, i.e. $c(t)=(E(t)-E_{min}(t))/(E_{max}(t)-E_{min}(t))$. Similarly, for any successful trajectory $\tau$, we define its episode physical cost as $c(\tau)=\frac{E(\tau)-E_{min}(\tau)}{E_{max}(\tau)-E_{min}(\tau)}$, which is normalized over the episode physical cost of all successful episodes so far. $c(t)$ and $c(\tau)$ will be used in our reward function to measure the physical expense of an action or action sequence.

We experiment in the 3D simulated environments built upon classroom scenes from OpenRooms dataset [18] by integrating Bullet [19] physics engine and iGibson’s renderer [3]. Our tasks are performed by a simulated two-wheeled Fetch robot [20] under joint space control with abilities other than locomotion disabled. The robot has a state space of 3D positions and orientations of itself and the interactive objects. It has four discrete actions: $[v,v]$ (move forward), $[0.5v,-0.5v]$ (turn left), $[-0.5v,0.5v]$ (turn right) and $[0,0]$ (stop), where parameter $v$ is the maximum value of the robot’s wheel angular velocity.

We trained the agents using proximal policy optimization (PPO) [21] algorithms and evaluated the learned policies on the proposed tasks to answer the following questions:

Does our physics-inspired reward function correctly guide the agent to learn a physically efficient policy? We compare the policies learned with (Blue) and without (Red) the physical cost in Fig. 2 and 3. The policies learned under the physics-inspired reward function has a clear drop on the mean energy cost of the successful episodes while converges to same near optimal success rate.

Does the policy learned under physics-inspired reward function reflect the agent’s awareness of mass and friction? We examine the distributions of evaluation trajectories on the preferences over low vs. high friction in Fig. 5 and light vs. heavy box in Fig. 5 with or without physical cost. The policy learned with physical cost has shown a clear preference while the counterpart without physical cost made the decision with nearly equal probabilities.

How does our definition of physical cost compare to other design strategies on learning successful and efficient policies? We ablate our physical cost to understand the role of the following factors: (1) Using pushing energy instead of robot output energy. (2) Normalizing the cost by running bounds instead of fixed bounds, which are estimated according to extra prior knowledge about the environment and the task. The comparisons of the training curves over the combinations of these factors are shown in Fig. 2 and 3. We observe that in most cases, removing either of these factors will lead to a drop in success rate or physical efficiency.

Our work made a step towards learning physics-aware policies which are responsive to different masses and frictions. Potential future works include: (1) Extending the method to vision-based RL for better generalization of the physical risk. (2) Transferring the physics-inspired policies, value functions, experiences to downstream safe-critic tasks to avoid risky behaviors such as falling down on slippery floor, pushing unmovable objects, etc.

We thank NSF CAREER 1751365, CHASE-CI and generous gifts from Adobe, Google and Qualcomm.