Efficient Reinforcement Learning of Task Planners for Robotic Palletization through Iterative Action Masking Learning

The problem of 3D bin packing, a complex variant of the classical bin packing problem, involves efficiently packing objects of various sizes into a finite number of bins or containers with fixed dimensions in three dimensions. The offline version of the problem, where all information about the objects to be packed is known in advance, has been particularly challenging due to its NP-hard nature. Early strategies were heavily reliant on heuristic methods, as initially demonstrated by [14] that extended 2D BPP algorithms to 3D context. This is followed by many research efforts that propose various heuristic and meta-heuristic algorithms to approximate the solutions [15, 16, 17]. More recently, [18, 19] propose to use deep reinforcement learning (DRL) to learn the policy for offline 3D BPP. Our work differs from these works by focusing on online 3D BPP instead of offline version.

In contrast to offline scenarios, online 3D bin packing problems involve making packing decisions without knowledge of future items, presenting unique challenges in achieving optimality and efficiency. Research in this domain has focused on developing adaptive heuristics and algorithms that can handle the sequential nature of item arrivals. Notably, Karabulut et al. [2] proposes deep-bottom-left (DBL) heuristics and Ha et al. [3] extends  [2] by employing a strategy that orders empty spaces using the DBL method and positions the current item into the earliest suitable space.  [4] introduces a Heightmap-Minimization technique aimed at reducing the volume expansion of items in a package when viewed from the loading perspective. Aside from heuristics-based methods, there are also research attempts utilizing DRL to solve the problem of online 3D BPP [5, 6]. DRL for online 3D BPP typically suffers from the problem of large action space, making the RL algorithms hard to optimize.  [5] address this challenge by implementing straightforward heuristics to reduce the action space, yet their demonstrations are confined to scenarios with limited discretization resolution. Conversely, another study by [6] seeks to alleviate this issue by devising a packing configuration tree that employs more intricate heuristics to identify a subset of feasible actions, marking a progression from their prior work. Our research is distinct from these approaches in two critical aspects. Firstly, the scenario we investigate is notably more complex and mirrors real-world conditions more closely. Specifically, we introduce a ”buffer” area, enabling the agent to select any item from a queue of upcoming boxes and rotate it along any axis, thereby significantly expanding the action space. Furthermore, our model accounts for translational and rotational uncertainties that occur when a robot places a box onto the pallet—factors that are overlooked by [6, 5] but are prevalent in actual physical systems. Secondly, from a methodological perspective, our approach diverges from the reliance on varying heuristics to trim the action space. Instead, we introduce a supervised learning-based pipeline to develop an action masking mechanism.

The online 3D BPP can be formulated as a Markov decision process (MDP), denoted by $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$, where $\mathcal{S}$ signifies the state space, $\mathcal{A}$ denotes the action space, $\mathcal{P}$ represents the transition probability between states, and $\mathcal{R}$ encapsulates the real-valued rewards. The objective of RL is to find a policy, $\pi(a|s)$, that can maximize the accumulated discounted reward $J(\pi)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{t}\right]$, where $\gamma$ represents the discounted factor. In the following, we details the components of the MDP in this study.

The principal challenge in applying RL to address the online 3D BPP resides in the expansive dimensionality of the action space, which complicates the optimization of RL algorithms. To counteract this issue, we embrace the strategy of ”invalid action masking”  [8, 9, 10], which serves to alleviate the aforementioned challenge. This section elucidates our methodology for developing the action masking model through supervised learning. The fundamental premise behind action space masking is the exclusion of invalid placements that could culminate in unstable pallet configurations. While prior studies have implemented similar strategies to exclude unstable placements and thereby simplify the learning process[7, 5], their approaches to action masking rely predominantly on heuristic algorithms. We posit that heuristic-based action masking exhibits several limitations: (i) The necessity for multiple hyperparameters in most action masking methods poses a challenge in tuning these parameters to accommodate diverse scenarios, such as variations in box sizes and weights. ii) Heuristic methods, particularly those involving the analysis of the center-of-mass (CoM), fail to account for uncertainties inherent in palletization execution, necessitating further elaboration. For instance, consider a heuristic that uses CoM analysis to prevent tipping by restricting box placements that significantly shift the CoM. However, this doesn’t consider real-world uncertainties like slight placement inaccuracies due to robot sensor errors, leading to unexpected CoM shifts and potentially unstable configurations despite heuristic approval. This example underscores the limitations of CoM-based heuristics, highlighting their lack of accommodation for practical uncertainties in robot execution, which questions their reliability in real-world scenarios. In response, our work investigates the feasibility of adopting supervised learning to construct the action masking model. Figure 2 delineates the framework of our proposed methodology for learning action masking. Specifically, the learning process is segmented into the following phases:

While being effective, the learned action masking model can still be inaccurate in certain states, particularly for the out-of-distribution (OOD) states encountered during the RL training process. Such discrepancies arise because during RL training the agent might explore states that diverge significantly from those within the training dataset $\mathcal{D}={(\mathcal{C}_{t},b_{t},g_{t})}$, leading to a misalignment between the distributions of the training data and the actual scenarios the model faces.

Inspired by DAgger [13], we introduce an iterative enhancement pipeline for refining the learning of the action masking model. This process begins with the application of a heuristic-based method for initial action masking during RL training, yielding the initial policy $\pi_{0}$. Concurrently, we systematically collect state samples during $\pi_{0}$ training, assembling dataset $\mathcal{D}_{0}$ via offline collection as detailed in Section III-B. Utilizing $\mathcal{D}_{0}$, we then train the neural network-based action masking model $f_{\theta}^{0}$, as depicted in Section III-B. This model, $f_{\theta}^{0}$, is subsequently integrated into a new RL training cycle to derive an enhanced policy $\pi_{1}$. This cycle of training, data collection to form $\mathcal{D}_{i}$, and model updating to $f_{\theta}^{i}$ is repeated, each iteration aiming to embed the updated action masking model in the subsequent RL training phase. This iterative process continues until the RL policies cease to show significant improvements, thereby optimally aligning the action masking model with the encountered state distributions. The pseudo code of the proposed method is presented in Algorithm 1.

To explore the task planning challenges associated with the palletization problem, we developed a simulated palletization task in MuJoCo [21], reflecting a realistic logistic scenario, as depicted in Figure 3. This simulation environment features 80 boxes of 5 distinct sizes, with dimensions (in inches) of $10\times 8\times 6$, $9\times 6\times 4$, $6\times 6\times 6$, $6\times 4\times 4$, $4\times 4\times 4$. The pallet is specified to have dimensions of $25\times 25$ inches, with a stipulation that the maximum height of stacked boxes cannot exceed 25 inches. For our experiments, we applied a discretization resolution of 1 inch to both the boxes and the pallet.

At each planning step, the task planner receives the current pallet configuration, represented as a height map, and the dimensions of $N$ forthcoming boxes in the buffer, then generates a task plan. This plan involves selecting one of the $N$ boxes, orienting it as desired, and placing it on the pallet, in accordance with the methodology described in Section III-A. To expedite the learning process, the actual ’pick-rotate-place’ actions by the robot are bypassed; instead, the chosen box is immediately positioned at the planner’s determined goal pose. To account for uncertainties inherent in physical execution, each box placement is subjected to random translational noise on the xy plane $\delta_{t}\sim\mathcal{N}(\textbf{0},\textbf{0.05})$ and rotational noise around the z-axis $\delta_{r}\sim\mathcal{N}(0,5)$, measured in inches and degrees, respectively. The sequence in which the 80 boxes are presented to the policy is randomized and unknown at the start of each RL training episode, and the weights of the boxes are also randomized to further enhance the simulation’s fidelity to real-world variability.

Our objective is to validate the efficacy of the learned action masking model within the RL training framework. To this end, we implemented two baseline methodologies, NoMask and HeuristicsMask, for comparison against our proposed LearnedMask method, detailed in Section III-B. The implementation specifics are outlined as follows:

In this section, we provide evaluation results under different experimental settings, aiming to address the three questions posed at the outset of Section IV.

We further tested our RL-based palletization task planner in a real-world physical system. Due to limitations in space and hardware, it was not feasible to implement our algorithm on an industrial-scale robot handling large boxes. Therefore, we utilized a Franka Panda robotic arm as a prototype for our physical system setup, depicted in Figure 7(a). The robot is equipped with a suction-type gripper for box manipulation and an Intel Realsense RGB-D camera mounted in-hand for box detection and localization on the conveyor. Space constraints restrict the camera’s perception to a single box at a time, resulting in an upcoming box count $N=1$. The boxes, showcased in Figure 7(b), come in 5 different sizes and contain various items to simulate the diversity in weights found in real palletization tasks.

Our methodology involved training the task planner in a simulated environment tailored to the physical system’s specifications regarding pallet and box dimensions. This trained planner was then directly applied to the real-world setup. The robotic system demonstrated proficiency by successfully stacking 35 boxes on the pallet, achieving a space utilization rate of $72.0\%$ in a single episode. The final pallet configuration, viewed from different angles, is presented in Figure 8. The ability to stack 35 boxes while maintaining compactness and stability showcases the effectiveness and robustness of our learned palletization planner policy. A demo video of the palletization process can be accessed at here.