Learning Coordinated Pushing Actions for Grasp-Based Occluded Object Retrieval in Cluttered Environments

Learning Coordinated Pushing Actions for Grasp-Based Occluded Object Retrieval in Cluttered Environments 1st Matteo Colombo Politecnico di Milano [email protected] 2nd Alessio Nozza Bielli Politecnico di Milano [email protected] 3rd Paolo Rocco Politecnico di Milano [email protected] 4th Andrea M. Zanchettin Politecnico di Milano [email protected] Abstract—We propose a push-to-grasp framework that coordinates pushing and grasping actions through three Deep QNetworks orchestrated by a perception-based oracle module, to achieve occluded object retrieval from a cluttered environment. The networks are trained through Deep Reinforcement Learning (DRL) in simulation, and then deployed on a robot, successfully executing retrieval tasks in the real world. Index Terms—Robotic manipulation, Deep reinforcement learning, Occluded object retrieval. I. INTRODUCTION The retrieval of objects in cluttered environments is a significant challenge in the field of robotic manipulation. Industrial and domestic applications increasingly demand systems capable of interacting effectively with complex environments where objects are frequently occluded or obstructed by clutter. Traditional grasp strategies often fail in such conditions, particularly when the target object is hidden beneath multiple layers or surrounded by obstacles. This work introduces a novel pushto-grasp framework, a modular approach that synergistically integrates both prehensile (grasp) and non-prehensile (push) actions. The idea of synergistically coordinating push and grasp actions in order to retrieve an object from a cluttered scene was first introduced in [1], where two fully convolutional networks are trained jointly in a DQN framework [4] and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. [1] is then extended [3] to try and learn policies that are more efficient and employ a reduced number of push actions to free the target object from surrounding clutter. Among these, [2] achieves the highest success rate. They use a predictive network that provides visual foresight of the scene and is exploited in a tree search as a state transition function, in the space of scene images. The tree search returns a sequence of consecutive push actions yielding the best arrangement of the clutter for grasping the target object. The main drawback of these methodologies is that they often rely on the target being visible to predict the next action. In this work, we propose a novel framework, shown in Figure 1, designed to operate with minimal assumptions about the This study was carried out within the MICS (Made in Italy – Circular and Sustainable) Extended Partnership and received funding from Next-Generation EU (Italian PNRR – M4 C2, Invest 1.3 – D.D. 1551.11-10-2022, PE00000004). CUP MICS D43C22003120001. Fig. 1. Our push-to-grasp framework uses an RGB and a Depth input image. An occlusion checker coordinates two agents that perform pushing actions on the cluttered scene. ϕP F acts to find the object in the environment, while ϕP R acts to free the object, which is grasped through the action selected by ϕP G. environment, relying primarily on visual input from an RGB-D camera. Leveraging Deep Reinforcement Learning, the agents try to learn optimal policies for the selection and execution of actions that maximise task success. Unlike prior state-of-theart approaches, our framework builds on the basis of [1] and extends decision-making to three dimensions, incorporating height selection for push actions. This enables the system to efficiently retrieve targets even in cases of severe occlusion. We also show that the proposed methodology, trained in simulation, can be transferred to a real robotic arm without the need for further retraining. II. METHODOLOGY Our framework is composed of three Deep Q Networks (DQN) Φψthat are trained in a self-supervised manner through the Deep Q Networks (DQN) methodology. These are coordinated by an oracle function: a vision-based module that, given the scene representation st, selects the type of action ψto take. stis the masked RGB-D image of the environment, while ψ can be either a find,release or grasp action. The find and release actions are both pushing actions of fixed length, while grasp is a grasping action, predicted by the corresponding 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629898 253 Fig. 2. An example of action selection starting from the current state s0 networks. The action value a∗is the output of each network and takes the form: Φψ(st) = max a(Q(st, at)) = Q(st, a∗ t)→a∗ t(1) where at= (q, z, θ)depending from the corresponding action type ψ.atis composed by q, which maps the pixel where ψshould be executed to (x, y)coordinates in robot space. zis the action execution height, while θassumes a different meaning according to the action type. For the find and release pushing actions, it represents the orientation in which the linear push is going to be executed, according to the base image reference frame. For the grasp action, it represents the gripper orientation during the grasp. Figure 2 shows how the three networks Φψare coordinated through the oracle function. The oracle takes as input an RGB and depth image of the scene, applying a semantic segmentation through a state-ofthe-art MaskRCNN network, which divides the scene into blobs representing the single objects and, if possible, detects the presence of the target object to be retrieved. The output of the grasp network ΦGis used to infer if there is a feasible grasp on it. If the output score a∗ t, which represents the best possible grasp on the target, exceeds a certain threshold QG, the object is grasped and the task is successfully concluded. Otherwise, find and release pushing actions are chained to find the desired object and to release it from the surrounding clutter. The oracle uses an occlusion checker based on Region of Interest (ROI) and a layer filter to select whether to employ the action resulting from ϕP F or ϕP R.ϕP F acts when the oracle detects that the target is occluded or not visible, while ϕP R pushes the revealed object or its surroundings with the goal of increasing the grasp score. To this end, ϕP F and ϕP R are trained cooperatively on the same scenarios, using DQN, with the following reward structure: Rat(st, st+1)=Rcurr +γRfut (2) The total reward Rat(st, st+1)for the specific action (release or find) is computed as a weighted sum over γ, the discount factor, of a current and a future reward term. We use Rcurr to reward consistency in the pushing action, while Rfut is shaped according to the specific goal of each of the two push networks. III. EXPERIMENTS AND VALIDATION The proposed framework was evaluated through a series of simulation benchmarks and real-world validations. Simulation tests were conducted in PyBullet with a UR5 manipulator and RGB-D sensing, while real-world experiments used a similar robotic arm with comparable perception hardware. Performance was assessed using standard metrics: task success rate, grasp and push success rates, number of actions, and execution time. We benchmarked the framework against VFT, proposed in [2]. To make the comparison fair, we stuck to densely cluttered scenes where the object is visible, removing ϕP F . This was motivated by the inability of VFT to handle scenarios where the object is fully occluded. Despite requiring only 7000 training iterations (compared to 200000 for VFT), the components of our framework achieved competitive performance. Notably, our method handled scenes with nearly double the object density of prior benchmarks, while reducing average execution time to 13 seconds versus the 610 seconds of VFT. Moving to tests with partially visible targets, our system reported a task success rate of 86.7%, with grasp and push primitives succeeding in 96% of executions. When extended to more challenging scenarios with vertical occlusions and fully hidden targets, the framework maintained robustness: our strategy solved 86.7% of tasks with 3.42 pushes on average. Deploying the framework to a real ABB CRB 15000 robot arm, the experiments confirmed the effectiveness and transferability of individual components, as all three of the policies were able to retain their effectiveness without the need for retraining, although we highlight that failures were mostly caused by noisy or inaccurate depth images, which resulted in wrong primitives execution heights. These results demonstrate that the proposed method matches state-of-the-art retrieval methods rates, but also introduces the ability to manage hidden and vertically occluded objects with lighter computation and training demands. REFERENCES [1] Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018, October). Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4238-4245). IEEE. [2] Huang, B., Han, S. D., Yu, J., & Boularias, A. (2021). Visual foresight trees for object retrieval from clutter with nonprehensile rearrangement. IEEE Robotics and Automation Letters, 7(1), 231-238. [3] Mohammed, M. Q., Kwek, L. C., Chua, S. C., Al-Dhaqm, A., Nahavandi, S., Eisa, T. A. E., ... & Alandoli, E. A. (2022). Review of learningbased robotic manipulation in cluttered environments. Sensors, 22(20), 7938. [4] Huang, Y. (2020). Deep Q-networks. In Deep reinforcement learning: fundamentals, research and applications (pp. 135-160). Singapore: Springer Singapore. 254