Incremental Acquisition and Composition of Robotic Manipulation Skills from Virtual Demonstrations

Incremental Acquisition and Composition of Robotic Manipulation Skills from Virtual Demonstrations Giuseppe Rauso Department of Electrical Engineering and Information Technology University of Naples “Federico II” Naples, Italy [email protected] Riccardo Caccavale Department of Electrical Engineering and Information Technology University of Naples “Federico II” Naples, Italy [email protected] Alberto Finzi Department of Electrical Engineering and Information Technology University of Naples “Federico II” Naples, Italy [email protected] Abstract— We propose a unified framework for robotic manipulation that integrates incremental learning from demonstrations, reinforcement learning, and symbolic task planning. The system supports the training of manipulation skills from Virtual Reality (VR) demonstrations, exploiting the ease of integration provided by Unity for recording demonstrations and training policies. A key feature is incremental learning, where policies acquired in early stages (e.g., proximity grasps with multi-fingered hands) are frozen and reused to accelerate the learning of more complex tasks (e.g., grasp-and-lift). Learned and predefined behaviors are stored in a structured repository, which can be flexibly queried during execution. Task composition is achieved through Hierarchical Task Network (HTN) planning, enabling the generation of long-horizon activities. Validation is performed in CoppeliaSim, transferring trained policies to different manipulators and end-effectors and testing with novel objects not seen during training. Preliminary results from the integrated works underscore the potential of the proposed approach, suggesting its scalability toward increasingly complex robotic manipulation tasks. Keywords— incremental learning, imitation learning, reinforcement learning, robotic manipulation, task planning I. INTRODUCTION Robotic manipulation has advanced considerably through reinforcement learning (RL) and imitation learning (IL), but enabling robots to incrementally acquire, reuse, and compose skills in a modular way remains challenging. Learning from Demonstration (LfD) has been widely adopted to accelerate training, with demonstrations collected via teleoperation, videos, kinesthetic teaching, or motion capture [1] [2] [3]. Virtual Reality (VR) has emerged as a particularly effective medium, as shown in methods such as DAPG [3], which combine demonstrations with behavioral cloning. However, most existing approaches are not incremental or modular, and often assume simple grippers with limited dexterity. Incremental skill acquisition offers a more flexible alternative: policies are trained progressively and reused to bootstrap more complex behaviors, such as extending a proximity grasp into a grasp-and-lift [4]. This strategy reduces training complexity and supports dexterous hands as well as simpler grippers. In parallel, the compositionality of tasks has been explored through policy combination [5], predefined libraries [6], or integration with task and motion planning (TAMP) [7] [8]. Yet these methods often intertwine learning and planning, limiting modularity and reuse across platforms. Our framework builds on these insights by merging two complementary directions: (i) incremental learning of dexterous and simple skills from VR demonstrations, and (ii) symbolic composition of learned and predefined behaviors within a repository queried by a Hierarchical Task Network (HTN) planner. Training and demonstrations are carried out in Unity, exploiting its VR integration, while validation is performed in CoppeliaSim with different manipulators, endeffectors, and objects not seen during training. This integration supports scalable and lifelong learning robotic manipulation tasks. II. INCREMENTAL LEARNING FROM DEMONSTRATIONS The first pillar of the framework is incremental skill acquisition from demonstrations. Demonstrations are collected in Virtual Reality (VR) using Unity, which offers a natural interface for human teleoperation and seamless integration with ML-Agents for policy training. This setup allows operators to intuitively provide demonstrations by moving a virtual manipulator, while the system records trajectories of end-effector motion and grasp commands. Policies are trained with a combination of Behavioral Cloning (BC), Proximal Policy Optimization (PPO), and Generative Adversarial Imitation Learning (GAIL). BC is employed to quickly align the policy with expert behavior, while PPO ensures stable policy updates and GAIL introduces imitation-based intrinsic rewards. A key contribution of the incremental approach is policy reuse. For instance, when using a multi-fingered BarrettHand, an initial policy is trained for proximity grasping, focusing exclusively on dexterous control of the hand. Once acquired, this policy is frozen and leveraged in a subsequent phase where the full manipulator learns a grasp-and-lift task. The manipulator must guide the hand to the correct region around the object and then invoke the already learned proximity grasp. This sequential training significantly reduces complexity and promotes modularity. The same incremental principle can be applied to simpler end-effectors such as parallel grippers. In this case, policies for grasp-and-lift can serve as building blocks for more complex behaviors like placement, which start from states sampled from successful grasps. By structuring training in stages, each new task capitalizes on previously acquired skills, lowering the number of demonstrations and training episodes required. III. REPOSITORY OF LEARNED BEHAVIORS The second pillar is the construction of a repository of skills that bridges low-level learned behaviors and high-level symbolic planning. Each learned policy is stored along with its symbolic specification, including preconditions, effects, and execution constraints. This representation allows policies to be used as operators within a planning domain, side by side 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629838 203 with predefined primitives such as move or release. The repository thus supports: • Learned primitives, incrementally acquired through VR demonstrations and RL (e.g., proximity grasp, grasp-and-lift, placement). • Predefined skills, which provide basic capabilities and ensure robustness in task execution. • Non-learned or missing behaviors, which are represented symbolically and can be composed with available skills until new demonstrations are recorded. By combining these elements, the repository acts as a procedural memory that grows over time, enabling continual learning and lifelong adaptation. Notably, it can store heterogeneous skills: dexterous multi-fingered grasps alongside simpler gripper-based manipulations. This design allows the system to flexibly switch between different platforms and levels of dexterity. IV. TASK COMPOSITION AND VALIDATION Task generation and execution are managed by an HTN planner, which decomposes complex goals into sequences of primitive skills retrieved from the repository. For example, a pick-and-place task can be constructed by composing a graspand-lift operator with a placement operator, while more complex activities, such as clear-and-place, require recursive decomposition into multiple subtasks. A distinctive aspect of the framework is cross-platform validation. While training and demonstration collection are performed in Unity, validation occurs in CoppeliaSim. This transfer allows policies to be tested with different manipulators (e.g., Franka Emika Panda) and end-effectors (different grippers from those used during training). Importantly, evaluation also includes objects not seen during training. In incremental learning with the BarrettHand, policies trained on primitive shapes generalized to novel objects such as bottles and hammers. Similarly, in the repository-based framework, policies learned for cylinders were successfully reused for pick-and-place and clear-andplace tasks involving glasses and bottles. These results highlight the adaptability of the approach: skills learned in simplified virtual settings can be transferred to new environments, new robotic platforms, and novel objects, demonstrating the feasibility of the proposed integration. V. CONCLUSION We presented a unified framework for robotic manipulation that combines incremental learning from VR demonstrations with symbolic task composition through an HTN planner. The system supports both dexterous anthropomorphic hands and simpler grippers, enabling progressive acquisition of manipulation skills and their reuse in structured tasks. A repository stores learned and predefined behaviors, acting as procedural memory and supporting longterm adaptability. Ongoing experimental evaluations indicate that incremental learning can reduce the complexity of training multi-fingered hands [4], while repository-based planning supports cross-platform transfer and the robust execution of long-horizon tasks. Together, these elements point toward a scalable and lifelong robotic task manipulation learning, where robots can progressively expand their skillset and flexibly compose behaviors to address complex goals. ACKNOWLEDGMENT This work was partially supported by the projects: EU Horizon INVERSE (grant 101136067); Melody (PRIN PNRR CUP E53D23017550001); SPACE IT UP (ASI MUR Contract 2024-5-E.0 - CUP I53D24000060005). References [1] M. Vecerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. M. O. Heess, T. Rothörl, T. Lampe e M. A. Riedmiller, «Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards,» in ArXiv, abs/1707.08817, 2017. [2] R. Caccavale, M. Saveriano, A. Finzi e D. Lee, «Kinesthetic teaching and attentional supervision of structured tasks in human-robot interaction,» Autonomous Robots, vol. 43, n. 6, pp. 1291--1307, 2019. [3] A. Rajeswaran, V. Kumar, A. S. J. Gupta, E. Todorov e S. Levine, «Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,» in ArXiv, abs/1709.10087, 2017. [4] G. Rauso, R. Caccavale e A. Finzi, «Incremental Learning of Robotic Manipulation Tasks through Virtual Reality Demonstrations,» Proc. of IROS, pp. 5176-5181, 2024. [5] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel e S. Levine, «Composable Deep Reinforcement Learning for Robotic Manipulation,» ICRA, pp. 6244-6251, 2018. [6] S. Nasiriany, H. Liu e Y. Zhu, «Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks,» Proc. of ICRA, pp. 74777484, 2022. [7] M. J. McDonald e D. Hadfield-Menell, «Guided Imitation of Task and Motion Planning,» Proc. of CoRL, pp. 630-640, 2022. [8] S. Cheng e D. Xu, «LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation,» IEEE Robotics and Automation Letters, vol. 8, n. 10, pp. 64516458, 2023. Fig. 1. Execution of a composite task to clear the table of cylinders before grasping the bottle and placing it in the box. The final behavior is composed of the learned grasp-and-lift and place policies, together with non-learned skills. 204