D4.1 - ASSEMBLY DATASET

This project has received funding from the European Union’s Horizon Europe - Research and Innovation program under the grant agreement No 101092100. This report reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. D4.1 ASSEMBLY DATASET Deliverable ID: D4.1 Project Acronym: HARTU Grant: 101092100 Call: HORIZON-CL4-2022-TWIN-TRANSITION-01 Project Coordinator: TEKNIKER Work Package: WP4 Deliverable Type: DATA (dataset) Responsible Partner: DFKI Contributors: DFKI, AIMEN, TEKNIKER Edition date: 28 June 2024 Version: 06 Status: Final Classification: [PU] D4.1 Assembly Dataset 2 HARTU Consortium HARTU “Handling with AI-enhanced Robotic Technologies for flexible manufactUring” (Contract No. 101092100) is a collaborative project within the Horizon Europe – Research and Innovation program (HORIZON-CL4-2022-TWIN-TRANSITION-01-04). The consortium members are: 1 FUNDACION TEKNIKER (TEK) 20600 Gipuzkoa | Spain Contact: Iñaki Maurtua [email protected] 2 DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH (DFKI) 67663 Kaiserslautern | Germany Contact: Vinzenz Bargsten vinzenz.barg[email protected]e 3 ASOCIACIÓN DE INVESTIGACIÓN METALÚRGICA DEL NOROESTE (AIMEN) 36418 Pontevedra| Spain Contact: Jawad Masood jawad.maso[email protected] 4 ENGINEERING INGEGNERIA INFORMATICA S.P.A. (ENG) 00144 Rome| Italy Contact: Riccardo Zanetti riccardo.zanet[email protected] 5 TOFAS TURK OTOMOBIL FABRIKASI ANONIM SIRKETI (TOFAS) 34394 Istanbul | Turkey Contact: Nuri Ertekin [email protected] 6 PHILIPS CONSUMER LIFESTYLE BV (PCL) 5656 AG Eindhoven | Netherlands Contact: Erik Koehorst [email protected] 7 ULMA MANUTENCION S. COOP. (ULMA) 20560 Gipuzkoa | Spain Contact: Leire Zubia [email protected] 8 DEEP BLUE Srl (DBL) 00193 ROME | Italy Contact: Erica Vannucci [email protected] 9 FMI HTS DRACHTEN B.V. (FMI) NL-4622 RD Bergen Op Zoom, Netherlands Contact: Floris goet [email protected] 10 TECNOALIMENTI S.C.p.A (TCA) 20124 Milano | Italy Contact: Marianna Faraldi [email protected] 11 POLITECNICO DI BARI (POLIBA) 70126 Bari | Italy Contact: Giuseppe Carbone [email protected] 12 OMNIGRASP S.r.l. (OMNI) 70124 Bari | Italy Contact: Vito Cacucciolo [email protected] m 13 INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE INCORPORATED (ITRI) 310401 Hsinchu | Taiwan Contact: Curtis Kuan [email protected].tw 14 INFAR INDUSTRIAL Co., Ltd (INFAR) 504 Chang-hua County | Taiwan Contact: Simon Chen [email protected] D4.1 Assembly Dataset 3 Document history Date Version Status Author Description [30/04/2024] 01 Draft DFKI Table of Content definition [17/05/2024] 02 Draft DFKI Summary [29/05/2024] 03 Draft AIMEN File descriptions [19/06/2024] 04 Draft TEK Image dataset descriptions [21/06/2024] 05 Draft DFKI Demonstration dataset descriptions [28/06/2024] 06 FINAL DFKI Version submitted D4.1 Assembly Dataset 4 Executive Summary The overall topic of the HARTU project is to develop robotic technologies which allow a more flexible robotic automation of manufacturing and logistics tasks. One element towards achieving this goal is the application of data-driven machine learning approaches. In the course of the method development in the project, multiple datasets related to robotic assembly have been created for this purpose. This document describes a dataset of user demonstrations for learning and generalizing robotic assembly skills, a dataset of measurements of a novel type of force sensor using fiber optics, as well as image datasets of the objects being handled in various containers. The datasets have been published on Zenodo and are accessible at the links listed in the Summary chapter. The deliverable described by this document is part of work package WP4 of the HARTU project, which deals with Learning and control of contact-rich assembly skills and also includes some datasets generated in “WP2 Common software infrastructure for integration, simulation and perception”. D4.1 Assembly Dataset 5 Table of contents 1. Introduction ................................................................................................................................ 7 1.1. Purpose and scope of the datasets ...................................................................................... 7 1.2. Outline of the document...................................................................................................... 7 2. Assembly Dataset for Learning from Demonstration of Contact Based Tasks (DFKI) ................. 8 2.1. Description of the data acquisition process ........................................................................ 8 2.1.1. Experimental setup .......................................................................................................... 8 2.1.2. Data collection ................................................................................................................. 8 2.2. Dataset Contents.................................................................................................................. 9 2.2.1. Demonstration Subsets .................................................................................................... 9 2.2.2. Data in each recording ................................................................................................... 10 2.2.3. File format and directory structure ............................................................................... 10 2.3. Data processing .................................................................................................................. 11 3. Fiber Optic Force Sensor Experiments for Suction and Finger Grippers (AIMEN)..................... 12 3.1. Experimental setup ............................................................................................................ 12 1. Suction cup ......................................................................................................................... 12 2. Tactile fingers ..................................................................................................................... 12 3.2. Data collection and description of files ............................................................................. 12 1. Folder Suction cup ............................................................................................................. 12 2. Folder TactileFingers .......................................................................................................... 13 3.3. Data processing .................................................................................................................. 14 4. Images dataset (TEKNIKER) ........................................................................................................ 16 5. Summary .................................................................................................................................... 17 List of figures Figure 1. The experimental robotic setup used to acquire the user demonstrations ......................... 8 Figure 2. Interface for defining the visual appearance and physical characteristics of parts ........... 16 Figure 3. Interface to define a scene ................................................................................................. 16 Figure 4. Interface to define the dataset of images to be generated ............................................... 16 D4.1 Assembly Dataset 6 Acronyms List of the acronyms HARTU Handling with AI-enhanced Robotic Technologies for flexible manufactUring PbD Programming by Demonstration FBG Fiber Bragg Grating DOF Degrees of freedom D4.1 Assembly Dataset 7 1. Introduction 1.1. Purpose and scope of the datasets The overall topic of the HARTU project is to develop robotic technologies which allow a more flexible robotic automation of manufacturing and logistics tasks. To achieve this goal, machine learning methods play a major role, since they allow the use of data-driven solutions instead of solutions hand-crafted by experts for specific applications. More specifically, one of the tasks which has been defined to achieve the goals of the HARTU project is Learning assembly operations through demonstrations. In more general terms, this is also known as Programming by Demonstration (PbD): the idea is to allow users of a robotic system such as a manipulator arm to teach the motions it has to do for certain task and to avoid a software-based programming. Simplify such that configuration of a systems is less reliant on expert programming skills. In HARTU, contacts and generalization are considered in the problem. However, for these methods respective data are required. Therefore, a dataset containing a variety of recordings of a user demonstrating assembly operations to a robotic manipulator arm is described and published with this document. Since assembly relies on locating, grasping, and sensing the objects, the scope of this deliverable has been extended: it is not limited to the assembly operations itself, but also includes related datasets that have been created in the first period of the project. Specifically, the accurate sensing of forces and contacts is crucial for a robotic system to fulfill the assembly and handling, in particular grasping, of parts. For this purpose, a fiber optic-based sensor is developed, which can be integrated into gripper pads and suction cups. The corresponding dataset of the measurements of such sensor elements is described in this document. The last type of datasets described in this document are segmented and labelled virtual images of the objects being handled in the HARTU project, such as automotive parts, barrels, and vegetables located in different type of containers or palletized. These images can be used, for example, to develop machine-learning-based grasp strategies or object pose estimation methods. We have expanded the deliverable's scope to encompass not just assembly operations but also other pickand-place operations. This expansion includes additional datasets we've developed during this period. 1.2. Outline of the document The document is outline as follows. Each dataset published with this deliverable is described in a separate chapter: Chapter 2 describes the dataset related to the assembly demonstrations, Chapter 3 describes the dataset related to the development of a fiber optic force sensing method, and Chapter 4 describes multiple image datasets with objects being handled within HARTU. Chapter 5 summarizes the published datasets. D4.1 Assembly Dataset 8 2. Assembly Dataset for Learning from Demonstration of Contact Based Tasks (DFKI) This dataset is a detailed collection of motion and visual data during kinesthetic demonstration gathered for the HARTU project, specifically for two use cases: - PCL: assembly of plastic shaver housing parts - TOFAS Assembly: assembly of metal automobile axle and brake parts The dataset supports research and development of methods for learning from demonstrations of contact based tasks in robotics, focusing on motion planning, control, and computer vision applications related to contact based assembly tasks. This dataset has been uploaded to ZENODO. It has been assigned the following DOI as unique identifier and is accessible at this address: https://doi.org/10.5281/zenodo.12513790 2.1. Description of the data acquisition process 2.1.1. Experimental setup The experimental setup consists of a manipulator arm with 7 degrees of freedom (KUKA LBR iiwa 14 R820), an end-effector force torque sensor (Robotiq FT-300), a two-finger gripper (Robotiq 2F140), and RGBD camera (Intel Realsense D455). The manipulator arm is mounted vertically, onto a metal column, which is mounted on a table. At the front of the table, the parts to be assembled are located. The general setup is shown in the following figure: Figure 1. The experimental robotic setup used to acquire the user demonstrations 2.1.2. Data collection The robotic manipulator arm is controlled in a gravity compensating mode such that it can be moved freely by external forces. A human operator then guides the end-effector of the D4.1 Assembly Dataset 9 manipulator with his hands into the target positions and orientations of the segments defined by each task. Thus, a task is composed of a sequence of these demonstrations. During this procedure, joint torques, joint velocities, joint positions, end-effector positions, forces and torques, as well as RGBD camera images are recorded in the HDF5 format and simultaneously in rosbag format using the tool rosbag2. 2.2. Dataset Contents 2.2.1. Demonstration Subsets The following demonstrations of the assembly tasks have been recorded: a) Use case PCL, part 1 i) The complete assembly procedure (1) Trial 1 (2) Trial 2 ii) The procedure sub-divided into 4 separately recorded assembly steps (1) Trial 1 (2) Trial 2 b) Use case PCL, part 2 i) The complete assembly procedure (1) Trial 1 (2) Trial 2 ii) The procedure sub-divided into 3 separately recorded assembly steps (1) Trial 1 (2) Trial 2 c) Use case TOFAS, part 1 i) The complete assembly procedure (1) Trial 1 (2) Trial 2 ii) The procedure sub-divided into 3 separately recorded assembly steps (1) Trial 1 (2) Trial 2 d) Use case TOFAS, part 2 i) The complete assembly procedure (1) Trial 1 (2) Trial 2 ii) The procedure sub-divided into 3 separately recorded assembly steps (1) Trial 1 (2) Trial 2 In summary, 34 individual recordings are included in this dataset. D4.1 Assembly Dataset 16 4. Images dataset (TEKNIKER) This dataset contains rendered images of multiple objects from the HARTU use cases in different containers. The initial datasets generated in the project have been uploaded and shared to ZENODO, HARTU community. In all cases the datasets have been created with the tools developed in WP2, i.e., using the simulation environment and CAD of parts provided by end-users and others publicly available. The datasets have been assigned a DOI as unique identifier: • HARTU AUTOMOTION DATASET: https://doi.org/10.5281/zenodo.12155482 This dataset is created using CAD models of different automobile parts. There are four distinct datasets: part1, part2, mosaic and carboard boxes. For each image, the location of each part is provided at the box level with labels in YOLO format. Each dataset is divided into training, validation and test subsets. • HARTU PALLETIZING AND BOXING DATASET: https://doi.org/10.5281/zenodo.12155351 This dataset is created using different objects CAD models. There are two distinct datasets: palletizing and multiobject boxing. For each image, the location of each object is provided at the box level with labels in YOLO format. Each dataset is divided into training, validation and test subsets. • HARTU VEGETABLES DATASET: https://doi.org/10.5281/zenodo.12155113 This dataset is created using vegetables CAD models. There are three distinct datasets: tomatoes, eggplants, and zucchinis. For each image, the location of each fruit is provided at the box level with labels in YOLO format. Each dataset is divided into training, validation and test subsets. For the development of the dataset, the tools developed in T2.3 have been used, which offers three interfaces: one for defining the visual appearance of the parts, a second one for defining the scenes and a third one for defining the dataset. For more details, see “D2.3 Simulation infrastructure for handling component training”. Figure 2. Interface for defining the visual appearance and physical characteristics of parts Figure 3. Interface to define a scene Figure 4. Interface to define the dataset of images to be generated D4.1 Assembly Dataset 17 5. Summary To summarize, the following datasets have been published with this deliverable: Dataset DOI Assembly demonstrations https://doi.org/10.5281/zenodo.12513790 Fiber-optic force sensor measurements https://doi.org/10.5281/zenodo.11387093 Image dataset, automotive parts https://doi.org/10.5281/zenodo.12155482 Image dataset, palletizing and boxing of parts https://doi.org/10.5281/zenodo.12155351 Image dataset, vegetables https://doi.org/10.5281/zenodo.12155113