D1.2 - Initial setup of real world scenarios

This project has received funding from the European Union’s Horizon Europe - Research and Innovation program under the grant agreement No 101092100. This report reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. D1.2 Initial setup of real world scenarios Deliverable ID: D1.2 Project Acronym: HARTU Grant: 101092100 Call: HORIZON-CL4-2022-TWIN-TRANSITION-01 Project Coordinator: TEKNIKER Work Package: WP1 Deliverable Type: DEM Responsible Partner: FMI Contributors: ALL Edition date: 23 November 2023 Version: 04 Status: Final Classification: PU D1.2: Initial setup of real world scenarios Version 04 2 HARTU Consortium HARTU “Handling with AI-enhanced Robotic Technologies for flexible manufactUring” (Contract No. 101092100) is a collaborative project within the Horizon Europe – Research and Innovation program (HORIZON-CL4-2022-TWIN-TRANSITION-01-04). The consortium members are: 1 FUNDACION TEKNIKER (TEK) 20600 Gipuzkoa | Spain Contact: Iñaki Maurtua [email protected] 2 DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH (DFKI) 67663 Kaiserslautern | Germany Contact: Dennis Mronga [email protected]e 3 ASOCIACIÓN DE INVESTIGACIÓN METALÚRGICA DEL NOROESTE (AIMEN) 36418 Pontevedra| Spain Contact: Jawad Masood jawad.[email protected] 4 ENGINEERING INGEGNERIA INFORMATICA S.P.A. (ENG) 00144 Rome| Italy Contact: Riccardo Zanetti riccar[email protected] 5 TOFAS TURK OTOMOBIL FABRIKASI ANONIM SIRKETI (TOFAS) 34394 Istanbul | Turkey Contact: Nuri Ertekin nuri.erte[email protected] 6 PHILIPS CONSUMER LIFESTYLE BV (PCL) 5656 AG Eindhoven | Netherlands Contact: Erik Koehorst [email protected] 7 ULMA MANUTENCION S. COOP. (ULMA) 20560 Gipuzkoa | Spain Contact: Leire Zubia [email protected] 8 DEEP BLUE Srl (DBL) 00193 ROME | Italy Contact: Erica Vannucci erica[email protected] 9 FMI HTS DRACHTEN B.V. (FMI) NL-4622 RD Bergen Op Zoom, Netherlands Contact: Floris goet [email protected] 10 TECNOALIMENTI S.C.p.A (TCA) 20124 Milano | Italy Contact: Marianna Faraldi [email protected] 11 POLITECNICO DI BARI (POLIBA) 70126 Bari | Italy Contact: Giuseppe Carbone giuseppe.ca[email protected] 12 OMNIGRASP S.r.l. (OMNI) 70124 Bari | Italy Contact: Vito Cacucciolo [email protected] 13 INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE INCORPORATED (ITRI) 310401 Hsinchu | Taiwan Contact: Curtis Kuan [email protected] 14 INFAR INDUSTRIAL Co., Ltd (INFAR) 504 Chang-hua County | Taiwan Contact: Simon Chen [email protected] D1.2: Initial setup of real world scenarios Version 04 3 Document history Date Version Status Author Description 19/09/2023 01 Draft FMI Document template 18/10/2023 02 Draft FMI Partners contribution integrated 09/11/2023 03 Draft FMI DBL contribution on workshops, and minor changes 23/11/2023 04 Final FMI Version submitted D1.2: Initial setup of real world scenarios Version 04 4 Executive Summary In line with the user-centred and industrial driven approach followed by the HARTU project, the consortium has implemented an initial set of demonstrators as part of this strategy. The objectives are: • To create the setups for an iterative and incremental integration-test-redesign process • To have demonstrators where data acquisition campaigns can be carried out. This includes data for technical developments, but also to gather users feedback from the early stages of system building. A total of 11 prototypes are available, corresponding to the 8 use cases defined by the project. The broad spectrum of demonstrators includes robots and vision cameras of different brands (i.e., KUKA, FANUC, UR, OMRON robots or Photoneo and Zed2i cameras), which will ensure that the solutions are not hardware specific. The document includes also the methodology that is used for the User Research, as well as the initial insights on this topic which are the result of the literature review and the various workshops that have been held in the first 10 months of the project with different stakeholders from the 5 Industrial companies offering the validation scenarios. Finally, Legal and Ethical aspects that have to be considered are presented. D1.2: Initial setup of real world scenarios Version 04 5 1 Table of contents 1 Introduction ............................................................................................................................... 10 2 Overview of the first prototypes overview ................................................................................ 11 2.1 TOFAS – UC1 – Spare parts delivery preparation ............................................................... 12 2.1.1 Use case overview ........................................................................................................ 12 2.1.2 Prototype at TEK .......................................................................................................... 14 2.2 TOFAS – UC2a – Kitting in the automotive sector .............................................................. 17 2.2.1 Use case overview ........................................................................................................ 17 2.2.2 Prototype at TEK .......................................................................................................... 20 2.3 TOFAS – UC2b – Pre-assembly in the automotive sector .................................................. 22 2.3.1 Use case overview ........................................................................................................ 22 2.3.2 Prototype at DFKI ......................................................................................................... 22 2.4 PCL – UC3 – Handling for mass customization in the consumer good sector ................... 23 2.4.1 Use case overview ........................................................................................................ 23 2.4.2 Prototype at PCL .......................................................................................................... 24 2.4.3 Prototype at DFKI ......................................................................................................... 26 2.5 TCA – UC4 – Packaging operation in food sector ............................................................... 27 2.5.1 Use case overview ........................................................................................................ 27 2.5.2 Prototype at TEK .......................................................................................................... 28 2.5.3 Prototype at AIMEN ..................................................................................................... 30 2.6 INFAR – UC5 – Fixtureless assembly in hand tool manufacturing sector .......................... 33 2.6.1 Use case overview ........................................................................................................ 33 2.6.2 Prototypes .................................................................................................................... 34 2.7 ULMA – UC6 – Order preparation: pallet to pallet ............................................................ 37 2.7.1 Use case overview ........................................................................................................ 37 2.7.2 Prototype at ULMA ...................................................................................................... 38 2.7.3 Prototype at TEK .......................................................................................................... 40 2.8 ULMA – UC7 – Order preparation: box to box ................................................................... 42 2.8.1 Use case overview ........................................................................................................ 42 2.8.2 Prototype at TEK .......................................................................................................... 43 3 Updated risk assessment ........................................................................................................... 46 4 User research in HARTU’s Use-cases ......................................................................................... 48 4.1 SSH: Methodology and data collection methods ................................................................ 48 D1.2: Initial setup of real world scenarios Version 04 6 4.1.1 Phase I: Discovery ........................................................................................................ 48 4.1.2 Phase II: Understand .................................................................................................... 49 4.1.3 Phase III: Analyse.......................................................................................................... 51 4.1.4 Phase IV: Recommend ................................................................................................. 53 4.2 Use-Cases and data analysis ................................................................................................ 53 4.3 Findings ................................................................................................................................ 54 4.4 Co-designing workshops to delineate the TO-BE scenarios and next steps ....................... 68 5 Ethical and Legal aspects ........................................................................................................... 69 5.1 Ethics model ........................................................................................................................ 70 5.1.1 AI ethics evaluation framework in manufacturing ...................................................... 70 5.1.2 Ethics framework ......................................................................................................... 71 5.1.3 Implementation of AI ethics evaluation framework .................................................... 73 5.2 Legal Case ............................................................................................................................ 76 5.2.1 Purpose and scope of the method ............................................................................... 76 5.2.2 The process .................................................................................................................. 77 5.3 Next steps ............................................................................................................................ 78 6 Bibliography ............................................................................................................................... 78 7 Annexes ...................................................................................................................................... 80 8 Annex 1: User research data collection ..................................................................................... 80 9 Annex 2: Workshops Outcomes ............................................................................................... 207 List of figures Figure 1. Input box preparation in the warehouse (1) and process overview (2) ............................. 12 Figure 2. Order preparation at the workshop.................................................................................... 12 Figure 3. Example of different plastic bags used for packaging ........................................................ 13 Figure 4. Layout of the proposed preparation area (TOFAS)............................................................. 13 Figure 5. Simulation of TOFAS spare part order preparation scenario............................................. 13 Figure 6. Scaled-down version of UC1 in TEK .................................................................................... 14 Figure 7. Initial UC1 prototype ........................................................................................................... 14 Figure 8. Barcode reader .................................................................................................................... 15 Figure 9. Barcode reader in the demonstrator .................................................................................. 15 Figure 10. Products in special compartments.................................................................................... 17 Figure 11. Products in semi-structured configuration ....................................................................... 17 Figure 12. Products randomly distributed ......................................................................................... 17 Figure 13. Kitting preparation area. ................................................................................................... 18 D1.2: Initial setup of real world scenarios Version 04 7 Figure 14. Destination containers on the conveyor. The blue boxes contain the discs .................... 18 Figure 15. Push buttons to control the conveyor belt that transports the output containers ......... 18 Figure 16. Pick to light device above each input container ............................................................... 18 Figure 17. Operator taking a component from the input container (left side). It will then put on the destination container (right side) ...................................................................................................... 19 Figure 18. Design of the overall system ............................................................................................. 19 Figure 19. Scaled-down version of UC2a in TEK ................................................................................ 20 Figure 20. Parts are taken with a specific orientation but need to be re-picked to insert in the container ............................................................................................................................................ 21 Figure 21. Initial tool re-picking stations ........................................................................................... 21 Figure 22 Use case UC2b, pre-assembly of real wheel in automotive sector ................................... 22 Figure 23 Dual-Arm KUKA iiwa as demonstrator for UC2b ............................................................... 22 Figure 24. Side view proposed set-up ................................................................................................ 24 Figure 25. Top view ............................................................................................................................ 24 Figure 26. Current setup .................................................................................................................... 24 Figure 27 Prototype for UC3 at DFKI Robotics Innovation Center..................................................... 26 Figure 28 Kinaesthetic Teaching of an assemby task......................................................................... 26 Figure 29. Current process for zucchini sorting ................................................................................. 27 Figure 30. TCA Prototype concept ..................................................................................................... 28 Figure 31. TCA Prototype concept implemented at TEKNIKER .......................................................... 28 Figure 32. PSEN rd1.2 safety radar sensor ......................................................................................... 29 Figure 33. Detail of 3 of the ZED2i ..................................................................................................... 29 Figure 34. Vacuum gripper ................................................................................................................. 29 Figure 35. TCA demonstrator at AIMEN. A: Edge controller; B: Physical Demonstrator; C: Dummy Eggplants in container; D: Demonstrator concept from D1.1 ........................................................... 30 Figure 36: TCA demonstrator design with upgrades of perception, control and height. ................. 31 Figure 37. Main components of the wrench ..................................................................................... 33 Figure 38. Operators at the assembly tables ..................................................................................... 33 Figure 39. Three main assembly steps: Step 6 (left), Step 7 (middle), Step 8 (right) ........................ 34 Figure 40. Simulated components for assembly steps ...................................................................... 34 Figure 41. Simulated assembly process ............................................................................................. 35 Figure 42. 3D layout (left) and the AR605 (right) of the demonstrator ............................................ 35 Figure 43. Real example of output pallet at ULMA’s customer ......................................................... 37 Figure 44. Pallet to pallet process ...................................................................................................... 38 Figure 45. Example of multi-reference pallet .................................................................................... 38 Figure 46. Design of the demonstrator at ULMA ............................................................................... 38 Figure 47. Initial setup at ULMA......................................................................................................... 39 Figure 48. Tool changing station, with the two tools currently available ......................................... 40 Figure 49. Initial setup at TEK during the preparation phase ............................................................ 41 Figure 50. Manual picking .................................................................................................................. 42 Figure 51. Placing the products in the output box ............................................................................ 42 Figure 52. Design of the box to box order preparation ..................................................................... 43 Figure 53. Design of the demonstrator at TEK ................................................................................... 43 Figure 54. Setup pf he demonstrator at TEK ...................................................................................... 44 D1.2: Initial setup of real world scenarios Version 04 8 Figure 55. One of the 2 grippers with automatic quick-change mechanism ..................................... 45 Figure 56. The combined finger + suction gripper under study ........................................................ 45 Figure 57. Different phases of the approach undertaken ................................................................. 48 Figure 58. Human Factors Pie, readapted from EUROCONTROL ....................................................... 50 Figure 59. One of the three co-design workshops carried out during the progress meeting ........... 68 Figure 60. Framework dimensions and concept ................................................................................ 71 Figure 61. Ethical framework implementation process. .................................................................... 73 Figure 62. Example of possible results emerging from the analysis conducted in step 2 analysis ... 75 Figure 63. Example of possible dashboard of final results of ethical assessment ............................. 76 List of tables Table 1. Methods used for data collection ........................................................................................ 51 Table 2. Mapping of methods used for data collection with methods used for data analysis and representation ................................................................................................................................... 53 Table 3. Take Home Messages for each analysed Use Case: an overview ........................................ 54 Table 4. Description of dimension and sub-dimensions of the framework ...................................... 71 D1.2: Initial setup of real world scenarios Version 04 9 Acronyms List of the acronyms HARTU Handling with AI-enhanced Robotic Technologies for flexible manufactUring D1.2: Initial setup of real world scenarios Version 04 16 6. The control system provides the destination box according to the order to which it corresponds, and the label read. 7. The robot picks the part and navigates to the destination output box. 8. The robot calculates the position inside the output box and executes the release. 9. The robot takes an image of the output box once the part has been released. 10. The robot navigates to the position of the input box. (*) Step 5 can be executed during the navigation to the position of the output box. (**) Steps 3 and 10 can be executed simultaneously. D1.2: Initial setup of real world scenarios Version 04 17 2.2 TOFAS – UC2a – Kitting in the automotive sector 2.2.1 Use case overview This use case corresponds to the preparation of kits of components, an operation known as ‘kitting’ in the automotive sector. The subsequent pre-assembly step will be treated as a separate use case: UC2b pre-assembly at the corresponding assembly workstation. In the kitting area, products are taken from containers/boxes in which they can be arranged in two main configurations: (1) Product-specific individual compartments (Figure 10); (2) Semi-structured configuration (Figure 11), forming layers that are separated by means of separators (cardboard or plastic); randomly distributed products (Figure 12) are not included in the kitting operation, but are managed by the operators at the assembly station. Figure 10. Products in special compartments Figure 11. Products in semi-structured configuration Figure 12. Products randomly distributed Components that must be included in the kit are placed in containers on one side of the preparation area, and the destination containers on the other side, as shown in the next pictures. The only exception is the discs, which are placed inside blue plastic boxes on a shelf near the conveyor belt. D1.2: Initial setup of real world scenarios Version 04 18 Figure 13. Kitting preparation area. Figure 14. Destination containers on the conveyor. The blue boxes contain the discs To start the preparation of kits, the operator presses the button in Figure 15 and 10 empty output containers arrive at the preparation area on the conveyor belt. Figure 15. Push buttons to control the conveyor belt that transports the output containers Figure 16. Pick to light device above each input container Then, the operator starts the pick-and-place process: on top of each input container there is a pick-to-light device that shows the destination conveyor for each component ( Figure 16). The operator takes the component, leaves it in the corresponding output box on the conveyor and acknowledges the action on the pick-to-light device. D1.2: Initial setup of real world scenarios Version 04 19 Finally, 10 paper forms are taken and introduced in each destination box, and the operator presses the button to transport the full containers on the conveyor belt to the assembly workstation (at the back of Figure 13). Figure 17. Operator taking a component from the input container (left side). It will then put on the destination container (right side) In this UC, it is proposed to use the same mobile manipulator as in UC1 to handle the products, as it provides a more flexible alternative to other solutions like a robot mounted on linear tracks. Figure 18. Design of the overall system A prototype has been created at TEK and, after integration and validation of the HARTU results, it will be delivered to the TOFAS plant in Bursa for final demonstration. D1.2: Initial setup of real world scenarios Version 04 20 2.2.2 Prototype at TEK The demonstrator installed at TEK is a scaled-down version of the proposed overall system, due to the availability of physical space at the shopfloor. Figure 19. Scaled-down version of UC2a in TEK TEK is waiting for the delivery of components and containers from TOFAS. The demonstrator consists of the following elements: ● Mobile platform: Segway RMP omnidirectional mobile + KUKA IIWA 14. • Dimensions of the base: (W x L x H) 788 X 1350 X 897. ● A ZED2i camera mounted on the end of arm. ● A second ZED2i mounted on the platform to locate the output containers. Alternatively, an eye-in-hand camera configuration can be used for this purpose. ● 3 Grippers • One magnetic. • One 2-finger. • One 3-finger. ● Tool change station embarked on the robot. D1.2: Initial setup of real world scenarios Version 04 21 ● Part repositioning station. Some parts are picked using a 3-finger gripper or magnetic gripper, but the placement in the output container requires to re-pick the part in a different way using a 2-finger griper. This operation is done in this station. Figure 20. Parts are taken with a specific orientation but need to be repicked to insert in the container Figure 21. Initial tool re-picking stations The sequence of actions in the demonstrator will be: 1. The robot receives the list of items to be placed in each output box. For each of them: a) It navigates to the input container. b) It takes an image and identifies the best candidate. c) It picks the object with the 3-finger or magnetic gripper and places it on the repositioning station. 2. The robot navigates to the destination container. 3. For each part a) It takes an image of the area and locates the container accurately. b) The robot picks the part from the repositioning station and places it on the corresponding output container. c) It moves to the next container position. When necessary, the robot changes the gripper. D1.2: Initial setup of real world scenarios Version 04 22 2.3 TOFAS – UC2b – Pre-assembly in the automotive sector 2.3.1 Use case overview This use case corresponds to the pre-assembly of components after the kitting operation described in UC2a. The use case to be considered is shown in Figure 22. It shows the pre-assembly of the real-wheel drum, which includes the following steps (from left to right): (1) Washer loading, (2) Nut loading, (3) Nut pre-screwing, (4) Drum loading, (5) Pre-screwing, (6) Screwing. Figure 22 Use case UC2b, pre-assembly of real wheel in automotive sector 2.3.2 Prototype at DFKI The laboratory prototype set up at DFKI Robotics Innovation Center comprises two rigidly mounted KUKA iiwa industrial manipulators equipped with Robotiq 3-Finger grippers, an Ensenso RGB-D Camera which provides high-resolution images and point clouds for object detection, 4x ASUS XTion RGB-D Cameras with low resolution to be used for collision detection, as well as two Sick Laserscanners for workspace monitoring. The robot will be controlled via ROS2. The table behind the robot can be used as assembly area and will be set up accordingly for the UC2b use case. Figure 23 Dual-Arm KUKA iiwa as demonstrator for UC2b D1.2: Initial setup of real world scenarios Version 04 23 2.4 PCL – UC3 – Handling for mass customization in the consumer good sector 2.4.1 Use case overview Philips is a world leader in mass production of consumer goods. To do this efficiently, the current processes are designed for high volumes with little product variation. Currently there is a trend towards more personalization. This results in more product variation within a given process, requiring the equipment to be more flexible and also easily reconfigurable. A typical example of customization is the lacquering line, where parts are coated with a layer of lacquer to match the product design to the consumer’s need. A wide range of products is fixtured manually by operators on the jigs that are going into the spray booths. Image – fixturing at a loading/unloading station Image – loading/unloading stations The scope of the HARTU prototype is the automation of the insertion and removal of parts on/from the lacquering jigs. This prototype will demonstrate flexibility, as it should be able to work with different product variants and colors (e.g., chest pieces and front panels). Furthermore, it should be easy to reconfigure or train the system for new product introductions. Image – Chest panel Image – Front panel In addition, this prototype will show the adjustability of a complex fixturing motion to a changing environment (i.e a freely rotating jig). Placement of the parts on the jig is a complex wrist motion, D1.2: Initial setup of real world scenarios Version 04 24 that is different for each position on the jig. The parts are attached to the jig using a snap-fit connection. Correct placement can be checked by an audible click of the snap-fit joints. Finally, the removal of parts from the jigs must always be done carefully to avoid scratching the newly applied surface finish. 2.4.2 Prototype at PCL The demonstrator to be installed at PCL will focus on the placement and removal of products on the lacquering line. To demonstrate the repeatability of the action, this will be done in a continuous loop. Products are picked from a single tray and placed back in the same tray with the support of the perception system. Figure 24. Side view proposed set-up Figure 25. Top view The state of the current set-up is as shown in Figure 26. This is an initial set-up that allows the start of the data acquisition campaign. Figure 26. Current setup The initial setup at PCL consists of the following parts: D1.2: Initial setup of real world scenarios Version 04 25 - 6-axis robot with 700mm reach. Type: Kuka kr6 R700 sixx - 2 Structured-light 3D Scanners. Type: Photoneo PhoXi 3D scanner M - Manual gripper change system. Type: Schunk SHS, size 50 - 2-Finger servo-electric gripper. Type: Weiss CRG 30-050 - Gripper fingers. Type: Custom 3D printed fingers The sequence of steps for this prototype is as follows: A. Place the pieces until the jig is full: 1. Grab image of tray. 2. Estimate pick pose. 3. Pick product from tray. 4. Grab image of jig. 5. Estimate place pose. 6. Place product on jig. B. Remove the pieces until the jig is empty. 1. Grab image of jig. 2. Estimate pick pose. 3. Remove product from jig. 4. Grab image of tray. 5. Estimate place pose. 6. Place product in tray. Upcoming and potential set-up changes: - The system is controlled by a PLC and the robot is programmed using vendor specific software. In the next phase we will use the ROS based architecture integrating HARTU results. This might require changing the robot or using/developing an appropriate robot driver. The following robots are considered: Kuka Kr6 R700 six and TM5-700. - Currently the jig is in a fixed position, therefore estimation of the jig position is not required for the placement of a part. This functionality is expected when implementing HARTU results. - New gripper concepts can be tested in this setup. For mounting the gripper it is advised to comply with the specifications of the Kuka flange or the Schunk SHS 50 adapter plate. D1.2: Initial setup of real world scenarios Version 04 32 ▪ The six Zed2i cameras work as the edge perception module and send the processed information to the central PC. ▪ The Zed2i cameras can be moved along the infrastructure. ▪ The height of the infrastructure can also be adjusted. o Robotics setup -> acquiring forces/trajectories/etc: Grasping module and Continuous monitoring module. ▪ Universal robot is installed at the centre of the demonstrator. ▪ The new gripper technology will be installed on the TCP. ▪ The demonstrator will be connected to the pneumatic lines for the vacuum gripper technologies. D1.2: Initial setup of real world scenarios Version 04 33 2.6 INFAR – UC5 – Fixtureless assembly in hand tool manufacturing sector 2.6.1 Use case overview The INFAR use-case will focus on the ratchet wrench assembly. The main components to be assembled in the ratchet wrench are shown in Figure 37. Figure 37. Main components of the wrench Currently, operators assemble this ratchet wrench manually as shown in Figure 38. Figure 38. Operators at the assembly tables D1.2: Initial setup of real world scenarios Version 04 34 HARTU will focus on three main steps of the assembly process, as shown in Figure 39. • Assembly step 6 – Block insertion • Assembly step 7 – Ratchet insertion • Assembly step 8 – C-shaped ring insertion Figure 39. Three main assembly steps: Step 6 (left), Step 7 (middle), Step 8 (right) Automation of these assembly steps is challenging because of the small size of the parts that have to be manipulated. 2.6.2 Prototypes A multi-robotic assembly system has been proposed for automating the assembly steps described in this use case. The demonstrator concept and the physical implementation is shown in next figures. The main idea is to have the first robot arm to hold the main body of the ratchet wrench and the second robot to pick the components one by one following the assembly steps 6, 7, and 8. The assembly is performed via the coordination among the two robots. Before doing the assembly task, simulations on assembly steps are used to determine the best robot moving paths, configurations, coordination among robots, etc. as shown in Figure 40. Figure 40. Simulated components for assembly steps D1.2: Initial setup of real world scenarios Version 04 35 The demonstrator will be implemented following the simulation set-up shown in Figure 41. Figure 41. Simulated assembly process The components included in the real demonstrator are the following: • Two robots: AR605 or SJ605 (both are designed by ITRI) • Specifically designed gripper to grasp the ratchet and the wrench. (Gripper is being designed by ITRI). • Robot Cell Controller (eMIO designed by ITRI). • Centralised robot coordination of the two robots with high-level commands, such as grasping points on the workpieces. • Perception system, consisting of cameras at the ceiling or close to the robot end-effector to provide visual information for further object recognition, grasp planning, etc. FOVISION or SENSOPART products will be adopted. • Multi-axis force/torque sensors mounted at the end-effectors to provide contact force information for assembly tasks. • Workpiece loading/unloading mechanism for robot grasping of workpieces for next assembly movement. (designed by ITRI). Figure 42 shows the 3D layout and the AR605 of the demonstrator. Figure 42. 3D layout (left) and the AR605 (right) of the demonstrator Other HARTU results will be integrated as they become available. D1.2: Initial setup of real world scenarios Version 04 36 As described, currently simulation for this demonstrator has been used to verify possible solutions and designs, assembly procedures and algorithms. Since the specific designed gripper and the workpiece loading/unloading mechanism is under manufacturing by the contractor. The real implementation of the demonstrator is scheduled for Dec, 2023. The sequence of actions in the demonstrator will be: 1. Placing the main bodies of the ratchet wrench, the C-shaped rings, ratchets, and blockmodules into the loading mechanisms. 2. The first robot picks a main body of the ratchet from the loading mechanism. 3. The second robot picks the block-module from the loading mechanism. 4. The assembly step 6 is performed by the second robot to insert the block-module into the main body of the ratchet wrench held by the first robot. 5. The second robot picks the ratchet from the loading mechanism. 6. The assembly step 7 is performed by the second robot to place the ratchet into the main body of the wrench held by the first robot. 7. The second robot picks the C-shaped ring from the loading mechanism. 8. The assembly step 8 is performed by the second robot to place the C-shaped ring around the ratchet. 9. Then, the first robot will place the assembled ratchet wrench into a basket. D1.2: Initial setup of real world scenarios Version 04 37 2.7 ULMA – UC6 – Order preparation: pallet to pallet 2.7.1 Use case overview In logistics, there are different types of order preparation procedures depending on how the products arrive at the preparation area and how the orders are delivered. • Input o Products arrive on pallets, this is mainly the case of bulky products packaged in carboards, large cans and sacks. o Small size products arrive in boxes, sorted or randomly distributed. • Output o Products are stacked on pallets, either of one or multiple references. o Products are placed in boxes, sorted or unstacked (randomly). This use case corresponds to the case in which products arrive on mono-reference pallets and are delivered on multi-reference pallets. In the order preparation area operators pick units from the incoming pallet (the one that has been transported from the warehouse) and place them on the pallets that will be finally delivered to the customer, as shown in Figure 43. Some of the features of the use case are the following: • The incoming pallets (Euro Pallet, EPAL) are always mono-reference and the output boxes are, usually, multi-reference. • The warehouse management system informs the operator of the number of units that have to be picked from the incoming pallet. This information is available in a GUI and is displayed in a pick-to-light system. • Operators manipulate the product by hand, and with the help of industrial manipulators for the heaviest products (they can weight up to 30 kg). • In some few occasions, the incoming pallet transports a box with products inside, which must be manipulated individually to complete an order (e.g., to take a can from the box an put them on the output pallet). • Operators use their own criteria to create the output pallet, trying to find the best combination to create stable pallets. For that, sometimes they move the already placed items and reposition them. • . Figure 43. Real example of output pallet at ULMA’s customer D1.2: Initial setup of real world scenarios Version 04 38 Figure 44. Pallet to pallet process Figure 45. Example of multi-reference pallet Two prototypes will be created: one at TEKNIKER for the validation of partial results and the final one at ULMA. The main two differences among them are: • The type of robot used. • The way the input pallets are transported to the picking station (using a conveyor in the case of ULMA and manually in the case of TEKNIKER). Both prototypes are described in the following sections. 2.7.2 Prototype at ULMA The demonstrator concept and the physical implementation is shown in Figure 46 and Figure 47: Figure 46. Design of the demonstrator at ULMA The robot will pick products from the pallets transported on the conveyor and will create a new multi-reference pallet (an order). D1.2: Initial setup of real world scenarios Version 04 39 Figure 47. Initial setup at ULMA The components included in the demonstrator are the following: • Robot FANUC R-2000Ic/210F. • Conveyor for transporting the input pallets (circular path). • 1 Photoneo L above the picking station (input pallet). • 1 ZED2i mounted on the robotic arm to monitor the status of the output pallet. • 2 suction grippers of different sizes. The currently available grippers are: o JOULIN CG-VG 400x400-J-P20-3STx8 o JOULIN EGV2-VG-125x400-J-4P30-3STx1 • Tool exchanger station and exchange system rsp P1804; Sn: 0108 • A control PC D1.2: Initial setup of real world scenarios Version 04 40 Figure 48. Tool changing station, with the two tools currently available The sequence of actions in the demonstrator will be: 1. The system calculates the mosaic (position of each part) to be created for an order. 2. The warehouse management system delivers the input pallets based on the position of the products in the mosaic. 3. For each pallet arriving at the picking station: a) An image of the input pallet is captured with the fixed camera. b) The system decides which product has to be picked. c) The robot picks the product. d) The robot places the product in the corresponding position of the mosaic. e) The robot takes an image of the mosaic with the embedded camera. If there is a mismatch with respect to the proposed mosaic, it stops and an alarm is generated (light or message) to inform the operator. f) The sequence is repeated for the number of items to be picked. When finished, the pallet leaves the picking station and a new one arrives. 2.7.3 Prototype at TEK The prototype at TEK is like the one at ULMA, with two main differences: • Robot KUKA KR210 R2700-2 /FLR. • There is not a conveyor to transport the input pallets. Instead, they will be moved by hand. D1.2: Initial setup of real world scenarios Version 04 41 The sequence of actions is similar to that of ULMA, except that the input pallets are moved by hand. Figure 49. Initial setup at TEK during the preparation phase D1.2: Initial setup of real world scenarios Version 04 48 4 User research in HARTU’s Use-cases 4.1 SSH: Methodology and data collection methods User research is the practice of studying and understanding people’s behaviours, needs, pain points and their motivation about their experiences in usage of technologies. The understanding of users’ needs is essential to define effective requirements for the design of technologies and solutions. In HARTU, user research has been used as a central process to study in a comprehensive way the five pilots so as to propose design recommendation, needs, and requirements centred on the users’ perspective. This will ensure the design of solutions acceptable to users. A four-phase process (Figure 57) has been defined at the beginning of the project to define the current ASIS context and to support the partners involved in the initial set-up of the real-world scenarios (T1.2). The four phases of the user research approach (Explore, Understand, Analyse and Recommend) are described in the following paragraph, with the description of the main methods applied and the specific outcomes generated in each phase. The outcomes of the process are directed to the designers and technology providers and are specific insights providing take home messages on the context, working layout and processes in which the designed solution will need to operate. User needs are preliminary insights that are part of the wider working system that will be impacted by the integration of HARTU results and that will need to be considered to ensure a smooth technological transition. Figure 57. Different phases of the approach undertaken 4.1.1 Phase I: Discovery In the discovery phase, a general introduction to each Industrial Use Case has been carried out. The main objective for this phase was to understand the relevant scenarios for HARTU in each Industrial Use Case and D1.2: Initial setup of real world scenarios Version 04 49 to define the possible logistics of field-studies. A series of interviews with the Manufacturing Line / Logistic Line (ML/LL) representatives were conducted and relevant documentation from use cases was collected. Based on the analysis of the gathered data, a first version of the scope and objectives of each scenario was outlined. This exploratory phase clarified the key roles operating in the scenarios analysed, as well as their responsibilities and tasks. 4.1.2 Phase II: Understand The main scope of phase II was the planning of field visits to the Industrial Use Cases to better understand the interactions between the key roles identified in Phase I and machines (both in its software and hardware components), as well as other related factors relevant to their tasks that might have indirectly affect interactions or be a result of these interactions (e.g., the need to collaborate with other members of staff, skills and competences needed to work effectively with the provided systems, or the comfort of the working environment). Understanding the way that the work is organised in the ML/LL currently (e.g., different roles and responsibilities, or shift patterns), and identifying areas of opportunities where the impact of the new solution could positively influence the processes also at higher organisational levels (e.g., number of employees needed to conduct the work, redefinition of operators tasks and upskilling/reskilling of competencies) is very important when integrating a new system in the context. The overarching goal in this Phase was to delineate the current scenarios (AS-IS) in each use case taken into consideration, and to highlight all the socio-technical related aspects to be considered when designing HARTU’s technological solution. An in-depth evaluation of the context and the related production processes and environment was undertaken. To plan the data collection activity from a Human Factors perspective, a series of categories were considered as a lens to observe and analyse the scenarios. The categories chosen took inspiration from EUROCONTROL’s Human-Factor Pie (EUROCONTROL, 2011) which is a framework that considers a series of Human-Factor related categories and sub-categories to be used in order to analyse a changing context. After a readaptation of the framework, six categories and sub-categories were identified as relevant to explore in HARTU’s scenarios. The categories, illustrated in Figure 58 are namely: ● Working Environment: The workspace, the general equipment and machinery used, and the physical environment; ● Organisation of work: Organisational, production and people management within a work setting, consideration of personal and cultural factors and issues related to the management of organisational changes; ● Skills and Training: The systematic development of competencies required by individuals to adequately perform their work; ● Roles, Procedures and responsibilities: Actual/prescribed working methods, positions/functions in the organisation and expected tasks performed by relevant roles; ● Teams and communication: How people work and communicate with each other on shared goals and tasks. ● Human-machine interaction: The actions, reactions, and interactions between humans and other system components. D1.2: Initial setup of real world scenarios Version 04 50 Figure 58. Human Factors Pie, readapted from EUROCONTROL For each category, a rationale, a set of questions to pose and aspects to be observed in each pilot was developed accordingly. These points of inquiry were then selected prior to the field-visits in preparation of the interviews outlines for each key role and were used as prompts during the field-work helping in gathering useful information during the observations and in general in note taking during the visits. In the following the methods for data collection used during the field visits are better detailed but it should be noted that an overarching application of the Human Factor framework was applied to ensure a comprehensive methodology to be used across the different scenarios. Investigations at the ML/LL plants were carried out using a mixed-method approach consisting of three main different activities aimed at gathering qualitative data, namely: semi-structured interviews, and observations. The methods used a focused-on approach offering diverse levels of interaction with study participants and considering a range of data sources (e.g., existing materials, internal and external documents sourced from the use-cases, individual knowledge, and experience of key roles). Table 1 shows the methods, followed by a brief description and their objectives. D1.2: Initial setup of real world scenarios Version 04 51 Table 1. Methods used for data collection Methods Description Objectives Contextual interviews with different representatives of the current workforce In-depth understanding of the wider organisational context in which employees are working (also including aspects such as organisation of work, training provision and communication), together with further exploration of HMI-related pain points identified during Observations and Physical Ergonomics Assessment, and related opportunities. Direct observation of the users while they are conducting the work. During the observation, a physical Ergonomics assessment can be performed through direct observation and direct measurements (e.g., light, noise). Understanding the key tasks and points of interaction with interfaces and/or machines, the actual tasks performed and potential insights to discuss during interviews. The data collected provides an overview of potential improvement for the user, as well as informing the tech designers on ergonomics, organisational and technical aspects to consider. The data collected during the field-studies visits to the ML/LL was the starting base for the analysis done in Phase III. 4.1.3 Phase III: Analyse In this phase, the data collected in Phase II are analysed providing insights on the ASIS scenarios. In this phase the use case context, tasks, challenges, opportunities, as well as a physical ergonomic assessment of the key roles observed and interviewed is provided. The analysis of the data was carried out using specific user research methods described in the following. Personas Personas are realistic representations of user types. They are created based on user research, collecting data about real users. Personas are a common tool used in human-centred design to represent users’ goals, needs, skills and behaviours and are used for understanding users’ expectations and help during the design phase of a solutions in making the right questions and taking into consideration the users’ perspectives. Personas help the understanding of users’ needs, goals, and behaviours. The outputs outlined through personas are: ● Description of tasks ● Skills and Expertise D1.2: Initial setup of real world scenarios Version 04 52 ● Workspace ● Typical interactions with tools ● HumanHuman Interaction User Journey Maps (UJ) User Journey Maps represent the different steps a user does to reach a goal and/or to perform a task. User Journey Maps highlight interactions of the users with tools and systems, other people, and roles within the organisation. The User Journey Map makes it possible to briefly and visually describe a person’s experience, highlighting also their thoughts, pain points, and emotions. The challenges and opportunities for improvements related to the key activities are then identified in the applicable areas. This method is useful for understanding users' current challenges in reaching a goal and the identification of points of strength and weakness in it. It also provides information about the frequency of interactions at different steps of the process and the means in which these interactions take place with humans (e.g., email, face to face, phone call, document handling) and/or with tools (e.g., computers, software, machineries, forklifts etc.). The outputs outlined through a user journey map are: ● Workspace location ● Supporting roles ● Digital touchpoints ● Physical touchpoints ● Other supporting tools ● Challenges ● Opportunities Hierarchical Task Analysis (HTA) Hierarchical Task analysis describes a specific task by breaking it into specific smaller sub-tasks. It makes it possible to visualise all the different steps to complete a task. It maps the way in which users complete a task, and it is based on data collected and observations from the real world. The creation of a Hierarchical Task Analysis supports the identification of possible critical sub-tasks, which can be improved, simplified, or changed to smooth or improve the completion of the task. The critical sub-tasks are the ones where users can have the major struggles, or where specific difficulties (e.g., organisational, technical, contextual) can lead to suboptimal performances. Hierarchical Task Analysis help outline the following: ● Key tasks and sub-tasks ● Interactions with tools and other relevant roles. Physical Ergonomics Assessment The physical ergonomics assessment considers a number of factors that are currently involved in the pilot's role: workstation area, indoor lighting, use of tools, noise-related hazard, microclimate-related hazard: assessment of indoor temperature, considering also seasonal variations, pollutant-related hazard, vibrationrelated hazard, problems arising from the use of machineries, and biomechanical overload. The data collection is performed through direct observation and direct numerical measurements (e.g., light, noise). The analysis of the data of a Physical Ergonomics assessment identifies the physical factors that should be considered for the implementation of technology, focusing on the main problems identified during the observations. The assessment has been remodelled to better fit any pilot scenario, involving the usage of D1.2: Initial setup of real world scenarios Version 04 53 specific methods to investigate the case as the employ of RULA method for assessing postures. These analyses have been deployed to gain a better understanding of the main issues related to the physical ergonomics aspect of the work performed. This involves initially assessing the current state and then collecting data to determine what technology can achieve in order to reduce physical effort and redesign tasks. A physical ergonomic assessment provides the following information: ● Analysis of the current possibilities for improving the physical ergonomics state ● Considerations regarding how the redesigned tasks could mitigate the emerging issues. The methods used to analyse the qualitative data collected, offered different ways of analysing, presenting, and identifying meaningful patterns within data. In Table 2 a mapping on how the methods used for data collection are linked to the methods used for data analysis and representation is done. Table 2. Mapping of methods used for data collection with methods used for data analysis and representation Methods for data analysis and representation Methods for data collection Personas User Journey Maps Hierarchical Task Analysis Physical Ergonomics Assessment Semistructured Interviews x x Observations x x x 4.1.4 Phase IV: Recommend The results from the data analysis phase were used to create specific design recommendations for each Industrial Use Case. The form used for the recommendations are the “Take Home Messages” (THM). THMs include possible design opportunities and suggestions to consider while designing HARTU solutions to mitigate the challenges identified in each pilot. Analysing the data in the AS-IS scenario gives the opportunity to highlight relevant take-home-messages and considerations that might determine and advance the design of HARTU’s technological solutions, including to a higher degree the end-user needs. Moreover, having a detailed overview of the AS-IS scenarios will help thinking through the possible changes in roles, responsibilities, procedures, tasks, that will need to happen through the adoption of HARTU’s solutions. 4.2 Use-Cases and data analysis The data collection conducted during the field work in the first six months of the project, was analysed for each pilot taking into account the different use cases of interest. The data gathered was analysed through the methods described in Section 1.2.3. D1.2: Initial setup of real world scenarios Version 04 54 The analysed data has been included in Annex 1. The results and key findings deriving from each Use Cases analysis are displayed in Section 4.3. 4.3 Findings In Table 3 an overview of the main findings and a list of recommendations generated through the findings are outlined for each Use Case and scenario of interest. The findings and recommendations are to be considered as Take Home Messages (THM). These examples of preliminary user needs and requirements derived from the analysis of the data collected. The source codes and job-steps illustrated in the table refer to the specific use cases’ analysis from which the finding was extracted (Section 1.2). The source code is formed by the pilots’ name (e.g., PILOT), the use case analysed (e.g., PILOT-1), the output from which the finding was extrapolated (e.g., P as per “Personas”; HTA as per “Hierarchical Task Analysis”, PEA as per “Physical Ergonomics Assessment”, UJ as per “User Journey” etc.) and the key role to which the output is mapped on (e.g., OP as per “Operator”, TL as per Team Leader etc.). User requirements and needs will be finalised together with the partners in a joint effort that will be further discussed in Section 4.4. The results deriving from the data analysis will be updated at different stages throughout the evolution of HARTU’s project. Table 3. Take Home Messages for each analysed Use Case: an overview UC1 – TOFAS – Spare parts delivery preparation Ref Source code Job Step Finding Recommendation #1 TOFAS1/P1_OP 3.1 Inserts order’s related goods in the outbound boxes The operators often need to grasp very heavy materials (boxes weight more than 14kg) that are sometimes also slippery (e.g., plastic bags). This causes discomfort and biomechanical overload of upper limbs. Introducing a device/crane/machine to reduce the effort perceived by the operators during the picking, lifting, carrying, and positioning activities would significantly reduce their biomechanical overload of upper limbs. The device/crane/machine would support the operators in their daily workload. #2 TOFAS1/UJ1_OP 2.1 Area meeting and reallocation of resources The daily reallocation of personnel into the different working areas impacts the operators’ productivity and efficiency as they interact with machinery, tools and, Machines, tools, interfaces should have a common and homogeneous designed visual interface across the working area to increase their learnability. The operators would be facilitated by the D1.2: Initial setup of real world scenarios Version 04 55 eventually, interfaces, which do not always have the same structure and visual language across the plant. same visual information and commands to be displayed and this would in turn allow for an efficient reallocation of personnel. #3 TOFAS1/UJ1_TL 5.3 Reallocate resources across the area The daily reallocation of personnel into the different working areas impacts the operators’ productivity and efficiency as they interact with machinery, tools and, eventually, interfaces, which do not always have the same structure and visual language across the plant. Team Leaders should have the opportunity to choose among the different resources available considering a number of variables as the time a specific operators spend working in the different area or the familiarity they might have with the different machinery tools or interfaces. Those variables can be better studied in order to valorise this reallocation and increase efficiency and satisfactions of operators. #4 TOFAS1/UJ1_OP 3.1.1 Scans the inbound box and casually picks a part To grasp parts in the inbound box the operator needs to bend in order to reach parts that are positioned in different depths of the box. The parts to reach in the box should be higher placed and better angled, to ensure that the operators maintain a better posture. A platform could be placed under the inbound boxes to alleviate this issue. #6 TOFAS1/PEA1_OP 3.1.5 Position the part inside the outbound box To correctly position the picked parts in the outbound boxes instead of throwing them the operators necessarily need to bend in order to gently releasing them and avoiding breakages. Although, the outbound boxes, differently from the inbound boxes, are not designed to facilitate this operation because the walls of the boxes are straight, this increases the vertical displacement of Fixed postures imposed by the boxes could be alleviated by introducing angled or adjustable platforms where parts could be easily displayed, reducing operators’ discomfort postures such as deep bending of their trunks. D1.2: Initial setup of real world scenarios Version 04 56 loads. #7 TOFAS1/UJ1_OP 3.1.4 Move to the outbound boxes area and searches for the correct outbound box The usage of electrical and manual forklifts it is not always practical in tight spaces and implies the necessity of increasing spatial awareness to conduct technical actions to ensure safety moving of parts. When a forklift is used, the surroundings have to be free of obstacles, which means that operators’ around must not carry out their tasks while it is being used. Forklifts should be enhanced to operate safely within dynamic environments, and thorough studies are needed to ensure the system's safety and reliability. The addition of sensors capable of monitoring movements and automatically stopping the forklift to prevent potential collisions could significantly alleviate the operators' workload and enhance overall productivity efficiency. This would enable operators to use the machine while concurrently performing other tasks within the same working area, thereby expediting operations, and ensuring safe operation. #8 TOFAS1/UJ1_OP 3.1.2 Scan the label on the product with the barcode reader The barcode reader is often carried in the operators’ hands limiting the operators’ capabilities to grasp slippery and heavy objects. Finding a different way to carry the barcode reader would increase productivity and reduce the workload of the operator avoiding possible slips of the parts on the ground. The system could consider this issue and find another way to scan the parts and the inbound and outbound boxes, and/or could carry a barcode reader across areas in more efficient ways. As a result, the operator should be able to grasp objects with both hands when needed. #9 TOFAS1/UJ1_OP 3.1.7 Inserts the quantity needed in the box and confirms Operators use gloves to carry out most of their activities although they need to interact with some interfaces throughout the process. The gloves that they wear are an impediment to the usage The operators need to be able to interact with a system while wearing gloves, which means that if a new system to be implemented foresees the insertion of a touchscreen it should be designed enhancing the touch sensitivity feature to D1.2: Initial setup of real world scenarios Version 04 57 of the touchscreen, as the fingertips of the gloves are not designed to use a touch interface. Instead, the area on the back of their gloves does work on screens, resulting on the operators’ using their back side of the hand to unlock them. increase operators’ precision in the interaction with the interface, avoiding the use of the back side of their hands or the removal of the entire glove. UC2 – TOFAS – Kitting and pre-assembly Ref Source code Job Step Finding Recommendation #1 TOFAS2/UJ1_TL 5.3 Reallocate resources across the area The daily reallocation of personnel into the different working areas impacts the operators’ productivity and efficiency as they interact with machinery, tools and, eventually, interfaces, which do not always have the same structure and visual language across the plant. Machines, tools, interfaces should have a common and homogeneous designed visual interface across the working area to increase their learnability. The operators would be facilitated by the same visual information and commands to be displayed and this would in turn allow for an efficient reallocation of personnel. #2 TOFAS2/UJ1_OP & TOFAS-2/ PEA1_OP 1.Kitting The human-machine interaction and the communication among colleagues (human-human interactions) is affected by 83dB of sound pressure. The value of surrounding sound pressure should be considered during the design process of potential signals and/or alerts, in order to ensure a clear exchange of information between the system and the operators, and among the operators themselves. #3 PEA_2 [AA] 2. Assembly Biomechanical overload of upper limbs: the task is organized in cycles and characterized by similar working gestures for over 50% of the time. The force perceived for the task in a To reduce/improve/ the biomechanical overload of upper limbs an assessment on OP30 using the OCRA method is recommended. The results deriving from the assessment will help establishing the best D1.2: Initial setup of real world scenarios Version 04 64 from the conveyor line and through it to the wase box is a repetitive task that requires operators move boxes from the conveyor line (initial height) up to the highest point of the waste boxes (final height), weights vary. #9 TCA/UJ3_AS 8.4 Manages the creation of the outbound pallet. The process of supervising is on the area supervisor. Different actions are performed throughout the shift and there is the possibility of making mistakes, especially in a period of high workload. The technology has the potential to provide improved support for these operations by serving as a third eye on the tasks performed and assisting in error correction. It could enhance the accuracy and efficiency of tasks such as printing the right information to affix on boxes, counting operations, especially on the creations of pallets and weight registering and reporting, operations that nowadays are performed manually without any specific support or technology assistance. UC5 – INFAR – Fixtureless assembly in hand tool manufacturing sector Ref Source code Job Step Finding Recommendation #1 INFAR/UJ1_ OP 3.Perform the assembly Demanding tasks are assigned to specific operators who may develop specialized skills. However, the prolonged pinch posture required by these tasks can place significant strain on the operator, increasing the risk of injury or discomfort. To mitigate the challenges that the nature of the operations requires, careful considerations on the workstation design should be given. The workstation design, with a specific attention to the desk layout, should minimize frequent handling or movement of wrenches and parts, especially those that imply the presence of the pinch grip. The repetitiveness of the tasks should be avoided in order to create sequences D1.2: Initial setup of real world scenarios Version 04 65 where the operators can alternate the movements performed reducing stereotypies. #2 INFAR/UJ1_ OP 3.Perform the assembly The operator works mainly seated upright without backrest. Bending forward and frequent twisting of trunk are required to finish the task. Postures currently used are not recommended and require redesigning of the process. Biomechanical overload and awkward posture are to be considered when redesigning the work layout and implementing adjustable working desks and supports to reduce discomfort through operators. #3 INFAR/UJ_O P 1 Receives information on the activity to perform from the MES. The operators need to move across the area to receive the information provided from the MES. The information could be displayed in a different way reducing the necessity for the operators to move across the area. UC6/UC7 – ULMA – Order preparation Ref Source code Job Step Finding Recommendation #1 ULMA_UJ1_ OP 1.1. Check items to be added to the pallet The system currently slows the operation as it signals the amount of items needed for the order in a small part of the computer display, highlighting the current orders and not considering upcoming orders. The user should be provided with an interface where more detailed information about the orders are displayed (e.g., the type of items arriving, and the upcoming items after). This would significantly help the user to have a clear overview of current as well as upcoming orders to support the planning of his tasks and have a full supervision of the system. D1.2: Initial setup of real world scenarios Version 04 66 #2 ULMA_UJ1_ OP 1.5. Moves to the pickto-light area to extract items needed from light boxes Operators are slow down in their process whenever they need to include only one or two light items into an order as currently items are boxed in triplets. When this happens, operators need to unbox the items needed, extract the number of items required in the order, re-box them to insert them in the pallet and write on the box with the remaining items the number of items that can be found in the box after the extraction of the ones needed. The user should be provided with individually stored items to manage the handling of individual items. This would help the user in reducing time devoted to the unboxing of the items needed (if less than 3) and help the user in reducing possible errors counting items. #3 ULMA_UJ1_ OP 2. Picking rack restock The operator in the picking of the light boxes does a repetitive movement, as well as a cognitive effort in reminding how many boxes (or items) he putted in the rack, and how many he still needs to put. The display and the computer do signal the number of items needed and not the number of boxes (each box usually contains three items). So it can happen that, especially when there are a higher number of items to count the operator makes mistakes placing the wrong number of items on the rack. The user should be provided with external support to count the items loaded on the orders as well as those still missing. The system could help by making easier the retrieval of the right number of items and/or maintain the counting of the items while the operator is loading the order. This would reduce possible errors when counting items. If the operations were to be substituted by the new system, would still be significant for the operator to successfully monitor the process. #4 ULMA /PEA1_OP 1. Pallet Forming Operators often bend their trunk during pushing and pulling, and their hands are often held under 60cm during the forming of the pallet, and during the pushing and pulling of loads as it uses vertical The operators should assume better postures in case of manual settings of the pallet, avoiding holding loads below the 60cm. The redesigned line should consider a different height according to the standard ISO 11228 -1 & ISO D1.2: Initial setup of real world scenarios Version 04 67 force (partial lifting) 11228 - 2. #5 ULMA /PEA1_OP 1. Pallet Forming Operators currently use cranes implemented in the heavy weight picking area to help them handling heavy cans. However, the height of the prehensile handles is positioned above the elbows and higher than the maximum reached point for some operators whom, as a result, need to stretch the entire body to reach it. The operator could be facilitated in the reaching of the crane if the system were designed in such way that it could recognise the operators’ height and set and adjust its handling point to avoid scratches or jumps. #6 UJ_WM n/a Whenever the operator finds in the line a damaged good to be replaced, he/she needs to contact the warehouse manager who will promptly and manually re-insert the order into the warehouse management software. When a product is damaged or leaking, the new system should be able to: a) signal the problem to a monitoring operator that can intervene and call the warehouse manager as it is done now. b) automatically assess the damage and the number of items to be changed and communicate the information to the warehouse manager. c) automatically assess the damage and the number of items to be changed and replan the order from the warehouse. In the first case (a) the system should be provided with an alarm to recall the operator’s attention. D1.2: Initial setup of real world scenarios Version 04 68 #7 UJ_WM n/a The warehouse manager needs to promptly know how many damaged items were replaced to ensure correct inbound orders and to always maintain an updated and organised inventory. The warehouse manager should be always updated on the missing, damaged and/or replaced items to ensure an updated and organised inventory. 4.4 Co-designing workshops to delineate the TO-BE scenarios and next steps The Take-home-messages played a crucial role in addressing a set of needs and outlining considerations for further exploration in terms of technology and its design, or human intervention and the way humans will need to interact with the technology to assist, cooperate, or collaborate with it in different tasks of the overall processes. During the second face-to-face meeting of HARTU (23-25th October 2023), three workshops were conducted in order to explore, together with all the partners, the evolution of future TO-BE scenarios. The primary focus of the activity was to engage participants in reflecting on the role of humans in the imagined new scenarios and to imagine what is expected of them to support the system and intervene appropriately based on system requirements and needs. Understanding the system and delineating how the human role can get involved, being responsible for more cognitively engaging activities while also interacting with the technological system helps outline user requirements. To initiate the co-designing sessions, a quick revision of the AS-IS Hierarchical Task Analysis was conducted, emphasising the current roles of operators and considering how and where they perform their tasks. Then, the most advanced version of the technological concepts derived from D1.1 was presented, highlighting what the robot can/cannot do at the moment considering its demonstrators’ design, the associated risks identified by the technical partners as general limitations, and their possible mitigations. Figure 59. One of the three co-design workshops carried out during the progress meeting D1.2: Initial setup of real world scenarios Version 04 69 At this point, the exercise revolved around a printed canvas representing 4 main areas of interest: • Tasks (Operator) • Tasks (Technology) • Communication • Risks • Environment The current tasks carried out described in D1.1 were printed out on cards that could be moved around the canvas. The moderator read the first task related to the technology and placed it onto the “Tasks (technology)” line. At this point, the moderator started asking whether any risks could be occurring and whether information as an input or output was expected to be delivered from the technology to the operators (and vice versa) in those cases the moderator collected the answer on a set of post-it and placed them on the corresponding task of the canvas. Depending on the type of information or communication exchange needed at the time, tasks that a human role might need to carry out could arise. These tasks were added to the Tasks (Operator) template row. The purpose of the workshop exercise was to prompt reflection on the communication flows necessary between technology and humans to effectively support each other. This involved considering what kind of information is to be shared and when, and how humans can support the system in case of failures or unexpected situations. It also considered the surrounding environment and how it can affect the new task, sharing of information, and the associated risks. The development of a new task analysis focusing on the TO-BE scenarios and including two different actors (technology and humans) was the output of the workshops and can be found in Annex 2. The task analysis is to be considered a first step that should be repeated until the technological system comes to its completion and its final design. This essential exercise will be conducted individually in the different team groups, as well as jointly in the consortium to successfully design the new process. The interactions that emerged in the workshop will be used as input for T1.3 and T1.4. After this activity, and when the technological system is at a more mature stage it will be necessary a new set of reviewed HTA, User Journey, and Personas identifying user needs and setting parameters for the TO-BE context. 5 Ethical and Legal aspects Alongside user requirements, Ethical and Legal aspects cannot be underestimated. These may affect the solutions' trustworthiness as well as their acceptability in the medium-long term. Therefore, it is advisable to take into account these aspects from the first phases of the project, when designing a technical solution. This will facilitate the innovation process, preventing the possibility that legal and ethical issues would act as showstoppers for the development and deployment of new technologies. D1.2: Initial setup of real world scenarios Version 04 70 It should be noted that the ethics assessment methodology has been developed by Deep Blue and previously implemented by Deep Blue in the XMANAI Research Project. Similarly, the legal case methodology has been produced by Deep Blue and implemented in research projects in the aviation domain such as ARGON, AEON, and HAIKU." 5.1 Ethics model 5.1.1 AI ethics evaluation framework in manufacturing The Ethics Guidelines for Trustworthy Artificial Intelligence (AI) is a document prepared by the HighLevel Expert Group on Artificial Intelligence (AI HLEG), an independent expert group was set up by the European Commission in June 2018, as part of the AI strategy. The AI HLEG presented a first draft of the Guidelines in December 2018. Following further deliberations by the group in light of discussions on the European AI Alliance, a stakeholder consultation and meetings with representatives from Member States, the Guidelines were revised and published in April 2019. Based on these specific Ethics Guidelines for Trustworthy AI, an AI system should have three main characteristics: be lawful, ethical and robust, to do so seven main requirements have been defined by the expert group. While the document from the European Commission Expert Group is one of the most relevant addressing ethical issues for AI, many other contributions exist, and, as highlighted by Hagendorff (2020), there are some issues related to the implementation of the identified guidelines: there are not mechanisms to ensure the compliance with the various codes of ethics, there are no consequences in case of deviations. Also, there is the risk to use ethics as a marketing strategy. There is the need to bridge the abstract ethics values and the technical implementations, to have effective ethical AI systems. Thus, the implementation of ethical guidelines should be tailored accordingly to the specific context of AI application (Floridi, 2019) and starting from the design phase, with an AI ethics that look at individual situations and specific technical assemblages (Hagendorff, 2020). As discussed, there are many recommendations for how to ethically design AI, but very few frameworks to support ethical AI evaluation and development. Moreover, when it comes to the failures of such systems, very little is documented on how their consequences can be contained. Most literature seems to simply warn of the risks of its failure, not on their mitigation and response. Furthermore, at the moment there is a lack toward specific framework for the manufacturing sector, while at the same time several studies highlight the ethical risks of introducing AI in the industrial sector such as in the case of loss of human skills (Torresen, 2018), and automated decision-making (Mpofu & Nicolaides, 2019). Another emergent risk, particularly relevant also in the manufacturing sector, is the liability and responsibility of AI activities (Coeckelbergh, 2020), which should be clearly defined. Explainability and transparency are a way to improve and answer liability concerns. D1.2: Initial setup of real world scenarios Version 04 71 5.1.2 Ethics framework The AI ethics evaluation framework is applicable to any AI technology and solution, already existing or to be designed. The approach, that it is explained in the next section, wants to be a holistic approach, that considers both the AI technology to be implemented and the context where it is implemented (see Figure 60). Figure 60. Framework dimensions and concept A set of dimensions covering different ethical aspects have been defined starting from the proposal of the European Commission High-Level Expert Group on AI, with the addition of a specific dimension added to evaluate specifically the risks regarding the liability of AI. The framework is designed to be applied with an iterative approach along the duration of the project to evaluate and monitor, during the different stage of HARTUs implementation, the ethical risks, and understand if the mitigation measures identified during the first iteration are effective. In the next section is presented the implementation process of the framework. The framework dimensions are presented and described in Table 3 . Description of dimension and sub-dimensions of the framework. Table 4. Description of dimension and sub-dimensions of the framework Dimension Description Sub-dimensions Privacy and Data Governance Only essential data for achieving the process is to be used for processing by the AI system. Suitable anonymity is to be kept at the time of algorithm supervision. · Respect for privacy – Full GDPR compliance at all stages Quality and integrity of data – Checks in place to ensure truthful inputs and outputs D1.2: Initial setup of real world scenarios Version 04 72 Dimension Description Sub-dimensions Legitimised access to data – Protection measures in place for users when accessing their data Technical Robustness and Safety AI algorithms should include security measures protecting the parties whose data is being processed by the AI. The integrity of the data needs to be protected by breaches and system failures. Resilience to attack and security – Security measures put in place to protect data Fault tolerance and general safety – Emergency measures in case of system failure Accuracy – Measures in place to maintain accuracy and avoid misinputs or human error Reliability – Tested for consistency before deployment Reproducibility – Sufficiently documented for easy reproduction in case of failure Diversity, Nondiscrimination and Fairness Inclusion and diversity must be considered, to avoid biases that could be included in the models due to data used. Biases could lead to discrimination and harm to certain groups of people. Consequently, action must be taken to avoid said biases. Avoidance of unfair bias – Checks on the algorithm preventing marginalisation of groups Accessibility – The system must be accessible to stakeholders with disabilities and impairments Universal design – Stakeholders involved throughout design and operational lifecycle Societal and Environmental Well-being Clear system documentation for the upskilling and reskilling of upcoming AI system operators. Environment variables taken into account for environmentally-conscious system design and operation. Adaptability – Impact on workers (reskilling/upskilling) Environmental impact – Monitoring for environmentally friendly decision-making Social impact – Monitored effect on communities affected by the AI decision-making Fundamental rights – Sustaining human rights Human Agency and Oversight AI algorithms should be accessible for oversight from all stakeholders. Human supervision is required to reestablish agency to human supervisors and regain control over decision making processes involving AI. Human oversight – human supervision for every stage of the algorithm Human agency – The ability for human modification of AI at any point Transparency AI should have systems in place to make the algorithm traceable and explainable at every stage of the process. Model weights should be accessible and Traceability – Full operational traceability ensuring correct operations at each stage of the AI process D1.2: Initial setup of real world scenarios Version 04 73 Dimension Description Sub-dimensions understandable through component analysis. Inputs and outputs need to be clearly outlined. Explainability – Each stage of the process should be explainable to stakeholders Awareness – Humans need to be aware of the AI system and its capabilities/limitations Accountability Automatic auditing measures should be put in place to easily identify and highlight flaws in the system. Version control must be used to revert harmful changes and identify failing areas for easier and faster modification. Auditability – Measures in place to make the system easy to examine and evaluate Impact – Negative impacts considered and reported, measures in place to minimise it Redress – Easy modification of failing areas/systems Liability All parties involved with the AI system and products of its operation need to be made aware of the possible hazards and failures and how reliability can be traced back to each party. Liability clarity – It is clear where the liability of the systems can be traced back to each actors involved 5.1.3 Implementation of AI ethics evaluation framework The AI ethics evaluation framework aims to facilitate a systemic and contextual approach to the ethical issues related to the design, development, and implementation of these new technological solutions. In particular, the main three goals of this tool are: 1. promoting a user-friendly approach from a fair reading of contextual ethical issues; 2. facilitating a dynamic and ongoing understanding of ethical principles in practice; 3. introducing a consistent assessment methodology for a handy and fair trade-off of the competing interests at stake, also considering each new technological solution that could be implemented. The AI ethics evaluation framework implementation is based on a four steps iterative process (see Figure 61). Figure 61. Ethical framework implementation process. D1.2: Initial setup of real world scenarios Version 04 80 7 Annexes 8 Annex 1: User research data collection