CARPOS Deliverable 1- Requirements, use-cases and KPIs of CARPOS

The project is carried out within the framework of the National Recovery and Resilience Plan Greece 2.0, funded by the European Union - NextGenerationEU (Implementation Body: HFRI) CARPOS Coachable Robot for Fruit Picking Operations D1Requirements, use-cases and KPIs of CARPOS Abstract: The purpose of the present deliverable is to report on the user needs and expectations from the CARPOS robotic system and describe the user requirements, as well as the CARPOS project use cases that derived from the conducted analysis. In this scope, the present deliverable reports the results of the user requirement collection and analysis performed. More specifically, the requirements, related measures and respective Key Performance Indicators (KPIs) for autonomous robotic fruit picking operations are investigated and defined, based on the Objectives set to the description of action (Technical Document). Both the requirements and the related measures are in turn drafted to both the technical experts and the agriculturerelated scientists of the project for evaluating their importance. Based on this evaluation, the KPIs of the project are finally defined. Furthermore, from among the fruit picking operations, the process of drafting the use-cases is conducted, that capture and put to challenge the system requirements for the proposed technologies. The system requirements defined, as well as their importance will be utilized for constructing the overall architecture of the solution and alongside with the use-cases and the KPIs will constitute the pillars for the experimental evaluation and testing conducted in WP6 of the project Dissemination Level: PU CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS Περίληψη στα Ελληνικά: Σκοπός του παρόντος παραδοτέου είναι η αναφορά των αναγκών και των προσδοκιών των χρηστών από το ρομποτικό σύστημα CARPOS και η περιγραφή των απαιτήσεων των χρηστών, καθώς και των σεναρίων χρήσης του έργου CARPOS που προέκυψαν από την ανάλυση που διενεργήθηκε. Προς αυτή την κατεύθυνση, το παρόν παραδοτέο αναφέρει τα αποτελέσματα της συλλογής και ανάλυσης των απαιτήσεων χρήστη που πραγματοποιήθηκαν. Πιο συγκεκριμένα, διερευνώνται και καθορίζονται οι απαιτήσεις, οι σχετικές μετρικές (Related measures - RMs) και οι αντίστοιχοι Βασικοί Δείκτες Επίδοσης (Key Performance Indicators - KPIs) για αυτόνομες ρομποτικές εργασίες συλλογής φρούτων, με βάση τους στόχους που έχουν τεθεί στην περιγραφή της δράσης (Τεχνικό Έγγραφο). Τόσο οι απαιτήσεις όσο και oι σχετικές μετρικές συντάσσονται με τη σειρά τους τόσο στους τεχνικούς εμπειρογνώμονες όσο και στους συνεργαζόμενους γεωπόνους του έργου για την αξιολόγηση της σημασίας τους. Βάσει της βαθμολόγησης των παραπάνω, εξάγονται τελικά οι δείκτες αξιολόγησης του έργου. Επιπλέον, μεταξύ των εργασιών συλλογής φρούτων, διεξάγεται η διαδικασία σύνταξης των σεναρίων χρήσης, που βασίζονται στις απαιτήσεις συστήματος για τις προτεινόμενες τεχνολογίες. Οι απαιτήσεις συστήματος, όπως αυτές καθορίζονται στο παρόν έγγραφο, καθώς και η σημασία τους θα χρησιμοποιηθούν για την κατασκευή της συνολικής αρχιτεκτονικής της λύσης και παράλληλα με τα σενάρια χρήσης και τους δείκτες επίδοσης θα αποτελέσουν τους πυλώνες για την πειραματική αξιολόγηση και δοκιμή που θα πραγματοποιηθεί στο Πακέτο Εργασίας 6 (WP6) του έργου. Document Status Document Change Log Each change or set of changes made to this document will result in an increment to the version number of the document. This change log records the process and identifies for each version number of the document the modification(s) which caused the version number to be incremented. Change Log Version Date Template Creation 0.1 Apr 11th 2024 First draft sent to partners for gathering their inputs 0.2 Apr 25th 2024 Answers from the technology developers received 0.3 May 1st 2024 Document Title Requirements, use-cases and KPIs of CARPOS Version 1.0 Work Package 1 Deliverable 1 Contributors Dimitrios Papageorgiou (PI), Michael Sfakiotakis and Ioannis Louloudakis Due Date 16/07/2024 Completion Date 16/07/2024 Confidentiality PU CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS Input from agricultural scientist received 0.4 May 31st 2024 Agricultural scientist answers 0.5 Jul 2nd 2024 Final consolidation 1.0 Jul 16th 2024 1. INTRODUCTION Current harvesting techniques performed by humans, is mainly performed either using a cutting tool/device, such as the harvesting of grapes, or without cutting tools, i.e. by hand, such as apple or peach picking. In order to distinguish between the two categories, we will henceforth refer to the latter one as “fruit picking” operation. Harvesting through fruit picking ensures the safety of the plant, the fruit, the farmer and/or the surrounding human workers, while in some cases it is easier to perform than harvesting using a cutting tool/device. In particular, there are many cases in which the fruit is very close or even attached to the surrounding environment, e.g. branches, leaves etc., without leaving cutting affordances (avoiding in this way the transmission of phytopathological diseases between healthy and diseased plants). Furthermore, when picking the fruit without a cutting tool the safety of the farmer and the surrounding human workers is ensured, as there are no dangerous tools involved in the process. Some examples of such fruit picking operations are detailed in Chapter 2 of this deliverable. To automate fruit picking using a robotic system, currently available solutions propose the explicit programming of the robot involving the specification of exact motion and/or force patterns in order to detach the fruit from the plant. This process requires considerable technical knowledge, it is time-consuming and lacks flexibility and adaptability to different types of crops. Moreover, in such cases, the life-long experience of farmers is not fully exploited to improve the effectiveness and performance of the robotic fruit picking action. The CARPOS project aims at providing a novel, coachable and human-friendly mobile robotic system, equipped with a robotic manipulator and multiple sensors, able of capturing and exploiting the lifelong experience of farmers through Learning by Demonstration (LbD), in order to successfully perform autonomous fruit picking operations. To achieve this, the system will be capable of intuitively accumulating knowledge through both kinesthetic and/or visual coaching by the human farmers, during its operation, thereby providing easy deployment, as well as flexibility and adaptability to different types of crops. Through the use of an intuitive humanmachine interface, the farmer will be able to monitor the process performed by the robot, and correct it whenever deemed necessary, by using the above coaching means. The goal of this deliverable is to conduct a review of users' needs and expectations in order to determine the CARPOS robot ecosystem's requirements and define the project's target use cases. More specifically, in Chapter 2, some representative fruit picking operations currently performed by human farmers are briefly presented. From those scenarios, two are selected as the use cases of the CARPOS project, namely that of the tomato (Elpida variety) and the prickly pear picking. The selected use cases are presented in detail in Chapter 3. In Chapter 4, the system requirements, related measures (RMs) and Key Performance Indicators (KPIs) are detailed. The system requirements and KPIs are built based on the Objectives of the project, via the RMs, and reflect the system targeted functionalities, e.g. the fruit picking performance, the safety of the fruit, the plant etc. CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS 1.1 RELATION TO OTHER DELIVERABLES The present deliverable will provide the basis for defining the final CARPOS system architecture and technical specifications. Both the use cases and the KPIs will serve as the guidelines for the final testing of the system, i.e. D8 - System Integration and testing, but also will give the scientific directions and aims for all the technical deliverables, i.e.: D3 - Fruit picking action identification and skill extraction from visual observation, D4 - Control schemes for occlusion-free active perception, D5 - Controller for unintentional and intentional pHRI during the coaching and autonomous operation, D6 - Graphical human-machine interface and skill extraction through proprioceptive data, D7 - Multimodal skill encoding and generalization in fruit geometry variations. The KPIs will play the role of our main metrics for evaluating the system during the development of the different modules, while the use cases will serve as the main testing setups for testing and developing the modules. 2. FRUIT PICKING OPERATIONS PERFORMED BY HUMAN Understanding the way humans grasp and detach the tomato fruits, including the factors and limitations that affect each type of grasp, as well as the typical grasp used in tomato picking, holds great scientific importance for ergonomic design and the development of efficient harvesting robot finger strategies. Human grasps can be categorized as power grasp and precision grasp (Napier, 19561). The power grasp is used to grasp larger size fruits such as apples, pears and peach while precision grasp (three-finger pinch) is usually employed in small size fruits such as berries, prickly pears, and cherries. Concomitantly, after each individual fruit has been detached from the branch, the farmer places them in specially designed storage crates minimizing post-harvest losses. Additionally, fruit sorting and packaging takes place either immediately after harvest in the field (i.e. strawberries) or after they have been transferred in the packing house (i.e. tomatoes), depending on a specific physiological characteristic of each plant species which determines the ripening process of the fruit before and after they have been harvested from the plants. Climacteric fruits have a fast period of ripening during which they change color and develop flavor and aroma and will continue ripen after they are harvested (i.e. bananas, tomatoes, pears, mangos, peaches, apples, and avocados) while non-climacteric plants produce fruits which stop ripen after the product has been harvested. Types of human-like tomato grasps As per Feix et al. (2016)2, the grasp taxonomy defines four distinct grasp types: Power palmThumb abduction, Power palm-Thumb adduction, Power pad-Thumb abduction and Pinch. In the Power palm-Thumb abduction type, the fingers, palm, and thumb wrap simultaneously around the tomato fruit, with the thumb generally opposed to the other fingers. In the Power palm-Thumb 1 NAPIER JR. The prehensile movements of the human hand. J Bone Joint Surg Br. 1956 Nov;38-B(4):902-13. doi: 10.1302/0301-620X.38B4.902. PMID: 13376678. 2 T. Feix, J. Romero, H. -B. Schmiedmayer, A. M. Dollar and D. Kragic, "The GRASP Taxonomy of Human Grasp Types," in IEEE Transactions on Human-Machine Systems, vol. 46, no. 1, pp. 66-77, Feb. 2016, doi: 10.1109/THMS.2015.2470657. CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS adduction type, the fingers and palm wrap around the tomato fruit, while the thumb is adducted on the side of the fingers. In the Power pad-Thumb abduction type, the fingers and thumb wrap around the tomato fruit, with at least two contact regions with each finger; the thumb opposes the other fingers, and the tomato fruit does not directly come into contact with the palm. In the Pinch type, the fingers and thumb squeeze the opposite sides of the tomato using fingertips without coming into contact with the palm. Figure 1. Four grasping types according to Feix etal. (2016), i) power palm-thumb abduction type, ii) power palm-thumb adduction type, iii) power pad-thumb abduction type and iv) pinch type (from left to right). 3. THE USE CASES OF CARPOS Fruit picking is currently mainly performed manually, e.g. by farmers. It imposes high physical and cognitive load to the worker, as it often involves non-ergonomic postures (kneeling or lifting the elbow above the level of shoulder) over long working hours and thus it is considered to risk the farmer’s health, in both short and life-long terms. To demonstrate and test the targeted functionalities, the objectives fulfillment, the effectiveness and the impact of the proposed solutions, we employ two scenarios of fruit picking, involving different fruit geometries, motion patterns, requirements and restrictions. The use cases are designed to reveal the potential added value gained from our developed methodologies and the generalization of our solutions to different crop types. Notably, these particular use cases cover a representative set in which the developed solutions will be applied, as they involve different motion patterns and different non-ergonomic positions, when performed by a human, namely that of kneeling (possibly the case in tomato picking) and that of lifting the elbow above the shoulder level (possibly the case in prickly pears picking). The selected use cases will sufficiently demonstrate the impact of the CARPOS solutions to this challenging task, once undertaken by our envisioned AI-powered robot. All scenarios assume an initial posture for the plant, i.e. that the robot is already in proximity of the plant. The scenarios involve the teaching of the activity by the human, as stated in the task description of the project. More specifically, in both use cases the following functionalities will be tested: - The ability of the robot to observe the human motion and request (if needed) iterations for gathering more information - The ability of the robot to kinesthetically correct the motion, when interrupted by the human when performing the task. CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS - The ability of the robot to reproduce the motion captured by both visual and kinesthetic means. - The human-machine interface - The ability of the system to generalize the task in various orientations and positions of the fruits. For the initial experimentation and testing, mockup setups with plastic fruits will be utilized, such as the one depicted in Fig. 2a. However, for more realistic experimentation, a realplant setup is under construction, depicted in Fig. 2b, which will be used extensively after its setup. In order to achieve continuous supply of tomato plants for the robotic experiments, 30 tomato seedling (var. ELPIDA) were transplanted in the greenhouse facilities of the Hellenic Mediterranean University. Tomato plants were cultivated in an open loop hydroponic system and at the stage of 2 fully mature tomato clusters, were transferred in a real-plant set up. The real-plant setup, in its final form, involves two fully developed tomato plants, along with the closed loop hydroponic system and a plant support system. The hydroponic system comprises of a single gutter, a nutrient solution tank (25 L), a pump (capacity of 3000 L/H) and a drip irrigation system with an individual emitter per plant at a flow rate of 4 L h–1. a) Mockup setup b) real plant setup Figure 2 – Experimental setups of the project CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS The two selected scenarios are listed below: Scenario 1. Harvesting (fruit picking) of tomato (Elpida variety) The tomato fruit-bearing system is defined as shown in Fig. 3 and involves several parts biologically distinguished between main stem and fruit like peduncle, pedicel, abscission zone, and calyx. The peduncle is a branch from the main stem, and it holds single or multiple pedicels where a branch holds an individual fruit. The pedicel can be divided anatomically into the distal and proximal pedicels based on the abscission zone, and the ripe fruit is detached physiologically by the abscission process in this zone. The abscission zone is important for harvesting as it comprises the fruit detachment point while retaining both the calyx and distal pedicel. The farmer places their hand at the bottom part of the fruit with their thump against the abscission zone, and moves the wrist against it, effectively using the fruit-calyx-distal pedicel apparatus as leverage for the zone to detach from the rest of the fruit cluster. Figure 3: Harvesting motion on Tomato (Elpida variety) Scenario 2. Prickly pear picking The harvest process of prickly pears, shown in Fig. 4, is considered one of the most difficult tasks during the cultivation process of this species due to the morphology and the presence of spikes both in shoots and fruits of the plant. It is preferable to harvest during the morning hours or in the afternoon, when the fruits appear some looseness and easily detach. Fruit harvesting is carefully performed by the farmer, using thick gloves or tools such as tweezers to avoid minor injuries during the process. The motion, regardless the use of a tool or not is always such the fruit rotates around the axis attached to the leaf, then when the fruit is lose it is either snapped with a downward motion of the wrist or if a knife is used, the remaining tissue is cut off. Additionally, fruit storage is delicately performed in spongy boxes to preserve the ideal postharvest characteristics of the fruit and increase shelf life of the product. CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS Figure 4: Prickly pear harvesting motion (yellow) and detachment point from the leaf (red). A plastic surface is used for the protection of the hand of the human. 4. SYSTEM REQUIREMENTS AND KEY PERFORMANCE INDICATORS The system requirements are defined based on the Objectives of the project which are the following (as taken from the description of action): O1: To develop novel Learning by Demonstration (LbD) methods that combine visual data (from the camera) and kinesthetic data (position and forces) for the incremental learning of fruit picking skills. O2: To develop new compliant control schemes that take into account the complete kinematic chain, i.e. the gripper, the robotic manipulator and the robotic platform will be considered as a single kinematic chain. The purpose of this compliant control scheme will be twofold: on the one hand it will enable the kinesthetic teaching of the picking skills (intentional pHRI) and on the other hand it will guarantee the safety of the surrounding humans (human-teacher, or workers in general) in case of possible unintentional collision. O3: To utilize and apply currently available computer vision solution to the problem of identifying: a) the fruit’s geometry and pose, b) the configuration of the human hands and c) the contact points between the human fingers and the fruit. O4: To develop control schemes for the active perception (tracking) of the human hand, avoiding possible occlusions, in order to optimize the robot’s observations of the human’s motion pattern during the visual demonstration performed by the farmer. O5: To combine a mobile robotic platform, a robotic manipulator, a robotic gripper, an in-hand RGB-D camera (attached to the manipulator’s end-effector), static RGB-D cameras (attached to the platform), force/torque sensors and 3D-printed robotic fingers, appropriately designed for fruit picking operations, as conceptually shown in Figure 3a, for the implementation and testing of the developed methodologies. CARPOS Coachable Robot for Fruit Picking Operations D1 - Requirements, use-cases and KPIs of CARPOS Version: 1.0 CARPOS To test and evaluate the fulfilment of the defined System Requirements (SRs), several Related Measures (RM) were defined for each system requirement. The SRs and RMs related to Objective 5 are defined in order to evaluate the whole system performance and assess the impact of the proposed solution in real-life agricultural applications. After the definition of both SRs and RMs, their importance was quantified by the agricultural scientists of the project, based on the needs and specifications of both farmers and agricultural experts, as well as by the technical/research scientists of the project responsible for developing the different modules of the system. The SRs and RMs are given in Table 1 and Table 2 respectively, alongside with the quantified importance, provided by both the agricultural and the technology-development experts. In both Table 1 and 2, the correspondence between the SRs and RMs is also provided. Table 1 – CARPOS System Requirements Rel Obj SR No Description of the System Requirement Related Metrics SR Importance (experts) Agric. T.Dev. O1 SR1.1 The robotic system should be able to learn the fruit picking skill from observing the human once RMs 1.1-1.3 Very low Low SR1.2 The robotic system should be able to learn the fruit picking skill from observing the human executing the task multiple times, to learn it more precisely RMs 1.1-1.2, 1.4-1.5 Very high High SR1.3 The robotic system should be able to learn the fruit picking skill from physical guidance by the human RMs 1.6-1.8 Low Moderate SR1.3 The robotic system should be able to correct the fruit picking skill learned, through physical guidance by the human RMs 1.6-1.9 High High SR1.4 The robotic system should be able to combine learning through visual observations and learning through physical guidance by the human RMs 1.10 Low High SR1.5 The system should include an easy and intuitive Human-machine interface RMs 1.11, 1.12 High High O2 SR2.1 The robot should be safe for working close to humans RMs 2.1 – 2.4 Very high Very high SR2.2 The robot should be compliant when the human exerts forces to it RMs 2.1 – 2.4, 2.6 Very high Very high SR2.3 During the execution of the fruit picking task the robot should follow an exact motion pattern RM 2.5 Moderate Low SR2.4 The robot should be compliant, when physically guided by the human for motion corrections. RMs 2.3 – 2.4, 2.6 High High