Generative AI for automated task modelling and task allocation in human robot collaborative applications

Generative AI for automated task modelling and task allocation in human robot collaborative applications Nikos Dimitropoulos, Michalis Kaipis, Stavros Giartzas, George Michalos (2)* Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and Aeronautics, University of Patras, Patras, 26504, Greece ARTICLE INFO Article history: Available online 27 April 2025 ABSTRACT Task modelling and assignments generation is a complex and time-consuming activity despite the availability of modern CAx and planning tools. This paper proposes an AI based framework using Large Multi-Modal Models and a Digital Twin to automatically create task models, sequences and assignment plans through the processing of video streams involving visual and audio cues on the recorded resources, tools, and tasks. The same LMMs perform the task-to-resource allocation considering metrics such as human factors and resource workload. A case study on the assembly of white goods showcases reduction in manual planning, enhanced resources utilization and improved human-robot collaborative applications. © 2025 The Author(s). Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/) Keywords: Production planning Human robot collaboration Large multi-modal models 1. Introduction The increase in industrial adoption of Human-Robot Collaboration (HRC) has brought new challenges related to task allocation and flow optimization for the newly created production systems. Empirical assignment methods and algorithms based on predefined rules are the most common techniques which are used to address such issues but are quite rigid in application and have to be manually applied using complex task modelling methods and planning tools [1,2]. However, the emergence of Artificial Intelligence (AI) as exemplified by large language models (LLMs) and large multi-modal models (LMMs), has vastly improved the performance of computational systems and subsequently their application potential in cases where human factors (behaviour, language, expression etc.) and real-world scenes must be processed and used in a meaningful way [3,4]. Recent studies have suggested the deployment of AI-based frameworks that combine task allocation with perception and reasoning [5]. Vision-based AI models have demonstrated capabilities in detecting and recognizing objects, tools, and humans within structured environments, supporting perception tasks in HRC [6]. Multi-agent reinforcement learning frameworks for dynamic task planning in reconfigurable production systems have been investigated [7], combined with Digital Twins (DTs) for optimized HRC [8,9]. LLMs have been enhanced beyond prompt engineering through methods such as Retrieval-Augmented Generation (RAG) and fine-tuning. Such advancements have expanded their applicability for interpreting complex multimodal inputs and generating taskspecific instructions [10], while proper prompt engineering plays a crucial role for receiving tailormade responses [11]. In addition, recent developments such as function calling enable LLMs to interface with external tools or APIs, including optimization engines and digital twins [12]. LMMs are innovative deep learning frameworks that focus on text, vision, and a range of different types of information, integrating them into a single model. These models can process videos, extract structured information from them and proceed to reason upon it in order to provide logical and understandable output and recommendations. Except for their use in task allocation systems, video processing models have been used in other contexts such as enhanced robot cognition [13]. Moreover, applications in quality control and defect detection exist [14]. This work proposes a consolidated approach that merges the use of LMMs and DTs to solve the task modelling and allocation problem in HRC applications. In comparison to prior work, the novelty of our framework lies on the adoption of AI technologies that enable the use of multi-modal input such as user narrated videos to create a representation of the desired production process and use it to allocate tasks to humans and robots, with no programming skills on the user side. Its main aim is to address the complexity of deploying and operating modern manufacturing environments (including flexible resources and humans), ensuring balanced workloads, optimal resource utilization, and enhanced safety. Section 2 presents the architecture of the proposed approach, while the implementation a system prototype is described in Section 3. Section 4 provides the evaluation on a case study from the white goods sector and Section 5 provides a discussion on the findings while outlining areas of future work. 2. Approach A high-level overview of the approach is shown in Fig. 1. The workflow starts with the manual recording of video footage of an existing assembly environment, as well as visual (pointing to objects/ resources, picking components) and verbal (narration, description etc.) cues provided by the recording engineer. The video is then fed to an LMM which processes both the audio (using Natural Language * Corresponding author. E-mail address: [email protected] (G. Michalos). https://doi.org/10.1016/j.cirp.2025.04.081 0007-8506/© 2025 The Author(s). Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/) CIRP Annals - Manufacturing Technology 74 (2025) 711 Contents lists available at ScienceDirect CIRP Annals - Manufacturing Technology journal homepage: https://www.editorialmanager.com/CIRP/default.aspx Processing) and video frames to a) detect available resources (Human workers, collaborative robots, tools) and b) generate a list of tasks (and their characteristics) that need to be executed. This list is then fed to a pretrained LMM (Gemini Flash 1.5 [15]) which is configured to consider human safety and resource productivity metrics for performing workload distribution. One of the key points that the LMM is focusing on while allocating tasks is the enablement of parallel work between human and robotic agents for the sake of fluid workflow execution. To do so, the framework is generating executable actions respecting predefined custom templates of JSON objects for each action. The dispatching of the tasks to humans and resources is undertaken by an executor server, that is responsible to communicate with all the available agents of the production line. As an extension, the DT component enables the simulation and evaluation of task allocation scenarios in a safe, virtual environment providing at the same time real time status updates of the actual system. Through this capability, real-time adaptability can be achieved in terms of planning based on the current status of the workstation and the task progress. Time generated data and simulation results are stored for future planning loops to increase task allocation efficiency. This ensures that the system can handle complex industrial setups with multiple workers and robots while maintaining balanced workloads and maximum throughput. Finally, the engineers can interact with the system through web interface and CLI tools to supervise and verify the quality of the generated results and rectify possible errors, such as task order logic issues and potential inefficiencies in task resource assignments. This corresponds to a very small percentage of the effort that would otherwise be required for a fully manual task generation and allocation which could involve hundreds or different actions for a particular case. To ensure scalability, the components of the system are designed to operate efficiently across diverse industrial environments, ranging from small teams to large-scale operations. The execution main server uses conflict resolution strategies, such as semaphores, to prevent resource contention. Furthermore, it includes mechanisms for evaluating system effectiveness through metrics presented in Section 3.6. 3. Implementation The architecture of Fig. 1 has been translated into a series of software components which are presented in this section. 3.1. Video analysis component (VAC) VAC is powered by Gemini 1.5 Flash, a Multi-modal model used for the video understanding process, through the ‘google.generativeai’Python library. To achieve good quality in the analysis of the video the following principles apply for the recoding: a) it should capture the complete workspace clearly meaning that items of interest such as resources (machines, robots), objects (parts, tools etc.) and any related information on them (name, id, capabilities) are clearly visualized b) it should either contain a visual demonstration of the desired process and/or an audio description of how the process should be carried out (sequence, tools used in each step, process parameters, etc.). Typically, shorter videos in length were observed to provide better results. The analysis process consists of the following steps: a) After uploading the video, the LMM is requested to describe the video. This allows for a first information extraction which can also cover the case of not having audio narration in the video and the LMM provides it through its description generation ability. This initial information greatly improves the accuracy of subsequent, more complex prompts. b) In the same session the LMM is prompted to use the video and its own generated description to identify, the resources and tools that are depicted and report then in a JSON-formatted list. The same process takes place for the parts to be assembled and their relative positions. c) Finally, the engineer can add extra prompts to improve or add detections that may have been misinterpreted or omitted in the above lists. An indicative frame of the teaching phase as well as examples of the identified objects and resources can be found in Fig. 2. 3.2. Task planner component (TPC) In a similar approach to the VAC component, the LMM is further provided with a standard output structure for representing tasks. It then prompted to produce high-level task descriptions, considering task sequencing constraints, task parallelization, weight of the items and resources capabilities. It specifically instructed to prioritize human tasks based on ergonomics and prefers HRC in the cases where task parallelization can yield lower cycle times. The main principles that are used to build the prompts involve: a) consider process ergonomics (postures, part weight etc.) for tasks that can be assigned to humans b) create a recommendation for tasks that can run in parallel, c) create links between tasks and their parents to avoid recursive dependencies and d) retrieve information by the DTC (Section 3.5) on the current resource workload in order to suggest an efficient allocation of tasks to each resource. The session is interactive in the sense that the LMM informs the engineer on the logic behind each decision and receives feedback for adjusting it according to the user’swish. 3.3. Task execution server component (TESC) TESC serves as the main coordinator, overseeing the order of tasks and the assignment to the agents. It processes the JSON output from TPC, generating task objects that include attributes such as status, description, and assigned resource ID. It offers a stateful API for agents to interact with, built with Python FastAPI, allowing them to: a) allocate a task, b) indicate task completion (join) and c) retrieve a list of pending tasks. It uses semaphores in avoiding race conditions since each resource can have a single activity at a time. The defined semaphores for our multi-agent system prevent two resources from being allocated to the same task concurrently. This ensures that there Fig. 1. Flowchart of the proposed framework. Fig. 2. Video frame with the identified objects and voice over prompts by the recording engineers. 8N. Dimitropoulos et al. / CIRP Annals - Manufacturing Technology 74 (2025) 711 are no conflicts among the independent agents. As the execution component can handle concurrent requests, it utilizes a semaphore flag whenever an agent requests a task allocation; if the semaphore flag is “on”, then the response is withheld until the release, preventing data inconsistencies. The web method ‘Web Events Source’ pushes real-time updates on any changes in task status. At the beginning of the execution, all tasks are marked as having a ’created’ state, while the ones without dependencies are switched to ’pending’state. Agents will search for tasks in ’pending’state and try to take ownership of those for execution. Once an assignment of the task is successfully performed on an agent, it enters the ’running’or ‘done’state. 3.4. Agent execution component (AEC) When a task is allocated to an agent, the agent’sLMMprocessesthe task description as well as resource capabilities in order to compose low level execution actions which are used to drive the resource. The later are defined exploiting user prompts in form of ‘templates’.Eachtemplate consists of 3 parts: a) skill definition (pre-execution), b) properties input (during execution) and c) the result object (post-execution). Each action triggers a specific handler that validates parameters in simulation environment (DTC). Once all actions are generated by the LMM, they are executed sequentially, with results stored for future use. Upon completing the final action, the agent notifies TESC of task completion and retrieves the next task. 3.5. Digital twin component (DTC) and human-in-the-loop interface (HILI) Two components were developed to facilitate the testing, finetuning and integration of the components above in real-life scenarios. The DTC is used during the testing and finetuning of the model, allowing the simulation of alternative scenarios and settings while exploiting the DT capability of mirroring the system status in real time. In the execution phase the DTC serves as a representation of the real robotic cell, utilizing data from a variety of sensors located in the cell (e.g. robot pose, gripper state, operator position, operator pose etc.) and is then used to generate a collision free robot trajectory using MoveIt! Framework, following the plan defined by the AEC. The DTC is not a core development of this paper, but it was essential to validate the effectiveness of solution before commissioning. The HILI offers a high-level tool for evaluating the planning results. It is built upon a web interface to providing a real time visualization of the system state and each task state as well as the pending list of tasks. The engineer has access to the executor terminal and can interact with the system via prompts to validate and/or modify the logical order of the generated tasks and their dependencies. 3.6. Data storage and analysis To assess the validity and effectiveness of the model, three main KPI categories are defined: a) parts and resources detection precision, b) task generation, c) tasks sequencing and assignment efficiency. The specific metrics for the evaluation of the process are: Presources: Precision of identified resources. This KPI measures the number of the resources (e.g. tools, machinery, robots, humans) identified through the video analysis and human prompting, when applicable. Ptools: Precision of identified tooling. This KPI assesses the tools being identified by the VAC compared to the known ground truth. Ptasks: Precision of identified tasks. This KPI assesses the correctness and completeness of the tasks being generated by the VAC, compared to the known ground truth. Etasks: High-level tasks to resources allocation efficiency. This KPI assesses the applicability of the assignments of tasks to the relevant resources done by the TPC based on their capabilities. Eactions: Low-level actions order logic. This KPI assesses the effectiveness of AEC to generate executable low-level actions/instructions. Texec: Total execution time based on the assignments generated. This calculates the total time needed to execute the whole assembly, either in virtual or physical environment. Tprep: Total preparation time. This is related to the time needed to generate an executable plan for a new scenario. To quantify system performance and identify areas for improvement, losses are calculated for each metric. Losses are derived by comparing the model output to known ground truth values. In particular, Eq. (1) calculates the number of detected resources in comparison with the actual ones shown in the video. Similarly, Eqs. (2) to (7) calculate the respective loss function for the number of tools, tasks, actions and durations as described above: Lresources ¼aresources ¢1Presources ðÞ ð1Þ Ltool ¼atool ¢1Ptool ðÞ ð2Þ Ltasks ¼atasks ¢1Ptasks ðÞ ð3Þ Ltasks alloc ¼atasks alloc ¢1Etasks ðÞ ð4Þ Lactions ¼aactions ¢1Eactions ðÞ ð5Þ Ltotal_time ¼atotal_time 1eTexec atotal_time_scale  ð6Þ Lprep ¼aprep 1eTprep aprep  ð7Þ The a i parameters are user defined weights representing the importance of each metric to the total solutions loss. The bigger the value the greater the effect of the identifier to the total system loss. The system’s root mean square error (RMSE) across all metrics provides a consolidated performance measure: RMSE ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 7X 7 i¼1 L2 i v u u tð8Þ 4. Case study 4.1. Reference scenario The case study scenario involves the assembly of induction kitchen hobs. A human operator and a light payload collaborative robot (UR-10) work on the same workpiece simultaneously (Fig. 3). The operator is in charge of loading the coils into the feeding system and setting the cable terminals at a defined location. The robot, equipped with a dedicated end-effector and a multi-sensor system can pick/place hobs, perform screwing operations and install connectors, checking their correct installation in real-time. This takes place at a specific workbench while an adjacent one facilitates tools and raw materials storage. Fig. 3. Reference scenario: a) real-world, b) DT based simulation. N. Dimitropoulos et al. / CIRP Annals - Manufacturing Technology 74 (2025) 711 9 4.2. Evaluation metrics and assessment The use case presented in the previous section was selected due to the high number of parts included, the complexity of the processes and potential for parallel task execution. Both simulation-based testing using the DTC as well as real-world validation on physical setup was performed. Two (2) scenarios were tested, differentiating on the number of components to be assembled (from 8 to 12), available tools and resources. The process was repeated 15 times for each scenario. To determine the impact of input data to the KPIs defined in Section 3.6, three approaches were followed: a) video; the input to VAC is only a video showing in detail a manual execution of the assembly, b) video + voice over; on top of the video, the engineer explains in detail the steps followed, c) video + voice over + prompting; on top of the video and the voice over, the engineer enhances the dataset with dedicated prompts or any other hint that the TPC or AEC should take into consideration. The correlation between data input, detection performance and planning output for the reference case study is shown in Fig. 4.Thegreen font indicates the ideal identification and/or assignment of tools/parts/ resources/tasks while the black font denotes partial. This example includes 8 tasks: 5 operator tasks and 3 robot tasks. Additionally, KPIs 7 and 8 corresponding to the total execution and preparation time have been compared against a conventional planning approach using heuristics [16]. The mean values of the KPIs across all 15 iterations and 2 scenarios, expressed in percentage of the ground truth, are shown in Table 1. Analyzing the results and as can be expected, by providing more details about what the video streams are showing, the model is able to detect more efficiently the relevant resources, tools and tasks as indicated by the RMSE value (the lower the better). Additional prompting has positive impact in task definition as well as low-level actions definition. The generated number of tokens by the LMM varied between 4 and 8, due to the interesting fact that some tasks could be logically grouped into one. Also, many steps of the use cases were assigned to therobotbytheTPCduetothe3%algorithmicdifferencenoticedinthe time and setup or use of any resource in relation to the human method, as task modelling and manual data input are process eliminated. 5. Conclusions future work This paper proposes a novel approach using LMMs assisted by DT technology to perform task modelling and allocation in HRC scenarios. The proposed approach is highly automated as it only requires video and audio streams which are elaborately processed by an LMM that is capable to extract all product/process/ resource data needed for the task planning and execution of the demonstrated operation. The aforementioned approach was validated in a use case derived from the white goods industry, showing the ability of an LMM to solve advanced industrial problems. The quality of the produced task sequences was considered reasonable, sensible and well aligned with expert’s plans, indicating its potential towards streamlining task allocation in shared workplaces. The proposed approach does not aim to replace classical optimization methods or CAPP systems, which remain a reliable solution for structured manufacturing environments, as it targets flexible/ dynamic settings where full digital twins, geometric models, or detailed process plans are often unavailable or impractical to maintain. The generated output, despite not being guaranteed-optimal, could help towards accelerating early-stage planning, enabling hybrid (humanrobot) task allocation, and facilitating operator involvement, while existing CAPP tools can help to refine it. Future work will seek to improve the system with regards to operating in more complex and greater variety of industrial settings. Such improvements shall include better task allocation through reinforcement learning as well as further autonomous decision making by the agents. Scalability to larger and more compound setups as well as integration of more sophisticated human-machine interfaces for improved operation and supervision will be investigated. Integration with classical solvers will aim to optimize the GenAI output for cases requiring strict adherence to cycle time limits or robot path constraints. Uncertainty handling mechanisms will limit the generation of incorrect or suboptimal plans. Strategies for detecting human execution errors and leveraging human recovery skills will be investigated, with a view to integrate human error modelling and recovery pathways in the overall workflow. The use of multiple input variations, structured task libraries, and simulation-based evaluation to support alternative plans generation will be evaluated. Function calling capabilities may also be leveraged to connect to external knowledge sources or simulation systems for validating such alternatives before implementation. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. CRediT authorship contribution statement Nikos Dimitropoulos: Writing original draft, Validation, Supervision, Project administration, Methodology, Investigation, Conceptualization. Michalis Kaipis: Writing original draft, Visualization, Validation, Software, Investigation, Formal analysis, Data curation, Conceptualization. Stavros Giartzas: Writing original draft, Visualization, Validation, Software, Investigation, Data curation. George Michalos: Writing review & editing, Methodology, Funding acquisition, Formal analysis, Conceptualization. Acknowledgments This research has been supported by the EU project “CONVERGING Social industrial collaborative environments integrating AI, Big Data and Robotics for smart manufacturing”. This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101058521. Supplementary materials Supplementary material associated with this article can be found in the online version at doi:10.1016/j.cirp.2025.04.081. Fig. 4. LMM input vs detection performance and planning output. Table 1 Metrics table of validation KPIs. Input video Input video + voice over Input video + voice over + prompting Heuristics search task planning Presources 80% 100% 100% N/A Ptools 76% 100% 100% N/A Ptasks 65% 88% 97% N/A Etasks 72% 100% 100% N/A Eactions 62% 86% 98% N/A Texec N/A N/A »276s »280s Tprep »18min »22min »26min »300min RMSE 0.6154 0.5595 0.4307 N/A All results represent mean values across 15 iterations cycles and 2 scenarios using DTC & real-world. 10 N. Dimitropoulos et al. / CIRP Annals - Manufacturing Technology 74 (2025) 711 References [1] Chryssolouris G (2006) Manufacturing systems: theory and practice, 2ndEdition Springer-VerlagNew York. [2] Makris S, Michalos G, Dimitropoulos N, Krueger J, Haninger K (2024) Seamless HumanRobot collaboration in industrial applications. CIRP Novel Topics in Production Engineering 1:39–73. [3] Sai S, Sai R, Chamola V (2024) Generative AI for industry 5.0: analyzing the impact of ChatGPT, DALLE, and other models. IEEE Open Journal of the Communications Society 6. 11. [4] Gkournelos C, Konstantinou C, Makris S (2024) An LLM-based approach for enabling seamless Human-Robot collaboration in assembly. CIRP Annals 73/1: 9–12. [5] Wu H, Ghadami A, Epureanu BI (2024) Dynamic task planning for autonomous reconfigurable manufacturing systems by knowledge-based multi-agent reinforcement learning. CIRP Annals 73/1:353–356. [6] Ansari F, Kohl L, Sihn W (2023) A competence-based planning methodology for optimizing human resource allocation in industrial maintenance. CIRP Annals 72/ 1:389–392. [7] Zheng P, Li C, Fan J, Wang L (2024) A vision-language-guided and deep reinforcement learning-enabled approach for unstructured human-robot collaborative manufacturing task fulfilment. CIRP Annals 73/1:341–344. [8] Liu S, Wang XV, Wang L (2022) Digital twin-enabled advance execution for human-robot collaborative assembly. CIRP Annals 71/1:25–28. [9] Xu Q, Zhou G, Zhang C, Chang F, Cao Y, Zhao D (2024) Generative AI and DT integrated intelligent process planning: a conceptual framework. International Journal of Advanced Manufacturing Technology 133/5:1–25. [10] Huang D, Yan C, Li Q, Peng X (2024) From large language models to large multimodal models: a literature review. Applied Sciences 14/12:5068. [11] Tian Y, Liu A, Dai Y, Nagato K, Nakao M (2024) Systematic synthesis of design prompts for large language models in conceptual design. CIRP Annals 73/1:85–88. [12] Kim S, Moon S, Tabrizi R, Lee N, Mahoney MW, Keutzer K, Gholami A (2024) An LLM compiler for parallel function calling. Forty-first International Conference on Machine Learning,. [13] Aristeidou C, Dimitropoulos N, Michalos G (2024) Generative AI and neural networks towards advanced robot cognition. CIRP Annals 73/1:21–24. [14] Tulbure AA, Danciu DP, Dulf EH, Tulbure AA (2024) A study on multi-modal LLM reasoning for defect detection. IEEE International Symposium for Design and Technology in Electronic Packaging (SIITME), 153–157. [15] Google Gemini Flash, https://deepmind.google/technologies/gemini/flash/, last accessed on 12/1/24. [16] Kousi N, Dimosthenopoulos D, Matthaiakis AS, Michalos G, Makris S (2019) AI based combined scheduling and motion planning in flexible robotic assembly lines. Procedia CIRP 86:74–79. N. Dimitropoulos et al. / CIRP Annals - Manufacturing Technology 74 (2025) 711 11