ETP4HPC SRA 6 White Paper - Scientific Applications & Workflows

WHITE PAPER Scientific Applications & Workflows November 2025 An SRA 6 White Paper Scientific Applications and Workflows 2 Co-leaders Sagar Dolas (SURF), [email protected] Marco Verdicchio (SURF), [email protected] Table of contents Executive Summary ..................................... 3 Introduction ............................................... 4 State of the Art - Scientific Applications and Workflows Ecosystem ................................. 6 Simulation, Modelling and Design............... 6 Instrument-Driven Discovery ...................... 6 Digital Twins ................................................ 7 Model training, HyperParameter Optimisation and Inference (AI/ML) ........... 8 Combining Physical and Digital Environments .................................................................... 8 Cross-Cutting Challenges ........................... 10 Support for the Scientific Process ............. 10 Workflow Specification ............................. 11 Workflow Orchestration ............................ 11 Data Aware Workflow Design ................... 12 Security, Identity and Compliance ............. 12 Knowledge and Skills ................................. 13 Key R&I Recommendations ....................... 14 Ecosystem Harmonisation ......................... 14 Actions .................................................. 14 Challenges addressed............................ 15 Workflow-driven Infrastructure Design ..... 15 Actions .................................................. 15 Challenges addressed............................ 16 Sustainable Application Workflow Development ............................................ 16 Actions .................................................. 16 Challenges addressed............................ 16 Conclusion ................................................ 17 Contributing authors ................................. 18 Appendix A - Elements of Workflow Design 19 Compute ................................................... 20 Time .......................................................... 21 Data........................................................... 22 Automation ............................................... 23 Appendix B - Table of recommended actions ................................................................ 24 Scientific Applications and Workflows 3 Executive Summary This White Paper is part of European Technology Platform’s Strategic Research Agenda 6. Scientific discovery is increasingly driven by workflows: complex, multistep processes that integrate data, computation, and collaboration across diverse infrastructures. As science becomes more data-intensive, AI-driven, and distributed across infrastructure, institutions and borders, workflows are no longer peripheral; they are central to research productivity and innovation. Workflow-readiness of Scientific Applications has the potential to bridge the gap between performance and productivity, amplifying the impact of investments in advanced infrastructure. This paper provides a (1) high-level overview analysis of the Scientific Applications and Workflows Ecosystem, (2) identifies cross-cutting challenges, and (3) translates them into actionoriented recommendations for policy bodies, funding agencies, and infrastructure providers. Scientific applications and their corresponding workflows exhibit diverse patterns of computation, data movement, and automation. Across this diversity, there are recurring challenges that cut across domains and workflow types:  Support for the Scientific Process: Ensuring reproducibility, reusability, and provenance remains difficult due to tight coupling between workflows and computational environments, as well as uneven adoption of standards.  Workflow Specification: Translating scientific ideas into reliable, executable workflows requires both domain expertise and workflow engineering skills, making specification a persistent bottleneck.  Workflow Orchestration: Managing workflows end-to-end across heterogeneous, distributed infrastructures demands robust orchestration, automation, and portability that current systems struggle to provide.  Data (Management) Aware Workflow Design: Workflow systems often do not fully support the integration and interoperability with data management processes and storage systems.  Security, Identity, and Compliance: Fragmented authentication, varying legal frameworks, and unclear accountability complicate secure, cross-institutional workflow execution.  Knowledge and Skills: Limited training and fragmented expertise hinder adoption, create duplication of effort, and slow the spread of best practices across scientific domains. The paper concludes with actionable guidance derived from the ecosystem overview and identified challenges. The proposed recommendations provide concrete suggestions for intervention, detailing specific sets of short-term and long-term actions across three key areas:  Ecosystem Harmonisation: Calls for lowering adoption barriers and converging on common standards, workflow languages, and shared APIs; coherent security and identity practices; and an easily discoverable, interoperable knowledge ecosystem.  Workflow-driven Infrastructure Design: Next-generation infrastructures should be designed with usability, portability, and sustainability of workflows as core principles, guided by capability-focused procurement strategies and supported by dedicated performance metrics.  Sustainable Application Workflow Development: A stronger focus on the financial and technical sustainability of workflows is required, with funding bodies explicitly supporting their continuous development and promoting collaborations and integration across projects and initiatives. There is an opportunity to contribute to workflow-enabled science by treating workflows as first-class research infrastructure. By acting on the recommendations outlined, policymakers, funders, and infrastructure providers can empower researchers to focus on scientific discovery rather than technical hurdles. Scientific Applications and Workflows 4 Introduction Scientific computing is continually evolving in parallel with the increasing complexity, scale, and collaborative nature of modern research needs and processes. Computational workflows are now central in many scientific domains, and understanding how they are structured, executed, and evolved is crucial.0F 1 1F 2 As current scientific challenges demand crossdisciplinary methods, the integration of heterogeneous infrastructures, and fast, reliable computational results, workflows have become central to how science is performed. Whether modelling multiscale and multiphysics systems, training AI models, or integrating instruments, research workflows are essential for orchestrating complex, multistage computational and data processes and for enabling reproducible, scalable science.2F 3 3F 4 Beyond execution and automation, workflows embody scientific processes and are crucial for capturing provenance, ensuring FAIR data practices, and supporting the reproducibility of results. They have become essential tools in modern science, enabling structured and automated scientific processes; their evolution is not only a software concern but also a key enabler of innovation across science and engineering domains. Investing in their design, 1 The Scientific Case for Computing in Europe 20182026, PRACE, ISBN 9789082169492, https://praceri.eu/prace-research-infrastructure/third-scientificcase/ 2 National Academies of Sciences, Engineering, and Medicine. 2022. Automated Research Workflows for Accelerated Discovery: Closing the Knowledge Discovery Loop. Washington, DC: The National Academies Press. https://doi.org/10.17226/26532 . 3 Ferreira da Silva, R., et al. Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows. Zenodo, 18 Oct. 2024, https://doi.org/10.5281/zenodo.13844759 4 Computational Needs for Accelerated Scientific Discovery, https://www.nwo.nl/sites/nwo/files/media-files/Computational%20needs%20for%20accelerated%20scientific%20discovery.pdf execution environments, and standardisation is therefore essential for accelerating the pace of discovery and translating research into societal impact.4F 5 Scientific application workflows have evolved from linear, monolithic batch scripts into complex and adaptive systems that orchestrate heterogeneous tasks across a large number and variety of computing environments. In the past, computational workflows were often tightly coupled to specific applications and infrastructures, with minimal interaction and limited flexibility. These early models were mainly domainspecific and typically structured as static pipelines with clearly defined inputs and outputs. As research questions become progressively interdisciplinary and data-intensive, workflows are evolving to incorporate more computational complexity, and data-driven processes, and to accommodate real-time data ingestion, advanced statistical analysis, and visualisation. The rise and adoption of machine learning and AI methods in science have further transformed workflows, which are now increasingly capable of dynamic parameter tuning, real time refinement, and informed decision-making5F 6 6F 7. This trend is particularly evident, for example, in self-driving labs and digital twin applications, 5 Etz, Brian D., David M. Rogers, Michael J. Brim, Ketan Maheshwari, Kellen Leland, Tyler J. Skluzacek, Jack Lange, et al. 2025. “Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows.” arXiv [Cs.DC]. arXiv. http://arxiv.org/abs/2506.01744 . 6 Badia, Rosa M., Laure Berti-Equille, Rafael Ferreira da Silva, and Ulf Leser. 2024. “Integrating HPC, AI, and Workflows for Scientific Data Analysis (Dagstuhl Seminar 23352).” Schloss Dagstuhl - LeibnizZentrum für Informatik. https://doi.org/10.4230/DAGREP.13.8.129 . 7 Ward, Logan, J. Gregory Pauloski, Valerie HayotSasson, Yadu Babuji, Alexander Brace, Ryan Chard, Kyle Chard, Rajeev Thakur, and Ian Foster. 2024. “Employing Artificial Intelligence to Steer Exascale Workflows with Colmena.” arXiv [Cs.DC]. arXiv. http://arxiv.org/abs/2408.14434 . Scientific Applications and Workflows 5 where simulations and data streams continuously interact to support adaptive modelling and provide predictive information and insights. At the same time, the increased use of cloud-native architectures, the edge-to-HPC continuum, and federated infrastructures has created the need to rethink computational workflows in terms of portability, elasticity, modularity, and robustness. Modern workflows now span from experimental instruments to HPC systems, incorporating data management, compute orchestration, and an abstraction to backend infrastructures.7F 8 Today, scientific workflows are not just execution techniques, but modular, reusable tools central to the scientific method itself. Their evolution reflects the broader transformation of science, requiring investment in standards, interoperability, and skills to support next-generation discovery at scale. The scientific community increasingly recognises the importance of common standards and shared specifications to enable portable, interoperable, and sustainable workflows across diverse environments. Initiatives such as the Common Workflow Language (CWL) and the Workflow Description Language (WDL) are laying the groundwork for describing, executing, and exchanging workflows beyond individual platforms and domains. These efforts help lower integration barriers and foster collaboration across scientific communities. However, despite these advances, adoption remains fragmented, with many workflow solutions still evolving in isolation and significant differences in scope, maturity, and interoperability persisting across domains.8F 9 Continued investment in 8 Organizational Chart. 2024. “Berkeley Lab Issues Request for Proposals for Next-Gen HPC System, NERSC-10.” NERSC: National Energy Research Scientific Computing Center. March 13, 2024. https://www.nersc.gov/news-andevents/news/berkeley-lab-issues-request-for-proposals-for-next-gen-hpc-system-nersc-10 open standards, coordinated adoption strategies, and cross-community collaboration will therefore be essential to move beyond isolated efforts and towards a truly integrated, scalable, and sustainable scientific computing ecosystem. This white paper aims to establish a common understanding and provide direction for the development and adoption of scientific workflows across various scientific applications. The sections below present an overview of the current and emerging scientific applications and workflows ecosystem, characterising their structural and computational requirements. This overview is followed by a set of cross-cutting challenges that affect the various scientific applications, workflows, and contexts. The paper concludes with actionable guidance derived from the ecosystem overview and identified challenges. These challenges and recommendations were also presented during the ETP4HPC Webinar “Research Applications & Workflows – Insights & Community Discussion”9F 10, where feedback from attendees was collected regarding the relevance and prioritisation of the proposed challenges and recommendations. This feedback informed the refinement and validation of the recommendations presented in this white paper. The proposed recommendations offer concrete intervention suggestions, detailing both short-term and long-term actions. These recommendations should guide policymakers, funders, and infrastructure providers in creating a more interoperable, efficient, and scientifically driven computational ecosystem that empowers researchers to focus on scientific discovery rather than technical hurdles. 9 A Terminology for Scientific Workflow Systems. Future Generation Computer Systems 174:107974 https://doi.org/10.1016/j.future.2025.107974 10 ETP4HPC Webinar – Research Applications & Workflows – Insights & Community Discussion, https://etp4hpc.eu/event/etp4hpc-webinar-research-applications-workflows-insights-community-discussion/ Scientific Applications and Workflows 6 State of the Art - Scientific Applications and Workflows Ecosystem In this section, we outline some of the main application areas where advanced computational approaches are transforming how we understand and interact with complex systems, sophisticated experimental setups and the increasing amount of data currently available to scientists. For each class of application, we analyse its composition through the core workflow design elements: Compute, Time, Data, and Automation (Appendix A: Elements of Workflow design). Simulation, Modelling and Design Simulation, Modelling and Design represent a class of applications built around the execution of complex, coupled, and scalable simulations. These are often multiphysics and multiscale, demanding high levels of parallelism, performance optimisation, and adaptive load balancing. From a workflow design perspective, they are dominated by their Compute element, specifically Highly demanding computations that require extreme scale-up or scale-out capabilities on HPC systems. Future workflow designs may also incorporate hybrid quantum–AI–HPC executions, where quantum resources, AI models, and classical HPC components are interleaved within a single run, potentially adding new layers of interoperability. The Time dimension is characterised by long-running campaigns rather than low-latency Time-criticality. The primary Data pattern is Data Staging, where large input datasets are prepared before a simulation and massive output datasets are archived for post-hoc analysis. Automation is typically Semi-automated, with researchers initiating jobs, but can become Fully-automated in, for 11 https://max-centre.eu/ 12 https://bioexcel.eu/ 13 https://cheese2.eu/ example, large ensemble simulations or parameter sweeps. Several leading European initiatives exemplify this approach. Projects such as MaX10F 11 and BioExcel11 F 12 focus on atomistic and molecular simulations, driving the optimisation of widely-used codes for quantum mechanics and molecular dynamics to support materials design and biomolecular research. Likewise, initiatives like ChEESE12F 13 and ESiWACE13F 14 develop large-scale simulation frameworks for natural hazard prediction, climate and weather modelling, and ocean system analysis. Plasma-PEPSC14F 15 focuses on predictive modelling through coupled, highresolution plasma codes. CAELESTIS1 5F 16 proposes a multistage and multiphysics workflow for probabilistic design and predictive manufacturing of aircraft structures. Instrument-Driven Discovery This class of applications integrates data-generating instruments, such as particle detectors, telescopes, microscopes, and environmental sensors, with a continuum of computing and data infrastructure. The aim is to enable realtime data acquisition, processing, and analysis across diverse and distributed computational environments, thereby supporting scientific discovery at scale. A tight coupling between Data and Time defines these workflows. The dominant Data pattern is high-volume Data Streaming directly from an instrument, which demands immediate processing. This imposes significant Timecriticality, requiring low-latency analysis to filter, classify, or make decisions about the incoming data. The Compute element is federated and heterogeneous, combining Flexible and 14 https://www.esiwace.eu/ 15 https://plasma-pepsc.eu/ 16 https://caelestis-project.eu/ Scientific Applications and Workflows 7 Composable processing at the edge for realtime filtering with Highly demanding computations on grid or HPC resources for offline analysis. Automation is crucial, with Event triggered data acquisition and Fully-automated pipelines to manage the data flow from capture to archival. Representative examples include major initiatives in high-energy physics and astrophysics. At the Large Hadron Collider (LHC), projects like ATLAS16F 17 and CMS17F 18 exemplify this set of applications through their use of advanced data filtering, simulation, and statistical inference over petabytes of collision data. Similarly, in radio astronomy, the LOFAR18F 19 and SKA19 F 20 initiatives produce massive volumes of observational data that require low-latency pipelines, federated storage, and distributed compute resources. Digital Twins The focus of digital twins lies on creating and leveraging high-fidelity, dynamic digital replicas of real-world physical systems. They are underpinned by complex, coupled, and scalable simulations, often multiphysics and multiscale in nature, which demand high levels of computational parallelism. Digital twins also rely on the integration and processing of vast, heterogeneous datasets, which can originate from a wide array of sources. Artificial intelligence and machine learning can further enhance these systems by enabling intelligent data assimilation, pattern discovery, and adaptive modelling. Workflows for digital twins are uniquely complex, synthesising all four design elements. The Compute component is hybrid, combining highly demanding computations from the underlying physical models with AI-enabled simulations to achieve predictive accuracy and 17 https://atlas.cern/ 18 https://cms.cern/ 19 https://www.astron.nl/telescopes/lofar/ 20 https://www.skao.int/ 21 Vázquez-Novoa, Fernando, Javier Conejero, and Rosa M. Badia. "Model generation architecture for digital twins and deployment system." The speed. The Time element is often paramount, with many applications having high Time-criticality and, in cases like disaster response, extreme Urgency. The Data model is equally multifaceted, combining Data Staging for initialisation with continuous Data Streaming from sensors to keep the twin synchronised with reality. These systems strive for fully automated operation, often using event-triggered mechanisms to assimilate new data and update the model state without human intervention. Methodologies to develop digital twins based on these hybrid workflows have been proposed20F 21. The Destination Earth (DestinE)21F 22 initiative aims to develop high-precision digital twins of the Earth to address challenges such as climate change, natural disasters, and enhancing environmental resilience. Projects like BioDT22F 23 and DTO-BioFlow23F 24 are advancing digital representations of biodiversity systems and biological networks by integrating genomic, ecological, and climate datasets. DT-GEO2 4F 25 focuses on developing a digital twin framework for geophysical extremes, enabling improved forecasting and risk assessment of events such as earthquakes, volcanic eruptions, and tsunamis through data assimilation and high-performance simulation. International Journal of High Performance Computing Applications (2024): 10943420251379123. 22 https://destination-earth.eu/ 23 https://biodt.eu/ 24 https://dto-bioflow.eu/ 25 https://dtgeo.eu/ Scientific Applications and Workflows 8 Model training, HyperParameter Optimisation and Inference (AI/ML) This class of applications centres on the computational processes involved in training, finetuning, and deploying artificial intelligence and machine learning models. These processes often require access to large-scale datasets, significant computational resources (especially for deep learning and foundation models), and efficient data pipelines. Model training is typically compute-intensive, involving iterative optimisation across billions of parameters. At the same time, inference must often meet strict latency and scalability requirements to be deployed in real-time or large-scale environments. These applications span a wide range of domains, from natural language processing and computer vision to scientific computing and digital biology, enabling new forms of knowledge extraction, automation, and decision support. The design of AI/ML workflows are distinctly split between training and inference phases. Training is characterized by Highly demanding computations, often requiring extreme-scale GPU clusters for weeks or months. Its primary Data pattern is the Data Staging of massive, static datasets. Inference, by contrast, may require Flexible and Composable processing to be deployed at scale and often has high Timecriticality, needing to deliver results with low latency. Automation is key across the lifecycle, from Fully-automated hyperparameter tuning to MLOps pipelines that are Event triggered by the availability of new data to retrain and deploy models. Several European initiatives are advancing large-scale AI and ML workflows to support scientific discovery and innovation. Projects such as GPT-NL25F 26 and NorLM26F 27 focus on developing large language models adapted to Dutch and Nordic languages, supporting research, public sector, and industry applications. OpenEuroLLM27F 28 aims to build open, 26 https://gpt-nl.nl/ 27 https://www.nora.ai/nora-projects/ai-fjord/norallm/ transparent, and sovereign European foundation models as shared research infrastructure. Complementary efforts, such as domain-specific initiatives in medicine, environmental science, and materials research, among others, are integrating machine learning into scientific workflows, demonstrating the growing importance of scalable AI training, optimisation, and inference capabilities across the European research landscape. Combining Physical and Digital Environments This is an emerging class of applications where automated laboratory platforms and highthroughput experimentation, sometimes coupled with AI-driven decision-making, autonomously generate, test, and refine scientific hypotheses. These workflows must automate analysis and, when appropriate, directly control instruments or experimental parameters, while remaining flexible enough to incorporate human-in-the-loop validation where necessary. There is often tight integration among laboratory automation, computational simulation, and AI tools, leveraging advanced robotics, machine learning for predictive modelling, and orchestration infrastructure. The overarching goal is to accelerate scientific discovery through a data-centric, iterative learning process, based on continuous data acquisition, data-driven 28 https://openeurollm.eu/ Scientific Applications and Workflows 9 modelling, and feedback for adaptive experimentation. These automated labs operate at the frontier of autonomous science, reducing human intervention while enhancing reproducibility, scalability, and exploratory efficiency. These workflows represent the pinnacle of integration across the design elements. Automation is their defining feature, aiming for a Fullyautomated, closed-loop system where human intervention is minimal. The workflow is inherently Event triggered, as the analysis of one experiment immediately informs the next. This creates a high degree of Time-criticality within the loop to ensure rapid progress. The Compute element is hybrid, using AI-enabled simulations for predictive modelling and Flexible processing for robotic control and real-time analysis. Data flows continuously and in low latency from instruments to the AI decision engine and back to the robotic hardware, representing a tight integration of Data Streaming and control. BigChemistryNL28F 29, a Dutch national program, aims to digitise chemistry research through the integration of high-throughput experimentation, artificial intelligence, and robotic automation, enabling closed-loop discovery cycles in chemistry and materials science. Other European efforts are similarly pushing boundaries; for example, ADAM (Autonomous Discovery of Advanced Materials)29F 30 is developing coupled computational and experimental engines that combine evolutionary exploration of chemical space with autonomous synthesis and materials testing via active robotics and AI. Similarly, the A-Lab at Lawrence Berkeley National Laboratory30F 31 and the Self-Driving Fluidic Lab at NC State31F 32 combine robotics with an AI-enabled closed-loop to speed up discovery in material science. 29 https://bigchemistry.nl/ 30 http://erc-adam.eu/ 31 https://newscenter.lbl.gov/2023/04/17/meetthe-autonomous-lab-of-the-future/ 32 https://www.abolhasanilab.com/ Scientific Applications and Workflows 16 readiness of research infrastructures for workflow execution.  Develop reference workflows leveraging solutions such as the EuroHPC Federated Platform and European Data Spaces, to serve the needs of the Scientific Applications. Long term (2+ years)  Include workflow benchmarking principles into ESFRI and EuroHPC JU roadmaps to ensure the procurement of infrastructures that are optimised for enabling workflows. Challenges addressed Current infrastructure procurement prioritises raw performance (e.g. FLOPS) over workflow portability, reproducibility, and sustainability. Sustainable Application Workflow Development A persistent obstacle in the European research ecosystem is the misalignment between funding structures and the long-term evolution of scientific applications and their workflows. Most middleware, tools, and workflow solutions are developed within short-term project cycles, each dependent on new competitive calls. As a result, researchers frequently develop innovative workflows throughout a project’s lifetime, but when funding ends, maintenance and adaptation cease, leading to fragmentation, duplication, and the loss of knowledge. This cycle undermines reproducibility, hinders cumulative progress, and limits the broader impact of public investments. To break this pattern, a shift towards sustained, workflow-aware co-design is essential. Funding bodies, research infrastructures, and scientific communities must jointly support the continuous co-evolution of applications, workflows, and infrastructure. This means treating workflows not as project-specific artefacts but as shared, evolving assets within the European research ecosystem, which will need development support through long-term funding programmes. A stronger emphasis on workflowaware co-design is required, where funding bodies explicitly support the continuous coevolution of applications, workflows, and infrastructure. Workflow solutions can only endure if they achieve broad applicability, scalability, and usability across scientific domains. Workflows must also handle provenance, reproducibility, and reusability as intrinsic properties rather than add-on features. By ensuring that workflows are user-friendly, scalable, and fully aligned with FAIR and open-science principles, we foster a sustainable ecosystem where workflow solutions remain adaptable, interoperable, and impactful across scientific domains. Technically sustainable workflows should therefore evolve in step with advancing infrastructures, evolving scientific applications, and continuously expanding data ecosystems. Actions Short Term (1 to 2 years)  Enforce workflow sustainability and maintenance plans in new funding calls and project proposals combined with corresponding initiatives that assist projects in meeting their sustainability goals. Long term (2+ years)  Establish a European Workflow Competence Centre that introduces communitydriven open development programmes for workflow software, building on the European open-source strategy. This centre will coordinate expertise, training, and shared development tools, fostering collaboration and innovation across the continent. Challenges addressed Short-term project funding fails to sustain longterm workflow and software evolution, leading to obsolescence and duplication. Scientific Applications and Workflows 17 Conclusion Scientific applications and their corresponding workflows are becoming the backbone of modern research, enabling discoveries across a diverse ecosystem of Scientific Applications. Yet, as this paper has shown, the growing sophistication of workflows brings with it equally complex challenges. From reproducibility and provenance to orchestration, heterogeneity, and security, the obstacles are cross-cutting and touch every stage of the research lifecycle. Addressing these challenges requires a shift in perspective. Workflows must no longer be treated as domain-specific artefacts but as firstclass scientific infrastructure, essential to ensuring that research is reproducible, scalable, and sustainable. This means recognising that the barriers are as much organisational and cultural as they are technical: they involve skills, standards, funding structures, and governance models as much as they involve schedulers, APIs, and accelerators. The recommendations outlined in this paper aim to translate these insights into action. Moving towards ecosystem harmonisation, through common standards, federated platforms can reduce fragmentation and lower barriers to adoption. At the same time, adopting workflow-driven infrastructure design that aligns with evolving requirements, from procurement strategies to heterogeneous computing, along with sustainable application workflow development, will ensure that scientific applications remain well-supported in the long term. This white paper is conceived as an agile and evolving document. It is intended as a living document and a tool to assess the implementation and effectiveness of the proposed recommendations periodically. The idea is to regularly evaluate the state of the art and identify any new or emerging challenges, thereby verifying the validity and alignment with the EU's strategic roadmaps. Ultimately, the success of this work depends on a collective effort. Policy bodies, funders, infrastructure providers, and scientific communities must work together to bridge gaps, consolidate resources, and promote workflows as shared, reusable assets. By doing so, Europe can lead in building a coherent and sustainable workflow ecosystem that empowers researchers to tackle the scientific and societal challenges of the coming decades. Scientific Applications and Workflows 18 Contributing authors Manuel Rodrigues (SURF) Tim Kok (SURF) Martin Golasowski (IT4I) Jan Martinovic (IT4I) Maike Gilliot (CEA) Rafael Da Silva (ORNL) Venkatesh Kannan (ICHEC) Rosa M. Badia (BSC) Guillaume Houzeaux (BSC) Acknowledgements to the participants of the “ETP4HPC Webinar – Research Applications & Workflows – Insights & Community Discussion” in which an early version of the recommendations in this whitepaper was presented, and feedback was collected via a Mentimeter survey. Scientific Applications and Workflows 19 Appendix A - Elements of Workflow Design Examining application workflow patterns more closely reveals fundamental elements that remain consistent across various domains, patterns, and use cases. These workflows are built from core components and their combinations, sometimes differing only in scale, dataset size, and computational complexity. Gaining a better understanding of these core elements, how they interact, and their impact on application workflows enables us to design effective infrastructure solutions, tools, and services. Scientific application workflows can be viewed as specific implementations of a sequence of computational steps designed to achieve scientific objectives. These steps typically involve executing repeatable computational tasks and managing data dependencies within constraints such as time, resource capacity, and infrastructure policies across national, international, or geographical boundaries. Notably, such workflows often involve managing hundreds or thousands of computational tasks to utilise HPC platforms and environments, facilitating effective accelerated discovery. In this section, we focus on some of the fundamental design elements that are independent of the execution environment, fostering a shared understanding of workflow design and composition. We define four core design elements:  Compute  Time  Data  Automation Together, these elements help identify workflow complexity, infrastructure demands, and design constraints for applications. They describe how computations can be structured, planned, and orchestrated across spatial, temporal, and data continua. Their composition defines a specific workflow archetype and, ultimately, a scientific application. Scientific Applications and Workflows 20 Compute This design category addresses the computational or processing demands within workflows or steps, within the constraints of computing platforms or environments. These are further subdivided into: Sub-Element Description Highly demanding computations Extreme Scale: Scale-up or Scale-out, suitable for iterative, long-running campaigns, multistep or ensembles, large-scale data processing or I/O, AI model training, hyperparameter tuning or optimisation, and batch-based or static planning of flows and tasks. E.g. Supercomputers, Cloud native supercomputing, HPC Clusters/systems, Flexible and Composable processing Flexible, malleable and scalable provisioning, cloud bursting, composability, and disaggregation, as well as dynamic scheduling, ease of use and access— examples include microservices deployment and visualisation. AI-enabled simulations AI-enabled simulations combine artificial intelligence with classical computing to accelerate, optimise, and enhance the accuracy of complex scientific and engineering simulations. Hybrid quantum, AI and HPC workflows Workflows which interleave in single run quantum resources, AI models / agents and classical HPC. Federation Deployment across the computational facilities, each with unique capabilities and policies. This design criterion can induce seamless execution of workflows across the continuum. Coherence/coordination is essential. Contractual and Quality of service - role of federation engine. E.g. Federated learning in Health, integration of edge/Cloud/IoT with HPC. Scientific Applications and Workflows 21 Time This design element addresses temporal sensitivity and responsiveness, as time-criticality is an essential aspect & requirement for certain classes of application workflows, particularly in cases involving big data science or interactive computing, such as Manufacturing, flood, or tsunami response running on shared infrastructures such as public HPC. Sub-Element Description Time-criticality This is defined by Real-time, near-real-time or low-latency needs. It demands greater flexibility, interactivity, steering capability, and proximity to environments, which are often not available in many HPC facilities42 F 43. The time sensitive workflows differ from urgent workflows, since they should coexist with other jobs on shared infrastructure for a longer time (i.e. imaging campaign from an instrument running for several days). Urgency Urgency dictates that completion occurs within a specific time frame; results or outcomes must be available promptly often at the cost of pre-emption of other jobs, stopping or suspending them before their completion These involve critical implications, impacts, and consequences. A subset of high time-sensitive workflows includes, for example, digital twins with actuators for the electrical grid or simulations during natural disasters occurring This means that an urgent workflow, by definition, is highly time-sensitive; however, not all workflows with high time-sensitivity are urgent. Long running jobs Another side of the time component are the long running workflows composed of one or many jobs in complex patterns. Examples of these jobs can be simulations taking up to 2 weeks to finish in a single workflow run. Various issues emerge in this time span like need for checkpointing, since probability of hardware error increases with time or validity of ephemeral credentials like API tokens and their refreshing. Scheduled / triggered runs Apart from manually triggered runs, the workflows can be run also periodically or be triggered by some external service. Delegation policy must be defined since it is not the user itself triggering the run but a scheduling service executing the run on behalf of the user. Secure handling of the user credentials must be taken into account as well as proper notification channels (e-mail, chatbot) on workflow status change. Request based computation / AI prompts Special cases of an interactive computing job are services waiting for user input like AI inference applications. They usually demand significant resources (multiple GPUs) as well as the initial set-up and model loading is a time consuming process. Appropriate scheduling and caching mechanisms must be used in a shared batch job environments such as HPC to avoid resource wasting by occupying GPUs which wait for incoming requests. 43 Reuther, Albert, Nick Brown, William Arndt, Johannes Blaschke, Christian Boehme, Antony Chazapis, Bjoern Enders, Robert Henschel, Julian Kunkel, and Maxime Martinasso. 2024. “Interactive and Urgent HPC: Challenges and Opportunities.” arXiv [Cs.DC]. arXiv. http://arxiv.org/abs/2401.14550 Scientific Applications and Workflows 22 Data This design element captures the relationship between data and its scale, particularly how it is accessed, prepared, moved, transported, managed, and staged during or after computations. We further classify it into different subelements. Sub-Element Description Data Staging Data is positioned or prepared before execution ensuring appropriate data locality. Often involves bulk transfer and careful orchestration (i.e. long data transfers should not be done on GPU nodes since it is done only on CPUs). . Data Transfer Data is moved or transferred (i) between user and target storage system and (ii) between workflow steps, workflow runs or even different workflows. Appropriate transfer protocols must be used including parallel TCP streams, taking into account possible instability of user network connection and at the same time efficient usage of high bandwidth networks in datacenters. Data Streaming Continuous flow of data between producers and consumers, often pipelined. Enables real-time analytics, progressive computation, and low-latency feedback. Consumption and buffering of the stream data must be carefully orchestrated especially in batch job environments on HPC systems to ensure optimal usage of the compute node resources. Data Temperature Data temperature describes how often and recently data is accessed, helping determine the right storage tier. Hot data is frequently used and stored on fast, usually all-flash media (scratch filesystems on HPC). Warm data is accessed occasionally and kept on mid-tier storage such as HDDs. Cold data is rarely used, residing in low-cost, high-latency storage like tape banks. Frozen or archival data is almost never accessed but retained long-term on tape or deep archive systems for compliance or legal needs. Scientific Applications and Workflows 23 Automation Automation refers to the characteristic of a workflow that coordinates its decision logic, computational tasks, data movement, and staging. It is relevant in enabling efficiency and scalability, especially in complex or large-scale workflows where manual orchestration would be impractical. Automation enables computational workflows to adapt to changes in data availability, environmental conditions, and responsiveness to external triggers. Effective automation abstracts the operational complexity from users, allowing them to focus on scientific or domain-specific objectives rather than the logistics of execution. Sub-Element Description Semi-automated Some tasks are orchestrated automatically, but human input or intervention is required at defined stages, such as validation, approval, or configuration tuning. Fully-Automated The entire workflow proceeds without human intervention, relying on predefined logic and rules or supported by AI steering and intelligent scheduling systems. Event triggered Workflow execution is initiated or modified in response to external events or system conditions, such as the arrival of new data, completion of a prior task, or a scheduled trigger. Scientific Applications and Workflows 24 Appendix B - Table of recommended actions Ecosystem Harmonisation Workflow-driven Infrastructure Design Sustainable Application Workflow Development Short term (1-2 years)  Establish a European Workflow Interoperability Task Force to align on workflow description languages, APIs, and metadata.  Create and maintain a catalogue of workflow standards, interfaces, frameworks and registries that relate solutions to the different types of Scientific Applications and representative research.  Establish pilots for federated identity and compliance, utilising existing AAI infrastructures (e.g. MyAccessID, EGI Checkin) to ensure the secure execution of workflows across sites and borders.  Define workflow-driven criteria into infrastructure procurement and evaluation (e.g., reproducibility, portability, and scientific ROI and TCO), e.g., through a Workflow Compatibility Index to assess the readiness of research infrastructures for workflow execution.  Develop reference workflows leveraging solutions such as the EuroHPC Federated Platform and European Data Spaces, to serve the needs of the Scientific Applications.  Enforce workflow sustainability and maintenance plans in new funding calls and project proposals combined with corresponding initiatives that assist projects in meeting their sustainability goals Long term (2+ years)  Align national and EU policies on security, authentication, and accountability across infrastructures, providing the basis for European governance on AAI.  Create an EU Workflow & Infrastructure Observatory to monitor the adoption of standards, compliance, progress in harmonisation, and establish a workflow validation scheme to promote harmonisation and interoperability.  Include workflow benchmarking principles into ESFRI and EuroHPC JU roadmaps to ensure the procurement of infrastructures that are optimised for enabling workflows.  Establish a European Workflow Competence Centre that introduces community-driven open development programmes for workflow software, building on the European open-source strategy. This centre will coordinate expertise, training, and shared development tools , fostering collaboration and innovation across the continent. Scientific Applications and Workflows 25 Cite as: Dolas S. et al., ETP4HPC SRA6 White Paper – Scientific applications & workflows, 2025, ETP4HPC. https://doi.org/10.5281/zenodo.17635534 DOI: 10.5281/zenodo.17635534 © ETP4HPC 2025