I/O Simulation: From Resources to Complex Workloads

1 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY I/O Simulation: From resources to complex workloads September 2, 2025 –Edinburgh, Scotland Fred Suter REX-IO 2025 2 Evolution of I/O and Data Management App. Disk Tape POSIX Files, raw scientific data Files, raw scientific data Output Archival Large-scale Distributed High Perf. I/O framework Self-described, Reduced, Refactored, Surrogated Parallel & Distributed File System Streaming Self-described, Reduced, Refactored, Surrogated Output Retrieval Query Flash vs. HDD DNA storage QoIs, RoIs, Inference App. Disk Tape POSIX Files, raw scientific data Files, raw scientific data Input Preload On a shared and limited infrastructure 3 Overarching Questions in That Context Can we •Build better I/O and data management software to accelerate science? •Optimize resource utilization and handle dynamic changes? •Reduce manual interventions in complex data management tasks? Yes, if we can take the right decisions and select the right levers at the right time What is needed for that? •Performance models that are fast, scalable, dynamic / interactive •And can capture the entire HW/SW stack and have predictive value In other words, we need a comprehensive Digital Twin 4 Why Simulate I/Os and Storage? An important performance driver to understand •Independent of scale and type of the computing infrastructure •As much important as computing and networking Specifics and concerns of storage subsystems may vary •Data Centers  Hierarchical (mass) storage subsystems  Different types of media involved •Supercomputers  Large-scale dedicated storage network  High-speed network interconnect •Clusters  Specific and tuned file system  Reliable, scalable, and simple •Grids and Clouds  Services offered by multiple data centers  Hidden underlying infrastructure Versatility is key! 5 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY A Brief History of 6 Distributed Systems as Scientific Objects to Study Clusters, supercomputers, peer-to-peer systems, grids, clouds, . . . How to study these systems and their applications on my laptop? 7 The SimGrid Toolkit Open Project since 1998 •2,200+ citations and 665+ usages Key strengths •Usability: Fast, Reliable, User-oriented APIs •Validated performance models: Open Science  Predictive Power •Versatility: Grid, P2P, HPC, Cloud, Fog, … A scientific instrument on your laptop https://simgrid.org 8 SimGrid History 2014–2025: Utilisability and Extensibility 1998 –2001: Factor student code (DAG scheduling) Casanova, H. Simgrid: a toolkit for the simulation of application scheduling 2001–2005: CSP and improved network models Legrand, A., Marchal, L., Casanova H. Scheduling distributed applications: the simgrid simulation framework 2005–2014: Versatility, Accuracy, Scalability Casanova, H., Legrand, A., Quinson, M. Simgrid: A generic framework for large-scale distributed experiments Casanova, H., Giersch, A., Legrand, A., Quinson, Suter, F. Versatile, scalable, and accurate simulation of distributed applications and platforms SG1 SG2 SG3 SG4 Casanova H., Giersch A. Legrand, A., Quinson M, Suter, F. Lowering Entry Barriers to Developing Custom Simulators ofDistributed Applications and Platforms with SimGrid 9 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY SimGrid in a Nutshell 16 Back in 2015 –Ground Truth Data Acquisition Testbed −Grid'5000 experimental platform (https://www.grid5000.fr) −Three types of disk: SATA-II, SAS, and SATA/SSD Methodology −Randomized FIO benchmarks −Synchronous, non-buffered I/O operations −Independent: From 32kiB to 2GiB with a fixed block size of 32KiB −Concurrent: 1 to 15 operations −For 10, 50, 100, 500, 1024, and 2048 MiB files http://dx.doi.org/10.6084/m9.gshare.1175156 17 Deriving Models from Experimental Data SSD HDD •Linear w.r.t. bandwidth •No latency •Heteroscedastic behavior −Variability proportional to size Concurrent accesses •Modify resource capacity as concurrency increases •Reevaluate each time a transfer begins or ends https://simgrid.org/doc/latest/Calibrating_the_models.html#i-o-calibration 18 File System Plugin, I/O Streams, and JBOD File System Plugin (ca. 2017) – Better separation of concerns −SimGrid models: raw byte streams on disk (read/write bandwith) −Plugin: concept of file and standard operations, Posix-like operations on file descriptors I/O streams – Speed up simulations −Model [read] – transfer – [write] from a host to another as a fluid activity −Disk to disk, disk to memory, or memory to disk −Fluid? Works as if doing store-and-forward at a very fine grain −I/O and Comm activities progress together at the limiting bandwidth speed JBOD Plugin – Modeling RAID systems −New concept of compound activity −Combines several activities and wait for the completion of the last one 19 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY I/Os in SimGrid –File system level 20 Wrench •Project initiated in 2016 –With R. Ferreira da Silva (ORNL) and H. Casonova (UH Manoa) •Objectives –A virtual lab to study WMS –Improve SimGrid expresiveness •DSL-like approach: –High level concepts –Composable modules –Different levels of APIs https://wrench-project.org 21 Wrench Overview 22 Implementation FIVES: a Simulator of High-Performance Storage Systems Courtesy of F. Tessier and J. Monniot 23 FIVES’ Compound Storage Service: Courtesy of F. Tessier and J. Monniot ●Generic model of a distributed file system ●Supports splitting a file in parts and distributing it on multiple Storage Services ●Integrated as a service into WRENCH Internally ●File Index →Free MDS ●Allocator →User provided allocation policy −FIVES comes with the Lustre round-robin/weighted policy 24 File System Module Motivations •Factor development of similar capabilities between SimGrid and WRENCH •Replace the old and simplistic file system plugin Objective •Implement a simulated file system on top of SimGrid •Support the notion of partitions that store directories that store files •Standard operations: create, move, unlink files, unlink directories, check for existence) •Support the notion of a file descriptor with POSIX-like operations (open, seek, read, write, close). Approach •Developed as a standalone library to be used in any SimGrid-based simulator Future work •Integrate the development of a Lustre file system made in FIVES https://github.com/simgrid/file-system-module 25 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY DTLMod – Versatile Simulated Data Transport Layer More details on DTLMod Thursday Sep 4th Session 7 on Performance Modelling and Optimisation in Pentland room 32 Ground-truth execution data from target system Simulator of target system Parameter ranges to calibrate −I/O bandwith, RAM page cache, WAN bandwidth, … Loss function Calibration algorithm −Grid, Random, Gradient Approach: Automated Calibration Procedure Courtesy of J. McDonald https://github.com/wrench-project/simcal 33 Determining Levels of Detail for Simulators Evaluation of Workflow Scheduling strategies Ground truth •5 application workflows •5 sizes (#tasks) •5 per-task CPU work amounts •4 data footprints 34 ORNL IS MANAGED BY UT-BATTELLE LLC FOR THE US DEPARTMENT OF ENERGY I/O simulation – The ground truth challenge 35 There can’t be any (useful) simulation model without data Application Comm.Computing I/O CPU Network StorageGPU Resource manager MPI runtime cuDA, HIP, Kokkos, … ADIOS, HDF5, NetCDF, … Lustre, DAOS, VectorDB LCF HPSS, DNA MGARD, SZ, ZFP, … Time Resources 36 Finding Relevant Datasets is Hard and Requires Processing! Darshan logs from Theta (ANL) →Year 2022, ~18,000 jobs Need for filtering and clustering of monitored jobs: ●Removing jobs with no I/O activity, classification based on bw performance, etc ●Reducing heterogeneity / addressing technical limitations of simulators Courtesy of F. Tessier and J. Monniot 37 Challenges Performance data management faces the same challenges as scientific data management Challenge 1: How can we capture and efficiently export and store performance data? Design monitoring tools along the same principles as for science data Challenge 2: How can disparate information from multiple sources regarding data management activities be fused into useful knowledge? Build AI surrogates of DM workloads Challenge 3: How can we reproduce Data Management behavior in a controlled fashion for “what if” investigations? Design multi-scale, multi-fidelity performance evaluation tools And some more … Data sharing vs. policies, integration of new advanced storage, coordination beyond DM, … 38 Conclusion Simulation can be used to assess the performance of I/Os and data management −From the resources (a.k.a. disks) −To application workflows −Through (distributed) file systems Models and tools exist −SimGrid, FSMod, FIVES, WRENCH, DTLMod … and many others outside the SimGrid ecosystem −Contributions welcomed! But challenges remain •Ground truth data acquisition and accessibility •Calibration of the simulation models •Selection of the appropriate level of detail 39 Questions? Thank you for your attention