Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing

Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing Zhaobin Zhu, Leonie Derstroff, Sarah M. Neuwirth Johannes Gutenberg University Mainz, Germany [email protected] REX-IO 2025 Workshop, IEEE CLUSTER, September 2025 Background & Motivation Why HPC I/O Analysis Matters Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 2 Application Network File System • Number of processes • Request sizes • Access patterns • I/O operation • Data vo lume • Message sizes • Network topology • Network paths • Network type • Type of file system • Disk types • Stripe sizes • File hierarchy • Shared access Application High-level I/O Libraries Parallel File System HDD DAOS SSD NVM Tape Object Store Application TensorFlow Pandas Application Virtual Machine S3 Cloud Store ... ... ... Low-level I/O Libraries MPI-IO I/O Forwarding Layer RAID Compute paradigms Storage paradigms Diverse Workloads Diverse Software Diverse Architecture Diverse Hardware I/O bottlenecks limit scalability and throughput Profiling & tracing tools essential for optimization HPC workflows increasingly data-intensive Background & Motivation Landscape of I/O Tools Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 3 •Many tools, different design philosophies •Challenge: Which tool to use for which purpose? Tool Languages Profiling Tracing Monitoring Darshan C, Python, Perl    Recorder C, C++, Python  Score-P C, C++, Fortran    TAU C, C++, Java    Scalasca 2.x C, C++    Beacon C, Python, JavaScript  SIOX C, C++, Python   DFTracer C, C++, Python  Background & Motivation Problem Statement Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 4 Multiple performance analysis tools for I/O available Tools produce different metrics & perspectives Users struggle to compare or combine results Our questions: −How do Darshan and Recorder differ in practice? −What are their trade-offs in scalability vs. detail? Tools in Focus Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz5 Tools in Focus Darshan and Recorder in a Nutshell Darshan Recorder Output Darshan binary Recorder binary Analysis Tools / Format Darshan logs, Darshan util scripts, Gauge, DXT explorer (trace analysis) Recorder logs, recorder-viz, Chromium trace file, Parquet Default Instrumentation LD_PRELOAD LD_PRELOAD Interface Categorization MPI-IO, POSIX, HDF5, PnetCDF, Lustre and STDIO MPI-IO, POSIX, HDF5, PnetCDF, NetCDF I/O Grouping •Functions grouped into read & write for each module •Non read or write are not displayed in the trace •Original function names are not retained in the DXT Original function names are retained with additional categories such as module and operation type (read / write) I/O Trace Scope Trace only covers intercepted I/O read / write operations Several POSIX functions are intercepted and several other MPI calls that does not include MPI_Init and MPI_Finalize Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 6 Recorder: Parallel I/O Tracing Tool •Tracing tool at function-call level •Captures POSIX, MPI-IO, full HDF5 (749 calls) •Includes timestamps, parameters, process-level details Darshan: I/O Characterization Tool •Lightweight profiling tool •Aggregated statistics (counts, sizes, bandwidth) •Optional DXT module for detailed POSIX/MPI-IO traces •Produces compact summaries, low runtime overhead Tools in Focus Darshan and Recorder: Key Differences Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 7 Snyder, S., 2022. Darshan: Enabling Insights into HPC I/O Behavior. ECP Community BoF Days. Wang, C. et al., 2025. Recorder: Comprehensive Parallel I/O Tracing and Analysis. Tools in Focus Comparison of Intercepted Functions POSIX MPI-IO HDF5 Both Tools open, open64, creat, creat64, dup, dup2, fileno, read, write, pread, pwrite, pread64, pwrite64, readv, writev, lseek, lseek64, __xstat(+64), __lxstat(+64), __fxstat(+64), mmap(+64), fsync, fdatasync, close, rename, fopen(+64), fdopen, fclose, fwrite, fprintf, fread, fseek, fseeko, fflush PMPI_File_{close, iread at, iread, iread_shared, iwrite at, iwrite, iwrite_shared, open, read_all_begin, read_all, read_at_all_begin, read_at_all, read_at, read, read_ordered_begin, read_ordered, read_shared, set_view, sync, write_all_begin, write_all, write_at_all_begin, write_at_all, write_at, write, write_ordered_begin, write_ordered, write_shared} H5Fcreate, H5Fopen, H5Fflush, H5Fclose, H5Dcreate1/2, H5Dopen1/2, H5Dread, H5Dwrite, H5Dclose, H5Oopen, H5Oclose, H5Dflush, H5Oopen_by_addr, H5Oopen_by_idx, H5Oopen_by_token Darshan only __open_2, openat(+64), dup3, mkstemp, mkostemp, mkstemps, mkostemps, preadv(+64/2), pwritev(+64/2), aio_read/write(+64), aio_return(+64), lio_listio(+64), freopen(+64), fputc, putw, fputs, printf, vfprintf, vprintf, fgetc, getw, _IO_getc, _IO_putc, __isoc99_fscanf, fscanf, vfscanf, fgets, fseeko64, fsetpos(+64), rewind – – Recorder only msync, getcwd, mkdir, rmdir, chdir, link, unlink, linkat, symlink(+at), readlink(+at), chmod, chown(+lchown), utime, opendir, readdir, closedir, rewinddir, __xmknod(+at), fcntl, pipe, mkfifo, umask, access, faccessat, tmpfile, truncate, ftruncate, ftell, remove, ftello PMPI_File_{set_size, seek, seek_shared, get_size, iwrite_at_all, iwrite_all} Recorder provides full HDF5 API coverage Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz 8 * Functions are taken from the tools’ source code. Functions in blue color are categorized as STDIO/ISO-C by the tools. Evaluation Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz9 Related Work & Context Fragmented Tooling Landscape Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz16 •Siloed Tool Views: Each tool sees a layer. None explain the whole system. => In case of I/O: App-level profilers (e.g., Darshan) vs. system tools (LDMS, DCDB) •Workflow Scripts Instead of Workflows: Custom scripts per experiment = unscalable, unrepeatable. => Benchmarking tools (e.g., iperf, sockperf) often require client/server logic incompatible with SLURM •Need: integrated multi-layer explainability Category Examples Srengths Limitations Application-level Darshan, Recorder, Score-P Fine-grained function tracing and lightweight profiling No visibility into system-wide interactions System-level LDMS, DCDB, TACCStats Aggregated I/O performance metrics Cannot correlate application performance with system metrics End-to-end Ganglia, Nagios, Apollo Holistic view of system utilization Lacks deep profiling at kernel and network levels Related Work & Context Mango-IO: I/O Metrics Consistency Analysis •Problem: metrics differ between tools → comparability issue •Solution: Mango-IO converts Darshan/Recorder traces → OTF2 for consistency •Finding: once normalized, discrepancies shrink Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz17 Liem, Radita, Sebastian Oeste, Jay Lofstead, and Julian Kunkel. Mango-IO: I/O Metrics Consistency Analysis. In 2023 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops), pp. 18-24. IEEE, 2023. Related Work & Context XIO: eXplainable I/O Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz18 Current tools answer what, not why Neuwirth, S. and Devarajan, H., Wang, C., and Lofstead, J.., 2025. XIO: Toward eXplainable I/O for HPC Systems. SSDBM’25. XIO proposes Master Architectural Plan (MAP) + DataCrumbs Related Work & Context XIO: eXplainable I/O Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz19 DataCrumbs: LowOverhead Multi-Layer Profiling for enabling Explainable I/O Different kernel stack calls can help identified buffered vs unbuffered read calls. DataCrumbs: eBPF-based, kernel + user tracing => causal explanations Conclusions Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz20 Conclusions Recommendations Future: Hybrid workflows combining both (or other tools); consistency via Mango-IO? When system-level behavior (e.g., metadata-heavy workloads, STDIO filtering) matters or needs control, Recorder’s fine-grained policies are more adaptable than Darshan’s fixed aggregation model. Choose Recorder for fine-grained, trace-level inspection or when working with applications using complex or layered I/O stacks like HDF5. Use Darshan when profiling needs to scale across many nodes, runtime perturbation must be minimal, and aggregated I/O summaries suffice. Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz21 Conclusions Summary & Outlook Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz22 Tools are not Interchangeable •Workload & goal matter •Profiling vs. tracing vs. explainability: each gives a different truth Metadata Standardization •Can we converge on trace metadata schemas across tools? •How do we ensure trace context is captured and preserved? From Metrics to Meaning •What constitutes a verified insight? •Can we establish common ground between correctness verification and performance validation? Vision: Integrated, Multi-layer Ecosystem for I/O Analysis •Combining: Efficiency (Darshan-like), Detail (Recorder-like), Consistency (e.g., Mango-IO), Explainability (XIO/DataCrumbs) Thank you for your Attention! Dr. Sarah M. Neuwirth Professor of Computer Science Johannes Gutenberg University Mainz Email: [email protected] Website: https://www.hpca-group.de/ NHR South-West HPC Center: https://nhrsw.de/ Benchmarking Darshan and Recorder for HPC I/O Profiling and Tracing • ©Sarah M. Neuwirth • Johannes Gutenberg University Mainz23