ALCF Polaris Darshan Log Collection

ALCF Polaris Darshan Log Collection Description This repository contains Darshan logs captured from production jobs running on ALCF Polaris. These logs have been anonymized to protect sensitive user identifying information. In this document we provide an overview on the collected Darshan data, the ALCF Polaris system, tools and best practices for analyzing Darshan data, as well as recommended ways for acknowledging this work. Darshan Background Darshan is an application-level I/O characterization tool that is specifically designed to help understand the behavior of HPC I/O workloads. It captures detailed information (counters, timers, and other statistics) describing access patterns for each file accessed by an application. Darshan utilizes lightweight and transparent instrumentation methods that enable its full-time deployment on production HPC systems like ALCF Polaris. Darshan generates a single, compressed log file for each application it instruments. Further details on Darshan’s design and on the data it captures is provided in our previous work [2]. Darshan also provides multiple tools for extracting and analyzing log data: •PyDarshan (preferred) – A new Python package providing interfaces for extracting Darshan log data into Pythonic data formats (e.g., Numpy arrays or Pandas DataFrames), as well as tools for analyzing this data (e.g., the PyDarshan job summary tool). – More details on PyDarshan and its usage can be found in these docs and in our previous work [3]. •darshan-util – Traditional C library and corresponding tools (e.g., darshan-parser , darshan-job-summary.pl ) for extracting and analyzing Darshan log data. –More details on darshan-util can be found in these docs. Darshan Data Overview The complete Polaris Darshan log collection is available via Globus and can be directly accessed HERE. The Darshan log data in this collection is organized in a directory tree by year/month/day (based on when the job finished). Within each day directory, there is a tarball named logs.tar.gz containing all Darshan logs collected on that day. A helper script, unpack-darshan-logs.sh , is available at the top-level of the collection to assist in unpacking all Darshan log data. Invoking this script with no arguments will unpack all Darshan data in the collection. To limit the unpacking to a particular directory, pass the directory name as an argument to the script. For example: # for all logs from year 2024 unpack-darshan-logs.sh /path/to/collection/2024 # for all logs from day 5/8/2024 unpack-darshan-logs.sh /path/to/collection/2024/5/8 To mask any sensitive user identifying information, various parts of the logs have been anonymized. This includes the log file names themselves which have been renamed to ‘[jobid]-[random_val]’. The following fields within the logs have also been anonymized and replaced with a hash value: •uid •exe (command line) •file name suffix (but not the preceding file system mount point) 1 This hash is consistently applied to all logs in the collection (e.g., real user IDs map one-to-one to anonymized user IDs across the entire collection). ALCF Polaris System Overview Polaris is a 520-node HPE Apollo 6500 Gen 10+ based system deployed at the ALCF. Each node has one 2.8 GHz AMD EPYC Milan 7543P 32-core CPU (with 512 GB of DDR4 RAM), four Nvidia A100 GPUs, and two Slingshot 11 network adapters. Polaris offers a few different storage systems intended for users’ data: • Eagle/Grand, Lustre scratch file systems composed of 160 object storage targets (OSTs) and 40 metadata targets (MDTs), with a total capacity of 100 PiB and peak performance of 650 GiB/s –mounted at /lus/eagle and /lus/grand •home, a Lustre file system meant for storage of non-I/O intensive user data –mounted at /home • temporary node-local storage for jobs, using two SSDs (XFS formatted) with a total capacity of 3.2 TiB and peak performance of 6 GiB/s –mounted at /local/scratch FAQ Why do I get compression errors like the following when trying to read Darshan logs from this collection? darshan-parser ff54db84-3a92-47a7-81a8-e5e97f2acaba.darshan Error: invalid compression type. Error: failed to initialize decompression data structures. To avoid this error, users need to ensure that the darshan-util package was built with bzip2 support: 1. For source installations, make sure you have the bzip2 development package installed (e.g., using APT, atp-get install libbz2-dev) before building Darshan. • If needed, you may need to pass the bzip2 install prefix to Darshan’s configure script using the --with-bzlib=DIR option. 2. For Spack installations of darshan-util , make sure to specify the +bzip2 variant (i.e., spack install darhan-util+bzip2). Do these logs cover all applications ran on Polaris over this time frame? No, these logs only cover a portion of jobs ran on Polaris, for a number of reasons: • Applications that do not use MPI (or MPI applications that do not call MPI_Finalize() ) are likely not instrumented unless users specifically enabled Darshan’s non-MPI mode (which is not used by default). •Some applications explicitly disable Darshan instrumentation. Is there any documentation on the meaning of various counters captured by Darshan? Yes, see this description of currently supported counters in the darshan-util docs. Acknowledging/Citing This Data If you use this data in your own work, we encourage you to cite the Zenodo record [1] and corresponding publication about how the repository was created [4]. Note that we will updated the reference information for [4] with additional detail once the ACM proceedings have been published for CUG 2025. Additionally, please include the following ALCF acknowledgment in any publications that use data from this repository: This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory and is based on 2 research supported by the U.S. DOE Office of Science-Advanced Scientific Computing Research Program, under Contract No. DE-AC02-06CH11357. ALCF Public Data Repository For access to more ALCF datasets visit: https://reports.alcf.anl.gov/data/index.html. References [1] ALCF Polaris Darshan Log Collection. (2025). doi:10.5281/zenodo.15052603. [2] S. Snyder, P. Carns, K. Harms, R. Ross, G. K. Lockwood, and N. J. Wright, “Modular HPC I/O Characterization with Darshan,” in 2016 5th Workshop on Extreme-Scale Programming Tools (ESPT), 2016. [3] J. Luettgau, S. Snyder, T. Reddy, N. Awtrey, K. Harms, J. L. Bez, R. Wang, R. Latham, and P. Carns, “Enabling Agile Analysis of I/O Performance Data with PyDarshan,” in Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, 2023. [4] Shane Snyder, Philip Carns, Kevin Harms, Rob Latham, and Robert Ross. “Expanding Community Access to Real-world HPC Application I/O Characterization Data using Darshan”. In Proceedings of the Cray Users Group meeting (CUG2025), May 2025. 3