A framework and methodology for performance prediction of HPC workloads

A Framework and Methodology for Performance Prediction of HPC Workloads J´ ulia Orteu∗, Marc Clasc` a∗, Marta Garcia-Gasulla∗, Jes´ us Labarta∗†, Elise Jennings‡ ∗Barcelona Supercomputing Center, Barcelona, Spain †Universitat Polit` ecnica de Catalunya, Barcelona, Spain ‡ParTec AG, M¨ unchen, Germany E-mail: {julia.orteu, marc.clasca, marta.garcia, jesus.labarta}@bsc.es [email protected] Keywords—HPC Workloads, Performance Prediction, Runtime Hardware Counters, Instructions per Cycle (IPC), Performance Tools, Parallel Applications, Regression trees, ML & AI I. EXTENDED ABSTRACT A. Introduction The presented poster outlines an approach for predicting the performance of High-Performance Computing workloads (HPC). By utilizing data gathered from runtime hardware counters across a range of HPC applications and benchmarks, we develop an artificial intelligence model based on ensemble tree algorithms. This model is capable of forecasting the performance of other HPC applications. This work differs from current research by focusing on the granularity of training and prediction. Specifically, our model is developed utilizing individual computation bursts as input samples for training. Through this approach, we prove that a prediction of the instructions per cycle (IPC) metric of unseen applications is possible based on architectural performance counters that can be obtained easily with already used and convenient performance tools. B. Research and Development The research line focuses on exploring the possibility of predicting the performance, and ultimately the execution time, of known workloads in HPC machines prior to their execution. Predicting the performance of HPC applications or workloads is a complex task with a widely discussed set of approaches and methodologies. The latest research work on this topic focuses on data-driven methodologies using machine learning, but previous studies present different types of prediction methods: analytical methods, that can be derived from a machine representation, or can be the result of a statistical work; and non-analytical methods, which comprise the artificial intelligence techniques and the simulation methods [1] . Our approach distinguishes itself from existing research by its focus on the granularity of training and prediction. Using performance analysis tools Extrae and Paraver [2] [3], developed in Barcelona Supercomputing Center (BSC), we extract data at the computational burst level and target the IPC metric of every burst as a performance measure. Fig. 1. Extrae and Paraver generalization of burst A burst is defined as the time interval of active operation between two successive events in a process. In Figure 1, we provide a visual example of a timeline from a execution representation of an application running two processes. The activities within these processes have been categorized into two types: communication bursts, indicated in red, and computation bursts, represented in blue. These categorizations are derived from MPI event markers, with green flags initiating computation and red flags marking the start of communication. In our study, we concentrate on computation bursts, which we refer to as useful bursts. These bursts are periods where the application is actively engaged in data processing and executing instructions. These individual useful bursts serve as the input samples to the model, as one entry of our dataset, highlighting that this approach provides a very precise performance prediction of a very specific application’s part. C. Features and workload characterization To facilitate our study, we consider a set of Performance Application Programming Interface (PAPI) counters [4] as the foundational data. These counters include the total number of instructions completed (N), the total cycles seen by thread (Cyc), memory load instructions (NLD), memory store instructions (NSR), branch instructions (NBR), and total cache miss events at both L1 and L3 levels (missL1,missL3). We normalize the data in each burst by using the ratio of instructions and cache misses instead of absolute counters values. This allows us to compare any individual burst from any part of a trace and from any application. From the counters, we characterize the variables using the following computations: •Instruction mixes, which are ratios of specific instruction types to the total number of instructions, such as rLD =NLD Nfor loads, rSR =NSR Nfor stores, and rBR =NBR Nfor branches. •Cache miss rates, calculated as rL1=missL1 NLD+NSR for the L1 cache and rL3=missL3 NLD+NSR for the L3 cache, which are indicators of memory access efficiency. •The average node concurrency during core execution, through the integral of the Parallelism function over the time from Tbegin to Tend:RTend Tbegin P ar(t)dt. •IPC, which is the target performance metric, given by the equation IP C =N Cyc . D. Data sources The process involves the selection of a specific set of benchmarks and kernels to extract data and adapt it for the purpose of training the models. Additionally, a separate set of applications has been chosen for evaluating the performance of the trained models (testing). The selection of kernels and benchmarks for the training set is a pivotal decision that facilitate the representation of the burst space, enabling the generalization to new applications. For each application selected for the training set, we’ve varied the problem sizes to capture a comprehensive dataset. The nature of the variation depends on the application’s characteristics—it could be the size of an array, the granularity of a mesh, or the complexity of inputs. Additionally, we’ve scaled the computational workload by altering the number of processes within a single node for each variant of problem size. This methodical approach allows us to construct a training dataset that covers a wide range of scenarios. E. Data extraction Framework Fig. 2. Flowchart of the data extraction part of the training framework The data extraction process for each application, both for training and testing datasets, is outlined in Figure 2. The depicted data extraction process begins with the execution of a known application on a known machine. In this case, we have used the MareNostrum 4 supercomputer’s architecture for reference. We obtain the useful bursts from a program execution using the BSC tools, which are then processed into a features format. F. AI Model The study investigates a range of diverse machine learning algorithms with the aim of training an effective predictive model. In this exploration, we have employed a 10-fold cross-validation method on our training dataset to evaluate the performance and compatibility of these algorithms with our type of data. The empirical results highlight a clear advantage of using Boosting Ensembles that rely on decision trees, leading to a focus on XGBoost method [5] . This also matches with previous research conclusions [6]. To ensure a fair assessment of the models, we’ve developed a method to inject these predictions into Paraver traces and we’ve devised specific error metrics into the trace tailored to evaluate the performance outcomes of each model. This allows for a more precise analysis of how well the models predict application performance. G. Summary The poster shows the main findings and discusses our exploration of this topic. We present a method of data collection and preprocessing based on low-effort program instrumentation and automatic tools, as it is well known in the literature that being able to collect data automatically and building the model effortlessly is critical to end up with valuable and convenient training and prediction workflow. We also studied how changing the feature set, the training data size or the machine learning algorithm affects the accuracy. We discuss a method to characterize computational bursts based on instruction mix features and instantaneous machine concurrency that is later able to classify, using trees, the IPC of unseen bursts. Therefore, we prove that it is possible to foresee the performance of a whole unseen application trace based on this characterization method. II. ACKNOWLEDGMENT This work bas been published in proceedings of the 11th International BSC Severo Ochoa Doctoral Symposium, 2024. REFERENCES [1] J. Flores-Contreras et al., “Performance prediction of parallel applications: a systematic literature review,” The Journal of Supercomputing, vol. 77, no. 4, pp. 4014–4055, 2021. [2] V. Pillet et al., “Paraver: A tool to visualize and analyze parallel code,” in Proceedings of WoTUG-18: transputer and occam developments, vol. 44, 1995, pp. 17–31. [3] H. Servat et al., “Framework for a productive performance optimization,” Parallel Computing, vol. 39, no. 8, pp. 336–353, 2013. [4] S. Browne et al., “Papi: A portable interface to hardware performance counters,” 1999. [5] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 785–794. [6] J. Sun et al., “Automated performance modeling of hpc applications using machine learning,” IEEE Transactions on Computers, vol. 69, no. 5, pp. 749–763, 2020. J´ ulia Orteu is a student in the first cohort of the Bachelor’s degree in Artificial Intelligence at Universitat Polit` ecnica de Catalunya (UPC). Since 2023, has been a Junior Research Engineer at the Barcelona Supercomputing Center (BSC) in the Best Practices for Performance and Programmability (BePPP) group, where she currently works as a performance prediction analyst for HPC Application Workloads.