HiPC25 Artifact: Energy-Aware Runtime Resource Harmonizer for Co-running Applications

1 Energy-Aware Runtime Resource Harmonizer for Co-running Applications (Artifact) I. GETTING STARTED This artifact provides the complete implementation and evaluation framework for Harmonizer, along with two additional runtimes - NuPoCo and Mapper - which were used for performance comparison in the paper. It includes detailed build instructions, benchmark setup procedures, and scripts required to reproduce all experimental results presented in the paper. The applications were compiled using LLVM version 17.0.0 with the -O3 optimization flag. All experiments and evaluations were conducted on a quad-socket Intel Xeon Gold 5318H system comprising 72 physical cores (18 cores per socket), with each core supporting two hardware threads, resulting in 144 logical cores. The system was equipped with 512 GB of RAM and ran Ubuntu 20.04.2 LTS. Turbo Boost was disabled to reduce performance variability and ensure consistent measurements(See Section 3 in the paper). II. PRE-REQUISITES INSTALLATION This section details the step-by-step installation of all required tools and system dependencies. A. Mandatory System Configuration and Dependencies System Configuration: The runtimes in this artifact were evaluated on Intel processors. The support for AMD processors is not provided, as according to publicly available documentation, it is unclear whether they provide support for UFS or uncore PMU measurements. Additionally, the entire software stack and tooling used in this artifact operate on Linux-based systems. The following steps require sudo access. Users must ensure that they have administrative privileges before proceeding. To ensure correct and reproducible behavior of the frequency control mechanisms used by the runtimes, the system must support CPU frequency scaling through the acpi-cpufreq driver. This tool allows the runtime to dynamically manage core frequencies. Without proper ACPI and frequency scaling support, frequency control may not function as intended, leading to unreliable performance results. Install the required system dependencies using: sudo apt update sudo apt install -y build-essential numactl msr-tools linux-headers-$(uname -r) cpufrequtils acpi-support acpid acpi hwloc libhwloc-dev Dependency Overview: •build-essential:Provides core compilation tools such as gcc,g++, and make, which are required to build the runtimes and benchmarks. •linux-headers-$(uname -r):Ensures kernel-level compatibility for low-level modules and interfaces used by MSR access and performance monitoring frameworks. •cpufrequtils:Enables CPU frequency scaling and governor management via the acpi-cpufreq driver. •acpi, acpid, acpi-support:Provide the ACPI subsystem support required for power management and frequency control mechanisms. •msr-tools:Provides access to Model Specific Registers (MSRs), which are required for writing to uncore frequency control registers. •numactl:Utility for inspecting NUMA topology and retrieving system-related information. •hwloc, libhwloc-dev:Used for detailed hardware topology discovery, including cores, sockets, caches, and NUMA hierarchy. To verify NUMA layout, system topology and list machine related information, run: numactl --hardware Verify that the system is using the acpi-cpufreq driver: cpupower frequency-info Enable Model Specific Register (MSR) access using: sudo modprobe msr MSR access is mandatory for reading and writing to uncore frequency control registers. Failure to configure these settings may result in incorrect frequency control behavior and unreliable experimental results. 2 B. Install LLVM 17 Compilation with gcc/g++ is supported; however, the evaluation in this paper uses binaries compiled with LLVM. wget https://apt.llvm.org/llvm.sh chmod +x llvm.sh sudo ./llvm.sh 17 After installation, verify the version: llvm-config --version C. Install LIKWID (Version 5.2.2) LIKWID is required to check the list of core and uncore frequencies supported by the machine. It is also used to set and reset the system to the default frequency settings before any test. wget https://github.com/RRZE-HPC/likwid/archive/refs/tags/5.2.2.tar.gz tar -xzf 5.2.2.tar.gz cd likwid-5.2.2 make sudo make install echo ’export PATH=/usr/local/bin:$PATH’ >> ˜/.bashrc source ˜/.bashrc To list the available core frequencies: likwid-setFrequencies -l To check the current frequency settings: likwid-setFrequencies -p D. Install libpfm4 libpfm4 is used to access hardware performance counters in a portable way. The runtimes used in this artifact use performance monitoring events, such as INSTRUCTION_RETIRED(instructions retired), MEM_LOAD_RETIRED.L3_HIT(L3 hits), and UNC_CHA_IMC_READS_COUNT.NORMAL (memory read requests) to collect profiling information during execution. git clone https://github.com/wcohen/libpfm4.git cd libpfm4 make install E. Install HCLib HCLib is a task-based parallel programming model library and is used to compile the benchmark ”CoHMM” for testing and evaluation. The library is already cloned in the benchmarks directory. cd Harmonizer/benchmarks/hclib ./clean.sh && ./install.sh source /home/<USER>/Harmonizer/benchmarks/hclib/hclib-install/bin/hclib_setup_env.sh III. ARTIFACT ACCESS Download the artifact from the given link and unzip the file: Link: https://zenodo.org/records/17481166 After downloading, extract and navigate to the Harmonizer directory: unzip Hipc25_Harmonizer.zip cd Harmonizer A. Update Machine-Specific Configuration Update the machine-specific configurations in the following file: src/inc/machine_info.h The information needed to update this file can be obtained from the commands mentioned in Section II. 3 B. Enable PMU Counter Support To check the performance monitoring counters supported on the system, run: cd libpfm4/examples ./showevtinfo Update the following file to enable profiling counters required for Harmonizer and NuPoCo: src/inc/pmu.h The counter used for L3 hits may look like MEM_LOAD_UOPS_RETIRED.L3_HIT or MEM_LOAD_RETIRED.L3_HIT on a CooperLake machine, while it may appear only as MEM_LOAD_RETIRED.L3_HIT on a SapphireRapid system. Therefore, it is important to match and replace the counter names with thoseused on the testing machine. Details of the counter names and what each counter signify is mentioned in the file ’pmu.h’. C. Runtime Parameter Modifications For runtime specific modifications such as thresholds or daemon sleep duration (default 100ms), user can update the files listed below. These thresholds determine the percentage loss in IPS acceptable while applying DVFS and UFS. Additionally, the user can modify the number of cores to reduce at each step while exploring the core allocation using DCT. Detailed information for each of these thresholds is provided in the corresponding header files mentioned below. •Harmonizer runtime parameters: Harmonizer/src/Harmonizer_RTE/inc/harmonizer.h This file documents the purpose of each parameter used by the runtime. •NuPoCo runtime parameters: Harmonizer/src/NuPoCo_RTE/inc/nupoco.h Example: •The STEP SIZE controls the frequency level change. The STEP SIZE is set to 1 and if the available core frequencies are (2.5, 2.4, 2.3, ...)GHz. After measuring IPC at 2.4 GHz, the next exploration step will move to either 2.5 or 2.3 based on IPC improvement or degradation. •The DVFS and UFS thresholds define the permitted percentage loss in IPC that an application can tolerate to optimize energy. These thresholds can be tuned based on observed IPC response. •If reducing uncore frequency by one level causes IPC to drop within 1% at the optimal set frequency but overall EDP worsens, then reducing the threshold to 0.8% may improve performance. •Similarly, DCT thresholds can be tuned to control: –Number of cores deallocated in each step –IPC threshold before locking a benchmark to further reduce the cores To enable the debug information for Harmonizer, set DEBUG=-DDEBUG in the Makefile at Harmonizer/src/Harmonizer_RTE. The following is a sample debug output demonstrating the IPC variation and DVFS exploration behavior of a memory-bound benchmark : Initial Slab == 1 Curr_freq_CF: 0 Initial_IPC: 39.388493522941808, Curr_freq_CF: 7 Prev_IPC: 39.388493522941808, Curr_IPC: 56.925915246084180 Delta_IPC is: 44.52423 Prev_Feq: 7, Curr_Freq: 8 Prev_IPC: 56.925915246084180, Curr_IPC: 59.925740791927566 Delta_IPC is: 52.14022 Prev_Feq: 8, Curr_Freq: 9 Prev_IPC: 59.925740791927566, Curr_IPC: 58.274210159394684 Delta_IPC is: 47.94729 Prev_Feq: 9, Curr_Freq set as: 8 Prev_IPC: 58.274210159394684, Curr_IPC: 58.614200491987290 Delta_IPC is: 48.81047 4 Prev_Feq: 8, Curr_Freq: 9 Prev_IPC: 58.614200491987290, Curr_IPC: 57.846476544676591 Delta_IPC is: 46.86136 Prev_Feq: 9, Curr_Freq set as: 8 =========================================================== Optimal freq for bench_id: 1, set as: 9 after exploration rounds: 6 =========================================================== D. Build Runtimes Build Default Runtime (Default Linux): cd Harmonizer/src/Default make clean && make Build Harmonizer Runtime: cd Harmonizer/src/Harmonizer_RTE make clean && make Build NuPoCo Runtime: The code for NupoCo was implemented using the placement algorithm described the paper [1]. Since, the NuPoCo artifact code controls the concurrency using modified OpenMP as mentioned in the paper, we have implemented the placement algorithm separately. cd Harmonizer/src/NuPoCo_RTE make clean && make Build MAPPER Runtime: The code for Mapper [2] uses the open-source implementation provided by the authors, with modifications to ’mapper.h’ to adjust the thresholds as specified in the paper using [3], [4]. The code provided in Harmonizer artifact modifies these thresholds according to our system. We have also added the code to record energy. cd Harmonizer/src/Mapper_RTE cd tests make clean && make cd .. make clean && make E. Building Benchmarks Compile the benchmarks used for runtime evaluation: cd Harmonizer/benchmarks source /home/<USER>/Harmonizer/benchmarks/hclib/hclib-install/bin/hclib_setup_env.sh make clean && make Note: To change the problem size, the details are mentioned in Section VIA. IV. TESTING AND EVALUATION The test scripts are provided for both quad-socket and dual-socket systems; however, the evaluations presented in the paper were conducted on a quad-socket machine. In scenarios involving application mixes with more number of applications than the sockets, the placement and UFS policy may not perform as decsribed in paper, since sockets are shared among multiple applications, which can reduce the performance benefits in both cases. Navigate to the test scripts directory: cd Harmonizer/test_scripts A. Testing on a Quad-Socket/Dual-Socket Machine Choose the appropriate directory according to your system configuration: cd test_scripts/4-socket OR cd test_scripts/2-socket 5 B. Configuration Files Modify the following files to configure evaluation and motivational tests. To increase the number of iterations (default is 1), modify the num_iterations parameter in the configuration scripts. To run a specific mix (e.g., Mix5), modify the batch3 or batch4 in the ’test config’ to execute only the required mix. test_config.sh motivation_config.sh These files allow you to set: •Maximum and minimum core and uncore frequency (to set system to default setting before launching the experiments) •Log directory path •Number of iterations •Benchmark mixes to launch C. Running Motivational Tests Inside the motivational experiments directory, run: cd Harmonizer/test_scripts/4-socket/motivational_experiments ./motivation.sh To run each motivation independently: chmod +x *.sh ./motivation_metric.sh ./motivation_core.sh ./motivation_uncore.sh ./motivation_dct.sh The same steps apply for the 2-socket configuration. D. Running Evaluation Tests To launch all evaluation experiments at once: cd Harmonizer/test_scripts/4-socket/test_experiments ./test.sh To execute each configuration independently: chmod +x *.sh ./test_script_Default_Batch3.sh ./test_script_Default_Batch4.sh ./test_script_Harmonizer_Batch3.sh ./test_script_Harmonizer_Batch4.sh ./test_script_NuPoCo_Batch3.sh ./test_script_NuPoCo_Batch4.sh ./test_script_Mapper_Batch3.sh ./test_script_Mapper_Batch4.sh The same steps are to be followed for the 2-socket run. V. LOGS AND REPRODUCIBILITY The log files will be generated in the directory specified by the LOG_DIR variable in the configuration files. Each test script produces performance logs under the corresponding result directories. A. Generating Plots To produce the plots for the results presented in the paper, the original logs are also provided inside logs directory. To plot all the graphs: cd Harmonizer/plot_scripts cd reproduce_plots ./plot.sh The generated plots will be saved at: reproduced_pngs To produce the plots from the log files generated on the testing machine: 6 cd Harmonizer/plot_scripts cd create_plots ./plot.sh The generated plots will be saved at: created_pngs VI. CHANGING PROGRAM PARAMETERS A. Modifying Benchmark Parameters To maintain application behavior as per the system, benchmark parameters can be adjusted to control memory-boundness and time. Stream In the following file, modify the value of Nto increase or decrease the memory-bound nature of the application: Harmonizer/benchmarks/stream/stream.cpp HPCCG Modify the values of nx,ny, or nz in: Harmonizer/benchmarks/hpccg/main.cpp MiniFE For MiniFE, parameters nx,ny, and nz specify memory usage. These can be modified directly in the test scripts, for example inside: test_scripts/4-socket/test_experiments/test_script_Harmonizer_Batch3.sh change nx,ny, and nz at [MF]="../../../benchmarks/Kokkos/miniFE/kokkos/src/miniFE_jobqueue.x nx 256 ny 256 nz 512" Increasing these values increases memory usage and memory-bound behavior. Note: Whenever parameters are modified inside C/C++ files, recompile the benchmarks to apply the changes. REFERENCES [1] Y. Cho, C. A. C. Guzman, and B. Egger, “Maximizing system utilization via parallelism management for co-located parallel applications,” in PACT, 2018. [Online]. Available: https://doi.org/10.1145/3243176.3243199 [2] S. Srikanthan, S. Chakraborti, P. Ferro, and S. Dwarkadas, “MAPPER: Managing application performance via parallel efficiency regulation,” ACM TACO, 2022. [Online]. Available: https://doi.org/10.1145/3501767 [3] S. Srikanthan, S. Dwarkadas, and K. Shen, “Data sharing or resource contention: Toward performance transparency on multicore systems,” in USENIX ATC 15, 2015. [Online]. Available: https://www.usenix.org/conference/atc15/technical-session/presentation/srikanthan [4] ——, “Coherence stalls or latency tolerance: Informed CPU scheduling for socket and core sharing,” in USENIX ATC 16, 2016. [Online]. Available: https://www.usenix.org/conference/atc16/technical-sessions/presentation/srikanthan