Comprehensive Performance Analysis of Portals4 Communication Primitives on BXI Hardware

Comprehensive Performance Analysis of Portals4 Communication Primitives on BXI Hardware Niklas J. Bartelheimer Johannes Gutenberg University Mainz, Germany [email protected] Sarah M. Neuwirth Johannes Gutenberg University Mainz, Germany [email protected] Abstract—This paper presents a comprehensive performance analysis of the BullSequana eXascale Interconnect (BXI) using the Portals4 programming model. The main contributions include: (1) the design and implementation of PtlBench, a Portals4 microbenchmark suite that evaluates low-level features such as bandwidth, latency, cache effects, and triggered operations; (2) a detailed comparison of Portals4-compatible MPI implementations (OpenMPI, ParaStationMPI) and our custom Portals4 device for the PGAS library GPI-2, covering point-topoint, one-sided, and collective operations; and (3) applicationlevel analysis using the Himeno and SSCA1 benchmarks to assess the impact on different communication patterns. These results provide valuable information of BXI’s capabilities and limitations for real-world HPC workloads and communication models. Index Terms—MPI, GASPI, PGAS, BXI, Portals4, Performance Study, Benchmarking, Performance Analysis I. INTRODUCTION High-performance computing (HPC) systems rely on efficient interconnects to enable fast, scalable communication across thousands of nodes. As workloads become increasingly complex, from scientific simulations to AI-driven applications, network performance becomes a critical bottleneck in achieving exascale computing capabilities. Interconnect technologies must provide high bandwidth, low latency, and efficient synchronization while minimizing CPU overhead to optimize computating resources. One emerging solution is the BullSequana eXascale Interconnect (BXI) [1], designed to meet the requirements of recent European supercomputing initiatives such as the European Pilot for Exascale (EUPEX) [2]. BXI utilizes Portals4 [3], an event-driven low-level communication API that supports both the Partitioned Global Address Space (PGAS) model [4] and the Message Passing Interface (MPI). By offloading communication processing to the hardware, Portals4 aims to improve scalability and efficiency for various HPC workloads. Despite its promising design, the real-world performance characteristics of BXI and its compatibility with Portals4based libraries have not yet been sufficiently explored. In order to fully leverage BXI in future exascale systems, a deeper performance analysis is required to evaluate the strengths and limitations of BXI in different programming models. Traditional high-performance interconnects, including InfiniBand, Cray’s Slingshot and Fujitsu’s Tofu interconnect, each have distinct advantages, but also come with trade-offs. BXI, on the other hand, uses a connectionless, hardwareaccelerated model using Portals4, that tries to strike a balance between efficiency and flexibility. However, existing studies lack comprehensive evaluations comparing Portals4’s performance on BXI with other interconnect solutions. In addition, little research has explored how well Portals4-based MPI implementations (e.g., OpenMPI, ParaStationMPI) and PGAS runtimes (e.g., GPI-2) perform on BXI hardware. To address these gaps, this paper presents an in-depth performance study of Portals4 on BXI hardware with three main contributions: (1) the development of PtlBench, a dedicated Portals4 microbenchmark suite that enables detailed analysis of key features and different configurations; (2) a comparison of Portals4-compatible MPI implementations (OpenMPI and ParaStationMPI) and our custom Portals4 backend for the GPI-2 PGAS runtime, focusing on the efficiency of point-to-point, one-sided, and collective communication; and (3) application-level evaluations using the Himeno and SSCA1 benchmarks, which reflect realworld communication patterns, highlighting the practical implications of Portals4’s design on BXI for HPC use cases. II. BACKGROUND AND RELATED WORK This section provides the necessary background on Portals4, BXI, and the PGAS programming model. We also review related benchmark suites and prior performance studies. A. Portals4 Network API Portals [3] is a low-level network API designed for efficient and scalable programming. The latest version, Portals4, supports both PGAS and MPI models. To improve scalability, Portals4 uses a reliable, connectionless architecture that avoids the complexity of connection-oriented networks like InfiniBand (IB) [5] and simplifies connection setup and shutdown. Portals4 provides a comprehensive set of communication primitives, including one-sided put/get operations and matching semantics for efficient two-sided communication. Libraries such as Sandia-OpenSHMEM [6] and OpenMPI [7] leverage Portals, and a reference implementation [8] supports both IB networks and shared memory systems. B. BullSequana eXascale Interconnect V2 (BXI) Designed as a next-generation HPC interconnect, BXI integrates hardware offloading, adaptive routing [9] and high-radix switching to support large-scale parallel workloads. Unlike traditional interconnects, BXI is designed on the basis of Portals4 and enables low-latency, high-bandwidth communication without the need of explicit connection management. It consists of two primary hardware components: a network interface card (NIC) optimized for PGAS and MPI operations and a 48-port high-radix switch providing up to 100 Gbps per port and a total bidirectional bandwidth of 9600 Gbps. This flexibility allows BXI to scale up to 64,000 nodes, ensuring high efficiency and Quality of Service (QoS) via Reliability, Availability, and Serviceability (RAS) features [1]. C. PGAS Model and GASPI The Partitioned Global Address Space (PGAS) model [4] offers an alternative to MPI by providing a shared-memory abstraction for distributed nodes while allowing fine-grained control over memory locality. This model simplifies data access patterns and reduces synchronization costs, making it attractive for large-scale applications. PGAS is implemented in various forms, including extensions such as Unified Parallel C (UPC), Co-Array Fortran (CAF), and libraries such as OpenSHMEM [10], although there is no strict definition. The Global Address Space Programming Interface (GASPI) [11], [12] is a PGAS-based communication API that uses asynchronous, one-sided communication with explicit remote reads and writes. Unlike MPI, GASPI avoids bulk-synchronous operations, allowing for a more efficient overlapping of communication and computation. Its only implementation, GPI-2 [13], supports notified communication and dynamic memory allocation, making it an ideal candidate for Portals4-based systems. D. Related Work Two areas of related work are particularly relevant to this paper: (1) benchmarks and (2) performance studies. Benchmarks are generally categorized into microbenchmarks and application benchmarks [14]. There are many microbenchmark suites for parallel programming models. For example, the OSU Micro-Benchmarks (OMB) [15] suite is widely used to evaluate MPI performance and also includes PGAS microbenchmarks for OpenSHMEM, UPC, and UPC++. Other notable PGAS microbenchmark suites include the UPC Operations Microbenchmarking Suite (UOMS) [16] and the GASPI Benchmark Suite [17]. In addition, the OpenSHMEM Benchmark Suite (OBS) [10] offers microbenchmarks, application kernels and applications specifically for OpenSHMEM. Network interconnect performance studies are essential for understanding data transmission efficiency, which directly impacts computing system scalability and performance. For example, Kalia et al. [18] examine how NIC architecture affects RDMA-based system performance. De Sensi et al. [19] conduct an experimental analysis of the Slingshot interconnect to guide researchers and system administrators. In addition, Li et al. [20] present an overview of modern interconnects in data centers and HPC clusters, along with representative benchmarks. These studies form the basis for our research. However, Portals4-based interconnects, particularly BXI, remain underInitiator Target NI NI LE LE LE ME MEME EQ CT CT EQ CT CT Non-Matching NI Matching NI MD CT EQ 1 2 3 PtlPut Data Acknowledgement (optional) Network Fig. 1. Illustration of a PtlPut operation, including the data structures needed for matching and non-matching communication. explored, with limited studies assessing their impact on MPI and PGAS-based workloads. This paper aims to fill this gap. III. PTLBENCH: PORTALS4 MICROBENCHMARK SUITE A detailed evaluation of BXI performance requires low-level access to Portals4 communication primitives. While the BXI software stack includes Ptlperf, a Server-Client benchmark offering basic metrics, it is not suitable for comprehensive performance studies. The Sandia Portals4 reference implementation [8] provides three microbenchmarks: NETPipe, Message Rate, and Round-Trip Time (RTT), which respectively measure point-to-point throughput, simulate real-world message traffic (inspired by the Sandia MPI Microbenchmark Suite [21]), and implement a ping-pong latency test. These tools use the Process Management Interface (PMI) for process orchestration and support job schedulers like Slurm. However, these benchmarks offer limited Portals4 API coverage and do not collect per-iteration data needed for detailed statistical analysis. To overcome these limitations, we develop a new microbenchmark suite tailored specifically for Portals4. This section begins with an analysis of key Portals4 components via the execution path of a PtlPut operation. We then describe the architecture and functionality of our benchmark suite. A. Exemplary Breakdown of a Portals4 Data Transfer To illustrate data movement in Portals4, we use the PtlPut operation as an example. Figure 1schematically illustrates the process. It begins with the Initiator issuing a PtlPut request to the Target. On the Target side, the request is processed by either the Matching Network Interface (MNI) or Non-Matching Network Interface (NMNI). MNIs use Match Bits (MB) in the message to locate the appropriate memory address, while NMNIs ignore MBs entirely. Each Logical Network Interface (LNI), representing per-process access to the hardware, abstracts a Physical Network Interface (PNI), with up to four LNIs per PNI. On the Target side, memory access occurs through a Portal, identified by an index in the Portal Table (PT). Each portal maintains a linked list of Matching Entries (MEs) or List Entries (LEs), which define accessible memory regions. These regions may be registered, specifying exact buffer details, or unregistered, covering the entire virtual address space by setting the start field to NULL and length to PTL_SIZE_MAX. BXI supports up to 248 −1addresses for unregistered memory. Match bits direct MNIs to the correct ME (red diamonds in Fig. 1), while NMIs default to the first LE. Persistent LEs are used in simple cases; in dynamic scenarios, the first LE is unlinked after use to support sequential access. Each portal maintains both a Priority List (shown) and an Overflow List (not shown). To monitor activity, PT entries can be associated with an Event Queue (EQ) for Full Events (FEs), or with Event Counters (CTs) for more granular tracking. On the Initiator side, a Memory Descriptor (MD) defines the source buffer for put operations or the destination for get operations, and can reference either registered or unregistered memory. Offsets specified in Portals4 calls define the precise region involved in communication. Once the PtlPut completes, the Target may optionally return a PTL_EVENT_ACK, which is recorded by the Initiator via EQ or CT [22]. B. Design and Implementation of PtlBench Unlike traditional low-level network benchmarks that rely on a Server-Client communication model, our Portals4 microbenchmark suite, PtlBench, uses MPI for orchestration. MPI was selected over PMI or PMIx due to its portability, ease of use, and rich feature set. Its compatibility with Portals4enabled NICs enables seamless communication between benchmark processes, while collective operations like barriers provide efficient synchronization. PtlBench includes five distinct microbenchmarks, each designed to evaluate a specific aspect of the Portals4 API. The following sections describe their implementation, key features, and intended use cases. 1) ptl bench: The ptl bench benchmark measures bandwidth and latency for PtlPut and PtlGet operations, modeled after OMB’s one-sided communication benchmarks. It supports both non-matching or matching NIs, and allows evaluation of event handling via either FEs or CTs. To assess cache effects, a custom cache saturation function is used to simulate cold-cache conditions by populating memory with random values. This enables controlled comparisons between cold and hot cache scenarios. On the target side, communication uses either a persistent LE or ME, where “persistent” denotes reuse of the same list entry throughout the benchmark. 2) ptl me none persistent: The ptl me none persistent benchmark measures the latency and bandwidth of Portals4’s matching network interface, focusing on communication patterns representative of MPI’s send-receive semantics. In contrast to ptl_bench, which relies on persistent list entries regardless of matching behavior, this benchmark emulates more dynamic, per-message matching scenarios. It uses two portal indices: one for the command channel and another for data transfer. On the target side, multiple list entries are created, each associated with a distinct memory buffer as specified by the window_size parameter. The initiator begins transmission after receiving a notification via the command channel. Messages are matched to list entries using configured match bits, and upon successful transmission, indicated by full events such as PTL_EVENT_PUT or PTL_EVENT_GET, the corresponding list entries are removed. This process repeats for each iteration. 3) ptl memory bench: This benchmark evaluates the BXI NIC’s virtual-to-physical address translation performance by measuring page fault latency in both local and remote access scenarios. It supports both one-sided and ping-pong communication patterns, using PtlPut and PtlGet operations. Memory is allocated using mmap to create page-sized virtual segments, relying on deferred physical allocation. This enables configuration of whether page faults are handled by the NIC or the host. Placement follows a first-touch policy, where initial access triggers physical allocation. Pages are classified as Hot (pre-allocated) or Cold (unallocated). Messages are constraint to 4 kB or less, with transfers targeting random offsets within each page. The ping-pong pattern mirrors the one-sided setup, enabling comparative analysis of translation overhead. 4) ptl ping pong: This benchmark measures Round-Trip Time (RTT) using a ping-pong communication scheme. It highlights Portals4’s Triggered Operations, in which standard calls like PtlPut and PtlGet are deferred until a CT reaches a defined threshold, allowing the NIC to execute the operation autonomously, bypassing CPU involvement. The benchmark compares RTT between standard PtlPut and its triggered variant PtlTriggeredPut, and also quantifies the additional setup latency introduced by the trigger mechanism. 5) ptl get ni props: As a hardware implementation of Portals4, BXI imposes inherent limitations on available resources. Portals4 allows customization of these limits by providing a pointer to a plt_ni_limits_t structure during NI creation. This structure can be initialized with INT_MAX,LONG_MAX, or zero before proceeding with NI initialization. The actual resource limits imposed by the Portals4 implementation are then retrieved from a separate structure after initialization. IV. COMPREHENSIVE PERFORMANCE EVALUATION To evaluate the real-world viability of Portals4 on BXI, we conducted a comprehensive, multi-phase performance study. This section presents our methodology and results, spanning low-level microbenchmarking with PtlBench, comparative analysis of MPI, specifically OpenMPI (OMPI) and ParaStationMPI (PSMPI), both running over Portals4, and the PGAS runtime GPI-2, as well as application-level evaluations using two representative HPC workloads. A. Test System Setup The evaluation was performed on the DEEP System [23], an MSA [24] prototype featuring a range of computing modules. Our research focused on the BXI Module (BM), which comprises four nodes, each equipped with an Intel Xeon Gold 5122 CPU, 48 GB of RAM, and a 100 Gbit BXIv2 interconnect. At the time of this writing, this was the only publicly available test system equipped with BXI. The nodes operate on Rocky Linux 8.10 Green Obsidian. Communication over the BXI interface utilized the pre-installed Portals4 version 2.1.9 by Eviden/Bull. 1 16 1024 65536 4194304 0 4000 8000 12000 0 4000 8000 12000 Message Size [B] Bandwidth [MB/s] PtlGet PtlPut Limit: 100Gbs Estimated (a) Bandwidth results for PtlPut and PtlGet. 1.0 1.1 1.2 1.3 1.4 1 16 1024 65536 4194304 Message Size in [B] Hot Cache Speedup PtlGet PtlPut (b) Hot Cache Speedup usage for PtlPut and PtlGet. PtlGet PtlPut 1 16 1024 65536 4194304 0 3000 6000 9000 0 3000 6000 9000 Message Size [B] Bandwidth [MB/s] ME−none−persistent ME−persistent (c) Bandwidth for persistent & non-persistent MEs. 2.4 2.5 2.6 PtlGet PtlPut Latency [us] Event Counter Full Event (d) Latency results for PtlPut & PtlGet for different event tracking types (message size: 1024 bytes). 0 1 2 3 64 1024 Message Size [B] Round Trip Time (RTT) [us] PtlPut PtlTiggeredPut + Setup Time PtlTriggeredPut (e) Round-Trip Time for PtlPut and PtlTriggeredPut (message sizes: 64 bytes and 1024 bytes). 0 30 60 90 cold−cold cold−hot hot−cold hot−hot Page State [local − remote] Latency [us] PtlGet PtlPut (f) Latency results for PtlPut and PtlGet targeting Hot and Cold pages. Fig. 2. Overview of performance results measured with PtlBench. B. Benchmark Methodology Our performance study follows a hierarchical, multi-stage methodology to systematically evaluate the performance of the BXI under the Portals4 communication model. The methodology is structured in three phases. 1) Portals4 Microbenchmarking on BXI with PtlBench: The first phase focuses on low-level communication benchmarking using our PtlBench suite1(see Section III). Experiments systematically varied message sizes (1 byte to 4 MB), cache states (hot vs. cold), communication types (PtlPut vs. PtlTriggeredPut), and memory allocation (pre-allocated vs. on-demand) to isolate key performance characteristics of Portals4 on BXI. Each benchmark was executed 100 times (referred to as experiments), with each experiment consisting of 10 warm-up and 1000 timed iterations. The per-experiment median was calculated from iteration results, and the overall median was derived from the 100 experiment medians. For benchmarks involving multiple message sizes, this procedure was applied independently to each size. Bandwidth measurements were performed by sending 64 messages per iteration within a message window. All benchmarks allocated a single memory buffer per message size, aligned to page boundaries using posix_memalign, or mmap in the case of ptl_memory_bench. As the test system featured a single CPU with a single NUMA domain, no process binding was applied. 2) Comparative Evaluation of MPI and PGAS Communication Models: The second phase evaluates the communication efficiency of Portals4-based MPI implementations (OpenMPI 5.0.4, ParaStationMPI 5.9.2) and the GPI-2 PGAS runtime, using our custom GASPI Benchmark Suite (GBS) [17] inspired by the OSU Micro-Benchmarks (OMB) and fine-tuned with 1https://github.com/NeuwirthLab/PtlBench the insights from Sections IV-D and IV-C. The analysis covers point-to-point (two-sided and one-sided), one-sided RMA synchronization, and Allreduce collective operation, with performance measured across message sizes from 1 byte to 4 MB and multiple process counts to capture scalability trends. All libraries are compiled with GCC 12.3.0, and MPI variants are built using their respective compiler wrappers. GPI-2, which lacks native Portals4 support, is extended with a custom Portals4 backend1developed for this study [25]. Each benchmark scenario consists of 100 experiments with at least 1,000 inner iterations per run, enabling statistically robust comparisons. Median results and boxplots are used to present findings. 3) Application-Level Performance Evaluation: Phase three connects low-level benchmark results with real-world HPC workloads by evaluating Portals4’s impact on BXI through two representative applications: the Himeno and SSCA1 benchmarks. The Himeno benchmark, based on the Jacobi method for solving Poisson equations, features a highly parallelizable stencil computation with a high communication-tocomputation ratio, ideal for assessing structured point-to-point messaging efficiency. In contrast, SSCA1 [10] implements the Smith-Waterman sequence alignment algorithm with dynamic programming and gap scoring, stressing fine-grained memory access and frequent small-message communication, issuing multiple Puts and Gets per iteration. Both benchmarks are executed using OpenMPI, ParaStationMPI, and our custom Portals4-enabled GPI-2 backend, under twoand four-node configurations. Metrics such as execution time, communication overhead, and synchronization efficiency are collected under varying process decompositions. C. Portals4 Microbenchmarking with PtlBench We evaluated Portals4’s performance on BXI using a series of experiments. Bandwidth measurements for PtlPut and TABLE I LIMITS OF THE BXIV2HARDWARE RESOURCES. Parameter Value Parameter Value Parameter Value max entries 4080 max unexpected headers 16319 max mds 1024 max eqs 960 max cts 1024 max pt index 255 max iovecs 0 max list size 16582 max triggered ops 7167 max msg size 67108864 max atomic size 1024 max fetch atomic size 64 max waw ordered size 2048 max war ordered size 64 max volatile size 64 PtlGet with event counters are shown in Figure 2(a). While BXIv2 advertises a theoretical maximum bandwidth of 12.5 GB/s (solid black line), our results aligned more closely with the 11 GB/s saturation point predicted by Derradji et al. [1] (dashed line). Figure 2(b) illustrates latency improvements under hot cache conditions relative to cold cache scenarios. A speed-up of 1.0 indicates no improvement. For small messages (<512 bytes), PtlPut latency improves by over 30%, with messages <64 bytes showing gains exceeding 20%. The dip at 64 bytes corresponds to the NIC’s transition from Programmed I/O (PIO) to Direct Memory Access (DMA). Cache effects significantly impact performance: cold cache conditions introduce higher latency due to frequent cache misses in the Portals4 library, especially for data structures like memory descriptors. These effects reflect real-world scenarios, where such structures are often evicted during compute-intensive phases. In contrast, hot cache conditions reduce latency, except for larger messages that better tolerate cache delays. Using the ptl_me_none_persistent benchmark, we analyzed the impact of tag-matching semantics on point-topoint communication. Figure 2(c) compares bandwidth for persistent versus non-persistent matching list entries. The persistent configuration (green curve) approaches saturation quickly for both PtlGet and PtlPut, while the nonpersistent setup (orange curve) incurs additional overhead due to a rendezvous-like protocol, in which the target pre-posts 64 entries (the window size) before notifying the initiator, resulting in a noticeably flatter slope. Portals4 supports two mechanisms for tracking network events: Full Events (FEs) and Event Counters (CTs). Figure 2(d) shows latency comparisons for PtlGet and PtlPut. Boxplots reveal a right-skewed distribution, with most latencies clustered near the lower quartile. Delays, likely due to congestion, retransmissions, or NIC queue saturation, affect only a small subset of messages. The median latency difference between CTs and FEs is roughly 0.05 µs for both operations. FEe exhibit more outliers (black dots) due to interruptdriven handling with PtlEQWait/PtlEQPoll, whereas CTs rely on busy-waiting with PtlCTWait/PtlCTPoll, which may account for reduced variance. The next experiment evaluates the achievable Round-Trip Time (RTT) for ping-pong communication using PtlPut and PtlTriggeredPut. Unlike PtlPut, which executes immediately, PtlTriggeredPut introduces a setup phase to arm the trigger at a specified CT value, adding additional latency. Figure 2(e) shows RTT results for 64-byte (orange) and 1024-byte (turquoise) messages, comparing PtlTriggeredPut with and without setup overhead. For small messages, PtlTriggeredPut achieves a lower RTT than PtlPut, though this advantage diminishes slightly when setup latency is included. For larger messages, the setup overhead becomes negligible, and PtlTriggeredPut consistently outperforms PtlPut. We also analyzed the impact of page faults managed by the BXI NIC. Figure 2(f) depicts the latency for local and remote page faults in PtlPut and PtlGet, distinguishing hot (pre-allocated) and cold (unallocated) pages. For PtlPut, remote page state has little impact unless the local page is hot. Cold local pages increase latency due to NIC-triggered faults, indicating that performance is primarily governed by Portals4’s memory descriptor state. In contrast, PtlGet is more sensitive to both local and remote page states. In the cold-cold case, its latency nearly doubles that of PtlPut, reflecting the need to resolve both remote portal and local memory descriptor information. D. BXI Limitations and Properties The following outlines BXI’s limitations and features as observed during our evaluation. As a hardware implementation of Portals4, BXI imposes inherent resource constraints. Portals4 permits querying and customizing these limits through the plt_ni_limits_t structure during NI creation. We developed a test program that initializes this structure with INT_MAX,LONG_MAX, or zero, then retrieves the actual values after initialization. The results are shown in Table I. Key limits include allocable resources such as EQs and CTs. Some parameters indicate extended functionality, such as max_iovec, which defines support for IOVECs (similar to Scatter-Gather Lists). Although BXI sets this value to zero, implying no support, code inspection revealed a partial, non-Portals4-compliant implementation. The max_msg_size limits messages to 64 MiB, relatively small compared to InfiniBand’s 2 GiB. Portals4 Triggered Operations (TOs), which enable deferred communication based on counter thresholds, are partially supported on BXI, allowing up to 7,167 outstanding operations (max_triggered_ops). The max_volatile_size parameter specifies the maximum size for put or atomic operations using the PTL_MD_VOLATILE flag, enabling PIO for reduced latency in small transfers, as discussed in Section IV. These and other capabilities are grouped into three feature categories defined by Portals4 in the NI limits structure. PTL_TARGET_BIND_INACCESSIBLE indicates that not all memory described by LEs, MEs, or MDs needs to be allo- 0 3000 6000 9000 1 16 1024 65536 4194304 Message Size [B] Bandwidth [MB/s] OMPI PSMPI (a) MPI point-to-point bandwidth results. 0 3000 6000 9000 1 16 1024 65536 4194304 Message Size [B] Bandwidth [MB/s] GPI−2 read GPI−2 write OMPI get OMPI put PSMPI get PSMPI put (b) Put / Get bandwidth results. 3 10 30 100 300 1 16 1024 65536 4194304 Message Size [B] Latency [us] GPI−2 read GPI−2 write OMPI get OMPI put PSMPI get PSMPI put (c) Put / Get latency results. 10 20 30 1 2 4 8 16 32 64 128 Number of Reduction Elements Latency [us] GPI−2−p#2 GPI−2−p#4 OMPI−p#2 OMPI−p#4 PSMPI−p#2 PSMPI−p#4 (d) Allreduce latency results using one process per node using two and four nodes. Process Count: 2 Node Count: 2 6000 6250 6500 6750 7000 MFLOPS Process Count: 4 Node Count: 4 13000 13200 13400 13600 GPI−2 OMPI PSMPI MFLOPS (e) Performance in MFLOPS for the Himeno benchmark using two different domain decompositions. Process Count: 2 Node Count: 2 60 80 100 120 Runime [s] Process Count: 4 Node Count: 4 50 75 100 125 GPI−2 OMPI PSMPI Runtime [s] (f) Runtime results for SSCA1 for different 2 and 4 processes using one process per node. Fig. 3. OpenMPI, ParaStationMPI and GPI-2 performance results. cated or accessible by the application. BXI supports this via a hardware-based virtual-to-physical address translation cache. PTL_TOTAL_DATA_ORDERING defines the ordering behavior for messages and enforces strict ordering for short messages, as defined by max_waw_ordered_size and max_war_ordered_size. BXI implements this behavior internally, disregarding user-specified limits. PTL_COHERENT_ATOMICS, which guarantees coherence between Portals4 and processor atomic operations, is not supported by the current BXI hardware. Our source code analysis revealed that functions like PtlBundleStart and PtlBundleEnd are implemented as no-ops. These functions, designed to optimize memory synchronization, particularly for high-rate, strided remote memory access, are critical for maintaining performance in demanding communication patterns [22]. E. MPI Point-to-Point Communication Point-to-point communication is fundamental to many HPC applications, making its performance a key focus in our evaluation. We measured the bandwidth for both OMPI and PSMPI, with results shown in the top half of Figure 3(a). The bandwidth curves highlight the protocol switch between the Eager and Rendezvous protocols under default settings. For OMPI, this switch occurs at message sizes above 16,384 bytes, while PSMPI transitions at 32,768 bytes. The Eager protocol sends messages immediately, even if the corresponding receive has not yet been posted. This approach favors small messages, where the receiver is likely to be ready and can process data promptly. In contrast, the Rendezvous protocol, designed for larger messages, uses a handshake mechanism: the sender first notifies the receiver, which then allocates a buffer and signals readiness before data transfer begins. This added coordination introduces latency, leading to noticeable performance drops for both OMPI and PSMPI at the transition point. It should be noted that PSMPI is optimized for MSAaware communication. As such, its performance reflects trade-offs aligned with the restrictions and requirements of the Modular Supercomputing Architecture (MSA) [24]. F. GPI-2 and MPI One-Sided Communication Since GASPI, and by extension GPI-2, follows the PGAS communication model, we evaluate its performance using our Portals4 device and compare it to the one-sided communication routines provided by OMPI and PSMPI. While the PGAS model is designed for a global address space, MPI-based one-sided communication relies on Access Epochs and requires explicit synchronization. For this evaluation, we use the Passive Target Communication model with the MPI_Win_Fence access pattern. This method places a fence at the beginning of the communication phase to mark the start of an access epoch and another at the end to ensure that all memory operations are completed both locally and remotely before continuing execution. Figure 3(b) illustrates the bandwidth results for Put and Get operations using one-sided communication. As expected, GPI2 consistently outperforms both MPI implementations, benefiting from its relaxed memory model and efficient asynchronous semantics. OMPI performs competitively, approaching GPI-2’s bandwidth in many cases, while PSMPI falls short, failing to achieve comparable throughput. Notably, PSMPI’s bandwidth progression for both Put and Get operations resembles that of point-to-point communication, including a visible protocol switch. This behavior suggests that PSMPI internally emulates one-sided operations using point-to-point mechanisms, which may contribute to its reduced performance in this context. Flush Flush_local 1 16 1024 65536 4194304 1 16 1024 65536 4194304 1 10 100 Message Size [B] Latency [us] OMPI get OMPI put PSMPI get PSMPI put Fig. 4. MPI one-sided passive target communication latency using flush (left) and flush_local (right). Figure 3(c) shows the corresponding latency results. The results follow a similar trend as the bandwidth data, with OMPI and GPI-2 achieving the lowest latencies. For small message sizes, GPI-2 outperforms PSMPI, by a factor of approximately 3.5×for Get operations and 2×for Put operations. OMPI’s latency profile reflects the absence of PTL_MD_VOLATILE usage, which would otherwise reduce latency for small messages. In GPI-2, the latency data indicates a switch from PIO to DMA transfers for Put operations at message sizes above 256 bytes. For Get operations, this switch happens earlier, at sizes exceeding 64 bytes. These thresholds align with the max_volatile_size NI limit, suggesting that PIO is used only below this limit. We also observed inconsistencies in how MPI implementations interpret the specification regarding flush behavior on MPI memory windows. Figure 4shows the latency for MPI-based Put and Get operations using MPI Win flush (left) and MPI Win flush local (right) for synchronization. According to the MPI specification, MPI Win flush guarantees that all RMA operations targeting a specific rank’s window are completed both locally and remotely, while MPI_Win_flush_local only ensures local completion. The latency results confirm this distinction. OMPI and PSMPI perform similarly for MPI_Win_flush, consistent with the earlier latency trends. However, for MPI_Win_flush_local, PSMPI exhibits a substantial improvement, whereas OMPI’s performance remains unchanged. Source code analysis revealed that OMPI’s Portals4 module enforces remote completion even for local flushes, effectively treating both flush modes the same. To verify whether this behavior is specific to PSMPI, we repeated the experiments using vanilla MPICH v4.1.0 over InfiniBand. The results matched those of PSMPI, confirming that this behavior is inherited from the underlying MPICH implementation. G. Collective Operations Collective operations facilitate communication and synchronization among groups of processes in parallel programs. The GASPI specification defines two collective operations: Barrier and Allreduce. A Barrier ensures that all participating processes reach a synchronization point before continuing, while Allreduce performs a global reduction (e.g., computing the maximum value across processes) and distributes the result to all participants. Figure 3(d) shows the latency results for the Allreduce operation with reduction elements ranging from 1 to 128. The limit of 128 reduction elements was chosen on the premise that GPI-2 in its default configuration only supports 255 reduction elements. 128 is the largest power of two that is contained in this intervall. The benchmark was conducted using one process per node on both two-node and four-node configurations. PSMPI achieves the best performance for reduction elements up to two, after which latency increases but remains relatively stable up to 128 elements, consistent across both configurations. OMPI demonstrates the best overall performance, maintaining near-constant latency across all tested counts. Its Portals4 module implements the Allreduce algorithm using triggered operations, a feature whose advantages were highlighted in Section IV-C. GPI-2, except for a single reduction element, shows the highest latency, with a linear increase as the number of reduction elements grows. This is because the Allreduce operation is not natively implemented within the Portals4 device abstraction but is built in the upper layer of the GPI-2 software stack. This highlights potential opportunities for optimization in the selection and implementation of the Allreduce algorithm within GPI-2. H. Application-level Performance Results 1) Himeno Benchmark: For our evaluation, we used the MPI-based static memory allocation implementation, the only publicly available C version. We modified it to support dynamic memory allocation, added OpenMP pragmas for multithreading, and integrated GPI-2 communication primitives. The MPI version utilizes vector data types, but OMPI and PSMPI could not effectively leverage these over BXI due to the Portals4 IOVEC implementation’s non-conformance with the specification. Consequently, we replaced vector types with a sequence of MPI_Isend and MPI_Irecv calls. Additionally, the original benchmark is designed to run for a fixed duration (e.g., one minute); we modified it to execute a fixed number of iterations for reproducibility. Figure 3(e) shows performance in Mega Flops per Second (MFLOPS) for a 3D domain of size 256x256x512. The upper plot reports results for a decomposition along the X-dimension, with two processes (one per node). The GPI-2 implementation achieves the highest performance, peaking at around 9,950 MFLOPS, followed closely by OMPI, which is approximately 100 MFLOPS lower based on the median. PSMPI’s performance is slightly below that of OMPI. The lower plot shows results for a domain further partitioned along the Y-dimension, effectively doubling the number of processes and nodes. The ranking of the communication libraries remains unchanged, with performance nearly doubling as expected. This is attributed to GPI-2’s efficient notified one-sided communication. The performance gap between OMPI and PSMPI is discussed further in Section IV-E. 2) Scalable Synthetic Compact Applications 1 (SSCA1): We extended the SSCA1 codebase, which uses C macros to abstract put and get semantics across communication libraries, to support GPI-2. Unlike MPI’s one-sided routines, which allow source data to reside in stack-allocated memory, GPI-2 requires data to be placed in a designated memory segment. To comply with this constraint, our GPI-2 implementation uses a dedicated message buffer, requiring an explicit copy of each data item before transmission. Figure 3(f) shows the runtime results for the SSCA1 benchmark executed with two and four processes on two and four nodes, respectively. The results indicate that OMPI consistently achieves the lowest runtimes, while both GPI-2 and PSMPI exceed 120 seconds in the four-node configuration. These results reaffirm the performance gap between OMPI and PSMPI discussed in Section IV-F. GPI-2’s comparatively weaker performance is likely due to its message buffering strategy, which introduces additional overhead. Given SSCA1’s reliance on frequent small put and get operations, the cost of copying data into message buffers significantly impacts overall runtime. V. CONCLUSION AND OUTLOOK This work introduced a Portals4 microbenchmark suite to analyze BXI performance. We compared the performance of Portals4-compatible MPI implementations with our custom Portals4-based backend for the GPI-2 communication library. For point-to-point communication, OMPI and PSMPI performed well, with OMPI showing a slight edge. In one-sided communication, GPI-2 outperformed both MPI implementations, though OMPI came close, while PSMPI’s reliance on point-to-point communication introduced significant overhead. For collective operations, GPI-2 showed strong Barrier performance but weaker Allreduce results. PSMPI excelled in Allreduce for small reduction element counts, surpassing OMPI. Overall, Portals4 over BXI demonstrates promising performance, with simpler implementation compared to InfiniBand. PSMPI, as the only MSA-aware MPI implementation, offers additional features but could benefit from improved OSC design for better resource utilization on homogeneous systems. Future work will include extending the microbenchmark suite to cover all Portals4 functionalities. For GPI-2, we plan to explore accelerator-to-accelerator communication over Portals4 and develop more efficient Allreduce algorithms. ACKNOWLEDGMENT This research is supported by EUPEX, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under GA No 101033975. The JU receives support from the European Union’s Horizon 2020 research and innovation program, France, Germany, Italy, Greece, United Kingdom, Czech Republic, and Croatia. The authors would also like to thank Gr´ egoire Pichon from Eviden for his support and valuable comments. REFERENCES [1] S. Derradji, T. Palfer-Sollier, J.-P. Panziera, A. Poudes, and F. W. Atos, “The bxi interconnect architecture,” in 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects, pp. 18–25, IEEE, 2015. [2] European Pilot for Exascale. https://eupex.eu/about-the-project/. accessed 08-August-2024. [3] Sandia National Laboratories, “Portals 4.0.” https://www.sandia.gov/ portals/portals-4-0/. Online; accessed: 2024-07-19. [4] M. De Wael, S. Marr, B. De Fraine, T. Van Cutsem, and W. De Meuter, “Partitioned Global Address Space Languages,” ACM Comput. Surv., vol. 47, May 2015. [5] InfiniBand Trade Association, “InfiniBand.” https://www.infinibandta. org. Online; accessed: 2024-08-02. [6] B. W. Barrett, S. Smith, J. Dinan, K. Seager, and R. E. Grant, “Sandia OpenSHMEM.” https://www.osti.gov/servlets/purl/1312730, 2016. [7] E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. H. Castain, D. J. Daniel, R. L. Graham, and T. S. Woodall, “Open MPI: Goals, concept, and design of a next generation MPI implementation,” in Proceedings, 11th European PVM/MPI Users’ Group Meeting, 2004. [8] Sandia National Laboratories, “Portals4 Reference Implementation.” https://github.com/sandialabs/portals4. GitHub repository. [9] P. Vign´ eras and J.-N. Quintin, “The bxi routing architecture for exascale supercomputer,” The Journal of Supercomputing, vol. 72, no. 12, pp. 4418–4437, 2016. [10] T. Naughton, F. Aderholdt, M. Baker, S. Pophale, M. G. Venkata, and N. Imam, “Oak Ridge OpenSHMEM Benchmark Suite,” in Workshop on OpenSHMEM and Related Technologies, pp. 202–216, Springer, 2018. [11] D. Gr¨ unewald and C. Simmendinger, “The GASPI API specification and its implementation GPI 2.0,” in 7th International Conference on PGAS Programming Models, vol. 243, p. 52, 2013. [12] C. Simmendinger, M. Rahn, and D. Gruenewald, “The GASPI API: A Failure Tolerant PGAS API for Asynchronous Dataflow on Heterogeneous Architectures,” in Sustained Simulation Performance 2014 (M. M. Resch, W. Bez, E. Focht, H. Kobayashi, and N. Patel, eds.), (Cham), pp. 17–32, Springer International Publishing, 2015. [13] Fraunhofer ITWM, “GPI-2: Global Address Programming Interface 2.” https://github.com/cc-hpc-itwm/GPI-2. accessed 24-August-2022. [14] S. Neuwirth and A. K. Paul, “Parallel I/O Evaluation Techniques and Emerging HPC Workloads: A Perspective,” in 2021 IEEE International Conference on Cluster Computing (CLUSTER), pp. 671–679, 2021. [15] D. K. Panda, “OSU Micro-Benchmarks (OMB).” https://mvapich.cse. ohio-state.edu/benchmarks/. Online; accessed 04-August-2024. [16] D. A. Mall´ on, Design of Scalable PGAS Collectives for NUMA and Manycore Systems. PhD thesis, University of A Coru˜ na, Spain, 2014. [17] N. Bartelheimer and S. Neuwirth, “Toward Reproducible Benchmarking of PGAS and MPI Communication Schemes,” in IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS), 2023. [18] A. Kalia, M. Kaminsky, and D. G. Andersen, “Design guidelines for high performance rdma systems,” in 2016 USENIX Annual Technical Conference (USENIX ATC 16), pp. 437–450, 2016. [19] D. De Sensi, S. Di Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler, “An in-depth analysis of the slingshot interconnect,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14, IEEE, 2020. [20] Y. Li, H. Qi, G. Lu, F. Jin, Y. Guo, and X. Lu, “Understanding hot interconnects with an extensive benchmark survey,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 2, no. 3, p. 100074, 2022. [21] Sandia National Laboratories, “Sandia MPI Microbenchmark Suite.” https://github.com/sandialabs/SMB. GitHub repository. [22] B. W. Barrett, R. Brightwell, R. E. Grant, W. Schonbein, S. Hemmert, K. Pedretti, K. Underwood, R. Riesen, M. Barbe, L. H. S. Filho, A. Ratchov, and A. B. Maccabe, “The Portals 4.3 Network Programming Interface,” Tech. Rep. SAND2022-8810, SNL, 2022. [23] Forschungszentrum Juelich, “DEEP Testcluster – System Overview.” https://deeptrac.zam.kfa-juelich.de:8443/trac/wiki/Public/User Guide/ System overview. accessed 10-18-2023. [24] S. Neuwirth, “Modular Supercomputing and its Role in Europe’s Exascale Computing Strategy,” PoS, vol. LATTICE2022, p. 245, 2023. [25] N. Bartelheimer and S. Neuwirth, “Leveraging Portals4 Microbenchmarks to Enhance GASPI Performance on BXI Networks,” in 2024 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops), pp. 176–177, 2024.