Scalable RDMA performance in PGAS languages

Scalable RDMA performance in PGAS languages Montse Farreras†, George Alm´ asi‡, C˘ alin Cas¸caval‡, Toni Cortes† †Department of Computer Architecture, Universitat Politecnica de Catalunya Barcelona Supercomputing Center, Barcelona, Spain {mfarrera, toni}@ac.upc.es ‡IBM T.J. Watson Research Center, Yorktown Heights, NY {gheorghe, cascaval}@us.ibm.com Abstract Partitioned Global Address Space (PGAS) languages provide a unique programming model that can span shared-memory multiprocessor (SMP) architectures, distributed memory machines, or cluster of SMPs. Users can program large scale machines with easy-to-use, shared memory paradigms. In order to exploit large scale machines efficiently, PGAS language implementations and their runtime system must be designed for scalability and performance. The IBM XLUPC compiler and runtime system provide a scalable design through the use of the Shared Variable Directory (SVD). The SVD stores meta-information needed to access shared data. It is dereferenced, in the worst case, for every shared memory access, thus exposing a potential performance problem. In this paper we present a cache of remote addresses as an optimization that will reduce the SVD access overhead and allow the exploitation of native (remote) direct memory accesses. It results in a significant performance improvement while maintaining the run-time portability and scalability. 1. Introduction Parallel programming for both multicores and large scale parallel machines is becoming evermore challenging; adequate programming tools offering both ease of programming and productivity are essential. However, while the productivity of developing applications for these machines is important, the users of massively parallel machines are expecting the same level of performance as obtained by manual tuning of MPI applications. Partitioned Global Address Space (PGAS) languages, such as UPC, Co-Array Fortran, and Titanium, extend existing languages (C, Fortran and Java, respectively) with constructs to express parallelism and data distributions. These languages provide a simpler, shared memory-like programming model, with control over the data layout. The performance of these languages relies on two main factors: (i) the programmer to tune for locality by specifying appropriate data layouts; and (ii) the compiler and runtime system to efficiently implement the locality directives. The discussion in this paper will focus on the UPC language, however, the optimizations apply to any of the PGAS languages. The memory model of UPC follows the PGAS model, with each thread having access to a private, a shared local, and a shared global section of memory. Threads have exclusive, low latency, access to the private section of memory. Typically the latency to access the shared local section is lower than the latency to access the shared global section. The UPC memory and threading model can be mapped to either distributed memory machines, shared memory machines or hybrid (clusters of shared memory machines, such as MareNostrum [3]). In this paper we describe an optimization in the IBM XLUPC runtime system to exploit hardware features, such as RDMA, to improve application performance and maintain a scalable design. The IBM XLUPC runtime system uses a Shared Variable Directory (see Section 2) to provide a scalable infrastructure that has been demonstrated to scale to hundreds of thousands of threads on the BlueGene/L computer [8]. However, in this scalable design, nodes keep only local information. To exploit RDMA, we implemented a cache of remote addresses on two different platforms: the MareNostrum [3] supercomputer, and a cluster of Power5 SMP machines. We measured the performance of this optimization on a set of benchmarks, and we demonstrate a reduction in execution time of up to 40% and 30%, on each of the platforms, respectively. The effects of using RDMA are the largest for very short messages; these are the kinds of performance improvements that conventional two-sided messaging systems cannot achieve because of design limitations (e.g. MPI point to point relies on message matching on the receiver, which rules out RDMA transfers). This paper makes the following main contributions: •describes the Shared Variable Directory (SVD), which is crucial to the scalability of the UPC RunTime System (RTS). •presents a runtime optimization that improves its Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. performance while being scalable and portable. •introduces a UPC parallel implementation of a subset of the DIS Stressmark Suite and evaluates these benchmarks on our system. The rest of the paper is organized as follows. Section 2 describes XLUPC RTS. Section 3 describes the runtime optimization implemented to speed up communication. Section 4 outlines the experiments and the results obtained from running the benchmarks on MareNostrum supercomputer and the AIX cluster. The related work is presented in Section 5 and finally conclusions and future work are discussed in Section 6. 2. The IBM XLUPC runtime The runtime component of the IBM XLUPC compiler has multiple roles: it spawns and collects UPC threads, implements accesses to shared data, performs pointer arithmetic on pointers to shared objects and implements all the UPC intrinsic function calls (such as upc_phaseof,upc_barrier and upc_memget). The XLUPC runtime defines an external API that is used by the UPC compiler when generating code. The overall architecture of the XLUPC runtime (Figure 1) is similar to that of GASNet [6]. It provides a platform-independent interface that can be implemented on top of a variety of architectures, SMP or distributed. The XLUPC runtime is designed for a hybrid mode of operation on clusters of SMP nodes: UPC threads communicate through shared memory when available, and send messages through one of several available transports where necessary. The Pthreads library is used to spawn multiple UPC threads on systems with SMP nodes. Implemented messaging methods include TCP/IP sockets, LAPI [23], Myrinet/GM transport [19] and the BlueGene/L messaging framework [1]. In this paper we focus on optimizing the hybrid and distributed-memory implementations of the runtime. 2.1. The Shared Variable Directory Shared objects (i.e. data structures accessible from all UPC threads) form the basis of the UPC language. The XLUPC runtime recognizes several kinds of shared objects: shared scalars (including structures/unions/enumerations), shared arrays (including multi-blocked array [7]), shared pointers (with either shared or private targets) and shared locks. All UPC shared objects have an affinity property. A shared object is affine to a particular UPC thread if it is local to that thread’s memory. Shared arrays are distributed in a block-cyclic fashion among the threads, so different pieces of the array have affinity to different threads. Access to shared objects presents a scalability problem that all UPC implementations share, namely that the addresses of locally allocated portions of shared objects are needed by other nodes in order to access shared data. There are multiple solutions to address this problem: •Ensure that shared objects have the same addresses in all nodes. Unfortunately this approach does not work too well with dynamic objects: it tends to fragment the address space and it is cumbersome to implement, sometimes requiring changes to the system memory allocator. •A distributed table of size O(nodes×objects)can be set up to track the addresses of every shared object on every node. For a large number of nodes or threads, this can be prohibitively expensive and thus, directly impacting scalability. It also requires extensive communication when the virtual to physical mapping changes on a particular node. •The third solution involves a distributed symbol table. Shared objects are known by their handles (unique identifiers for every shared object). Translation from shared object handles to memory locations only happens on the home node of the shared object. In the XLUPC runtime we opted for the last approach. Shared objects are organized into a distributed symbol table called the Shared Variable Directory (SVD). The SVD manages the life cycle of shared objects (allocation, freeing, use). On a system with nUPC threads the SVD consists of n+ 1 partitions. Partition k,0≤k < n holds a list of those variables affine to thread k. The last partition (called the ALL partition) is reserved for shared variables allocated statically or through collective operations. Each partition in the SVD is a list of shared objects. Shared objects that are locally allocated have an associated control structure containing the memory addresses in question (Figure 2). Shared objects are referred to by their SVD handles, opaque objects that internally index the SVD. An SVD handle contains the partition number in the directory, and the index of the object in the partition. Multiple replicas of the SVD exist in a running XLUPC system. The SVD often changes at runtime because of UPC routines for dynamic data allocation, such as upc_global_alloc,upc_all_alloc, and upc_local_alloc. The SVD has to be kept internally consistent. Partitioning greatly aids this process, because it allows the SVD to be kept consistent with a minimal effort and without any bottlenecks: 1) UPC threads can allocate and de-allocate shared variables independently of each other. Each thread updates its own partition, and sends notifications to other threads; 2 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. Figure 1: Software organization of XLUPC runtime 2) Because the SVD is partitioned, and each partition has a single writer, memory allocation events do not require locks. The ALL partition is only updated by collective memory allocation operations that are already synchronized. 2.2. The performance compromise of the SVD design We consider the SVD essential to the scalability of the XLUPC runtime. Unfortunately there is a price to be paid for translating SVD handles to memory addresses only at the target node. Modern communication transports (like Myrinet [19] and LAPI [15]) have one-sided RDMA communication primitives that require no CPU involvement on the remote end. However, these primitives typically require the physical address of the shared object at the target to be known by the initiator of the communication. Figure 3 contrasts the implementation of remote GET when the address of the object is known (b), and when it is not (a). RDMA GET and PUT functions cannot be used in a naive SVD implementation, since SVD lookups require CPU involvement. This both lengthens communication latency and burdens the target CPU with work, contributing to the scalability bottleneck. In the next section we outline an optimization that allows the use of RDMA operations while preserving the SVD design, by caching the remote addresses of shared objects on an as-needed basis. 3. Remote Address Cache The goal of the remote address cache (hereafter address cache) is to enable small message transfers via RDMA, and thus reduce latency and improve the performance of the XLUPC runtime system. It does this by caching remote addresses as needed by computation. Unlike a full table of remote addresses, the cache’s memory requirements can be controlled and offset against performance. The address cache is implemented as a hash table. Each entry in the cache correlates a universal SVD handle and a node identifier ID with the physical base address for the shared variable identified by the SVD handle on the remote node ID. During a GET or PUT operation the initiating node consults the address cache for the base address on the target node. A cache hit guarantees that the final remote address (base address + offset) can be calculated on the initiator node, allowing the message transfer to be executed as an RDMA operation. Conversely, if the address lookup in the cache fails the operation cannot be executed as an RDMA operation. However, the slower operation can be harnessed to retrieve the remote base address for the next operation to the same node. We have modified the default (nonRDMA) one-sided messaging protocol to retrieve the base address of the remote shared object during the transfer by piggybacking it either on the data stream or on the ACK message. To ensure the correctness of RDMA transfers, the remote node has to guarantee that the remote memory address has not changed between accesses. To ensure that the physical address (typically required by RDMA operations) of a shared array is fixed, we need to pin the array in memory [28], [24]. To this end we augmented the address cache with a table of registered (pinned) memory locations. The pinned address table is tagged by local virtual addresses and contains physical addresses in the format needed by RDMA operations. Figure 4 portrays a typical runtime snapshot. In the figure the address cache of node A caches two remote addresses on node B. Both these entries exist in the pinned address table of node B. Node C also caches 3 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. distributed shared array part 0 part 1 part 2 part 3 ALL part 0 part 1 part 2 part 3 ALL All−0 part 0 part 1 part 2 part 3 ALL UPC thread n−1 ... var 0−0 All−0 All−0 var 0−1 var 0−2 var 1−0 var 1−1 UPC Thread 0 UPC thread 1 ... ... ... Figure 2: Shared Variable Directory in a distributed memory machine. Gray boxes represent the local memory of each of n UPC threads, each with a copy of the SVD. The SVD has n+ 1 partitions; each partition has a list of control blocks, one each for shared objects known locally. Addresses are only held for the local or ALL partitions. Distributed shared array “All-0” has a different local address on every node. (a) Default GET (b) RDMA GET Figure 3: Protocols for GET: (a) when the remote address is unknown a Request To Send (RTS) message needs to be sent before every data transfer. Acknowledge message may be required depending on the implementation (i.e. to signal completion or to carry the remote address if data transfer is performed thought one-sided communication). And (b) with RDMA operation when the remote address is known. one of these entries, as well as another entry on node A (shaded appropriately to show the correspondence). 3.1. Implementation details In our implementation, before an address can be tagged in another node’s address cache it needs to be pinned locally. Nevertheless, the pinning strategy is not the goal of this paper and for the sake of simplicity a greedy “pin everything” approach is presented here. We resolve that: (i) the entire memory allocated for a shared object is pinned at once on a particular node. For example, if any element of a shared array Ais accessed on node n,all of the memory related to array Ais pinned on that node, making it all available for RDMA operations. And also, (ii) once a shared object is pinned it remains pinned until it is freed. The address cache is eagerly invalidated when a shared object is deallocated. These decisions simplify the implementation of the address cache, because the cache tags can simply be the SVD handles and consistency of the remote address cache is not an issue. However, these assumptions limit the implementation in two important respects. First, it does not deal with network transports that limit the amount of contiguous memory pinned by a single call, and also limits on the total amount of memory pinned are ignored. We have successfully implemented a more elaborated technique to deal with both these issues obtaining similar results. 4 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. Figure 4: Example snapshot of address cache on three nodes. A detailed explanation can be found in [10]. 3.2. Considerations for a LAPI based implementation The LAPI implementation of the XLUPC low level messaging API uses Active Messages (LAPI_Amsend) (as shown in Figure 5 (a)). Implementation of the remote address cache required only trivial changes in the messaging library to enable the process of populating the address cache. Cache hits result in messages that bypass the standard messaging system completely and use RDMA directly. LAPI limits the amount of memory that can be assigned to a single registered memory handle (a configuration parameter, 32 MBytes on our machines). 3.3. Considerations for the Myrinet/GM implementation GM is a message-based communication system for Myrinet [19]. The XLUPC runtime Myrinet port was implemented on top of GM instead of MX, mainly because GM is currently the driver installed on our testing platform (the MareNostrum supercomputer [3]). Moreover, GM provides one-sided support and primitives to directly expose the RDMA capabilities of the hardware. However, three considerations need to be taken into account: first, Myrinet requires memory registration for any data transfer, and it is handled by the programmer in GM. Second, the largest message GM can send or receive is limited by the amount of DMAable memory the GM driver is allowed to allocate by the operating system. (1 GByte on our test machines). Finally, memory registration is an expensive operation; memory deregistration even more so. Our original XLUPC Myrinet port implements multiple transfer protocols depending on message size [20]. Short messages are copied to avoid memory registration costs. Long messages are transferred with an MPIlike rendezvous protocol with memory registration/deregistration embedded into the phases of the protocol. As an optimization a cache of registered memory regions was implemented [26] with lazy memory deregistration. Again, the changes required in the messaging library to enable the remote address cache were trivial (Figure 5 (b)) and very similar to the ones in LAPI. 4. Evaluation The scalability of the XLUPC runtime with the Shared Variable Directory has been demonstrated in our previous work [8]. In this paper we aim to demonstrate the effect of the address cache on messaging performance. Our experiments were performed on comparatively smaller machines: 512 nodes of the MareNostrum supercomputer and a 28-node Power5 cluster running AIX. We defined a confidence coefficient of 95% and ran each experiment multiple times to reduce the standard error. We assumed experiments to be independent, therefore the formulas associated with a normal distribution apply [14]. 4.1. Evaluation environment: MareNostrum MareNostrum [3] is a cluster of 2560 JS21 blades, each with two dual core IBM PPC 970-MP processors sharing 8 GBytes of main memory. The processors are equipped with a 64 KByte instruction/32 KByte data L1 cache and a 1024 KBytes of L2 cache and run the SLES9 (Linux) operating system. 5 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. (a) LAPI (b) GM Figure 5: Enabling address cache population in the standard XLUPC implementation of GET (changes shown in italics), in (a)LAPI and (b)GM. The initiating process sends an Active Message that triggers the execution of a header handler on the passive target. The header handler performs address translation and memory registration. The reply sent back to initiator process contains requested user’s data plus the address. GM requires different protocols for different message sizes. The MareNostrum’s interconnection network is Myrinet with a 3-level crossbar, resulting in 3 different route lengths (1 hop, when two nodes are connected to the same crossbar aka. linecard, and 3 hops or 5 hops depending on the number of intervening linecards). As mentioned before, we implemented the XLUPC runtime transport on top of the Myrinet/GM messaging library. 4.2. Evaluation environment: Power5 cluster The second machine we performed our experiments on is a 28-node cluster of Power5 servers. Each node is with 16 GBytes of RAM and 8 2-way SMT Power5 cores running at 1.9 GHz. The nodes are connected with an IBM High-Performance Switch (HPS). The operating system is AIX 5.3; we used the LAPI library provided as part of the Parallel Operating Environment on the system as the basis for the XLUPC transport implementation. 4.3. Microbenchmarks Our first set of experiments sought to quantify the maximum benefit obtainable by the address cache. We wrote and executed microbenchmarks to compare GET roundtrip latencies and PUT overheads of the XLUPC runtime with and without cache operation. Figure 6 shows the relative gains (i.e. execution time reduction) caused by deploying the address cache for PUT and GET on both LAPI and Myrinet/GM transports. We see three distinct modes on all platforms. •For very small messages (up to 1 KByte in size) the gains in GET roundtrip latency (left panel of Figure 6) are in 30% and 16% range respectively for GM and LAPI. This is the optimization that we had been targeting. •For medium message size range messages (1 KByte to 16 KByte) there are even larger gains (around 40%). This is likely due to the rising cost of memory copies in the non-cached case. The gain is more visible on LAPI, fadding out at 2 MByte, than on Myrinet because the rated bandwidth of the HPS switch is 8x that of Myrinet. Roundtrip latencies of both networks are in the 4-8 microsecond range. Thus, a fixed amount of overhead reduction (of the order of the roundtrip latency) affects the HPS switch for much higher message sizes than Myrinet. •As expected, differences between cached and uncached behavior diminish as message size increases and communication becomes bandwidth dominated instead of overhead dominated. With PUT messages (right panel of Figure 6), in GM we do not see any benefit of using the address cache for small message transfers, up to 2 KBytes. PUT execution time is not significantly affected by address translation overheads, and the load on the remote 6 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. xlupc_distr_get latency improvement using the cache of svd addresses 0 5 10 15 20 25 30 35 40 45 1 4 16 64 256 1024 4096 16384 65536 262144 1048576 4194304 message size (in bytes) percentade of improvement GM LAPI xlupc_distr_put latency improvement using the cache of svd addresses -250 -200 -150 -100 -50 0 50 100 1 4 16 64 256 1024 4096 16384 65536 262144 1048576 4194304 message size (in bytes) percentage of improvement GM LAPI Figure 6: Latency improvement by using the address cache in both platforms: LAPI(o) and GM (x) considering different message sizes. The Yaxis represents the performance benefit in terms of execution time reduction by using the address cache, expressed in percentage ( 100(Z−W) Z, with Zbeing the average regular latency, and Wbeing the average latency by making use of the address cache.Xaxis shows the message size. Latency comparision of xlupc_distr_get with and without the cache of svd addresses in GM 0 10 20 30 40 50 60 70 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 message size (in bytes) time (in usecs) without cache with cache Latency comparision of xlupc_distr_get with and without the cache of svd addresses in LAPI 0 5 10 15 20 25 30 35 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 message size (in bytes) time (in usecs) without cache with cache Figure 7: GET latency with and without the address cache in both platforms: GM and LAPI, considering short message sizes. The Yaxis represents the latency. The Xaxis is the message size. CPU is not measured on this microbenchmark. The situation is different for LAPI, where we see a net decrease in performance of up to 200% by using the address cache. The cause of this performance decrease is the IBM switching hardware, which offers excellent throughput in RDMA mode, at the cost of higher latency. In a GET operation the higher latency is offset by the fact that the remote node’s CPU time is not part of the roundtrip time; in the case of PUT the remote CPU’s operation is overlapped with the next send. Following these results, we disabled the address cache for the PUT operations in LAPI. Figure 7 shows another view of the GET roundtrip latency data: the absolute latencies in microseconds for small message GET operations with and without address caching on both our test platforms. 4.4. DIS Stressmark Suite We have implemented a subset of the DIS Stressmark Suite [2] and ran it with and without address caching. DIS benchmarks have been chosen over e.g. the NAS benchmarks because they recreate the access patterns of data-intensive real applications, whereas most NAS benchmarks are computerather than data-intensive. Four of the benchmarks have been implemented and evaluated (Figure 9): •The Pointer Stressmark is repeatedly following pointers (hops) to randomized locations in memory until a condition becomes true. The entire process is performed multiple times. Each UPC thread runs the test separately with different starting and ending positions on the same shared array. •The Update Stressmark is a pointer-hopping benchmark similar to the Pointer Stressmark. The major difference is that in this code more than one 7 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. remote memory location is read - and one remote location is updated - in each hop. All this is done by UPC thread 0, while the other threads idle in a barrier. This benchmark is designed to measure the overhead of remote accesses to multiple threads. •The Neighborhood Stressmark is a stencil code prototype. It deals with data that is organized in multiple dimensions. It requires memory accesses to pairs of pixels with specific spatial relationships.Computation is performed in parallel based on the locality of the shared array. The twodimensional pixel matrix is block-distributed in a row major fashion. Accesses are local or remote depending on stencil distances and pixel positions. •The Field Stressmark emphasizes regular access to large quantities of data. It searches an array of random words for token strings, that delimit the sample sets, from which simple statistics are collected. The delimiters themselves are updated in memory. When all instances of a token are found and statistics computed, the process is repeated with a different token. The string array is blocked in memory (i.e. with a block size of dN T HREADS e, with Nbeing the array size). Because the array is updated in every run, the outermost loop (which iterates over multiple tokens) cannot be parallelized. Parallelization is done instead in the inner loop, where each UPC thread searches the local portion of the data string for tokens. Because a token may span the boundary of two segments affine to different threads, the threads must overlap their search spaces by at least the width of a token to guarantee that all tokens are found. Since token lengths are typically much smaller than the string array’s blocking factor, most accesses in the algorithm are local. 4.5. Cache size considerations Cache size is an important metric that may affect overall application performance. The space needed by the address cache depends on both the number of shared variables declared by the UPC application and the communication pattern in the running application. Most UPC applications (with a few notable exceptions) declare a relatively small number of shared variables and have static and well defined communication patterns that result in insignificantly small caches even on large machines. Our selected subset of benchmarks covers both types of applications. Field and Neighborhood follow the well defined communication patterns of typical UPC applications. As it is shown for Neighborhood in figure 8 (b), it results in insignificantly small caches, only a few cache entries are used and the hit ratio keeps constant as we scale. Whereas Pointer and Update belong to the group of rare UPC applications that unpredictably access remote memory locations along the whole shared memory space, which results in address caches that grow with the number of nodes. Figure 8 (a) shows for Pointer hit ratio degradation as we scale, with a prompt starting point as cache size is reduced. For this kind of applications we have a compromise between memory usage and speedup. The Address Cache is currently implemented as a dynamic hash table. Its size is allowed to increase on demand to a fixed limit of 100 entries. The next section shows a considerably performance improvement, even for applications following an unpredictable communication pattern. Concerning the Pinned Address Table, its appropriate size depends exclusively on the number of shared variables. It is our experience that in most UPC programs (i.e. the NAS benchmarks, the HPC Challenge benchmarks, a suite of NSA benchmarks and others) the number of shared variables is relatively small. Shared variables enhance productivity but decrease performance; most good UPC programs therefore limit their number. Our experiments show that a table of 10 entries is more than enough for well defined UPC applications. 4.6. Stressmark evaluation on MareNostrum We evaluated the 4 Stressmarks on a 512 blade subset of the MareNostrum computer, with 4 UPC threads running on each blade. A representative subset of our experiments is shown in Figure 9. Since UPC threads running in the same blade share memory, remote communication between these threads does not involve the network hardware. This can affect measured performance improvements, since no benefit from the address cache can be expected when the network hardware is not in use. The Pointer Stressmark shows good performance, between 30% and 60% improvement depending on the total number of remote accesses in the benchmark. The performance of the Pointer Stressmark is gated by network latencies and overheads; any reduction in these results in performance gains. In hybrid execution mode the network device is shared by all UPC threads running on a blade; the improvement caused by smaller CPU overheads was augmented by smaller network device overheads; with four threads competing for the same network device any improvement in network device access time is magnified fourfold. Since in cached mode the DMA is doing the work of data transfer, the four threads spend much less time queueing up for access to 8 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply. Pointer: Hit Rate depending on cache size 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 8 - 2 16 - 4 32 - 8 64 - 16 128 - 32 256 - 64 512 - 128 1024 - 256 2048 - 512 # threads - # nodes Cache Hit Rate 4 cache entries 10 cache entries 100 cache entries (a) Pointer Neighborhood: Hit Rate depending on the Cache size 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 8 - 2 16 - 4 32 - 8 64 - 16 128 - 32 256 - 64 512 - 128 1024 - 256 2048 - 512 # threads - # nodes Cache Hit Rate 4 cache entries 10 cache entries 100 cache entries (b) Neighborhood Figure 8: Address Cache Size Evaluation using DIS Stressmark Suite. The vertical axes show the percentage of hits on the Address Cache. The horizontal axes of the graphs show the threads and nodes used. Each line represents a different cache configuration in terms of cache size. Results are shown considering a random thread. DIS: Address Cache evaluation in hybrid GM 0 10 20 30 40 50 60 70 8 - 2 16 - 4 32 - 8 64 - 16 128 - 32 256 - 64 512 - 128 1024 - 256 2048 - 512 # threads - #nodes Percentage of improvement Pointer Update Neighborhood Field (a) GM DIS: Address Cache Evaluation in hybrid LAPI -10 0 10 20 30 40 50 60 4 - 2 8 - 2 16 - 2 32 - 2 64 - 4 128 - 8 256 - 16 448 - 28 #threads - #nodes Percentage of improvement Pointer Update Neighborhood Field (b) LAPI Figure 9: Address Cache Evaluation on GM (a) and LAPI (b) using DIS Stressmark Suite The vertical axes show performance improvement when using the remote address cache: 100(Z−W) Z, where Zand Ware the regular and address-cache-enabled runtimes respectively. The horizontal axes of the graphs show the threads and nodes used. the network device. This effect could not be appreciated in the microbenchmark because it ran on 1 active thread in each node. The Update Stressmark shows a 11% to 22% performance improvement when the address cache is enabled, which corresponds to the performance measured by the PUT and GET microbenchmarks. We do not see performance improvement caused by two threads per node, because only thread 0 initiates communication. The Neighborhood Stressmark shows 10% to 20% improvement. The stencil used in this experiment (with a stencil distance of 10) causes about 3 16 of memory accesses to be potentially remote, depending on the number of nodes, therefore the improvement is mostly along the lines we expected based on the microbenchmark. The Field Stressmark shows a 35% to 40% performance improvement. We analyzed the behavior of this benchmark using the Paraver performance analysis toolkit [3]. The trace showed that the remote GET and PUT access times at the “overhangs” were abnormally large when address cache was not in use. As a reminder, each thread in the benchmark scans the local portion of a distributed array. The scan extends into “overhangs” that belong to the two neighboring threads. With normal XLUPC runtime operation the remote node’s CPU is part of every remote access; but the Myrinet/GM transport does not overlap communication and computation. While a CPU is busy with the local 9 Authorized licensed use limited to: UNIVERSITAT POLIT?CNICA DE CATALUNYA. Downloaded on March 25,2010 at 10:34:55 EDT from IEEE Xplore. Restrictions apply.