HBM performance on FPGAs

HBM performance on FPGAs Elias Perdomo∗†, Teresa Cervero∗, Xavier Martorell∗† Behzad Salami∗, ∗Barcelona Supercomputing Center, Barcelona, Spain †Universitat Polit` ecnica de Catalunya, Barcelona, Spain E-mail: {elias.perdomo, teresa.cervero, xavier.martorell, behzad.salami}@bsc.es Keywords—HBM, FPGA, performance, pseudo-channel, microswitches I. EXTENDED ABSTRACT Main memory access has become an increasing performance bottleneck for traditional and High-Performance Computing (HPC) applications. High Bandwidth Memory (HBM) emerged as an alternative to conventional DRAMs, offering higher bandwidth, lower power, and higher integration capabilities to meet the demands of contemporary applications. The transition of most advanced FPGAs from DDR to HBM confirms this paradigm shift. However, users face substantial challenges due to the scarcity of technical documentation on maximizing HBM features when using FPGAs. We addressed the knowledge gap for HBM characteristics within FPGAs, aiming to standardize its utilization in the complex HPC domain. Our Memory Sandbox enables analysis within and across HBM pseudo-channels. We show that HBM achieves 99.99% of its nominal peak bandwidth with long sequential memory accesses. However, we observe a performance drop to 0.17% with reduced burst size and random data access patterns. Our study spotlights the necessity for meticulous management of concurrent accesses and strategic data placement in HBM, offering critical considerations for optimizing HBM performance in FPGA-based systems. A. HBM in the current computer architecture environment Custom hardware – from workstations to PCs– has experienced tremendous improvements in the past decades. However, while the speed of commercial microprocessors has increased by approximately 70% every year, the speed of commodity DRAM has improved by only around 50% in the past decade. As a result, computer systems are experiencing difficulties in achieving high processing efficiency[1]. The traditional approach for boosting performance in systems, particularly those at the edge where huge amounts of data need to be processed locally or regionally; consists of adding more computational capability into a chip and bringing more memory on-chip. But that approach no longer scales since we are now getting to the boundaries of the trifecta of von Neumann architectures, Moore’s Law and Dennard scaling. Consequently, engineers have begun focusing on solving the bottleneck between processors and memories by turning out new architectural designs at a rate no one would have anticipated before. An alternative, based on recent technological advances, is moving processing elements closer to, or even into the 42% 260% 200% 50% Power consumption per chip is reduced over 68% with HBM2 Memory Bandwidth per Chip increases 530% with HBM2 Bandwidth (GB/s) Ratio [mW/Gbps/Pin) Memory Versions Memory Versions 256 192 160 128 96 64 32 0 GDDR5 HBM1 HBM2 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 GDDR5 HBM1 HBM2 Fig. 1. Bandwidth and power consumption comparison. memory. This solution looks to avoid the penalty for replicating processing elements, which provides an acceptable tradeoff. When utilizing wide short buses (HBM being the most common example [2], [3]), designers avoid the penalty of going outside the die for access to memory and recover some of the performance tradeoffs. HBM systems can overcome all DRAM challenges as an enabler of architectures for high-performance and/or low-power computing, while its low speed/pin consumption also improves power efficiency [19], [20](Fig. 1). This trend is followed by Xilinx, one of the two market giants in the area of FPGAs and the leader in adaptive computing. Xilinx is firmly committed to a transition to HBM memory as a solution to memory bottlenecks, as demonstrated in recent years. In October 2018, Xilinx launched the Alveo U200 with no HBM memory [4]. Only 1 month after, in November 2018 the new Alveo U280 already included 8GB of HBM2 and halved DDR capacity [5]. Their last Alveo Data center card release, the Alveo U55C, doubled HBM capacity and rescinded the DDR memory banks’ use [6]. B. HBM performance analysis To shed some light on the intrinsic details of HBM, we developed the Memory Sandbox tool providing higher configurability, more control over measurements, and further insights (i.e. clock cycles of each memory transaction) than the current HBM monitor offered by Xilinx. Our configurable environment is structured in two main pieces: a front-end piece as a user interface for setting up the experiments to be executed, and a back-end piece composed of a set of hardware IPs to run the experiments in the FPGA, according to the data introduced in the front end. Thus, the most relevant IP we have developed is a highly Configurable Pattern Generator, which mimics processor threads data requests with sequential and pseudo-random memory access patterns. 0 50 100 150 200 250 300 350 400 450 500 WR RD WR RD WR RD WR RD WR RD WR RD WR RD RCB BRC BRGCG RBC RGBCG RBC true RBC false Address Mapping Policies 32 PsCHs (Read and Write) Theoretical Maximum 460.8GB/s 58.35% 61.26% 63.36% 63.76% 44.41% 49.23% 93.50% 99.99% 94.49% 99.99% 94.82% 99.99% 74.39% 74.39% default ID feature 0 5 10 15 20 25 30 35 40 WR RD WR RD WR RD WR RD BRC RBC RCB RCBI Address Mapping Policies 2 DDR4 modules (Read and Write) Throughput (GB/s) Theoretical Maximum 38.4GB/s 7.98% 11.81% 8.28% 11.98% 77.23% 86.18% 93.82% 93.82% DDR HBM (a) (b) Fig. 2. Throughput Results for (a)DDR4 and (b) HBM Address Mapping Policies. An initial analysis of typical memory access patterns allows us to implement benchmarks to reveal the subjacent characteristics of HBM and DDR in FPGAs. For this purpose, we emulate the Repetitive Sequential Traversal (RST) a typical sequential access pattern widely used in FPGA programming and sparse accesses with pseudo-random accesses. The first scenarios intend to stress HBM and DDR to measure the actual throughput peak (bandwidth) when using our Memory Sandbox. For this purpose, we perform sequential accesses (RST) in vertically attached pseudo-channels or banks. We enabled outstanding transactions and burst sizes were set to the maximum (16 and 256 beats, respectively). Address mapping policies microbenchmarks results are shown in Fig. 2. Most modern computer applications require large amounts of memory access. In HBM, as each pseudo-channel has a size of 256MB, multiple pseudochannels will likely be accessed by most applications. From the previous experiments, we know that the performance of a single pseudo-channel is the result of any address mapping policy in Fig. 3 divided by the total amount of pseudochannels (32). Fig. 3 shows the results of accessing different HBM pseudo-channels emulating a single-threaded processing element connected to AXI Port 0. These experiments are performed with a sequential access pattern (RST), a burst size of 16 and RBC true as address mapping policy, which offers the best performance for this type of access pattern according to our experiments. Two main conclusions can be drawn from these experiments: • Pseudo-channels on the same micro-switch show the same performance regardless of the AXI port accessing them. • Throughput experiences an average degradation of 50% if the processing element performs memory accesses outside the pseudo-channel to which it is directly connected. This performance loss is the same for the adjacent micro-switch or the furthest one. There is no linear degradation. The performance is either the same for the 4 pseudo-channels within the same micro-switch or 50% in the other 28 pseudo-channels. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 WR RD Ps-CH in different micro-switches Theoretical Maximum 14.4GB/s Throughput (GB/s) µSW0 µSW1 µSW2 µSW3 µSW4 µSW5 µSW6 µSW7 Fig. 3. Throughput Results for HBM accessing different micro-switches. C. Conclusion HBM appears as a solution being integrated into FPGAs to face the memory wall issue and large companies are already committed to its use. As expected, the throughput performance was more than 12 times better when using all 32 pseudochannels in the HBM in parallel than when using the 2 memory banks in the DDR. The different address mapping policies, the burst size, accesses within a micro-switch or external ones, and the randomization of the address can have a huge impact on the HBM throughput. REFERENCES [1] Todd Carl Mowry, “Tolerating latency through software-controlled data prefetching,” PhD Thesis, Stanford University, Mar. 1994. [2] Copyright © 2022 Samsung ALL rights reserved, “Next-level performance,” Samsung HBM, Tech. Rep., 2022. [Online]. Available: https://www.samsung.com/semiconductor/dram/hbm/ [3] M. Ujald´ on, “HPC Accelerators with 3D Memory,” 2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES), pp. 320–328, Aug. 2016, doi: 10.1109/CSEEUC-DCABES.2016.203. [4] Xilinx Inc., “ALVEO™ Product Selection Guide Datasheet,” Xilinx Inc., Tech. Rep. XMP451 (v1.7), 2021. [Online]. Available: https://www.xilinx.com/support/documentation/selectionguides/alveo-product-selection-guide.pdf [5] ——, “Xilinx Extends Data Center Leadership with New Alveo U280 HBM2 Accelerator Card,” Xilinx Inc., Tech. Rep., Nov. 2018. [Online]. Available: https://www.xilinx.com/news/press/2018/xilinx-extends-datacenter-leadership-with-new-alveo-u280-hbm2-accelerator-card-dell-emcfirst-to-qualify-alveo-u200.html [6] Xilinx Inc., “Xilinx Launches Alveo U55C, Its Most Powerful Accelerator Card Ever, Purpose-Built for HPC and Big Data Workloads,” Xilinx Inc., Tech. Rep., Nov. 2021. [Online]. Available: https://www.xilinx.com/news/press/2021/xilinx-launches-alveo-u55c-itsmost-powerful-accelerator-card-ever-purpose-built-for-hpc-and-big-dataworkloads.html Elias Perdomo received a B.Eng. degree in Automation Engineering in 2012, and a M.Sc. degree in Digital Systems in 2018 both from the Technological University of Havana, Cuba. In addition, he received a M.Sc. Advanced Microelectronic Systems Engineering from the University of Bristol, UK in 2019. Since 2020, he has been working with FPGA Team of the Barcelona Supercomputing Center (BSC) as well a PhD student at the Compuer Architecture Department of the Universitat Politecnica de Catalunya (UPC), Barcelona, Spain. His current research interests include Embedded Systems, RTL and RISCV SoC design, FPGAs, Heterogeneous Computing, automatic design generation and memory management for HPC and programming models.