An energy-efficient near-data processing accelerator for DNNs to optimize memory accesses

Contents lists available at ScienceDirect Journal of Systems Architecture journal homepage: www.elsevier.com/locate/sysarc An energy-efficient near-data processing accelerator for DNNs to optimize memory accesses✩ Bahareh Khabbazan∗,Mohammad Sabri,Marc Riera,Antonio González Universitat Politècnica de Catalunya (UPC), Barcelona, Spain A R T I C L E I N F O Keywords: DNN NDP Accelerators Quantization Exponential Transformer A B S T R A C T The constant growth of DNNs makes them challenging to implement and run efficiently on traditional computecentric architectures. Some accelerators have attempted to add more compute units and on-chip buffers to solve the memory wall problem without much success, and sometimes even worsening the issue since more compute units also require higher memory bandwidth. Prior works have proposed the design of memorycentric architectures based on the Near-Data Processing (NDP) paradigm. NDP seeks to break the memory wall by moving the computations closer to the memory hierarchy, reducing the data movements and their cost as much as possible. The 3D-stacked memory is especially appealing for DNN accelerators due to its high-density/low-energy storage and near-memory computation capabilities to perform the DNN operations massively in parallel. However, memory accesses remain as the main bottleneck for running modern DNNs efficiently. To improve the efficiency of DNN inference we present QeiHaN, a hardware accelerator that implements a 3D-stacked memory-centric weight storage scheme to take advantage of a logarithmic quantization of activations. In particular, since activations of FC and CONV layers of modern DNNs are commonly represented as powers of two with negative exponents, QeiHaN performs an implicit in-memory bit-shifting of the DNN weights to reduce memory activity. Only the meaningful bits of the weights required for the bit-shift operation are accessed. Overall, QeiHaN reduces memory accesses by 25% compared to a standard memory organization. We evaluate QeiHaN on a popular set of DNNs. On average, QeiHaN provides 4.3𝑥speedup and 3.5𝑥energy savings over a Neurocube-like accelerator. Contents 1. Introduction ...................................................................................................................................................................................................... 2 2. Background & Related work ............................................................................................................................................................................... 3 2.1. Modern DNNs ........................................................................................................................................................................................ 3 2.2. DNN quantization................................................................................................................................................................................... 3 2.3. Dataflows in DNN accelerators ................................................................................................................................................................ 3 2.4. 3D-stacked memory................................................................................................................................................................................ 3 2.5. 3D-stacked DRAM-based DNN accelerators ............................................................................................................................................... 4 3. LOG2 quantization analysis ................................................................................................................................................................................ 4 4. QeiHaN accelerator............................................................................................................................................................................................ 5 4.1. Architecture........................................................................................................................................................................................... 6 4.2. Memory organization.............................................................................................................................................................................. 7 4.3. Dataflow................................................................................................................................................................................................ 7 5. Methodology ..................................................................................................................................................................................................... 8 6. Evaluation......................................................................................................................................................................................................... 8 6.1. 3D-stacked memory accesses ................................................................................................................................................................... 9 6.2. Performance........................................................................................................................................................................................... 9 6.3. Energy consumption ............................................................................................................................................................................... 9 ✩New Paper, Not an Extension of a Conference Paper. ∗Corresponding author. E-mail address: [email protected] (B. Khabbazan). https://doi.org/10.1016/j.sysarc.2024.103320 Received 11 July 2024; Received in revised form 25 November 2024; Accepted 5 December 2024 Journal of Systems Architecture 159 (2025) 103320 Available online 12 December 2024 1383-7621/© 2024 Elsevier B.V. All rights are reserved, including those for text and data mining, AI training, and similar technologies. B. Khabbazan et al. 6.4. Area...................................................................................................................................................................................................... 10 6.5. Comparison with a TPU-like accelerator................................................................................................................................................... 10 7. Conclusions ....................................................................................................................................................................................................... 10 CRediT authorship contribution statement ........................................................................................................................................................... 10 Declaration of competing interest........................................................................................................................................................................ 10 Acknowledgments .............................................................................................................................................................................................. 10 Data availability ................................................................................................................................................................................................ 10 References......................................................................................................................................................................................................... 11 1. Introduction Deep Neural Networks (DNNs) represent the state-of-the-art solution to a broad range of machine learning applications such as natural language processing (NLP) and image classification. Modern DNNs can outperform human-level accuracy in many of these applications at the expense of high computational cost, memory requirements, and energy consumption. Complex DNN models are composed of hundreds of layers of artificial neurons with billions of model parameters and operations. The constant growth of DNNs makes them challenging to implement and run efficiently, even in the most recent accelerators [1] based on traditional computing architectures due to the memory wall problem. On the other hand, some recent research has focused on a new paradigm named Near-Data Processing (NDP) [2,3], which seeks to break the memory wall by moving the computations closer to the memory hierarchy. Conventional DNN accelerators dedicate a significant part of their area to processing elements (PEs) to accelerate the frequent dot-product operations in DNN layers. Many previous designs exploit data and thread-level parallelism through large PE arrays, which further amplifies memory bandwidth demands, often creating a bottleneck in feeding data to the PEs. Despite efforts to reduce off-chip memory accesses and improve on-chip data reuse, the memory wall continues to be a major constraint in compute-centric architectures. Data movement, which accounts for 62.7% of total energy consumption according to recent studies, remains the dominant factor in energy overhead, far surpassing the energy required for computations. Additionally, much of this data transfer is triggered by simple operations and primitives that could be efficiently handled with low-cost hardware implementations. These observations, along with the increasing size of DNN models, motivate the transition from conventional compute-centric to datacentric architectures for more efficient processing of data-intensive workloads. Over the last few years, researchers have been exploring novel memory-centric architectures based on the so-called Near-Data Processing (NDP) paradigm to accelerate neural network algorithms by moving most of the computations ’’in/near-memory’’ and, hence, reducing the data movements and their cost as much as possible [4,5]. NDP has gained a lot of attention with the introduction of the 3D stack memory technology, which allows the integration of logic and memory in the same chip, providing high-speed connections between a high-density memory and a logic die. Micron’s Hybrid Memory Cube (HMC) [6], High Bandwidth Memory (HBM) [7] from AMD/Hynix, and Samsung’s Wide I/O [8] are popular examples implementing this trending technology. NDP architectures based on 3D-stacked memory attack the memory wall by increasing storage capacity, memory bandwidth, and reducing power consumption [9]. Compared with the conventional 2D DRAM, 3D memory provides an order of magnitude higher bandwidth (160 to 250 GBps) with up to 5𝑥better energy efficiency and, hence, 3D memory is an excellent option for meeting the high throughput, low energy requirements of scalable DNN accelerators [5,10,11]. All 3Dstacked memory system implementations provide highly parallel access to memory which is well suited to the highly parallel architecture of the DNN accelerators [2,12]. Neurocube [13] and TETRIS [14] are popular NDP 3D-stacked memory architectures that offer promising performance and energy consumption for accelerating DNNs. However, there is still large room for improvement, since these architectures and memory technology present multiple challenges to extend their adoption. First, architectures based on 3D-memory require to rethink of the design of on-chip buffers in the logic die as well as the location where the computations are executed. For example, performing simple operations on the DRAM dies can drastically reduce the amount of memory movements and the need for big on-chip buffers. Second, new approaches for dataflow scheduling and partitioning of the DNN computations are also required to reduce the memory pressure. Thus, changing the memory organization and data placement can fully exploit the features of 3D-stacked architectures. In addition, the area of the logic die is constrained by the package, and there are tight thermal constraints that limit the power dissipation of the system. Consequently, it is critical to propose solutions that improve in these aspects. In this paper, we show how to efficiently exploit a logarithmic base2 quantization (LOG2) of activations on FC and CONV layers of typical DNN models to reduce memory movements due to weights. First, we perform an analysis of the exponents obtained after the LOG2 quantization, and observed that a huge percentage of the activations are represented with negative exponents, that is, their original value is in the range of [−1, 1]. LOG2 quantization has been proposed in previous works to reduce the numerical precision of either activations/weights and exploited to substitute multiplications by a bit-shifting of the other operand. Based on these observations we propose an implicit in-memory bit-shifting of the DNN weights to reduce the memory movements. Weights are uniformly quantized and stored at the bit-level granularity into different memory regions, that is, each bit of a set of weights is stored in a different memory bank to exploit the inherent parallelism of 3D-stacked architectures. Next, we propose a mechanism to avoid accessing the bits of the weights that are not useful due to the right bit-shifting of the negative exponents of the logarithmically quantized activations. Then, we present QeiHaN, a novel NDP accelerator that implements the above LOG2 quantization-shifting engine and efficient weight storage scheme for high-performance low-energy DNN inference. QeiHaN is implemented on top of a Neurocube-like architecture, but extended with an enhanced input stationary dataflow. The extra hardware required for our technique is modest since most of the components are already available in the baseline. QeiHaN only requires a small set of additional comparators and integer adders to perform the LOG2 quantization. Then, we also replace the multipliers by simple bit-shift logic, reducing the computational cost and the overall area of the PEs. Our experimental results show that the overheads are minimal compared to the savings in memory accesses and multiplications. To summarize, this paper focuses on efficient DNN inference leveraging logarithmic quantization in NDP 3D-stacked DRAM-based accelerators. The main contributions are: •We analyze the distribution of exponents of the logarithmically (i.e. LOG2) quantized activations in multiple layers of modern DNNs including CNNs, RNNs, and Transformers. We observe that a huge percentage of the exponents are negative, leading to potential memory savings as a result of reducing the accesses to only the useful bits of the weights. Journal of Systems Architecture 159 (2025) 103320 2 B. Khabbazan et al. •We propose a novel data layout and an optimized dataflow to exploit the bank-level parallelism of 3D-stacked memory together with the LOG2 quantization of activations. Each memory bank stores a different subset of the bits of the uniformly quantized weights to allow for parallel accesses to the required bits of the bit-shifting operations. On average, we reduce the memory accesses due to the weights by 25% compared to a standard memory organization. •We present QeiHaN, a 3D-stacked DRAM-based hardware accelerator that implements our data layout and dataflow for efficient DNN inference. We evaluate QeiHaN for several DNNs. QeiHaN improves performance by 1.4𝑥and reduces energy consumption by 1.3𝑥on average over NaHiD, a baseline accelerator implementing the same dataflow and quantization as QeiHaN but with a standard memory organization for weights. Compared to Neurocube [13], QeiHaN achieves 4.3𝑥speedup and 3.5𝑥energy savings on average. The rest of the paper is organized as follows. Section 2introduces some preliminaries for QeiHaN and provides a summary of works related to 3D memory DNN accelerators. Section 3discusses the observations on the logarithmic quantization of activations for a modern set of DNNs. Section 4describes the architecture of QeiHaN including the implementation details of the main hardware components. Section 5 presents the evaluation methodology and Section 6discusses the experimental results of QeiHaN on different networks. Finally, Section 7 concludes the paper by summarizing the key insights of this design alongside the overall performance. 2. Background & Related work In the following subsections we review some terminology and concepts that may be helpful throughout this paper. First, we give a general description of DNNs, including the main categories and different types of layers. Next, we review DNN quantization and common dataflows of DNN accelerators. Finally, we discuss 3D memory architectures, which offer more opportunities to implement a highly efficient DNN accelerator in terms of both performance and energy consumption. 2.1. Modern DNNs Deep Neural Networks (DNNs) fall into three main categories. First, Multi-Layer Perceptrons (MLPs) consist of multiple Fully-Connected (FC) layers where each input neuron is connected, via synapses with particular weights, to every output neuron. Second, Convolutional Neural Networks (CNNs) employ convolutional layers to extract features, often followed by one or more FC layers for classification. For instance, AlexNet [15] demonstrated remarkable efficiency in image and video processing. Finally, Recurrent Neural Networks (RNNs) [16] comprise stacked cells with feedback connections that store information from past computations to enhance future predictions. The most popular RNN architecture, Long–Short Term Memory (LSTM), uses FC layers arranged in gates. PTBLM [17], an LSTM-based RNN, is widely applied in language modeling, speech recognition, and machine translation tasks. Attention-based models, such as the Transformer [18] and BERT [19] variants, have become state-of-the-art for key tasks like natural language processing [20], computer vision [21,22], question answering [23], and video analysis [24]. These models use attention mechanisms to gather contextual information, implemented via FC layers, allowing them to handle large inputs effectively. Despite their efficiency, these networks suffer from long execution times due to their large memory footprint and lower computation-to-memory access ratio compared to convolutional layers. FC layers are more memory intensive as weights are not reused by different neurons. Most DNNs consist primarily of FC and convolutional layers, which dominate the computational workload. Pooling layers reduce spatial dimensions, normalization layers stabilize training, and activation functions introduce non-linearity, contributing to overall network performance. However, these layers represent a smaller fraction of the execution time. This paper focuses on optimizing hardware accelerators for FC and convolutional layers in MLPs, CNNs, RNNs, and Transformers. 2.2. DNN quantization Quantization is a highly popular technique to map values from a continuous range to a discrete set. The main purpose of quantization is to compress the original DNN models to reduce the memory footprint and the computational cost with a minor impact on accuracy. Eq. (1) shows an example of a function that quantizes real values (in floating-point, FP, precision) and maps them to an integer range. 𝑄(𝑟) =𝐼 𝑁 𝑇(𝑟∕𝑠) −𝑧(1) where 𝑄(𝑟)is the quantized value, 𝑟is a FP value, 𝑠is a scaling factor, and 𝑧is an integer offset. The 𝐼 𝑁 𝑇function is a rounding to the nearest value. This method is also referred to as linear uniform quantization since the resulting quantized values (a.k.a. quantization levels) are uniformly spaced. Recently, non-uniform quantization schemes have been proposed to further reduce the memory pressure. These methods have been designed for DNN models with tensors that have a bell-shaped long-tailed distribution of weights and activations [25,26]. Logarithm quantization is an example of a non-uniform scheme, where the quantization levels increase exponentially instead of linearly [27]. The Logarithmic Quantization (LQ) [28–30] offers smaller numerical precision (i.e. bitwidth) with lower accuracy loss compared to the linear quantization by exploiting the non-uniform distribution of tensors. QeiHaN employs uniform quantization for the weights and a logarithmic base-2 (LOG2) quantization for the activations of all the FC/CONV layers. Section 3 provides more details on the LOG2 quantization and its main benefits. 2.3. Dataflows in DNN accelerators The dataflow of a DNN accelerator is defined as the mapping and scheduling of the computations as well as the data partitioning across compute units. The dataflow that is most effective to reduce the memory accesses and data movements to optimize performance and energy efficiency depends on the target cognitive computing task and hardware architecture [31]. The dataflow determines the storage requirements and communication patterns among main memory, local on-chip buffers inside PEs, and compute units. In previous works [32,33], the election of the dataflow is based on minimizing the data movement of the inputs, outputs or weights. Therefore, DNN accelerators tend to follow one of these dataflows: Weight Stationary (WS), Output Stationary (OS), and Input Stationery (IS). In OS, each PE computes an output neuron at a time [34]. In the WS/IS dataflows, each PE pre-loads a set of weights/inputs from memory to local buffers, and those are used to perform all associated computations [35]. QeiHaN uses an input stationary dataflow, which means that each input of a given layer is read and reused, until all the related computations are done, before reading the next input. The IS dataflow is the most suitable for our logarithmic quantization of DNN activations and efficient weight storage scheme. We compare QeiHaN with two baseline accelerators, one with OS dataflow and the other with IS dataflow. 2.4. 3D-stacked memory Conventional architectures rely on external accelerators (e.g., GPUs, TPUs) for computation, increasing energy consumption due to frequent large-scale data transfers between memory and compute units. These architectures also face limitations like low memory bandwidth and inefficient data reuse, making them less energy-efficient for data-intensive Journal of Systems Architecture 159 (2025) 103320 3 B. Khabbazan et al. applications, such as DNN inference. In contrast, NDP architectures utilizing 3D-stacked memory mitigate these inefficiencies by bringing computation closer to memory. By reducing data movement, NDP architectures improve both energy efficiency and performance. High-density 3D memory is a promising technology for the memory system of DNN and other domain-specific accelerators [36,37]. It consists of stacking multiple memory dies on top of each other, which increases the memory capacity and bandwidth compared to 2D memory, and also reduces the access latency due to the shorter on-chip wiring interconnection. These aspects lead to an overall improvement in both energy efficiency and performance. The 3D memory dies are commonly based on DRAM, but the integration of other memory technologies is being actively researched with very promising results. On the other hand, recent advances in lowcapacitance through-silicon vias (TSVs) technology have enabled 3D memory that includes a few DRAM dies on top of a logic chip, within a single package [38]. Although there are numerous implementations of 3D-stacked memory technologies, until now, the Hybrid Memory Cube (HMC) [39] by Micron and the High Bandwidth Memory (HBM) [7,40] from AMD/Hynix are the preferred choices for most DNN accelerator proposals [11,13,14,41]. HBM and HMC are designed for high performance data-centric applications. Both are composed of vertically stacked DRAM dies with a single logic layer at the bottom. These memory technologies take advantage of Through-Silicon Vias (TSVs) to enable high-bandwidth and low-latency communication between the stacked memory layers. In HBM, each DRAM die is partitioned horizontally, and different partitions on different dies are treated as independent memory channels. On the other hand, in HMC, each DRAM die is divided into multiple partitions in a 2D grid where the corresponding partitions in the vertical direction form a single vault. Both HBM and HMC can exploit memorylevel parallelism by organizing the large number of TSVs into multiple independently-operated channels. This allows multiple partitions in the DRAM die to be accessed simultaneously, further enhancing memory bandwidth and overall system performance. NDP systems employing HBM or HMC associate the PEs of the logic die with each channel or vault to efficiently utilize the memorylevel parallelism and achieve high data processing throughput. The choice between HBM and HMC would depend on the specific requirements of the NDP system, and the desired trade-offs between memory bandwidth, energy efficiency, and integration with the host processor. 2.5. 3D-stacked DRAM-based DNN accelerators Neurocube [13] is a programmable DNN accelerator integrated into the logic layer of a 3D stack DRAM-based HMC. The Neurocube architecture consists of clusters of processing engines (PE) connected by a 2D mesh NoC in the processing layer. Each PE of the logic layer is associated to a single memory vault, and can operate independently, and communicate through the TSVs and a vault controller (VC). The organization of each PE includes multiple memory buffers to store weights and inputs as well as some units to perform MAC operations. In addition, each vault controller includes a Programmable Neurosequence Generator (PNG) unit that generates the commands to orchestrate the corresponding operations of the DNN layers. The PNGs employ a simple finite state machine (FSM) with counters that are initialized depending on the number of MAC units in each PE and the DNN layer topology. Fig. 1shows a general overview of the Neurocube architecture and a PE. Neurocube partitions input feature maps for CONV layers and outputs for FC layers across different vaults and exploits the output stationary dataflow. While it is highly optimized for CNN execution, its programmability and scalability enable the mapping and execution of various DNN architectures. In this work, we implement a Neurocube-like baseline accelerator to assess the performance improvement and energy savings of QeiHaN. Fig. 1. Neurocube organization. Source: Adapted from [13]. In the same line of research, TETRIS [14] stands out as another popular DNN accelerator based on HMC. Like Neurocube, TETRIS features an optimized hardware architecture coupled with software scheduling and partitioning techniques that exploit the inherent characteristics of 3D memory. First, the authors show that the high throughput and low energy characteristics of 3D memory allow the rebalance of the NN accelerator design, using more area for processing elements and less area for SRAM buffers. Second, they move some portions of the NN computations close to the DRAM banks to decrease the bandwidth pressure and increase performance and energy efficiency. Finally, they develop an optimized dataflow scheduler and hybrid partitioning scheme that parallelizes the DNN computations within and across multiple vaults and stacks. In particular, TETRIS emphasizes a partitioning strategy that is primarily focused on accelerating CONV layers. However, FC layers, which are very common in current DNN workloads such as Transformers, still follow the same dataflow and partitioning strategy of Neurocube, with minor improvements. 3. LOG2 quantization analysis DNN quantization allows to reduce the numerical precision of activations and weights, which in turn favors the memory footprint and the computational cost of hardware accelerated DNN architectures. Therefore, quantization techniques have been widely explored in previous studies as described in Section 2.2. In particular, logarithmic quantization takes advantage of the non-uniform distribution of tensors to significantly reduce the numerical precision of input activations and/or weights with a minor impact in accuracy. This section analyzes the effects of the LOG2 quantization of activations on multiple DNN models and layers. First, we explore the benefit of the logarithmic encoding of activations to simplify the dot-product operations. Then, we provide some hints on reducing the number of accesses to the main memory by exploiting the characteristics of the 3D memory and bit-shifting operation. Some prior works have used linear uniform quantization to compress the DNN parameters. However, we observe that activations and weights of most DNNs do not follow a uniform distribution, which causes a huge impact in terms of accuracy loss when the precision is further reduced to very low bitwidths (i.e. <8𝑏). Especially in recent DNNs that are extremely deep and can have hundreds of layers, the error is propagated and expanded among layers. On the other hand, logarithmic base-2 (LOG2) quantization [28– 30] leverages the usually non-uniform distribution of activations and weights in a pre-trained DNN. The study in [29] compared the impact of linear and LOG2 quantization on activations and weights of VGG16 and AlexNet. Their analysis shows an exponential distribution of activation values around 0. They also concluded that activations are more robust to LOG2 quantization than weights for several reasons. First, Journal of Systems Architecture 159 (2025) 103320 4 B. Khabbazan et al. CONV layers reuse the weights multiple times when computing the dotproducts, propagating the error across the inputs/outputs of all layers. Second, the range of the weights is not as wide as the activations [42], and their density is often higher than that of the activations, that is, the amount of weights is huge, and their range is narrow. Additionally, the DNA-TEQ [43] study provides a detailed analysis of the distribution of weights and activations across various DNNs, demonstrating that the error from fitting weight distributions to a logarithmic model is higher than that for activations. We performed an experiment applying LOG2 quantization to the activations and weights of modern DNNs, together and individually, and reached similar conclusions regarding weights being more sensitive to the LOG2 quantization error than activations. This suggests that the base-2 may not be the best-fitting exponential base for quantizing the weights. In this paper, we apply logarithmic base-2 (LOG2) quantization to the input activations of all the FC and CONV layers of a set of DNNs. On the other hand, we apply INT8 uniformly distributed linear quantization to the weights of these layers based on Eq. (1). These layers represent close to 100% of the total execution time for typical neural networks. Next, we analyze the distribution of exponents of the quantized activations, and the accuracy loss due to the LOG2 quantization. This scheme also allows us to efficiently re-organize offline the weights in-memory without additional expensive hardware, and exploit some of the intrinsic characteristics of the 3D-stacked memory, as described below. For each input 𝑥and each layer 𝑙, the LOG2 quantization is applied according to the following equations: 𝐿𝑜𝑔 𝑄𝑢𝑎𝑛𝑡(𝑥) ={0𝑥= 0 2𝑥 otherwise.(2) 𝑥 =𝐶 𝑙 𝑖𝑝(𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2(|𝑥|))), 𝑚𝑖𝑛, 𝑚𝑎𝑥, (3) where 𝐶 𝑙 𝑖𝑝(𝑥, 𝑚𝑖𝑛, 𝑚𝑎𝑥) =⎧ ⎪ ⎨ ⎪ ⎩ 𝑚𝑖𝑛 𝑥⩽𝑚𝑖𝑛 𝑚𝑎𝑥 𝑥⩾𝑚𝑎𝑥 𝑥otherwise. (4) The exponent 𝑥 is computed based on Eq. (3). The 𝑅𝑜𝑢𝑛𝑑 function is defined as rounding to the nearest integer, and the clipping function in Eq. (4) forces the exponent values to be in the range of [𝑚𝑖𝑛, 𝑚𝑎𝑥], where 𝑚𝑖𝑛 = −(2𝑛−1)and 𝑚𝑎𝑥 = (2𝑛−1 − 1). Assuming an n-bit exponential quantization (e.g. 𝑛= 4), the number of unique intervals is 2𝑛− 1. We store an extra bit for the sign of the value, but in most layers it is not necessary since the activations are all positive. The 𝑚𝑖𝑛 exponent is also used as a special case to represent the exactly zero activation value, so all small activations are effectively pruned due to the clipping. The main benefit of the LOG2 quantization is that it not only reduces the numerical precision but also eliminates the bulky digital multipliers by using simple shift and ADD operations. The approximated activation values 𝑥 are stored as exponents to reduce the memory pressure and the computational complexity. Eq. (5) shows the transformed dot-product operation with the bit-shifting of 𝑤𝑖weights by the 𝑥𝑖exponents of the base-2 powers representing the activations, where 𝑥𝑖is quantized to an integer exponent using Eq. (2). Note that the positive exponents will lead to a shift to the left, while negative exponents result in a shift to the right. 𝑤𝑇𝑥= 𝑛 ∑ 𝑖=1 𝑤𝑖×𝑥𝑖≃ 𝑛 ∑ 𝑖=1 𝑤𝑖× 2𝑥𝑖 = 𝑛 ∑ 𝑖=1 𝐵 𝑖𝑡𝑠ℎ𝑖𝑓 𝑡(𝑤𝑖, 𝑥𝑖) (5) In order to further exploit the LOG2 quantization of the input activations, a key observation is that, if a given activation is represented with a base-2 power of a negative exponent, the bit-shifting to the right will discard the least significant bits (LSB) of the weights that are multiplied by the corresponding activation. In other words, during Fig. 2. Histograms of the LOG2 Quantization (LogQuant) of activations from all the FC and CONV layers of AlexNet, Transformer, PTBLM, BERT-Base and BERT-Large. Fig. 3. Estimated memory savings for our set of DNNs. the right bit-shift operation, and assuming that weights are uniformly quantized to 8 bits, only 1⩽8 −|𝑥|⩽7bits of the weights are required while the rest can be avoided, reducing the memory accesses at a fine granularity. To demonstrate the potential of this idea, we perform an analysis of the exponents resulting from a LOG2 4-bit quantization of the activations in all FC/CONV layers of a popular set of DNNs from different domains. All the evaluated networks have been re-trained, reducing the accuracy loss after quantization to less than 1% in all cases. Fig. 2shows the distribution of the non-zero, quantized activations. On average, more than 71% of the activations have negative exponents. PTBLM (98%), BERT-Base (82%), and BERT-Large (85%) have a similar distribution of exponents with a high concentration of negative values centered around −3, while the Transformer (57%) and AlexNet (36%) have the most symmetric distribution resulting in the lowest amount of negative exponents. QeiHaN, our proposed solution for efficient DNN inference, is based on exploiting this observation. We define the estimated memory savings as the percentage of bits from the weights that can be ignored because the negative exponents of the base-2 activations render those bits useless when performing the bit-shifting operation. Fig. 3shows that the memory savings are directly related to the histograms of the quantized activations. On average, 25% of the memory accesses can be avoided. In addition, zero-activations are pruned in both, the baseline and our proposal, further reducing memory accesses. However, the conventional storage of weights in-DRAM is not suitable to exploit this optimization. The following section describes how to re-organize the weights in-memory to take full advantage of the LOG2 quantization. 4. QeiHaN accelerator This section describes the hardware support required to implement QeiHaN. First, we present the main hardware components of the QeiHaN accelerator. Next, we describe the memory organization of weights and activations. Finally, we show how FC and CONV layers are executed in the accelerator using QeiHaN with an enhanced input stationary dataflow. Journal of Systems Architecture 159 (2025) 103320 5 B. Khabbazan et al. Fig. 4. Architecture of the QeiHaN accelerator including the organization of a single Processing Element (PE). 4.1. Architecture The goal of QeiHaN is to optimize the memory pressure by performing an implicit in-memory bit-shifting of the weights in the FC and CONV layers of different DNNs. QeiHaN leverages a large number of negative exponents after the LOG2 quantization of activations, and an efficient weight storage scheme, to save memory accesses. Similar to Neurocube [13] and TETRIS [14], QeiHaN is based on NDP architectures [10,39] that leverage 3D stacked memory for highperformance, low-energy DNN inference. As described in Section 2, the 3D memory consists of multiple DRAM dies connected via TSVs to a logic die. DRAM dies are divided into vertical partitions named vaults that resemble conventional DDRx channels, which can operate independently. In addition, each vault is connected to a tile in the logic die to perform arithmetic computations on the stored data. Fig. 4shows a high-level schematic of the QeiHaN architecture. Each tile in the logic die consists of a single PE, a Vault Controller (VC), a Router (R), and a PE Controller (PEC). The VC manages all the memory operations within the corresponding vault. The router provides local access between a given PE and its related vault, as well as remote access to the other vaults/PEs through a 2D mesh network. In addition, the PEC orchestrates the communication between the PE and the router by generating the addresses of the required data in each PE. Finally, the PE is the core of the tile, and is responsible for accelerating the DNN operations. The main components of a PE include the blocks of SRAM used for storing the inputs (IB), outputs (OB), and weights (WB), the LOG2 Quantization (LOG2-Quant) unit, the Weight Decoder and Shifter (D&S) unit, the ADD array, and the Special Function Unit (SFU). Below is a detailed description of each component: Memory Buffers: Each PE in the logic die has three individual onchip SRAM buffers to store and reuse the data fetched from the main memory according to the dataflow of the accelerator. First, a small Input Buffer (IB) stores blocks of input FP16 activations until filling the whole buffer space. Second, an Output Buffer (OB) stores the partial and final results that are produced during the execution of a DNN layer. Third, a Weights Buffer (WB) keeps the required bits of the weights for the bit-shifting operations. All the SRAM memories are double buffered to load data from main memory while performing computations, avoiding stalls in the pipeline, and highly multi-banked to achieve the bandwidth required to feed a large number of functional ADD units. In addition, all these buffers are sized considering the worst case scenarios, that is, the biggest layer for the I/O buffer, and all the 8 bits of 𝑀weights for the WB, where 𝑀is the bus size of a vault in the 3D-stacked memory. LOG2-Quant Unit: This unit is in charge of the LOG2 quantization of input activations from FP16 to base-2 exponential values according to Eq. (2).Fig. 5shows the hardware required to compute the 𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2(|𝑥|)) function of Eq. (3). Unlike previous works that use relatively complex hardware [44–46], we implement this function with a very simple scheme. In particular, we perform a comparison between Fig. 5. Hardware implementation of 𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2|𝑥|). the fractional part of the value |𝑥|and the √2using a simple comparator. The standard half precision (FP16) format of a value 𝑥is encoded with a sign bit, mantissa 𝑚, and exponent 𝑒. The exponent 𝑒is already expressed as an integer in base-2 format, so the LOG2 function of |𝑥|can be implemented by applying the logarithm on the mantissa 𝑚as shown in Eq. (6). Taking into account the hidden bit of the mantissa, 𝑚is always a value between [1, 2). Therefore, the term 𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚)can be further simplified by Eq. (7). In the next step, each quantized value 𝑥 in QeiHaN is represented by a 4-bit exponent through a clipping function (i.e. Eq. (4)), resulting in a range of [−8,7]. An extra bit may be used for the actual sign of the activations, except for when all are known to be positive due to the ReLU activation function. In addition, all zero activations will skip the quantization and all the related computations and memory accesses. Similarly, all the small activations clipped to −8 will be effectively pruned (rounded to zero). Finally, each quantized activation is sent to the D&S unit for further processing. 𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2|𝑥|) =𝑒+𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚)(6) 1⩽𝑚 <2←←→ 0⩽𝑙 𝑜𝑔2𝑚 <1⇒𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚) ={0𝑚 <√2 1𝑚⩾√2(7) Weight Decoder & Shifter Unit (D&S): The weights that multiply nonzero activations are decoded from a compressed stream and bit-shifted by appending the necessary amount of zeros based on the exponent from the LOG2-Quant unit. According to the exponent value 𝑥, the PE controller determines the required bits of the weights that have to be loaded from DRAM and stored into the WB. A non-negative exponent requires loading all 8 bits of each weight, and the D&S unit shifts the weights 𝑥 positions to the left before sending the results to the ADD array. Otherwise, we only need to fetch the 8 −|𝑥|MSBs of the weights. For example, given a negative exponent 𝑥 = −3, only the 5 MSBs of each weight are loaded into the WB. Then, the D&S unit reads the selected bits of the weights from the WB and generates a set of 16-bit 𝑑values, where 𝑑is the amount of adders in the ADD array. Note that the bit-shifted weights are the result of the traditional multiplication of activations and weights. In order to use this unit efficiently, QeiHaN reorganizes the weights in-memory to a bit-level granularity as described below in Section 4.2. ADD Array: This array is made of 𝑑independent ADD units that are used to accumulate the products of each activation by the corresponding weights. According to the sign of the activation value, not the exponent, the bit-shifted weight is added/subtracted to/from the partial outputs computed in previous cycles and stored in the OB. The LOG2 quantization removes the need for any multiplier, so the partial outputs are loaded from the OB and the bit-shifted weights come from the D&S. As a result, in a single execution all the adders compute partial outputs related to the same input activation from 𝑑different convolutional kernels or output neurons. Special Function Unit (SFU): The SFU is composed of units to perform non-linear activation functions, pooling, and normalization, among others. These functions are usually applied to the final outputs of the FC/CONV layers at the end of their execution, and tend to require more numerical precision in order not to lose accuracy. Thus, QeiHaN de-quantizes the resulting 16-bit integer outputs back to FP16 before using those functions. The non-linear functions are implemented with Look-Up-Tables (LUTs). Journal of Systems Architecture 159 (2025) 103320 6 B. Khabbazan et al. Fig. 6. Organization of the I/O Buffer. 4.2. Memory organization This section describes the memory organization of the accelerator, which refers to the data layout of weights and activations inside the DRAM of each vault and the on-chip buffers of the PEs. To illustrate it, the top of Fig. 6shows an example of a small CONV layer with an input feature map (IFM) size of four channels (IC1–IC4), and an output feature map (OFM) size of two channels (OC1–OC2). On the other hand, the bottom of Fig. 6shows how the input/output activations of the different channels are partitioned and distributed among the I/O buffers of two different PEs/Vaults. In QeiHaN, the input activations are divided channel-wise across all vaults, that is, all inputs of a given channel are stored in the same vault. In contrast, each vault allocates a portion of the corresponding partial outputs of all the channels. In CONV layers, the dimensionality of the inputs/outputs may be quite large, so we employ a blocking scheme to reduce the on-chip storage requirements by segmenting the IFM and OFM into 𝑁blocks or tiles per channel. The I/O Buffer only stores a subset of blocks for each assigned channel of the IFM and OFM, the block size being significantly smaller than the dimensions of the feature maps. Note that each Vault/PE is working on a different set of inputs but producing partial outputs of the same OFM channels. Hence, a reduction is required at the end of the execution to obtain the final outputs. Likewise, FC layers are a special case of CONV, where there is just a single block and input per channel (i.e. 𝑁= 1). Fig. 7shows the layout of 𝑀filters or kernels with 𝑃weights per channel each in the DRAM dies of each vault, where each partition includes 4 banks, for the example of Fig. 6. Similar to the activations, the weights of each kernel are also distributed channel-wise across all vaults. The bits of the weights of the corresponding channels are interleaved in the different banks and partitions of the same vault. That is, the least-significant bit (LSB) of a subset of weights is stored in the first bank of a vault, then the next bit in the second bank and so on. This layout simplifies the implementation of our implicit bit-shifting scheme, as it is easy to locate all the bits of the weights that are required to operate with a given input in case some have to be skipped and others accessed. In addition, most 3D-stacked DRAM-based operations use aClosedPage Policy to reduce power consumption [47]. Consequently, applications benefit from Bank-Level Parallelism but not from spatial locality. QeiHaN remaps the data to avoid internal organization bottlenecks and, hence, requests to different banks can be concatenated/overlapped by the simultaneous activation of the same row across the memory banks within a vault. This overlapping effectively hides the latency associated with activating another row while reading data from a different bank, it balances the load, prevents bottlenecks, and ensures the efficient utilization of memory’s bandwidth. Note that weights are known statically so their organization can be pre-arranged offline. Fig. 7. Organization of weights in the 3D-stacked memory. 4.3. Dataflow Neurocube [13] follows an output stationary (OS) dataflow in which each PE computes a subset of outputs at a time. This dataflow is inefficient to exploit the resources of the 3D memory, as demonstrated by our results in Section 6. On the other hand, QeiHaN uses an enhanced input stationary (IS) dataflow coupled with a blocking scheme to efficiently exploit the LOG2 quantization of the input activations, minimizing the memory accesses to both weights and activations. Fig. 8 illustrates the dataflow of the QeiHaN accelerator with a flowchart. The proposed dataflow includes three main stages marked in different colors: Pre-Processing (Gray), Execution (Orange), and Post-Processing (Blue). In the Pre-Processing stage, each PE reads input activations from DRAM until filling the input buffer space. That is, inputs (outputs) are pre-loaded (processed) on-demand by blocks, activations are stored in FP16 format, and the size of the blocks is computed according to the feature map sizes and the I/O buffer capacity. In the IS dataflow, each PE of the accelerator fetches and processes one input of a block at a time from the I/O buffer, and performs all the associated computations before moving to the next input. First, the LOG2 quantization and clipping function is applied to obtain the 4-bit exponent 𝑥. Then, QeiHaN also performs a zero and small activation pruning. Concurrently, the reading of input blocks from DRAM continues in the background, as long as there is space in the buffers, to hide the memory latency while doing computations of the current blocks. In the Execution stage, and based on the value of the exponent 𝑥, a set of 𝑀useful bits of INT8 uniformly quantized weights of 𝑀different kernels related to the input are read from DRAM at a time, where 𝑀 is determined by the internal 3D-stacked memory bus size (e.g. 32-bit). Thus, in each request, the bits in the same position for 𝑀different weights are loaded into the weights buffer, and multiple requests are made until all the required bits are retrieved. Next, the bits of the weights are decoded and bit-shifted by appending the corresponding zeros, resulting in 16-bit integer values. These results are grouped and sent to the ADD array unit in batches of 𝑑values, where 𝑑is the number of adders (e.g. 16). In parallel, the partial outputs from previous executions are loaded from the output buffer. Then, the accelerator performs 𝑑ADDs to accumulate the results of each output with the shifted weights, followed by the write-back to the output buffer. This stage is repeated until all the weights of all filters related to the current activation are processed. Journal of Systems Architecture 159 (2025) 103320 7 B. Khabbazan et al. Fig. 8. Dataflow and Execution scheme. Table 1 DNNs employed for the experimental evaluation of QeiHaN. The model size accounts for the parameters in FP32 and INT8 of the FC/CONV layers where the LOG2 quantization is applied. The accuracy shown is after quantization. DNN model FP32-Size (MB) INT8-Size (MB) Accuracy (QeiHaN/FP32) AlexNet [15] 144 36 57.05/57.48% (Top-1) PTBLM [17] 136 34.2 79.40/78.09 (Perplexity) Transformer [18] 336 84 28.28/28.44 (BLEU-uncased) BERT-Base [19] 440 110 86.75/86.96% (F1) BERT-Large [19] 1320 330 89.72/89.86% (F1) Finally, in the Post-Processing stage, QeiHaN reduces the partial outputs of each PE. The reduction starts as soon as enough activations complete all their operations. Then, in a centralized PE, the final results are de-quantized, and the SFU performs the activation and pooling operations before distributing and storing the corresponding activations back to each vault. After processing all the blocks of inputs the layer execution is completed. Note that all the main steps are carried out in parallel in a deep pipeline. 5. Methodology This section presents the methodology for evaluating QeiHaN, our NDP accelerator for DNN inference. Workloads. Our objective is to prove that our scheme provides important savings for multiple applications and different DNN models. To this end, we evaluate QeiHaN on five state-of-the-art DNN workloads from different domains, summarized in Table 1. Their model sizes range from medium to large scale with several hundreds of MBytes in memory footprint. In particular, we include the ILSVRC 2012 winner, AlexNet [15] (5 CONV and 3 FC layers), one of the most popular CNNs for image classification with the ImageNet dataset, and PTBLM [17] (2 LSTM layers), an RNN that consists of LSTM cells for language modeling using the Penn Treebank dataset. In addition, we employ three attention-based networks: Transformer (6 Encoders, 6 Decoders), BERTBase (12 Encoders, 110M Parameters), and BERT-Large (24 Encoders, 340M parameters). The Transformer [18] model is evaluated on the machine translation task of Newtest2014 (English to German) which contains 3003 sentences. BERT-Base [19], and its larger variant BERTLarge, are evaluated on the question-answering task of SQuADv1 [48]. Finally, all these networks have been re-trained in order to recover the accuracy after quantization, that is, less than 1% loss. Accuracy is reported as Top-1 for image classification (higher is better), perplexity for language modeling (lower is better), bilingual evaluation understudy (BLEU) for machine translation (higher is better), and weighted average of the precision and recall (F1) for question-answering (higher is better). System models and simulation. We have developed a simulator that accurately models three different systems, QeiHaN and two baseline accelerators. The first baseline is inspired in Neurocube [13], described in Section 2, but with some optimizations, such as a lower quantization bitwidth, to isolate the effects of our proposal when comparing the two. The second baseline, named NaHiD, implements the same architecture, dataflow, and quantization scheme as QeiHaN but with a standard memory organization of the weights. That is, NaHiD also replaces multiplications by bit-shift operations and additions but, in contrast to QeiHaN, it requires loading all the bits of the weights from memory. This comparison allows us to infer the main benefits due to the QeiHaN’s efficient 3D memory-centric weight storage scheme. Among the related works described in Section 2, we only compare against Neurocube because TETRIS has a similar HW and dataflow for FC layers. Most of the TETRIS improvements come from optimizing the dataflow of CONV layers in CNNs. Note that the majority of modern DNNs are mainly composed of FC layers, hence, we would expect the comparison with TETRIS to have minor impact in our results. In all three systems, We focus on accelerating FC and CONV layers that use static weights, as these are key components of DNN workloads. Matrixmatrix multiplication layers, which are not explicitly targeted by our approach, are assumed to be processed on the host system using FP16 precision. Table 2shows the parameters of the experiments. For a fair comparison, we set most of the configuration parameters to match the Neurocube baseline: a 3D-stacked memory of 4 GB with 4 DRAM dies partitioned into 4×4 vaults and PEs, an internal 3D memory bandwidth of 10 GB/s per vault, about 2.5 kB of SRAM per PE, 16 MAC/ADD units per PE, and a frequency of 300 MHz in the logic die. QeiHaN and NaHiD require slightly smaller memory buffers (i.e. 2 kB of OB, 64B of IB, and 64B of WB) due to the different dataflow. Regarding area and energy consumption evaluation, the logic components are implemented in Verilog, including all the additional components required by QeiHaN, and synthesized to obtain the delay, area, and power using the Synopsys Design Compiler [49], the modules of the DesignWare library and the technology library of 28/32 nm from Synopsys. On the other hand, we characterize the memory buffers of the accelerator by obtaining the delay, energy per access, and area using CACTI-P [50]. We use the configurations optimized for low power and a supply voltage of 0.78 V for three systems. Finally, the energy consumption of the 3D-stacked memory is estimated by using an HMC configuration of DRAMSim3 [51]. The results obtained with the aforementioned tools are combined with the activity factors and memory traces provided by our simulator to obtain the dynamic and static power of the accelerators. 6. Evaluation This section evaluates the performance, energy efficiency, and memory activity of our proposal. First, we introduce an analysis of the total number of memory accesses to the 3D-stacked DRAM dies after applying the QeiHaN scheme. Then, we present the speedups and energy Journal of Systems Architecture 159 (2025) 103320 8 B. Khabbazan et al. Table 2 Parameters for the accelerators. Common parameters Technology 32 nm Logic die frequency 300 MHz #DRAM dies, #Banks per vault per die 4 #Vaults, #PEs 16 3D-stacked memory total size 4 GB 3D-stacked memory bandwidth per vault 10 GB/s Neurocube parameters Dataflow OS Weights precision 8-bit Input activations precision 8-bit Weights/inputs quantization Uniform/Uniform #MAC units per PE 16 Total SRAM buffers size per PE 2.5 kB NaHiD & QeiHaN parameters Dataflow IS Weights precision 8-bit Input activations precision 4-bit Weights/inputs quantization Uniform/LOG2 #ADD units per PE 16 Total SRAM buffers size per PE 2.1 kB savings achieved by QeiHaN compared to the Neurocube and NaHiD baselines. Finally, we discuss the accelerator overheads. Notably, the averages presented for each evaluation are calculated using the arithmetic mean. These averages primarily serve a summarization purpose, but the results for each model type should be examined individually, as they provide a more accurate reflection of performance for specific workloads. 6.1. 3D-stacked memory accesses Fig. 9reports the normalized total 3D memory accesses of QeiHaN over the two baseline accelerators. This total includes both, memory accesses for reading/writing the weights and the input activations. On average for our set of DNNs, QeiHaN reduces the total DRAM accesses by 72.4% and 25% over Neurocube and NaHiD, respectively. The great reduction of memory accesses with respect to the baselines is mainly due to constraining the accesses to only the required bits of the weights for the bit-shifting operations. Moreover, QeiHaN shows a higher reduction of memory accesses over Neurocube due to two main reasons. First, the enhanced IS dataflow of QeiHaN requires each input activation to be accessed just once during the execution of a layer. In contrast, the OS dataflow of Neurocube may require multiple accesses to the activations. Second, QeiHaN performs pruning of zero and small activations after applying the quantization, removing all the related memory accesses to the weights. The efficiency of the activation pruning is limited in Neurocube due to its OS dataflow, so it is not implemented. On the other hand, compared to NaHiD, the reduction is well correlated to the estimated memory savings due to the huge amount of negative exponents as discussed in Section 3. Both QeiHaN and NaHiD use the same dataflow and pruning scheme, so both access the same input activations, and the savings come from the weights. 6.2. Performance Fig. 10 shows the speedups achieved by QeiHaN. Compared to Neurocube, QeiHaN provides consistent speedups for the five DNNs that range from 8.69𝑥(AlexNet) to 1.24𝑥(Transformer), achieving an average performance improvement of 4.25𝑥. The reduction in execution time is due to QeiHaN’s efficient memory organization and enhanced IS dataflow. The number of memory accesses is dramatically reduced since only the meaningful bits of the weights required by the shift operations are loaded. In addition, QeiHaN employs a novel weight Fig. 9. Normalized total memory accesses for each DNN. Fig. 10. Speedups of QeiHaN over Neurocube &NaHiD. storage scheme to exploit the bank-level parallelism of the 3D memory. Moreover, QeiHaN overlaps the different stages of the dataflow in a deep pipeline, shortening the critical path of the execution. As shown in Fig. 9,AlexNet and PTLBM exhibit the highest reduction in memory accesses and, hence, they obtain the largest performance improvements. The difference in speedup between these two networks and the attention-based models is in the percentage of zero and small activations that are effectively pruned in QeiHaN, skipping part of the execution and post-processing stages. The effect of activation pruning is minor in Transformer (3%), BERT-Base (7%), and BERT-Large (13%), but significant in AlexNet (47%) and PTLBM (55%). Compared to NaHiD, the benefits of QeiHaN are more modest but still quite important, achieving an average speedup of 1.38𝑥. The main reason is that both accelerators benefit from the same architecture, dataflow, quantization, and activation pruning scheme. Therefore, the improvements come mainly from the novel memory layout for storing the weights in the 3D memory, and the corresponding reduction of memory accesses by leveraging the logarithmic quantization. PTBLM obtains the largest benefits, achieving an speedup of 1.86𝑥whereas AlexNet gets the lowest improvements, that is, 1.07𝑥speedup. These results are directly proportional to the percentage of negative exponents shown in Fig. 2. 6.3. Energy consumption Fig. 11 reports normalized energy savings. On average, QeiHaN reduces the energy consumption of the accelerator by 3.52𝑥and 1.28𝑥 over Neurocube and NaHiD, respectively. As we observed for performance, the energy savings are well correlated with the number of negative exponents and the corresponding reduction of memory accesses. These energy savings are due to two main reasons. First, dynamic energy is reduced due to the savings in multiplications and memory accesses. Second, the performance improvements shown in Fig. 10 provide a reduction in static energy. Again, PTBLM obtains the largest benefits, achieving a reduction of 8.2𝑥and 1.6𝑥in energy compared to both Neurocube and NaHiD respectively. Fig. 12 shows the energy breakdown of QeiHaN and NaHiD over Neurocube. The figure shows results for the five neural networks including the percentage of energy consumed by each major hardware block of the accelerators. As can be seen, the DRAM of the 3D-stacked memory (i.e. HMC) consumes most of the energy in all cases. The energy savings achieved by our proposal are significant, and are especially Journal of Systems Architecture 159 (2025) 103320 9