Con en s lis s a ailable a ScienceDi ec
Jou nal o Sys ems A chi ec u e
jou nal homepage: www.else ie .com/loca e/sysa c
An ene gy-e icien nea -da a p ocessing accele a o o DNNs o op imize
memo y accesses✩
Baha eh Khabbazan∗,Mohammad Sab i,Ma c Rie a,An onio González
Uni e si a Poli ècnica de Ca alunya (UPC), Ba celona, Spain
A R T I C L E I N F O
Keywo ds:
DNN
NDP
Accele a o s
Quan iza ion
Exponen ial
T ans o me
A B S T R A C T
The cons an g ow h o DNNs makes hem challenging o implemen and un e icien ly on adi ional compu e-
cen ic a chi ec u es. Some accele a o s ha e a emp ed o add mo e compu e uni s and on-chip bu e s o
sol e he memo y wall p oblem wi hou much success, and some imes e en wo sening he issue since mo e
compu e uni s also equi e highe memo y bandwid h. P io wo ks ha e p oposed he design o memo y-
cen ic a chi ec u es based on he Nea -Da a P ocessing (NDP) pa adigm. NDP seeks o b eak he memo y
wall by mo ing he compu a ions close o he memo y hie a chy, educing he da a mo emen s and hei
cos as much as possible. The 3D-s acked memo y is especially appealing o DNN accele a o s due o i s
high-densi y/low-ene gy s o age and nea -memo y compu a ion capabili ies o pe o m he DNN ope a ions
massi ely in pa allel. Howe e , memo y accesses emain as he main bo leneck o unning mode n DNNs
e icien ly.
To imp o e he e iciency o DNN in e ence we p esen QeiHaN, a ha dwa e accele a o ha implemen s
a 3D-s acked memo y-cen ic weigh s o age scheme o ake ad an age o a loga i hmic quan iza ion o
ac i a ions. In pa icula , since ac i a ions o FC and CONV laye s o mode n DNNs a e commonly ep esen ed
as powe s o wo wi h nega i e exponen s, QeiHaN pe o ms an implici in-memo y bi -shi ing o he DNN
weigh s o educe memo y ac i i y. Only he meaning ul bi s o he weigh s equi ed o he bi -shi ope a ion
a e accessed. O e all, QeiHaN educes memo y accesses by 25% compa ed o a s anda d memo y o ganiza ion.
We e alua e QeiHaN on a popula se o DNNs. On a e age, QeiHaN p o ides 4.3𝑥speedup and 3.5𝑥ene gy
sa ings o e a Neu ocube-like accele a o .
Con en s
1. In oduc ion ...................................................................................................................................................................................................... 2
2. Backg ound & Rela ed wo k ............................................................................................................................................................................... 3
2.1. Mode n DNNs ........................................................................................................................................................................................ 3
2.2. DNN quan iza ion................................................................................................................................................................................... 3
2.3. Da a lows in DNN accele a o s ................................................................................................................................................................ 3
2.4. 3D-s acked memo y................................................................................................................................................................................ 3
2.5. 3D-s acked DRAM-based DNN accele a o s ............................................................................................................................................... 4
3. LOG2 quan iza ion analysis ................................................................................................................................................................................ 4
4. QeiHaN accele a o ............................................................................................................................................................................................ 5
4.1. A chi ec u e........................................................................................................................................................................................... 6
4.2. Memo y o ganiza ion.............................................................................................................................................................................. 7
4.3. Da a low................................................................................................................................................................................................ 7
5. Me hodology ..................................................................................................................................................................................................... 8
6. E alua ion......................................................................................................................................................................................................... 8
6.1. 3D-s acked memo y accesses ................................................................................................................................................................... 9
6.2. Pe o mance........................................................................................................................................................................................... 9
6.3. Ene gy consump ion ............................................................................................................................................................................... 9
✩New Pape , No an Ex ension o a Con e ence Pape .
∗Co esponding au ho .
E-mail add ess: [email p o ec ed] (B. Khabbazan).
h ps://doi.o g/10.1016/j.sysa c.2024.103320
Recei ed 11 July 2024; Recei ed in e ised o m 25 No embe 2024; Accep ed 5 Decembe 2024
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
A ailable online 12 Decembe 2024
1383-7621/© 2024 Else ie B.V. All igh s a e ese ed, including hose o ex and da a mining, AI aining, and simila echnologies.
B. Khabbazan e al.
6.4. A ea...................................................................................................................................................................................................... 10
6.5. Compa ison wi h a TPU-like accele a o ................................................................................................................................................... 10
7. Conclusions ....................................................................................................................................................................................................... 10
CRediT au ho ship con ibu ion s a emen ........................................................................................................................................................... 10
Decla a ion o compe ing in e es ........................................................................................................................................................................ 10
Acknowledgmen s .............................................................................................................................................................................................. 10
Da a a ailabili y ................................................................................................................................................................................................ 10
Re e ences......................................................................................................................................................................................................... 11
1. In oduc ion
Deep Neu al Ne wo ks (DNNs) ep esen he s a e-o - he-a solu ion
o a b oad ange o machine lea ning applica ions such as na u al
language p ocessing (NLP) and image classi ica ion. Mode n DNNs can
ou pe o m human-le el accu acy in many o hese applica ions a
he expense o high compu a ional cos , memo y equi emen s, and
ene gy consump ion. Complex DNN models a e composed o hund eds
o laye s o a i icial neu ons wi h billions o model pa ame e s and
ope a ions. The cons an g ow h o DNNs makes hem challenging o
implemen and un e icien ly, e en in he mos ecen accele a o s [1]
based on adi ional compu ing a chi ec u es due o he memo y wall
p oblem. On he o he hand, some ecen esea ch has ocused on a
new pa adigm named Nea -Da a P ocessing (NDP) [2,3], which seeks
o b eak he memo y wall by mo ing he compu a ions close o he
memo y hie a chy.
Con en ional DNN accele a o s dedica e a signi ican pa o hei
a ea o p ocessing elemen s (PEs) o accele a e he equen do -p oduc
ope a ions in DNN laye s. Many p e ious designs exploi da a and
h ead-le el pa allelism h ough la ge PE a ays, which u he ampli-
ies memo y bandwid h demands, o en c ea ing a bo leneck in eeding
da a o he PEs. Despi e e o s o educe o -chip memo y accesses
and imp o e on-chip da a euse, he memo y wall con inues o be
a majo cons ain in compu e-cen ic a chi ec u es. Da a mo emen ,
which accoun s o 62.7% o o al ene gy consump ion acco ding o
ecen s udies, emains he dominan ac o in ene gy o e head, a
su passing he ene gy equi ed o compu a ions. Addi ionally, much o
his da a ans e is igge ed by simple ope a ions and p imi i es ha
could be e icien ly handled wi h low-cos ha dwa e implemen a ions.
These obse a ions, along wi h he inc easing size o DNN models,
mo i a e he ansi ion om con en ional compu e-cen ic o da a-
cen ic a chi ec u es o mo e e icien p ocessing o da a-in ensi e
wo kloads.
O e he las ew yea s, esea che s ha e been explo ing no el
memo y-cen ic a chi ec u es based on he so-called Nea -Da a P o-
cessing (NDP) pa adigm o accele a e neu al ne wo k algo i hms by
mo ing mos o he compu a ions ’’in/nea -memo y’’ and, hence, e-
ducing he da a mo emen s and hei cos as much as possible [4,5].
NDP has gained a lo o a en ion wi h he in oduc ion o he 3D
s ack memo y echnology, which allows he in eg a ion o logic and
memo y in he same chip, p o iding high-speed connec ions be ween a
high-densi y memo y and a logic die. Mic on’s Hyb id Memo y Cube
(HMC) [6], High Bandwid h Memo y (HBM) [7] om AMD/Hynix,
and Samsung’s Wide I/O [8] a e popula examples implemen ing his
ending echnology.
NDP a chi ec u es based on 3D-s acked memo y a ack he memo y
wall by inc easing s o age capaci y, memo y bandwid h, and educing
powe consump ion [9]. Compa ed wi h he con en ional 2D DRAM,
3D memo y p o ides an o de o magni ude highe bandwid h (160
o 250 GBps) wi h up o 5𝑥be e ene gy e iciency and, hence, 3D
memo y is an excellen op ion o mee ing he high h oughpu , low
ene gy equi emen s o scalable DNN accele a o s [5,10,11]. All 3D-
s acked memo y sys em implemen a ions p o ide highly pa allel access
o memo y which is well sui ed o he highly pa allel a chi ec u e o he
DNN accele a o s [2,12].
Neu ocube [13] and TETRIS [14] a e popula NDP 3D-s acked
memo y a chi ec u es ha o e p omising pe o mance and ene gy
consump ion o accele a ing DNNs. Howe e , he e is s ill la ge oom
o imp o emen , since hese a chi ec u es and memo y echnology
p esen mul iple challenges o ex end hei adop ion. Fi s , a chi ec-
u es based on 3D-memo y equi e o e hink o he design o on-chip
bu e s in he logic die as well as he loca ion whe e he compu a ions
a e execu ed. Fo example, pe o ming simple ope a ions on he DRAM
dies can d as ically educe he amoun o memo y mo emen s and
he need o big on-chip bu e s. Second, new app oaches o da a low
scheduling and pa i ioning o he DNN compu a ions a e also equi ed
o educe he memo y p essu e. Thus, changing he memo y o ganiza-
ion and da a placemen can ully exploi he ea u es o 3D-s acked
a chi ec u es. In addi ion, he a ea o he logic die is cons ained by
he package, and he e a e igh he mal cons ain s ha limi he
powe dissipa ion o he sys em. Consequen ly, i is c i ical o p opose
solu ions ha imp o e in hese aspec s.
In his pape , we show how o e icien ly exploi a loga i hmic base-
2 quan iza ion (LOG2) o ac i a ions on FC and CONV laye s o ypical
DNN models o educe memo y mo emen s due o weigh s. Fi s , we
pe o m an analysis o he exponen s ob ained a e he LOG2 quan-
iza ion, and obse ed ha a huge pe cen age o he ac i a ions a e
ep esen ed wi h nega i e exponen s, ha is, hei o iginal alue is in
he ange o [−1, 1]. LOG2 quan iza ion has been p oposed in p e ious
wo ks o educe he nume ical p ecision o ei he ac i a ions/weigh s
and exploi ed o subs i u e mul iplica ions by a bi -shi ing o he
o he ope and. Based on hese obse a ions we p opose an implici
in-memo y bi -shi ing o he DNN weigh s o educe he memo y
mo emen s. Weigh s a e uni o mly quan ized and s o ed a he bi -le el
g anula i y in o di e en memo y egions, ha is, each bi o a se o
weigh s is s o ed in a di e en memo y bank o exploi he inhe en
pa allelism o 3D-s acked a chi ec u es. Nex , we p opose a mechanism
o a oid accessing he bi s o he weigh s ha a e no use ul due o
he igh bi -shi ing o he nega i e exponen s o he loga i hmically
quan ized ac i a ions.
Then, we p esen QeiHaN, a no el NDP accele a o ha implemen s
he abo e LOG2 quan iza ion-shi ing engine and e icien weigh s o -
age scheme o high-pe o mance low-ene gy DNN in e ence. QeiHaN
is implemen ed on op o a Neu ocube-like a chi ec u e, bu ex ended
wi h an enhanced inpu s a iona y da a low. The ex a ha dwa e e-
qui ed o ou echnique is modes since mos o he componen s a e
al eady a ailable in he baseline. QeiHaN only equi es a small se
o addi ional compa a o s and in ege adde s o pe o m he LOG2
quan iza ion. Then, we also eplace he mul iplie s by simple bi -shi
logic, educing he compu a ional cos and he o e all a ea o he
PEs. Ou expe imen al esul s show ha he o e heads a e minimal
compa ed o he sa ings in memo y accesses and mul iplica ions.
To summa ize, his pape ocuses on e icien DNN in e ence le e -
aging loga i hmic quan iza ion in NDP 3D-s acked DRAM-based accel-
e a o s. The main con ibu ions a e:
•We analyze he dis ibu ion o exponen s o he loga i hmically
(i.e. LOG2) quan ized ac i a ions in mul iple laye s o mode n
DNNs including CNNs, RNNs, and T ans o me s. We obse e ha
a huge pe cen age o he exponen s a e nega i e, leading o
po en ial memo y sa ings as a esul o educing he accesses o
only he use ul bi s o he weigh s.
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
2
B. Khabbazan e al.
•We p opose a no el da a layou and an op imized da a low o
exploi he bank-le el pa allelism o 3D-s acked memo y oge he
wi h he LOG2 quan iza ion o ac i a ions. Each memo y bank
s o es a di e en subse o he bi s o he uni o mly quan ized
weigh s o allow o pa allel accesses o he equi ed bi s o
he bi -shi ing ope a ions. On a e age, we educe he memo y
accesses due o he weigh s by 25% compa ed o a s anda d
memo y o ganiza ion.
•We p esen QeiHaN, a 3D-s acked DRAM-based ha dwa e accel-
e a o ha implemen s ou da a layou and da a low o e icien
DNN in e ence. We e alua e QeiHaN o se e al DNNs. QeiHaN
imp o es pe o mance by 1.4𝑥and educes ene gy consump ion
by 1.3𝑥on a e age o e NaHiD, a baseline accele a o imple-
men ing he same da a low and quan iza ion as QeiHaN bu
wi h a s anda d memo y o ganiza ion o weigh s. Compa ed o
Neu ocube [13], QeiHaN achie es 4.3𝑥speedup and 3.5𝑥ene gy
sa ings on a e age.
The es o he pape is o ganized as ollows. Sec ion 2in oduces
some p elimina ies o QeiHaN and p o ides a summa y o wo ks
ela ed o 3D memo y DNN accele a o s. Sec ion 3discusses he obse -
a ions on he loga i hmic quan iza ion o ac i a ions o a mode n se
o DNNs. Sec ion 4desc ibes he a chi ec u e o QeiHaN including he
implemen a ion de ails o he main ha dwa e componen s. Sec ion 5
p esen s he e alua ion me hodology and Sec ion 6discusses he ex-
pe imen al esul s o QeiHaN on di e en ne wo ks. Finally, Sec ion 7
concludes he pape by summa izing he key insigh s o his design
alongside he o e all pe o mance.
2. Backg ound & Rela ed wo k
In he ollowing subsec ions we e iew some e minology and con-
cep s ha may be help ul h oughou his pape . Fi s , we gi e a gene al
desc ip ion o DNNs, including he main ca ego ies and di e en ypes
o laye s. Nex , we e iew DNN quan iza ion and common da a lows
o DNN accele a o s. Finally, we discuss 3D memo y a chi ec u es,
which o e mo e oppo uni ies o implemen a highly e icien DNN
accele a o in e ms o bo h pe o mance and ene gy consump ion.
2.1. Mode n DNNs
Deep Neu al Ne wo ks (DNNs) all in o h ee main ca ego ies. Fi s ,
Mul i-Laye Pe cep ons (MLPs) consis o mul iple Fully-Connec ed
(FC) laye s whe e each inpu neu on is connec ed, ia synapses wi h
pa icula weigh s, o e e y ou pu neu on. Second, Con olu ional Neu-
al Ne wo ks (CNNs) employ con olu ional laye s o ex ac ea u es,
o en ollowed by one o mo e FC laye s o classi ica ion. Fo ins ance,
AlexNe [15] demons a ed ema kable e iciency in image and ideo
p ocessing. Finally, Recu en Neu al Ne wo ks (RNNs) [16] comp ise
s acked cells wi h eedback connec ions ha s o e in o ma ion om
pas compu a ions o enhance u u e p edic ions. The mos popula
RNN a chi ec u e, Long–Sho Te m Memo y (LSTM), uses FC laye s
a anged in ga es. PTBLM [17], an LSTM-based RNN, is widely applied
in language modeling, speech ecogni ion, and machine ansla ion
asks.
A en ion-based models, such as he T ans o me [18] and BERT
[19] a ian s, ha e become s a e-o - he-a o key asks like na u-
al language p ocessing [20], compu e ision [21,22], ques ion an-
swe ing [23], and ideo analysis [24]. These models use a en ion
mechanisms o ga he con ex ual in o ma ion, implemen ed ia FC
laye s, allowing hem o handle la ge inpu s e ec i ely. Despi e hei
e iciency, hese ne wo ks su e om long execu ion imes due o hei
la ge memo y oo p in and lowe compu a ion- o-memo y access a io
compa ed o con olu ional laye s. FC laye s a e mo e memo y in ensi e
as weigh s a e no eused by di e en neu ons.
Mos DNNs consis p ima ily o FC and con olu ional laye s, which
domina e he compu a ional wo kload. Pooling laye s educe spa ial
dimensions, no maliza ion laye s s abilize aining, and ac i a ion unc-
ions in oduce non-linea i y, con ibu ing o o e all ne wo k pe o -
mance. Howe e , hese laye s ep esen a smalle ac ion o he exe-
cu ion ime. This pape ocuses on op imizing ha dwa e accele a o s o
FC and con olu ional laye s in MLPs, CNNs, RNNs, and T ans o me s.
2.2. DNN quan iza ion
Quan iza ion is a highly popula echnique o map alues om a
con inuous ange o a disc e e se . The main pu pose o quan iza ion
is o comp ess he o iginal DNN models o educe he memo y oo -
p in and he compu a ional cos wi h a mino impac on accu acy.
Eq. (1) shows an example o a unc ion ha quan izes eal alues (in
loa ing-poin , FP, p ecision) and maps hem o an in ege ange.
𝑄(𝑟) =𝐼 𝑁 𝑇(𝑟∕𝑠) −𝑧(1)
whe e 𝑄(𝑟)is he quan ized alue, 𝑟is a FP alue, 𝑠is a scaling ac o ,
and 𝑧is an in ege o se . The 𝐼 𝑁 𝑇 unc ion is a ounding o he nea es
alue. This me hod is also e e ed o as linea uni o m quan iza ion
since he esul ing quan ized alues (a.k.a. quan iza ion le els) a e
uni o mly spaced.
Recen ly, non-uni o m quan iza ion schemes ha e been p oposed
o u he educe he memo y p essu e. These me hods ha e been de-
signed o DNN models wi h enso s ha ha e a bell-shaped long- ailed
dis ibu ion o weigh s and ac i a ions [25,26]. Loga i hm quan iza ion
is an example o a non-uni o m scheme, whe e he quan iza ion le els
inc ease exponen ially ins ead o linea ly [27]. The Loga i hmic Quan-
iza ion (LQ) [28–30] o e s smalle nume ical p ecision (i.e. bi wid h)
wi h lowe accu acy loss compa ed o he linea quan iza ion by ex-
ploi ing he non-uni o m dis ibu ion o enso s. QeiHaN employs uni-
o m quan iza ion o he weigh s and a loga i hmic base-2 (LOG2)
quan iza ion o he ac i a ions o all he FC/CONV laye s. Sec ion 3
p o ides mo e de ails on he LOG2 quan iza ion and i s main bene i s.
2.3. Da a lows in DNN accele a o s
The da a low o a DNN accele a o is de ined as he mapping
and scheduling o he compu a ions as well as he da a pa i ioning
ac oss compu e uni s. The da a low ha is mos e ec i e o educe
he memo y accesses and da a mo emen s o op imize pe o mance
and ene gy e iciency depends on he a ge cogni i e compu ing ask
and ha dwa e a chi ec u e [31]. The da a low de e mines he s o age
equi emen s and communica ion pa e ns among main memo y, local
on-chip bu e s inside PEs, and compu e uni s.
In p e ious wo ks [32,33], he elec ion o he da a low is based
on minimizing he da a mo emen o he inpu s, ou pu s o weigh s.
The e o e, DNN accele a o s end o ollow one o hese da a lows:
Weigh S a iona y (WS), Ou pu S a iona y (OS), and Inpu S a ione y
(IS). In OS, each PE compu es an ou pu neu on a a ime [34]. In
he WS/IS da a lows, each PE p e-loads a se o weigh s/inpu s om
memo y o local bu e s, and hose a e used o pe o m all associa ed
compu a ions [35].
QeiHaN uses an inpu s a iona y da a low, which means ha each
inpu o a gi en laye is ead and eused, un il all he ela ed compu-
a ions a e done, be o e eading he nex inpu . The IS da a low is he
mos sui able o ou loga i hmic quan iza ion o DNN ac i a ions and
e icien weigh s o age scheme. We compa e QeiHaN wi h wo baseline
accele a o s, one wi h OS da a low and he o he wi h IS da a low.
2.4. 3D-s acked memo y
Con en ional a chi ec u es ely on ex e nal accele a o s (e.g., GPUs,
TPUs) o compu a ion, inc easing ene gy consump ion due o equen
la ge-scale da a ans e s be ween memo y and compu e uni s. These
a chi ec u es also ace limi a ions like low memo y bandwid h and ine -
icien da a euse, making hem less ene gy-e icien o da a-in ensi e
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
3
B. Khabbazan e al.
applica ions, such as DNN in e ence. In con as , NDP a chi ec u es
u ilizing 3D-s acked memo y mi iga e hese ine iciencies by b inging
compu a ion close o memo y. By educing da a mo emen , NDP
a chi ec u es imp o e bo h ene gy e iciency and pe o mance.
High-densi y 3D memo y is a p omising echnology o he mem-
o y sys em o DNN and o he domain-speci ic accele a o s [36,37].
I consis s o s acking mul iple memo y dies on op o each o he ,
which inc eases he memo y capaci y and bandwid h compa ed o 2D
memo y, and also educes he access la ency due o he sho e on-chip
wi ing in e connec ion. These aspec s lead o an o e all imp o emen
in bo h ene gy e iciency and pe o mance.
The 3D memo y dies a e commonly based on DRAM, bu he
in eg a ion o o he memo y echnologies is being ac i ely esea ched
wi h e y p omising esul s. On he o he hand, ecen ad ances in low-
capaci ance h ough-silicon ias (TSVs) echnology ha e enabled 3D
memo y ha includes a ew DRAM dies on op o a logic chip, wi hin a
single package [38]. Al hough he e a e nume ous implemen a ions o
3D-s acked memo y echnologies, un il now, he Hyb id Memo y Cube
(HMC) [39] by Mic on and he High Bandwid h Memo y (HBM) [7,40]
om AMD/Hynix a e he p e e ed choices o mos DNN accele a o
p oposals [11,13,14,41].
HBM and HMC a e designed o high pe o mance da a-cen ic
applica ions. Bo h a e composed o e ically s acked DRAM dies wi h
a single logic laye a he bo om. These memo y echnologies ake
ad an age o Th ough-Silicon Vias (TSVs) o enable high-bandwid h
and low-la ency communica ion be ween he s acked memo y laye s.
In HBM, each DRAM die is pa i ioned ho izon ally, and di e en pa i-
ions on di e en dies a e ea ed as independen memo y channels. On
he o he hand, in HMC, each DRAM die is di ided in o mul iple pa -
i ions in a 2D g id whe e he co esponding pa i ions in he e ical
di ec ion o m a single aul . Bo h HBM and HMC can exploi memo y-
le el pa allelism by o ganizing he la ge numbe o TSVs in o mul iple
independen ly-ope a ed channels. This allows mul iple pa i ions in he
DRAM die o be accessed simul aneously, u he enhancing memo y
bandwid h and o e all sys em pe o mance.
NDP sys ems employing HBM o HMC associa e he PEs o he
logic die wi h each channel o aul o e icien ly u ilize he memo y-
le el pa allelism and achie e high da a p ocessing h oughpu . The
choice be ween HBM and HMC would depend on he speci ic equi e-
men s o he NDP sys em, and he desi ed ade-o s be ween memo y
bandwid h, ene gy e iciency, and in eg a ion wi h he hos p ocesso .
2.5. 3D-s acked DRAM-based DNN accele a o s
Neu ocube [13] is a p og ammable DNN accele a o in eg a ed
in o he logic laye o a 3D s ack DRAM-based HMC. The Neu ocube
a chi ec u e consis s o clus e s o p ocessing engines (PE) connec ed
by a 2D mesh NoC in he p ocessing laye . Each PE o he logic
laye is associa ed o a single memo y aul , and can ope a e inde-
penden ly, and communica e h ough he TSVs and a aul con olle
(VC). The o ganiza ion o each PE includes mul iple memo y bu e s
o s o e weigh s and inpu s as well as some uni s o pe o m MAC
ope a ions. In addi ion, each aul con olle includes a P og ammable
Neu osequence Gene a o (PNG) uni ha gene a es he commands o
o ches a e he co esponding ope a ions o he DNN laye s. The PNGs
employ a simple ini e s a e machine (FSM) wi h coun e s ha a e
ini ialized depending on he numbe o MAC uni s in each PE and he
DNN laye opology. Fig. 1shows a gene al o e iew o he Neu ocube
a chi ec u e and a PE. Neu ocube pa i ions inpu ea u e maps o
CONV laye s and ou pu s o FC laye s ac oss di e en aul s and
exploi s he ou pu s a iona y da a low. While i is highly op imized o
CNN execu ion, i s p og ammabili y and scalabili y enable he mapping
and execu ion o a ious DNN a chi ec u es. In his wo k, we imple-
men a Neu ocube-like baseline accele a o o assess he pe o mance
imp o emen and ene gy sa ings o QeiHaN.
Fig. 1. Neu ocube o ganiza ion.
Sou ce: Adap ed om [13].
In he same line o esea ch, TETRIS [14] s ands ou as ano he
popula DNN accele a o based on HMC. Like Neu ocube, TETRIS
ea u es an op imized ha dwa e a chi ec u e coupled wi h so wa e
scheduling and pa i ioning echniques ha exploi he inhe en cha ac-
e is ics o 3D memo y. Fi s , he au ho s show ha he high h oughpu
and low ene gy cha ac e is ics o 3D memo y allow he ebalance o
he NN accele a o design, using mo e a ea o p ocessing elemen s
and less a ea o SRAM bu e s. Second, hey mo e some po ions
o he NN compu a ions close o he DRAM banks o dec ease he
bandwid h p essu e and inc ease pe o mance and ene gy e iciency.
Finally, hey de elop an op imized da a low schedule and hyb id
pa i ioning scheme ha pa allelizes he DNN compu a ions wi hin and
ac oss mul iple aul s and s acks. In pa icula , TETRIS emphasizes a
pa i ioning s a egy ha is p ima ily ocused on accele a ing CONV
laye s. Howe e , FC laye s, which a e e y common in cu en DNN
wo kloads such as T ans o me s, s ill ollow he same da a low and
pa i ioning s a egy o Neu ocube, wi h mino imp o emen s.
3. LOG2 quan iza ion analysis
DNN quan iza ion allows o educe he nume ical p ecision o ac i-
a ions and weigh s, which in u n a o s he memo y oo p in and
he compu a ional cos o ha dwa e accele a ed DNN a chi ec u es.
The e o e, quan iza ion echniques ha e been widely explo ed in p e-
ious s udies as desc ibed in Sec ion 2.2. In pa icula , loga i hmic
quan iza ion akes ad an age o he non-uni o m dis ibu ion o enso s
o signi ican ly educe he nume ical p ecision o inpu ac i a ions
and/o weigh s wi h a mino impac in accu acy. This sec ion analyzes
he e ec s o he LOG2 quan iza ion o ac i a ions on mul iple DNN
models and laye s. Fi s , we explo e he bene i o he loga i hmic
encoding o ac i a ions o simpli y he do -p oduc ope a ions. Then,
we p o ide some hin s on educing he numbe o accesses o he
main memo y by exploi ing he cha ac e is ics o he 3D memo y and
bi -shi ing ope a ion.
Some p io wo ks ha e used linea uni o m quan iza ion o com-
p ess he DNN pa ame e s. Howe e , we obse e ha ac i a ions and
weigh s o mos DNNs do no ollow a uni o m dis ibu ion, which
causes a huge impac in e ms o accu acy loss when he p ecision is
u he educed o e y low bi wid hs (i.e. <8𝑏). Especially in ecen
DNNs ha a e ex emely deep and can ha e hund eds o laye s, he
e o is p opaga ed and expanded among laye s.
On he o he hand, loga i hmic base-2 (LOG2) quan iza ion [28–
30] le e ages he usually non-uni o m dis ibu ion o ac i a ions and
weigh s in a p e- ained DNN. The s udy in [29] compa ed he impac
o linea and LOG2 quan iza ion on ac i a ions and weigh s o VGG16
and AlexNe . Thei analysis shows an exponen ial dis ibu ion o ac i-
a ion alues a ound 0. They also concluded ha ac i a ions a e mo e
obus o LOG2 quan iza ion han weigh s o se e al easons. Fi s ,
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
4
B. Khabbazan e al.
CONV laye s euse he weigh s mul iple imes when compu ing he do -
p oduc s, p opaga ing he e o ac oss he inpu s/ou pu s o all laye s.
Second, he ange o he weigh s is no as wide as he ac i a ions [42],
and hei densi y is o en highe han ha o he ac i a ions, ha is, he
amoun o weigh s is huge, and hei ange is na ow. Addi ionally, he
DNA-TEQ [43] s udy p o ides a de ailed analysis o he dis ibu ion o
weigh s and ac i a ions ac oss a ious DNNs, demons a ing ha he
e o om i ing weigh dis ibu ions o a loga i hmic model is highe
han ha o ac i a ions. We pe o med an expe imen applying LOG2
quan iza ion o he ac i a ions and weigh s o mode n DNNs, oge he
and indi idually, and eached simila conclusions ega ding weigh s
being mo e sensi i e o he LOG2 quan iza ion e o han ac i a ions.
This sugges s ha he base-2 may no be he bes - i ing exponen ial
base o quan izing he weigh s.
In his pape , we apply loga i hmic base-2 (LOG2) quan iza ion
o he inpu ac i a ions o all he FC and CONV laye s o a se o
DNNs. On he o he hand, we apply INT8 uni o mly dis ibu ed linea
quan iza ion o he weigh s o hese laye s based on Eq. (1). These
laye s ep esen close o 100% o he o al execu ion ime o ypical
neu al ne wo ks. Nex , we analyze he dis ibu ion o exponen s o he
quan ized ac i a ions, and he accu acy loss due o he LOG2 quan-
iza ion. This scheme also allows us o e icien ly e-o ganize o line
he weigh s in-memo y wi hou addi ional expensi e ha dwa e, and
exploi some o he in insic cha ac e is ics o he 3D-s acked memo y,
as desc ibed below. Fo each inpu 𝑥and each laye 𝑙, he LOG2
quan iza ion is applied acco ding o he ollowing equa ions:
𝐿𝑜𝑔 𝑄𝑢𝑎𝑛𝑡(𝑥) ={0𝑥= 0
2𝑥 o he wise.(2)
𝑥 =𝐶 𝑙 𝑖𝑝(𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2(|𝑥|))), 𝑚𝑖𝑛, 𝑚𝑎𝑥, (3)
whe e
𝐶 𝑙 𝑖𝑝(𝑥, 𝑚𝑖𝑛, 𝑚𝑎𝑥) =⎧
⎪
⎨
⎪
⎩
𝑚𝑖𝑛 𝑥⩽𝑚𝑖𝑛
𝑚𝑎𝑥 𝑥⩾𝑚𝑎𝑥
𝑥o he wise.
(4)
The exponen 𝑥 is compu ed based on Eq. (3). The 𝑅𝑜𝑢𝑛𝑑 unc ion
is de ined as ounding o he nea es in ege , and he clipping unc ion
in Eq. (4) o ces he exponen alues o be in he ange o [𝑚𝑖𝑛, 𝑚𝑎𝑥],
whe e 𝑚𝑖𝑛 = −(2𝑛−1)and 𝑚𝑎𝑥 = (2𝑛−1 − 1). Assuming an n-bi
exponen ial quan iza ion (e.g. 𝑛= 4), he numbe o unique in e als
is 2𝑛− 1. We s o e an ex a bi o he sign o he alue, bu in mos
laye s i is no necessa y since he ac i a ions a e all posi i e. The 𝑚𝑖𝑛
exponen is also used as a special case o ep esen he exac ly ze o
ac i a ion alue, so all small ac i a ions a e e ec i ely p uned due o
he clipping.
The main bene i o he LOG2 quan iza ion is ha i no only educes
he nume ical p ecision bu also elimina es he bulky digi al mul iplie s
by using simple shi and ADD ope a ions. The app oxima ed ac i a ion
alues 𝑥 a e s o ed as exponen s o educe he memo y p essu e and he
compu a ional complexi y. Eq. (5) shows he ans o med do -p oduc
ope a ion wi h he bi -shi ing o 𝑤𝑖weigh s by he 𝑥𝑖exponen s o he
base-2 powe s ep esen ing he ac i a ions, whe e 𝑥𝑖is quan ized o an
in ege exponen using Eq. (2). No e ha he posi i e exponen s will
lead o a shi o he le , while nega i e exponen s esul in a shi o
he igh .
𝑤𝑇𝑥=
𝑛
∑
𝑖=1
𝑤𝑖×𝑥𝑖≃
𝑛
∑
𝑖=1
𝑤𝑖× 2𝑥𝑖
=
𝑛
∑
𝑖=1
𝐵 𝑖𝑡𝑠ℎ𝑖𝑓 𝑡(𝑤𝑖, 𝑥𝑖)
(5)
In o de o u he exploi he LOG2 quan iza ion o he inpu
ac i a ions, a key obse a ion is ha , i a gi en ac i a ion is ep esen ed
wi h a base-2 powe o a nega i e exponen , he bi -shi ing o he
igh will disca d he leas signi ican bi s (LSB) o he weigh s ha
a e mul iplied by he co esponding ac i a ion. In o he wo ds, du ing
Fig. 2. His og ams o he LOG2 Quan iza ion (LogQuan ) o ac i a ions om all he
FC and CONV laye s o AlexNe , T ans o me , PTBLM, BERT-Base and BERT-La ge.
Fig. 3. Es ima ed memo y sa ings o ou se o DNNs.
he igh bi -shi ope a ion, and assuming ha weigh s a e uni o mly
quan ized o 8 bi s, only 1⩽8 −|𝑥|⩽7bi s o he weigh s a e equi ed
while he es can be a oided, educing he memo y accesses a a ine
g anula i y.
To demons a e he po en ial o his idea, we pe o m an analysis o
he exponen s esul ing om a LOG2 4-bi quan iza ion o he ac i a-
ions in all FC/CONV laye s o a popula se o DNNs om di e en
domains. All he e alua ed ne wo ks ha e been e- ained, educing
he accu acy loss a e quan iza ion o less han 1% in all cases.
Fig. 2shows he dis ibu ion o he non-ze o, quan ized ac i a ions. On
a e age, mo e han 71% o he ac i a ions ha e nega i e exponen s.
PTBLM (98%), BERT-Base (82%), and BERT-La ge (85%) ha e a simila
dis ibu ion o exponen s wi h a high concen a ion o nega i e alues
cen e ed a ound −3, while he T ans o me (57%) and AlexNe (36%)
ha e he mos symme ic dis ibu ion esul ing in he lowes amoun o
nega i e exponen s. QeiHaN, ou p oposed solu ion o e icien DNN
in e ence, is based on exploi ing his obse a ion.
We de ine he es ima ed memo y sa ings as he pe cen age o bi s
om he weigh s ha can be igno ed because he nega i e exponen s
o he base-2 ac i a ions ende hose bi s useless when pe o ming
he bi -shi ing ope a ion. Fig. 3shows ha he memo y sa ings a e
di ec ly ela ed o he his og ams o he quan ized ac i a ions. On
a e age, 25% o he memo y accesses can be a oided. In addi ion,
ze o-ac i a ions a e p uned in bo h, he baseline and ou p oposal,
u he educing memo y accesses. Howe e , he con en ional s o age
o weigh s in-DRAM is no sui able o exploi his op imiza ion. The
ollowing sec ion desc ibes how o e-o ganize he weigh s in-memo y
o ake ull ad an age o he LOG2 quan iza ion.
4. QeiHaN accele a o
This sec ion desc ibes he ha dwa e suppo equi ed o imple-
men QeiHaN. Fi s , we p esen he main ha dwa e componen s o he
QeiHaN accele a o . Nex , we desc ibe he memo y o ganiza ion o
weigh s and ac i a ions. Finally, we show how FC and CONV laye s
a e execu ed in he accele a o using QeiHaN wi h an enhanced inpu
s a iona y da a low.
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
5
B. Khabbazan e al.
Fig. 4. A chi ec u e o he QeiHaN accele a o including he o ganiza ion o a single
P ocessing Elemen (PE).
4.1. A chi ec u e
The goal o QeiHaN is o op imize he memo y p essu e by pe -
o ming an implici in-memo y bi -shi ing o he weigh s in he FC
and CONV laye s o di e en DNNs. QeiHaN le e ages a la ge num-
be o nega i e exponen s a e he LOG2 quan iza ion o ac i a ions,
and an e icien weigh s o age scheme, o sa e memo y accesses.
Simila o Neu ocube [13] and TETRIS [14], QeiHaN is based on
NDP a chi ec u es [10,39] ha le e age 3D s acked memo y o high-
pe o mance, low-ene gy DNN in e ence. As desc ibed in Sec ion 2,
he 3D memo y consis s o mul iple DRAM dies connec ed ia TSVs
o a logic die. DRAM dies a e di ided in o e ical pa i ions named
aul s ha esemble con en ional DDRx channels, which can ope a e
independen ly. In addi ion, each aul is connec ed o a ile in he logic
die o pe o m a i hme ic compu a ions on he s o ed da a.
Fig. 4shows a high-le el schema ic o he QeiHaN a chi ec u e.
Each ile in he logic die consis s o a single PE, a Vaul Con olle
(VC), a Rou e (R), and a PE Con olle (PEC). The VC manages all he
memo y ope a ions wi hin he co esponding aul . The ou e p o ides
local access be ween a gi en PE and i s ela ed aul , as well as emo e
access o he o he aul s/PEs h ough a 2D mesh ne wo k. In addi ion,
he PEC o ches a es he communica ion be ween he PE and he ou e
by gene a ing he add esses o he equi ed da a in each PE. Finally, he
PE is he co e o he ile, and is esponsible o accele a ing he DNN
ope a ions. The main componen s o a PE include he blocks o SRAM
used o s o ing he inpu s (IB), ou pu s (OB), and weigh s (WB), he
LOG2 Quan iza ion (LOG2-Quan ) uni , he Weigh Decode and Shi e
(D&S) uni , he ADD a ay, and he Special Func ion Uni (SFU). Below
is a de ailed desc ip ion o each componen :
Memo y Bu e s: Each PE in he logic die has h ee indi idual on-
chip SRAM bu e s o s o e and euse he da a e ched om he main
memo y acco ding o he da a low o he accele a o . Fi s , a small
Inpu Bu e (IB) s o es blocks o inpu FP16 ac i a ions un il illing he
whole bu e space. Second, an Ou pu Bu e (OB) s o es he pa ial
and inal esul s ha a e p oduced du ing he execu ion o a DNN
laye . Thi d, a Weigh s Bu e (WB) keeps he equi ed bi s o he
weigh s o he bi -shi ing ope a ions. All he SRAM memo ies a e
double bu e ed o load da a om main memo y while pe o ming
compu a ions, a oiding s alls in he pipeline, and highly mul i-banked
o achie e he bandwid h equi ed o eed a la ge numbe o unc ional
ADD uni s. In addi ion, all hese bu e s a e sized conside ing he wo s
case scena ios, ha is, he bigges laye o he I/O bu e , and all he
8 bi s o 𝑀weigh s o he WB, whe e 𝑀is he bus size o a aul in
he 3D-s acked memo y.
LOG2-Quan Uni : This uni is in cha ge o he LOG2 quan iza ion
o inpu ac i a ions om FP16 o base-2 exponen ial alues acco d-
ing o Eq. (2).Fig. 5shows he ha dwa e equi ed o compu e he
𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2(|𝑥|)) unc ion o Eq. (3). Unlike p e ious wo ks ha use
ela i ely complex ha dwa e [44–46], we implemen his unc ion wi h
a e y simple scheme. In pa icula , we pe o m a compa ison be ween
Fig. 5. Ha dwa e implemen a ion o 𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2|𝑥|).
he ac ional pa o he alue |𝑥|and he √2using a simple com-
pa a o . The s anda d hal p ecision (FP16) o ma o a alue 𝑥is
encoded wi h a sign bi , man issa 𝑚, and exponen 𝑒. The exponen
𝑒is al eady exp essed as an in ege in base-2 o ma , so he LOG2
unc ion o |𝑥|can be implemen ed by applying he loga i hm on he
man issa 𝑚as shown in Eq. (6). Taking in o accoun he hidden bi o
he man issa, 𝑚is always a alue be ween [1, 2). The e o e, he e m
𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚)can be u he simpli ied by Eq. (7). In he nex s ep, each
quan ized alue 𝑥 in QeiHaN is ep esen ed by a 4-bi exponen h ough
a clipping unc ion (i.e. Eq. (4)), esul ing in a ange o [−8,7]. An
ex a bi may be used o he ac ual sign o he ac i a ions, excep o
when all a e known o be posi i e due o he ReLU ac i a ion unc ion.
In addi ion, all ze o ac i a ions will skip he quan iza ion and all he
ela ed compu a ions and memo y accesses. Simila ly, all he small
ac i a ions clipped o −8 will be e ec i ely p uned ( ounded o ze o).
Finally, each quan ized ac i a ion is sen o he D&S uni o u he
p ocessing.
𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2|𝑥|) =𝑒+𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚)(6)
1⩽𝑚 <2←←→ 0⩽𝑙 𝑜𝑔2𝑚 <1⇒𝑅𝑜𝑢𝑛𝑑(𝑙 𝑜𝑔2𝑚) ={0𝑚 <√2
1𝑚⩾√2(7)
Weigh Decode & Shi e Uni (D&S): The weigh s ha mul iply non-
ze o ac i a ions a e decoded om a comp essed s eam and bi -shi ed
by appending he necessa y amoun o ze os based on he exponen
om he LOG2-Quan uni . Acco ding o he exponen alue 𝑥, he PE
con olle de e mines he equi ed bi s o he weigh s ha ha e o be
loaded om DRAM and s o ed in o he WB. A non-nega i e exponen
equi es loading all 8 bi s o each weigh , and he D&S uni shi s
he weigh s 𝑥 posi ions o he le be o e sending he esul s o he
ADD a ay. O he wise, we only need o e ch he 8 −|𝑥|MSBs o he
weigh s. Fo example, gi en a nega i e exponen 𝑥 = −3, only he
5 MSBs o each weigh a e loaded in o he WB. Then, he D&S uni
eads he selec ed bi s o he weigh s om he WB and gene a es a
se o 16-bi 𝑑 alues, whe e 𝑑is he amoun o adde s in he ADD
a ay. No e ha he bi -shi ed weigh s a e he esul o he adi ional
mul iplica ion o ac i a ions and weigh s. In o de o use his uni
e icien ly, QeiHaN eo ganizes he weigh s in-memo y o a bi -le el
g anula i y as desc ibed below in Sec ion 4.2.
ADD A ay: This a ay is made o 𝑑independen ADD uni s ha
a e used o accumula e he p oduc s o each ac i a ion by he co e-
sponding weigh s. Acco ding o he sign o he ac i a ion alue, no
he exponen , he bi -shi ed weigh is added/sub ac ed o/ om he
pa ial ou pu s compu ed in p e ious cycles and s o ed in he OB. The
LOG2 quan iza ion emo es he need o any mul iplie , so he pa ial
ou pu s a e loaded om he OB and he bi -shi ed weigh s come om
he D&S. As a esul , in a single execu ion all he adde s compu e
pa ial ou pu s ela ed o he same inpu ac i a ion om 𝑑di e en
con olu ional ke nels o ou pu neu ons.
Special Func ion Uni (SFU): The SFU is composed o uni s o pe o m
non-linea ac i a ion unc ions, pooling, and no maliza ion, among
o he s. These unc ions a e usually applied o he inal ou pu s o he
FC/CONV laye s a he end o hei execu ion, and end o equi e
mo e nume ical p ecision in o de no o lose accu acy. Thus, QeiHaN
de-quan izes he esul ing 16-bi in ege ou pu s back o FP16 be o e
using hose unc ions. The non-linea unc ions a e implemen ed wi h
Look-Up-Tables (LUTs).
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
6
B. Khabbazan e al.
Fig. 6. O ganiza ion o he I/O Bu e .
4.2. Memo y o ganiza ion
This sec ion desc ibes he memo y o ganiza ion o he accele a o ,
which e e s o he da a layou o weigh s and ac i a ions inside he
DRAM o each aul and he on-chip bu e s o he PEs.
To illus a e i , he op o Fig. 6shows an example o a small CONV
laye wi h an inpu ea u e map (IFM) size o ou channels (IC1–IC4),
and an ou pu ea u e map (OFM) size o wo channels (OC1–OC2).
On he o he hand, he bo om o Fig. 6shows how he inpu /ou pu
ac i a ions o he di e en channels a e pa i ioned and dis ibu ed
among he I/O bu e s o wo di e en PEs/Vaul s.
In QeiHaN, he inpu ac i a ions a e di ided channel-wise ac oss
all aul s, ha is, all inpu s o a gi en channel a e s o ed in he same
aul . In con as , each aul alloca es a po ion o he co esponding
pa ial ou pu s o all he channels. In CONV laye s, he dimensionali y
o he inpu s/ou pu s may be qui e la ge, so we employ a blocking
scheme o educe he on-chip s o age equi emen s by segmen ing he
IFM and OFM in o 𝑁blocks o iles pe channel. The I/O Bu e only
s o es a subse o blocks o each assigned channel o he IFM and OFM,
he block size being signi ican ly smalle han he dimensions o he
ea u e maps. No e ha each Vaul /PE is wo king on a di e en se o
inpu s bu p oducing pa ial ou pu s o he same OFM channels. Hence,
a educ ion is equi ed a he end o he execu ion o ob ain he inal
ou pu s. Likewise, FC laye s a e a special case o CONV, whe e he e is
jus a single block and inpu pe channel (i.e. 𝑁= 1).
Fig. 7shows he layou o 𝑀 il e s o ke nels wi h 𝑃weigh s pe
channel each in he DRAM dies o each aul , whe e each pa i ion
includes 4 banks, o he example o Fig. 6. Simila o he ac i a ions,
he weigh s o each ke nel a e also dis ibu ed channel-wise ac oss
all aul s. The bi s o he weigh s o he co esponding channels a e
in e lea ed in he di e en banks and pa i ions o he same aul . Tha
is, he leas -signi ican bi (LSB) o a subse o weigh s is s o ed in he
i s bank o a aul , hen he nex bi in he second bank and so on. This
layou simpli ies he implemen a ion o ou implici bi -shi ing scheme,
as i is easy o loca e all he bi s o he weigh s ha a e equi ed o
ope a e wi h a gi en inpu in case some ha e o be skipped and o he s
accessed.
In addi ion, mos 3D-s acked DRAM-based ope a ions use aClosed-
Page Policy o educe powe consump ion [47]. Consequen ly, applica-
ions bene i om Bank-Le el Pa allelism bu no om spa ial local-
i y. QeiHaN emaps he da a o a oid in e nal o ganiza ion bo lenecks
and, hence, eques s o di e en banks can be conca ena ed/o e lapped
by he simul aneous ac i a ion o he same ow ac oss he memo y
banks wi hin a aul . This o e lapping e ec i ely hides he la ency
associa ed wi h ac i a ing ano he ow while eading da a om a
di e en bank, i balances he load, p e en s bo lenecks, and ensu es
he e icien u iliza ion o memo y’s bandwid h. No e ha weigh s a e
known s a ically so hei o ganiza ion can be p e-a anged o line.
Fig. 7. O ganiza ion o weigh s in he 3D-s acked memo y.
4.3. Da a low
Neu ocube [13] ollows an ou pu s a iona y (OS) da a low in which
each PE compu es a subse o ou pu s a a ime. This da a low is
ine icien o exploi he esou ces o he 3D memo y, as demons a ed
by ou esul s in Sec ion 6. On he o he hand, QeiHaN uses an en-
hanced inpu s a iona y (IS) da a low coupled wi h a blocking scheme
o e icien ly exploi he LOG2 quan iza ion o he inpu ac i a ions,
minimizing he memo y accesses o bo h weigh s and ac i a ions. Fig. 8
illus a es he da a low o he QeiHaN accele a o wi h a lowcha .
The p oposed da a low includes h ee main s ages ma ked in di e en
colo s: P e-P ocessing (G ay), Execu ion (O ange), and Pos -P ocessing
(Blue).
In he P e-P ocessing s age, each PE eads inpu ac i a ions om
DRAM un il illing he inpu bu e space. Tha is, inpu s (ou pu s) a e
p e-loaded (p ocessed) on-demand by blocks, ac i a ions a e s o ed in
FP16 o ma , and he size o he blocks is compu ed acco ding o he
ea u e map sizes and he I/O bu e capaci y. In he IS da a low, each
PE o he accele a o e ches and p ocesses one inpu o a block a a
ime om he I/O bu e , and pe o ms all he associa ed compu a ions
be o e mo ing o he nex inpu . Fi s , he LOG2 quan iza ion and clip-
ping unc ion is applied o ob ain he 4-bi exponen 𝑥. Then, QeiHaN
also pe o ms a ze o and small ac i a ion p uning. Concu en ly, he
eading o inpu blocks om DRAM con inues in he backg ound, as
long as he e is space in he bu e s, o hide he memo y la ency while
doing compu a ions o he cu en blocks.
In he Execu ion s age, and based on he alue o he exponen 𝑥, a
se o 𝑀use ul bi s o INT8 uni o mly quan ized weigh s o 𝑀di e en
ke nels ela ed o he inpu a e ead om DRAM a a ime, whe e 𝑀
is de e mined by he in e nal 3D-s acked memo y bus size (e.g. 32-bi ).
Thus, in each eques , he bi s in he same posi ion o 𝑀di e en
weigh s a e loaded in o he weigh s bu e , and mul iple eques s a e
made un il all he equi ed bi s a e e ie ed. Nex , he bi s o he
weigh s a e decoded and bi -shi ed by appending he co esponding
ze os, esul ing in 16-bi in ege alues. These esul s a e g ouped and
sen o he ADD a ay uni in ba ches o 𝑑 alues, whe e 𝑑is he numbe
o adde s (e.g. 16). In pa allel, he pa ial ou pu s om p e ious
execu ions a e loaded om he ou pu bu e . Then, he accele a o
pe o ms 𝑑ADDs o accumula e he esul s o each ou pu wi h he
shi ed weigh s, ollowed by he w i e-back o he ou pu bu e . This
s age is epea ed un il all he weigh s o all il e s ela ed o he cu en
ac i a ion a e p ocessed.
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
7
B. Khabbazan e al.
Fig. 8. Da a low and Execu ion scheme.
Table 1
DNNs employed o he expe imen al e alua ion o QeiHaN. The model size accoun s o
he pa ame e s in FP32 and INT8 o he FC/CONV laye s whe e he LOG2 quan iza ion
is applied. The accu acy shown is a e quan iza ion.
DNN model FP32-Size (MB) INT8-Size (MB) Accu acy (QeiHaN/FP32)
AlexNe [15] 144 36 57.05/57.48% (Top-1)
PTBLM [17] 136 34.2 79.40/78.09 (Pe plexi y)
T ans o me [18] 336 84 28.28/28.44 (BLEU-uncased)
BERT-Base [19] 440 110 86.75/86.96% (F1)
BERT-La ge [19] 1320 330 89.72/89.86% (F1)
Finally, in he Pos -P ocessing s age, QeiHaN educes he pa ial
ou pu s o each PE. The educ ion s a s as soon as enough ac i a ions
comple e all hei ope a ions. Then, in a cen alized PE, he inal esul s
a e de-quan ized, and he SFU pe o ms he ac i a ion and pooling
ope a ions be o e dis ibu ing and s o ing he co esponding ac i a ions
back o each aul . A e p ocessing all he blocks o inpu s he laye
execu ion is comple ed. No e ha all he main s eps a e ca ied ou in
pa allel in a deep pipeline.
5. Me hodology
This sec ion p esen s he me hodology o e alua ing QeiHaN, ou
NDP accele a o o DNN in e ence.
Wo kloads. Ou objec i e is o p o e ha ou scheme p o ides
impo an sa ings o mul iple applica ions and di e en DNN models.
To his end, we e alua e QeiHaN on i e s a e-o - he-a DNN wo k-
loads om di e en domains, summa ized in Table 1. Thei model sizes
ange om medium o la ge scale wi h se e al hund eds o MBy es in
memo y oo p in . In pa icula , we include he ILSVRC 2012 winne ,
AlexNe [15] (5 CONV and 3 FC laye s), one o he mos popula CNNs
o image classi ica ion wi h he ImageNe da ase , and PTBLM [17] (2
LSTM laye s), an RNN ha consis s o LSTM cells o language mod-
eling using he Penn T eebank da ase . In addi ion, we employ h ee
a en ion-based ne wo ks: T ans o me (6 Encode s, 6 Decode s), BERT-
Base (12 Encode s, 110M Pa ame e s), and BERT-La ge (24 Encode s,
340M pa ame e s). The T ans o me [18] model is e alua ed on he
machine ansla ion ask o New es 2014 (English o Ge man) which
con ains 3003 sen ences. BERT-Base [19], and i s la ge a ian BERT-
La ge, a e e alua ed on he ques ion-answe ing ask o SQuAD 1 [48].
Finally, all hese ne wo ks ha e been e- ained in o de o eco e
he accu acy a e quan iza ion, ha is, less han 1% loss. Accu acy is
epo ed as Top-1 o image classi ica ion (highe is be e ), pe plexi y
o language modeling (lowe is be e ), bilingual e alua ion unde -
s udy (BLEU) o machine ansla ion (highe is be e ), and weigh ed
a e age o he p ecision and ecall (F1) o ques ion-answe ing (highe
is be e ).
Sys em models and simula ion. We ha e de eloped a simula-
o ha accu a ely models h ee di e en sys ems, QeiHaN and wo
baseline accele a o s. The i s baseline is inspi ed in Neu ocube [13],
desc ibed in Sec ion 2, bu wi h some op imiza ions, such as a lowe
quan iza ion bi wid h, o isola e he e ec s o ou p oposal when com-
pa ing he wo. The second baseline, named NaHiD, implemen s he
same a chi ec u e, da a low, and quan iza ion scheme as QeiHaN bu
wi h a s anda d memo y o ganiza ion o he weigh s. Tha is, NaHiD
also eplaces mul iplica ions by bi -shi ope a ions and addi ions bu ,
in con as o QeiHaN, i equi es loading all he bi s o he weigh s
om memo y. This compa ison allows us o in e he main bene i s due
o he QeiHaN’s e icien 3D memo y-cen ic weigh s o age scheme.
Among he ela ed wo ks desc ibed in Sec ion 2, we only compa e
agains Neu ocube because TETRIS has a simila HW and da a low o
FC laye s. Mos o he TETRIS imp o emen s come om op imizing he
da a low o CONV laye s in CNNs. No e ha he majo i y o mode n
DNNs a e mainly composed o FC laye s, hence, we would expec he
compa ison wi h TETRIS o ha e mino impac in ou esul s. In all
h ee sys ems, We ocus on accele a ing FC and CONV laye s ha use
s a ic weigh s, as hese a e key componen s o DNN wo kloads. Ma ix-
ma ix mul iplica ion laye s, which a e no explici ly a ge ed by ou
app oach, a e assumed o be p ocessed on he hos sys em using FP16
p ecision.
Table 2shows he pa ame e s o he expe imen s. Fo a ai com-
pa ison, we se mos o he con igu a ion pa ame e s o ma ch he
Neu ocube baseline: a 3D-s acked memo y o 4 GB wi h 4 DRAM
dies pa i ioned in o 4×4 aul s and PEs, an in e nal 3D memo y
bandwid h o 10 GB/s pe aul , abou 2.5 kB o SRAM pe PE, 16
MAC/ADD uni s pe PE, and a equency o 300 MHz in he logic die.
QeiHaN and NaHiD equi e sligh ly smalle memo y bu e s (i.e. 2 kB
o OB, 64B o IB, and 64B o WB) due o he di e en da a low.
Rega ding a ea and ene gy consump ion e alua ion, he logic com-
ponen s a e implemen ed in Ve ilog, including all he addi ional com-
ponen s equi ed by QeiHaN, and syn hesized o ob ain he delay, a ea,
and powe using he Synopsys Design Compile [49], he modules o
he DesignWa e lib a y and he echnology lib a y o 28/32 nm om
Synopsys. On he o he hand, we cha ac e ize he memo y bu e s o
he accele a o by ob aining he delay, ene gy pe access, and a ea
using CACTI-P [50]. We use he con igu a ions op imized o low
powe and a supply ol age o 0.78 V o h ee sys ems. Finally, he
ene gy consump ion o he 3D-s acked memo y is es ima ed by using
an HMC con igu a ion o DRAMSim3 [51]. The esul s ob ained wi h
he a o emen ioned ools a e combined wi h he ac i i y ac o s and
memo y aces p o ided by ou simula o o ob ain he dynamic and
s a ic powe o he accele a o s.
6. E alua ion
This sec ion e alua es he pe o mance, ene gy e iciency, and mem-
o y ac i i y o ou p oposal. Fi s , we in oduce an analysis o he o al
numbe o memo y accesses o he 3D-s acked DRAM dies a e apply-
ing he QeiHaN scheme. Then, we p esen he speedups and ene gy
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
8
B. Khabbazan e al.
Table 2
Pa ame e s o he accele a o s.
Common pa ame e s
Technology 32 nm
Logic die equency 300 MHz
#DRAM dies, #Banks pe aul pe die 4
#Vaul s, #PEs 16
3D-s acked memo y o al size 4 GB
3D-s acked memo y bandwid h pe aul 10 GB/s
Neu ocube pa ame e s
Da a low OS
Weigh s p ecision 8-bi
Inpu ac i a ions p ecision 8-bi
Weigh s/inpu s quan iza ion Uni o m/Uni o m
#MAC uni s pe PE 16
To al SRAM bu e s size pe PE 2.5 kB
NaHiD & QeiHaN pa ame e s
Da a low IS
Weigh s p ecision 8-bi
Inpu ac i a ions p ecision 4-bi
Weigh s/inpu s quan iza ion Uni o m/LOG2
#ADD uni s pe PE 16
To al SRAM bu e s size pe PE 2.1 kB
sa ings achie ed by QeiHaN compa ed o he Neu ocube and NaHiD
baselines. Finally, we discuss he accele a o o e heads. No ably, he
a e ages p esen ed o each e alua ion a e calcula ed using he a i h-
me ic mean. These a e ages p ima ily se e a summa iza ion pu pose,
bu he esul s o each model ype should be examined indi idually,
as hey p o ide a mo e accu a e e lec ion o pe o mance o speci ic
wo kloads.
6.1. 3D-s acked memo y accesses
Fig. 9 epo s he no malized o al 3D memo y accesses o QeiHaN
o e he wo baseline accele a o s. This o al includes bo h, memo y
accesses o eading/w i ing he weigh s and he inpu ac i a ions.
On a e age o ou se o DNNs, QeiHaN educes he o al DRAM
accesses by 72.4% and 25% o e Neu ocube and NaHiD, espec i ely.
The g ea educ ion o memo y accesses wi h espec o he baselines
is mainly due o cons aining he accesses o only he equi ed bi s o
he weigh s o he bi -shi ing ope a ions. Mo eo e , QeiHaN shows
a highe educ ion o memo y accesses o e Neu ocube due o wo
main easons. Fi s , he enhanced IS da a low o QeiHaN equi es each
inpu ac i a ion o be accessed jus once du ing he execu ion o a
laye . In con as , he OS da a low o Neu ocube may equi e mul iple
accesses o he ac i a ions. Second, QeiHaN pe o ms p uning o ze o
and small ac i a ions a e applying he quan iza ion, emo ing all he
ela ed memo y accesses o he weigh s. The e iciency o he ac i a ion
p uning is limi ed in Neu ocube due o i s OS da a low, so i is no
implemen ed. On he o he hand, compa ed o NaHiD, he educ ion
is well co ela ed o he es ima ed memo y sa ings due o he huge
amoun o nega i e exponen s as discussed in Sec ion 3. Bo h QeiHaN
and NaHiD use he same da a low and p uning scheme, so bo h access
he same inpu ac i a ions, and he sa ings come om he weigh s.
6.2. Pe o mance
Fig. 10 shows he speedups achie ed by QeiHaN. Compa ed o
Neu ocube, QeiHaN p o ides consis en speedups o he i e DNNs
ha ange om 8.69𝑥(AlexNe ) o 1.24𝑥(T ans o me ), achie ing an
a e age pe o mance imp o emen o 4.25𝑥. The educ ion in execu ion
ime is due o QeiHaN’s e icien memo y o ganiza ion and enhanced
IS da a low. The numbe o memo y accesses is d ama ically educed
since only he meaning ul bi s o he weigh s equi ed by he shi
ope a ions a e loaded. In addi ion, QeiHaN employs a no el weigh
Fig. 9. No malized o al memo y accesses o each DNN.
Fig. 10. Speedups o QeiHaN o e Neu ocube &NaHiD.
s o age scheme o exploi he bank-le el pa allelism o he 3D mem-
o y. Mo eo e , QeiHaN o e laps he di e en s ages o he da a low
in a deep pipeline, sho ening he c i ical pa h o he execu ion. As
shown in Fig. 9,AlexNe and PTLBM exhibi he highes educ ion
in memo y accesses and, hence, hey ob ain he la ges pe o mance
imp o emen s. The di e ence in speedup be ween hese wo ne wo ks
and he a en ion-based models is in he pe cen age o ze o and small
ac i a ions ha a e e ec i ely p uned in QeiHaN, skipping pa o he
execu ion and pos -p ocessing s ages. The e ec o ac i a ion p uning
is mino in T ans o me (3%), BERT-Base (7%), and BERT-La ge (13%),
bu signi ican in AlexNe (47%) and PTLBM (55%).
Compa ed o NaHiD, he bene i s o QeiHaN a e mo e modes bu
s ill qui e impo an , achie ing an a e age speedup o 1.38𝑥. The main
eason is ha bo h accele a o s bene i om he same a chi ec u e,
da a low, quan iza ion, and ac i a ion p uning scheme. The e o e, he
imp o emen s come mainly om he no el memo y layou o s o ing
he weigh s in he 3D memo y, and he co esponding educ ion o
memo y accesses by le e aging he loga i hmic quan iza ion. PTBLM
ob ains he la ges bene i s, achie ing an speedup o 1.86𝑥whe eas
AlexNe ge s he lowes imp o emen s, ha is, 1.07𝑥speedup. These e-
sul s a e di ec ly p opo ional o he pe cen age o nega i e exponen s
shown in Fig. 2.
6.3. Ene gy consump ion
Fig. 11 epo s no malized ene gy sa ings. On a e age, QeiHaN
educes he ene gy consump ion o he accele a o by 3.52𝑥and 1.28𝑥
o e Neu ocube and NaHiD, espec i ely. As we obse ed o pe o -
mance, he ene gy sa ings a e well co ela ed wi h he numbe o neg-
a i e exponen s and he co esponding educ ion o memo y accesses.
These ene gy sa ings a e due o wo main easons. Fi s , dynamic
ene gy is educed due o he sa ings in mul iplica ions and memo y
accesses. Second, he pe o mance imp o emen s shown in Fig. 10
p o ide a educ ion in s a ic ene gy. Again, PTBLM ob ains he la ges
bene i s, achie ing a educ ion o 8.2𝑥and 1.6𝑥in ene gy compa ed o
bo h Neu ocube and NaHiD espec i ely.
Fig. 12 shows he ene gy b eakdown o QeiHaN and NaHiD o e
Neu ocube. The igu e shows esul s o he i e neu al ne wo ks in-
cluding he pe cen age o ene gy consumed by each majo ha dwa e
block o he accele a o s. As can be seen, he DRAM o he 3D-s acked
memo y (i.e. HMC) consumes mos o he ene gy in all cases. The en-
e gy sa ings achie ed by ou p oposal a e signi ican , and a e especially
Jou nal o Sys ems A chi ec u e 159 (2025) 103320
9
