Open-Sou ce GEMM Ha dwa e Ke nels Gene a o :
Towa d Nume ically-Tailo ed Compu a ions
Louis Ledoux∗†, Ma c Casas∗†
∗Ba celona Supe compu ing Cen e , Ba celona, Spain
†Uni e si a Poli `
ecnica de Ca alunya, Ba celona, Spain
E-mail: {louis.ledoux, ma c.casas}@bsc.es
Keywo ds—GEMMs, ma ix-ma ix-mul iply, ull s ack ame-
wo k, au oma ed pipeline, lopoco, OpenCAPI, OpenBLAS, High
Pe o mance Compu ing, app oxima e/ ans/ex ended p ecision.
I. EXTENDED ABSTRACT
Many scien i ic compu ing p oblems can be educed o
Ma ix-Ma ix Mul iplica ions (MMM), making he Gene al
Ma ix Mul iply (GEMM) ke nels in he Basic Linea Alge-
b a Sub ou ine (BLAS) o in e es o he high-pe o mance
compu ing communi y. Howe e , hese wo kloads ha e a wide
ange o nume ical equi emen s. Ill-condi ioned linea sys ems
equi e high-p ecision a i hme ic o ensu e co ec and ep o-
ducible esul s [1]. In con as , eme ging wo kloads such as
deep neu al ne wo ks, which can ha e millions up o billions
o pa ame e s, ha e shown esilience o a i hme ic inke ing [2]
and p ecision lowe ing [3].
Gene al pu pose a i hme ic uni s and compu e o ma s
such as he IEEE754 s anda d na u ally unde pe o m in his
as e land o scena ios. We p opose he gene a ion o nu-
me ically ailo ed ci cui s whe e he necessa y and su icien
in e nal p ecision is gene a ed o a ge he compu a ions
equi emen s in e ms o nume ical quali y while imp o ing
he ene gy cos .
A. Open Sou ce SW/HW co-designed amewo k o nume i-
cally ailo ed MMMs
As depic ed by Fig. 1, ou amewo k is composed o
wo dis inc phases, he p io Ha dwa e gene a ion low and
he un ime execu ion low. Because MMMs a e basically
made o a bi a y long do p oduc s, we design a cus om
Fused Do P oduc (FDP) ope a o ha is agnos ic o he
compu e o ma and suppo s posi , IEEE754, and b loa 16
a ia ions, while ne e ounding be ween wo accumula ions.
The in e media e p ecision o he ixed-poin accumula o
used in he do -p oduc is a key aspec o his wo k, and is
xc u3p-ff c1517-2
OCXL
ke nel module
DMA
Engine
libocxl
OpenCAPI link ; DMA ans e @ ~20GB/s
I/O con ol; poin e passing; s a dma; egis e pooling
A p io i Ha dwa e gene a ion low
Run ime execu ion low
add ess
ansla ion
liboc-accel
ke nel &
HOST HW OS & use space
malloc()
cas ()
gemm()
OpenBLAS
use app
py o ch
numpy
POWER9 HOST Sha ed DDR4 AlphaDATA 9 3
Func ional
speci ica ion
1
3
4
6
5
2
oc-accel
OpenCAPI
T anscei e s
<5,-30,2>
Sys olic A ay
32x31
b loa 16
Pe o mance
speci ica ion
lopoco
EOB
Signy
Signx
Iy.Fy
Scaley
Scalex
isNaNy
isNaNx
unsigned
mul iplie
unsigned
adde
ωS
11 1
1 1
1
1
0
sign
ex ension
RCA
S
C
RCA
S
C
RCA
S
C
isNaNoA
CEOBQ
CSA adix-2k
ωQ
1
k-1
FTZ
0
1
0
1
0
1
ωF+1
Ix.Fx
2ωF+2
ωS+1
2ωF+2
shi al
gen
pa selec
ωShi
1
1
ba el shi
pad
LSBMSB
OVF
1
oo_big
oo_small
con ig. x hwgen.py
N,M
LSB,MSB,OVF
A i hme ic
Chip
F eq
Sys olic A ay
Gene a ion
buses
w apping
VUP.cpp
FDP.cpp
PE
PE
PE
PE
PE
PE
PE
PE
PE
A2S3 A2S3 A2S3
L2AL2AL2A
columns B
ows A
SOB
EOB
columns C
A2S3
A2S3
A2S3
Legend:
HSSD da a
inpu da a (inpu o ma )
in e media e da a (S3 o ma )
con ol
ou pu da a (ou pu o ma )
FF_delay
LUT_delay
C8_delay
Adde model
A
B
C
D
Fig. 1. O e iew o he 2 phases amewo k. Le is Run ime execu ion low
and igh is Ha dwa e gene a ion low.
27,7×
5×
15,1×
5,6×
Fig. 2. Sea Su ace Heigh compu a ion compa ing IEEE-754 double-,
quad- pecision FMAs and a 91-bi FDP w nume ical quali y and powe
consump ion.
con igu able h ough he leng h o he sc a chpad delimi ed
by he pa ame e s MSB (Mos Signi ican Bi ) and LSB
(Leas Signi ican Bi ). We le e age he au oma ed pipeline
ea u e o lopoco [4] which is an e ec i e ool o e icien ly
explo ing he wide ange o unc ional speci ica ions along
wi h pe o mance speci ica ions o p oduce MMM ke nels wi h
he necessa y basic elemen s (LUTs, FFs, Ca y chains, DSPs)
o a a ge ed (chip, equency)couple (see Fig. 1- B
).
The essence o his wo k is o make in e media e p ecision
weakings om he ha dwa e accessible o high-end so wa e
code as anspa en as possible. We achie e ha by aking
in o accoun ha many HPC codes ely on BLAS lib a ies o
pe o m MMM ope a ions. Such lib a ies ecei e he unc ion
call o pe o m a GEMM and dispa ch adequa ely o he
unde lying ha dwa e a hei disposal.
B. HPC wo kloads esul s
We expe imen wi h wo amilies o eal HPC wo kloads
wi h con as ing nume ical equi emen s, namely A i icial
In elligence (AI) and Sea Su ace Heigh (SSH), whose e-
spec i e esul s can be obse ed in Fig. 3 and Fig. 2.
Fig. 3. Top1 Accu acy s alida ion da ase in e ence Ene gy cos o a ious combina ions o da ase s,models,compu e o ma s, and accumula o s.
Fo he SSH compu a ion, he esul s ob ained wi h
64-bi and 128-bi FPUs exhibi dec easing ep oducibil-
i y as he ec o size inc eases. In con as , ou 91-bi
⟨o : 30, msb : 30, lsb : 30⟩FDP main ains ep oducibili y
o all ec o sizes wi hou de ia ion. Ou p oposed FDP
consis en ly exhibi s 52 co ec bi s, which is a leas 5×and
27.7×mo e han quad-p ecision and double-p ecision. Ou
measu emen s on VU3P-2 FPGA a 200MHz show ha he
uni s powe consump ion a e 0.266, 0.549, and 0.491 wa s o
double-p ecision FMA, quad-p ecision FMA, and he 91-bi
FDP, espec i ely. Fo all e alua ed sizes, he 91-bi FDP yields
a leas 5.6×and 15.1×mo e co ec bi s o he same wa age
as quad-p ecision and double-p ecision FMAs, espec i ely.
Fo AI wo kloads, we employ Py o ch as a base
amewo k and link i o ou modi ied OpenBLAS. We use
popula neu al ne wo k models such as ResNe 18, ResNe 34,
ResNe 50, DenseNe 121, DenseNe 161, DenseNe 169, and
VGG11 wi h ba ch no maliza ion, and e alua e hem on
he CIFAR-10 and ImageNe da ase s. To measu e powe
consump ion and accu acy, we use he B ainFloa 16 and
IEEE-754 32-bi o ma s o ou compu a ions wi h a la ge
a ie y o accumula o s a ying hei OV F ,MSB, and
LSB pa ame e s. Fig. 3 shows he ela ionship be ween
powe consump ion and accu acy o di e en accumula o
and a i hme ic combina ions. Fo example, i 84% Top1
accu acy is sa is ying o Imagene wi h Resne 50, he mos
sui ed a i hme ic/accumula o combina ion is IEEE-754 32-
bi /⟨o : 9, msb : 6, lsb :−20⟩ ep esen ed by a ligh pu ple
hexagon as all o he ma ke s a e ei he on he igh o below.
C. Conclusion
O e all, ou wo k highligh s he impo ance o nume ically
ailo ed accumula o s o ep oducibili y in scien i ic compu -
ing applica ions. Ou esul s p o ide aluable insigh s in o
he ade-o s be ween powe consump ion and accu acy, and
we belie e ha ou esul s ha e he po en ial o in o m he
design o u u e AI and scien i ic compu ing sys ems, and we
encou age o he esea che s o explo e he possibili ies o low
p ecision accumula o s using ou open-sou ce amewo k.
II. ACKNOWLEDGMENT
Ma c Casas is suppo ed by G an RYC-2017-23269
unded by MCIN/AEI/ 10.13039/501100011033 and by “ESF
In es ing in you u u e”.
REFERENCES
[1] D. Bailey and J. M. Bo wein, “High-P ecision Compu a ion and
Ma hema ical Physics,” in P oceedings o XII Ad anced Compu ing
and Analysis Techniques in Physics Resea ch — PoS(ACAT08).
E ice, I aly: Sissa Medialab, Oc . 2009, p. 014. [Online]. A ailable:
h ps://pos.sissa.i /070/014
[2] J. Johnson, “Re hinking loa ing poin o deep lea ning,”
a Xi :1811.01721 [cs], No . 2018, a Xi : 1811.01721. [Online].
A ailable: h p://a xi .o g/abs/1811.01721
[3] M. Cou ba iaux, I. Huba a, D. Soud y, R. El-Yani , and
Y. Bengio, “Bina ized Neu al Ne wo ks: T aining Deep Neu al
Ne wo ks wi h Weigh s and Ac i a ions Cons ained o +1 o -
1,” a Xi :1602.02830 [cs], Feb. 2016, a Xi : 1602.02830. [Online].
A ailable: h p://a xi .o g/abs/1602.02830
[4] M. Is oan and F. de Dinechin, “Au oma ing he pipeline o a i hme ic
da apa hs,” in Design, Au oma ion & Tes in Eu ope Con e ence &
Exhibi ion (DATE 2017), Lausanne, Swi ze land, Ma . 2017. [Online].
A ailable: h ps://hal.in ia. /hal-01373937
Louis Ledoux ecei ed his BSc deg ee in 2016
in Compu e Science om Uni e si ´
e de Rennes1,
F ance. The ollowing yea s, he pu sued his MSc
deg ee in pa allel wi h an Enginee diploma om
´
Ecole Sup´
e ieu e d’Ing´
enieu s de Rennes (ESIR).
He concluded in 2018 his s udies in Rennes wi h
a posi ion o Ha dwa e Enginee a b<>com, a na-
ional esea ch labo a o y. This posi ion allowed him
o expe imen wi h he i s FPGAs in he cloud and
hei i ualiza ions. Since 2018, he has been a PhD
candida e a he Compu e A chi ec u e depa men s
o Ba celona Supe compu ing Cen e (BSC) and Uni e si a Poli `
ecnica de
Ca alunya (UPC), Spain.