Open-source GEMM hardware kernels generator: toward numerically-tailored computations

Open-Sou ce GEMM Ha dwa e Ke nels Gene a o : Towa d Nume ically-Tailo ed Compu a ions Louis Ledoux∗†, Ma c Casas∗† ∗Ba celona Supe compu ing Cen e , Ba celona, Spain †Uni e si a Poli ` ecnica de Ca alunya, Ba celona, Spain E-mail: {louis.ledoux, ma c.casas}@bsc.es Keywo ds—GEMMs, ma ix-ma ix-mul iply, ull s ack ame- wo k, au oma ed pipeline, lopoco, OpenCAPI, OpenBLAS, High Pe o mance Compu ing, app oxima e/ ans/ex ended p ecision. I. EXTENDED ABSTRACT Many scien i ic compu ing p oblems can be educed o Ma ix-Ma ix Mul iplica ions (MMM), making he Gene al Ma ix Mul iply (GEMM) ke nels in he Basic Linea Alge- b a Sub ou ine (BLAS) o in e es o he high-pe o mance compu ing communi y. Howe e , hese wo kloads ha e a wide ange o nume ical equi emen s. Ill-condi ioned linea sys ems equi e high-p ecision a i hme ic o ensu e co ec and ep o- ducible esul s [1]. In con as , eme ging wo kloads such as deep neu al ne wo ks, which can ha e millions up o billions o pa ame e s, ha e shown esilience o a i hme ic inke ing [2] and p ecision lowe ing [3]. Gene al pu pose a i hme ic uni s and compu e o ma s such as he IEEE754 s anda d na u ally unde pe o m in his as e land o scena ios. We p opose he gene a ion o nu- me ically ailo ed ci cui s whe e he necessa y and su icien in e nal p ecision is gene a ed o a ge he compu a ions equi emen s in e ms o nume ical quali y while imp o ing he ene gy cos . A. Open Sou ce SW/HW co-designed amewo k o nume i- cally ailo ed MMMs As depic ed by Fig. 1, ou amewo k is composed o wo dis inc phases, he p io Ha dwa e gene a ion low and he un ime execu ion low. Because MMMs a e basically made o a bi a y long do p oduc s, we design a cus om Fused Do P oduc (FDP) ope a o ha is agnos ic o he compu e o ma and suppo s posi , IEEE754, and b loa 16 a ia ions, while ne e ounding be ween wo accumula ions. The in e media e p ecision o he ixed-poin accumula o used in he do -p oduc is a key aspec o his wo k, and is xc u3p-ff c1517-2 OCXL ke nel module DMA Engine libocxl OpenCAPI link ; DMA ans e @ ~20GB/s I/O con ol; poin e passing; s a dma; egis e pooling A p io i Ha dwa e gene a ion low Run ime execu ion low add ess ansla ion liboc-accel ke nel & HOST HW OS & use space malloc() cas () gemm() OpenBLAS use app py o ch numpy POWER9 HOST Sha ed DDR4 AlphaDATA 9 3 Func ional speci ica ion 1 3 4 6 5 2 oc-accel OpenCAPI T anscei e s <5,-30,2> Sys olic A ay 32x31 b loa 16 Pe o mance speci ica ion lopoco EOB Signy Signx Iy.Fy Scaley Scalex isNaNy isNaNx unsigned mul iplie unsigned adde ωS 11 1 1 1 1 1 0 sign ex ension RCA S C RCA S C RCA S C isNaNoA CEOBQ CSA adix-2k ωQ 1 k-1 FTZ 0 1 0 1 0 1 ωF+1 Ix.Fx 2ωF+2 ωS+1 2ωF+2 shi al gen pa selec ωShi 1 1 ba el shi pad LSBMSB OVF 1 oo_big oo_small con ig. x hwgen.py N,M LSB,MSB,OVF A i hme ic Chip F eq Sys olic A ay Gene a ion buses w apping VUP.cpp FDP.cpp PE PE PE PE PE PE PE PE PE A2S3 A2S3 A2S3 L2AL2AL2A columns B ows A SOB EOB columns C A2S3 A2S3 A2S3 Legend: HSSD da a inpu da a (inpu o ma ) in e media e da a (S3 o ma ) con ol ou pu da a (ou pu o ma ) FF_delay LUT_delay C8_delay Adde model A B C D Fig. 1. O e iew o he 2 phases amewo k. Le is Run ime execu ion low and igh is Ha dwa e gene a ion low. 27,7× 5× 15,1× 5,6× Fig. 2. Sea Su ace Heigh compu a ion compa ing IEEE-754 double-, quad- pecision FMAs and a 91-bi FDP w nume ical quali y and powe consump ion. con igu able h ough he leng h o he sc a chpad delimi ed by he pa ame e s MSB (Mos Signi ican Bi ) and LSB (Leas Signi ican Bi ). We le e age he au oma ed pipeline ea u e o lopoco [4] which is an e ec i e ool o e icien ly explo ing he wide ange o unc ional speci ica ions along wi h pe o mance speci ica ions o p oduce MMM ke nels wi h he necessa y basic elemen s (LUTs, FFs, Ca y chains, DSPs) o a a ge ed (chip, equency)couple (see Fig. 1- B ). The essence o his wo k is o make in e media e p ecision weakings om he ha dwa e accessible o high-end so wa e code as anspa en as possible. We achie e ha by aking in o accoun ha many HPC codes ely on BLAS lib a ies o pe o m MMM ope a ions. Such lib a ies ecei e he unc ion call o pe o m a GEMM and dispa ch adequa ely o he unde lying ha dwa e a hei disposal. B. HPC wo kloads esul s We expe imen wi h wo amilies o eal HPC wo kloads wi h con as ing nume ical equi emen s, namely A i icial In elligence (AI) and Sea Su ace Heigh (SSH), whose e- spec i e esul s can be obse ed in Fig. 3 and Fig. 2. Fig. 3. Top1 Accu acy s alida ion da ase in e ence Ene gy cos o a ious combina ions o da ase s,models,compu e o ma s, and accumula o s. Fo he SSH compu a ion, he esul s ob ained wi h 64-bi and 128-bi FPUs exhibi dec easing ep oducibil- i y as he ec o size inc eases. In con as , ou 91-bi ⟨o : 30, msb : 30, lsb : 30⟩FDP main ains ep oducibili y o all ec o sizes wi hou de ia ion. Ou p oposed FDP consis en ly exhibi s 52 co ec bi s, which is a leas 5×and 27.7×mo e han quad-p ecision and double-p ecision. Ou measu emen s on VU3P-2 FPGA a 200MHz show ha he uni s powe consump ion a e 0.266, 0.549, and 0.491 wa s o double-p ecision FMA, quad-p ecision FMA, and he 91-bi FDP, espec i ely. Fo all e alua ed sizes, he 91-bi FDP yields a leas 5.6×and 15.1×mo e co ec bi s o he same wa age as quad-p ecision and double-p ecision FMAs, espec i ely. Fo AI wo kloads, we employ Py o ch as a base amewo k and link i o ou modi ied OpenBLAS. We use popula neu al ne wo k models such as ResNe 18, ResNe 34, ResNe 50, DenseNe 121, DenseNe 161, DenseNe 169, and VGG11 wi h ba ch no maliza ion, and e alua e hem on he CIFAR-10 and ImageNe da ase s. To measu e powe consump ion and accu acy, we use he B ainFloa 16 and IEEE-754 32-bi o ma s o ou compu a ions wi h a la ge a ie y o accumula o s a ying hei OV F ,MSB, and LSB pa ame e s. Fig. 3 shows he ela ionship be ween powe consump ion and accu acy o di e en accumula o and a i hme ic combina ions. Fo example, i 84% Top1 accu acy is sa is ying o Imagene wi h Resne 50, he mos sui ed a i hme ic/accumula o combina ion is IEEE-754 32- bi /⟨o : 9, msb : 6, lsb :−20⟩ ep esen ed by a ligh pu ple hexagon as all o he ma ke s a e ei he on he igh o below. C. Conclusion O e all, ou wo k highligh s he impo ance o nume ically ailo ed accumula o s o ep oducibili y in scien i ic compu - ing applica ions. Ou esul s p o ide aluable insigh s in o he ade-o s be ween powe consump ion and accu acy, and we belie e ha ou esul s ha e he po en ial o in o m he design o u u e AI and scien i ic compu ing sys ems, and we encou age o he esea che s o explo e he possibili ies o low p ecision accumula o s using ou open-sou ce amewo k. II. ACKNOWLEDGMENT Ma c Casas is suppo ed by G an RYC-2017-23269 unded by MCIN/AEI/ 10.13039/501100011033 and by “ESF In es ing in you u u e”. REFERENCES [1] D. Bailey and J. M. Bo wein, “High-P ecision Compu a ion and Ma hema ical Physics,” in P oceedings o XII Ad anced Compu ing and Analysis Techniques in Physics Resea ch — PoS(ACAT08). E ice, I aly: Sissa Medialab, Oc . 2009, p. 014. [Online]. A ailable: h ps://pos.sissa.i /070/014 [2] J. Johnson, “Re hinking loa ing poin o deep lea ning,” a Xi :1811.01721 [cs], No . 2018, a Xi : 1811.01721. [Online]. A ailable: h p://a xi .o g/abs/1811.01721 [3] M. Cou ba iaux, I. Huba a, D. Soud y, R. El-Yani , and Y. Bengio, “Bina ized Neu al Ne wo ks: T aining Deep Neu al Ne wo ks wi h Weigh s and Ac i a ions Cons ained o +1 o - 1,” a Xi :1602.02830 [cs], Feb. 2016, a Xi : 1602.02830. [Online]. A ailable: h p://a xi .o g/abs/1602.02830 [4] M. Is oan and F. de Dinechin, “Au oma ing he pipeline o a i hme ic da apa hs,” in Design, Au oma ion & Tes in Eu ope Con e ence & Exhibi ion (DATE 2017), Lausanne, Swi ze land, Ma . 2017. [Online]. A ailable: h ps://hal.in ia. /hal-01373937 Louis Ledoux ecei ed his BSc deg ee in 2016 in Compu e Science om Uni e si ´ e de Rennes1, F ance. The ollowing yea s, he pu sued his MSc deg ee in pa allel wi h an Enginee diploma om ´ Ecole Sup´ e ieu e d’Ing´ enieu s de Rennes (ESIR). He concluded in 2018 his s udies in Rennes wi h a posi ion o Ha dwa e Enginee a b<>com, a na- ional esea ch labo a o y. This posi ion allowed him o expe imen wi h he i s FPGAs in he cloud and hei i ualiza ions. Since 2018, he has been a PhD candida e a he Compu e A chi ec u e depa men s o Ba celona Supe compu ing Cen e (BSC) and Uni e si a Poli ` ecnica de Ca alunya (UPC), Spain.