Quantifying performance: Benchmarks: v.1.0

Quan i ying pe o mance Benchma ks V 1.0 José M. Cáma a ([email protected]) Mo i a ion Pe o mance mus be measu ed o: Assess he beha io o a compu ing sys em. Compa e a ious sys ems. Op imize u iliza ion. Remo e bo lenecks. Quan i ying pe o mance is a daun ing ask: Sys ems a e dis inc om one ano he . They a e e y complex. Handle a b oad a ie y o applica ions and da a. Me ics La ency: ime o comple e an ac ion: deli e a message, execu e a p og am, ul ill a eques … Acco ding o he scena io i can be exp essed in e ms o : execu ion ime o esponse ime oo. Th oughpu : asks comple ed pe ime uni : ins uc ions, messages, que ies… Th oughpu = 1/ la ency only when no o e lap is p oduced (ins uc ion o message pipe-line). O he wise h oughpu > 1/ la ency. Benchma ks Concep : applica ion p og am used o quan i y compu e ’s pe o mance. Goal: esul s mus be nume ic, objec i e and ai . Types:  Real p og ams. Syn he ic. Ke nels. Toys. Sui es. Benchma ks Real p og ams may seem he mos objec i e op ion bu in many occasions hei esul s a e ha d o in e p e since oo many compu e sys ems a e a ec ed by hem in unce ain and a iable ways. Syn he ic benchma ks a e designed o e lec he pe o mance o ce ain subsys ems. The e o e, in o de o e alua e all subsys ems, usually a sui e is p e e ed. Ke nels a e close o eal p og ams. They elimina e all ha is no ele an , such as use in e ace, calcula ion esul s, e c. Toy benchma ks a e sho p og ams ha p oduce esul s al eady known o he use . E alua ing esul s Benchma k sui es a e a common ool bu since hey a e composed by a numbe o p og ams. Sys ems may pe o m di e en ly unde each one. Di ec compa ison is no possible hen. A mo e elabo a ed me ic mus be ob ained: A i hme ic mean:𝐴𝐴𝐴𝐴 = ∑ 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 𝑁𝑁. No accu a e when he es s a e un ela ed. The longes es ends o p e ail. Geome ic mean:𝐺𝐺𝐴𝐴 = ∏ 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 𝑁𝑁 . I s u ili y is unclea . Ha monic mean:𝐻𝐻𝐴𝐴 =𝑁𝑁 ∑ 1 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 Resul s 𝑟𝑟𝑖𝑖 may be absolu e execu ion imes (o hei in e ses) o speed-ups ( e e ed o a p ecise sys em). Compa ing means Le ’s hink o a se o es s whe e he execu ion imes a e: 1s, 4s and 10s. The a i hme ic mean o hem is 5s. This is co ec bu , aking in o accoun ha he sho es p og ams may be execu ed mo e imes han he longes , maybe hey should weigh mo e on he a e age. The ha monic mean is 3/1,35 = 2,22s. Tha could e lec mo e accu a ely he expec ed pe o mance o he sys em. Selec ing benchma ks Dis inc sys ems equi e dis inc benchma ks. We a e conside ing wo ypes o se e s: supe compu e s & da a cen e s. Fo supe compu e s, ega dless he subsys em we in end o s ess, wha ma e s is execu ion ime ( h oughpu is also ele an in many cases). Resul s a e usually gi en in e ms o FLOPS. Fo da a cen e s he esponse ime is wha he use pe cei es as “pe o mance”. Th oughpu doesn’ make much sense in his en i onmen . Resul s may be gi en in e ms o SLO/SLA a io. Fo example: 99% o esponses below 100ms. Benchma king supe compu e s By a , he mos popula benchma k o supe compu e s is Linpack. P o ides pe o mance in FLOPS unde he execu ion o a ke nel based on he esolu ion o sys ems o linea equa ions. I has a numbe o a ia ions o adap i sel o many compu e a chi ec u es. Is he es bed o he Top500 supe compu e ank. NAS NPB: sui e o ke nels de eloped by NASA. Ke nels y o e lec he co e o calcula ions commonly pe o med by luid mechanics applica ions and o he usual p og ams ela ed o i s ac i i y. All benchma ks impose hea y calcula ion on he sys em; di e on he p oblem hey sol e and he amoun o communica ion a ic hey gene a e: EP: Emba assingly Pa allel. I in ol es calcula ion bu e y li le communica ion be ween p ocesso s. MG: Mul ig id. Unlike he o me one, i equi es communica ion be ween bo h close and emo e p ocesso s. CG: Conjuga e G adien . Communica ion is low and sca e ed among close and emo e nodes. FT: Fou ie T ans o m. Hea y communica ion pa e n e enly dis ibu ed. IS: In ege So . Communica ion, al hough hea y and uni o m is no as hea y and uni o m as in he p e ious case. LU, SP & BT: add ess he same ma hema ical p oblem using di e en algo i hms. BT is “less pa allel” han he es .