Smi h-Wa e man Accele a ion in Mul i-GPUs:
A Pe o mance pe Wa Analysis
Jes´us P´e ez Se ano(1), Edans Fla ius De Oli ei a Sandes(2),
Alba C is ina Magalhaes Al es de Melo(2), Manuel Ujald´on(1),
(1)Compu e A chi ec u e Depa men , Uni e si y o Malaga, Spain
(2)Compu e Science Depa men , Uni e si y o B asilia, B azil
Abs ac . We p esen a pe o mance pe wa analysis o CUDAlign
4.0, a pa allel s a egy o ob ain he op imal alignmen o huge DNA se-
quences in mul i-GPU pla o ms using he exac Smi h-Wa e man me hod.
Speed-up ac o s and ene gy consump ion a e moni o ed on diffe en
s ages o he algo i hm wi h he goal o iden i ying ad an ageous sce-
na ios o maximize accele a ion and minimize powe consump ion. Ex-
pe imen al esul s using CUDA on a se o GeFo ce GTX 980 GPUs
illus a e hei capabili ies as high-pe o mance and low-powe de ices,
wi h a ene gy cos o be mo e a ac i e when inc easing he numbe o
GPUs. O e all, ou esul s demons a e a good co ela ion be ween he
pe o mance a ained and he ex a ene gy equi ed, e en in scena ios
whe e mul i-GPUs do no show g ea scalabili y.
Keywo ds: GPGPU, CUDA, DNA sequences alignmen , ene gy cos s.
1 In oduc ion
The ad en o he Human Genome P ojec has b ough o he o eg ound o
pa allel compu ing a b oad spec um o da a in ensi e biomedical applica ions
whe e biology and compu e science join as a happy alliance be ween demanding
so wa e and powe ul ha dwa e. Since hen, he bioin o ma ics communi y gen-
e a es compu a ional solu ions o suppo genomic esea ch in many sub ields
such as gene s uc u e p edic ion [5], phylogene ic ees [32], p o ein docking
[23], and sequence alignmen [12], jus o men ion a ew o an ex ensi e lis .
Huge olumes o da a p oduced by geno yping echnology pose challenges
in ou capaci y o p ocess and unde s and da a. Ul a high densi y mic oa ays
now con ain mo e han 5 million gene ic ma ke s, and nex gene a ion sequenc-
ing is enabling he sea ch o causal ela ionship o a ia ion close o he single
nucleo ide le el. Fu he mo e, cu en clinical s udies include hund eds o hou-
sands o pa ien s ins ead o housands gene ically inge p in ed ew yea s ago,
ans o ming bioin o ma ics in o one o he lagships o he big da a e a.
In mode n imes o compu ing, when da a olume pose a compu a ional
challenge, he GPU immedia ely comes o ou minds. CUDA (Compu e Uni ied
De ice A chi ec u e) [21] and OpenCL [31] ha e es ablished he mechanisms
o da a in ensi e gene al pu pose applica ions o exploi GPUs ex ao dina y
2 Lec u e No es in Compu e Science: Au ho s’ Ins uc ions
powe in e ms o TFLOPS (Te a Floa ing-Poin Ope a ions Pe Second) and
da a bandwid h. Being GPUs he na u al pla o m o la ge-scale bioin o ma ics,
esea che s ha e al eady analyzed aw pe o mance and sugges op imiza ions
o he mos popula applica ions. This wo k ex ends he s udy o ene gy con-
sump ion, an issue o g owing in e es in he HPC communi y once GPUs ha e
ecen ly conque ed he g een500.o g supe compu e s lis .
Ou wo k ocuses on biological sequences alignmen in o de o ind he deg ee
o simila i y be ween hem. Wi hin his con ex , we may dis inguish wo basic
app oaches: Global alignmen , in an a emp o align he en i e leng h o he
sequence when a pai o sequences a e e y simila in con en and size, and local
alignmen , whe e egions o simila i y be ween he wo sequences a e iden i ied.
Needleman-Wunsch (NW) [20] p oposed a me hod o global compa ison based
on dynamic p og amming (DP), and Smi h-Wa e man (SW) [30] modi ied he
NW algo i hm o deal wi h local alignmen s. Compu a ional equi emen s o
SW a e o e whelming, so esea che s ei he elax hem using heu is ics as in he
well-know BLAST ool [16], o ely on high pe o mance compu ing o sho en
he execu ion ime. We ha e chosen commodi y GPUs o explo e he la e .
The es o his pape is o ganized as ollows. Sec ion 2 comple es his sec-
ion wi h some ela ed wo k. Sec ion 3 desc ibes he p oblem o compa ing wo
DNA sequences. Sec ion 4 summa izes ou p e ious s udies. Sec ions 5 and 6
in oduce ou in as uc u e o measu ing he expe imen al numbe s, which a e
la e analyzed in Sec ion 7. Finally, Sec ion 8 d aws conclusions o his wo k.
2 Rela ed Wo k
SW has become e y popula o e he las decade o compu e (1) he exac
pai wise compa ison o DNA/RNA sequences o (2) a p o ein sequence (que y)
o a genomic da abase in ol ing a bunch o hem. Bo h scena ios ha e been
pa allelized in he li e a u e [8], bu ine-g ained pa allelism applies be e o
he i s scena io, and he e o e i s be e in o many-co e pla o ms like In el
Xeon Phis [13], N idia GPUs using CUDA [26], and e en mul i-GPU using
CUDAlign 4.0 [27], which is ou depa u e poin o analyze pe o mance, powe ,
ene gy and cos along his wo k.
On he o he hand, ene gy consump ion is gaining ele ance wi hin sequence
alignmen , which p omo es me hodologies o measu e ene gy in genomic se-
quence compa ison ools.
In [4], i is minimized he powe consump ion o a sequence alignmen accele -
a o using applica ion speci ic in eg a ed ci cui (ASIC) design low. To dec ease
he ene gy budge , au ho s educe clock cycle and scale equency.
Hasan and Za a [9] p esen pe o mance e sus powe consump ion o bioin-
o ma ics sequence alignmen using diffe en ield p og ammable ga e a ays
(FPGAs) pla o ms implemen ing he SW algo i hm as a linea sys olic a ay.
Zou e al. [33] analyze pe o mance and powe o SW on FPGA, CPU and
GPU, decla ing he FPGA as he o e all winne . Howe e , hey do no measu e
eal- ime powe dynamically, bu simpli y wi h a s a ic alue o he whole un.
Smi h-Wa e man in Mul i-GPUs 3
Mo eo e , hey use models om he i s and second N idia GPU gene a ions
(GTX 280 and 470), which a e, by a , he mos inefficien CUDA amilies as a
as ene gy consump ion is conce ned. Ou analysis measu es wa s on physical
wi es and using Maxwell GPUs, he ou h gene a ion whe e he ene gy budge
has been op imized up o 40 GFLOPS/W, down om 15-17 GFLOPS/W in he
hi d gene a ion and jus 4-6 GFLOPS/W in he p e ious ones.
3 DNA Sequence Compa ison
A DNA sequence is ep esen ed by an o de ed lis o nucleo ide bases. DNA
sequences a e ea ed as s ings composed o cha ac e s o he alphabe σ=
A, T, G, C. To compa e wo sequences, we place one sequence abo e he o he ,
possibly in oducing spaces, making clea he co espondence be ween simila
cha ac e s [14]. The esul o his placemen is an alignmen .
Gi en an alignmen be ween sequences S0 and S1, a sco e is assigned o
i as ollows. Fo each pai o cha ac e s, we associa e (a) a punc ua ion ma,
i bo h cha ac e s a e iden ical (ma ch); o (b) a penal y mi, i he cha ac e s
a e diffe en (misma ch); o (c) a penal y g, i one o he cha ac e s is a space
(gap). The sco e is he addi ion o all hese alues. Figu e 1 p esen s one possible
alignmen be ween wo DNA sequences, whe e ma=+1, mi=1 and g=2.
Fig. 1. Example o alignmen and sco e.
3.1 Smi h-Wa e man
The SW algo i hm [30] is based on dynamic p og amming (DP), ob aining he
op imal pai wise local alignmen in quad a ic ime and space. I is di ided in
wo phases: calcula e he DP ma ix and ob ain he alignmen ( aceback).
Phase 1. This phase ecei es as inpu sequences S0 and S1, wi h sizes |S0|=m
and |S1|=n. The DP ma ix is deno ed Hm+1,n+1, whe e Hi,j con ains he sco e
be ween p e ixes S0[1..i] and S1[1..j]. A he beginning, he i s ow and column
a e illed wi h ze oes. The emaining elemen s o H a e ob ained om Eq. 1.
Hi,j = max







Hi−1,j−1+ (i S0[i] = S1[j] hen ma else mi)
Hi,j−1+g
Hi−1,j +g
0
(1)
4 Lec u e No es in Compu e Science: Au ho s’ Ins uc ions
In addi ion, each cell Hi,j con ains in o ma ion abou he cell ha was used
o p oduce he alue. The highes alue in Hi,j is he op imal sco e.
Phase 2 ( aceback). The second phase o SW ob ains he op imal local align-
men , using he ou pu s o he i s phase. The compu a ion s a s om he cell
ha has he highes alue in H, ollowing he pa h ha p oduced he op imal
sco e un il he alue ze o is eached.
Figu e 2 p esen s a DP ma ix wi h sco e = 5. The a ows indica e he
alignmen pa h when wo DNA sequences wi h sizes m= 12 and n= 8 a e
compa ed, esul ing in a 913 DP ma ix. In o de o compa e Megabase sequences
o , say, 60 Million Base Pai s (MBP), a ma ix o size 60,000,001 ×60,000,001
(3.6 Pe a cells) is calcula ed.
Fig. 2. DP ma ix o sequences S0 and S1, wi h op imal sco e = 5. The a ows
ep esen he op imal alignmen .
The o iginal SW algo i hm assigns a cons an cos g o each gap. Howe e ,
gaps end o occu oge he a he han indi idually. Fo his eason, a highe
penal y is usually associa ed o he i s gap and a lowe penal y is gi en o he
emaining ones (affine-gap model). Go oh [7] p oposed an algo i hm based on
SW ha implemen s he affine-gap model by calcula ing h ee alues o each
cell in he DP ma ix: H , E and F, whe e alues E and F keep ack o gaps in
each sequence. As in he o iginal SW algo i hm, ime and space complexi ies o
he Go oh algo i hm a e quad a ic.
3.2 Pa allel Smi h-Wa e man
In SW, mos o he ime is spen calcula ing he DP ma ices and, he e o e, is a
candida e p ocess o be pa allelized. F om Eq. 1, we can see ha cell Hi,j depends
on h ee o he cells: Hi−1,j ,Hi−1,j−1and Hi,j−1. This kind o dependency is
well sui ed o be pa allelized using he wa e on me hod [22], whe e he DP
ma ix is calcula ed by diagonals and all cells on each diagonal can be compu ed
in pa allel.
Figu e 3 illus a es he wa e on me hod. In s ep 1, only one cell is calcula ed
in diagonal d1. In s ep 2, diagonal d2has wo cells, ha can be calcula ed in
pa allel. In he u he s eps, he numbe o cells ha can be calcula ed in
Smi h-Wa e man in Mul i-GPUs 5
pa allel inc eases un il i eaches he maximum pa allelism in diagonals d5 o d9,
whe e i e cells a e calcula ed in pa allel. In diagonals d10 o d12, he pa allelism
dec eases un il only one cell is calcula ed in diagonal d13. The wa e on s a egy
limi s he amoun o pa allelism du ing he beginning o he calcula ion ( illing
he wa e on ) and he end o he compu a ion (emp ying he wa e on ).
Fig. 3. The wa e on me hod.
4 CUDAlign implemen a ion on GPUs
GPUs calcula e a single SW ma ix using all many-co es, bu da a dependencies
o ce neighbou co es o communica e in o de o exchange bo de elemen s. Fo
Megabase DNA sequences, he SW ma ix is se e al Pe aby es long, and so,
e y ew GPU s a egies [11, 26] allow he compa ison o Megabase sequences
longe han 10 Million Base Pai s (MBP). SW# [11] is able o use 2 GPUs
in a single Megabase compa ison o calcula e he Mye s-Mille [15] linea space
a ian o SW. CUDAlign [26] ob ains he alignmen o Megabase sequences wi h
a combined SW and Mye s-Mille s a egy. When compa ed o SW#, CUDAlign
p esen s sho e execu ion imes o huge sequences on a single GPU [11].
Compa ing Megabase DNA sequences in mul iple GPUs is mo e challenging.
GPUs a e a anged logically in a linea way so ha each GPU calcula es a subse
o columns o he SW ma ix, sending he bo de column elemen s o he nex
GPU. Asynch onous CPU h eads will send/ ecei e da a o/ om neighbo GPUs
while GPUs keep compu ing, ha way o e lapping he equi ed communica ions
wi h effec i e compu a ions whene e easible.
4.1 CUDAlign e sions
CUDAlign was implemen ed using CUDA, C++ and p h eads. Expe imen al
esul s collec ed in a la ge GPU clus e using eal DNA sequences demons a e
good scalabili y o up o 16 GPUs [27]. Fo example, using he inpu da a se
desc ibed in sec ion 5, execu ion ime was educed om 33 hou s and 20 minu es
on a single GPU o 2 hou s and 13 minu es on 16 GPUs (14.8x speedup).

6 Lec u e No es in Compu e Science: Au ho s’ Ins uc ions
Table 1 summa izes he se o imp o emen s and op imiza ions pe o med
on CUDAlign since i s incep ion, and Table 2 desc ibes all s ages and phases o
he 4.0 e sion, he one used along his wo k.
Table 1. Summa y o CUDAlign e sions.
Ve sion Majo con ibu ions Re .
1.0 Compa es on GPUs sequences o un es ic ed size using he affine [24]
gap model o SW. I p o ides he op imal sco e and he end
coo dina es o he op imal alignmen , bu no he ull alignmen .
2.0 Inco po a es he Mye s-Mille (MM) algo i hm o e ie e [25]
he ull alignmen o wo sequences in linea space.
2.1 Imp o emen s on six s ages: 1-3 un on GPUs, 4-6 on CPUs. [26]
3.0 Mul i-GPU o SW phase 1 o dis ibu e he DP ma ix, [29]
and o e lap compu a ions wi h communica ions o he CPU.
4.0 Mul i-GPU o SW phase 2, including Pipeline T aceback (PT) [27]
and Inc emen al Specula i e T aceback (IST) o es ima e
he poin whe e op imal alignmen will c oss bo de columns.
MASA Mul i-pla o m A chi ec u e o Sequence Aligne , enabling e sions [28]
o un on (1) a se ial CPU, (2) mul ico e CPU using OmpsSs,
(3) manyco e GPU using CUDA, and (4) Xeon Phi using OpenMP.
Table 2. Summa y o CUDAlign 4.0 s ages, including he SW phase i belongs o and
he p ocesso whe e i is execu ed.
S age Desc ip ion Phase Who
1 Ob ains he op imal sco e. 1 GPU
2 Pa ial aceback. 2 GPU
3 Spli ing pa i ions. 2 GPU
4 Mye s-Mille wi h balanced spli ing and o hogonal exec. 2 CPU
5 Ob aining he ull alignmen . 2 CPU
6 Ex e nal isualiza ion (op ional). 2 CPU
5 Expe imen al se up
We ha e conduc ed an expe imen al su ey on a compu e endowed wi h an In el
Xeon se e and an N idia GeFo ce GTX 980 GPU om Maxwell gene a ion.
See Table 3 o a summa y o majo ea u es.
Fo he inpu da a se , we ha e used eal DNA sequences coming om he
Nacional Cen e o Bio echnology (NCBI) [19] da abase. Resul s shown in Table
4 use sequences om asso ed ch omosomes, whe eas Tables 5 and 6 use as inpu
a pai o sequences om he ch omosome 22 compa ison be ween he human
(50.82 MBP - see [18], accession numbe NC 000022.11) and he chimpanzee
(37.82 MBP - see [17], accession numbe NC 006489.4).
To execu e he equi ed s ages on a mul i-GPU en i onmen , we pe o med
wo modi ica ions in CUDAlign 4.0: (1) a subpa i ioning s a egy o i each
Smi h-Wa e man in Mul i-GPUs 7
Table 3. Cha ac e iza ion o he in as uc u e used along ou expe imen al analysis.
CPU GPU
P ocesso Xeon E5-2620 4 GeFo ce GTX 980
8 co es @ 2100 MHz 2048 co es @ 1126 MHz
Memo y 64 GB DDR4 @ 2400 MHz 4 GB GDDR5 @ 7000 MHz
256 bi s, 76.8 GB/s 384 bi s, 336 GB/s
So wa e O.S. Ubun u 14.04.4 LTS 64 bi s CUDA 8.0
pa i ion in ex u e memo y, and (2) w i ing ex a ows in he ile sys em as
ma ks o be used in la e s ages o ind he c osses wi h he op imal alignmen .
Mo eo e , we ocus ou expe imen al analysis on he i s h ee s ages o he SW
algo i hm, which a e he ones ex ensi ely execu ed on GPUs as Table 2 e lec s.
6 Moni o ing ene gy
We ha e buil a sys em o measu e cu en , ol age and wa age based on a
Beaglebone Black, an open-sou ce ha dwa e [3] combined wi h he Accelpowe
module [6], which has eigh INA219 senso s [1]. Inspi ed by [10], wi es aken in o
accoun a e wo powe pins on he PCI-exp ess slo (12 and 3.3 ol s) plus six
ex e nal 12 ol s pins coming om he powe supply uni (PSU) in he o m o
wo supplemen a y 6-pin connec o s (hal o he pins used o g ounding).
Accelpowe uses a modi ied e sion o pmlib lib a y [2], a so wa e package
speci ically c ea ed o moni o ing ene gy. I consis s o a se e daemon ha
collec s powe da a om de ices and sends hem o he clien s, oge he wi h a
clien lib a y o communica ion and synch oniza ion wi h he se e .
Fig. 4. Wi es, slo s, cables and connec o s o measu ing ene gy on GPUs.
The me hodology o measu ing ene gy begins wi h a s a -up o he se e
daemon. Then, he sou ce code o he applica ion whe e he ene gy wan s o be
measu ed has o be modi ied o (1) decla e pmlib a iables, (2) clea and se he
wi es which a e connec ed o he se e , (3) c ea e a coun e and (4) s a i .
Once he code is o e , we (5) s op he coun e , (6) ge he da a, (7) sa e hem
o a .cs ile, and (8) inalize he coun e .
8 Lec u e No es in Compu e Science: Au ho s’ Ins uc ions
Table 4. Powe , execu ion imes and ene gy consump ion on ou GPUs o diffe en
alignmen sequences.
Sequence S age 1 S age 2 S age 3
A e age powe (wa s pe GPU)
ch 22 101.11 W 116.26 W 77.27 W
ch 21 102.11 W 116.47 W 78.89 W
47M 104.37 W 117.12 W 76.33 W
ch Y 103.25 W 119.63 W 0.00 W
Execu ion ime (seconds) To al ime
ch 22 11161.92 s 185.20 s 14.25 s 11361.38 s
ch 21 9687.36 s 61.49 s 11.03 s 9759.89 s
47M 6694.95 s 88.25 s 9.05 s 6792.26 s
ch Y 6798.12 s 3.99 s 0.00 s 6802.11 s
Ene gy consump ion (kilojules pe GPU) To al ene gy To al cos (∗)
ch 22 1128.63 kJ 21.53 kJ 1.10 kJ 4x 1151.27 kJ 0.1660 e
ch 21 989.26 kJ 7.16 kJ 0.87 kJ 4x 997.29 kJ 0.1440 e
47M 698.82 kJ 10.34 kJ 0.69 kJ 4x 709.85 kJ 0.1024 e
ch Y 701.94 kJ 0.48 kJ 0.00 kJ 4x 702.42 kJ 0.1012 e
(∗)Ene gy cos s a e shown o all ou GPUs and on an a e age a e o 0.13 e/kWh.
7 Expe imen al esul s
We s a showing execu ion imes and ene gy spen by ou diffe en sequences
on a mul i-GPU en i onmen composed o ou GeFo ce GTX 980 GPUs. Those
sequences equi e a ound 5-6 hou s on a single GPU, and he ime is educed
o less han a hal using 4 GPUs. I is no a g ea scalabili y, bu we al eady
an icipa ed he exis ence o dependencies among GPUs, hus hu ing pa allelism.
Table 4 includes he numbe s coming om his ini ial expe imen . We can see
ha s age 1 p edomina es o he execu ion ime, and ha wa age keeps s able
a ound 100 wa s o all sequences. Powe goes down o less han 80 wa s in he
hi d s age, bu i s weigh is low (negligible o he case o he ch Y sequence,
whe e s age 2 also akes li le ime).
Once we ha e seen he beha iou o all hese sequences, we ha e selec ed jus
ch 22 as he mo e s able o cha ac e ize SW om now on.
Table 5 shows he esul s o ch 22 when SW is execu ed on a mul i-GPU
en i onmen . As expec ed, powe consumed by each GPU emains s able ega d-
less o he numbe o GPUs ac i e du ing he pa alleliza ion p ocess. Execu ion
imes keep showing he al eady announced scalabili y on s age 1. Those imes
a e somehow uns able o s age 2, and inally each good scalabili y on s age 3.
Because GPUs keep compu ing on s age 1 mos o he ime, he o e all ene gy
cos is hea ily in luenced by his s age. Basically, en e ing mul i-GPU om a
single GPU execu ion doubles he ene gy cos , and hen emains s able o 3
and 4 GPUs, whe e execu ion imes a e g ea ly educes. Tha way, he pe o -
mance pe wa a io is disappoin ing when mo ing om single o win GPUs,
bu hen e ol es nicely o 3 and 4 GPUs.
Smi h-Wa e man in Mul i-GPUs 9
Table 5. Powe , execu ion imes and ene gy consump ion on diffe en numbe o GPUs
o he ch 22 alignmen sequence.
No. GPUs S age 1 S age 2 S age 3
A e age powe (wa s pe GPU)
4 101.11 W 116.26 W 77.27 W
3 101.53 W 108.16 W 78.79 W
2 100.30 W 114.68 W 76.74 W
1 102.95 W 114.44 W 81.27 W
Execu ion ime (seconds) To al ime
4 11161.92 s 185.20 s 14.25 s 11361.38 s
3 14719.32 s 253.72 s 17.70 s 14990.76 s
2 22080.04 s 159.77 s 23.17 s 22262.99 s
1 22302.24 s 291.50 s 46.65 s 22640.40 s
Ene gy consump ion (kilojules pe GPU) To al ene gy To al cos (∗)
4 1128.63 kJ 21.53 kJ 1.10 kJ 4x 1151.27 kJ 0.1660 e
3 1494.60 kJ 27.45 kJ 1.40 kJ 3x 1523.44 kJ 0.1650 e
2 2214.77 kJ 18.32 kJ 1.78 kJ 2x 2234.88 kJ 0.1614 e
1 2296.22 kJ 33.36 kJ 3.79 kJ 2333.37 kJ 0.0842 e
(∗)Ene gy cos s a e shown o all GPUs in ol ed and on an a e age a e o 0.13 e/kWh.
Table 6 summa izes gains (in ime educ ion) and losses (as ex a ene gy
cos s) on all scena ios o ou mul i-GPU execu ion o he ch 22 sequence com-
pa ison. S age 3 is he mo e ewa ding one wi h he highes ime sa ings and
he lowes ene gy penal ies, bu un o una ely, SW keeps compu ing he e jus a
ma ginal pe iod o ime. S age 2 se s eco ds in ene gy cos s, and s age 1 keeps on
an in e media e posi ion, which is wha inally cha ac e izes he whole execu ion
gi en i s hea y wo kload. The swee es scena io is s age 2 using 2 GPUs, whe e
we a e able o cu ime in hal and spend less ene gy o e all. In he opposi e
side, he wo s case goes o s age 1 using 2 GPUs, whe e ime is educed jus one
pe cen o almos double he ene gy spen . Finally, we ha e a solid conclusion on
ou GPUs, wi h ime being educed 50% a he expense o doubling he ene gy
budge . Figu e 5 p o ides de ails abou he dynamic beha iou o e ime o
each o he s ages when unning he ch 22 sequence compa ison on ou GPUs.
Table 6. Sa ings (in execu ion ime) and penal ies (in ene gy cos ) when accele a ing
SW ch 22 sequence compa ison on 4, 3 and 2 GPUs e sus a baseline on a single GPU.
S age 1 S age 2 S age 3 To al
No. Sa ings Penal y Sa ings Penal y Sa ings Penal y Sa ings Penal y
GPUs ( ime) (ene gy) ( ime) (ene gy) ( ime) (ene gy) ( ime) (ene gy)
4 49.96% 96.60% 36.47% 158.15% 69.46% 6.09% 49.82% 97.35%
3 34.01% 95.26% 12.97% 146.85% 62.06% 0.81% 33.79% 95.86%
2 1.00% 92.90% 45.20% 9.83% 50.34% -6.07% 1.67% 91.55%