... manusc ip No.
(will be inse ed by he edi o )
Mul i-De ice Con olle s: A Lib a y To Simpli y
The Pa allel He e ogeneous P og amming
Ana Mo e on-Fe nandez ·A u o
Gonzalez-Esc ibano ·Diego R. Llanos
Recei ed: da e / Accep ed: da e
Abs ac Cu en HPC clus e s a e composed by se e al machines wi h di -
e en compu a ion capabili ies and di e en kinds and amilies o accele a o s.
P og amming e icien ly o hese he e ogeneous sys ems has become an im-
po an challenge. The e a e many p oposals o simpli y he p og amming and
managemen o accele a o de ices, and he hyb id p og amming, mixing ac-
cele a o s and CPU co es. Howe e , he po abili y comp omises in many cases
he e iciency on di e en de ices, and he e a e de ails abou he coo dina ion
o di e en ypes o de ices ha should be s ill ackled by he p og amme .
In his wo k we in oduce he Mul i-Con ole (MC l), an abs ac en i y
implemen ed in a lib a y, ha coo dina es he managemen o he e ogeneous
de ices, including accele a o s wi h di e en capabili ies and se s o CPU-
co es. Ou p oposal imp o es s a e-o - he-a solu ions, simpli ying he da a
pa i ion, mapping, and anspa en deploymen o bo h, simple gene ic ke -
nels po able ac oss di e en de ice ypes, and specialized implemen a ions
de ined and op imized using speci ic na i e o endo p og amming models
(such as CUDA o NVIDIA’s GPUs, o OpenMP o CPU-co es). The un-
ime sys em au oma ically selec s and deploys he mos app op ia e imple-
men a ion o each ke nel o each de ice, managing he da a mo emen s, and
hiding he launching de ails. Resul s o an expe imen al s udy wi h i e s udy
cases indica es ha ou abs ac ion allows he de elopmen o lexible and
high e icien p og ams, ha adap o he he e ogeneous en i onmen .
Keywo ds Accele a o s, Hyb id compu a ion, GPUs, Ke nel cha ac e iza-
ion, Memo y ans e s
A. Mo e on-Fe nandez ·A. Gonzalez-Esc ibano, ·D.R. Llanos
Depa amen o de In o m´a ica, Uni e sidad de Valladolid, Valladolid, Spain
A. Mo e on-Fe nandez E-mail: [email p o ec ed]a.es
A. Gonzalez-Esc ibano E-mail: [email p o ec ed]a.es
D. R. Llanos E-mail: [email p o ec ed]a.es
2 Ana Mo e on-Fe nandez e al.
1 In oduc ion
Cu en HPC clus e s a e composed by se e al machines wi h di e en compu-
a ion capabili ies and accele a o s, such as G aphics P ocessing Uni s (GPUs)
o XeonPhi cop ocesso s [20]. I has been p o ed ha he use o hese he e oge-
neous sys ems imp o es he pe o mance in many applica ions [18]. Howe e ,
p og amming e icien ly o hese he e ogeneous sys ems has become an im-
po an challenge. The di e en compu a ional uni s (CPUs, GPUs, XeonPhi,
..) ha can o m a clus e , ypically ha e di e en p og amming equi emen s
and cons ain s o achie e he bes pe o mance. Thus, di e en p og amming
models a e used o each kind o de ice. Fo example, CUDA p og amming
model achie es he bes pe o mance in NVIDIA GPUs [8], and OpenMP has
been shown o be an e icien p og amming model o mul i-co es in sha ed-
memo y sys ems o XeonPhi cop ocesso s.
The e a e many p oposals o simpli y he p og amming and managemen
o accele a o de ices, and he hyb id p og amming mixing accele a o s and
CPU co es. Howe e , he po abili y comp omises in many cases he e iciency
on di e en de ices. Depending on he p oposal, some de ails abou he co-
o dina ion o di e en ypes o de ices a e s ill ackled by he p og amme ,
such as compu a ion pa i ion and balance, da a mapping and locali y, o da a
mo emen coo dina ion ac oss di e en memo y hie a chies.
In his wo k we in oduce he Mul i-Con ole (MC l), an abs ac en i y
implemen ed in a lib a y, ha coo dina es he managemen o he e ogeneous
de ices, including accele a o s wi h di e en capabili ies and se s o CPU-
co es. I helps he p og amme o handle he compu a ion pa i ion, mapping,
and anspa en execu ion o complex asks in such hyb id and he e ogeneous
en i onmen , independen ly o he a ge de ices selec ed a un- ime. Ou
p oposal allows he exploi a ion o simple gene ic ke nels ha can i in any
de ice, e y specialized ke nels de ined and op imized by he p og amme o
each a chi ec u e, and e en w appe s o call hi d-pa y p ede ined lib a ies
(such as e.g. cuBLAS [14]). This allows he exploi a ion o na i e o endo
speci ic p og amming models, in a highly e icien way. The mos app op ia e
ke nels o each a ge de ice a e au oma ically selec ed by he en i y du ing
he p og am execu ion.
Ou wo k is de eloped on he concep o Con olle p esen ed in [1,12].
While he Con olle anspa en ly manages he da a mo emen s and he
launching o se ies o ke nels on a gi en a ge de ice, he Mul i-Con olle
coo dina es se e al Con olle s associa ed o di e en de ices o g oups o
CPU-co es. I is implemen ed as an ex ensible lib a y, and i can use he bes
p og amming models, ools, and compile s o each po en ial de ice.
We p esen an expe imen al s udy wi h i e case s udies. We show ha
ou app oach is highly lexible, wi h minimum p og amming e o o chang-
ing he a ge de ices. The esul s o a pe o mance s udy compa ing ou
app oach wi h op imized e e ence codes show ha ou implemen a ion does
no in oduce signi ican pe o mance penal ies.
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 3
The es o he pape is o ganized as ollows: Sec ion 2 p esen s he ela ed
wo k. Sec ion 3 in oduces he lib a ies used o build ou p oposal. Sec ion 4
explains he p oposed model. Sec ion 5 shows he expe imen al s udy, and
inally Sec . 6 exposes he conclusions and u u e wo k.
2 Rela ed wo k
In his sec ion we analyse di e en wo ks p oposed in he li e a u e ha a -
ge he simpli ica ion o pa allel p og amming on he e ogeneous sys ems, com-
posed by di e en compu a ional de ices, including accele a o s. These p o-
posals a oid he need o using a manual combina ion o he speci ic p o-
g amming models o each compu a ion de ice. They in oduce uni ied p o-
g amming models o ool abs ac ions o manage he a chi ec u al di e ences
be ween compu a ional uni s o di e en na u e o capabili ies.
A widesp ead p og amming amewo k o deal wi h he e ogeneous de ices
is OpenCL [19]. The OpenCL con ex abs ac ion allows he managemen o
mul iple de ices o he same na u e (using he same pla o m in OpenCL
no a ion). Howe e , he coo dina ion o de ices o di e en na u es, and he
managemen o da a sha ing o pa i ioning, compu a ion mapping, load bal-
ancing, and communica ion ac oss hem a e icky, and should be manually
sol ed and coded by he p og amme . Mo eo e , he abs ac ions in oduced
by OpenCL de i e in many cases in no eaching he bes possible pe o -
mance (see e.g. [8]). Many lib a ies o highe le el o abs ac ion, ha ely
on OpenCL as execu ion laye , ypically inhe i some o hese p oblems (see
e.g. [19]). An in e es ing example o abs ac ion buil on op o OpenCL is
he Maa lib a y [16]. I p o ides a uni ied con ex wi h an abs ac iew e-
ga dless o he numbe and na u e o de ices, o GPU and CPU pla o ms.
The main di e ences wi h ou p oposal a e ela ed o ou choice o in e nally
using na i e o endo low-le el p og amming models. We p opose he possi-
bili y o decla ing bo h uni ied and specialized/op imized ke nels o di e en
a chi ec u es, which a e selec ed a un- ime o each pa icula de ice. Ou
p oposal allows he exploi a ion o ea u es o he na i e p og amming models,
o specialized hi d-pa y lib a ies op imized o he de ice. We in oduce mo e
lexibili y o selec he desi ed de ices associa ed o a mul icon olle , applying
echniques such as g ouping CPU-co es as a single de ice, o exploi na i e
mul i- h eaded pa allelism inside a ke nel. The pe o mance esul s p esen ed
o he Maa lib a y a e only compa ed wi h o he OpenCL implemen a ions.
I has no been s a ed ye he pe o mance e ec s compa ing wi h using mo e
speci ic p og amming models such as combina ions o CUDA o GPUs, and
OpenMP o mul ico e CPUs.
Mo e gene al app oaches p opose comple e in eg a ed amewo ks ha ex-
ploi lowe -le el speci ic p og amming models. Some examples include OM-
PICUDA [9], Casheme e [6], S a PU [7] o he skele on p og amming ame-
wo k based on i , SkePU [2]. PACXX [4] is a ans o ma ion sys em in eg a ed
in o he LLVM compile amewo k. I gene a es code o di e en kinds o
4 Ana Mo e on-Fe nandez e al.
de ices, and i ans o ms explici pa allel cons uc ions ha use he concep
o ke nel and launching in an abs ac an elegan o m. Scogland e al [17]
also p oposes a mixed app oach by adap ing OpenMP p agmas o he e oge-
neous sys ems. In gene al all hese app oaches hide he coo dina ion de ails o
he p og amme , o he poin o cons aining he po en ial op imiza ions ha
could be achie ed manually. Techniques o selec launching pa ame e s like he
h eadblock size a e o example ackled in SkePU using ial-and-e o , wi h
no possibili y o ex apola e he esul s o o he ke nel codes o a chi ec u es.
Unlike p e ious app oaches, ou lib a y p oposes a lexible ool, ha allows
he p og amming wi h gene ic and po able ke nels, and a he same ime he
in eg a ion o highe le els o op imiza ion using he na i e o endo p o ided
p og amming models, ools, o lib a ies, o highe e iciency and pe o mance.
The lexibili y in e ms o p og amme con ol o he de ice selec ion and
coo dina ion a un- ime also imp o es p e ious echniques.
3 Backg ound
The Mul i-Con olle lib a y we a e p esen ing in his pape is buil on op o
wo p e ious ools. The i s one, named Hi map [3], is a lib a y o manage
he pa i ion and mapping o da a s uc u es. I is used in ou model o man-
age he da a dis ibu ion ac oss de ices, and o p o ide a common in e ace
o implemen da a managemen inside gene ic po able ke nels. The second
one, named Con olle [1,12], is a lib a y ha de ines an abs ac en i y o
anspa en ly manage he da a mo emen s and ke nel launching o a single
de ice. Ou p oposal combines hem wi h a new laye o abs ac ion o co-
o dina e he use o se e al de ices o di e en a chi ec u es o na u es. This
sec ion in oduces he eade o bo h p e ious ools, be o e desc ibing he new
abs ac ion p oposed.
3.1 Hi map lib a y
Hi map is a lib a y designed o hie a chical iling and mapping o dense and
spa se a ays, o g aphs. Hi map is based on a dis ibu ed SPMD p og amming
model, using abs ac ions o decla e da a s uc u es wi h a global iew. I
au oma izes he pa i ion, mapping, and communica ion o hie a chies o iles,
while s ill deli e ing good pe o mance [3].
An objec Hi Shape ep esen s a subspace o domain indexes. Fo dense
a ays, i is de ined as an n-dimensional ec angula pa allelo ope. The limi s
on each dimension a e ep esen ed wi h a Hi Sig objec , con aining he ange
limi s. Each Hi Sig objec is a uple o h ee in ege numbe s S= (b, e, s)
(begin, end, and s ide), ep esen ing he indexes in one o he axis o he
domain. Begin and s ide membe s o a Signa u e ep esen he coe icien s o
a linea unc ion S(x) = sx +b.
Hi map de ines he Hi Tile s uc u e, an abs ac en i y o n-dimensional
a ays and iles. A Hi Tile s uc u e is a handle o s o e a ay me a-da a,
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 5
1/* Ke nel cha ac e iza ions */
2KERNEL_CHAR(Ma Add,1, ull,low,low)
3
4/* Gene ic ke nel codes o any de ice */
5KERNEL(Ma Add, 3, OUT, Hi Tile_ loa , C, IN, Hi Tile_ loa , A,
6IN, Hi Tile_ loa , B ) {
7in x = h ead.x;
8in y = h ead.y;
9hi _ ileElem( C, 2, x, y ) =
10 hi _ ileElem( A, 2, x, y ) + hi _ ileElem( B, 2, x, y );
11 }
Fig. 1 Ke nel de ini ion and con igu a ion o a ma ix addi ion using he Con olle lib a y.
along wi h he poin e o he ac ual memo y space o he da a. A Hi Tile
maps ac ual da a elemen s o he index subspace de ined by a shape. The e
a e only ou unc ions o Hi map needed o wo k wi h ou Mul i-Con olle
lib a y. Fi s , hi ileDomain and hi ileDomainAlloc a e used o decla e he
index domains o a ile a ay. The second one also alloca es he memo y o
he da a. The unc ion hi ileF ee is used o ee he da a memo y and clean
he handle . The unc ion hi ileElem is used in hos o ke nels code o access
he elemen s o a ile. I ecei es a ile name, a numbe o dimensions, and
he indexes alues o he desi ed elemen . The da a a e accessed in ow majo
o de in all cases, independen ly o he implemen a ion.
Hi map lib a y includes many o he unc ionali ies. Fo example, he man-
agemen o hie a chical subselec ions o pa s o he iles, and he anspa en
managemen o dis ibu ed-a ays, wi h abs ac pa i ion and communica-
ion unc ionali ies ha in e nally use a message-passing pa adigm (exploi -
ing MPI). Hi map has been ex ended wi h a much smalle handle (Hi KTile),
wi h he minimum in o ma ion needed o da a accesses o mul idimensional
a ays h ough a da a poin e , ha is use ul o anspa en ly po he da a
s uc u es o accele a o s in a mo e e icien o m.
3.2 Con olle s lib a y
The second abs ac ion upon we ha e buil ou p oposal is implemen ed in he
Con olle s lib a y. I in oduces an abs ac en i y ha allows he anspa en
launching o se ies o asks on a single accele a o de ice, also conside ing a
g oup o CPU co es as a many-co e de ice. The Con olle model p esen s se -
e al impo an ea u es: (1) A mechanism o de ine common ke nels eusable
ac oss di e en ypes o de ices, o specialized ke nels o speci ic de ice kinds;
(2) A anspa en mechanism o memo y managemen , including op imized
communica ions o he da a s uc u es be ween he hos and he co espond-
ing images in he accele a o s; (3) An op imiza ion sys em o selec p ope
alues o ke nel-launching con igu a ion pa ame e s (such as he h eadblock

6 Ana Mo e on-Fe nandez e al.
geome y), guided by simple quali a i e code cha ac e iza ion p o ided by he
p og amme .
In ou p oposal we use he Con olle en i y o manage each de ice inside
he new Mul i-con olle ha coo dina es hem. In he Con olle lib a y, a
ke nel is decla ed by using he p imi i e KERNEL < ype>. Whe e ype may be
emp y o indica e a ke nel usable on any kind o de ice, o a speci ic alue o
a specialized code o a gi en ype o de ice. This is use ul when di e en op i-
miza ions on he ke nel code a e equi ed o di e en de ices. Cu en ly, he
lib a y suppo s he speci ic decla a ions KERNEL GPU o CUDA code a ge -
ing NVIDIA’s GPUs, KERNEL CPU o hos machine code a ge ing se s o CPU
co es, and KERNEL GPU WRAPPER,KERNEL CPU WRAPPER, o hos machine code
which includes calls o specialized GPU o CPU lib a ies, such as cuBLAS o
MKL ou ines. Mo eo e , he wo k p esen ed in [11] ex ended he Con olle
lib a y o suppo he Xeon Phi co-p ocesso .
The ke nel-de ini ion p imi i es decla e in b acke s he numbe o pa am-
e e s o he ke nel, wi h a uple o in o ma ion o each pa ame e . The pa-
ame e in o ma ion includes i s ype, name, and inpu /ou pu ole:
–IN: o inpu Hi Tile pa ame e s, whose elemen s a e only ead.
–OUT: o ou pu Hi Tile pa ame e s, whose elemen s a e only w i en.
–IO: o inpu and ou pu Hi Tile pa ame e s, wi h elemen s can be bo h
ead and w i en.
–INVAL: o inpu pa ame e s o any ype passed by alue.
The p og amme can also p o ide a ke nel cha ac e iza ion, in e ms o
code ea u es, ha helps o au oma ically de e mine p ope ke nel launching
pa ame e s. In he cu en p o o ype, he CPU h eads g anula i y is de e -
mined by a simple egula blocking policy, ha does no equi e a speci ic
ke nel cha ac e iza ion. Fo GPU ke nels, he lib a y in eg a es he model
p esen ed in [15,21]. This model allows he de e mina ion o con igu a ion pa-
ame e s (g id, h eadblock, and L1 cache memo y sizes), o NVIDIA’s GPUs.
The p imi i e KERNEL CHAR ecei es he ke nel name, he numbe o dimensions
o he h ead space (1, 2, o 3), and desc ip i e alues o he cha ac e iza ion
model. These alues a e a quali a i e desc ip ion o cha ac e is ics o he ke -
nel code p o ided by he p og amme . They a e ela ed o: (a) The coalescing
p ope y o he global memo y access pa e ns ( ull, medium, sca e ); (b) The
a io o a i hme ic/logic ope a ions pe global memo y access (high, medium,
low); and (c) The a io o da a sha ing accesses in a block pe global memo y
access (high, medium, low).
Figu e 1 shows an example o he ke nel o a ma ix addi ion compu a ion.
We see a ke nel cha ac e iza ion in line 2 and a ke nel de ini ion in lines 5 and 6.
4 Mul iple-De ice Con olle (MC l) lib a y
The Mul iple-De ice Con olle (MC l) lib a y p o ides a simpli ied way o
p og am applica ions a ge ing he e ogeneous sys ems wi h di e en kinds o
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 7
Execu ion
policyQueue
C l-0
Execu ion
policyQueue
C l-1
Execu ion
policyQueue
C l-2
Many-co e
Memo y
De -0
De -1
De -2
A[0]
Mul i-co e
Memo y
A[1]
Many-co e
Memo y
A[2]
Ke nel
di ision
Da a
pa i ion
MC l
Hos
MA ach
MLaunch
MDe ach
Hi Tile A, B
A
Code
Memo y
A ached
Mul iCon olle De ices
CPU
CPU
CPU
CPU
A
BB
B[0]
B[1]
B[2]
In e nal
X
Fig. 2 Diag am o he Mul iple-De ice Con olle lib a y (MC l).
compu a ional uni s. In his pape , we de ine compu a ional uni / a ge de ice
as an accele a o s (GPU, Xeon Phi, e c.) o a g oup o CPU-co es conside ed
as a single independen de ice. The goal o his lib a y is: (1) o au oma ize he
da a pa i ion and da a ans e be ween he hos and mul iple a ge de ices,
as well as (2) o anspa en ly coo dina e he di ision and execu ion o he
compu a ion among di e en compu a ional uni s, independen ly o he kind
o a ge de ice exploi ed (GPUs, g oup o CPU-co es, e c).
The lib a y has an objec -o ien ed design, despi e he ac ha i is mainly
de eloped in C language. The classes a e implemen ed as C s uc u es wi h
associa ed unc ions. The Mul i-Con olle model a chi ec u e is p esen ed in
he Fig. 2.
The Mul i-Con olle objec p o ides unc ions o manage:
– Mul i-de ice coo dina ion: The Mul i-Con olle is associa ed wi h a
se o di e en de ices a cons uc ion. I in e nally c ea es Con olle ob-
jec s o in e ac wi h each de ice. The Mul i-Con olle p o ides an ab-
s ac in e ace ha enables o manage i as a single compu a ional de ice,
independen ly o he in e nally associa ed de ices.
– Da a s uc u es: The Mul i-Con olle abs ac ion c ea es an uni ied
memo y con ex o all o he associa ed de ices, whe e in e nal da a s uc-
u es can be c ea ed, o da a s uc u es om he main hos h ead can be
a ached. Da a s uc u es can be eplica ed o pa i ioned and dis ibu ed
ac oss he de ices as he p og amme equi es. In he cu en p o o ype,
a simple s a ic pa i ioning me hod has been included. I pa s he s uc-
u es in as many i egula pa s as numbe o de ices we e selec ed in
he Mul i-Con olle cons uc ion. The size o each pa is calcula ed p o-
po ionally o a lis o weigh s. Da a mo emen s ac oss di e en de ice
memo y hie a chies a e anspa en ly managed by he in e nal Con olle
objec s associa ed o each de ice.
– Ke nel de ini ion and launching: The Mul i-Con olle model in e-
g a es he Con olle s idea o mul i- e sion ke nel de ini ion. Thus, ke nel
launching in a Mul i-Con olle simply uses a ke nel name. The in e nal
8 Ana Mo e on-Fe nandez e al.
Con olle s selec s, a un- ime, he mos app op ia e ke nel e sion o
implemen a ion o he associa ed de ice, among hose p o ided by he
p og amme . The Mul i-Con olle in e nally di ides he compu a ion as-
socia ed o a ke nel launching among he di e en de ices. The ke nel exe-
cu ion on a de ice is pe o med asynch onously wi h espec o he ke nels
execu ion in he es o de ices. Synch oniza ions a e equi ed only by da a
eques s on he main hos h ead.
4.1 Mul i-Con olle cons uc ion
A Mul i-Con olle objec is cons uc ed o manage a speci ic collec ion o
de ices. I s cons uc ion unc ionali y ecei es an o de ed lis o de ices spec-
i ica ions. Each de ice speci ica ion is used o in e nally c ea e a Con olle
objec associa ed o he compu a ional esou ce. In he cu en p o o ype we
suppo de ice speci ica ions ha include: (1) NVIDIA’s GPUs, speci ying
hei CUDA de ice numbe , and (2) G oups o main hos CPU-co es, speci-
ying a ange o co e iden i ie s, acco ding o hei in e nal numbe ing in he
CPU in o ma ion p o ided by he ope a ing sys em.
The Mul i-Con olle in e nally c ea es a queue o empo a ily s o e he
eques s o ke nel launchings, be o e di iding he compu a ion and mapping
i o he queues o he in e nal de ice Con olle s. The synch oniza ion and
coo dina ion ope a ions o each Con olle a e execu ed on i s own ask, which
makes asynch onous he use o a hos h ead only when ac i i y is needed, o
minimal in e e ence wi h o he hos h eads. In he cu en p o o ype, he
in e nal de ice Con olle s a e implemen ed using OpenMP asks.
4.2 Da a s uc u es and domains
One o he objec i es o he Mul i-con olle lib a y is o p o ide an homo-
geneous in e ace o wo k wi h da a s uc u es in di e en de ice ypes, p e-
se ing he coalescing o ec o iza ion p ope ies o he code due o he da a
accesses o de . The p e ious Con olle s lib a y uses Hi map o p o ide such
in e ace.
Fo Mul i-Con olle we p opose o exploi and ex en Hi map unc ional-
i ies o p o ide a anspa en abs ac ion o he pa i ion, subselec ion, and
mapping o pa s o he da a s uc u es o he di e en de ices associa ed
o a Mul i-Con olle . The Mul i-Con olle model p oposes a single memo y
con ex o he whole se o associa ed he e ogeneous de ices. Da a s uc u es
om he main hos h ead can be a ached o he Mul i-Con olle con ex ,
and hey should no be manipula ed on he main hos h ead un il hey a e
de ached om he Mul i-Con olle . The Mul i-Con olle can decide when
he eal da a mo emen s should be done, synch onously o asynch onously, o
he ac ual de ices, depending on he ke nels enqueued o execu ion, and hei
da a dependencies.
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 9
To a oid edundan da a mo emen s ac oss memo y hie a chies, he model
p o ides he p og amme wi h a lexible a achmen unc ionali y. The cu en
p oposal is ocused on applica ions whe e: (1) No da a ans e s be ween de-
ices a e needed ac oss se e al ke nel execu ions; and (2) any pa o he com-
pu a ion needs ei he , a whole da a s uc u e, o a subse o he da a s uc u e
ha does no o e lap wi h o he subpa s. Thus, da a s uc u es should be
assigned as a whole o all he de ices, o pa i ioned in non-o e lapping pa s,
assigning one o each de ice.
To suppo his model, we ha e ex ended he Hi Tile objec s in he Hi map
lib a y wi h he capabili y o pa i sel in se e al sub-selec ions, and s o e he
in o ma ion abou he pa i ion inside he objec . In e nally, when a Hi Tile
is a ached o a Mul i-Con olle , i pe o ms he ollowing s eps:
1. Fi s , i checks ha he Hi Tile is no al eady a ached o any Mul i-
Con olle . I ha pa icula Hi Tile objec is al eady a ached, he p o-
g am aises an e o , as a second a achmen could lead o ace condi ions
due o he concu en execu ion o ke nels in di e en Mul i-Con olle s.
2. I i is an a achmen wi hou he pa i ion op ion, he whole space o
indexes o he da a s uc u e a e mapped o each de ice. I i is an a ach-
men wi h he pa i ion op ion ac i a ed, he Mul i-Con olle pa s he
index space o he Hi Tile da a s uc u e in a numbe o pa s equal o he
numbe o de ices de ined in he Mul i-Con olle . The pa i ion policies
in oduced in Hi map a e esponsible o di iding he da a s uc u e wi h
no-o e lapped domains. The pa i ion size co esponding o each de ice is
p opo ional o he weigh s p o ided in an a ay o loa ing poin numbe s,
one o each de ice. Mo e sophis ica ed pa i ion policies can be easily
added in he u u e hanks o he modula plug-ins sys em in he Hi map
lib a y. The in o ma ion abou he mapping is s o ed in he Hi Tile objec
o u he e e ence.
3. Finally, he Mul i-Con olle c ea es a Hi Tile s uc u e o he space o
sub-space o indexes mapped o each de ice, and p oceeds o do he da a
ans e s o he assigned de ice when needed. T ans e s a e no needed
o g oups o CPU-co es o he hos , o accele a o s ha can sha e he
hos memo y space. The ans e policies inside he Mul i-Con olle can
ake decisions abou when and how make he ans e s. Fo example, he
cu en Mul i-Con olle p o o ype implemen bo h, immedia e and lazy
ans e s. The implemen a ion o asynch onous ans e s is cu en ly an
on-going wo k.
When he da a s uc u e is de ached, he Mul i-Con olle objec ensu es
he consis ency o he whole da a s uc u e in he main hos h ead. This may
imply da a ans e s om some o all o he associa ed de ices. The in o ma ion
s o ed in he objec s abou he index space mapped o each de ice is used o
he ans e s, and elimina ed a he end o he de achmen p ocedu e. The
seman ic o his ope a ion makes i synch onous. The main hos h ead should
block un il i s s a e is consolida ed.
16 Ana Mo e on-Fe nandez e al.
Table 1 De elopmen e o measu es o he ou benchma ks when hey a e p og ammed
using Cuda, OpenMP, and he p oposed Mul i-Con olle lib a y.
Benchma k Code Lines Cycloma ic Hals ead
o Code Complexi y Measu e
Ma ix Cuda 72 7 202361
Addi ion OpenMP 49 11 99783
MC l 61 5 103528
Ma ix Cuda 142 5 409862
Mul . OpenMP 45 9 81136
MC l 97 6 201242
Black- Cuda 211 7 742735
Scholes OpenMP 134 8 389556
MC l 163 6 486956
Mandel- Cuda 49 3 159481
b o OpenMP 36 5 66610
MC l 55 3 124212
5.1.4 Mandelb o algo i hm
The escape ime algo i hm is he simples algo i hm o gene a ing a ep esen-
a ion o ac al geome ic images o he Mandelb o se . This example p esen s
an i egula wo kload pe h ead. Ne e heless, he compu a ion space can be
di ided be ween wo GPUs gi ing he same amoun o wo k o each GPU,
and hus p oducing good esul s. This example will clea ly show he balancing
e ec o pa i ioning he wo kload among se e al GPUs. The applica ion does
no need any da a ans e o s a he compu a ion, only o e u n he esul s.
The pa ame e s chosen o he expe imen s a e: An image size o 2048 ×2048,
and a limi o 60,000 i e a ions pe pixel. A isual ep esen a ion o he chosen
a ea is shown in Fig. 6. The GPU h eadblock size used is 32 ×32 o all he
expe imen s wi h his benchma k. In ou implemen a ion, he same gene ic
ke nel de ini ion is used o bo h GPUs and CPUs.
5.2 De elopmen e o
In his sec ion we compa e, in e ms o de elopmen e o , he use o ou
lib a y wi h he mos common na i e p og amming models o NVIDIAS’s
GPUs and mul i-co e CPUs, which a e CUDA and OpenMP espec i ely. Fo
his compa ison, we use h ee classical de elopmen e o me ics: COCOMO
lines o code, McCabe’s cycloma ic complexi y [10], and Hals ead de elopmen
e o [5]. McCabe’s cycloma ic complexi y measu e is a quan i a i e measu e
o he numbe o linea ly independen pa hs h ough a p og am’s sou ce code,
and he Hals ead de elopmen e o me ic is also a quan i a i e numbe based
on he numbe on ope a o s and ope ands in he sou ce code. Low cycloma ic
complexi y and Hals ead de elopmen e o esul s imply codes simple o de-
elop and debug. These me ics a e ypically used in he assessmen o so wa e
design complexi y. The me ics a e applied o he pa s o code ha include:
ke nel de ini ions, ke nel cha ac e iza ions, he coo dina ion code in he main
hos h ead wi h he Mul i-con olle managemen , and da a s uc u es man-

Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 17
agemen . We igno e code de o ed o e o o esul s checking, pe o mance
ins umen a ion, and w i ing messages o he s anda d ou pu .
Table 1 shows he di e en measu es o he di e en codes e alua ed. The
esul s show ha ou lib a y implies less de elopmen e o o he p og am-
me han using CUDA o all he s udy cases. On he o he hand, al hough
he OpenMP p og amming model needs a less olume o lines o code, he
cycloma ic complexi y o ou p oposed is less because ou abs ac ion hides
some un- ime decisions and checkings.
T ans o ming a CUDA p og am in o an OpenMP e sion, o in he opposi e
way, is no a i ial ask. Remind he Fig. 5, whe e we show wo codes, using he
Mul i-Con olle abs ac ion, ha pe o m a ma ix addi ion using di e en
a ge de ice combina ions. When we compa e bo h codes, we obse e ha he
e o equi ed by he p og amme o change he p og am in o de o exploi
1 GPU + 10 CPU-co es, o wo GPU de ices, only in ol es 4 lines o code.
We can see hese ou lines highligh ed on bo h codes in he code.
This simpli ica ion makes anspa en o he p og amme all he di e ences
on da a ans e s, ke nel launchings, and da a managemen s o di e en kind
o compu a ional de ices such as accele a o s o g oups o CPU-co es.
5.3 Pe o mance esul s
In his sec ion we p esen pe o mance esul s: (1) compa ing ou p oposal
wi h pu e CUDA e e ence p og ams, and (2) e alua ing he impac o using
in ou model di e en compu a ional uni s o he i e case s udies selec ed.
The goal o his s udy is o de e mine he po en ial pe o mance penal y in-
oduced by using ou app oach, as well as he pe o mance gain ob ained
when exploi ing a combina ion o he e ogeneous de ices wi h di e en compu-
a ional capabili ies.
The expe imen s ha e been execu ed on a hos machine named Hyd a, wi h
wo CPUs In el Xeon E5-2609 3 @1.90GHz, 64Gb DDR3 main memo y, and
wo GPUs: an NVIDIA’s GeFo ce Ti an Z (named GPU-0) and a Ti an Black
X (named GPU-1). We exploi he wo GPU de ices, and mul iple CPU-co es
o ganized in a single i ual de ice in ou model. Fo his es , we ha e decided
o a oid pe o mance e ec s de i ed om o e susc ip ion o hype h eading.
As ou Mul i-Con olle lib a y uses one hos h ead o each de ice o be
con olled, he numbe o CPU-co es we use o compu e and execu e ke nels
is 10.
The p og ams ha e been compiled using he CUDA Toolki 8.0, and GCC
4.8.3. We ha e used he lags, -O3, and - openmp o exploi pa allelism when
using a g oup o CPU-co es as a compu a ional uni . We ha e execu ed all he
expe imen s en imes, egis e ing he lowes o al execu ion imes. We ha e
also measu ed sepa a ely he imes spen in copying da a o h o and back
om he a ge de ices, and he compu a ion ime o he ke nels. We include
he ime spen by ou queue sys em inside he compu a ion imes.
We ha e es ed h ee kinds o codes:
18 Ana Mo e on-Fe nandez e al.
0
0.2
0.4
0.6
0.8
1
1.2
80 85 90 92.5 95 97.5 100 Cuda
Execu ion ime (sec.)
CPU % da a
Ma ix Addi ion CPU+GPU, SIZE=20000x20000
Compu a ion
Da a ans e s
0
0.1
0.2
0.3
0.4
0.5
0.6
80 85 90 92.5 95 97.5 100 Cuda
Execu ion ime (sec.)
CPU % da a
BlackScholes CPU+GPU, SIZE= 3*10^8
Compu a ion
Da a ans e s
0
200
400
600
800
1000
1200
1400
80 85 90 92.5 95 97.5 100 Cuda
Execu ion ime (sec.)
CPU % da a
Ma ix Mul iplica ion CPU+GPU, SIZE=12800x12800
Compu a ion
Da a ans e s
0
1
2
3
4
5
6
7
8
9
80 85 90 92.5 95 97.5 100 Cuda
Execu ion ime (sec.)
CPU % da a
BlackScholes_2048 CPU+GPU, SIZE= 1*10^6
Compu a ion
Da a ans e s
0
0.5
1
1.5
2
2.5
80 85 90 92.5 95 97.5 100 Cuda
Execu ion ime (sec.)
CPU % da a
Mandelb o CPU+GPU, SIZE=2048 x 2048 ITERATIONS=60000
Compu a ion
Da a ans e s
Fig. 7 Pe o mance esul s (in seconds) o expe imen s on Hyd a using a g oup o 10 CPU-
co es and a GPU. The igh -mos columns show he esul o he e e ence CUDA p og ams
un on he same GPU.
–A na i e CUDA implemen a ion o he di e en benchma ks es ed. We
p esen measu es ob ained in one o bo h GPUs in ou a ge sys em de-
pending on he s udy. See he igh -mos and/o he le -mos columns in
he cha s discussed bellow. Cuda Re : Measu es in NVIDIA’s GeFo ce
Ti an Black Z. Cuda Re -2: Measu es in NVIDIA’s GeFo ce Ti an Black
X.
– CPU+GPU: This code, p og ammed using he MC l lib a y, execu es
he p og ammed applica ion on wo de ices, a g oup o 10 CPU-co es, and
an NVIDIA’s GeFo ce Ti an Black Z. Di e en mappings ha e been es ed,
de e mined by he pe cen age o da a and compu a ion assigned o each
de ice.
– GPU+GPU: This code, p og ammed using he MC l lib a y, execu es
he applica ion on he wo GPUs a ailable in he a ge sys em. Again,
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 19
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Cuda_Re 100 70 65 60 55 50 0 Cuda_Re -2
Execu ion ime (sec.)
GPU-0 % da a
Ma ix Addi ion GPU+GPU, SIZE=20000x20000
Compu a ion
Da a ans e s
0
0.5
1
1.5
2
2.5
3
3.5
Cuda_Re 100 70 65 60 55 50 0 Cuda_Re -2
Execu ion ime (sec.)
GPU-0 % da a
BlackScholes GPU+GPU, SIZE= 3*10^8
Compu a ion
Da a ans e s
0
1
2
3
4
5
6
7
8
9
10
Cuda_Re 100 70 65 60 55 50 0 Cuda_Re -2
Execu ion ime (sec.)
GPU-0 % da a
Ma ix Mul iplica ion GPU+GPU, SIZE=12800x12800
Compu a ion
Da a ans e s
0
0.05
0.1
0.15
0.2
0.25
Cuda_Re 100 70 65 60 55 50 0 Cuda_Re -2
Execu ion ime (sec.)
GPU-0 % da a
BlackScholes_2048 GPU+GPU, SIZE= 1*10^6
Compu a ion
Da a ans e s
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Cuda_Re 100 80 75 70 65 60 0 Cuda_Re -1
Execu ion ime (sec.)
GPU-0 % da a
Mandelb o GPU+GPU, SIZE=2048 x 2048 ITERATIONS=60000
Compu a ion
Da a ans e s
Fig. 8 Pe o mance esul s (in seconds) o expe imen s on Hyd a using wo di e en GPUs.
The le - and igh -mos columns show he esul s o he e e ence CUDA p og ams un on
each o he wo GPUs conside ed in he s udy.
di e en mappings ha e been es ed, de e mined by he pe cen age o da a
and compu a ion assigned o each de ice.
Figu e 7 shows he pe o mance esul s ob ained by ou p oposal when
he da a, and hus he compu a ion, is di ided among he g oup o CPU-
co es and a GPU, he NVIDIA’s GeFo ce Ti an Black Z. In he applica ions
whe e da a ans e s domina e he o al ime (Ma ix Addi ion and Black-
Scholes benchma ks) we can achie e a be e pe o mance by gi ing pa o
he compu a ion o he g oup o CPU-co es. Despi e he compu a ional powe
o he GPU accele a o , he compu a ion di ision imp o es pe o mance by
educing he ime spen in da a ans e s o/ om he GPU.
When he compu a ional load is high, such as in he ma ix mul iplica ion,
and Black-Scholes 2048, he bes pe o mance is ob ained doing he whole
compu a ion in he GPU. This is ypical o his kind o p og ams ha e-
ally sui he GPU compu a ional model. Fo he Mandelb o benchma k, we
20 Ana Mo e on-Fe nandez e al.
obse e ha he pa i ion o he compu a ion ob ains ma ginally be e pe -
o mance esul s han he CUDA e e ence codes. This pa icula beha iou is
because o he i egula wo kload o his benchma k.
Figu e 8 shows he pe o mance esul s ob ained when we di ide he com-
pu a ion be ween he wo GPUs. When he compu a ion load is eally low and
he e a e mul iple ke nel launchings, he ime spen in he queue managemen s
( emind ha hese imes a e aken in o accoun in he compu a ion ime) can
be no iceable, such in he BlackScholes 2048 case (i akes app oxima ely 0.04
seconds).
In ou i s p o o ype o he lib a y, da a ans e s o se e al GPUs a e
s ill sequen ialized. Thus, in hose applica ions whe e da a ans e s lead he
execu ion ime, he bes pe o mance is ob ained when he applica ion is exe-
cu ed only in he mos powe ul GPU. Howe e , when he compu a ion ime
is much highe han he communica ion imes, a di ision o he compu a ion
among he GPUs, p opo ional o hei ela i e compu a ion powe o his
p oblem, imp o es he pe o mance. Fo example in ou s udy, wi hou aking
in o accoun he da a ans e s, he ke nel execu ion imes a e educed he
34%, 25%, and 41% compa ed o he bes CUDA e e ence codes which use a
single de ice, o he ma ix mul iplica ion, Black-Scholes 2048 and Mandel-
b o benchma ks espec i ely. This beha iou is shown in he expe imen a ion
esul s.
6 Conclusion
In his pape we p esen he Mul i-Con olle (MC l), an abs ac en i y im-
plemen ed in a lib a y, ha coo dina es he managemen o he e ogeneous de-
ices, including accele a o s wi h di e en capabili ies and se s o CPU-co es.
This en i y o e s a global iew o he compu a ion, anspa en ly managing he
coo dina ion, da a pa i ion, mapping, and execu ion o whole compu a ions
on hei associa ed de ices. Ou solu ion allows he use o simple gene ic ke -
nels (po able ac oss di e en de ice ypes), o specialized implemen a ions de-
ined and op imized using speci ic na i e o endo p og amming models (such
as CUDA o NVIDIA’s GPUs, o OpenMP o CPU-co es). The un- ime sys-
em au oma ically selec s and deploys he mos app op ia e implemen a ion
o each ke nel o each de ice, managing he da a mo emen s, and hiding he
launching de ails. Resul s o an expe imen al s udy wi h i e s udy cases indi-
ca e ha ou abs ac ion allows he de elopmen o lexible and high e icien
p og ams, ha adap o he he e ogeneous en i onmen . On-going and u u e
wo k includes he s udy o he e ec o mo e sophis ica ed echniques o da a
mo emen which o e lap communica ion and compu a ion, and a new expe -
imen al s udy including i egula benchma ks whose da a should be di ided
using mo e complex mechanisms.
Acknowledgemen s This esea ch has been pa ially suppo ed by MICINN (Spain) and
ERDF p og am o he Eu opean Union: HomP og-He Sys p ojec (TIN2014-58876-P), CAPAP-
Mul i-De ice Lib a y: Simpli ying Pa allel He e ogeneous P og amming 21
H6 (TIN2016-81840-REDT), and COST P og am Ac ion IC1305: Ne wo k o Sus ainable
Ul ascale Compu ing (NESUS).
Re e ences
1. Alonso-Mayo, A., O ega-A anz, H., Gonzalez-Esc ibano, A.: Communica o s: An ab-
s ac ion o ease he use o accele a o s. In: HLPGPU’2016 (2016)
2. Das gee , U., Enmy en, J., Kessle , C.W.: Au o- uning SkePU: A Mul i-backend Skele-
on P og amming F amewo k o mul i-GPU Sys ems. In: P oc. IWMSE’11, pp. 25–32.
ACM, New Yo k, NY, USA (2011)
3. Gonzalez-Esc ibano, A., To es, Y., F esno, J., Llanos, D.R.: An ex ensible sys em o
mul ile el au oma ic da a pa i ion and mapping. IEEE T ansac ions on Pa allel and
Dis ibu ed Sys ems 25(5), 1145–1154 (2014)
4. Haidl, M., Go la ch, S.: PACXX: Towa ds a Uni ied P og amming Model o P og am-
ming Accele a o s using C++14. In: P oc. LLVM-HPC’14. IEEE (2014)
5. Hals ead, M.H.: Elemen s o So wa e Science (Ope a ing and p og amming sys ems
se ies). Else ie Science Inc. (1977)
6. Hijma, P., Jacobs, C.J., an Nieuwpoo , R.V., Bal, H.E.: Cashme e: He e ogeneous
many-co e compu ing. In: Pa allel and Dis ibu ed P ocessing Symposium (IPDPS),
2015 IEEE In e na ional, pp. 135–145. IEEE (2015)
7. Hugo, A.E., Gue mouche, A., Wac enie , P.A., Namys , R.: Composing Mul iple
S a PU Applica ions o e He e ogeneous Machines: A Supe ised App oach. In: P oc.
IPDPSW’13 PhD Fo um, pp. 1050–1059. IEEE, Washing on, D.C., USA (2013)
8. Ka imi, K., Dickson, N.G., Hamze, F.: A pe o mance compa ison o cuda and opencl.
a Xi p ep in a Xi :1005.2581 (2010)
9. Liang, T., Li, H., Chiu, J.: Enabling Mixed OpenMP/MPI P og amming on Hyb id
CPU/GPU Compu ing A chi ec u e. In: P oc. IPDPSW’12, PhD Fo um, pp. 2369–
2377. IEEE, Washing on, D.C., USA (2012). DOI 10.1109/IPDPSW.2012.294
10. McCabe, T.J.: A complexi y measu e. So wa e Enginee ing, IEEE T ansac ions on 4,
308–320 (1976)
11. Mo e on-Fe nandez, A., Rod iguez-Gu iez, E., Gonzalez-Esc ibano, A., Llanos, D.R.:
Suppo ing he xeon phi cop ocesso in a he e ogeneous p og amming model. In: Eu-
opean Con e ence on Pa allel P ocessing, pp. 457–469. Sp inge , Cham (2017)
12. Mo e onFe nandez, A., O egaA anz, H., GonzalezEsc ibano, A.: Con olle s: An ab-
s ac ion o ease he use o ha dwa e accele a o s. The In e na ional Jou nal o High
Pe o mance Compu ing Applica ions (2017). DOI 10.1177/1094342017702962. URL
h p://dx.doi.o g/10.1177/1094342017702962
13. NVIDIA: NVIDIA CUDA C P og amming Guide 7.5 (2015). URL {h p://docs.
n idia.com/cuda/pd /CUDA _C _P og amming _Guide.pd }. Las isi : No embe 16 h,
2015
14. N idia, C.: Cublas lib a y. NVIDIA Co po a ion, San a Cla a, Cali o nia 15, 27 (2008)
15. O ega-A anz, H., To es, Y., Gonzalez-Esc ibano, A., Llanos, D.R.: Op imizing an
APSP implemen a ion o NVIDIA GPUs using ke nel cha ac e iza ion c i e ia. The
Jou nal o Supe compu ing 70(2), 786–798 (2014). DOI 10.1007/s11227-014-1212-z
16. P´e ez, B., Bosque, J.L., Bei ide, R.: Simpli ying p og amming and load balancing o
da a pa allel applica ions on he e ogeneous sys ems. In: P oceedings o he 9 h Annual
Wo kshop on Gene al Pu pose P ocessing using G aphics P ocessing Uni , pp. 42–51.
ACM (2016)
17. Scogland, T.R., Roun ee, B., Feng, W.c., de Supinski, B.R.: He e ogeneous ask
scheduling o accele a ed openmp. In: Pa allel & Dis ibu ed P ocessing Symposium
(IPDPS), 2012 IEEE 26 h In e na ional, pp. 144–155. IEEE (2012)
18. Shen, J., Va banescu, A.L., Lu, Y., Zou, P., Sips, H.: Wo kload pa i ioning o accel-
e a ing applica ions on he e ogeneous pla o ms. IEEE T ansac ions on Pa allel and
Dis ibu ed Sys ems 27(9), 2766–2780 (2016)
19. S one, J.E., Goha a, D., Shi, G.: Opencl: A pa allel p og amming s anda d o he -
e ogeneous compu ing sys ems. Compu ing in science & enginee ing 12(1-3), 66–73
(2010)

22 Ana Mo e on-Fe nandez e al.
20. TOP500.o g: Top500 supe compu ing si es. WWW (2017). On h p://www. op500.
o g/
21. To es, Y., Gonzalez-Esc ibano, A., Llanos, D.R.: uBench: exposing he impac o CUDA
block geome y in e ms o pe o mance. The Jou nal o Supe compu ing 65(3), 1150–
1163 (2013). DOI 10.1007/s11227-013-0921-z