Quan i ying Se e less Elas ici y: The gume e
Benchma k Sui e
Ge mán T. Eizagui e1, En ique Molina-Giménez1, Ge a d Finol1, Ca los
Molina1, and Ped o Ga cía-López1
Uni e si a Ro i a i Vi gili, Ta agona, Spain
Abs ac . Se e less compu ing has eme ged as a powe ul pa adigm
o dis ibu ed wo k lows, o e ing ine-g ained, low-la ency esou ce p o-
isioning o p ecisely mee job demands. While wo k lows o en u ilize
hund eds o concu en CPUs, exis ing se e less benchma king sui es
equen ly concen a e on small-scale pa allelism and mic ose ice wo k-
loads. Fu he mo e, hese benchma ks ypically conside only Func ion-
as-a-Se ice (FaaS ) backends, o e looking hei na u al cloud successo s,
Con aine -as-a-Se ice (CaaS ). These limi a ions ha e c ea ed a signi -
ican gap in e alua ing a c ucial ea u e o se e less pla o ms: hei
abili y o accu a ely handle sudden changes in esou ce alloca ion, also
known as elas ici y. To add ess his, we in oduce gume e , a new bench-
ma king sui e speci ically designed o e alua e he elas ici y o se e less
pla o ms (bo h FaaS and CaaS ) o highly pa allel dis ibu ed wo k-
lows. gume e acili a es a ho ough assessmen o an unde lying pla -
o m using a " i e-and- o ge " execu ion model and minimal use in e -
en ion. I le e ages a se o comp ehensi e pipelines ha sample a ious
ypical scaling beha io s, p o iding an in-dep h analysis o elas ici y, ex-
ecu ion ime, cos , and e iciency. We apply gume e o e alua e he elas-
ici y o h ee popula se e less pla o ms: AWS Lambda, Google Cloud
Run, and IBM Code Engine. Ou esul s e eal signi ican di e ences in
elas ici y ac oss hese pla o ms, showing ha FaaS o e ings s ill ou -
pe o m CaaS in elas ici y by a ac o o up o 6.5x. Howe e , despi e a
lowe pe o mance, CaaS can be up o 64.5% less expensi e han FaaS
in speci ic scena ios, he eby un eiling an in e es ing op imiza ion space
o cloud wo k lows.
Keywo ds: Se e less compu ing ·Func ion-as-a-Se ice (FaaS) ·Con aine -
as-a-Se ice (CaaS) ·Benchma king
1 In oduc ion
Se e less compu ing has seen apid adop ion in cloud compu ing, eme ging as
a solu ion o he ope a ional complexi ies o adi ional cloud models, like cloud
i ual machines. By o e ing managed esou ce p o isioning and se up abs ac-
ions, se e less pla o ms elimina e he need o de elope s o manage in as-
uc u e o un ime en i onmen s. Ins ead, hey can ocus solely on applica ion
logic while he cloud p o ide handles se e alloca ion, scaling, and execu ion.
2 Eizagui e e al.
While Func ion-as-a-Se ice (FaaS) pla o ms such as AWS Lambda [3] we e
he i s o popula ize se e less compu ing, he pa adigm now encompasses a
b oad ecosys em. This includes scalable objec s o age (e.g., AWS S3 [2]), da a
analy ics se ices (e.g., GCP BigQue y [8]), and o he managed solu ions. De-
spi e hei di e si y, all se e less pla o ms adhe e o key de ining ea u es [18]:
Managed esou ce alloca ion. Se e less abs ac s in as uc u e managemen ,
handling no jus indi idual esou ces, bu dynamically o ches a ing dis ibu ed
esou ce pools behind he scenes. This is especially use ul o wo kloads wi h high
and a iable pa allelism, whe e he numbe o equi ed esou ces scales in and
ou equen ly.
Pay-as-you-go billing. Use s a e cha ged only o he ac ual execu ion ime o
hei unc ions, wi h no cos incu ed du ing idle pe iods. This model elimi-
na es was e and op imizes cos s o applica ions wi h luc ua ing o unp edic able
wo kloads.
Decoupled compu e and s o age. Se e less sepa a es compu e and s o age, allow-
ing each o scale independen ly. This imp o es e iciency and cos -e ec i eness
by ailo ing esou ce alloca ion o an applica ion’s speci ic needs.
Se e less Compu ing o Dis ibu ed Wo k lows
The p ope ies o se e less makes i a pa icula ly well-sui ed pa adigm o
wo kloads wi h dynamic pa allelism demands. Such is he case o dis ibu ed
wo k lows, whe e wo kloads a e o en in e mi en and bu s y and esou ce e-
qui emen s change dynamically. This ansla es in o highly pa allelism needs
bo h wi hin and be ween jobs.
Wo k lows specially bene i om se ices p o iding compu a ional esou ces
in a se e less manne . We call such se ices Se e less Compu ing Pla o ms,
as hey enable ins an esou ce deploymen p ecisely when needed and s ic ly
o he equi ed du a ion. This allows ma ching he equi ed pa allelism almos
ins an ly, in a p oac i e way (as opposi e o eac i e scaling o adicional clus e
se ices, whe e esou ces a e (de)p o isioned in esponse o eal ime wo kload
me ics). No ably, FaaS pla o ms can p o ision housands o CPU ins ances
wi hin seconds, making hem excep ionally appealing o emba assingly pa allel
wo kloads [17, 25].
Howe e , FaaS a e no he only cloud se ices ha p o ide compu a ional
esou ces in a se e less manne . To his day, al e na i e se ice ypes, such as
Con aine -as-a-Se ice (CaaS) (like IBM Code Engine [16]), p o ide highly sim-
ila APIs o hose o FaaS, bu wi h mo e con ol o e he p o isioned esou ces
and essen ial, unde lying a chi ec u al shi s. These se ices, howe e , a e o en
o e looked in discussions o se e less compu ing.
Wha i s in A Se e less Compu ing Pla o m?
To delinea e he scope o his wo k, we i s de ine wha can be conside ed a
se e less compu ing pla o m. This de ini ion emains ambiguous in p ac ice, as
Quan i ying Se e less Elas ici y: The gume e Benchma k Sui e 3
mode n in e p e a ions o se e less ex end beyond o iginal se e less se ices
o include a ious managed componen s. In ac , cloud p o ide s ha e b oad-
ened he se e less designa ion o encompass managed se ices ha main ain
co e se e less a ibu es. No able examples include managed clus e se ices
like AWS EMR Se e less [1] and GCP Da ap oc Se e less [9], which abs ac
in as uc u e managemen while o e ing u nkey da a analy ics and AI capa-
bili ies.
♦We conside a se e less compu ing pla o m ha wi h (1) apid com-
pu a ional esou ce p o isioning and (2) e ec i ely unbounded ho izon al
scalabili y.
An analysis o comme cial and esea ch se e less wo k low p ocessing pla -
o ms [17, 23, 6, 29] e eals consis en a chi ec u al pa e ns aligned wi h se e -
less p inciples: (1) a uni o m, scalable se e less compu ing pla o m o da a
p ocessing, (2) cloud s o age o da a exchange and aul ole ance, and (3) aux-
ilia y o ches a ion se ices. Figu e 1 illus a es his canonical a chi ec u e.
P og amme
Se e less
amewo k
Se e less compu ing
pla o m
1 2 ...
0
<wo k low>
Cloud esou ces
∞
Se e less s o age backend
Auxilia y se ices
(e.g. un ime eposi o ies,
logging se ices)
Fig. 1: Seg ega ing Componen s in Se e less Wo k low P ocessing. A
se e less da a analy ics pla o m p o ides a p og amming amewo k while ab-
s ac ing he unde lying execu ion low. Behind he scenes, i ea u es decoupled
s o age and compu e backends, along wi h auxilia y se ices (e.g., o logging o
message passing). Se e less compu e backends o e unlimi ed ho izon al scal-
ing, adap i e esou ce alloca ion, and a pay-as-you-go billing model. S o age
backends may include in-memo y s o age, objec s o age, o managed da abase
se ices.
De ining a se e less compu ing pla o m is one hing, bu eaching a con-
sensus on he "bes " pla o m o dis ibu ed wo k low p ocessing emains an
elusi e goal. In his wo k, we’ e ocusing on p ecisely ha challenge.
4 Eizagui e e al.
Do We Need Ye Ano he Se e less Benchma k Sui e?
P ope ly e alua ing se e less compu ing pla o ms equi es cap u ing speci ic
pe o mance de ails while accoun ing hei speci ic limi a ions. The li e a u e on
se e less benchma king is b oad [5, 22, 24, 30, 19, 12, 27], including unc ion
wo k lows wi h modes esou ce equi emen s [28, 13, 20]. Howe e , o he bes
o ou knowledge he e is no speci ic benchma king sui e speci ically designed
o highly pa allel se e less wo k lows wi h c oss-pla o m compa ibili y.
Se e less wo k lows exhibi dis inc i e p ope ies ha complica e hei e al-
ua ion. They can expe ience pa allelism luc ua ions spanning wo o h ee o -
de s o magni ude wi hin job [31, 26, 21], c ea ing dynamic esou ce equi emen s
ha exis ing benchma ks canno adequa ely assess. Howe e , cu en solu ions
emain cons ained by hei ocus on ei he modes pa allelism scena ios ( ypi-
cally limi ed o ens o unc ions [28, 20]) o isola ed unc ion e alua ion.
To b idge his gap, we in oduce gume e , a comp ehensi e benchma king
sui e speci ically designed o assess se e less compu ing pla o ms o dis-
ibu ed wo k lows. gume e enables holis ic e alua ion h ough ep esen-
a i e wo kloads, inco po a es elas ici y-awa e pe o mance me ics and in-
eg a es cloud-speci ic cos modeling while main aining c oss-pla o m com-
pa ibili y. Implemen ed as an open-sou ce solu ion, gume e suppo s bo h
immedia e benchma king needs and u u e ex ensions wi h bo h u he
pla o ms and use -de ined wo k lows.
The key con ibu ions o his pape a e:
–A comp ehensi e benchma king sui e o se e less da a analy ics o e alua e
pe o mance, cos , and elas ici y o mul iple a chi ec u es, enabling di ec
compa ison o se e less compu ing pla o ms.
–A me hodology o assess he c i ical iad o elas ici y, pe o mance, and cos
in se e less wo k low execu ion, add essing a key gap in cu en e alua ion
app oaches.
–A de ailed compa ison o ele an compu ing pla o ms, including AWS Lambda,
GCP Cloud Run and IBM Code Engine, disclosing impo an and p ac ical
ade-o s.
2 Design
2.1 Measu ing Elas ici y in Wo k low Execu ion
Measu ing cloud elas ici y in da a wo k lows emains a challenge, as no consen-
sus exis s on a s anda d me ic despi e i s c i ical impo ance. While esou ce
elas ici y is no exclusi e o he cloud, i is inhe en ly associa ed wi h i : since
cloud se ices p o ide on-demand esou ces, i is essen ial o assess how as
hese esou ces adap o an applica ion’s needs. Howe e , a uni e sally accep ed
elas ici y me ic emains elusi e—a challenge ecognized o o e a decade [14].
Quan i ying Se e less Elas ici y: The gume e Benchma k Sui e 5
Elas ici y is de ined by h ee key ea u es [4]: (1) esou ce (de)p o isioning la-
ency, (2) he maximum numbe o esou ces ha can be acqui ed concu en ly,
and (3) billing g anula i y. Howe e , ou de ini ion o a se e less backend as-
sumes i ually unlimi ed esou ces and a pay-pe -use billing model, inhe en ly
sa is ying (2) and (3). Thus, we ocus on measu ing how quickly a backend adap s
compu ing esou ces o ma ch wo kload demands.
Isola ing elas ici y as a unc ion o esou ce (de)p o isioning speed simpli ies
he p oblem and enables he use o a mo e in ui i e me ic. Building on p e ious
p oposals [7], we aim o de ine an indica o analogous o he physics concep
o elas ici y. In ma e ials science, Young’s modulus—he ea e e e ed o as
s i ness—quan i ies he maximum s e ch a ma e ial unde goes unde a gi en
ension. Deno ing ϵas he applied ension, and Las he ma e ial’s leng h, we
de ine s i ness Sas:
S=σ
△L/L (1)
To adap his concep o cloud sys ems, we ep esen he applied ension
as he equi ed esou ces, C , and he leng h as he esou ces p o ided by he
sys em, Cp. Thus, he sys em’s s i ness a a gi en poin in he applica ion’s
execu ion is de ined as:
S=C
max(Cp,1) (2)
012345
Time
0
500
1000
Resou ces
Requi ed P o ided S i ness
Fig. 2: Measu ing s i ness in se e less wo k low execu ion. Assuming
esou ces can scale ou inde ini ely, we quan i y he a io be ween esou ces e-
qui ed by he wo k low and he ac ual esou ces p o ided by he sys em.
A s a ic measu e o s i ness alone is insu icien o ully assess a da a analy ics
pipeline. As discussed p e iously, esou ce demands a y ac oss pipeline s ages,
each wi h di e en equi emen s. The e o e, in eal-wo ld scena ios, e alua ing
6 Eizagui e e al.
he s i ness o a se e less backend equi es measu ing i s esou ce p o isioning
accu acy h oughou he en i e execu ion. We calcula e i as he a ea be ween
he equi ed and p o isioned esou ce cu es, illus a ed in Figu e 2. Le he
absolu e esou ce misma ch o e he in e al [0, T]be:
S=ZT
0
(C ( )−Cp( )) d (3)
This ep esen s he cumula i e amoun o unde -p o isioned esou ces. To
no malize his a ea, we de ine he maximum possible misma ch as:
Smax =ZT
0
C ( )d (4)
which co esponds o he case whe e no esou ces a e p o isioned a all, i.e.,
Cp( )=0 o all .
Then, we de ine he no malized elas ici y as:
Ea ea = 1 −S
Smax
(5)
This o mula ion yields alues in he ange [0,1], whe e Ea ea = 1 indica es
a pe ec ma ch be ween p o isioned and equi ed esou ces, and Ea ea = 0 in-
dica es comple e ine iciency (no esou ces we e p o isioned a any ime). E,
deno ing elas ici y coe icien , ep esen s a use ul measu e o he elas ici y o
di e en pla o ms hos ing he same wo k low. Howe e , since Eis highly de-
penden on he speci ic esou ce demands o a gi en wo k low, i is no sui able
o c oss-wo k low compa isons.
2.2 A Rep esen a i e Ye P ac ical Wo k low Se
Table 1 p esen s a de ailed lis o all wo k lows included in gume e , ou lining
hei unc ionali y, execu ion plan, and obse able insigh s. These wo k lows a e
designed o encompass a a ie y o ypical scaling beha io s, om s eep scale-
ou s (e.g., FLOPS, Mon e Ca lo S ock P edic ion) and d as ic scale-ins (e.g.,
Mon e Ca lo Pi Es ima ion, Mon e Ca lo S ock P edic ion) o scena ios wi h
changing pa allelism wi hin he wo k low i sel (e.g., Mandelb o , Te aSo ).
Fo p ac ical applica ion, we’ e aligned he benchma k pa allelisms wi h he
s anda d se ice quo as o AWS, GCP, and IBM Cloud, assuming one CPU pe
se e less ins ance.
Quan i ying Se e less Elas ici y: The gume e Benchma k Sui e 7
Table 1: O e iew o gume e wo k lows.
Wo k low Desc ip ion S ages Key Insigh s
FLOPS Each unc ion indepen-
den ly execu es loa ing-
poin ope a ions o e wo
N-dimensional ma ices.
Single s age wi h
200 unc ions.
(1) Measu es he
compu a ional powe
o he pla o m. (2)
Re eals s eep scale-
ou capabili ies.
Mon e
Ca lo S ock
P edic ion
MapReduce implemen a ion
o a Mon e Ca lo algo i hm
o p edic s ock a ia ion
ends.
Map s age: 10
unc ions; Re-
duce s age: 1
unc ion.
Re eals mode a e
scale-in capabili ies.
Mon e
Ca lo Pi
Es ima ion
MapReduce implemen a ion
o a Mon e Ca lo algo i hm
o app oxima e he alue o
π.
Map s age: 100
unc ions; Re-
duce s age: 1
unc ion.
Re eals s eep scale-
in capabili ies.
Te aSo Dis ibu ed so o e a 5GB
Te aSo da ase .
Fi s s age: 50
unc ions; Second
s age: 100 unc-
ions.
(1) E alua es scal-
abili y o long-
las ing unc ions. (2)
Re eals mode a e
scale-in capabili ies.
Mandelb o Runs se e al s ages, each
calcula ing he Mandelb o
se on a egion o he linea
space.
Se en consecu-
i e s ages wi h
[4, 16, 36, 64,
100, 144, 196]
unc ions.
Re eals i e a i e,
p og essi e scale-ou
capabili ies.
3 Implemen a ion
We use li hops [25], an open sou ce backend lis ed in he hi d-pa y in eg a-
ions o Code Engine1as he cen al building block o gume e .li hops is
a se e less amewo k ha allows use s o un Py hon unc ions on se e less
pla o ms. I p o ides a uni ied in e ace o execu ing unc ions ac oss di e -
en cloud p o ide s, making i sui able o benchma king pu poses. li hops
suppo s a ious backends, including AWS Lambda and IBM Code Engine, al-
lowing us o e alua e he elas ici y o hese pla o ms using he same codebase.
Fo benchma king pu poses, we ex end li hops wi h an addi ional GCP Cloud
Run backend and p o e i s ex ensibili y o o he se e less pla o ms.
1h ps://cloud.ibm.com/docs/codeengine? opic=codeengine-suppo ed-in eg a ions
8 Eizagui e e al.
4 E alua ion
4.1 Benchma ked Pla o ms and Se up
To demons a e gume e ’s u ili y, we compa ed he pe o mance o h ee ep-
esen a i e se e less pla o ms om di e en cloud p o ide s. The se ups em-
ployed a e de ailed in Table 2. We selec ed a pu e FaaS pla o m (AWS Lambda),
a pu e CaaS pla o m (IBM Code Engine), and a hyb id al e na i e (Google
Cloud Run). We ex ac ed execu ion ime, cos , and elas ici y me ics om each
o p o ide a compa a i e analysis.
Table 2: O e iew o e alua ed se e less pla o m se ups.
Pla o m Cloud P o ide Type S o age
Lambda [3] AWS FaaS AWS S3 [2]
Cloud Run [11] GCP CaaS /FaaS GCP Cloud S o age [10]
Code Engine [16] IBM Cloud CaaS IBM COS [15]
All expe imen s an in he us-eas 1 egion o i s equi alen . We p e-wa med
ins ances o all expe imen s wi h h ee eplicas o 200 unc ions o ensu e a
su icien ly wa m execu ion en i onmen . Each unc ion was p o isioned wi h 1
CPU. In AWS Lambda, CPU alloca ion is di ec ly linked o memo y, wi h 1
CPU co esponding o 1769MB o RAM. Fo Code Engine and Cloud Run,
which o e mo e lexible CPU-memo y con igu a ions, we alloca ed 2048MB o
memo y pe unc ion o closely ma ch Lambda’s con igu a ion.
4.2 Elas ici y
We e alua e he elas ici y o each pla o m when handling di e en wo kloads.
Figu e 3 illus a es he p o isioned CPUs (equi alen o he numbe o unc ions)
o e ime, o e ing a isual ep esen a ion o each wo kload’s execu ion imeline.
FaaS backends demons a e as e and mo e s able scaling compa ed o
CaaS. A low pa allelism (Figu e 3a), he p ima y di e ence appea s o be in
scaling la ency. Howe e , as pa allelism inc eases (Figu e 3c), Code Engine ex-
hibi s inconsis en scale-ou pa e ns a he han a mono onic inc ease in p o-
isioned CPUs. This beha io is pa icula ly p oblema ic wi h equen changes
in pa allelism, as shown in Figu e 3d.
Code Engine’s scaling issues may s em om i s a chi ec u e. Unlike FaaS
pla o ms, Code Engine ins ances a e p o isioned wi hin a Kube ne es clus e
and equi e a membe ship p o ocol o p o isioning. This could in e e e wi h
he scaling la ency o Code Engine esou ces.
Lambda and Cloud Run show simila pe o mance, wi h Lambda demon-
s a ing sligh ly lowe scale-ou la ency. In e es ingly, ask execu ion ime in
Quan i ying Se e less Elas ici y: The gume e Benchma k Sui e 9
Te aSo appea s signi ican ly slowe in Cloud Run. This could be a ibu ed
o he I/O pe o mance o GCP Cloud S o age, gi en ha a dis ibu ed so
in ol es a da a-in ensi e all- o-all exchange be ween i s s ages.
0 5 10 15 20 25 30
Execu ion Time (s)
0
10
# CPUs
AWS Lambda
GCP Cloud Run
IBM Code Engine
(a) Mon e Ca lo S ock P edic ion.
0 5 10 15 20 25
Execu ion Time (s)
0
100
# CPUs
(b) Mon eca lo Pi Es ima ion.
0 20 40 60 80 100
Execu ion Time (s)
0
100
# CPUs
(c) Te aSo .
0 20 40 60 80 100
Execu ion Time (s)
0
50
# CPUs
(d) Mandelb o .
Fig. 3: CPU p o isioning imelines o each wo k low, on he e alua ed pla o ms.
Figu e 3 se es as a alida ion o ou p oposed elas ici y me ic, which we
depic in Figu e 4. As commen ed be o e, he elas ici y o CaaS does no ma ch
ha o FaaS, specially as pa allelism inc eases and becomes mo e a iable.
