Techniques o P o ec
Con iden iali y and In eg i y o
Pe sis en and In-Memo y Da a
A disse a ion submi ed owa ds he deg ee
Doc o o Enginee ing
o he Facul y o Ma hema ics and Compu e Science
o Saa land Uni e si y
by
Anjo Lucas Vahldiek-Obe wagne
Saa b ücken
Oc obe , 2018
Da e o Colloquium: Feb ua y 5 h, 2019
Dean o Facul y: P o . D . Sebas ian Hack
Chai o he Commi ee: P o . D . Ge Smolka
Repo e s
Fi s Re iewe : P o . Pe e D uschel, Ph.D.
Second Re iewe : Deepak Ga g, Ph.D.
Thi d Re iewe : S e an Sa oiu, Ph.D.
Academic Assis an : Engel Le aucheux, Ph.D.
II
III
Abs ac
Today compu e s s o e and analyze aluable and sensi i e da a. As a esul we need
o p o ec his da a agains con iden iali y and in eg i y iola ions ha can esul
in he illici elease, loss, o modi ica ion o a use ’s and an o ganiza ion’s sensi i e
da a such as pe sonal media con en o clien eco ds. Exis ing echniques p o ec ing
con iden iali y and in eg i y lack ei he e iciency o a e ulne able o malicious
a acks. In his hesis we sugges echniques, Gua da and ERIM, o e icien ly and
obus ly p o ec pe sis en and in-memo y da a.
To p o ec he con iden iali y and in eg i y o pe sis en da a, clien s speci y
pe - ile
policies o Gua da decla a i ely, concisely and sepa a ely om code. Gua da
en o ces policies by media ing I/O in he s o age laye . In con as o p io echniques,
we p o ec agains acciden al o malicious ci cum en ion o highe so wa e laye s.
We p esen he design and p o o ype implemen a ion, and demons a e ha Gua da
e icien ly en o ces example policies in a web se e .
To p o ec he con iden iali y and in eg i y o in-memo y da a, ERIM isola es
sensi i e da a using In el Memo y P o ec ion Keys (MPK), a ecen x86 ex ension
o pa i ion he add ess space. Howe e , MPK does no p o ec agains malicious
a acks by i sel . We p e en malicious a acks by combining MPK wi h call ga es
o us ed en y poin s and ahead-o - ime bina y inspec ion. In con as o exis ing
echniques, ERIM e icien ly p o ec s equen ly-used session keys o web se e s,
an in-memo y e e ence moni o ’s p i a e s a e, and managed un imes om na i e
lib a ies. These use cases esul in high swi ch a es o he o de o 10
5
–10
6
swi ches/s.
Ou expe imen s demons a e less hen 1% un ime o e head pe 100,000 swi ches/s,
hus ou pe o ming exis ing echniques.
IV
Ku zda s ellung
Compu e speiche n und analysie en we olle und sensi i e Da en. Das ha zu Folge,
dass wi diese Da en gegen Ve aulichkei s- und In eg i ä s e le zungen schü zen
müssen. Ande n alls d oh die une laub e F eigabe, de Ve lus ode die Modi ika ion
de Da en. Exis ie ende Me hoden schü zen die Ve aulichkei und In eg i ä
unzu eichend, da sie ine izien und an ällig ü mu willige Ang i e sind. In diese
Dok o a bei s ellen wi zwei Me hoden, Gua da und ERIM, o , die pe sis en e
Da en und Da en im A bei sspeiche e izien und wide s ands ähig beschü zen.
Um die Ve aulichkei und In eg i ä pe sis en e Da en zu schü zen, e knüp en
Nu ze ü jede Da ei Rich linien in Gua da . Gua da übe p ü diese Rich linien ü
jeden Zug i und se z diese im Speiche medium du ch. Im Gegensa z zu exis ie en-
den Me hoden, beschü z Gua da o mu willigem Umgehen. Wi besch eiben die
Me hode, eine Implemen ie ung und e aluie en die E izienz on Beispiel ich linien.
Um die Ve aulichkei und In eg i ä on Da en im A bei sspeiche zu schü zen,
isolie ERIM sensi i e Da en mi Hil e on In el Memo y P o ec ion Keys (MPK),
eine neue x86 E wei e ung, um den A bei sspeiche au zu eilen. Da MPK alle dings
nich gegen mu willige Ang i e schü z , e hinde ERIM diese, indem es MPK mi
wide s ands ähigen Wechseln de Speiche be eiche und eine Binä codeübe p ü ung
kombinie . Im Gegensa z zu exis ie enden Me hoden, beschü z ERIM e izien
häu ig genu z e Si zungsschlüssel, Zus ands a iablen eines Re e enzmoni o s und
e wal e e Lau zei umgebungen on na i en Biblio heken. Unse e Expe imen e
zeigen, dass wenige als 1% Lau zei meh au wand je 100.000 Wechselope a ionen p o
Sekunde no wendig sind.
V

Publica ions
Pa s o his hesis ha e appea ed in he ollowing publica ions.
•
“Gua da : En o cing da a policies a he s o age laye ”. Anjo Vahldiek-
Obe wagne , Eslam Elnike y, Aas ha Meh a, Deepak Ga g, Pe e D uschel,
Ansley Pos , Rod igo Rod igues, Johannes Geh ke. In P oceedings o he
Eu opean Con e ence on Compu e Sys ems (Eu oSys), 2015.
•
“ERIM: Secu e and E icien In-p ocess Isola ion”, Anjo Vahldiek-Obe wagne ,
Eslam Elnike y, Nuno O. Dua e, Deepak Ga g, Pe e D uschel. Unde e iew
and echnical epo (a Xi :1801.06822), 2018.
Addi ional publica ions no included in his hesis.
•
“P o ec ing Da a In eg i y wi h S o age Leases”, Anjo Vahldiek, Eslam Elnike y,
Ansley Pos , Pe e D uschel, Rod igo Rod igues. MPI-SWS Technical Repo
2011-008.
•
“Tho h: Comp ehensi e Policy Compliance in Da a Re ie al Sys ems”, Eslam
Elnike y, Aas ha Meh a, Anjo Vahldiek-Obe wagne , Deepak Ga g, Pe e
D uschel. In P oceedings o he USENIX Secu i y Symposium, 2016.
•
“Ligh -Weigh Con ex s: An OS Abs ac ion o Sa e y and Pe o mance”,
James Li on, Anjo Vahldiek-Obe wagne , Eslam Elnike y, Deepak Ga g, Bobby
Bha acha jee, Pe e D uschel. In P oceedings o he USENIX Symposium on
Ope a ing Sys ems Design and Implemen a ion (OSDI), 2016.
VI
•
“Pesos: Policy Enhanced Secu e Objec S o e”, Robe K ahn, Bohdan T ach,
Anjo Vahldiek-Obe wagne , Thomas Knau h, P amod Bha o ia, Ch is o Fe ze .
In P oceedings o he Eu opean Con e ence on Compu e Sys ems (Eu oSys),
2018.
VII
Fynn, Julius, Timon and Ke s in.
VIII
Acknowledgmen s
I would like o ex end my hanks o many people. They so gene ously con ibu ed
o he wo k p esen ed in his hesis and helped me du ing he ups and downs o
g adua e li e.
Fi s ly, I would like o exp ess my since e g a i ude o my ad ise s Pe e D uschel
and Deepak Ga g o hei con inuous suppo . Thei guidance helped me in esea ch
and w i ing o his hesis. I’m e y g a e ul o hem o gi ing me he ime o aise
my amily and be wi h my child en.
I would like o hank he es o my hesis commi ee: Paul F ancis, and S e an
Sa oiu, o hei insigh ul commen s and encou agemen , bu also o he ha d
ques ions which helped me o u he imp o e my esea ch om a ious pe spec i es.
I ha e been e y o una e o collabo a e wi h an as ic ellow PhD s uden s,
Eslam Elnike y and Aas ha Me ha. Wi h hem, he coun less and i ing deadlines
u ned in o ad en u e ips. I canno imagine going h ough g adua e li e wi hou
hei iendship and suppo .
Besides Eslam and Aas ha, I had he g ea pleasu e o collabo a e wi h Bobby
Bha acha jee, Nuno Dua e, Johannes Geh ke, James Li on, Rod igo Rod igues,
and Ansley Pos .
Fu he mo e, I would like o hank ellow s uden s and pos docs a MPI-SWS
o c ea ing a g ea place o wo k, in pa icula A pan, Bimal, Bilal, Cheng, Ekin,
Ezgi, Felipe, Filip, Geo g, Jan-Oli e , James, Juhi, Manoha , Na acha, Nancy, Nuno,
Oana, Paa ijaa , Ped o, P amod, Reinha d, Sco , and Vik o .
IX
Lis o Figu es
2.1 Re e ence moni o implemen a ion scena ios . . . . . . . . . . . . . . 8
3.1 Gua da implemen a ion in a SAN se e . . . . . . . . . . . . . . . . 40
3.2 Absolu e Gua da la ency o e head . . . . . . . . . . . . . . . . . . . 47
3.3 La ency wi h an SSD, ela i e o iSCSI . . . . . . . . . . . . . . . . . 48
3.4 Absolu e la ency wi h SSD . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5 Absolu e la ency wi h HDD . . . . . . . . . . . . . . . . . . . . . . . 49
3.6 SSDI/O h oughpu ........................... 50
3.7 FS benchma ks ead and w i e ( /w) pe o mance . . . . . . . . . . . 53
3.8 Web se e h oughpu . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.9 La ency wi h MAL, olun a y and no logging . . . . . . . . . . . . . 57
4.1
SPEC CPU o e head o CPI/CPS wi h ERIM and an emu-
la ion o WRPKRU (EMUL-CPI/CPS), ela i e o no p o ec ion . . . 90
4.2
SPEC CPU o e head o CPI/CPS and ERIM-CPI/CPS,
ela i e o no p o ec ion. . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3 Nginx h oughpu wi h one wo ke wi h and wi hou ERIM . . . . . 98
4.4 Nginx h oughpu wi h one wo ke wi h emula ed ERIM and lwCs . . 102
5.1 S eps owa ds an isola ed c yp og aphic lib a y in se e applica ions 116
XVI

CHAPTER 1
In oduc ion
Today compu e s assis people in mos daily ac i i ies such as social in e ac-
ions, lea ning, and in o ma ion sha ing. People en us compu e sys ems wi h
hei aluable da a, e.g., pe sonal media con en , inancial and heal h eco ds, and
c yp og aphic keys. Compu e sys ems ough o p o ec he con iden iali y and
in eg i y o such da a. Con iden iali y gua an ees ha only au ho ized eads o he
da a succeed. In eg i y p e en s unau ho ized upda es o he da a.
Viola ing he con iden iali y and in eg i y o sensi i e da a can esul in i s
leak, loss o modi ica ion. As a esul clien s and o ganiza ions may ace he loss o
highly sen imen al da a and epu a ional o inancial loss. Common causes o da a
con iden iali y and in eg i y iola ions [
15
] include so wa e bugs, secu i y ulne -
abili ies, miscon igu a ion and ope a o e o . Fi s , a bug, e.g., in an applica ion
may o e w i e exis ing iles, iola ing in eg i y. Second, secu i y ulne abili ies in
online se ices may be used by malicious a acke s o ex ac sensi i e da a such as
c yp og aphic keys, iola ing con iden iali y. Thi d, miscon igu a ions may lead o
acciden al da a eads, iola ing con iden iali y. Fou h, an adminis a i e ope a o
o a sys em may acciden ally dele e da a om he sys em, iola ing in eg i y.
To p e en hese iola ions, many echniques o p o ec da a con iden iali y
and in eg i y ha e been p oposed. Model checking, language-based s a ic analysis,
es ing and e e ence moni o s a e b oad classes o such echniques. Model checking
ensu es ha an applica ion ollows a gi en speci ica ion, and hus p o ides he
1
s onges gua an ees compa ed o he emaining gua an ees. Howe e , model checking
o e e yday so wa e (e.g., ope a ing sys ems o web b owse s) is complex and
consequen ly di icul and expensi e, which limi s he use o model checking o
speci ic componen s in high- isk applica ions. Language-based s a ic analysis en o ces
speci ic p og am in a ian s o e sou ce code and ma ks in a ian iola ions such
as bugs and ulne abili ies du ing de elopmen . Due o he app oxima ion o
un ime alues, analysis ools su e in p ac ice om high alse posi i es a es. In
addi ion, limi ed suppo o mul i-language so wa e sys ems hinde s hei b oad
adop ion [
104
]. Tes ing, on he o he hand, p o ides a easonable and bes -e o
co e age o iola ions o e a subse o applica ion inpu s. Al hough widely used,
es ing-based app oaches do no p o ide o mal gua an ees, since es ing e e y
possible inpu is usually un easible, especially when conside ing malicious a acks.
Thus, none o hese echniques p o ides he abili y o en o ce con iden iali y and
in eg i y sys ema ically ac oss applica ions and independen o he applica ion
implemen a ion.
In con as , e e ence moni o s [
6
] en o ce con iden iali y and in eg i y o da a by
obse ing applica ions a un ime, media ing ele an e en s (such as I/O o memo y
accesses) and denying accesses which iola e con iden iali y o in eg i y. T ea ing
he applica ion as a black box allows e e ence moni o s o en o ce con iden iali y
and in eg i y independen o he applica ion implemen a ion and applica ion size,
and sys ema ically ac oss applica ions. This allows adop ion ac oss a wide ange o
use cases including legacy applica ions wi h no access o sou ce code. Compa ed
o model checking o s a ic analysis, which p o e co ec ness o an applica ion,
e e ence moni o s educe he p oo o co ec ness o a smalle and simple piece o
code, namely he e e ence moni o . In con as o he o he echniques, e e ence
moni o s induce un ime o e head on he p oduc ion sys ems o which hey a e
applied. Reducing his o e head is an impo an design conside a ion.
2
Re e ence moni o s ha e been applied o a ious use cases. When p o ec ing da a
con iden iali y and in eg i y, we di e en ia e be ween p o ec ing in- ansi ,pe sis en ,
o in-memo y da a.In- ansi da a is ypically p o ec ed using c yp og aphic me hods
which enc yp he con en s p o iding con iden iali y, and sign he con en s p o iding
in eg i y. Fo secu e communica ion oday’s compu e sys ems ely on he SSL/TLS
s anda d wi h eadily a ailable implemen a ions in se e al c yp og aphic lib a ies
such as OpenSSL [
96
]. Howe e , o bo h, pe sis en and in-memo y da a, exis ing
echniques do no e icien ly and comp ehensi ely p o ec da a con iden iali y and
in eg i y [134, 45, 110, 124, 48, 3, 74, 59, 29, 68, 75, 113, 58, 38, 29].
To p o ec he con iden iali y and in eg i y o pe sis en da a, exis ing moni o ing
echniques [
48
,
134
,
45
,
118
,
139
,
19
,
74
,
5
,
124
,
110
] media e applica ion I/O by in
a lib a y, ile sys em, ope a ing sys em, hype iso , o in he s o age laye . Howe e ,
media ion in a laye o he han he s o age laye can be easily bypassed, and none
o he s o age laye echniques suppo gene al con iden iali y and in eg i y policies.
Hence, a s ong and gene al policy en o cemen echnique o pe sis en da a is
cu en ly missing.
To p o ec he con iden iali y and in eg i y o in-memo y da a om accesses by
an un us ed applica ion, p io wo k elies on memo y isola ion h ough language and
un ime [
38
,
72
,
129
,
143
,
68
], p ocess-based [
74
,
59
,
29
,
20
], o andomiza ion-based
echniques [
113
,
58
]. Fi s , language and un ime echniques isola e by inse ing
checks in o he applica ion bina y o p o ec agains a bi a y da a accesses. Al hough
obus agains malicious a acks, hese echniques su e om un ime o e heads
o pe o m he checks. Second, p ocess-based isola ion spli s he execu ion o an
applica ion in o sepa a e ha dwa e-p o ec ed p ocesses. Simila o language and
un ime isola ion, p ocess-based isola ion is obus agains malicious a acks. The
e iciency depends on he cos o con ex swi ches be ween applica ion p ocesses,
which is usually high. Thi d, andomiza ion-based isola ion uses he huge add ess
space o hide sensi i e da a a a andom loca ion. While andomiza ion-based
3
echniques a e e icien (no checks o high swi ch cos s), malicious a acke s can
ind he sec e loca ion and b eak gua an ees [
113
,
60
,
39
,
49
,
94
]. No exis ing
echnique e icien ly isola es memo y wi h low un ime o e head, and o e s s ong
p o ec ion agains malicious a acke s. As a esul , no exis ing isola ion echnique is
su icien o se e al impo an use cases such as p o ec ing c yp og aphic keys o
na i e lib a ies in managed un imes.
Con ibu ions:
This disse a ion con ibu es Gua da , which e icien ly en o ces ex-
p essi e con iden iali y and in eg i y policies a he s o age laye p o ec ing pe sis en
da a, and ERIM, which s ongly and e icien ly isola es in-memo y da a.
Gua da :
In con as o p e ious echniques [
48
,
134
,
45
,
118
,
139
,
19
,
74
] ha
in e cep a he applica ion o a sys em so wa e laye , Gua da p o ec s he con-
iden iali y and in eg i y o pe sis en da a a he s o age laye . Thus, Gua da
minimizes he size and a ack su ace o he us ed compu ing base (TCB) elied
upon o en o cemen .
P o ec ing a he s o age laye limi s a ailable access in o ma ion o block
add esses. As a esul , exis ing s o age laye echniques echniques [
5
,
124
,
110
] do
no en o ce gene ic con iden iali y and in eg i y policies. To o e come he lack o
clien in o ma ion such as ile names and access c eden ials, clien s communica e wi h
Gua da h ough secu e channels, unneling h ough un us ed sys em laye s like he
ope a ing sys em. Clien s use his communica ion o send addi ional in o ma ion
such as ile names o access c eden ials. Using his in o ma ion, Gua da en o ces
con iden iali y and in eg i y o e e y da a access while elying only on i s own
en o cemen logic.
Wi h Gua da , con iden iali y and in eg i y equi emen s a e speci ied as pe - ile
policies by use s, de elope s, o adminis a o s. Policies speci y he condi ions unde
which a ile may be ead, upda ed, o ha e i s policy changed. These condi ions,
w i en in a decla a i e language, may depend on clien au hen ica ion, he ini ial
4
and inal s a es o he ile (size and con en ) in an upda e ansac ion, o signed
s a emen s by ex e nal us ed componen s (ce i ying, o ins ance, he cu en
wall-clock ime). Gua da s o es he policy as pa o i s own me ada a and ensu es
ha each access o he ile complies wi h he policy.
Fo example, use s can ely on Gua da o mi iga e se ious h ea s: To p e en
he inse ion o malicious code in execu ables, a policy can p o ec execu able iles
by allowing only upda es signed by a us ed pa y; o p e en sys em logs om
co up ion and ampe ing, a policy can p o ec log iles by making hem append-only;
o p e en he illici elease o a use ’s p i a e da a, a policy can p o ec he use ’s
da a by equi ing an au hen ica ed secu e session o ead da a; o p e en a bi a y
ile accesses and allow audi ing o accesses, a policy can p o ec iles by equi ing a
manda o y log en y be o e accessing a ile.
We e alua e he e iciency o Gua da using he policy examples desc ibed
abo e. We show ha Gua da en o ces con iden iali y and in eg i y policies wi h low
o e head. When p o ec ing a web se e ’s con en om accesses by unau ho ized
use s, and bina ies om unau ho ized upda es, he h oughpu o e head is less han
1% compa ed o no p o ec ion.
ERIM:
ERIM is a amewo k o s ong, e icien isola ion o in-memo y da a. I
allows pa i ioning an applica ion in o a us ed and an un us ed componen wi hin
a single add ess space. Fo his, ERIM elies on Memo y P o ec ion Keys (MPK) [
64
],
a ecen x86 ex ension o pa i ion he add ess space in o up o 16 disjoin memo y
domains. Wi h ERIM he us ed and he un us ed componen ’s da a eside in
di e en domains and ERIM con ols access o each domain. A new use -mode CPU
ins uc ion (WRPKRU) swi ches access pe missions o domains e icien ly (abou 60
cycles pe swi ch), wi hou ke nel in e en ion. Al hough e icien , his ins uc ions
allows malicious a acke s o escala e hei access pe missions. Hence, by i sel MPK
5

is no su icien o gua an ee secu i y agains malicious o comp omised un us ed
componen s.
ERIM’s con ibu ion is o build secu e memo y isola ion using MPK by (1)
p o iding call ga es o secu ely ans e con ol o he us ed componen a p ede ined
en y poin s wi hou ke nel in e en ion, and (2) use bina y inspec ion o emo e
exploi able bina y code ensu ing ha he swi ch ins uc ion canno be exploi ed.
As a esul , o gain access o sec e da a, an un us ed componen has o in oke
a call ga e ans e ing con ol o he us ed componen . In con as o p io
echniques, ERIM’s memo y isola ion signi ican ly educes he swi ch cos be ween
he us ed and un us ed componen , does no slow down un us ed componen like
language-based echniques, and p o ec s agains malicious a acke s.
We apply ERIM’s design o challenging and p e iously high-o e head use
cases [
74
,
29
,
72
]. Fi s , we isola e equen ly used OpenSSL session keys o a
web se e (nginx) and show scalabili y. Second, we isola e he sa e egion in an
implemen a ion o code-poin e in eg i y (CPI) [
72
]. Thi d, we isola e a managed
un ime (node.js) om an un us ed na i e lib a y (SQLi e). Ou esul s show ha
ERIM p o ides obus memo y isola ion wi h a low o e head o less han 1% o
100,000 swi ches pe second.
O e iew:
In he emainde o his hesis we u he desc ibe he backg ound and
ela ed wo k (Chap e 2), ollowed by de ailed desc ip ion o he design, implemen a-
ion, and e alua ion o Gua da (Chap e 3) and ERIM (Chap e 4). Finally, we
conclude and desc ibe u u e wo k (Chap e 5).
6
CHAPTER 2
Backg ound
In his chap e we p o ide an o e iew o he backg ound wo k on e e ence
moni o s and b ie ly desc ibe exis ing echniques o p o ec con iden iali y and
in eg i y o pe sis en and in-memo y da a. This chap e is only mean o se e
as a backg ound ma e ial o unde s anding he hesis. A de ailed compa ison o
exis ing wo k is p o ided in Sec ions 3.8 and 4.6.
Re e ence moni o ing en o ces secu i y policies a un ime wi hou insis ing ha
he applica ion be bug- ee. Re e ence moni o s in e cep all ele an ope a ions,
e alua e each ope a ion agains he equi ed policy and deny ope a ions when
iola ions a e imminen . We summa ize s a e-o - he-a echniques o e e ence
moni o ing.
Figu e 2.1 depic s possible implemen a ion scena ios o e e ence moni o s.
Re e ence moni o s ha e been implemen ed a di e en abs ac ion laye s wi hin
he so wa e and ha dwa e s ack (see Figu e 2.1a). Each abs ac ion laye gua ds
access o he esou ces p o ided o highe laye s. Re e ence moni o s in highe laye s
(e.g., applica ion, da abase o ile sys em) ely on p o ec ion gua an ees p o ided by
lowe laye s, inc easing he TCB and isk o ci cum en ion o he e e ence moni o .
While moni o ing a an abs ac ion laye , e e ence moni o s can be implemen ed
by isola ing so wa e componen s in us ed execu ion en i onmen s (TEE) [
82
] (see
Figu e 2.1b) o a sepa a e applica ion p ocess [
22
] (see Figu e 2.1c), by sandboxing
7
Ope a ing Sys em
Applica ion
VMM
CPU
Remo e
Hos
Da abase RM
RM
RM
RM
RM
(a) Re e ence moni o a each le el o abs ac ion
CPU
Applica ion
TEE
T us ed
Componen
RM
(b)
Re e ence moni o in us ed execu-
ion en i onmen (TEE)
Ope a ing Sys em
Applica ion RM
(c)
Re e ence moni o in sepa a e appli-
ca ion
Ope a ing Sys em
RM
Applica ion
(d)
Sandboxing applica ion inside a e -
e ence moni o
Ope a ing Sys em
Applica ion
RM
(e) Inlined e e ence moni o
Figu e 2.1: Re e ence moni o implemen a ion scena ios
an applica ion [
143
] (see Figu e 2.1d), o by inlining moni o s in o he applica ion
i sel [1, 72, 29] (see Figu e 2.1e).
En o cemen echniques in non-applica ion laye s e icien ly media e all accesses
o ele an esou ces (e.g., memo y o iles). Usually lowe abs ac ion laye s, such
as he ope a ing sys em (OS) o i ual machine moni o (VMM), in e cep e en s
wi h coa se-g ain in o ma ion om he applica ion. Each laye abs ac s in o ma ion
wi h help om he applica ion. Fo ins ance, implemen ing a pe ile con iden iali y
policy is only possible wi hin he ile sys em laye o abo e. A hese laye s he
accessed ile and i s associa ed policy is s ill a ailable.
Moni o ing a he applica ion laye o e s he mos de ailed in o ma ion abou he
applica ion s a e and execu ion a he cos o a la ge TCB and isk o ci cum en ion.
Inlining he media ion and en o cemen in o he applica ion [
37
,
72
] o e s he abili y
o p o ec he in eg i y o he con ol low o an applica ion a he cos o addi ional
checks o e e y indi ec jump and e u n. En o cing such applica ion le el gua an ees
8
a a lowe laye (e.g., he OS) is in easible, since e e y check would incu high swi ch
cos s be ween he laye and he applica ion.
While nume ous e e ence moni o ing echniques ha e been sugges ed, his
disse a ion ocuses on p o ec ing he con iden iali y and in eg i y o pe sis en and
in-memo y da a. We desc ibe nex he s a e o he a in p o ec ing pe sis en and
in-memo y da a.
2.1 P o ec ing pe sis en da a
In he ollowing we desc ibe echniques o p o ec pe sis en da a om illici elease,
co up ion o dele ion due o bugs, miscon igu a ions, ope a o e o o malicious
a acks. We do no conside ha dwa e ailu es, since eplica ion (such as RAID [
98
])
o da a enc yp ion mi iga e hese h ea s easily.
In gene al, he da a con iden iali y and in eg i y gua an ees in oday’s compu e
sys ems depend on, and a e sp ead ac oss he applica ion, a da abase managemen
sys em, he OS (including he ile sys em) and i ual machine moni o s. Fo example,
each laye en o ces i s own use access con ol p o ec ing agains illici accesses and in
some cases also keeps da a hashes o p o ec he in eg i y. Compa ed o applica ion
laye p o ec ion, lowe laye s p o ide a s onge p o ec ion agains ci cum en ion, bu
ypically do no p o ide a gene ic policy en o cemen and ins ead ocus on speci ic
uses and policies.
Hype iso /OS da a p o ec ion
Nexus [
118
] and TAOS [
139
] a e wo OS-le el
echniques ha en o ce au ho iza ion policies on OS in e aces (e.g., iles, in e -
p ocess communica ion, memo y mappings o p ocess managemen ) p o ec ing da a
con iden iali y. Nexus op ionally main ains a Me kle hash ee o he ile sys em
o p o ide da a in eg i y. In con as o Nexus and TAOS which en o ce policies,
Dune [
19
] and lwC [
74
] a e amewo ks o build a e e ence moni o a he OS
abs ac ion laye media ing he sys em call in e ace and bo h show use cases o
9

CHAPTER 3
Gua da : En o cing da a policies a
he s o age laye
This chap e desc ibes Gua da , a sys em o en o ce con iden iali y and in eg i y
policies on pe sis en da a. B ie ly, he p oblem is ha compu e and s o age sys ems
inc ease in complexi y and so does he isk o da a con iden iali y and in eg i y
om so wa e bugs, secu i y ulne abili ies and human e o . In addi ion, da a
is inc easingly s o ed on hi d-pa y pla o ms, in oducing addi ional isks like
unau ho ized da a use by he hi d pa y. Da a s o ed in hi d-pa y pla o ms ely
on he us and eliabili y o he hi d-pa y p o ide . Today’s sys ems en o ce he
applicable secu i y policy o a ile implici ly in hei code. Fu he mo e, he policy
speci ica ion and en o cemen may sp ead o e di e en subsys ems, inc easing he
isk o ci cum en ion and miscon igu a ion.
Gua da in oduces a e e ence moni o a he s o age laye o ackle hese
challenges. I p o ides a single-poin o policy speci ica ion, con igu a ion and
en o cemen a he s o age laye elying only on i s own policy in e p e e , en o cemen
logic and explici policy dependencies, hus minimizing he TCB and a ack su ace.
The ollowing sec ions desc ibe Gua da ’s design and API, i s decla a i e policy
language, example use cases, an implemen a ion, ela ed wo k and an expe imen al
e alua ion o a p o o ype implemen a ion.
17
3.1 Design
Gua da ’s design was guided by ou p inciples:
1.
Gua da policies a e a ached o iles, sepa a e om code, and speci ied in a
cus om decla a i e policy language. The e o e, he policy o a ile’s da a can
be speci ied concisely in one place and audi ed easily.
2.
Gua da en o ces policies in he s o age laye o minimize he isk o policy
ci cum en ion. Ou implemen a ion o Gua da in a SAN se e , o ins ance,
allows a scalable con igu a ion whe e policies a e en o ced by block se e s
in a machine oom, while clien compu e s and he en e p ise ne wo k a e
un us ed.
3.
Gua da policies s a e me ely wha accesses a e allowed unde which condi ions,
lea ing i o un us ed code how o demons a e compliance wi h a policy. This
sepa a ion keeps he policy language small and policies concise, while shi ing
complexi y o un us ed so wa e and o e head o clien compu e s.
4.
Gua da elies on c yp og aphic ile a es a ions o b idge he seman ic gap
be ween pe - ile policies and block-le el en o cemen . By eques ing an a es a-
ion o a ile’s policy, name and con en hash, an applica ion can e i y ha
Gua da associa es da a and policy co ec ly, independen o he ilesys em o
i s me ada a.
En o cemen a he s o age laye is p e e able, since i minimizes he isk o ci -
cum en ion, and makes i easy o physically p o ec he us ed Gua da componen s
in a machine oom. A design en o cing a a highe laye (e.g., NAS ile se e , VMM
o clien OS laye ) would ex end us o addi ional, and likely mo e dis ibu ed,
componen s. Mo eo e , Gua da is able o b idge he seman ic gap be ween iles and
blocks as wi hou elying on he un us ed ilesys em and i s me ada a.
18
Da a s o ed in Gua da is o ganized in o iles. Fo each policy-p o ec ed ile,
Gua da main ains i s own shadow me ada a, consis ing o an o de ed lis o ex en s,
a unique nume ic iden i ie , a ex ual name s ing ( ypically used o s o e he ile’s
pa hname(s)—mul iple in he case o ha d links), and a e e ence o a policy in
e ec o he ile. The se o nume ic iden i ie s o m a la namespace, while he
se o names ypically encode a con en ional namespace hie a chy main ained by an
un us ed ilesys em. Each ile can ha e i s own policy bu , ypically, a collec ion o
iles sha e he same policy.
The policy o a ile consis s o ou ules, one o each o he pe missions
ead
,
upda e
,
des oy
and
se policy
. Each ule speci ies condi ions on he con ex and
en i onmen unde which he espec i e pe mission holds. Abs ac ly, he
ead
ule
ep esen s he ile’s con iden iali y policy; he
upda e
ule encodes he ile’s in eg i y
policy; he
des oy
ule go e ns when he ile’s iden i ie (name) can be ecycled;
and he
se policy
ule desc ibes when he policy can be changed. S o age commands
ha ead o upda e a ile o i s me ada a check condi ions o he co esponding
policy ules.
Gua da in eg a es wi h ilesys ems. The (un us ed) ilesys em as usual assigns
names and s o age blocks o a ile and ansla es ile eques s in o block eques s
using i s me ada a. Gua da uses i s own shadow me ada a o look up he ile and
policy associa ed wi h a block eques secu ely and e icien ly. Gua da also assigns
i s own unique ile iden i ie s, which can be eused only unde policy con ol.
File a es a ions ie he GDC’s iew o a ile as a sequence o ex en s o an
applica ion’s iew o a named ile, he eby emo ing he need o us he ilesys em
and i s me ada a. By eques ing an a es a ion a e a ile is w i en o ead, an
applica ion can e i y ha i s iew o he ile is iden ical o he GDC’s. Gua da has
suppo o spa se iles. The cu en design assumes ha a block is assigned o a
mos one ile; block sha ing o suppo de-duplica ion, o ins ance, could be added
easily.
19
Gua da ’s p og am logic, called he Gua da con olle o GDC, is in eg a ed
wi h a s o age block de ice and en o ces policies on e e y ead and w i e. The
GDC ex ends he s anda d block-de ice in e ace wi h a ile-le el in e ace, which
allows highe so wa e laye s o (a) c ea e, dele e, ead and upda e se s o ex en s
( iles) using simple ansac ions, (b) associa e policies wi h iles, (c) c yp og aphically
au hen ica e and es ablish secu e sessions, (d) p o ide c eden ials and o he e idence
o policy compliance, and (e) ob ain a es a ions on s o ed iles and hei policies. The
ile-le el in e ace can be used by a Gua da -awa e ilesys em, o by an applica ion
lib a y in combina ion wi h a legacy ilesys em ia IOCTL calls.
3.2 Th ea model
The GDC, me ada a and da a mus be physically p o ec ed om unau ho ized access
and unde ec ed ampe ing. In ou implemen a ion (see Sec ion 3.6.1), da a and
me ada a s o age de ices a e assumed o be physically p o ec ed, e.g., in a machine
oom wi h es ic ed access. Gua da policies a e en o ced, subjec o ex e nal policy
dependencies, ega dless o bugs, miscon igu a ions, o secu i y inciden s ou side he
s o age de ice, including inciden s on any numbe o clien machines.
We make s anda d assump ions abou policies: Co ec policies mus be ins alled
when da a is i s s o ed, and ex e nal dependencies o policies like ime se e s,
clien au hen ica ion keys, and admin au hen ica ion keys mus be us wo hy (in
pa icula , admin au hen ica ion keys can o en be s o ed o line and p o ec ed phys-
ically). Unde hese assump ions, Gua da de ends agains h ea s o con iden iali y
and in eg i y o s o ed da a. In addi ion, Gua da can p o ec he in eg i y and
con iden iali y o iles ans e ed be ween Gua da de ices, and be ween a Gua da
de ice and a clien de ice h ough a secu e channel. This includes h ea s due o
bugs and ulne abili ies in in e media e so wa e laye s including ope a ing sys ems,
ilesys ems, s o age se ices buil on op o Gua da , and ne wo ks, and h ea s due
20
o human negligence and oppo unis ic malice. Gua da is no conce ned wi h da a
a ailabili y. To mask he e ec s o a ha dwa e o media ailu e, loss, o des uc ion
o a Gua da de ice, da a mus be eplica ed on mul iple Gua da de ices wi h
independen ailu e modes.
3.3 In e ace
Gua da ex ends he s anda d block de ice in e ace wi h means o es ablish sessions,
c ea e, upda e and dele e iles, ins all policies, p o ide e idence o policy compliance,
and ob ain a es a ions. In he ollowing, we desc ibe he unc ionali y p o ided by
he in e ace. Table 3.1 shows all Gua da API calls.
3.3.1 Session in e ace
A use applica ion (also called a clien ) in e ac s wi h Gua da in a session. A
secu e, au hen ica ed session mus be used o access iles whose policy equi es clien
au hen ica ion. To access o he iles, no explici session is equi ed. Such use is
concep ually ea ed as pa o a de aul , un us ed session.
A session is es ablished wi h a s anda d handshake p o ocol in which he clien
and Gua da au hen ica e each o he using hei p i a e keys. As pa o he p o ocol,
new, session-speci ic keys a e c ea ed. These keys a e used o enc yp and/o
au hen ica e ( h ough message au hen ica ion codes) all subsequen communica ion
in he session. This p o ec s in- ansi da a and commands om snooping and
modi ica ion in in e media e laye s. Mo eo e , he clien ’s public key (which ac s as
a clien iden i ie ) becomes a ailable du ing e e y policy e alua ion in he session;
hence, Gua da can en o ce policies ha es ic access o a speci ic use . A he
end o he handshake, Gua da e u ns a unique session iden i ie (sId) ha links
la e commands o he session. In he desc ip ion o he emaining in e ace, we omi
21

Session API:
message,sId handshake1(message) Ini ia es he session es ablishmen .
in handshake2(sId,message) Finalizes he session es ablishmen .
in endsession(sId) Te mina es he session sId.
T ansac ion API:
Id openTx(sId,objname) S a s a ansac ion on ile named objname.
in endTx(sId, Id) Commi s a ansac ion.
in
se Policy
(sId, Id,
pId)
Se policy pId o objname.
in
euse
(sId, Id,o ,len,
o ’)
Takes con en o in e al [o ,o +len - 1] and inse s
con en a o ’.
in
esh
(sId, Id,b,len,
bu ,o [, cache lag])
W i e con en o block and add o objname a o se o .
bu
eadTx
(sId, Id,o ,
len [, cache lag])
Reads om objname a o se o .
File/Policy API:
pId c ea ePolicy(sId,policy) S o es policy and e u ns a unique iden i ie pId.
in des oy(sId,objname) Dele es objname’s me ada a and con en .
Con en Hashing API:
hId ini Hash( Id,cu O New)
C ea es hash iden i ie o cu en o new objname.
ce i ica e closeHash(hId)
Compu es hash, c ea es ce i ica e and s o es hash
in cache.
Ce i ica e API:
nonce ge Nonce(sId) Re u ns pseudo- andom nonce alue.
in se Ce i ica e(sId,ce i ica e) P o ide ce i ica e o Gua da .
ce i ica e a es (sId,objname,nonce)
C ea es a ce i ica e a es ing he s a e o obj-
name.
Replica ion/Mig a ion API:
bu pickle(sId,objname, a ge GdKey)
C ea es an enc yp ed bu e bu including
he con en and policy o ile objname which
can only be enc yp ed by a Gua da de ice
wi h a ge GdKey as public key.
in unpickle(sId,bu ,objname)
Dec yp s bu o ex ac con en cand policy
p; c ea es policy pand ile named objname
wi h con en c; associa es he p e iously c e-
a ed policy.
Table 3.1: Gua da In e ace Calls
22
he sId a gumen as i appea s in e e y call. Gua da can wo k wi h any clien -side
in as uc u e o c ea ing, managing and dis ibu ing public keys.
3.3.2 T ansac ion in e ace
Rich policies may equi e mo e han one ead o w i e ope a ion o ansi ion a
ile om one complian s a e o ano he . Fo ins ance, a ile’s in eg i y policy may
equi e ha each upda e inc emen s an embedded e sion coun e . Fo his pu pose,
Gua da suppo s ansac ions consis ing o a sequence o eads and upda es on a
single ile. T ansac ions a e a omic: ei he all he upda es a e pe sis ed o hey a e
all disca ded. Policies may e e o bo h he cu en and new con en o a ile in
a ansac ion, as well as he con en o o he iles. The policy is checked once a
he end o he ansac ion, which commi s i he policy check succeeds, and abo s
o he wise.
We ind his design use ul in encoding policy s a e machines and access-accoun ing
policies, as illus a ed in Sec ion 3.5. Howe e , he design comes wi h a ade-o :
To a oid bu e ing a po en ially unbounded numbe o upda es du ing a ansac ion,
Gua da o bids des uc i e upda es as pa o a ansac ion. Ins ead, new con en
mus be w i en o esh (no cu en ly alloca ed o a policy-p o ec ed ile) ex en s
on disk. This choice mi o s mode n ilesys em designs wi h copy-on-w i e block
alloca ion, e.g., in WAFL, ZFS, and B s [
56
,
125
,
23
]. Ou side a ansac ion,
des uc i e w i es succeed i allowed by he policy.
The ansac ion API adds 5 new commands: openTx, endTx, euse, esh and
eadTx, se Policy. The call
openTx
(sId,objname) s a s a new ansac ion on he ile
named objname. Gene a ing objname is up o he (un us ed) highe laye s, e.g., he
ilesys em. I objname does no exis , a new emp y ile is c ea ed and gi en his name
( his is he only way o c ea e a ile in Gua da ). The call e u ns a ansac ion id
( Id) ha links la e calls o he ansac ion and he session. A ile is upda ed by
23
eusing con en om i s cu en e sion and adding esh con en o c ea e a new
e sion. The call
euse
( Id,o ,len,o ’) akes con en in he logical ange [o ,o +len-1]
om he cu en e sion and inse s i a o se o ’ in he new e sion (inse ion is
pu ely a me ada a ope a ion). The call
esh
( Id,blk,len,bu ,o ) w i es len by es
om bu e bu o he ex en s a ing a by e numbe bon disk and adds he esul ing
ex en o he new e sion a logical o se o . Be o e w i ing he ex en , Gua da
checks ha i is no occupied by any ile (including he ile being modi ied). The
new e sion o he ile may be gi en a new policy wi h he call
se Policy
( Id,pId).
The call bu
eadTx
( Id,o ,len) eads len by es o he ile s a ing a logical o se o
in he ile and e u ns he esul o he bu e bu . The ead ule o he ile’s policy is
e alua ed be o e eading o bu ; i i denies access, he call ails. This en o ces da a
con iden iali y. No e ha we allow by e-le el add essing on iles, so policies can be
e y ine-g ained.
The upda es in a ansac ion a e commi ed wi h he call
endTx
( Id). Gua da
e alua es he upda e ule o he ile’s policy be o e commi ing he new e sion. This
en o ces da a in eg i y. The upda e ule has access o he cu en and new con en o
he ile, as well as ele an me ada a, e.g., he o se s and leng hs o eads and w i es
in he ansac ion. Addi ionally, i he policy has been upda ed, Gua da e alua es
he se policy ule o he ile’s policy; his p o ec s he policy i sel om unau ho ized
changes.
3.3.3 File/Policy in e ace
While ile c ea ions a e implemen ed as ansac ions, ile des uc ion and policy
c ea ion exis as addi ional calls. The
des oy
(objname) call emo es he con en
and me ada a o he ile named objname om Gua da a e success ully e alua ing
he des oy pe mission o he associa ed policy. To educe he equi ed me ada a
space, Gua da allows mul iple iles o be p o ec ed by he same policy. The e o e,
24
he
c ea ePolicy
call e u ns a policy Id (pId) which can be used mul iple imes in
se Policy calls du ing a ansac ion.
3.3.4 Con en cache in e ace
Gua da policies may be con ingen on he cu en con en o one o mo e iles and
he p oposed new con en o he upda ed ile in he con ex o a ansac ion. To
enable he e icien e alua ion o such policies, wo Gua da caches hold ile con en
o use in policy e alua ion. A pe -session cache con ains en ies ha e e o cu en
ile con en s, ei he as a sequence o by es a a gi en ile o se and leng h, o as
he hash o such a sequence. A pe - ansac ion cache con ains he same ypes o
en ies bu e e s o en a i e upda es o a ile. En ies a e added o he cache as a
side-e ec o ead, w i e, esh o eadTx commands wi h app op ia e lags (cache lag).
When a ansac ion commi s, any en ies in he ansac ion cache a e mo ed in o
he session cache, and any exis ing session cache en ies hey supe sede a e e ic ed.
When a ansac ion abo s, he en ies in he ansac ion cache a e disca ded. To
sa is y a policy ha e e s o cu en o pending ile con en , un us ed clien code
is expec ed o ill app op ia e cache en ies by issuing ead/w i e commands be o e
a emp ing a ansac ion commi .
In o de o i e a i ely build con en hashes, Gua da o e s he
ini Hash
call o
s a he hash compu a ion. I he e u ned iden i ie (hId) is speci ied as cache lag
du ing a ead, w i e, esh o eadTx call, hen he espec i e con en is added o
he hash compu a ion. A e a clien inishes he ead/w i e sequence, she closes
he hash ia he
closeHash
call which inalizes he hash compu a ion and s o es
he esul in he espec i e session o ansac ion cache o la e use du ing policy
e alua ion. In addi ion a c yp og aphically signed ce i ica e including he compu ed
hash, ile name and a hash o he associa ed policy is e u ned.
25
We belie e ha policies will be w i en mos ly by p i acy and secu i y expe s.
Fo any applica ion, he e will be a limi ed numbe o basic use ul policies, and
mos sys em adminis a o s, use s o de elope s will me ely selec om a lib a y o
policies, pe haps wi h mino cus omiza ion.
3.5 Policy examples
We illus a e Gua da ’s capabili ies by p esen ing example policies o p o ec exe-
cu ables, log iles and backups. I he
ead
o
upda e
ule o a policy is omi ed,
hen he pe mission is always allowed and i a
se policy
o
des oy
ule is omi ed,
hen ha pe mission is ne e allowed.
3.5.1 P o ec ed execu ables
Fo an execu able ile, i is desi able o p e en acciden al o malicious o e w i ing
o ollback o a p io e sion. A ep esen a i e Gua da policy o accomplish his
is shown below. The policy s a es ha he new con en o he execu able a e any
upda e mus be signed by he so wa e endo (called “Vendo ”) as being e sion 10
o la e . Mo eo e , any policy changes mus be ce i ied wi h he adminis a o ’s
key, kad.
upda e :- ile_name_is(F)∧new_leng h_is(L)∧
(0, L)willHa eHash Nh ∧key_is(K, “Vendo ”)∧
Ksigns ok_hash(F, N, Nh)∧(N≥10)
se policy :- ile_name_is(F)∧
new_pol_hash_is(Nph)∧
kad signs good_policy(F, Nph)
The i s ule allows an upda e o he ile only i he e is a public key
K
belonging
o “Vendo ” (condi ion
key_is
(
K, “Vendo ”
)), which signs ha he ile’s new con en
32

hash,
Nh
, is he
N
h e sion o he execu able (condi ion
Ksigns ok_hash
(
F, N, Nh
))
and
N≥
10. The p edica es
key_is
(
K, “Vendo ”
)and
Ksigns ok_hash
(
F, N,Nh
)a e
e i ied om clien -p o ided ce i ica es signed by a ce i ying au ho i y and he
endo , espec i ely. The second ule allows a change o he execu able’s policy only
i he hash o he new policy, called
Nph
, has been ce i ied by he adminis a o
(condi ion kad signs good_policy(F, Nph)).
P ope ies:
As long as he in eg i y o he endo ’s and admin’s keys is main ained,
iles p o ec ed by he policy canno be o e w i en excep wi h con en signed by he
endo and e sion
≥
10, e en i he en i e sys em is comp omised (w i e in eg i y).
A a ian o his policy can limi con en on he sys em’s boo sec o o endo -signed
boo images, hus p o ec ing he boo sequence om ojans and oo ki s.
3.5.2 Append-only logs
The ollowing policy speci ies an append-only ile ha may be ex ended by anyone
bu modi ied in-place (e.g., o a ed) only by an adminis a o iden i ied by he public
key
kad
. The policy p e en s acciden al o malicious manipula ion o sys em log iles.
upda e :- session_is(kad)∨
(old_leng h_is(Lo)∧new_leng h_is(Ln)∧(Ln ≥Lo)∧
upda ed_loca ions_a e(M)∧disjoin (M, [0, Lo]))
The policy allows an upda e i ei he he session is au hen ica ed by he adminis a o
(condi ion
session_is
(
kad
)) o he ile’s new leng h
Ln
exceeds i s cu en leng h
Lo
and he i s Lo by es o he ile a e no modi ied.
P ope ies:
As long as he in eg i y o he admin’s key is main ained, he policy is
en o ced e en i he sys em is comp omised.
33
3.5.3 P o ec ed backup
Backup iles can be p o ec ed om acciden al o malicious modi ica ion o a ixed
pe iod o ime using he ollowing policy.
upda e :- key_is(K, “TimeSe e ”)∧
Ksigns ime(T)a Ti∧
coun _is(Tj)∧(T+Tj−Ti>endT)
The policy allows modi ica ion o he ile only i he cu en ime exceeds a p e-
de e mined ime
endT
. To en o ce such policies, Gua da elies on signed ce i i-
ca es om ime se e s and a sho - ange in e nal iming coun e . In de ail, he
policy says ha he e should be a key
K
belonging o a ime se e (condi ion
key_is
(
K, “TimeSe e ”
)), which issued a ce i ica e ha he ime was
T
when he
Gua da in e nal coun e had alue
Ti
(condi ion
Ksigns ime
(
T
)
a Ti
), he cu en
in e nal coun e alue is
Tj
(condi ion
coun _is
(
Tj
)) and he cu en ime (calcula ed
as T+Tj−Ti) exceeds he backup end ime endT.
P ope ies:
As long as he in eg i y o he ime se e and i s signing key is
main ained, a ile wi h his policy canno be modi ied be o e he designa ed ime,
e en i he sys em, he admin’s and he ile owne ’s p i a e keys a e comp omised.
3.5.4 Manda o y access logging (MAL)
Legisla ion and o ganiza ional policies o en manda e ha all ead and w i e access
o sensi i e in o ma ion like medical eco ds be logged. Al hough applica ion-le el
solu ions o en o ce such manda o y access logging (MAL) exis , en o cing he policy
in Gua da is desi able because i would inc ease secu i y.
34
Fo his exposi ion, le
P
be he sensi i e ile which mus be p o ec ed by MAL
and le
L
be i s log ile. We assume ha he log ile is append-only, h ough he
policy desc ibed ea lie . The MAL equi emen is h ee- old:
Comple eness
Fo e e y ead on
P
, an en y in
L
should desc ibe who ead and
om whe e in
P
. Fo e e y w i e, a simila en y mus exis in
L
and i mus
addi ionally con ain a hash o he con en w i en.
Causali y
Gi en wo w i e en ies in
L
, he o de in which hey we e applied o
P
should be e iden and, simila ly o a ead and a w i e en y.
P ecision
Call a w i e en y in
L
dangling i i does no co espond o an ac ual
w i e on
P
. Then, ei he dangling en ies should no be allowed in
L
o hey
should be de ec able.
Dangling ead en ies a e usually no a p oblem, because i is in he clien ’s
in e es o es ablish ha i did no ead ce ain da a and, hence, no c ea e dangling
ead en ies. We also desc ibe la e how ead en ies can be made p ecise.
We s a wi h an ob ious s awman policy o
P
, which is comple e, bu does
no p o ide causali y and p ecision. We e ine he design la e . We de ine wo kinds
o en ies o
L
:
may_ ead
(
K, S
), which indica es ha he clien wi h public key
K
has po en ially ead he se
S
o (o ,len) anges om
P
; and
change
(
K, S, H
),
which s a es ha con en wi h hash
H
has been w i en o he anges in
S
. To o ce
logging o eads, we equi e in he
ead
ule o
P
’s policy ha i he ange
R
is ead
by clien
K
, hen an en y
may_ ead
(
K, S
)wi h
R⊆S
exis in
L
. Simila ly, w i e
logging could be o ced h ough P’s upda e ule.
This s awman policy o
P
can be exp essed in he Gua da policy language
because he se
R
o loca ions ead o upda ed is a ailable h ough con ex ual
p edica es in he policy language, he clien
K
is a ailable h ough he p edica e
is_session
(
K
)and
L
’s con en is a ailable h ough he session cache (p edica e
says
).
35
The policy can also be easily sa is ied by he clien : P io o eading o w i ing,
he clien could append an app op ia e en y o
L
and ha e i cached o
P
’s
subsequen policy e alua ion. E en hough his policy sa is ies he MAL equi emen
o comple eness, i does no sa is y causali y and p ecision. No hing in
L
’s policy
p e en s he clien om c ea ing en ies ha a e ne e used and such en ies canno
be dis inguished om o he s ( his iola es p ecision). Mo eo e , no hing in
P
’s
policy p e en s use o L’s en ies ou -o -o de , which iola es causali y.
To ob ain causali y and p ecision, we e ine his s awman design. We embed a
coun e in each en y in
L
and en o ce h ough
L
’s policy ha he coun e inc ease
by 1a each successi e
change
en y and emain he same a each
may_ ead
en y. We
en o ce h ough
P
’s policy ha he alue o he coun e in he las
change
en y ha
has al eady been applied o
P
be w i en a a designa ed locus in
P
. Fu he , he
en y used o jus i y a ead mus ha e a coun e numbe ha ma ches he cu en
coun e in
P
. We desc ibe below how we en o ce hese equi emen s. Assuming ha
hey ha e been en o ced, bo h causali y and p ecision a e sa is ied. Causali y holds
because he policies jus desc ibed o ce ha
change
en ies apply o
P
in inc easing
o de o hei coun e numbe s, and ha a ead co esponding o a
may_ ead
is used
a e all
change
en ies wi h smalle o equal coun e numbe s ha e been applied.
P ecision holds because a
change
en y is dangling i and only i i s coun e numbe
is highe han he coun e in P.
The log’s en ies a e e ised o include coun e numbe s. They ake he o ms
may_ ead
(
N, K, S
)and
change
(
N, K, S, H
), whe e
N
deno es a coun e . We ese e a ixed
locus in
P
o a coun e , called
C
. The log is ini ialized wi h a dummy en y wi h
N
= 0 and
P
is ini ialized wi h
C
= 0. We desc ibe ele an policies o
L
and
P
in
wo ds, omi ing symbolic ep esen a ions o cla i y. We ha e o mally ep esen ed
hese policies in ou p o o ype implemen a ion; expe imen al esul s a e p esen ed in
Sec ion 3.7.5.
36
L’s upda e policy:
Only appends a e allowed and only en ies o he wo designa ed
o ms may be added. I he added en y has he o m
may_ ead
(
N, . . .
), hen
N
mus be copied om he p e ious en y and i he added en y has he o m
change
(
N, . . .
), hen
N
mus be one mo e han he p e ious en y’s coun e .
These equi emen s can be ep esen ed in he Gua da policy language because
he p e ious en y and he new en y a e accessible h ough he session and
ansa ion caches, espec i ely, du ing e alua ion o he upda e ule.
P’s ead policy: L
mus con ain a
may_ ead
en y wi h he same coun e numbe
as
C
and ange se la ge han he ac ual ange ead.
L
’s ele an en y and
C
a e accessible h ough he session cache du ing
P
’s policy e alua ion. In
pa icula ,
C
can be e e enced because Gua da suppo s by e-le el add essing
on iles and he locus o
C
is ixed in ad ance. The clien is esponsible o
speci ying which en y o Lin he session cache sa is ies he policy.
P’s upda e policy: L
mus con ain an en y desc ibing he upda e p ecisely. The
coun e in he en y mus be one mo e han
C
. The upda e mus also inc emen
C
by 1. When e alua ing
P
’s policy,
L
’s ele an en y and he old alue o
C
a e accessible h ough he session cache. The new alue o
C
is accessible
h ough he ansac ion cache.
MAL clien :
The MAL clien mus pe o m some bookkeeping s eps o sa is y he
MAL policy. P io o each access on
P
, app op ia e log en ies mus be c ea ed and
commi ed o Gua da . When c ea ing log en ies, lags mus be se o bu e hem
in he con en cache o use in
P
’s policy e alua ion. A log en y’s cache eco d is
also necessa y o c ea e he nex log en y. Simila ly, when
C
is upda ed, lags mus
be se o cache i o use in u u e policy e alua ions. This app oach ollows om
ou design p inciple o placing he bu den and complexi y o how o sa is y a policy
on he un us ed code.
37

The o e head o c ea ing log en ies o upda es can be educed by commi ing
ansac ions less equen ly (and, hence, equi ing ewe
change
en ies). Simila ly,
he o e head o c ea ing log en ies o eads can be educed by clubbing se e al
an icipa ed eads in o a single
may_ ead
en y. The pe o mance bene i o hese
op imiza ions is subs an ial and we epo on i in Sec ion 3.7.5. Applica ions
ha canno accu a ely es ima e hei ead-se s ahead o ime can simply c ea e
blanke
may_ ead
en ies ha co e he en i e ile and pe iodically commi ead-only
ansac ions accompanied by special log en ies ha speci y p ecisely wha has been
ead in he ansac ion. The p ecise ead se is a ailable o Gua da du ing a commi
ansac ion, so he log en y’s accu acy can be e i ied. This mode o use equi es a
second coun e in log en ies and he sensi i e ile o coun ead-only ansac ions.
3.5.5 O he policy idioms
Many o he common policies can be exp essed in Gua da . Examples include:
(a) Role-based policies whe e access depends on he clien ’s ole in an o ganiza ion
(ce i ica es can ela e clien s o oles), (b) Blacklis (whi elis ) policies whe e access
is denied (allowed) i he clien ’s iden i y exis s in a so ed ile ( he ile’s so edness
can also be en o ced using Gua da policies), and (c) His o y-based policies whe e
access depends on pas e en s ha a e isible o Gua da . The la e can be en o ced
by eco ding e en s in a dedica ed log ile and allowing access o he da a ile only
when he log ile is in ce ain s a es. The MAL policy is a simple his o y-based
policy ha allows access only when he e en o c ea ing an app op ia e log en y
has occu ed.
3.5.6 Exp essi eness
As hese examples demons a e, he Gua da policy language is exp essi e. I
can exp ess con en -based policies like MAL ha p io wo k on decla a i e policy
38
languages canno . Howe e , he language has limi a ions. I disallows ecu si ely-
de ined p edica es and, hence, canno exp ess layou s de ined by i e a ion o ecu sion,
e.g., i canno exp ess ha he con en o a ile be well- o med XML. Such cons ain s
may be checked by a us ed ex e nal e i ie using ce i ica es o communica e
be ween he e i ie and Gua da , o by ex ending he language wi h ecu si e
p edica es.
3.6 Implemen a ion
This sec ion desc ibes he p o o ype implemen a ion o Gua da in a SAN se e
and p esen implemen a ion al e na i es o he Gua da design.
3.6.1 P o o ype
Ou p o o ype is based on he iSCSI En e p ise Ta ge (IET) SAN se e , which
implemen s he se e -side iSCSI p o ocol and p o ides SCSI block s o age access
ia E he ne . IET is in p oduc ion use and a ailable o many Linux dis ibu ions.
Figu e 3.1 depic s he componen le el design. The se e accesses an SSD o he
Gua da me ada a and one o mo e payload disks which a e ei he magne ic- o
lash-based. IET consis s o a ke nel module, which implemen s block accesses, and
a use -le el daemon p ocess, which implemen s iSCSI managemen unc ions. To
implemen Gua da , we ex ended he ke nel module and added a second use -le el
daemon, which implemen s he Gua da in e ace and e alua es policies. The ke nel
module pe o ms upcalls o de e mine i iSCSI block accesses should be allowed. The
se e is con igu ed wi h a small SSD o s o ing Gua da me ada a, as well as one
o mo e magne ic disks o SSDs o he payload da a.
The Gua da daemon main ains wo B- ee index s uc u es on he me ada a
SSD: a block- o- ile index o ind he ile and policy associa ed wi h a gi en block
numbe , and a name- o- ile index o e ie e he ile in o ma ion (se o ex en s,
39
Figu e 3.1: Gua da implemen a ion in a SAN se e
policy, e c.) gi en a ile id. Fo pe o mance, he Gua da daemon main ains a
w i e- h ough DRAM cache o B- ee nodes and policies, backed by he SSD. Upda es
a e pe sis ed on he SSD du ing a ansac ion commi .
When he ke nel module ecei es a block access eques , i passes he access ype
( ead/w i e) and loca ion (disk o se , leng h) o he mul i- h eaded Gua da daemon,
which consul s he block- o- ile index. I he block loca ion is no associa ed wi h
a policy-p o ec ed ile, he access is g an ed. O he wise, he daemon e alua es he
policy and e u ns he esul o he ke nel module. Fo ead eques s, he block ead
is scheduled while checking he pe mission o educe la ency. Du ing a w i e eques ,
he block w i e mus be de e ed un il he Gua da daemon g an s he pe mission.
To educe he numbe o upcalls and policy e alua ions, he ke nel module main-
ains a cache o p e ious policy e alua ion esul s o he o m
hex en , pe missionsi
.
To eed his cache, he Gua da daemon always e u ns he la ges ex en encompass-
ing he p esen ly eques ed block o which he same pe missions hold. The cache is
lushed when a policy changes. This op imiza ion a oids policy e-e alua ion and
sa es he communica ion cos be ween ke nel module and he Gua da daemon in
many cases.
Ou p o o ype’s a ack su ace consis s o he IET managemen in e ace, he
block-de ice in e ace, he Gua da in e ace ex ensions as well as he policy language.
40
Despi e he ela i ely la ge IET codebase, which includes a minimally con igu ed
Linux ke nel, he esul ing a ack su ace is likely o be signi ican ly smalle han
ha o he sys ems and applica ions buil on op o Gua da in mos cases. Ou
Gua da implemen a ion adds less han 20,000 LOC o he exis ing IET codebase,
plus he OpenSSL and glib lib a ies i elies on.
3.6.2 Implemen a ion al e na i es
Gua da can be implemen ed in di e en ways depending on he deploymen and
h ea model. The GDC can be implemen ed using he ollowing mechanisms:
(a)
In a SAN se e o use in a da a cen e , as desc ibed in he p e ious p o o ype
sec ion.
(b)
In eg a ed wi h he mic ocon olle o a hyb id disk o use in an indi idual
machine.
(c)
In a us le wi hin a i ual machine moni o o ope a ing sys em, isola ed
using us ed ha dwa e ea u es like In el SGX [63] o ARM T us Zone [12].
Table 3.3 lis s examples o deploymen scena ios, hei h ea models and us
assump ions. As desc ibed in he h ea model each implemen a ion mus p o ec
he GDC, me ada a and da a om unau ho ized physical access and unde ec ed
ampe ing
Implemen a ion (a) elies on physical p o ec ion, e.g., in a machine oom wi h
access only by us ed employees. A possible deploymen scena io a a Cloud p o ide
p o ec s use da a om bugs and miscon igu a ions in i s in as uc u e and om
oppo unis ic access by employees. The use mus us he Cloud p o ide o p e en
physical access o he SAN se e by all bu us ed employees.
In implemen a ion (b), he GDC is implemen ed as pa o a mic ocon olle
embedded in a hyb id disk. He e, he me ada a and da a a e enc yp ed and au-
41
0.5
1
1.5
2
2.5
3
3.5
4
RR LR SR RW LW SW
Rela i e o e head
Wo kload
iSCSI
Gua da emp y
Gua da ile
Gua da policy
Figu e 3.3: La ency wi h an SSD, ela i e o iSCSI
Figu e 3.3 shows he esul ing a e age access la ency wi h he SSD, ela i e o
he plain iSCSI. E en wi h he as SSD as a block s o e de ice, he Gua da la ency
o e head is gene ally low, bu signi ican o andom w i es (2- old inc ease). The
ac ha ou block s o e SSD pe o ms andom w i es much as e han andom
eads (153
µ
s e sus 233
µ
s), p esumably due o w i e bu e ing in i s in e nal DRAM,
combined wi h he ac ha he policy check canno be o e lapped wi h he access
du ing a w i e, con ibu es o his high ela i e o e head.
No e ha he andom access wo kload is ex eme: The SSD block s o e de ice
is e y as , we a e measu ing he la ency o iny accesses (512 by es) a andom
loca ions o e he en i e disk, and he e a e many iles and policies. Inc easing he
eques size educes he o e head. Fo example, wi h a 4K eques size, he o e heads
dec ease om 29.3% o
RR
and 101.6% o
RW
o 17.7% and 96.1%, espec i ely.
Wi h 128K eques s, he o e heads go u he down o 0.9% and 23.5%, espec i ely.
Mo eo e , as we show nex , e en unde his wo kload he SSD e ains much o i s
la ency ad an age o e he HDD wi h Gua da , and Gua da ’s h oughpu o e head
is e y low on bo h he SSD and he HDD.
Figu es 3.4 and 3.5 compa e he absolu e la encies achie ed on a HDD and SSD
wi h and wi hou Gua da . Despi e Gua da ’s la ge ela i e o e heads o pu ely
andom w i es, he SSD e ains i s owe ing la ency ad an age on such accesses o e
48

0
0.1
0.2
0.3
0.4
0.5
0.6
RR LR SR RW LW SW
Response ime (ms)
Wo kload
iSCSI SSD Gua da policy SSD
Figu e 3.4: Absolu e la ency wi h SSD
0.01
0.1
1
10
RR LR SR RW LW SW
Log. esponse ime (ms)
Wo kload
iSCSI HDD Gua da policy HDD
Figu e 3.5: Absolu e la ency wi h HDD
he HDD (no e ha he y-axis is di e en o SSD and HDD). Wi h he magne ic
HDD, he Gua da la ency o e heads o all con igu a ions a e negligible (below 1%).
Compa ed o a locally a ached SSD, he a e age la ency o a emo ely connec ed
iSCSI SSD inc eases by 0.051 ms, a li le mo e han one ne wo k ound ip (0.047
ms).
3.7.2.2 Read/w i e h oughpu
Nex we examine he ead/w i e h oughpu o he Gua da p o o ype, using he same
con igu a ions as he la ency expe imen . The es clien issues ou 128KB eques s
concu en ly, which is su icien o achie e maximal ead and w i e h oughpu in
49
0
100
200
300
400
500
600
RR LR SR RW LW SW
Th oughpu (MB/s)
Wo kload
iSCSI
Gua da emp y
Gua da ile
Gua da policy
Figu e 3.6: SSD I/O h oughpu
he baseline iSCSI in all cases. Fo each access pa e n in each con igu a ion, we un
he h oughpu es 5 imes; each un issues a o al o 20,000 accesses and s a s a
a andom block wi hin he disk.
Figu e 3.6 shows he absolu e h oughpu wi h he SSD. The esul s shown
a e he a e ages o 5 uns, whe e e o ba s indica e he s anda d de ia ion. The
Gua da o e head is below 2% o all access pa e ns wi h he SSD. Wi h he HDD,
he o e heads a e in he same ange.
The high la ency o e head on andom w i es does no signi ican ly a ec he
h oughpu because policy e alua ion o di e en eques s can be pe o med in
pa allel by he mul i- h eaded Gua da daemon, and o e lapped wi h disk and SSD
accesses o me ada a and blocks.
Mo eo e , compa ed o a locally a ached SSD, he h oughpu o e head is a
mos 3% o all iSCSI and Gua da con igu a ions and wo kloads.
3.7.2.3 I/O pe o mance summa y
While Gua da adds li le la ency o HDD accesses and SSD accesses wi h good
locali y, i has a no iceable la ency o e head on small, pu ely andom w i es o
an SSD. Howe e , his o e head diminishes quickly wi h la ge eques sizes and
50
Policy size Domain size
1 2 4 8 16
12.2 3.4 5.8 10.7 20.4
24.6 10.4 28.9 95.1 345.8
37.0 24.0 121.2 770.5 5,518.1
49.4 50.9 485.3 6,156.4 88,319.3
511.9 104.9 1,951.3 49,234.7 1,411,800.8
Table 3.4:
E alua ion la ency in
µ
s o a ying policy size (numbe o p edica es
and a iables in he policy) and domain size (maximum numbe o cache en ies)
mo e locali y, and can be o e lapped wi h concu en accesses, so ha he SSD’s
h oughpu is no a ec ed.
3.7.2.4 Policy e alua ion o e head
Consis en wi h Da alog, he heo e ical wo s -case e alua ion ime o a policy
ule is in
O
(
m·Dn
), whe e
m
is he size o he ule (numbe o p edica es),
D
is
he size o he domain (bounded by he size o he Gua da cache) and
n
is he
numbe o a iables in he ule. In Table 3.4, we show he measu ed policy e alua ion
ime o syn he ic policies designed o ex ica e wo s -case execu ion om ou policy
in e p e e .
D
a ies along columns o he able and
m
and
n
a y along ows (
m
=
n
in all expe imen s). The esul s ma ch he expec ed complexi y
O
(
m·Dn
). The
able indica es (co ec ly) ha policy e alua ion could be a subs an ial bo leneck o
some policies bu we do no obse e his bo leneck in p ac ice. The a e age policy
e alua ion la ency o he mos complex policy e alua ed, MAL (Sec ion 3.7.5) is only
27.7
µ
s, e en hough he policy has
m
= 4,
n
= 4 and
D
= 40. This is because o a
ca e ul implemen a ion o he policy in e p e e o conside mo e ecen cache en ies
i s . Ou o he
example
policies e alua e e en as e ; he a e age e alua ion ime o
he ime-based policy om he la ency expe imen con igu a ion
Gua da policy
is
only 3.7µs.
51
3.7.2.5 Space equi emen s o me ada a
We quan i y he me ada a s o age equi emen s. Because he me ada a size depends
on he s uc u e o he payload da a, we analyzed he me ada a space equi emen s
o 70,825 ilesys em snapsho s collec ed by Ag awal e al. [
2
]. The snapsho s we e
aken om Windows sys ems wi hin Mic oso co po a ion be ween 2000 and 2004,
and con ain be ween 30k and 90k iles each wi h an a e age ile size be ween 108KB
and 189KB. Fo e alua ion pu poses, we gi e each ile in each snapsho an in eg i y
policy ha disallows modi ica ion p io o a gi en da e. The snapsho s a e mo e
han 10 yea s old a he ime o his w i ing. Because he a e age ile size in
oday’s sys ems has likely inc eased, howe e , ou analysis o Gua da ’s me ada a
equi emen s ela i e o he size o he da a is conse a i e.
The equi ed me ada a can be accommoda ed in a solid s a e memo y o 0.8% o
he da a size o 99.89% o he snapsho s. As a poin o e e ence, e en comme cially
a ailable hyb id disks p o ide a leas 0.8% Flash [
112
] a he ime o his w i ing.
Newe combina ions o Flash/disk de ices achie e much highe Flash o disk capaci y
a ios and his end is p ojec ed o con inue gi en he p ice and space educ ion
a es o lash memo y. Fo example, Apple’s Fusion D i e [
10
] has a a io o 128GB
Flash o a 1TB HDD, which can easily accommoda e all he snapsho s. In all ou
expe imen s, which use o he da a se s, he me ada a i in o only 0.2% o he da a
size.
3.7.2.6 Flash memo y wea
Because Flash memo y can endu e only a limi ed numbe o e ase/p og am cycles,
we mus check ha he SSD used o s o e me ada a will no wea quickly. To be
conse a i e, we assume ha he Flash mus las a leas 10 yea s. The li e ime is
in luenced by he size o he me ada a, he a e o me ada a upda es, and he Flash
capaci y. A smalle capaci y causes he Flash log o w ap a ound as e and leads o
52
0
50
100
150
200
250
300
iozone( /w) bonnie++( /w)
Pe o mance (MB/s)
iSCSI SSD
Gua da policy SSD
iSCSI HDD
Gua da policy HDD
Figu e 3.7: FS benchma ks ead and w i e ( /w) pe o mance
highe u iliza ion, which in u n educes cleaning e iciency and equi es e en mo e
Flash w i es.
Unde he con igu a ion o
Gua da policy
used abo e, we keep ack o how
much wea he Flash expe iences while p esen ed wi h a se ies o me ada a upda es,
i.e., adding and emo ing ex en s o a con en ile picked a andom. En e p ise
en i onmen s ypically deploy single-le el cell (SLC) Flash memo y, which has a
nominal li e ime o 100,000 e ase/p og am cycles. Using only 4GB o such memo y
we can accommoda e up o 19.5M upda es pe day (225 pe second). This is an
ex ao dina ily high upda e a e ha can accommoda e e en he mos w i e-in ensi e
applica ions. Cheape mul iple-le el cell (MLC) and iple-le el cell (TLC) Flash
memo y wi h nominal li e imes o 10,000 and 1,000 e ase/p og am cycles would
suppo up o 1.95M and 195,000 me ada a upda es pe day, espec i ely.
3.7.3 Filesys em benchma ks
Nex , we measu e he pe o mance o he Gua da p o o ype using he s anda d
ilesys em benchma ks
iozone
3.429 and
Bonnie++
1.03. The block s o e was
o ma ed unde ex 4.
iozone
uses ou wo ke h eads o w i e 1GB sequen ially
53

o ou sepa a e iles.
1
La e , each wo ke pe o ms a sequen ial ead o he ile
hey p e iously w o e. Simila ly,
Bonnie++
w i es hen eads 1GB each o 16
iles. Figu e 3.7 shows he pe o mance o he baseline and Gua da unde he
Gua da policy
con igu a ion. The esul s shown a e he a e ages o 5 uns and
e o ba s indica e he s anda d de ia ion. The Gua da o e heads a e below 1.0%
o bo h benchma ks on bo h he HDD and he SSD. No e ha
Bonnie++
uses he
C lib a y unc ions ge c and pu c o pe o m ile eads and w i es, and is he e o e
unable o sa u a e he disks.
Simila o he h oughpu expe imen , he iSCSI SSD esul s a e close o hose
achie ed wi h a locally a ached SSD (a mos 3.5% lowe ).
3.7.4 Use case: Web se e
Nex , we conside he pe o mance o he Gua da p o o ype as pa o a modi ied
Apache Web se e . The se e holds a 220GB s a ic snapsho o English language
Wikipedia a icles om 2008 [
137
] and Wikimedia images om 2005 [
136
], con aining
15 million iles wi h an a e age ile size o 15KB and maximum ile size o
∼
500KB.
The HTTP clien asynch onously eques s HTML pages om he Web se e , using
a wo kload based on he ac ual access coun s o Wikipedia pages du ing one hou on
Ap il 1, 2012 [
138
]. Because ou snapsho is much olde and had ewe a icles a
he ime, we igno e accesses o non-exis ing pages. In o al, abou 350,000 di e en
pages we e accessed in he ace, o which 250,000 a e pa o he 2008 snapsho .
Since we do no ha e access o ime s amps, we dis ibu ed he indi idual accesses
e enly wi hin an hou , and eplayed he i s 100,000 page eques s.
We use he ollowing Gua da policies o p o ec he se e ’s pe sis en s a e:
Con en :
Requi e con en upda es signed by owne s. We andomly assign one o
40,000 owne s o each con en ile.
1We used he command iozone -i 0 -i 1 - 512k -I -c -e -T - 4 -s 1g -F iles
54
200
250
0 10 20 30 40 50 60 70 80 90
Th oughpu (Reques s/s)
Numbe o concu en HTTP eques s
iSCSI
Gua da
Figu e 3.8: Web se e h oughpu
Execu ables/Con ig:
Requi e ha upda es o execu able and con igu a ion iles
be signed by he adminis a o .
Log iles:
The Apache log iles can only be appended, excep wi h an adminis a o
key used o o a e he log.
To sa is y he log ile policy, we added a o al o 51 lines o code o Apache. This
ex a code issues Gua da commands o send con en hashes o Gua da and lush
applica ion and ilesys em caches ( lush & sync) be o e e e y log ile upda e. The
policies p o ec ing he con en , execu ables and con igu a ion iles do no equi e any
modi ica ions o Apache.
Figu e 3.8 shows he a e age h oughpu o h ee uns as a unc ion o he numbe
o concu en HTTP accesses, o plain iSCSI and Gua da (s anda d de ia ion is
below 0.5%). Each un loads 100,000 Wikipedia pages. The h oughpu o e head o
he Gua da con igu a ion o e he unmodi ied iSCSI se e is 1.95% a 60 concu en
eques s, whe e iSCSI eaches i s peak h oughpu , and always wi hin 2.7%. This
esul shows ha he Gua da o e heads mos ly o e lap wi h o he ac i i ies in he
Web se e . The 100,000 page eques s esul in app oxima ely 350,000 Gua da
eads, o an a e age o 3.5 eads pe page. This shows ha a subs an ial numbe
o eads each he Gua da de ice and a e no abso bed by he ilesys em bu e
55
cache. In addi ion, Apache w i es 2.7MB o log eco ds in 170 ansac ions unde he
append-only policy. The e a e no upda es o con en , execu ables and con igu a ion
iles, no log o a ions in he wo kload, bu policies mus s ill be checked du ing each
access.
In e ms o unc ionali y, Gua da p o ec s con en , logs, con igu a ion and
execu able iles om ampe ing by unau ho ized pa ies, which we con i med h ough
aul injec ion expe imen s.
3.7.5 Manda o y access logging
In ou inal expe imen , we pe o m accesses o a ile wi h ou manda o y access
logging (MAL) policy. The policy equi es an app op ia e en y in a sepa a e log
ile o an access o be allowed by Gua da . We use a 64MB p ima y ile wi h o
wi hou he MAL policy in place. The p ima y ile and he log ile eside on di e en
HDDs a ached o he same Gua da IET se e . The e sion coun e embedded in
he p ima y ile is s o ed in Flash memo y no used by Gua da . The clien connec s
o he Gua da de ice and accesses he p ima y ile in h ee di e en con igu a ions.
no log: File accessed wi hou any logging and en o cemen . (ho izon al lines)
log: Accesses logged wi hou policy en o cemen .
Gua da MAL: Accesses logged and policy en o ced by Gua da .
Figu e 3.9 shows he a e age access la ency o 100,000 sequen ial 4KB eads and
w i es o he p ima y ile, a ying he numbe o accesses pe eco ded log en y om
1 o 512. E o ba s indica e he s anda d de ia ion. In he case o a single access
pe log en y, en o cing he MAL policy inc eases he ead/w i e la ency by 11.5%
and 50.6%, espec i ely, o e olun a y logging. The highe cos o logged w i es
compa ed o eads e lec s he need o upda e he e sion numbe . Bo h cos s can
be educed by issuing e sion coun e upda es, log w i es, and p ima y ile accesses
56
0.2
0.4
0.6
0.8
1
1.2
1.4
1 2 4 8 32 128 512
Access la ency (ms)
Accesses pe log en y
Read log
Read Gua da MAL
W i e log
W i e Gua da MAL
ead no log
w i e no log
Figu e 3.9: La ency wi h MAL, olun a y and no logging
in pa allel. Mo eo e , as shown in he igu e, he cos o MAL can be amo ized by
logging se e al accesses in a single log en y, and app oaches he cos o comple ely
unlogged accesses o 512 accesses pe log en y.
3.8 Rela ed wo k
Policy languages based on Da alog.
Many decla a i e policy language a e based
on Da alog and esemble he Gua da policy language in syn ax and seman ics. Some
examples a e Sou ei [
100
], Binde [
33
] and SecPAL [
18
]. Whe eas hese languages
a e gene ic, he Gua da policy language is domain-speci ic and con ains cus om-
designed, s o age- ele an p edica es (Sec ion 3.4). Sou ei, Binde and SecPAL allow
in ensional ( ecu si e, ule-de ined) p edica es, which he Gua da policy language
omi s o keep he implemen a ion simple. These p edica es can be added o Gua da
wi hou any concep ual challenges. DKAL [
52
] ex ends Da alog wi h decla a i e
ules o exchanging au ho iza ion c eden ials in dis ibu ed sys ems. Such ules can
be added o Gua da as well.
TCG s o age wo k g oup speci ica ion.
Al hough de eloped independen ly, he
Gua da a chi ec u e bea s some esemblance o s o age wo k g oup s anda ds o
he us ed compu ing g oup (TCG) [
127
]. Simila o Gua da , he TCG s anda d
57
is easy o deploy and amenable o an e icien implemen a ion, as demons a ed by
ou expe imen al e alua ion.
64

CHAPTER 4
ERIM: Secu e and E icien
In-p ocess Isola ion
The p e ious chap e desc ibed he design, implemen a ion and e alua ion o
Gua da , a s o age laye e e ence moni o en o cing con iden iali y, in eg i y and
accoun ing policies o e pe sis en da a. I en o ces hese policies independen o
highe abs ac ion laye s p o iding a s ong h ea model elying on a small TCB
and a ack su ace.
Howe e , he se o en o ceable policies a e limi ed by he obse able e en s
a he s o age laye . Gua da has no con ol o e da a eleased o an applica ion
wi h su icien access c eden ials. As a esul s an applica ion may leak da a o e he
ne wo k, due o bugs, malicious a acks o miscon igu a ions.
In con as o Gua da , ERIM media es an un us ed applica ion’s execu ion,
in e cep ing ele an applica ion ope a ions like accesses o p i a e da a o ope a ing
sys em se ices. To media e un us ed applica ions, ERIM pa i ions sensi i e da a
and code in o an isola ed and us ed componen , he eby limi ing he e ec s o bugs
and ulne abili ies in he un us ed componen o da a accessible in he un us ed
applica ion only. Fo ins ance, isola ing c yp og aphic keys om he emaining
applica ion can hwa ulne abili ies like he OpenSSL Hea bleed bug [
90
]; isola ing
jump ables can p e en a acks on he in eg i y o an applica ion’s con ol low; and
isola ing a managed language’s un ime can p o ec i s secu i y in a ian s om bugs
and ulne abili ies in co-linked na i e lib a ies.
65
Isola ion o emos equi es memo y isola ion, which p e en s an un us ed com-
ponen om di ec ly accessing he p i a e memo y o o he componen s. B oadly
speaking, memo y isola ion can be en o ced using one o wo app oaches. Fi s ,
we may ins umen he code o un us ed componen s wi h bounds checks p io o
indi ec memo y accesses, ensu ing ha memo y o o he componen s is no accessed
di ec ly, as in SFI [
129
]. Howe e , his app oach imposes o e head on all execu ion o
un us ed componen s due o bounds checks, and i equi es an addi ional echnique
o p e en ci cum en ion o he bounds checks in he ace o con ol- low hijacks [
68
].
The o al o e head is commonly o he o de o ens o pe cen poin s.
The second app oach is o use ha dwa e suppo o memo y isola ion such as
OS o hype iso (ex ended) page ables [
20
,
29
,
74
,
19
]. He e, as access checks in
ha dwa e p e en a componen om accessing he memo y o o he componen s, bu
he e is an o e head on swi ches be ween componen s, since ha dwa e p i ileges mus
be changed, e.g., by swi ching page ables and possibly in alida ing TLB en ies.
Recen wo k on in-p ocess isola ion such as Wedge [
20
], Sh eds [
29
], and ligh -weigh
con ex s (lwCs) [
74
] has educed he cos o ha dwa e-based isola ion somewha .
None heless, swi ching s ill equi es a sys em call and i s cos is signi ican (a 1 us
pe swi ch [
74
] a conse a i e swi ch a e o 100,000 imes a seconds amoun s o 10%
o e head).1
Consequen ly, he e is need o an isola ion echnique ha does no impose
con inuous o e head while a componen execu es and ha also has e y low swi ching
cos on componen ansi ion. ERIM achie es his goal by building on a ecen x86
ISA ex ension called memo y p o ec ion keys o MPKs, also simply called p o ec ion
keys [
64
]. MPKs allow agging each page wi h one o 16 domains, hus pa i ioning
a p ocess’ add ess space in o disjoin domains. A special pe -co e egis e , PKRU,
1
Using x86 memo y segmen a ion ins ead o page ables, as in Na i e Clien [
143
], can educe
he swi ch cos . Howe e , suppo o segmen a ion wi h 64-bi add essing is limi ed and Na i e
Clien has been dep eca ed in a o o he memo y-sa e language WebAssembly [53].
66
de e mines which domains a e accessible. Swi ching pe missions equi es only w i ing
he PKRU egis e wi h a use -mode ins uc ion, which is a ela i ely quick ope a ion
(11–260 cycles on cu en In el CPUs in ou expe imen s).
Howe e , since he PKRU-upda e ins uc ion is use -mode, MPK by i sel
in insu icien o secu i y: Comp omised o malicious componen s can execu e
he ins uc ion o gain unau ho ized access o he memo y o o he componen s.
To p e en his, ERIM addi ionally elies on bina y inspec ion o ensu e ha all
occu ences o his ins uc ion (called WRPKRU) in he bina y a e sa e, i.e., hey
canno be exploi ed o gain unau ho ized access. By design, his p ope y holds e en
i he e is a con ol- low hijack in he un us ed componen . Hence, he e is no need
o complemen ERIM wi h con ol- low in eg i y, which would add o e head.
ERIM dis inguishes i sel om p io wo k on applica ions ha ha e e y high
swi ching a es (~10
5
/s o mo e) and ha addi ionally spend a non i ial amoun o
ime in un us ed componen s. The e a e many such applica ions. We e alua e ou
p o o ype o ERIM on h ee such applica ions. Fi s , in he web se e nginx, we
show ha ERIM can isola e session keys. P o ec ing session keys is a meaning ul
goal, since a acks a ge ed a indi idual use s’ p i acy only need o comp omise
session keys. Second, we show ha ERIM can e icien ly isola e he sa e egion in
code-poin e in eg i y [
72
]. Thi d, we show ha ERIM can be used o isola e a
managed language un ime om possibly buggy na i e lib a ies. In all cases, we
obse e swi ching a es o o de s a leas 10
5
imes/s pe co e. ERIM p o ides s ong
ha dwa e isola ion wi h o e heads less han 1% o e e y 100,000 swi ches/s, which
is conside ably lowe han ha o exis ing echniques.
The ollowing sec ions desc ibe he design, h ee use cases, a p o o ype imple-
men a ion, ela ed wo k and he e alua ion using h ee use cases.
67
4.1 Design
Like p io wo k, ERIM enables a us ed applica ion componen o isola e sensi i e
da a om he es o he un us ed applica ion. Unlike p io wo k, ERIM suppo s
such isola ion wi h low o e head e en a high swi ching a es be ween he un us ed
applica ion and he us ed componen , and wi hou elying on any o he (possibly
expensi e) p o ec ion mechanism, e.g., con ol- low in eg i y. By way o example,
he us ed componen may be a c yp o lib a y ha wan s o isola e c yp og aphic
keys, an inlined e e ence moni o ha wan s o isola e sensi i e me a da a (such as
a ain map o jump ables), o i may be a managed language un ime ha wan s
o isola e om a buggy na i e lib a y. We use he le e
T
o deno e he us ed
componen and U o deno e he emaining un us ed applica ion.
The main p imi i e ERIM p o ides is memo y isola ion—i ese es a egion o
he add ess space accessible exclusi ely om he us ed componen
T
. This ese ed
egion is deno ed
MT
and i can be used by
T
o s o e sensi i e da a. The es o
he add ess space, deno ed
MU
, holds he applica ion’s egula heap and s ack and
is accessible om bo h
U
and
T
. ERIM p e en s
U
om ha ing di ec access o
MT
;
access o
MT
is enabled a omically wi h a con ol ans e o designa ed en y poin s
in
T
, and disabled when
T
e u ns con ol o
U
. Mo e p ecisely, ERIM en o ces he
ollowing in a ian s:
(1) While con ol is in U, access o MT emains disabled.
(2)
Access o
MT
is enabled a omically wi h a con ol ans e o a designa ed en y
poin in T and disabled when T ans e s con ol back o U.
The i s in a ian p o ides isola ion o
MT
om
U
, while he second in a ian
p e en s
U
om con using
T
in o accessing
MT
imp ope ly by jumping in o he
middle o
MT
’s code. Due o he second in a ian , ERIM does no need suppo
om a solu ion o con ol- low in eg i y o secu i y.
68
Con ol ans e s om
U
o
T
and back, wi h he co esponding enabling and
disabling o access o
MT
a e acili a ed by special sequences o ERIM-p o ided code,
dubbed call ga es. Call ga es a e implemen ed in a manne ha p e en s exploi a ion
o hei bina y code o ele a ing p i ileges.
A call ga e enables access o
MT
, execu es a speci ied en y poin o
T
, hen
disables access o
MT
when ans e ing con ol o he us ed componen and
disables access on he way back. A call ga e ans e s con ol only o a designa ed
en y poin in he us ed componen . This en y poin may be a unc ion (i he
us ed componen is a lib a y) o a speci ic sequence o ins uc ions (i he us ed
componen is inlined in o he applica ion).
By design, ERIM imposes negligible o e head on he execu ion o code wi hin
he un us ed applica ion and wi hin he us ed componen , and i s call ga es a e
e y as . Addi ionally, ERIM’s isola ion is s ong—i is de i ed di ec ly om a
ha dwa e secu i y ea u e and i is absolu e, no p obabilis ic (unlike add ess space
layou andomiza ion (ASLR)). Bo h, as swi ching and he s ong isola ion, make
ERIM sui able o p o ec sensi i e da a in high-pe o mance applica ions, e en hose
ha swi ch be ween he applica ion and he lib a y e y equen ly.
4.1.1 Th ea model
ERIM makes no assump ions abou he un us ed componen (
U
) o an applica ion.
U
may beha e a bi a ily and may con ain memo y co up ion and con ol- low
hijack ulne abili ies ha may be exploi ed du ing i s execu ion.
Howe e , ERIM assumes ha he us ed componen
T
’s bina y does no ha e
such ulne abili ies and does no comp omise sensi i e da a by calling back in o
U
while access o
MT
is enabled, h ough in o ma ion leaks, o by mapping execu able
pages wi h unsa e/exploi able occu ences o he WRPKRU ins uc ion.
69

The ha dwa e, he OS ke nel, and a small lib a y added by ERIM o each p ocess
ha uses ERIM a e us ed o be secu e. We also assume ha he ke nel en o ces
s anda d DEP—an execu able page mus no be simul aneously mapped wi h w i e
pe missions. ERIM elies on a lis o legi ima e en y poin s in o
T
p o ided ei he
by he p og amme o he compile , and his lis is assumed o be co ec (see
Sec ion 4.1.5). The OS’s dynamic p og am loade /linke is us ed o in oke ERIM’s
ini ializa ion unc ion be o e any o he code in a new p ocess.
Side-channel and owhamme a acks, and mic oachi ec u al leaks, al hough
impo an , a e beyond he scope o his wo k. Howe e , ERIM is compa ible wi h
exis ing de enses.
4.1.2 In el Memo y P o ec ion Keys (MPK)
To ealize i s goals, ERIM uses he ecen MPK ex ension o he x86 ISA [
64
].
MPK allows associa ing one o 16 p o ec ion keys wi h each memo y page, hus
pa i ioning he add ess space in o up o 16 domains. A pe -co e egis e , called
PKRU, de e mines he cu en access pe missions ( ead, w i e, nei he o bo h) on
each domain o he code unning on ha co e. Access checks agains he PKRU a e
implemen ed in ha dwa e and impose no o e head on p og am execu ion.
Changing access p i ileges equi es w i ing new pe missions o he PKRU egis e
wi h a use -mode ins uc ion, WRPKRU. This ins uc ion is ela i ely as (11–260
cycles on cu en In el CPUs), does no equi e a syscall, changes o page ables, a
TLB lush, o in e -co e synch oniza ion.
Since WRPKRU can be execu ed in use -mode, un us ed code can execu e i
a any poin o ele a e p i ileges and MPK canno p o ide any memo y secu i y
agains un us ed applica ion code by i sel . To ge his p o ec ion, ERIM combines
MPK wi h addi ional bina y inspec ion o ensu e ha any WRPKRU occu ences
70
on execu able pages a e sa e, i.e., hey canno be exploi ed o imp ope ly ele a e
p i ilege.
The mains eam Linux ke nel ully suppo s page- able en ies agged wi h MPK
domains, syscalls o ag he en ies wi h speci ic domains and es o es PKRU egis e s
upon con ex swi ches. Since ha dwa e PKRU checks a e disabled in ke nel mode, he
ke nel has also been modi ied o check PKRU pe missions explici ly be o e accessing
any use space poin e . To elimina e he isk o signal handle s ele a ing p i ileges,
he ke nel upda es he PKRU egis e o i s ini ial se o p i ileges (only ead/w i e
access o domain 0) be o e h owing a signal o he use space.
4.1.3 High-le el o e iew o he design
ERIM can be con igu ed o p o ide ei he comple e isola ion o
MT
om
U
(con-
iden iali y and in eg i y), o only w i e p o ec ion (only in eg i y). Fo simplici y,
we desc ibe he design o comple e isola ion i s . Sec ion 4.1.7 desc ibes how o
con igu e ERIM sligh ly di e en ly o p o ide w i e p o ec ion only.
ERIM’s isola ion mechanism is concep ually simple: I maps
T
’s ese ed memo y,
MT
, and he applica ion’s gene al memo y,
MU
, o wo di e en MPK domains.
I manages MPK pe missions ( he pe -co e PKRU egis e s) o ensu e ha
MU
is
always accessible, while
MT
is ne e accessible when con ol is in
U
. I allows
U
o
secu ely ans e con ol o
T
and back ia call ga es. A call ga e enables access o
MT
using he WRPKRU ins uc ion and immedia ely ans e s con ol o a speci ied
en y poin o
T
, which may be an explici o inlined unc ion. When
T
is done
execu ing, he call ga e disables access o
MT
and e u ns con ol o
U
. This en o ces
ERIM’s wo in a ian s (1) and (2) om Sec ion 4.1. Call ga es ope a e en i ely in
use -mode ( hey don’ use syscalls) and a e desc ibed in Sec ion 4.1.4.
P e en ing WRPKRU exploi a ion
A key di icul y in ERIM’s design is p e-
en ing he un us ed
U
om exploi ing WRPKRU ins uc ions on execu able pages
71
in he add ess space o ele a e p i ileges, e.g. using con ol- low hijack o code-
injec ion a acks. To p e en such exploi s, ERIM elies on bina y inspec ion o
en o ce he in a ian ha only sa e WRPKRU occu ences appea on execu able
pages. A WRPKRU occu ence is sa e i i is immedia ely ollowed by one o he
ollowing:
(A) A p e-designa ed en y poin o T.
(B)
A speci ic sequence o ins uc ions ha checks ha he pe missions se by he
WRPKRU do no include access o
MT
and e mina es he p og am o he wise.
A sa e WRPKRU occu ence canno be exploi ed o execu e un us ed code wi h
access o
MT
. I he occu ence sa is ies (A), hen i does no gi e con ol o
U
a all;
ins ead, i en e s
T
a a designa ed en y poin . I he occu ence sa is ies (B), hen
i would e mina e he p og am immedia ely we e i used by a con ol- low hijack o
enable access o MT.
ERIM’s call ga es use only sa e WRPKRU occu ences and, he e o e, pass
ou bina y inspec ion. Ou modi ied ke nel inspec s any page p io o mapping
i in execu able mode, en o cing he in a ian ha all occu ences o WRPKRU
on execu able pages a e sa e. Sec ion 4.1.5 p o ides de ails o his ke nel bina y
inspec ion mechanism.
C ea ing sa e bina ies
An impo an ques ion is how o cons uc bina ies ha
do no ha e unsa e WRPKRUs. On x86, an inad e en o unin ended execu able
WRPKRU may a ise spanning he by es o wo adjacen ins uc ions o as a subse-
quence in a longe ins uc ion. To elimina e inad e en WRPKRUs, we de elop a
bina y ew i ing mechanism ha ew i es any sequence o ins uc ions con aining an
inad e en WRPKRU o a unc ionally equi alen sequence wi hou any WRPKRUs.
Simila ly, he mechanism also al e s delibe a e uses o WRPKRU which olun a ily
swi ch domains by inse ing p i ilege checks. The mechanism can be deployed as
72
1xo ecx , ecx
2xo edx , edx
3mo PKRU_ALLOW_TRUSTED, eax
4WRPKRU // cop ies eax o PKRU
6// Execu e us ed componen ’ s code
8xo ecx , ecx
9xo edx , edx
10mo PKRU_DISALLOW_TRUSTED, eax
11WRPKRU // cop ies eax o PKRU
12cmp PKRU_DISALLOW_TRUSTED, eax
13j e con inue
14s y s c a l l exi // e mina e p og am
15con inue :
16// co n ol e u ns o he un us ed a pplic a io n he e
Lis ing 4.1:
Call ga e implemen a ion in assembly. The code o he us ed
componen ’s en y poin may be inlined by he compile on line 6, o he e may be
an explici di ec call o i .
a compile pass, in eg a ed wi h ou bina y inspec ion, o by s a ically ew i ing
bina ies p io o hei use as explained in Sec ion 4.2
4.1.4 Call ga es
A call ga e ans e s con ol om
U
o
T
, enabling access o
MT
, hen uns code
om a designa ed en y poin o
T
, and la e e u ns con ol o
U
a e disabling
access o
MT
. This equi es wo WRPKRUs. The p ima y challenge in designing
he call ga e is ensu ing ha bo h hese WRPKRUs a e sa e in he sense explained
in Sec ion 4.1.3.
Lis ing 4.1 shows he assembly code o a call ga e. WRPKRU expec s he new
PKRU alue in he
eax
egis e and equi es ecx and edx o be 0. The call ga e
wo ks as ollows. Fi s , i se s PKRU o enable access o
MT
(lines 1–4). The mac o
PKRU_ALLOW_TRUSTED is a PKRU se ing ha allows access o
MT
. Nex ,
he call ga e passes con ol o he designa ed en y poin o
T
(line 6). The en y
poin ’s code may be in oked ei he by a di ec call, o i may be inlined he e.
73
he p esen alue o he PKRU is no PKRU_ALLOW_TRUSTED, indica ing ha
he syscall does no o igina e om
T
. To gain access o es ic ed esou ces,
U
has
o in oke T, which can ac as a e e ence moni o .
4.2 Rew i ing inad e en WRPKRUs
Fo secu i y, ou bina y inspec ion (see Sec ion 4.1.5) equi es bina ies o ha e only
sa e WRPKRU occu ences. WRPKRUs emi ed pu pose ully by a compile can
be made sa e by changing he compile sligh ly o inse he check on lines 12–15
o Figu e 4.1 a e e e y po en ially unsa e WRPKRU. Inad e en WRPKRUs—
hose ha occu unin en ionally as pa s o longe x86 ins uc ions o spanning wo
consecu i e x86 ins uc ions—a e mo e in e es ing. In his Sec ion, we desc ibe
a ew i e s a egy o elimina e such WRPKRUs. The s a egy is comple e: Any
sequence o x86 ins uc ions con aining an inad e en WRPKRU can be ew i en
o a unc ionally equi alen sequence wi hou any WRPKRUs.
4.2.1 Rew i e s a egy
WRPKRU is a 3 by e ins uc ion, 0x0F01EF. WRPKRU sequences ha span wo
o mo e ins uc ions can be “b oken” by inse ing a 1 by e nop like 0x90 be ween
any wo consecu i e ins uc ions. 0x90 does no coincide wi h any indi idual by e
o WRPKRU (0x0F, 0x01 and 0xEF), so his inse ion canno gene a e a new
WRPKRU.
A WRPKRU sequence ha lies en i ely wi hin a longe ins uc ion can be
elimina ed by inding an equi alen sequence o ins uc ions. Doing so sys ema ically
equi es unde s anding x86 ins uc ion coding. An x86 ins uc ion consis s o :
(i) An opcode ield possibly wi h p e ix.
80

(ii)
A MOD R/M ield ha de e mines he add essing mode and includes he code
o a egis e ope and.
(iii) An op ional SIB ield ha speci ies egis e s o indi ec memo y add essing.
(i )
Op ional displacemen and/o immedia e ields which speci y cons an o se s
o memo y ope a ions and o he cons an ope ands.
Ou s a egy o ew i ing an ins uc ion con aining WRPKRU as a subsequence
depends on he ields wi h which he WRPKRU subsequence o e laps. Table 4.1
summa izes ou s a egy. I he WRPKRU sequence lies en i ely in he opcode ield,
hen he ins uc ion is WRPKRU. As explained ea lie , his case is handled by
adding a check (B) a e he ins uc ion o make i sa e.
I he sequence o e laps wi h he MOD R/M ield, we change he egis e code in
he MOD R/M ield, which elimina es he WRPKRU sequence. This change equi es
a ee egis e . I one exis s, we use i , else we ew i e o push an exis ing egis e o
he s ack, use i in he ins uc ion, and pop i back. (Lines 2 and 3 in Table 4.1.)
I he sequence o e laps wi h he displacemen o he immedia e ield, we change
he mode o he ins uc ion o use a egis e ins ead o a cons an . The cons an
is compu ed in he egis e be o e he ins uc ion (lines 4 and 6). I a ee egis e
is una ailable, we push and pop one. Two ins uc ion-speci ic op imiza ions a e
possible. I he ins uc ion is jump-like, hen he jump a ge can be eloca ed in he
bina y; his changes he displacemen in he ins uc ion, elimina ing he need o a
ee egis e (line 5). I he ins uc ion is an associa i e ope a ion such as addi ion,
hen he ope a ion can be pe o med in wo inc emen s wi hou an ex a egis e
(line 7).
We ne e ew i e he SIB ield. This does no a ec he comple eness o ou
echnique since any WRPKRU mus o e lap wi h a leas one non-SIB ield ( he SIB
ield is 1 by e long while WRPKRU is 3 by es long).
81
O e lap wi h Cases Rew i e s a egy ID Example
Opcode Opcode = WRPKRU Inse p i ilege check a e WRPKRU 1
Mod R/M Mod R/M = 0x0F
Change o unused egis e + mo e com-
mand
2
add ecx, [ebx + 0x01EF0000]
→
mo eax, ebx; add ecx, [eax +
0x01EF0000];
Push/Pop used egis e + mo e com-
mand
3
add ecx, [ebx + 0x01EF0000]
→
push eax; mo eax, ebx; add ecx,
[eax + 0x01EF0000]; pop eax;
Displacemen Full/Pa ial sequence Change mode o use egis e 4
add eax, 0x0F01EF00
→
(push ebx;)
mo ebx, 0x0F010000; add ebx,
0x0000EA00; add eax, ebx; (pop
ebx;)
Jump-like ins uc ion
Mo e code segmen o al e cons an used
in add ess
5
call [ ip + 0x e 010 ]
→
call [ ip +
0x e 0100]
Immedia e Full/Pa ial sequence Change mode o use egis e 6
add eax, 0x0F01EF
→
(push ebx;)
mo ebx, 0x0F01EE00; add ebx,
0x00000100; add eax, ebx; (pop ebx;)
Associa i e opcode
Apply ins uc ion wice wi h di e en im-
media es o ge equi alen e ec
7
add ebx, 0x0F01EF00
→
add ebx,
0x0E01EF00; add ebx, 0x01000000
Table 4.1: Rew i e s a egy o in a-ins uc ion occu ences o WRPKRU
82
4.2.2 Implemen ing he ew i ing
Fo bina ies ha can be ( e)compiled om sou ce, ew i ing can be added o he
codegen phase o he compile , which con e s he in e media e ep esen a ion (IR)
o machine ins uc ions. Whene e codegen ou pu s an inad e en WRPKRU, he
su ounding ins uc ions in he IR can be eplaced wi h equi alen WRPKRU- ee
ins uc ions as desc ibed abo e, and codegen can be un again on he upda ed IR.
Fo bina ies ha canno be ecompiled, he ew i e s a egy can be in eg a ed
wi h ou bina y inspec ion handle (Sec ion 4.1.5). I he handle disco e s an unsa e
WRPKRU on an execu able page du ing i s scan, i can o e w i e he page wi h
1-by e ap ins uc ions, make i execu able, and s o e he o iginal page in ese e
wi hou enabling i o execu ion. Subsequen ly, i he e is a jump in o he execu able
page, a ap occu s and he ap handle disco e s an en y poin in o he page. I can
hen disassemble he ese ed page om ha en y poin on, ew i ing any disco e ed
WRPKRU occu ences, and copy he WRPKRU- ee ins uc ion sequences back o
he execu able page. To p e en o he h eads om execu ing pa ially o e w i en
ins uc ion sequences, we ac ually ew i e a esh copy o he execu able page wi h
he WRPKRU- ee sequences, and hen swap his ew i en copy o he execu able
page. This echnique is anspa en o he applica ion, has an o e head p opo ional
o he numbe o en y poin s in o o ending pages (we disassemble om e e y en y
poin only once) and main ains he in a ian ha only sa e WRPKRU sequences a e
execu able.
In con as o ew i ing a un ime, a bina y can be s a ically ew i en o emo e
all inad e en WRPKRUs. Compa ed o a compile o un ime app oach, s a ic
bina y ew i ing does no ely on sou ce code a ailabili y and does no imposes
addi ional un ime o e head. I s d awback is he dependence on a s a ic ew i e ool
which can success ully ew i e a bina y. In o de o success ully ew i e a bina y,
83
ools like Dynins [
34
] equi e a ull disassembly o he bina y. Recen ly Bauman e
al. [16] ha e p oposed a s a ic ew i e echnique emo ing his dependency.
Ou s a ic ew i e app oach, simila o he bina y inspec ion, pe o ms a simple
linea scan o he bina y o ind all hose inad e en occu ences o he 3-by e
WRPKRU sequence in execu able sec ions. Nex , using any bina y ew i ing ool, e.g.,
Dynins [
34
], we disassemble he bina y o he ex en possible, and ew i e ins uc ions
o elimina e hese inad e en occu ences. We use he p e iously desc ibed ew i ing
s a egy (see Sec ion 4.2.1 o able 4.1). Occu ences o WRPKRU in pa s ha we
canno disassemble a e handled by he bina y inspec ion and ew i ing a un ime as
desc ibed in he p e ious Sec ion.
We e alua e he e ec i eness o s a ically ew i ing bina ies in Sec ion 4.5.1.4.
4.3 Use Cases
ERIM di e s om p io wo k by p o iding e icien isola ion in applica ions whe e
swi ches be ween us ed and un us ed componen s a e e y equen , o he o de
o 10
5
o 10
6
imes a second. We desc ibe h ee such use-cases he e, and show in
Sec ion 4.5 ha ERIM’s o e head is low on all o hem.
4.3.1 Isola ing c yp og aphic keys in web se e s
Isola ing long- e m SSL keys o p o ec om web se e ulne abili ies such as he
Hea bleed bug [
90
] is well-s udied [
74
,
75
]. Howe e , long- e m keys a e accessed
ela i ely in equen ly (only a ew imes pe use session). Session keys ha a e
accessed a mo e equen ly (up o 10
6
imes a second pe co e in a high h oughpu
web se e like nginx) ha e no been isola ed so a . Isola ing sessions keys is also
ele an as hese keys p o ec he con iden iali y o indi idual use s. No exis ing
echnique can isola e session keys wi hou signi ican o e head.
84
Taking ERIM’s e icien isola ion in o accoun , an ERIM-p o ec ed componen
can isola e he c yp og aphic keys and c yp og aphic me hods. This esul s in a small
TCB and a ack su ace. OpenSSL does no implemen isola ion wi hin he lib a y,
hence we pa i ioned OpenSSL’s low-le el c yp o lib a y (libc yp o) o isola e he
session keys and basic c yp o ou ines, which un as
T
, om he es o he web se e ,
which uns as
U
. The ou e laye o OpenSSL p o ides he high-le el SSL/TLS
in e ace, whe eas he inne , isola ed laye secu ely s o es he c yp og aphic keys
and pe o ms c yp og aphic ope a ions. When using his ERIM-p o ec ed OpenSSL
wi hin a se e applica ion, a new SSL/TLS session c ea es a session key wi hin he
T
. Messages o his session can only be en-/dec yp ed wi hin he
T
. This e icien ly
p o ec s he c yp og aphic keys o se e applica ions om memo y ulne abili ies.
4.3.2 CPI/CPS
Code-poin e in eg i y (CPI) [
72
] is a compile ans o m ha p e en s con ol- low
hijacks by isola ing sensi i e objec s—code poin e s and objec s ha can lead o
code poin e s—in a sa e egion ha canno be w i en wi hou bounds checks. CPS
is a ligh e , less-secu e a ian o CPI ha isola es only code poin e s. Swi ching
a es o he sa e egion can be e y high in CPI, o he o de o 10
6
swi ches pe
second on s anda d benchma ks. A key ques ion in CPI/CPS is how o isola e he
sa e egion. The o iginal pape uses ASLR on x86-64 o i s e alua ion. ASLR
has almos no un ime o e head, bu i is now known o be ine ec i e o da a
isola ion [113, 60, 39, 49, 94].
We show ha ERIM can p o ide s ong isola ion o he sa e egion a low cos .
To do his, we o e ide he CPI/CPS-enabled compile ’s in insic unc ion o w i ing
he sensi i e egion o use a call ga e a ound an inlined sequence o
T
code ha
pe o ms a bounds check be o e he w i e. (MemSen y [
68
] also p oposes he use
85

o MPKs o isola ing he sa e egion, bu does no ac ually build o e alua e his
use-case.)
4.3.3 Na i e lib a ies in managed un imes
Applica ions unning on managed un imes such as a Ja a o Ja aSc ip VM o en
ely on hi d-pa y na i e code lib a ies. A ele an secu i y goal is o isola e he
managed un ime om bugs and ulne abili ies in he na i e lib a ies. ERIM can
be used o his pu pose by mapping he managed un ime o
T
and he na i e
lib a y(ies) o
U
. We es his by isola ing a na i e SQLi e plugin om Node.js.
SQLi e and Node.js a e, espec i ely, a s a e-o - he-a C da abase lib a y and a
s a e-o - he-a managed un ime o Ja aSc ip [121, 91].
4.4 Implemen a ion
We ha e implemen ed a p o o ype o ERIM on Linux. The p o o ype includes a 77
line Linux Secu i y Module (LSM) ha in e cep s all mmap and mp o ec calls o
p e en
U
om mapping pages in execu able mode, and p e en s
U
om o e iding
he bina y inspec ion handle . We also added 26 LoC in ke nel hooks needed o
his module. Ou implemen a ion also includes he ERIM un ime lib a y, which
p o ides a memo y alloca o o e
MT
, call ga es, he ERIM ini ializa ion code, and
bina y inspec ion. These comp ise 569 LoC.
Sepa a ely, we ha e implemen ed he ew i ing logic o elimina e inad e en
WRPKRU occu ences (abou 2250 LoC). While we ha e no ye in eg a ed he
logic in o ei he a compile o ou inspec ion handle , we ha e in eg a ed i in o a
s andalone bina y ew i ing ool ha uses Dynins [
34
] o disassemble bina ies. The
bina ies used in ou e alua ion do no ha e any unsa e WRPKRU occu ences and
do no load any lib a ies a un ime.
86
Call ype Cos (cycles)
Inlined call (no swi ch) 5
Di ec call (no swi ch) 8
Indi ec call (no swi ch) 19
Inlined call + swi ch 60
Di ec call + swi ch 69
Indi ec call + swi ch 99
ge pid sys em call 152
lwC swi ch [74] (Skylake CPU) 6050
Table 4.2: Cycle coun s o basic call and e u n
4.5 E alua ion
We e alua e ERIM on mic obenchma ks and on he h ee applica ions men ioned
in Sec ion 4.3. We pe o m ou expe imen s on Dell Powe Edge R640 machines
wi h 16-co e MPK-enabled In el Xeon Gold 6142 2.6GHz CPUs (wi h Tu bo Boos
and SpeedS ep disabled), 384GB memo y, 10Gbps E he ne links, unning Debian
8. Fo he CPI expe imen , we use he Le ee p o o ype 0.2 a ailable om
h p:
//dslab.ep l.ch/p oj/cpi/
and Clang 3.3.1 including i s CPI compile pass,
un ime lib a y ex ensions and link- ime op imiza ion. Fo he nginx expe imen , we
use nginx 1.12.1 and OpenSSL 1.1.1 and he ECDHE-RSA-AES128-GCM-SHA256
ciphe . Fo he managed language un ime expe imen , we use Node.js 9.11.1 and
SQLi e 3.22.0. Fo a compa ison base line we use SQLi e compiled o WebAssembly
ia emsc ip en 1.37.37’s WebAssembly backend [36].
4.5.1 Mic obenchma ks
4.5.1.1 Swi ch cos
We pe o med a mic obenchma k o measu e he o e head o in oking a unc ion
wi h and wi hou a swi ch o a us ed componen . The unc ion adds a cons an o
an in ege a gumen and e u ns he esul . Table 4.2 shows he cos o in oking
87
he unc ion, in cycles, as an inlined unc ion (I), as a di ec ly called unc ion (DC),
and as a unc ion called ia a unc ion poin e (FP). Fo e e ence, he able also
includes he cos o a simple syscall (ge pid) and he cos o a swi ch on lwCs, a
ecen in-p ocess isola ion mechanism based on s anda d page able p o ec ions [
74
].
In ou mic obenchma k, calls wi h an ERIM swi ch a e be ween 55 and 80 cycles
mo e expensi e han hei no-swi ch coun e pa s. The mos expensi e indi ec call
cos s less han he simples sys em call (ge pid). ERIM swi ches a e up o 100x
as e han lwC swi ches.
Because he CPU mus no eo de loads and s o es wi h espec o a WRPKRU
ins uc ion, he o e head o an ERIM swi ch depends on he CPU pipeline s a e
a he ime o he WRPKRUs in he swi ch. In expe imen s desc ibed la e in his
Sec ion, we obse ed a e age o e heads anging om 11 o 260 cycles pe swi ch. A
a clock a e o 2.6GHz, his co esponds o o e heads be ween 0.04% and 1.0% o
100,000 swi ches pe second, which is signi ican ly lowe han he o e head o any
bounds-check, ke nel- o hype iso -based isola ion.
4.5.1.2 Emula ing MPK’s swi ch cos
Following we desc ibe how o emula e he WRPKRU ins uc ion. This enables
us o compa e agains echniques who’s en i onmen does no suppo MPK. The
WRPKRU ins uc ion mo es he alue o he eax egis e o he PKRU egis e .
Howe e , since he ins uc ion impac s he alidi y o subsequen loads/s o es, he
ins uc ion canno be e-o de ed ela i e o su ounding load/s o e ins uc ions in
he execu ion pipeline. We emula e he cos o WRPKRU using a sequence o
xo ins uc ions ha ha e no ne unc ional e ec (excep consuming CPU cycles),
ollowed by RDTSCP, which causes a pipeline s all and p e en s ins uc ion e-
o de ing. The emula ion code is shown in Lis ing 4.2.
88
o ( i = 0; i < 5; i++) {
xo eax , ecx
xo ecx , eax
xo eax , ecx
}
d scp
Lis ing 4.2: WRPKRU emula ion using RDTSCP and Xo Swi ch
Benchma k Swi ches/sec CPI O e head (%)
ERIM EMUL
403.gcc 13,454,647 22.3 22.68
445.gobmk 1,055,994 1.77 1.76
447.dealII 1,270,582 0.56 0.17
450.soplex 408,192 0.6 2.56
464.h264 e 1,684,572 1.22 0.86
471.omne pp 36,578,718 144.02 142.26
482.sphinx 1,148,883 0.84 0.65
483.xalancbmk 21,448,977 52.22 51.74
Table 4.3:
Domain swi ch a es o selec ed SPEC CPU benchma ks and o e heads
o ERIM-CPI and EMUL-CPI, ela i e o s anda d CPI.
Valida ion
To alida e ha ou emula ion es ima es o e heads close o hose o he
ac ual WRPKRU ins uc ion, we e- un he CPI/CPS benchma ks o Sec ion 4.5.2
wi h WRPKRU emula ion in place o he ac ual WRPKRU ins uc ion. Figu e 4.1
ep oduces ERIM’s ela i e o e heads on a ious benchma ks om Figu e 4.2 bu
addi ionally lis s he ela i e o e heads using he WRPKRU emula ion (lines EMUL-
CPI and EMUL-CPS). Table 4.3 lis s he p ecise o e heads o CPI on benchma ks
ha ha e high swi ching a es. As can be seen, he o e heads o he emula ion a e
e y close o ac ual ERIM’s o e heads on all benchma ks.
No e om Table 4.3 ha ou emula ion is no pe ec , bu qui e close o he
ac ual in e ms o o e head. Emula ing he pe o mance o WRPKRU pe ec ly is
di icul since emula ion canno exac ly ep oduce he e ec s o WRPKRU on he
execu ion pipeline. (WRPKRU mus p e en he eo de ing o loads and s o es wi h
espec o i sel .) Depending on he speci ic benchma k, ou emula ion sligh ly o e -
89
Benchma k Swi ches/sec ERIM-CPI o e head
ela i e o o ig. CPI in %
403.gcc 16,454,595 22.30%
445.gobmk 1,074,716 1.77%
447.dealII 1,277,645 0.56%
450.soplex 410,649 0.60%
464.h264 e 1,705,131 1.22%
471.omne pp 89,260,024 144.02%
482.sphinx3 1,158,495 0.84%
483.xalancbmk 32,650,497 52.22%
Table 4.5:
Domain swi ch a es o selec ed SPEC CPU benchma ks and o e heads
o ERIM-CPI wi hou bina y inspec ion, ela i e o he o iginal CPI wi h ASLR.
Table 4.5 also shows he o e head o ERIM-CPI excluding bina y inspec ion,
ela i e o he o iginal CPI o e ASLR (no ela i e o an unp o ec ed baseline
as in Figu e 4.2). This ela i e o e head is exac ly he cos o ERIM’s swi ching.
Depending on he benchma k, i a ies om 0.03% o 0.16% o 100,000 swi ches
pe second o , equi alen ly, 7.8 o 41.6 cycles pe swi ch. These esul s indica e ha
ERIM can suppo inlined e e ence moni o s wi h swi ching a es o up o 10
6
imes
a second wi h low o e head. Beyond his a e, he o e head becomes no iceable.
4.5.2.2 CPS
The esul s o CPS a e simila o hose o CPI, bu he o e heads a e gene ally
lowe . Rela i e o anilla SPEC wi h no p o ec ion, he geome ic means o he
o e heads o he o iginal CPS and ERIM-CPS ac oss all benchma ks a e 1.1% and
2.4%, espec i ely. ERIM-CPS o e head ela i e o he o iginal CPS is wi hin 2.5%
on all benchma ks, excep excep pe lbench, omne pp and xalancbmk, whe e i
anges up o 17.9%.
4.5.3 P o ec ing session keys in nginx
Nex , we use ERIM o isola e SSL session keys in a high pe o mance web se e ,
nginx. We modi ied OpenSSL’s libc yp o o isola e he keys and he unc ions o
96

File
size
(KB)
Th oughpu Swi ches/s CPU
load
na i e
(%)
Na i e
( e-
q/s)
ERIM
el.
(%)
0
95,761
95.83 1,342,605 100.0
1
87,022
95.18 1,220,266 100.0
2
82,137
95.44 1,151,877 100.0
4
76,562
95.25 1,073,843 100.0
8
67,855
95.98 974,780 100.0
16
45,483
97.10 820,534 100.0
32
32,381
97.31 779,141 100.0
64
17,827
100.00 679,371 96.7
128 8,937 99.99 556,152 86.4
Table 4.6:
Nginx h oughpu wi h a single wo ke . The s anda d de ia ion is below
1.1% in all cases.
AES key alloca ion and enc yp ion/dec yp ion in o ERIM’s
T
and use ERIM call
ga es o in oke hese unc ions.
Ou goal is o measu e ERIM’s o e head on he peak h oughpu o nginx. To
s a , we con igu e nginx o un a single wo ke pinned o a CPU co e, and connec
o i emo ely om 4 concu en ApacheBench (
ab
) [
8
] ins ances o e HTTPS wi h
keep-ali e. Each ins ance simula es 75 concu en clien s. The clien s all eques
he same ile, whose size we a y om 0 o 128KB ac oss expe imen s. Figu e 4.3b
shows he h oughpu o ERIM-p o ec ed nginx ela i e o ou baseline (na i e
nginx wi hou any p o ec ion) o di e en eques sizes, measu ed a e an ini ial
wa m-up pe iod. Figu e 4.3a shows he absolu e h oughpu s in eques s/s in he
same expe imen . All numbe s a e a e ages o 10 uns.
ERIM-p o ec ed nginx p o ides a h oughpu wi hin 95.18% o he unp o ec ed
se e o all eques sizes. To explain he o e head u he , we lis he numbe o
ERIM swi ches pe second in he nginx wo ke and he wo ke ’s CPU u iliza ion
in Table 4.6 o eques sizes up o 128KB. The o e head shows a gene al end up
o eques s o size 32 KB: The wo ke ’s co e emains sa u a ed bu as he eques
size inc eases, he numbe o ERIM swi ches pe second dec ease, and so does
97
0
20000
40000
60000
80000
100000
0kb
1kb
2kb
4kb
8kb
16kb
32kb
64kb
128kb
Reques s/s
File size
Na i e
ERIM
(a) A e age numbe o eques s pe second o na i e and ERIM.
0
0.2
0.4
0.6
0.8
1
0kb
1kb
2kb
4kb
8kb
16kb
32kb
64kb
128kb
No malized Th oughpu
File size
Na i e ERIM
(b) No malized h oughpu o na i e (no p o ec ion).
Figu e 4.3:
Nginx h oughpu wi h one wo ke , wi h and wi hou ERIM p o ec ion,
wi h a ying eques sizes. S anda d de ia ions we e all below 1.1%.
98
ERIM’s ela i e o e head. The obse a ions a e consis en wi h an o e head o abou
0.31%–0.44% o 100,000 swi ches pe second. Fo eques sizes o 64KB and highe ,
he 10Gbps ne wo k ca d sa u a es and he wo ke does no u ilize i s CPU co e
comple ely in he baseline. The ee CPU cycles abso b ERIM’s CPU o e head, so
ERIM’s h oughpu ma ches ha o he baseline.
No e ha his is an ex eme es case o a web se e . He e, he web se e does
almos no hing and se es he same cached ile epea edly. To ge a mo e ealis ic
assessmen , we se up nginx o se e om a 571 MB co pus o 15,520 s a ic HTML
Wikipedia pages snapsho ed in 2006 [
137
]. File sizes a y om 417 by es o 522
KB (a e age size 37.7 KB). 75 keep-ali e clien s eques andom pages (selec ed
based on page iews on Wikipedia [
138
]). The a e age h oughpu wi h a single nginx
wo ke was 22,415 eques s/s in he base line and 21,802 eques s/s wi h ERIM (s d.
de s. below 0.6% in bo h cases). On a e age, he e we e 615,000 swi ches a second.
This co esponds o a o al o e head o 2.7%, o abou 0.43% o 100,000 swi ches a
second.
4.5.3.1 Scaling wi h mul iple wo ke s
To e i y ha ERIM scales wi h co e pa allelism, we e- an he i s expe imen abo e
wi h 3, 5 and 10 nginx wo ke s pinned o sepa a e co es, and su icien numbe s o
concu en clien s o sa u a e all he wo ke s. Table 4.7 shows he ela i e o e heads
wi h di e en numbe o wo ke s. Fo eques s la ge han hose shown in he able,
he ne wo k ca d sa u a es, and he spa e CPU cycles in he na i e base line abso b
ERIM’s o e head comple ely. Fo compa ison, he second and hi d columns o he
able epea he numbe s o he 1 wo ke con igu a ion o Table 4.6.
In he baseline, nginx’s h oughpu scales qui e well wi h he numbe o wo ke s.
Impo an ly, he ela i e o e head o ERIM’s p o ec ion does no inc ease wi h he
numbe o co es. Thus, ERIM scales wi h mul i-co e pa allelism indica ing ha
ERIM adds no addi ional synch oniza ion and scales pe ec ly wi h co e pa allelism.
99
File
size
(KB)
1 wo ke 3 wo ke s 5 wo ke s 10 wo ke s
Na i e
( e-
q/s)
ERIM
el.
(%)
Na i e
( e-
q/s)
ERIM
el.
(%)
Na i e
( e-
q/s)
ERIM
el.
(%)
Na i e
( e-
q/s)
ERIM
el.
(%)
0
95,761
95.83
276,736
96.05
466,419
95.67
823,471
96.40
1
87,022
95.18
250,565
94.50
421,656
96.08
746,278
95.47
2
82,137
95.44
235,820
95.12
388,926
96.60
497,778
100.00
4
76,562
95.25
217,602
94.91
263,719
100.00
8
67,855
95.98
142,680
100.00
Table 4.7:
Nginx h oughpu wi h mul iple wo ke s. The s anda d de ia ion is
below 1.5% in all cases.
This is unsu p ising gi en ha upda es o he PKRU o a co e a ec execu ion on
ha co e only.
4.5.3.2 Compa ison o ke nel-based isola ion
Using he single wo ke nginx expe imen , we compa e ERIM’s o e head o ha
o lwCs [
74
], a s a e-o - he-a sys em o in-p ocess isola ion based on s anda d
page- able p o ec ions. LwCs map each isola ed componen o a sepa a e add ess
space (in he same p ocess). A swi ch be ween componen s equi es ke nel media ion
o change page ables.
Since lwCs we e implemen ed only o F eeBSD, whose cu en ke nel suppo s
nei he MPKs no ou Xeon Gold machines, we un his compa ison expe imen
on an olde machine wi hou MPK suppo and use an emula ion o WRPKRU
o accoun o ERIM’s o e head as desc ibed in Sec ion 4.5.1.2. All expe imen s
desc ibed he e we e pe o med on Dell Op iPlex 7040 machines wi h 4-co e In el
Skylake i5-6500 CPUs clocked a 3.2GHz, 16GB memo y, 10 Gbps E he ne ca ds,
unning F eeBSD 11.
We use he exis ing single wo ke nginx expe imen (Sec ion 4.5.2) o compa e
he pe o mance o an ERIM-based isola ion o ha o lwC-based isola ion. As
opposed o ERIM swi ches, which ope a e en i ely in use space, lwC swi ches a e
syscalls. We c ea e a second ins ance o nginx ha uses an lwC (in place o an ERIM
100
componen ) o isola e session keys and basic c yp og aphic unc ions. We alloca e
da a bu e s in a memo y egion ha is sha ed be ween he p o ec ed lwC and he
web se e lwC o acili a e e icien da a sha ing. The o e head o WRPKRU is
emula ed as p e iously explained.
Figu e 4.4b depic s he o e head o he ERIM- and lwC-based a ian s ela i e
o he na i e baseline o an unmodi ied nginx, a e aged o e 20 uns. Nginx is
con igu ed o un one wo ke and se es 4 ApacheBench ins ances each simula ing
75 clien s accessing a s a ic ile ia HTTPS wi h keep-ali e.
The ERIM-based emula ion p o ides h oughpu wi hin 97.88% (wi hin 99%
o iles 64KB and la ge ) o he unp o ec ed na i e se e , whe eas he lwC-based
isola ion is limi ed o 50% o he na i e se e h oughpu o small iles and up o
80% o la ge (2MB) iles. The eason is he cos o lwC swi ch syscalls, which is oo
high gi en he a e o in oca ions o he enc yp ion unc ions.
Figu e 4.4a shows he absolu e numbe o se ed eques s pe second o he same
expe imen . The lwC-based nginx canno sus ain mo e han 26,500 eq/s, whe eas
ERIM pe o ms close o he na i e implemen a ion. A 64KB iles we sa u a e he
10Gbi ne wo k link esul ing in lowe eq/s o he na i e baseline and ERIM.
In summa y, we ind ha lwCs pe o m signi ican ly wo se han ERIM in his
expe imen : The h oughpu o nginx wi h lwC-based isola ion is ne e abo e 80%
o na i e nginx and, o small eques s, whe e he swi ch a e is highe , i is below
50% o na i e nginx. In con as , wi h ERIM’s isola ion, he h oughpu is wi hin
95% o na i e nginx in all con igu a ions. Hence, ERIM pe o ms signi ican ly be e
han ke nel-media ed isola ion a high swi ch a es.
4.5.4 Isola ing managed un imes
Nex , we es ERIM’s use o isola e a managed language un ime om an un us ed
na i e lib a y. Speci ically, we link he widely-used na i e da abase lib a y, SQLi e,
101

0
20000
40000
60000
80000
100000
0kb
1kb
2kb
4kb
8kb
16kb
32kb
64kb
128kb
256kb
512kb
1mb
2mb
Reques s/s
File size
Na i e
ERIM (emula ed)
LwC
(a) A e age numbe o eques s pe second, na i e, ERIM and lwC.
0
0.2
0.4
0.6
0.8
1
0kb
1kb
2kb
4kb
8kb
16kb
32kb
64kb
128kb
256kb
512kb
1mb
2mb
No malized Th oughpu
File size
Na i e
ERIM (emula ed)
LwC
(b) ERIM and lwC h oughpu no malized o na i e.
Figu e 4.4:
Nginx h oughpu wi h one wo ke , wi h emula ed ERIM p o ec ion
and wi h lwCs, wi h a ying eques sizes. S anda d de ia ions we e all below 1.1%.
102
o Node.js, a s a e-o - he-a Ja aSc ip un ime and use ERIM o isola e Node.js
om SQLi e by mapping Node.js’s un ime o
T
and he na i e lib a y o
U
. We
manually ins umen ed SQLi e’s en ypoin s o in oke call ga es. Addi ionally, since
we wan o isola e Node.js’s s ack om SQLi e, we un Node.js on a sepa a e s ack
in
MT
, and add code o swi ch o he s anda d s ack (in
MU
) p io o calling a
SQLi e unc ion. Finally, SQLi e uses he libc unc ion
memmo e
, which accesses libc
cons an s ha a e in
MT
, so we implemen ed a sepa a e
memmo e
o SQLi e. In
o al, we added 437 LoC.
We measu e un ime using he speed es 1 benchma k ha comes wi h SQLi e
and emula es a ypical da abase wo kload [
122
]. This benchma k pe o ms a o al o
32 sho es s ha s ess di e en da abase unc ions like selec s, joins, inse s and
dele es. We inc eased he i e a ions in each es by a ac o o ou o make he es s
longe . Ou base line o compa ison is anilla SQLi e linked o Node.js wi hou any
p o ec ion. We con igu e he benchma k o s o e he da abase in-memo y and epo
a e ages o 20 uns.
The geome ic mean o ERIM’s o e head on un ime ac oss all es s is 4.3%.
The o e head is below 6.7% on all es s excep hose wi h mo e han 10
6
swi ches
pe second. This sugges s ha ERIM can be used o isola ing na i e lib a ies om
managed language un imes wi h low o e heads up o a swi ching cos o he o de o
10
6
pe second. Beyond ha he o e head is no iceable. Table 4.8, columns 1–3, show
he ela i e o e heads o es s wi h swi ching a es o a leas 100,000/s. These a e
consis en wi h an a e age o e head be ween 0.07% and 0.41% o 100,000 swi ches/s.
The ac ual swi ch cos measu ed om di ec CPU cycle coun s a ies om 73 o 260
cycles ac oss all es s. The swi ch cos exceeds 100 cycles only on benchma ks whe e
he swi ch a e is e y low (less han 2,000 imes/s). We e i ied ha hese highe
cycle coun s a e due o ins uc ion cache misses—a e y low swi ch a es, he call
ga e is lushed ou o he ins uc ion cache be ween swi ches.
103
Tes
#Swi ches/s O e head (%)
ERIM WebAssembly
100 11,183,281 12.73% 132.48%
110 8,329,914 12.18% 135.44%
400 8,161,584 15.42% 156.04%
120 7,190,766 13.81% 145.19%
142 7,074,553 9.41% 165.88%
500 6,419,008 12.13% 119.15%
510 5,868,395 5.60% 113.76%
410 5,091,212 3.64% 122.77%
240 2,358,524 3.74% 126.63%
280 2,303,516 3.22% 100.05%
170 1,264,366 4.22% 104.87%
310 1,133,364 2.92% 81.71%
161 1,019,138 2.81% 138.64%
160 1,014,829 2.73% 136.27%
230 670,196 2.04% 193.42%
270 560,257 2.28% 92.78%
Table 4.8:
O e head ela i e o na i e execu ion o SQLi e speed es 1 es s wi h
mo e han 100,000 swi ches/s. S anda d de ia ions we e below 5.6% o na i e, and
ERIM and below 15.4% o WebAssembly.
4.5.4.1 Compa ison o isola ion wi h bounds checks (SFI)
We also use he abo e expe imen o compa e ERIM o isola ion based on bounds
checks. Fo his, we e-compile he SQLi e lib a y o na i e code indi ec ly h ough
WebAssembly, a new memo y-sa e, low-le el language designed speci ically o w i ing
sa e na i e plugins o Ja aSc ip en i onmen s [
53
]. The WebAssembly o na i e
code ansla ion inse s bounds checks p io o indi ec memo y accesses. Compila ion
ia WebAssembly is he cu en ly ecommended me hod o sa ely adding na i e
plugins o Google’s Ch ome web b owse ; mos majo web b owse s a e expec ed o
ecommend he same me hod in he nea u u e.
Ac oss all 32 es s, he geome ic mean o he ela i e o e head o WebAssembly-
based isola ion on un ime is 133.5%. The o e heads ange om 66.4% o 280.6%,
which is signi ican ly highe han ERIM’s o e heads. Howe e , WebAssembly’s
o e heads do no inc ease wi h he swi ching a e since i does no in e pose on
104
swi ches. Ins ead, i imposes a con inuous o e head while execu ion is in SQLi e.
O he wo k using bounds checks has ound simila ly high o e heads on pe o mance-
in ensi e benchma ks [95, 53].
4.6 Rela ed Wo k
Re e ence moni o s [
6
] media e p i ileged access by un us ed applica ions p o ec ing
da a con iden iali y and in eg i y, by sandboxing he applica ion o checks inse ed
in o he applica ions execu ion. All implemen a ions sha e he impo an p ope y
o p o ec ing he e e ence moni o ’s code and s a e om co up ion and deploy
isola ion echniques shielding he e e ence moni o . Leas -p i ilege and p i ilege
sepa a ion [
105
] de ine he basis o oday’s e e ence moni o mechanisms in ha dwa e
([
82
,
63
,
12
]), hype iso s ([
14
,
19
]), ope a ing sys ems ([
132
,
35
]) o applica ions
([
129
,
80
,
41
,
143
,
29
]). Two ecen su eys [
126
,
117
] show iable a acks and possible
coun e measu es o p o ec agains da a con iden iali y and in eg i y iola ions.
Fu he mo e hei in e cep ion g anula i y a ies om a sepa a e gues OS ([
14
]),
a single applica ion ([
19
,
20
,
132
]) o applica ion componen s ([
129
,
82
,
74
,
80
,
41
,
143
,
29
,
144
]). In e cep ing a ine g anula i y o e s isola ion ac oss applica ion
componen s, whe eas cou se-g ain in e cep ion isola es independen applica ions o
componen s. Due o i s equen in oca ions, ine-g ained in e cep ion solu ions
equi e isola ion echniques wi h low o e head. ERIM isola es applica ion componen s
and in e cep s ine-g ained secu i y ele an e en s wi hin he applica ion, simila ly
o ARMlock [
144
] o SFI-based isola ion [
129
,
80
,
41
,
143
]. This is in con as o
echniques using OS p ocess bounda ies ([74]) o CPU p i ilege le els ([19, 14]).
Koning e al. [
68
] su ey echniques o e icien da a encapsula ion wi hin a
p ocess, including SFI, dynamic enc yp ion o p i a e da a using he In el AES-NI
ISA ex ensions, app oaches ha use VT-x i ualiza ion ha dwa e, and hose ha
ely on he In el MPX and MPK ISA ex ensions. I hen p esen s a gene al isola ion
105

CHAPTER 5
Conclusion
Today compu e s s o e and analyze aluable and sensi i e da a such as pe sonal
mul imedia o clien eco ds. An impo an goal is o p o ec he con iden iali y
and in eg i y o such da a, minimizing he isk o illici elease, loss o modi ica ion.
Howe e , exis ing echniques o p o ec con iden iali y and in eg i y a e ulne able
o malicious a acks o a e ine icien . This hesis con ibu es wo new echniques,
Gua da and ERIM, p o iding con iden iali y and in eg i y o pe sis en and in-
memo y da a secu ely and e icien ly.
Gua da en o ces, a he s o age laye , ich pe - ile con iden iali y and in eg i y
policies wi h low o e head. The en o cemen a he s o age laye educes he a ack
su ace and he isk o ci cum en ion due o so wa e bugs, miscon igu a ions and
ope a o e o s in highe laye s. Gua da o e comes he gap be ween s o age laye
en o cemen and pe - ile policies by a es ing he s a e o iles and associa ed policies
h ough c yp og aphically-signed ce i ica es. To speci y policies, we de elop a
domain-speci ic language which allows da a accesses condi ioned on au hen ica ion,
us ed wall clock ime, and a ile’s s a e including he con en . We demons a e an
e icien implemen a ion o en o ce such policies in an iSCSI SAN se e and apply
Gua da o wo use cases p o ec ing he con en , execu able, and log iles o a web
se e , as well as en o cing manda o y access logging.
ERIM p o ides da a con iden iali y and in eg i y o in-memo y da a by isola ing
sensi i e da a om accesses by un us ed componen s. I isola es sensi i e da a
113
in o a sepa a e, us ed memo y componen using In el MPK and ensu es ha
only he us ed componen has access o sensi i e da a. To p e en malicious
a acks om escala ing p i ileges using he unp i ileged WRPKRU CPU ins uc ion,
ERIM addi ionally p o ec s he us ed componen ia secu e con ol ans e s and
bina y inspec ion. Secu e con ol ans e s ensu e ha he un us ed componen
canno ele a e access pe missions wi hou he in ol emen o he us ed componen .
Bina y inspec ion gua an ees ha no execu able bina y code sequence ele a es access
pe missions o he us ed componen , while execu ing un us ed code. ERIM’s
isola ion imposes no addi ional o e head on he execu ion and less han 1% un ime
o e head pe 100,000 swi ches/second. Unlike s a e-o - he-a isola ion echniques,
he low swi ch cos and no o e head on execu ion allows ERIM o e icien ly isola e
equen ly-used session keys in web se e s, an in-memo y e e ence moni o ’s p i a e
s a e, and managed un imes om na i e lib a ies as demons a ed in he e alua ion.
5.1 Fu u e Wo k
Gua da and ERIM independen ly p o ec he con iden iali y and in eg i y o sensi i e
pe sis en and in-memo y da a. While Gua da es ic s da a accesses a he s o age
laye , i does no p o ec da a eleased o an applica ion. In con as ERIM es ic s
access o applica ion da a, bu does no p o ec pe sis en da a om malicious
a acks. Al hough beyond he goal o his hesis, in his sec ion we discuss how o
o e come he limi a ions o each echnique and a ain an end- o-end con iden iali y
and in eg i y gua an y o sensi i e pe sis en da a h oughou i s in-memo y use in
an applica ion.
We can o e come he limi a ions o each indi idual sys em by connec ing an
ERIM-isola ed moni o o a Gua da de ice which s o es he sec e s. These pe sis-
en ly s o ed sec e s on a Gua da de ice could be p o ec ed om a bi a y da a
accesses by associa ing a policy which allows da a access only by au ho ized connec-
114
ions. Howe e , wi hou any changes o ERIM, an ERIM-isola ed moni o would
no be able o access hese sec e s. To gain access, he moni o has o connec o
he Gua da de ice and au hen ica e i sel . The cu en design o ERIM does no
p o ide a way o gene a e a unique au ho iza ion sec e ha is eliable and consis en
ac oss en i onmen s and eboo s. In o de o gene a e au ho iza ion sec e s, exis ing
echniques like us ed pla o m modules (TPM) o In el SGX gene a e au ho iza ion
sec e s o code by measu ing he code’s in-memo y oo p in as a secu e hash. This
hash is hen used o de i e a unique au ho iza ion sec e .
Simila o exis ing echniques, we sugges o change ERIM’s ini ializa ion o
gene a e an au ho iza ion sec e by measu ing he us ed moni o ’s oo p in . Once
measu ed, he sec e is placed in he us ed moni o ’s memo y (ou side o he
un us ed applica ion’s each). Using his sec e , he us ed moni o au hen ica es
i sel o he Gua da de ice. This app oach gua an ees ha Gua da only eleases
sec e s o an ERIM-isola ed moni o , while ERIM p o ec s he in-memo y copy o
he sec e om accesses by an un us ed applica ion.
By connec ing an ERIM-isola ed moni o o Gua da we p o ide an end- o-end
con iden iali y and in eg i y gua an ee which would be pa icula ly in e es ing o
se e applica ions ha ely on a secu e connec ion o clien s using asymme ic
c yp og aphy. In oday’s se e and c yp og aphic lib a y implemen a ions (see
Figu e 5.1a) he p i a e and session keys a e no isola ed in memo y o p o ec ed in
pe sis en s o age. Exis ing se e applica ions ead a p i a e key om pe sis en
s o age in o memo y and use he key o es ablish a secu e connec ion by nego ia ing
a session key. The session key is s o ed in memo y and used by bo h pa ies o
enc yp and dec yp messages. Hence, secu i y ulne abili ies and bugs in he
un us ed se e applica ion, he ope a ing sys em, o applica ions wi h access o
he s o age may esul in a con iden iali y o in eg i y iola ion. Fo example,
unau ho ized applica ions may ead he pe sis en p i a e key and elease hem,
iola ing con iden iali y. Simila ly, he key could be modi ied on disk, iola ing
115
(a)
Today’s se e applica-
ions
(b)
Isola ing a c yp o-
g aphic lib a y using ERIM
and connec ing i o a Gua -
da de ice
(c)
Combining ERIMwi h
So wa e Gua d Ex ensions
(SGX) o isola e om he
ope a ing sys em
Figu e 5.1: S eps owa ds an isola ed c yp og aphic lib a y in se e applica ions
in eg i y. Once he keys eside in memo y, malicious a acks like Hea bleed [
90
] can
elease o modi y he keys, iola ing bo h con iden iali y and in eg i y.
To p o ec he keys om hese ypes o h ea s, Gua da in combina ion wi h
ERIM can p o ec he pe sis en keys and he in-memo y keys e icien ly as shown in
Figu e 5.1b. The se e needs o be spli in o a us ed moni o which only holds he
c yp og aphic unc ions and an un us ed se e which handles he communica ion
and p o ides he se ice o he clien . Du ing ini ializa ion ERIM isola es he us ed
moni o ’s memo y om he un us ed se e applica ion. ERIM measu es he us ed
moni o and p o ides he au ho iza ion sec e o he us ed moni o . I hen allows
he us ed moni o o ini ialize, connec o a Gua da de ice, au ho ize using he
p e iously measu ed sec e , and ead he p i a e key. Gua da checks ha he
au hen ica ed clien ac ually is he ERIM-isola ed us ed moni o which is speci ied
in he policy. A e inishing he us ed moni o ’s ini ializa ion, ERIM s a s he
un us ed se e , which begins i s usual ope a ion wai ing o clien s o connec .
Once a clien connec s o es ablish a secu e connec ion, he un us ed se e accep s
he connec ion, s a s he SSL/TLS handshake p o ocol and swi ches o he us ed
moni o whene e enc yp ing o dec yp ing messages using he in-memo y p i a e
key and gene a ing new session keys.
116
In con as o he h ea model o Gua da which assumes in e media e laye s,
such as he OS, o be ulne able o malicious a acks o ci cum en ion, he echnique
p oposed abo e, howe e , assumes he OS o be us ed, since ERIM’s gua an ees
depend on he OS. As a esul , he p oposed solu ion would only p o ec con iden iali y
and in eg i y agains a acks om ou side, e.g., malicious clien s a acking he se e
like Hea bleed [
90
], and any h ea s on he ne wo k be ween he machine unning
he se e and he Gua da de ice. Such a h ea model is common o se e s
unning in he cloud.
Fu he ha dening ERIM agains OS ulne abili ies:
While he cloud h ea
model is commonly assumed, ecen a acks [
24
] show how o he cloud enan s
can access in-memo y sec e s iola ing con iden iali y and in eg i y. In addi ion,
highly sensi i e applica ions may no assume he cloud p o ide o be us ed and,
hence, in e media e laye s like he OS and VMM which p o ide isola ion a e no
longe us ed. In o de o s eng hen he h ea model o he p esen ed echnique,
he memo y isola ion gua an ees o ERIM ha e o be independen o in e media e
so wa e laye s like he OS o VMM.
To de end agains hese h ea s, us ed execu ion en i onmen s (TEE), in
pa icula In el SGX, can be used o shield sensi i e da a om he cloud pla o m
and o he enan s. SGX p o ides in-memo y encla es o s o e sensi i e da a and
execu e code independen o he unning OS o VMM. Se e al esea ch sys ems
[
17
,
13
,
70
] demons a e he use o In el SGX o shield an applica ion agains he
cloud pla o m.
While SGX p o ides s ong memo y isola ion gua an ees, i s high swi ch cos s
compa able o a con ex swi ch hinde s i s adop ion, and p e en s i om being
used o isola e equen ly-used sec e s. Exis ing wo k o e comes hese pe o mance
limi a ions by ei he isola ing in equen ly-used sec e s like p i a e keys o isola ing
an en i e applica ion. Howe e , pushing en i e applica ions (e.g., a se e ) in o
117

an encla e wi hou u he memo y isola ion lea es he applica ion ulne able o
malicious a acks, due o he size and complexi y o hese applica ions. Recen
wo k [
71
] sugges s u he p o ec ing applica ions in SGX encla es by adding memo y
bound checks. Howe e , such checks incu subs an ial un ime o e head, and i is
no su icien o ERIM o simply swap In el MPK o In el SGX. Ins ead, we need
o combine bo h app oaches o allow en i e applica ions o un wi hin SGX encla es,
shielding hem om he emaining cloud so wa e s ack, while isola ing sec e s wi hin
he applica ion using a mechanism simila o In el MPK.
To his end, we sugges amending he SGX speci ica ion, since i has no p o ision
o MPK-like memo y isola ion using pe -page domains and an access pe mission
egis e such as he PKRU egis e . The encla e memo y desc ip o s eside in
p ocesso ese ed memo y which is inaccessible o sys em so wa e (e.g., OS o
VMM). An impo an desc ip o is he encla e page cache map (EPCM), a able-like
s uc u e, which holds in o ma ion abou which memo y pages belongs o an encla e
and holds pe -page access pe mission bi s. Cu en ly he EPCM allows pages o be
accessible wi h ead, w i e, and execu e pe mission and does no allow pages o be
agged wi h a MPK domain.
To allow page-le el memo y isola ion wi hin SGX encla es, we sugges adding
memo y domain iden i ie s o he EPCM and ex ending he CPU’s memo y access
pe mission check o alida e he cu en access pe missions in he PKRU egis e
agains he memo y access’s EPCM domain iden i ie . This app oach ex ends he
each o In el MPK in o SGX encla es. Simila o he use o MPK in ERIM, his
solu ion is ulne able o malicious a acks and hence needs o be combined wi h
ERIM’s secu e con ol ans e s and bina y inspec ion. Fo code in encla es, we can
simpli y ERIM’s bina y inspec ion, since encla e memo y is alloca ed once a he
s a o an encla e and can only be ex ended wi h a special p o ocol including a s ep
in which he encla e app o es he ex ension [
83
]. The bina y inspec ion could scan
once a he s a o unsa e WRPKRUs in all execu able memo y and a un ime i
118
could only app o e new pages which do no con ain unsa e WRPKRUs. The e is no
need o ke nel modi ica ions o signal handle s. By amending he SGX speci ica ion
and combining i wi h ERIM’s secu e con ol ans e s and bina y inspec ion, we
can isola e equen ly-used sec e s wi hin SGX encla e wi hou us ing he sys em
so wa e like he OS (see Figu e 5.1c).
In his sec ion we ha e shown how o ex end he p o ec ion o pe sis en iles om
Gua da o in-memo y da a using ERIM. We discussed he challenge in au hen ica ing
an ERIM-isola ed us ed moni o o a Gua da de ice and desc ibe a echnique
o gene a e an au hen ica ion sec e using code measu emen s. We discuss i s use
in a commonly assumed h ea model o cloud en i onmen s. Fo highly sensi i e
applica ions, we desc ibe an ex ension o In el’s SGX o p o ec agains ogue cloud
p o ide s and o he cloud enan s.
119
Bibliog aphy
[1]
Ma ín Abadi, Mihai Budiu, Úl a E lingsson, and Jay Liga i. Con ol- low
in eg i y. In P oceedings o ACM SIGSAC Con e ence on Compu e and
Communica ions Secu i y (CCS), 2005.
[2]
Ni in Ag awal, William J. Bolosky, John R. Douceu , and Jacob R. Lo ch. A
i e-yea s udy o ile-sys em me ada a. ACM T ansac ions on S o age, 3(3),
2007.
[3]
Ma cos K. Aguile a, Minwen Ji, Ma k Lillib idge, John MacCo mick, E win
Oe li, Da id G. Ande sen, Mike Bu ows, Timo hy Mann, and Chand amohan
Thekka h. Block-Le el Secu i y o Ne wo k-A ached Disks. In P oceedings
o USENIX Con e ence o File and S o age Technologies (FAST), 2003.
[4]
Hussain M. J. Almoh i and Da id E ans. Fidelius Cha m: Isola ing Unsa e
Rus Code. In P oceedings o ACM Con e ence on Da a and Applica ion
Secu i y and P i acy (CODASPY), 2018.
[5]
Amazon. Amazon Simple S o age Se ice (Amazon S3).
h p://aws.amazon.
com/s3/, 2011.
[6] James P. Ande son. Compu e Secu i y Technology Planning S udy (Volume
II), 1972.
[7]
Jason Ansel, Pe Ma chenko, Ul a E lingsson, Elijah Taylo , B ad Chen,
De ek L. Schu , Da id Seh , Cli L. Bi le, and Benne Yee. Language-
independen sandboxing o jus -in- ime compila ion and sel -modi ying code.
In P oceedings o ACM SIGPLAN con e ence on P og amming language design
and implemen a ion (PLDI), 2011.
120
[8]
Apache HTTP Se e P ojec .
h ps://h pd.apache.o g/docs/2.4/
p og ams/ab.h ml.
[9] Apple Inc. Time Machine, 2007.
[10] Apple Inc. Apple Fusion D i e. h ps://www.apple.com/de/imac, 2017.
[11]
ARM. De elope guide: ARM memo y domains.
h p://in ocen e .a m.
com/help/, 2001.
[12]
ARM. ARM Secu i y Technology.
h p://in ocen e .a m.com/
help/ opic/com.a m.doc.p d29-genc-009492c/PRD29-GENC-009492C_
us zone_secu i y_whi epape .pd , 2009.
[13]
Se gei A nau o , Bohdan T ach, F anz G ego , Thomas Knau h, And e Ma in,
Ch is ian P iebe, Joshua Lind, Di ya Mu hukuma an, Ma k L. S illwell, Da id
Gol zsche, Da id Eye s, Pe e Pie zuch, and Ch is o Fe ze . SCONE: Secu e
Linux Con aine s wi h In el SGX. In P oceedings o USENIX Symposium on
Ope a ing Sys ems Design and Implemen a ion (OSDI), 2016.
[14]
Paul Ba ham, Bo is D ago ic, Kei F ase , S e en Hand, Tim Ha is, Alex
Ho, Rol Neugebaue , Ian P a , and And ew Wa ield. Xen and he a o
i ualiza ion. ACM SIGOPS Ope a ing Sys ems Re iew, 37(5), 2003.
[15]
F i zge ald Ba h, Vic o Chin, Moshe Fe be , Sean Hi el, Lau ie Jame-
son, Na heniel Mason, Ha deep Meh o a a, Ashish Meh a, Mihi Mohan y,
K ishna Na ayanswamy, and Michael Roza. Top h ea s o cloud compu ing
plus indus y insigh s.
h ps://www.couldsecu i yaliance.o g/download/
op- h ea s- o-cloud-compu ing-plus-indus y-insigh s/, 2017.
[16]
E ick Bauman, Zhiqiang Lin, and Ke in W. Hamlen. Supe se disassembly:
S a ically ew i ing x86 bina ies wi hou heu is ics. In P oceedings o Ne wo k
and Dis ibu ed Sys ems Secu i y Symposium (NDSS), 2018.
121
[65]
The iSCSI En e p ise Ta ge P ojec .
h p://iscsi a ge .sou ce o ge.
ne /, 2011.
[66]
T. Jim. SD3: A us managemen sys em wi h ce i ied e alua ion. In
P oceedings o IEEE Compu e Socie y Symposium on Resea ch in Secu i y
and P i acy, 2001.
[67]
Douglas Kilpa ick. P i man: A Lib a y o Pa i ioning Applica ions. In
P oceedings o USENIX Anual Technical Con e ence (ATC), 2003.
[68]
Koen Koning, Xi Chen, He be Bos, C is iano Giu ida, and Elias A hana-
sopoulos. No Need o Hide: P o ec ing Sa e Regions on Commodi y Ha dwa e.
In P oceedings o ACM Eu opean Con e ence on Compu e Sys ems (Eu oSys),
2017.
[69]
Ramak ishna Ko la, Tom Rodehe e , Ind aji Roy, Pa ick S uedi, and Ben-
jamin Wes e . Pas u e: Secu e O line Da a Access using Commodi y T us ed
Ha dwa e. In P oceedings o USENIX Symposium on Ope a ing Sys ems Design
and Implemen a ion (OSDI), 2012.
[70]
Robe K ahn, Bohdan T ach, Anjo Vahldiek-Obe wagne , Thomas Knau h,
P amod Bha o ia, and Ch is o Fe ze . Pesos: Policy Enhanced Secu e Objec
s o e. In P oceedings o ACM Eu opean Con e ence on Compu e Sys ems
(Eu oSys), 2018.
[71]
Dmi ii Ku aiskii, Oleksii Oleksenko, Se gei A nau o , Bohdan T ach, P amod
Bha o ia, Pascal Felbe , and Ch is o Fe ze . SGXBOUNDS: Memo y Sa e y o
Shielded Execu ion. In P oceedings o ACM Eu opean Con e ence on Compu e
Sys ems (Eu oSys), 2017.
128

[72]
Volodymy Kuzne so , László Szeke es, and Ma hias Paye . Code-poin e
in eg i y. In P oceedings o USENIX Symposium on Ope a ing Sys ems Design
and Implemen a ion (OSDI), 2014.
[73]
Ninghui Li and John C. Mi chell. Da alog wi h Cons ain s: A Founda ion o
T us Managemen Languages. In P oceedings o ACM Symposium on P ac ical
Aspec s o Decla a i e Languages (PADL), 2003.
[74]
James Li on, Anjo Vahldiek-Obe wagne , Eslam Elnike y, Deepak Ga g,
Bobby Bha acha jee, and Pe e D uschel. Ligh -Weigh Con ex s: An OS
Abs ac ion o Sa e y and Pe o mance. In P oceedings o USENIX Symposium
on Ope a ing Sys ems Design and Implemen a ion (OSDI), 2016.
[75]
Yu ao Liu, Tianyu Zhou, Kexin Chen, Haibo Chen, and Yubin Xia. Thwa ing
Memo y Disclosu e wi h E icien Hype iso -en o ced In a-domain Isola ion.
In P oceedings o ACM SIGSAC Con e ence on Compu e and Communica ions
Secu i y (CCS), 2015.
[76]
Boon Thau Loo. The Design and Implemen a ion o Decla a i e Ne wo ks.
PhD hesis, Uni e si y o Cali o nia, Be keley, 2006.
[77]
Boon Thau Loo, Tyson Condie, Joseph M. Helle s ein, Pe os Mania is, Timo-
hy Roscoe, Ion S oica, Thau Loo, Pe os Mania is, Tyson Condie, Timo hy
Roscoe, and Joseph M. Helle s ein. Implemen ing decla a i e o e lays. In
ACM SIGOPS Ope a ing Sys ems Re iew, olume 39, 2005.
[78]
Kangjie Lu, Chengyu Song, Byoungyoung Lee, Simon P. Chung, Taesoo Kim,
and Wenke Lee. ASLR-Gua d: S opping Add ess Space Leakage o Code
Reuse A acks. In P oceedings o ACM SIGSAC Con e ence on Compu e and
Communica ions Secu i y (CCS), 2015.
129
[79]
Michelle L. Mazu ek, Yuan Liang, William Meliche , Manya Sleepe , Lujo
Baue , G ego y R. Gange , Ni in Gup a, and Michael K. Rei e . Towa d s ong,
usable access con ol o sha ed dis ibu ed da a. In P oceedings o USENIX
Con e ence on File and S o age Technologies (FAST), 2014.
[80]
S ephen Mccaman and G eg Mo ise . E alua ing SFI o a CISC A chi ec u e.
In P oceedings o USENIX Secu i y Symposium, 2006.
[81]
Jona han M. McCune, Yanlin Li, Ning Qu, Zongwei Zhou, Anupam Da a,
Vi gil Gligo , and Ad ian Pe ig. T us iso : E icien cb educ ion and
a es a ion. In P oceedings o IEEE Symposium on Secu i y and P i acy
(Oakland), 2010.
[82]
Jona han M. McCune, B yan J. Pa no, Ad ian Pe ig, Michael K. Rei e , and
Hi oshi Isozaki. Flicke : An Execu ion In as uc u e o TCB Minimiza ion.
In P oceedings o ACM Eu opean Con e ence on Compu e Sys ems (Eu oSys),
2008.
[83]
F ank McKeen, Ilya Alexand o ich, I ai Ana i, D o Caspi, Simon Johnson,
Rebekah Leslie-Hu d, and Ca los Rozas. In el so wa e gua d ex ensions
suppo o dynamic memo y managemen inside an encla e. In P oceedings o
Ha dwa e and A chi ec u al Suppo o Secu i y and P i acy, 2016.
[84]
Aas ha Meh a, Eslam Elnike y, Ka u a Ha ey, Deepak Ga g, and Pe e
D uschel. Qapla: Policy compliance o da abase-backed sys ems. In P oceedings
o USENIX Secu i y Symposium, 2017.
[85]
Michael Mesnie , Feng Chen, Tian Luo, and Jason B. Ake s. Di e en ia ed
s o age se ices. In P oceedings o ACM Symposium on Ope a ing Sys ems
P inciples (SOSP), 2011.
130
[86]
Mike Mesnie , G ego y R. Gange , and E ik Riedel. Objec -based s o age.
IEEE Communica ions Magazine, 41(8), 2003.
[87]
Mic oso Co p. Bi locke .
h ps://docs.mic oso .com/en-us/windows/
secu i y/in o ma ion-p o ec ion/bi locke /bi locke -o e iew.
[88]
Mic oso Co p. Windows Backup and Res o e.
h p://www.mic oso .com/
a home/se up/backupda a.aspx{#} bid=l7X90d97alI.
[89]
Mic oso Co p. Wha Is Volume Shadow Copy Se ice?: Da a Re-
co e y.
h ps:// echne .mic oso .com/en-us/lib a y/cc757854( =ws.
10).aspx, 2003.
[90]
MITRE. CVE-2014-0160.
h ps://n d.nis .go / uln/de ail/
CVE-2014-0160, 2014.
[91] Node.js Founda ion. h ps://nodejs.o g.
[92] Oasis. eX ensible Access Con ol Ma kup Language, 2005.
[93]
OCZ Technology Inc. Dene a 2 Da a Shee .
h ps://d i e.google.com/
ile/d/0B4hWjkpwenosS0VMOWZkZ1B R1E/ iew?usp=sha ing, 2011.
[94]
Angelos Oikonomopoulos, Elias A hanasopoulos, He be Bos, and C is iano
Giu ida. Poking Holes in In o ma ion Hiding. In P oceedings o USENIX
Secu i y Symposium, 2016.
[95]
Oleksii Oleksenko, Dmi ii Ku aiskii, P amod Bha o ia, Pascal Felbe , and
Ch is o Fe ze . In el MPX Explained: A C oss-laye Analysis o he In el
MPX Sys em S ack. In P oceedings o ACM on Measu emen and Analysis o
Compu ing Sys ems, 2018.
[96]
OpenSSL. C yp o (OpenSSL c yp og aphic lib a y).
h p://www.openssl.
o g/, 2012.
131
[97]
B yan Pa no, Jona han M. McCune, and Ad ian Pe ig. Boo s apping T us
in Mode n Compu e s. In P oceedings o IEEE Symposium on Secu i y and
P i acy (Oakland), 2011.
[98]
Da id A. Pa e son, Ga h Gibson, and Randy H. Ka z. A case o edun-
dan a ays o inexpensi e disks (RAID). In P oceedings o ACM SIGMOD
in e na ional con e ence on Managemen o da a (SIGMOD), 1988.
[99]
Adam G. Penning on, John Linwood G i in, John S. Bucy, John D. S unk,
and G ego y R. Gange . S o age-Based In usion De ec ion. ACM T ansac ions
on In o ma ion and Sys em Secu i y, 13(4), 2010.
[100]
And ew Pimlo and Oleg Kiselyo . Sou ei, a Logic-Based T us -Managemen
Sys em. In P oceedings o In e na ional Symposium on Func ional and Logic
P og amming (FLOPS), 2006.
[101]
Sean Quinlan and Sean Do wa d. Ven i: a new app oach o a chi al da a
s o age. In P oceedings o USENIX Con e ence o File and S o age Technologies
(FAST), 2002.
[102]
E ik Riedel, Ch is os Falou os, Ga h a Gibson, and Da id Nagle. Ac i e Disks
o La ge Scale Da a P ocessing. Tc, 34(6), 2001.
[103] Rus language. h ps://www. us -lang.o g/.
[104]
Cai lin Sadowski, Edwa d A andilian, Alex Eagle, Liam Mille -Cushon, and
Cie a Jaspan. Lessons om building s a ic analysis ools a google. Commun.
ACM, 61(4):58–66, Ma ch 2018.
[105]
Je ome H. Sal ze and Michael D. Sch oede . The P o ec ion o In o ma ion
in Compu e Sys ems. In P oceedings o he IEEE, olume 63, 1975.
[106]
Samsung. 830 SSD da a shee .
h p://www.samsung.com/us/sys em/
consume /p oduc /mz/7p/c1/mz7pc128nam/830.pd , 2011.
132
[107]
Nuno San os, Rod igo Rod igues, K ishna P. Gummadi, and S e an Sa oiu.
Policy-sealed da a: A new abs ac ion o building us ed cloud se ices. In
P oceedings o USENIX Secu i y Symposium, 2012.
[108]
F ed B. Schneide , Ke in Walsh, and Emin Gün Si e . Nexus au ho iza ion logic
(NAL): Design a ionale and applica ions. ACM T ansac ions on In o ma ion
and Sys em Secu i y, 14(1), 2011.
[109]
Seaga e Technology LLC. Kine ic Open S o age Pla o m.
h p://www.
seaga e.com/solu ions/cloud/da a-cen e -cloud/pla o ms.
[110]
Seaga e Technology LLC. Sel -Enc yp ing Ha d Disk D i es in he Da a Cen e .
Technical Repo TP583, 2007.
[111]
Seaga e Technology LLC. Ba acuda Da a Shee .
h p://www.seaga e.com/
iles/s a ic iles/docs/pd /da ashee /disc/ba acuda-x -ds1696.
3-1102us.pd , 2012.
[112]
Seaga e Technology LLC. Momen us XT Da a Shee .
h p://www.seaga e.
com/docs/pd /da ashee /disc/ds_momen us_x .pd , 2012.
[113]
Ho a Shacham, Ma hew Page, Ben P a , Eu-Jin Goh, Nagend a Modadugu,
and Dan Boneh. On he e ec i eness o add ess-space andomiza ion. In
P oceedings o ACM SIGSAC Con e ence on Compu e and Communica ions
Secu i y (CCS), 2004.
[114]
Moni ul I. Sha i , Wenke Lee, Weidong Cui, and And ea Lanzi. Secu e in- m
moni o ing using ha dwa e i ualiza ion. In P oceedings o ACM SIGSAC
Con e ence on Compu e and Communica ions Secu i y (CCS), 2009.
[115]
Lei Shi, Yuming Wu, Yubin Xia, Na han Dau enhahn, Haibo Chen, Binyu
Zang, Haibing Guan, and Jiñming Li. Decons uc ing xen. In P oceedings o
Ne wo k and Dis ibu ed Sys em Secu i y Symposium (NDSS), 2017.
133

[116]
Jiwu Shu, Zhi ong Shen, and Wei Xue. Shield: A s ackable secu e s o age
sys em o ile sha ing in public s o age. J. Pa allel Dis ib. Compu ., 74(9),
Sep embe 2014.
[117]
Rui Shu, Peipei Wang, Sigmund A. Go ski III, Benjamin Andow, Adwai
Nadka ni, Luke Desho els, Jason Gion a, William Enck, and Xiaohui Gu. A
S udy o Secu i y Isola ion Techniques. ACM Compu ing Su eys, 49(3), 2016.
[118]
Emin Gün Si e , Willem de B uijn, Pa ick Reynolds, Alan Shieh, Ke in Walsh,
Dan Williams, and F ed B. Schneide . Logical a es a ion: an au ho iza ion
a chi ec u e o us wo hy compu ing. In P oceedings o ACM Symposium
on Ope a ing Sys ems P inciples (SOSP), 2011.
[119]
Gopalan Si a hanu, Swamina han Sunda a aman, and E ez Zadok. Type-sa e
disks. In P oceedings o USENIX Symposium on Ope a ing Sys ems Design
and Implemen a ion (OSDI), 2006.
[120]
Mu hian Si a hanu, Vijayan P abhaka an, Flo en ina I. Popo ici, Timo hy E.
Denehy, And e A paci-Dusseau, and Remzi A paci-Dusseau. Seman ically-
Sma Disk Sys ems. In P oceedings o USENIX Con e ence o File and S o age
Technologies (FAST), 2003.
[121] SQLi e. h ps://www.sqli e.o g.
[122] SQLi e. Speed es 1. h ps://www.sqli e.o g/ es ing.h ml.
[123]
S o age Wo k G oup o he T us ed Compu ing G oup. Sel -Enc yp ing D i es
Take o o S ong Da a P o ec ion.
h ps:// us edcompu ingg oup.o g/
sel -enc yp ing-d i es- ake-o -s ong-da a-p o ec ion/, 2010.
[124]
John D. S unk, Ga h R. Goodson, Michael L. Scheinhol z, C aig A. N.
Soules, and G ego y R. Gange . Sel -Secu ing S o age: P o ec ing Da a in
134
Comp omised Sys ems. In P oceedings o USENIX Symposium on Ope a ing
Sys ems Design and Implemen a ion (OSDI), 2000.
[125] Sun Mic osys ems. Sola is ZFS, 2009.
[126]
Laszlo Szeke es, Ma hias Paye , Tao Wei, and Dawn Song. SoK: E e nal
wa in memo y. In P oceedings o IEEE Symposium on Secu i y and P i acy
(Oakland), 2013.
[127]
TCG. TCG S o age A chi ec u e Co e Spec-
i ica ion.
h ps:// us edcompu ingg oup.o g/
cg-s o age-a chi ec u e-co e-speci ica ion/, 2007.
[128]
Eno The eska, Hi esh Ballani, G eg O’Shea, Thomas Ka agiannis, An Row-
s on, Tom Talepy, Richa d Black, and Timo hy Zhu. Io low: A so wa e-de ined
s o age a chi ec u e. In P oceedings o ACM Symposium on Ope a ing Sys ems
P inciples (SOSP), 2013.
[129]
Robe Wahbe, S e en Lucco, Thomas E. Ande son, and Susan L. G aham.
E icien so wa e-based aul isola ion. In P oceedings o ACM Symposium on
Ope a ing Sys ems P inciples (SOSP), 1993.
[130]
Ke in Walsh and F ed B. Schneide . Cos s o Secu i y in he PFS File Sys em.
Technical epo , Compu ing and In o ma ion Science, Co nell Uni e si y, 2012.
[131]
Da id H.D. Wa en. an Abs ac P olog Ins uc ion Se . Technical Repo
Technical No e 309, SRI In e na ional, 1983.
[132]
Robe N. M. Wa son, Jona han Ande son, Ben Lau ie, and K is Kennaway.
A as e o Capsicum. Communica ions o he ACM, 55(3), 2012.
[133]
Ca s en Weinhold and He mann Hä ig. VPFS: Building a i ual p i a e
ile sys em wi h a small us ed compu ing base. In ACM SIGOPS Ope a ing
Sys ems Re iew, 2008.
135
[134]
Ca s en Weinhold and He mann Hä ig. jVPFS: Adding Robus ness o a
Secu e S acked File Sys em wi h Un us ed Local S o age Componen s. In
P oceedings o USENIX Anual Technical Con e ence (ATC), 2011.
[135]
Jan We ne , Geo ge Bal as, Rob Dalla a, Na han O e ness, Ke in Z. Snow,
Fabian Mon ose, and Michalis Polych onakis. No-Execu e-A e -Read: P e-
en ing Code Disclosu e in Commodi y So wa e. In P oceedings o ACM
SIGSAC Asia Con e ence on Compu e and Communica ions Secu i y (Asia
CCS), 2016.
[136]
Wikimedia Founda ion. Image Dump.
h p://a chi e.o g/de ails/
wikimedia-image-dump-2005-11, 2005.
[137]
Wikimedia Founda ion. S a ic HTML dump.
h p://dumps.wikimedia.o g/
,
2008.
[138]
Wikimedia Founda ion. Page iew s a is ics Ap il 2012.
h p://dumps.
wikimedia.o g/o he /pagecoun s- aw/2012/2012-04/, 2012.
[139]
Edwa d Wobbe , Ma ín Abadi, Michael Bu ows, and Bu le Lampson. Au-
hen ica ion in he Taos ope a ing sys em. ACM T ansac ions on Compu e
Sys ems, 12(1), 1994.
[140]
Ted Wobbe , Aydan Yume e endi, Ma ín Abadi, And ew Bi ell, and Daniel R.
Simon. Au ho izing applica ions in singula i y. In ACM SIGOPS Ope a ing
Sys ems Re iew, olume 41, 2007.
[141]
Rubin Xu, Hassen Saïdi, and Ross Ande son. Au asium: P ac ical Policy
En o cemen o And oid Applica ions. In P oceedings o USENIX Secu i y
Symposium, 2012.
136
[142]
Yuanzhong Xu, Alan M. Dunn, Owen S. Ho mann, Michael Z. Lee, Syed Akba
Mehdi, and Emme Wi chel. Applica ion-de ined decen alized access con ol.
In P oceedings o USENIX Anual Technical Con e ence (ATC), 2014.
[143]
Benne Yee, Da id Seh , G ego y Da dyk, J. B adley Chen, Robe Mu h,
Ta is O mandy, Shiki Okasaka, Neha Na ula, and Nicholas Fullaga . Na i e
clien : A sandbox o po able, un us ed x86 na i e code. In P oceedings o
IEEE Symposium on Secu i y and P i acy (Oakland), 2009.
[144]
Yajin Zhou, Xiaoguang Wang, Yue Chen, and Zhi Wang. ARMlock: Ha dwa e-
based Faul Isola ion o ARM Yajin. In P oceedings o ACM SIGSAC Con e -
ence on Compu e and Communica ions Secu i y (CCS), 2014.
137