Deli e able D2.6
ML Model Ce i ica ion – 1
Edi o (s):
Ching-Yu Kao
Responsible Pa ne :
F aunho e Ins i u e o Applied and In eg a ed Secu i y (FhG
AISEC)
S a us-Ve sion:
Final – 1.0
Da e:
28.10.2024
Type:
OTHER
Dis ibu ion le el:
PU
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 2 o 19
www.eme ald-he.eu
P ojec Numbe :
101120688
P ojec Ti le:
EMERALD
Ti le o Deli e able:
D2.6 – ML model ce i ica ion – 1
Due Da e o Deli e y o he EC
31.10.2024
Wo kpackage esponsible o he
Deli e able:
WP2 – Me hodology o knowledge ex ac ion
Edi o (s):
Ching-Yu Kao (FhG)
Con ibu o (s):
--
Re iewe (s):
Ma inella Pe occhi CNR
C is ina Ma ínez, Juncal Alonso (TECNALIA)
App o ed by:
All Pa ne s
Recommended/manda o y
eade s:
WP1, WP2, WP3, WP4, and WP5
Abs ac :
This deli e able p esen s componen s o e idence
ex ac ion om machine lea ning models ha can be
in eg a ed wi h he ce i ica ion g aph.
I is he esul o wo k pe o med in Task 2.4. This
documen is a i s /in e im e sion, he inal e sion on
sou ce e idence ex ac o s will be epo ed in D2.7
Keywo d Lis :
Knowledge ex ac ion, machine lea ning, deep lea ning,
obus ness, secu i y, echnical e idence
Licensing in o ma ion:
This wo k is licensed unde C ea i e Commons
A ibu ion-Sha eAlike 4.0 In e na ional (CC BY-SA 4.0
DEED h ps://c ea i ecommons.o g/licenses/by-sa/4.0/)
Disclaime
Funded by he Eu opean Union. Views and opinions
exp essed a e howe e hose o he au ho (s) only and
do no necessa ily e lec hose o he Eu opean Union.
The Eu opean Union canno be held esponsible o
hem.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 3 o 19
www.eme ald-he.eu
Documen Desc ip ion
Ve sion
Da e
Modi ica ions In oduced
Modi ica ion Reason
Modi ied by
0.1
03.09.2024
Fi s d a e sion, ou line
Ching-Yu Kao (FHG
AISEC)
0.2
07.10.2024
Added con en s o AI-SEC
Ching-Yu Kao (FHG
AISEC)
0.3
18.10.2024
Finaliza ion
Ching-Yu Kao (FHG
AISEC)
0.4
26.10.2024
In e nal e iew
Ma inella Pe occhi
(CNR)
0.5
28.10.2024
Modi ica ion a e QA e iew
Ching-Yu Kao (FHG
AISEC)
0.6
28.10.2024
Final Re iew
C is ina Ma ínez/
Juncal Alonso
(TECNALIA)
0.7
29.10.2024
Modi ica ions a e inal e iew
Ching-Yu Kao (FHG
AISEC)
1.0
31.10.2024
Submi ed o he Eu opean
Commission
C is ina Ma ínez/
Juncal Alonso
(TECNALIA)
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 4 o 19
www.eme ald-he.eu
Table o con en s
Te ms and abb e ia ions ............................................................................................................... 5
Execu i e Summa y ....................................................................................................................... 6
1 In oduc ion ........................................................................................................................... 7
1.1 Abou his deli e able ................................................................................................... 7
1.2 Documen s uc u e ...................................................................................................... 7
2 ML model e idence ex ac o s in he EMERALD a chi ec u e .............................................. 8
3 AI-SEC..................................................................................................................................... 9
3.1 Func ional desc ip ion ................................................................................................... 9
3.2 Technical desc ip ion .................................................................................................. 10
3.2.1 P o o ype a chi ec u e.................................................................................... 10
3.2.2 Technical speci ica ions .................................................................................. 11
3.3 Deli e y and usage ...................................................................................................... 12
3.3.1 Package in o ma ion ....................................................................................... 12
3.3.2 Ins alla ion ...................................................................................................... 12
3.3.3 Ins uc ions o use ......................................................................................... 13
3.3.4 Example o Running he Tool ......................................................................... 14
3.3.5 Licensing in o ma ion ..................................................................................... 16
3.3.6 Download ........................................................................................................ 16
3.4 Limi a ions and u u e wo k ........................................................................................ 16
4 Conclusions .......................................................................................................................... 18
5 Re e ences ........................................................................................................................... 19
Lis o ables
TABLE 1. REQUIREMENT AI-SEC.01 - EXTRACTION OF SECURITY FEATURES FROM ML MODELS ...................... 9
TABLE 2. OVERVIEW AND DESCRIPTION OF PACKAGE STRUCTURE FOR THE AI-SEC ...................................... 12
TABLE 3. SETUP FOR THE ML MODEL USING MNIST DATASET ................................................................. 15
TABLE 4. SETUP FOR THE ML MODEL USING CIFAR10 DATASET .............................................................. 15
TABLE 5. RESULTS ON MNIST AND CIFAR10 USING CLEVER SCORE, SHAPR SCORE, DATA POISONING AND
LIME. .................................................................................................................................... 15
Lis o igu es
FIGURE 1. EMERALD COMPONENT OVERVIEW DIAGRAM [6]. THE RED RECTANGLE HIGHLIGHTS THE ML MODEL
EVIDENCE EXTRACTION COMPONENTS, WHICH ARE DESCRIBED IN THIS DELIVERABLE. ............................. 8
FIGURE 2. AI-SEC ARCHITECTURE. TO EXTRACT FEATURES FROM ML MODELS, WE NEED TO EVALUATE THE
POISONING LEVEL (ATTACK COMPONENT), ROBUSTNESS (DATA PROCESSOR1), PRIVACY LEVEL (DATA
PROCESSOR1) AND EXPLANATIONS (DATA PROCESSOR2) ................................................................. 10
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 5 o 19
www.eme ald-he.eu
Te ms and abb e ia ions
AI
A i icial In elligence
AI-SEC
AI Secu i y E idence Collec o
AMOE
Assessmen and Managemen o O ganisa ional E idence
API
Applica ion P og amming In e ace
BSI
Bundesam ü Siche hei in de In o ma ions echnik
BSI C4
A i icial In elligence Cloud Se ices Compliance C i e ia Ca alogue
Ce G aph
Ce i ica ion G aph
CIFAR
Canadian Ins i u e Fo Ad anced Resea ch
Codyze
S a ic Code Analyze om FHG
CSA o EU CSA
EU Cybe secu i y Ac
CSP
Cloud Se ice P o ide
CSV
Comma-Sepa a ed Values
CLEVER
C oss Lipschi z Ex eme Value o nE wo k Robus ness
DoA
Desc ip ion o Ac ion
EC
Eu opean Commission
eknows
Pla o m o So wa e Analysis om SCCH
GA
G an Ag eemen o he p ojec
KPI
Key Pe o mance Indica o
MEDINA
P edecesso p ojec o EMERALD
MIT
Massachuse s Ins i u e o Technology
MNIST
Modi ied Na ional Ins i u e o S anda ds and Technology da abase
LIME
Local In e p e able Model-agnos ic Explana ions
MEDINA
P edecesso p ojec o EMERALD
PNG
Po able Ne wo k G aphics
SW
So wa e
SHAP
SHapley Addi i e exPlana ions
TOM
Technical and O ganisa ional Measu e
TRL
Technology Readiness Le el
WP
Wo k Package
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 6 o 19
www.eme ald-he.eu
Execu i e Summa y
This deli e able p esen s he ini ial design, a chi ec u e, and implemen a ion s a e o he
machine lea ning (ML) model e idence ex ac o s o WP2, we call i AI-SEC. They con ibu e o
he key esul KR1-EXTRACT o EMERALD, a amewo k o con inuously ex ac knowledge om
well- ained ML models and p epa e sui able e idence based on hem.
EMERALD ollows a knowledge g aph-based app oach o p o ide a uni ied iew o he cloud
se ice unde ce i ica ion a di e en laye s o he se ice, anging om he in as uc u e laye
(e.g., i ual esou ces), o he business laye (e.g., policies and p ocedu es), o he
implemen a ion laye (e.g., sou ce code iles) and da a laye (e.g., inc easingly used AI models)
in cloud applica ions.
The ML model e idence ex ac o s, de eloped in Task 2.4 and desc ibed in his deli e able, aim
a iden i ying c i ical secu i y- ela ed ea u es, such as ad e sa ial obus ness, p i acy, secu i y
and explainable AI. O he ela ed deli e ables in WP2, all due a p ojec Mon h 12 (Oc obe
2024), p o ide unc ional and echnical de ails on u he e idence ex ac o s om di e en
sou ces, i.e., D2.2 [1] on sou ce code e idence ex ac ion, D2.4 [2] on e idence ex ac ion om
policy documen s in Task 2.3 and D2.8 [3] on un ime da a ex ac ion in Task 2.5. All hese de ails
con ibu ed o D2.1 [4] on he o e all in o ma ion model o he ce i ica ion g aph in Task 2.1.
The main pa o his deli e able p o ides unc ional and echnical desc ip ions o he e idence
ex ac o AI-SEC, including i s pu pose and scope, he (cu en and planned) co e age o he
EMERALD equi emen s, and he componen s’ in e nal a chi ec u e. These desc ip ions a e
complemen ed by in o ma ion on deli e y and usage, as well as on limi a ions and u u e wo k.
Finally, he documen concludes wi h a sho summa y.
The ML model e idence ex ac o s desc ibed in his deli e able con ibu e o KR1-EXTRACT by
p o iding nex -gene a ion e idence ga he ing ools and echniques based on a knowledge g aph
app oach. The p esen ed ex ac o s cu en ly ha e he ini ial p o o ypes implemen ed and
eady o be ( o some deg ee) in eg a ed wi h o he componen s o he EMERALD a chi ec u e.
Based on he wo k desc ibed in his deli e able, he ML model e idence ex ac o s will be
u he ex ended and in eg a ed in o he EMERALD amewo k. This is he i s i e a ion o he
deli e able coming om Task 2.4. The second and inal e sion o his deli e able (D2.7 [5] ) wi h
he upda ed ex ac o s will be deli e ed in p ojec Mon h 24 (Oc obe 2025). E idence will be
p epa ed acco ding o he in eg a ed, g aph-based model o seman ically linked and combined
e idence.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 7 o 19
www.eme ald-he.eu
1 In oduc ion
EMERALD aims o p o ide a nex gene a ion se o e idence ga he ing ools and echniques
based on a knowledge g aph app oach. KR1-EXTRACT suppo s an imp o ed and uni ied ool-
suppo ed app oach o con inuously ex ac knowledge om di e en laye s o a cloud se ice,
e.g., in as uc u e, pla o m, un ime in o ma ion, policy documen s, so wa e, and AI models.
The objec i e o WP2 is o es ablish a uni ied iew o he cloud se ice unde ce i ica ion by
ex ac ing and en iching knowledge o he di e en laye s o he se ice and p o iding sui able
e idence o secu i y me ics. A g aph-based model, called he ce i ica ion g aph (Ce G aph),
se es as a common s uc u e ha is illed by all e idence ex ac ion ools.
1.1 Abou his deli e able
This deli e able ocuses on he design, implemen a ion, and ini ial e alua ion o he ools and
echniques ha o m he backbone o EMERALD's e idence-ga he ing amewo k. The
deli e able emphasizes he ole o AI-SEC in c ea ing e idence om he ML models.
1.2 Documen s uc u e
The documen is s uc u ed as ollows.
In Sec ion 3 we epo on he design and implemen a ion o AI-SEC ML model ex ac o . Fo he
ML model ex ac o , unc ional and echnical desc ip ions a e p o ided, including hei pu pose
and scope, he (cu en and planned) co e age o he EMERALD equi emen s, he componen s’
in e nal a chi ec u e, hei subcomponen s, and de ails abou he p og amming language,
lib a ies, e c. used. These desc ip ions a e complemen ed by in o ma ion on deli e y and usage,
including package in o ma ion, ins alla ion ins uc ions, use manual, licensing and download
in o ma ion, as well as limi a ions and u u e wo k.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 8 o 19
www.eme ald-he.eu
2 ML model e idence ex ac o s in he EMERALD a chi ec u e
This sec ion desc ibes how he ML model e idence ex ac o s in e ac wi h (selec ed) EMERALD
componen s on a concep ual le el. Figu e 1 shows he EMERALD high-le el a chi ec u e as a
componen diag am, as desc ibed in D1.1 [6]. In EMERALD, a componen is de ined as “any pa
o he EMERALD ecosys em ha has a speci ic unc ionali y and can be conside ed a sepa a e
en i y wi h espec o o he componen s” (see D1.3 [7]).
The componen s o collec ing e idence abou echnical and o ganisa ional measu es, i.e.,
AMOE, eknows, AI-SEC, Cloudi o -Disco e y, and Codyze, a e ep esen ed a he bo om pa o
Figu e 1. The ML model e idence ex ac o AI-SEC, which ob ains echnical e idence om he
analysis o he ML model o cloud applica ions, is highligh ed using a hick ame. AI-SEC is a
newly de eloped componen in EMERALD and analyses AI models o se e al key e idence
ega ding obus ness agains ad e sa ial a acks, explainabili y, and ai ness.
Figu e 1. EMERALD componen o e iew diag am [6]. The ed ec angle highligh s he ML model
e idence ex ac ion componen s, which a e desc ibed in his deli e able.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 9 o 19
www.eme ald-he.eu
3 AI-SEC
In alignmen wi h he equi emen s de ined in Sec ion 6.2, "Secu i y & Robus ness Objec i e,"
o he BSI C i e ia Ca alogue C5, AI-SEC is designed o mee hese c i ical secu i y c i e ia o
ensu e comp ehensi e p o ec ion and compliance. Ou solu ion add esses ou key aspec s:
p i acy, ad e sa y esis ance, explainabili y, and da a leakage p e en ion. These elemen s a e
undamen al in es ablishing a obus and secu e sys em capable o wi hs anding po en ial
h ea s while main aining anspa ency and da a in eg i y.
3.1 Func ional desc ip ion
O e all pu pose. The p o o ype p o ides a comp ehensi e oolki o e alua ing and imp o ing
he secu i y o machine lea ning models by ocusing on ad e sa ial obus ness es ing, p i acy
ulne abili y assessmen , da a poisoning a acks, and model in e p e abili y. I e ec i ely
assesses ulne abili ies using echniques such as CLEVER sco e calcula ion [8], SHAP leakage
analysis [9], backdoo da a poisoning [10], and LIME-based explana ions [11].
These ea u es will be collec ed as e idence o he ce i ica ion g aph.
Con ex and scope. The oolki assumes ha use s al eady ha e p e- ained models a ailable,
which can be di ec ly u ilized o e alua ion. Addi ionally, i suppo s s anda d da ase s o
obus ness and p i acy assessmen s, while also allowing cus om da ase impo s o ailo ed
e alua ions.
Mo i a ion. The oolki aims o s eamline he secu i y e alua ion o machine lea ning models
by in eg a ing mul iple secu i y assessmen s in o a uni ied sys em.
Inno a ion. AI-SEC will ocus on he ollowing inno a ions:
• In eg a ion o mul iple secu i y assessmen s, obus ness, p i acy, and in e p e abili y
in o a single, uni ied sys em.
• Au oma ion o calcula ions and esul gene a ion h ough command-line inpu s
Requi emen s. The ele an equi emen s wi h hei espec i e implemen a ion s a e (pa ially
/ ully / no implemen ed) and a b ie desc ip ion o how hey a e / will be implemen ed a e
p o ided in Table 1.
Table 1. Requi emen AI-SEC.01 - Ex ac ion o secu i y ea u es om ML models
Field
Desc ip ion
Requi emen ID
AI-SEC.01
Sho i le
The ex ac o ool includes de ined c i e ia
Desc ip ion
The designed AI-SEC has he ea u es based on BSI AIC4
S a us
Wo k in P og ess
P io i y
Mus
Componen
AI-SEC
Sou ce
Componen , KPI
Type
Technical
Rela ed KR
KR5_AIPOC
Rela ed KPI
KPI 5.1
Valida ion accep ance
c i e ia
Code e iew: Re iew code and check i analysis me hods
wo k o di e en ML models.
P og ess
Pa ially implemen ed – 35%
Miles one
MS5: Componen s V2 (M24)
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 16 o 19
www.eme ald-he.eu
Da a
Poisoning
o iginal image
poisoned image
o iginal image
poisoned image
LIME-based
explana ions
o iginal image
model p edic ion
o iginal image
model p edic ion
3.3.5 Licensing in o ma ion
Since LIME and SHAP use pe missi e MIT licenses, we choose o license his ool unde he MIT
License.
3.3.6 Download
The cu en ly implemen ed pa s a e s o ed on EMERALD's Gi lab
11
.
3.4 Limi a ions and u u e wo k
The cu en es s using accessible models ha e shown easonable esul s, demons a ing he AI-
SEC po en ial. Howe e , a signi ican limi a ion is ha many cloud se ices do no g an di ec
access o hei models, which poses a challenge o comp ehensi e e alua ion. To add ess his,
a po en ial solu ion is o use a p oxy model as a subs i u e o he cloud-based model. While
p omising, u he expe imen a ion is equi ed o e alua e he e ec i eness o he p oxy model
in accu a ely e lec ing he beha iou o he o iginal cloud model. Addi ionally, op imiza ion o
11
h ps://gi .code. ecnalia.com/eme ald/public/componen s/ai-sec
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 17 o 19
www.eme ald-he.eu
he ool is necessa y o enhance i s e iciency and ensu e i can un smoo hly in a ious
en i onmen s.
Fu u e wo k will ocus on e ining he AI-SEC pe o mance and explo ing al e na i e me hods
o model e alua ion in scena ios whe e di ec access is es ic ed. These e o s will help
imp o e he adap abili y and obus ness o AI-SEC ac oss di e en use cases. Fu u e ac i i ies
will also co e he in eg a ion o he AI-SEC componen in he EMERALD CaaS amewo k as
e idence ex ac o ool and wi h he EMERALD UI. All hese changes will be epo ed in he sub-
sequen e sion o his deli e able, namely, D2.7 [5] in p ojec mon h M24.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 18 o 19
www.eme ald-he.eu
4 Conclusions
In his deli e able, as an ini ial ou pu o Task 2.4, we p esen ed he design, a chi ec u e, and
cu en implemen a ion s a us o he EMERALD model e idence ex ac ion componen s. These
componen s ollow he holis ic app oach o he EMERALD amewo k and a e aligned wi h he
echnical equi emen s ga he ed in WP1 (D1.3 [12]). The epo ou lines he ela ionship
be ween he p esen ed componen , AI-SEC and o he pa s o he EMERALD amewo k,
de ailing he in e nal s uc u e o he componen , i s subcomponen s, and ele an in o ma ion
abou i s echnical implemen a ion.
The componen in oduced in he epo , AI-SEC, suppo s e idence ex ac ion o machine
lea ning models. A he cu en s age o he p ojec , his componen , based on p elimina y wo k,
has a wo king p o o ype ha can (pa ially) in eg a e wi h o he EMERALD componen s and has
been es ed using accessible ML models. Fu u e wo k will in ol e es ing his ool wi h mo e
complex models.
The subsequen and inal i e a ion o his epo (D2.7 [5]), which will p o ide upda es on he
p og ess o he componen , is planned o p ojec mon h 24.
D2.6 - ML model ce i ica ion – 1 Ve sion 1.0 – Final. Da e: 31.10.2024
© EMERALD Conso ium Con ac No. GA 101120688 Page 19 o 19
www.eme ald-he.eu
5 Re e ences
[1]
EMERALD Conso ium, “D2.2 Sou ce E idence Ex ac o – 1,” 2024.
[2]
EMERALD Conso ium, “D2.4 AMOE – 1: E idence ex ac ion om policy documen s ha
can be in eg a ed wi h he ce i ica ion g aph,” 2024.
[3]
EMERALD Conso ium, “D2.8 Run ime e idence ex ac o – 1: E idence ex ac ion om
un ime da a ha can be in eg a ed wi h he ce i ica ion g aph,” 2024.
[4]
EMERALD Conso ium, “D2.1 G aph On ology o E idence S o age: Desc ip ion o a
uni o m schema o s o ing and linking he e ogenous da a,” 2024.
[5]
EMERALD Conso ium, “D2.7 ML model ce i ica ion– 2”.
[6]
EMERALD Conso ium, “D1.1 Da a modelling and in e ac ion mechanisms - 1,” 2024.
[7]
EMERALD Conso ium, “EMERALD Glossa y in D1.3- EMERALD solu ion a chi ec u e - 1,”
2024.
[8]
T.-W. Weng, H. Zhang, P.-Y. Chen, Y. Jin eng, D. Su, Y. Gao, C.-J. Hsieh and L. Daniel,
“E alua ing he obus ness o neu al ne wo ks: An ex eme alue heo y app oach,” a Xi
p ep in , 2018.
[9]
V. Duddu, S. Szylle and N. Asokan, “Shap : An e icien and e sa ile membe ship p i acy
isk me ic o machine lea ning,” a Xi p ep in , 2021.
[10]
H. Sou i, L. Fowl, R. Chellappa, M. Goldblum and T. Gols ein, “Sleepe agen : Scalable
hidden igge backdoo s o neu al ne wo ks ained om sc a ch,” Ad ances in Neu al
In o ma ion P ocessing Sys ems 35, 2022.
[11]
M. T. Ribei o, S. Singh and C. Gues in, “"Why should I us you?" Explaining he
p edic ions o any classi ie ,” P oceedings o he 22nd ACM SIGKDD In e na ional
Con e ence on Knowledge Disco e y and Da a Mining, pp. 1135-1144, 2016.
[12]
EMERALD Conso ium, “D1.3 EMERALD solu ion a chi ec u e - 1,” 2024.
[13]
A. Saha, A. Sub amanya and H. Pi sia ash, “Hidden igge backdoo a acks,” in
P oceedings o he AAAI con e ence on a i icial in elligence, , ol. 34, no. 07, pp. 11957-
11965, 2020.
