22/01/2025
2024 in Re iew:
Inciden s, Lea nings, and Plans
EGI CSIRT’s IRTF
EGI Inciden Response Task Fo ce
In oduc ion
Inciden Response Task Fo ce
•EGI CSIRT
–Coo dina es ope a ional secu i y ac i i ies wi hin EGI
•Inciden Response Task Fo ce (IRTF)
–A small eam o secu i y expe s, pa o he EGI CSIRT,
dis ibu ed ac oss mul iple coun ies and o ganiza ions.
–Take pa in an on-du y o a, ac as i s esponde s o
epo s o secu i y inciden s wi hin he EGI In as uc u e.
–Inciden esponse and digi al o ensics expe ise is
made a ailable o si es o he in es iga ion and
esolu ion o inciden s.
3
h ps://csi .egi.eu/ac i i ies/
Vulne abili ies
•Paki i agen s a e deployed on compu ing nodes o moni o and epo he
pa ching s a us o Linux sys ems.
•30 Ad iso ies sen by SVG in 2024
•53 Vulne abili ies epo ed by IRTF in 2024
–7 C i ical,! 17 High,! 1 Mode a e,! 28 O he s (unspeci ied, ad ise on con ig change).
4
h ps://paki i.egi.eu/
h ps://ope a ions-po al.egi.eu/
Communica ions Challenge
•Ensu e ha con ac in o ma ion is up- o-
da e and unc ional (biannual ac i i y)
–Enabling e icien coo dina ion du ing an
inciden .
•I is an email con aining de ails abou he
challenge along wi h a unique URL.
–Recipien s a e equi ed o access he URL,
which allows he esponse ime o be eco ded.
5
Communica ions Challenge
•New P ocedu e:
•EGI-Ope a ions is coo dina ing he ollow-up. Many hanks!
6
Communica ions Challenge - Si es
•The EGI Inciden Response policy s a es:
–“You shall ollow he inciden esponse p ocedu e
de ined by he e-In as uc u e".
•The associa ed p ocedu e (SEC01 EGI CSIRT
Secu i y Inciden Handling P ocedu e) de ines a
maximum esponse ime o 4 hou s.
•EGI si es no esponding p omp ly o secu i y
no i ica ions a e being suspended in GOC-DB
•Only Ce i ied si es a e being es ed.
–+30% o he si es a e no es ed! Simila o Paki i
–This in o ma ion is no consumed by VOs!
7
h ps://documen s.egi.eu/public/ShowDocumen ?docid=2935
h ps://con luence.egi.eu/display/EGIPP/SEC01+EGI+CSIRT+Secu i y+Inciden +Handling+P ocedu e
h ps://goc.egi.eu
Communica ions Challenge - VOs
•Communi y Ope a ions Secu i y Policy co e s he need o de ine a secu i y
con ac and eply o secu i y- ela ed eques s in a imely manne .
–The e a e no di ec penal ies o no eplying in due ime.
–We ely on building close ela ionships wi h VOs o demons a e he impo ance o
esponding o such es s and keeping hei con ac in o ma ion up o da e.
8
h ps://con luence.egi.eu/display/EGIPP/Communi y+Ope a ions+Secu i y+Policy
Resul s Communica ions Challenge - Si es
•Excluding echnical e o s, 13 si es (6%) and 30 si es (13%) ailed o espond.
•9 (4%) did no espond o ei he o he wo campaigns.
•
9
16
Inciden #1
Iden i y Mismanagemen
EGI Inciden Response Task Fo ce
[email p o ec ed]
h ps://con luence.egi.eu/display/EGIBG/CSIRT+PGP+key
Iden i y Mismanagemen
•All use s om a uni e si y IdP we e assigned he same EGI Check-in accoun
–This was due o he common de ini ion o oPe sonID.
–To mi iga e his issue:
•The oPe sonID ield was ini ially disabled o use s coming om his IdP.
•Use iden i ica ion was swi ched o ely only on eduPe sonUniqueId o ha speci ic IdP.
–The logs o se ices accessed using he sha ed Check-in iden i y we e analysed and
no malicious ac i i y was de ec ed.
–A e he uni e si y implemen ed he solu ion, he inciden was conside ed as
esol ed.
17
18
Inciden S #2
Use s Misbeha ing
EGI Inciden Response Task Fo ce
[email p o ec ed]
h ps://con luence.egi.eu/display/EGIBG/CSIRT+PGP+key
EGI Check-in use s misbeha ing
•A use a emp s o access mul iple, unconnec ed VOs’ esou ces.
A. Comp omised accoun
B. Gene ic accoun
C. Legi accoun
•The IdP was con ac ed, suspended he use and dele ed hei ac i e sessions.
•The e was no sign o any u he abuse.
•Some VOs which g an ed access o he use we e un esponsi e (also on he
CommsChallenge)
D. The VOs p o ided access o esou ces on a " ee ial" basis.
E. The use gained access wi h a c edible na a i e and a legi ima e passpo .
19
20
Inciden #3
Si e Comp omise
EGI Inciden Response Task Fo ce
[email p o ec ed]
h ps://con luence.egi.eu/display/EGIBG/CSIRT+PGP+key
De ec ion
•Unusual high load igge ed suspicion
•Admin epo ed and asked o help
•Access o a o ensics p oxy was p o ided
21
IRTF
Suspicious Se e Fo ensics P oxy
Ini ial Checks
•No excep ional load
•No suspicious p ocesses
•No unusual po s opened
•…bu one connec ion wi hou PID
22
Roo ki de ec ion
23
Payload
24
ke nel-dbus_s a .sh
Hide p ocess
Roo ki injec ion
Deac i a e SELinux
SSH s eale
C2 connec ion
Mine igge
Wipe logs
Check o connec ions
and s op he mine
Impac
•Mos o he nodes o he whole in as uc u e we e
comp omised
25
Execu ion and Pe sis ence
•Manual igge
•Scheduled Task/Job
•Boo o Logon Au os a Execu ion
•Ke nel Modules and Ex ensions
–h ps://gi hub.com/m0nad/Diamo phine
•Add SSH Au ho ized Keys
32
De ense E asion
•Masque ading ile name, se ice and loca ion
•Hide A i ac s
•Ob usca ed In o ma ion
–Open sou ce: node-bash-ob usca e
33
h ps://gi hub.com/willshiao/node-bash-ob usca e
De ense E asion
•Disable i ewall
•Compe i ion emo al
•Clean Logs and dele e malwa e
–Open Sou ce: mig
•Use non-S anda d Po
34
h ps://gi hub.com/Kabo /mig-logcleane - esu ec ed
P i ilege escala ion
•Valid Accoun s
–Comp omised accoun s wi h admin/sudo p i ileges.
•Exploi a ion o P i ilege Escala ion
–OpenSou ce ulne abili y explo e .
35
h ps://gi hub.com/The-Z-Labs/linux-exploi -sugges e /linux-exploi -sugges e .sh
Command and Con ol
•Bidi ec ional Communica ion using IRC (In e ne Relay Cha )
–IRC is a p o ocol using TCP/IP o eal- ime ex -based communica ion
36
IRC Connec
In ec ed Se e
Vic im in as uc u e
IRC Legi ne wo k
h ps://unde ne .o g/
Payload
•Ne wo k Flood (DDoS)
–M Scy heLULZ
•C yp o mining
–Nanomine and GMine
37
One mon h du ing he inciden
h ps://gi hub.com/nanopool/nanomine
h ps://gi hub.com/de elso wa e/GMine Release
h ps://gi hub.com/M Scy heLULZ/DDoS-Sc ip s
A ack Recipe
38
A ack Phase
Open Sou ce Tool
Ini ial Access
SSHP ank, CB u eK ag
Roo ki
Diamo phine
De ense E asion
Node-Bash-Ob usca e, Mig-Logcleane
La e al Mo emen
Masscan
P i ilege escala ion
TheZLabs Exploi Sugges e
C yp o mining
Nanomine , GMine
DDOS
Pea lbo
IRC
Unde ne , Dalne
Impac
•Ope a ing since a leas 2020
•200+ Linux se e s comp omised
•25+ o ganiza ions impac ed
•Two g oups ope a ing he same ools
•A ack sp ead quickly
•Found $16 000 in a single c yp o walle
39
Resea ch & Educa ion
Th ea In elligence Sha ing
•Ad iso ies
–Sha e de ailed in o ma ion
–De ec ion and mi iga ion ins uc ions
•MISP
–Sha e IOCs o immedia e de ec ion
•WLCG Secu i y Newsle e
–Coming soon. Subsc ibe!
40
h ps://misp.ce n.ch
h ps://e-g oups.ce n.ch/e-g oups/Eg oupsSubsc ip ion.do?eg oupName=wlcg-secu i y-newsle e
Lea nings
Ini ia i es
•De elop a s a egy wi h EGI Ope a ions and VOs o imp o e si e logging
capabili ies.
•Conduc ligh weigh secu i y exe cises o es and s eng hen si e inciden
esponse capabili ies.
•Collabo a e wi h WLCG o s eng hen ela ionships wi h VOs.
–Discuss secu i y opics in he WLCG Open Technical Fo um (OTF).
–Publish he WLCG Cybe secu i y Newsle e egula ly. Subsc ibe!
–Re iew policies and p ocedu es o ensu e alignmen wi h e ol ing needs.
•O ganize ainings including con e ences, CSC, and hands-on wo kshops.
•Implemen au oma ed scans o moni o EGI cen al se ices p oac i ely.
48
h ps://e-g oups.ce n.ch/e-g oups/Eg oupsSubsc ip ion.do?eg oupName=wlcg-secu i y-newsle e
Hands-On Wo kshop
•Technical wo kshop o sysadmins
•Lea n he mos impo an poin s o
–Respond e ec i ely o an inciden
–Ga he e idence wi hou ampe ing i
–Handle and s uc u e in es iga ions da a
–Pe o m digi al o ensics analysis
49
h ps://indico.ce n.ch/e/secu i y-wo kshop25
h ps://e-g oups.ce n.ch/e-g oups/Eg oupsSubsc ip ion.do?eg oupName=secu i y-wo kshop25
50
Ques ions / Feedback
EGI Inciden Response Task Fo ce
[email p o ec ed]
h ps://con luence.egi.eu/display/EGIBG/CSIRT+PGP+key
