Análisis de código e ins umen ación de los
en íos pa a jueces en línea
Code analysis and ins umen a ion o
submissions o online judges
T abajo de Fin de G ado
Cu so 2023–2024
Au o es
Luis Es eban Velasco
Juan T illo Ca e as
Rod igo Bu gos Sosa
Di ec o
Rubén Ra ael Rubio Cuélla
G ado en Ingenie ía In o má ica
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
Análisis de código e ins umen ación de los
en íos pa a jueces en línea
Code analysis and ins umen a ion o
submissions o online judges
T abajo de Fin de G ado en Ingenie ía In o má ica
Au o es
Luis Es eban Velasco
Juan T illo Ca e as
Rod igo Bu gos Sosa
Di ec o
Rubén Ra ael Rubio Cuélla
Con oca o ia: Junio 2024
G ado en Ingenie ía In o má ica
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
27 de mayo de 2024
Ag adecimien os
A nues as amilias po su apoyo, al equipo de DOMjudge y inalmen e a Rubén
po su en usiasmo y dedicación como u o de es e abajo de in de g ado.

Resumen
Análisis de código e ins umen ación de los en íos
pa a jueces en línea
La p ác ica au ónoma es undamen al en el ap endizaje de la p og amación y en
los di e sos con ex os donde se enseña es a disciplina los co ec o es au omá icos de
p oblemas son muy u ilizados. Los llamados jueces en línea o ecen un epe o io de
p oblemas y pe mi en en ia a los usua ios sus soluciones pa a que sean e aluadas en
el momen o. Pa a e alua las, el juez ecibe el código del usua io, lo compila y lo eje-
cu a sob e dis in os ejemplos pa a después comp oba que los esul ados coincidan
con los espe ados. El esul ado de esa e aluación pa a el es udian e se limi a habi-
ualmen e a un e edic o que indica sin de alles si la solución es co ec a o no, a da
demasiado o alla al ejecu a se. El p o eso , en cambio, e las di e encias en b u o
en e la salida del p og ama y la salida espe ada. En los concu sos de p og amación,
es a in o mación es más que su icien e, pe o una e oalimen ación más ins uc i-
a y signi ica i a se ía de g an ayuda pa a el es udian e que es á ap endiendo a
p og ama .
El obje i o de es e abajo es aplica écnicas de análisis de es á ico e ins u-
men ación en los jueces au omá icos pa a p opo ciona e oalimen ación más sig-
ni ica i a a es udian es y p o eso es.
Se ha es udiado la e ec i idad de un pequeño conjun o de he amien as sob e
en íos eales de años an e io es. Las conclusiones de es e análisis, in e esan es po sí
mismas, nos han se ido pa a decidi qué comp obaciones son más ele an es pa a
su implemen ación al juez. Hemos ex endido el juez au omá ico DOMjudge, amplia-
men e u ilizado en la acul ad de In o má ica de la UCM y en o as uni e sidades,
pa a in eg a es as comp obaciones y mos a los esul ados en la e oalimen ación
a los es udian es. Es o ha eque ido modi icaciones en el código y en el p oceso de
ins alación del juez au omá ico, y el desa ollo de he amien as pa a p ocesa la
in o mación de los analizado es.
Palab as cla e
Juez en línea, Análisis es á ico, Ins umen ación, C++, DOMjudge
ii
Abs ac
Code analysis and ins umen a ion o submissions o
online judges
Au onomous p ac ice is undamen al in lea ning p og amming and, in he a ious
con ex s whe e p og amming is augh , au oma ic p oblem checke s a e ex ensi ely
used. The so-called online judges o e a epe oi e o p oblems and allow use s o
submi hei solu ions o on- he-spo e alua ion. The judge ecei es he code om
he use , compiles i , and uns i on di e en es inpu s, hen checks ha he esul s
ma ch he expec ed ones. The esul o his e alua ion o he s uden is usually
limi ed o a e dic ha indica es wi hou de ails whe he he solu ion is co ec
o no , akes oo long o ails o un. The eache , on he o he hand, sees he
aw di e ences be ween he p og am ou pu and he expec ed ou pu . In p og am-
ming compe i ions, his in o ma ion is mo e han enough, bu mo e ins uc i e and
meaning ul eedback would be o g ea help o he s uden lea ning o code.
The aim o his wo k is o apply s a ic code analysis echniques and ins umen-
a ion on au oma ic judges o p o ide mo e meaning ul eedback o s uden s and
eache s.
The e ec i eness o a small se o ools has been s udied on ac ual submissions
om p e ious yea s. The conclusions o his analysis, in e es ing in hei own, ha e
helped us decide which checks a e mos ele an o implemen on he judge. We
ha e ex ended he online judge DOMjudge, (widely used in he Facul y o Com-
pu e Science a UCM and o he uni e si ies) o in eg a e hese checks and include
he esul s in he eedback. This has equi ed modi ica ions o he code and he
se up p ocess o he au oma ic judge, and he de elopmen o ools o p ocess he
in o ma ion om hose ools.
Keywo ds
Online Judge, S a ic analysis, Ins umen a ion, C++, DOMjudge
ix

Chap e 1
In oduc ion
In his sec ion, we p esen he mo i a ion o his hesis, as well as he p oposed
objec i es, he wo k plan, a b ie o e iew o his hesis’ o ganiza ion, and he
eposi o ies ha include he so wa e de eloped.
Online judges a e widely used pla o ms o es he unc ionali y o code o a
speci ied p oblem. They a e commonly used in compe i i e p og amming con es s,
bu hei applica ion is also p esen in many educa ional ins i u ions. Code is sub-
mi ed o he online judge, which will be in cha ge o compiling he code and es
i as an app oxima ion o check i s co ec ness; hese es s usually in ol e inpu ing
o he submi ed code a p ede ined inpu and compa ing he ou pu agains he
expec ed one. The code can be w i en in any p og amming languages he online
judge suppo s. Addi ional cons ain s, such as ime and memo y limi s, can be
imposed. Use s will hen ecei e he judge’s e dic , accep ed o w ong answe ( he
possible e dic s o DOMjudge appea in Sec ion 2.1.1).
These a e no he only wo possible e u ned e dic s. A submission can be
e alua ed as inapp op ia e o a p oblem i i does no compile o execu e co ec ly.
Ensu ing ha a piece o code sa is ies hese condi ions alls on he use s hemsel es
and, in many cases, i can be a non- i ial challenge (specially o hose jus s a ing
o lea n o code). One can a emp o ead hei code in an a emp o ind e o s
on i . Ve i y whe e a iables a e de ined and used, hei ypes, he possible alues
ha hey can ha e du ing code execu ion, o wha e ec hey can ha e on an ac ion.
This is, in essence, s a ic analysis o code, and he e exis many ools ha pe o m
his beha io on a much b oade scale.
S a ic analyze s a e g ea ools o ind issues in code as i is w i en, as hey
do no equi e o execu e he code. They pe o m a deepe analysis on he code
han he one a pe son could easonably de elop, in ol ing he examina ion o code
s uc u es, syn ax, and seman ics o de ec issues such as inadequa ely used a i-
ables, bug p one code, and iola ions o coding con en ions. They a e commonly
implemen ed in code de elopmen wo k lows o de ec e o s in he code ea ly in
he p ocess. In addi ion o s a ic analyze s, hese ype o wo k lows can also use
code ins umen a ion, such as sani ize s, o complemen he capabili ies o s a ic
code analyze s. They a e un ime ools (in con as o s a ic analyze s) and hey
can iden i y addi ional p oblems in he code, such as memo y leaks and unde ined
1
2Chap e 1. In oduc ion
beha io .
Au oma ically analyzing a p og am and explaining i s e o s is an inhe en ly
di icul ask, as seman ic analysis o he code will gene ally be undecidable. This
is p o ed by Rice’s heo em, ha s a es ha all non- i ial seman ic p ope ies o
p og ams a e undecidable; non- i ial p ope ies a e ones ha a e nei he ue o
e e y p og am no alse o e e y p og am. [28] So, i is impossible o asse ha a
ce ain p og am has a p ope y jus by looking a how i beha es (i s seman ics),
unless ha p ope y is i ial. Mo eo e , an example o i can be seen in he hal ing
p oblem, whe e i is demons a ed ha i is impossible o w i e a p og am ha ,
ecei ing as inpu ano he p og am, can decide whe he i will s op o no .
Howe e , a ious o mal and in o mal echniques can p o ide pa ial answe s o
some o hese p oblems. These me hods ha e been es ic ed o he academy o he
analysis o c i ical p og ams in he indus y, bu hey ha e ecen ly become mo e
popula wi h he appea ance o ools aimed a he s anda d p og amme and he
inclusion o analyze s in he mos widely used compile s o languages as popula
as C++. Fo ins ance, he la es e sions o he GNU compile and Clang include
s a ic analyze s o de ec e o s in he code wi hou execu ing i , allow p og ams
o be ins umen ed wi h p ede ined checks a un ime, and p o ide esou ces o
ad anced use s o easily de ine hei own checks.
These posi i e ou comes could be a welcome addi ion o he epe oi e o possi-
bili ies ha an online judge o e s.
1.1. Mo i a ion
In compu e science, i is o en he case ha eaching ins i u ions use p ac ical
labo a o y lec u es as complemen o hei heo e ical ones, specially ea ly in he
deg ee. I is a popula and p o en ac ha one lea ns he mos by doing, hence he e
exis many incen i es o include online judges in labo a o y classes as a main ool
o apply his p inciple. Online judges can aid in s uc u ing an ac i e lea ning pa h
by which s uden s can apply hei acqui ed knowledge du ing heo e ical lessons o
w i e code ha es s hei p oblem-sol ing skills.
The main disad an age ha his kind o ools b ing o a s udying en i onmen a e
he o en lacking eedback capabili ies ha hey p o ide. Ins uc o s a e, in many
cases, in cha ge o p o iding assis ance o hose ha a e s uck wi h a doub o ace
a ne e seen be o e (o en echnical) compila ion o execu ion e o . Gene ally, i
is un ealis ic o ins uc o s o be able o a end and sa is y e e y single s uden ’s
needs, as i is o assume ha mul iple o hem can be p esen simul aneously. The e
is e en he case o no such ins uc o p esen o a ailable a a gi en ime, and
ha ing a ool as online judges, ha a e able o p o ide his so o aid, despi e no
as specialized, can o e g ea alue.
We, he eam behind his hesis, ha e amilia i y wi h hese si ua ions. Unde -
s anding he us a ion ha comes wi h no g asping he eason o an unaccep ed
submission o compila ion e o has de e mined o be essen ial in unde s anding
he app ecia ion o his kind o in e ac ion. The e ha e been mul iple easons o
equi e o pe o m a manual analysis o he code in o de o iden i y he e o s and
1.2. Goals 3
inconsis encies con ained in i , essen ially pe o ming s a ic analysis; he applica ion
o a ool ha ul ills his pu pose and p o ides eedback based on i is, in ou belie ,
a aluable addi ion. A s a ic analyze is able o ind e o s in he code wi hou
execu ing i by in e p e ing he code ollowing symbolic execu ion. These means
ha , as he alues o he a iables a e unknown ou side execu ion ime, he ana-
lyze mus use a symbol o ep esen hese alues, and ollow he possible pa hs o
execu ion ha he code can ake. Each pa h con ains a possible se o alues ha
he a iable could be assigned o, and i an e o is ound in ha pa h, i is epo ed
back o he use .
S a ic analyze s can be an addi ional s ep included du ing compila ion o unning
o code inside an online judge, and i s ou pu can be used o gene a e a eedback
message o s uden s acco dingly. They would se e well wi hin hem, specially
when accompanied by addi ional ools such as code ins umen a ion. The di e -
ence be ween bo h ools is ha he code needs o be execu ed o ools pe o ming
ins umen a ion o de ec e o s inside i .
G an ed, ins umen a ion will, in con as o s a ic analyze s, consume mo e un
ime, bu being able o cus omize he possible execu ions o code inside he judges
hemsel es esul s in a e y a ac i e p oposi ion.
To add o he ma e , code analysis is a ield ha ex ends o he ea ly days
o he appea ance o p og amming languages, which has p o en o con ain leng hy
esea ch and many con ibu o s o i s de elopmen . Ve y ad anced ools a e p esen
wi h ou s anding pe o mance and wo k behind, ha could be implemen ed o he
ma e being ea ed.
By combining s a ic analysis and lin ing wi h un ime ins umen a ion, achie ing
a mo e comp ehensi e app oach o lea ning is possible. The challenge esides in
applying he igh balance o he eedback, and no gi e oo much away as means
o g an ing assis ance, which would p oduce he opposi e esul ha wha is ying
o be achie ed.
1.2. Goals
Ou main goal is o p o ide mo e meaning ul eedback o s uden and ins uc o s
using online judges by le e aging on s a ic analysis and ins umen a ion. A he end,
we wan o ha e hem in eg a ed in o DOMjudge, bu o achie e his goal we i s
need o unde s and he echniques and ools and ind he mos app op ia e way o
use hem. We conside he ollowing speci ic goals:
Unde s and s a ic analysis and ins umen a ion as p o ided by se e al main-
s eam ools.
S udy he beha io o a selec ed se o ools (Clang-Tidy, Cppcheck, ASan,
and UBSan) on s uden s submissions and decide which a e mos use ul o
meaning ul eedback.
Ex end DOMjudge o allow he execu ion o code analyze s and ins umen ed
submissions and p esen s hei indings as eedback o s uden s and ins uc o s.
4Chap e 1. In oduc ion
1.3. Wo k Plan
This p ojec has been de eloped in wo main phases: s udy phase and imple-
men a ion phase. The ime managemen o hese phases can be seen in igu e 1.1.
Du ing he ini ial phase, we s udied he necessa y concep s in o de o unde -
s and he basics o how s a ic analyze s and ins umen a ion wo k and a e used.
Following his, a e iew on he mos popula ools in ha domain was ca ied ou ;
his p ocess was undamen al in o de o de e mine which would be included la e
in he implemen a ion. Once ools we e chosen, we analyzed hei pe o mance on
a se o eal s uden submissions p o ided by ou u o . This da a was pa o eal
s uden submissions o exams and class p oblems, which was bene icial because i
mean ha we could pe o m an analysis on ac ual eal submissions in o de o
ob ain ep esen a i e da a.
Following his, he i s s ep owa ds being able o implemen hese ools was un-
de s anding how DOMjudge ecei es a submission and judges i . This is a necessa y
s ep o iden i y he ele an code ha should be modi ied in he upcoming s eps.
Secondly, we p epa ed and se up a Docke en i onmen o deploy a DOMjudge
ins ance in ou machines o es ing and de eloping pu poses. Once he equi ed
changes we e included in ou ex ended DOMjudge, we pu e e y hing done so a
oge he o speci y he igh checks ha should be included o he s a ic analyze s
and ins umen a ion.
Wi h his, i emained o e i y he unc ionali y o he modi ied judge wi h
ealis ic submissions o check he co ec ness o he implemen a ion and documen
e e y hing.
The ollowing Gan diag am po ays he ime managemen du ing his whole
p ocess:
Figu e 1.1: Gan diag am po aying ime managemen
1.4. O ganiza ion o his hesis 5
1.4. O ganiza ion o his hesis
The gene al s uc u e o he hesis consis s o he ollowing sec ions:
Chap e 1 (In oduc ion): This pa se s ou he easons o ca ying ou he
wo k, he objec i es o be achie ed, he o ganiza ion o he hesis as well as
linking he eposi o y o he code used in o de o ca y ou his hesis.
Chap e 2 (S a e o he a ): In he s a e o he a we ocus on explaining he
pla o m we a e going o modi y and he ools ha we a e going o use o he
ealiza ion o his hesis.
Chap e 3 (O line analysis on eal s uden submissions): This chap e we
analyze he ou pu o he s a ic analyze s and sani ize s on eal submissions.
We also show he p ocedu e o ob ain and analyze he da a, he esul s o ha
analysis and he conclusions we ha e a i ed o.
Chap e 4 (In eg a ion o s a ic and un ime analysis in DOMjudge): his
chap e ocuses on he modi ica ions made o he DOMjudge pla o m o in-
eg a e he ools necessa y o c ea ing he eedback.
Chap e 5 (Real use case scena io): In his chap e we show wo es s ca ied
ou in o de o e i y he judge’s unc ionali y. Fi s ly, a p epa ed submission
ha se es as p oo o concep and, secondly, a made up submission ha a -
emp s o be as close o eali y as possible .We will show he eedback p o ided
by he judge.
Chap e 6 (Conclusions and Fu u e Wo k): This chap e se s ou he discussion
o he hesis whe e we discuss he main p oblems we encoun e ed and how we
sol ed hem, i also shows he conclusions we ha e a i ed o by de eloping
his hesis as well as he u u e wo k ha could be ca ied ou .
Appendix A: able con aining an in dep h lis ega ding occu ences o Clang-
Tidy diagnos ics in submissions, he comple e lis o ele an checks o he
ools o Cppcheck and UBSan as well as a able compa ing all he code analysis
checks o no o e lap hem.
1.5. Sou ce code and da a eposi o ies
We ha e de eloped and main ained a se ies o eposi o ies du ing he de elop-
men o his p ojec ha collec ou e o s o comple e he di e en miles ones.
The Gi Hub eposi o y con ains he mos ele an sc ip s and code ha we ha e
de eloped o his hesis: sc ip s o analyzing he in o ma ion p o ided by he ools
used in he s uden submissions, sc ip s o pa sing he ou pu o he analyze s, code
o c ea ing g aphs o unde s and be e he esul s o he analyze s, he code o
he es case and eal use case scena ios, he pa se s used in DOMjudge o show he
eedback, and he composi ion o he Docke :

6Chap e 1. In oduc ion
h ps://gi hub.com/Jan i-3/analysis-o -DOMjudge-submissions
This eposi o y is he esul o a common e o o he whole eam.
The modi ica ions applied o he sou ce code o DOMjudge o in eg a e ou code
analysis ea u es is a ailable in a o k o he o icial eposi o y a :
h ps://gi hub.com/ obu02/DOMjudge
The changes made o he sou ce code will be explained in Chap e 4. Mo eo e ,
Docke Hub eposi o ies ha e been used o main ain Docke images o he modi ied
DOMjudge. These images ha e been c ea ed by us om he modi ied DOMjudge
sou ce code. The DOMSe e image is a
h ps://hub.docke .com/ / obuso02/domse e
and he judgehos image a
h ps://hub.docke .com/ / obuso02/judgehos
The c ea ion o hese images will be explained in Sec ion 4.4.
Chap e 2
S a e o he a
This chap e will co e he main aspec s o he hesis o unde s and online judges
pla o ms and he ools we a e going o implemen on he DOMjudge in o de o
gi e eedback o he s uden .
2.1. Online judges
The online judges we e c ea ed o acili a e code co ec ions o ins uc o s a
uni e si ies, we will s a by de ining wha is an online judge. Online judges a e
pla o ms (usually wi h web in e aces) used o code assessmen playing a pi o al
ole in educa ion, compe i i e p og amming, and ca ee aining. [21]
Online judges ake examples o inpu s and hei co esponding expec ed ou pu
p epa ed by he ins uc o , hen in oduce ha same inpu o he code uploaded
and compa e he ou pu wi h he expec ed one o check i i ma ches, i i does, he
judge quali ies he submission as co ec Figu e 2.1.
Some judges used in educa ion ins i u ions in Spain a e DOMjudge and Ju ge.o g [27].
Ju ge.o g is de eloped by he Poly echnic Uni e si y o Ca alonia in o de o p o ide
a pla o m o lea ning and p ac icing p og amming ( ailo ed speci ically o hei
compu e science s uden s).
As men ioned be o e, online judges a e also used in p og amming con es s.
Some no able examples a e: he In e na ional Collegia e P og amming Con es
(ICPC) [12], an algo i hmic p og amming con es o college s uden s whe e DOM-
judge is used, and he In e na ional Olympiad in In o ma ics (IOI) [25] con es s o
seconda y and high school s uden s whe e he judge used is CMS [5], as well as hei
egional quali ie s.
Some o he amous online judges a e P ojec Eule [16], Codeche [6]and DMOJ [2].
In his p ojec , we will discuss DOMjudge Figu e 2.2, he o icial judge o he
ICPC, as i is he one we use a ou Uni e si y.
7
8Chap e 2. S a e o he a
Figu e 2.1: P ocess o a classic online judge
Figu e 2.2: DOMjudge logo
2.1.1. DOMjudge
As s a ed on hei websi e, “DOMjudge is a sys em o unning p og amming
con es s, like he ICPC egional and wo ld inals p og amming con es s.” [7]. How-
e e in he speci ic case o uni e si ies, i is used o gi ing s uden s assignmen s
and o co ec hem by an au oma ic p ocess. These assignmen s consis o coding
p oblems ha ead inpu acco ding o he p oblem inpu speci ica ion (gi en by
he p oblem se e ) and y o make a code ha w i es he co ec , co esponding
ou pu .
To e u n a e dic , he judge pe o ms se e al s eps on a submission. Fi s ly,
i will compile he code (i no an in e p e ed language), since he use s p o ide
hei sou ce code. Compila ion is pe o med based on he p og amming language
o he submission, which is speci ied by he use a he momen o submission. The
esul ing execu able ile is hen execu ed, ecei ing as inpu he es case speci ied
o he p oblem a emp ing o be sol ed. The ou pu p oduced du ing he execu ion
phase is sa ed o be compa ed wi h he expec ed ou pu o he p oblem. The ou pu
will be pa sed o e i y i s simila i y wi h he solu ion.
The p e ious s eps can be cus omized o he needs o he p oblem. DOMjudge
2.1. Online judges 9
o e s he possibili y o using di e en compa e and un iles. I hey a e no speci ied,
de aul s will be used. These special un and compa e iles allow o specialized and
speci ic checks o equi emen s o s eps o a judgemen o a speci ic p oblem. To
be used, hey ha e o be selec ed in he special_ un and special_compa e ields
o he p oblem.
Depending on he esul s o he execu ion and he expec ed ou pu , a e dic is
ob ained o each es case ou pu s a esul , hese esul s a e hen agg ega ed in o
a main e dic . The e dic gi en can be:
WA (W ong Answe ): he code p oduced an ou pu di e en om he ex-
pec ed.
CE (Compila ion E o ): he code has compila ion e o s.
RTE (RunTime Excep ion): du ing he execu ion o he code an e o has
occu ed.
TLE (Time Limi Excep ion): he ime limi o code execu ion has been
exceeded.
AC (ACcep ed): he code has inished wi hin he ime gi en and has p oduced
he expec ed ou pu .
Al hough hese e dic s a e no equen , hey may some imes happen:
NO (No Ou pu ): he code has p oduced no ou pu .
OLE (Ou pu Limi Excep ion): he code p oduced oo much ou pu .
When he e’s mo e han one es case hen he esul s a e gi en based on he i s
highes p io i y esul . This p io i y is decided by he adminis a o o he judge,
in ou case he p io i ies a e:
AC: 1 ≪NO: 10 ≪WA: 30 ≪TLE, RTE, OLE: 99 (maximum p io i y)
Then, i WA and TLE a e he esul s in he es cases o a p oblem he submission
esul would be TLE as i has highe p io i y.
Some o he ea u es ha DOMjudge p o ides a e: Web in e ace; Modula
sys em o plugging in languages/compile s and alida o s; Ju y in o ma ion (such
as numbe o submissions, judging and di s) and op ions ( ejudge, cla i ica ions,
esubmi ). Howe e , he e is no ac ual ool p o ided o gi ing de ailed eedback,
hey jus gi e he s uden he esul as explained be o e.
The main de elope s o his pla o m a e Jaap Elde ing, Nicky Ge i sen, Kei h
Johnson, Thijs Kinkho s , Ma Pluijmaeke s, Michael Vasseu and Tobias We h.
16 Chap e 3. O line analysis on eal s uden submissions
Table 3.1: Numbe o diagnos ics in compiled submissions
Name Submission Team P oblem
pe o mance-a oid-endl 564 263 48
bugp one-na owing-con e sions 481 241 38
pe o mance- o - ange-copy 117 64 11
pe o mance-unnecessa y- alue-pa am 78 56 21
clang-analyze -deadcode.DeadS o es 57 32 15
bugp one-b anch-clone 56 25 12
bugp one-suspicious-semicolon 38 16 11
bugp one-in ege -di ision 32 28 1
clang-analyze -cplusplus.NewDele e 27 13 2
misc- edundan -exp ession 14 3 1
clang-analyze -op in.cplusplus.Unini ializedObjec 1 1 1
clang-analyze -op in.cplusplus.Unini ializedObjec : Unini ialized ields
in objec s c ea ed a e a cons uc o call.
In Table 3.1 we show he numbe o occu ences o each diagnos ic wi hin all
submissions, shown in he second column. Bu , in o de o a oid any kinds o bias
ela ed o diagnos ics appea ing mul iple imes wi hin a eam o a p oblem we added
wo o he columns. The hi d column coun s he numbe o eam’s submissions o
a p oblem which con ained he diagnos ic, and, inally, he ou h column con ains
he numbe o p oblems which ha e submissions con aining he diagnos ic. See he
ull e sion o Table 3.1 in Table A.1.
F om he Clang-Tidy s a is ics we can each he ollowing conclusions o each o
he ele an diagnos ics:
pe o mance-a oid-endl: I has 18 ansi ions om TLE o AC and e en i
he 43% o AC submissions is simila o he 48% global pe cen age o AC, as i
has he mos ansi ions o AC, and i is a nice oppo uni y o each s uden s
mo e abou IO ope a ions, i would be a nice one o communica e o s uden s
using he judge.
bugp one-na owing-con e sions: I also has a pe cen age o AC e y
simila o global one due o no always being a c i ical e o , bu i is p obably
always unin en ional and i also appea s in many p oblems.
pe o mance- o - ange-copy and pe o mance-unnecessa y- alue-pa am:
As i was expec ed he pe o mance do no appea o ha e ele ancy when
looking a AC a es o ansi ions, bu as hey each an impo an lesson and
appea many imes, we ha e decided i is bene icial o communica e i o he
s uden s

3.3. Code analysis esul s 17
clang-analyze -deadcode.DeadS o es: I has a high accep ance a e (38%)
because i can be due o a le o e alue ha is no used in he end, bu , i has 3
ansi ions o AC showing i could be help ul o ix e o s in some submissions.
bugp one-b anch-clone: 37% o AC submissions is no signi ican enough,
bu as many imes i is an e o due o copy pas ing and hen o ge ing o
change some hing, i could be communica ed wi h lowe p io i y.
bugp one-in ege -di ision: I has a e y low accep ance a e, bu i only
appea s in one p oblem, could be communica ed only in some p oblems, i is
also ha d o spo i you do no know abou i .
bugp one-suspicious-semicolon: I is no always a c i ical e o bu as
i was obse ed in some submissions i is e y icky o spo . Some s uden s
abandon he p oblem be o e eaching he solu ion and o he s each he solu ion
wi h he diagnos ic s ill p esen , o example a s uden used a e na y ope a o
wi hin an i s a emen wi h a semicolon a e wa ds, his is why his diagnos ic
has a low numbe o ansi ions o AC.
clang-analyze -cplusplus.NewDele e: I has 0 ansi ions o AC and a
high accep ance a e (48%). This is due o some alse posi i es.
misc- edundan -exp ession: We decided o include his check e en hough
i appea s a e y small amoun o imes, due o he ac ha i is e y icky
o ind some imes, i also has 0 ansi ions o AC, bu ha is due o eams
abandoning he p oblem be o e eaching an AC submission, one eam wen as
a as submi ing 11 solu ions wi hou eaching an AC. When analyzing hose
submissions code we no iced ha i was a logic e o ha was easily sol able.
clang-analyze -op in.cplusplus.Unini ializedObjec : I appea s only once
in a submission wi h RTE.
3.3.2. Cppcheck
Cppcheck checks ha appea ed on he da ase a e di ided in o 5 majo ca e-
go ies:
in o ma ion : 3 ypes o checks appea ed, jus 1 is impo an
s yle : 24 ypes appea ed, 13 o hem ele an ones
wa ning : 7 ypes appea ed, 6 a e impo an
pe o mance : 4 ypes appea ed, all o hem a e didac ic
e o : 8 ypes appea ed, jus 3 a e ac ually ele an
Cppcheck is a ex ensi e s a ic analyze as explained in he s a e o he a . This
is why, we can’ ind any simila checks be ween each o he . Howe e , as hey
a e qui e speci ic, we can ac ually join a ious checks in o jus one. This can be
18 Chap e 3. O line analysis on eal s uden submissions
seen on he able o appendix C, whe e we make a de ailed explana ion o each Cp-
pcheck and Clang. idy check and de e mine ha 4 di e en Cppcheck’s checks can be
co e ed by jus one Clang-Tidy check. The checks we e " edundan Condi ion",
"mul iCondi ion", "duplica eExp ession" and "cons S a emen ", all o hem
could be simpli ied by le ing he Clang-Tidy check misc- edundan -exp ession
localize and explain o he s uden he lawed coding p ac ice hey had. This can
be seen on he able in he Appendix A.4.
A e an exhaus i e s udy o he checks, we ha e educed he numbe o checks
aised om 46 di e en ele an checks o jus 27 ele an checks. All ele an
checks co e comple ely di e en a eas o bad p ac ices o he code ( his is a esul
o he said cha ac e is ics o Cppcheck). The checks a e e y di e se and could no
be educed anymo e as hey did no o e lap a all. Howe e , he e is ac ually 3
checks ha co e di e en "a eas" o he code bu could be easily joined in o one,
The ShadowVa iable,ShadowFunc ion and ShadowA gumen . As he names
sugges , hey a e all ela ed o collisions on named a iables/ unc ions o a gumen s
used on an inne unc ion which o e laps he meaning. This is a poo p og amming
p ac ice and mus be communica ed o he s uden .
The ollowing sec ion is included in he appendix a he han he main ex ,
no due o i s lack o ele ance, bu o a oid o e whelming he eade . Al hough
his in o ma ion is qui e in e es ing and in luen ial, he o ma is epe i i e as i
de ails all aspec s o he e o s iden i ied by Cppcheck. None heless, we s ongly
ecommend dedica ing ime o he Appendix A.2.
3.4. Ins umen a ion esul s
In his sec ion we show he ins umen a ion’s esul s o each o he sani ize s
3.4.1. Memo y e o s by ASan
Ou o 3663 submissions, 172 ha e ASan e o s. On a e age, he e we e 2.94
e o s pe submission, o a o al o 506 e o s. You also ha e o accoun ha each
p oblem has a di e en numbe o es cases and ASan only gene a es ou pu when
aul y code is eached. F om he ASan ou pu s we ex ac ed da a abou he numbe
o each e o ’s occu ences, abou he dis ibu ion o esul s on each es case wi h
an e o , and on each submission wi h e o s, and abou he ansi ions om a no
accep ed es case o an accep ed one when he ASan e o s a e ixed. They can be
ound in he eposi o y, again in he analyze _ou pu s di ec o y.
We can obse e ha mos o he submission wi h ASan e o s we e ejec ed
by he online judge 3.2, he e o s ha appea he mos a e heap bu e o e lows,
segmen a ion aul s and s ack o e lows, and hey a e no commi ed by a ew eams,
no in a ew p oblems, meaning hey a e common e o s. Run ime excep ion is he
mos common esul o all e o s, excep o memo y leaks, his is expec ed as hey
do no usually cause he p og am o s op, unless i uns ou o memo y.
Su p isingly, he e a e submissions wi h a al ASan e o s ha a e accep ed by
he judge. Ou o he 20 es cases wi h e o s ha we e accep ed: 2 we e memo y
3.4. Ins umen a ion esul s 19
Figu e 3.2: Ve dic in es cases and submissions wi h ASan e o s
3.95% AC
1.19% NO
TLE
5.34%
RTE
66.6%
WA
22.92%
(a) By es case (506 e o s)
1.78% AC
0.58% NO
TLE
8.72%
RTE
72.67%
WA
16.28%
(b) By submission (172 e o s)
leaks and 18 we e heap o e lows. Al hough memo y leaks should be a oided, hey
a e no e o s which should be a al o he execu ion o he code, as i should
con inue execu ion i he e is enough memo y. Heap o e lows on he o he side a e
mo e dange ous, hey a e due o ou o bounds accesses, hey only s op execu ion
i he posi ion besides he ec o is accessible by he p og am. E en hough ha
posi ion in memo y is no supposed o be accessed, he p og am can luckily ead
he igh alue o s o e i in a iable ha is no used, hence he solu ion can s ill
be accep ed.
3.4.2. Unde ined beha io by UBSan
The e a e 408 submissions ou o a o al o 4119 ha gene a e one o mo e
ubsan.log iles con aining he e o s. Gene a ing a o al o 930 ubsan.log.
On a e age 2.28 iles a e gene a ed pe submission, again he e a e p oblems
wi h mo e cases han o he s. As we ha e epea ed e o s, we need o elimina e any
edundancies ound. We can ind se e al e o s on an assignmen epea ed on all
es cases o said assignmen . Wi hin he same case, we can ha e se e al e o s, so
we needed o make a Py hon sc ip o clean he da a.
A e cleaning he da a, we need o make e en mo e modi ica ions o he da a
in o de o associa e e o s by ype as UBSan dis inguishes e o s by de ails ha
he s uden s don’ need (such as “null poin e wi h base 0xACDFFFFFF. . . ”). We
managed o iden i y 14 di e en ypes o e o s ia egula exp essions.
The majo i y o UBSan e o s de ec ed esul in no accep ed quali ica ion by
he judge wi h only 23 ou o 408 submissions being accep ed.Figu e 3.3
UBSan e o s de ec ed should be a al o he execu ion in any case(meaning
ha hey will p obably in e up he execu ion). Howe e , he p og amme could
be “lucky”, as he memo y space no in ended by he p og amme o eading could
ha e a no null alue o con ain some hing which made sense in he code. This is
he main eason ha all UBSan e o s aised should be no i ied o he p og amme .
I is clea ly obse ed when checking he e o s aised ha mos e o s a e ela ed
o poin e s. Al hough his is expec ed, i is in e es ing.
E e y ime a “load o null poin e ”, “e o e e ence binding o misaligned ad-
20 Chap e 3. O line analysis on eal s uden submissions
Figu e 3.3: Ve dic in es cases wi h UBSan e o s
O he e dic s
94.36%
AC
5.64%
d ess”, “load o misaligned add ess”, “s o e o null poin e ”, “s o e o misaligned
add ess”, “membe call on null poin e ” and “null poin e passed as a gumen , which
is decla ed o ne e be null” appea s on he ubsan.log’s gene a ed, he esul o
he judge on ha es case is always RTE . These esul s make sense as he unde-
ined beha io e o s a e usually a al a un ime as said be o e (a leas wi h he
p e iously men ioned, his is no he case o example wi h in ege o e lows). An
example o in e es is “s o e o null poin e ” which emind us o some e o s ASan
also de ec s as “segmen a ion aul ” which is a memo y add ess e o . These e o s
can appea oge he when unning bo h ools on he same submission.
To see he es o he lis o UBSan checks please e e o he Appendix A.3.
3.5. Conclusions
A e an exhaus i e analysis o he s a ic analyze s and he sani ize s we conclude
ha hey a e o g ea impo ance in o de o p o ide a de ailed eedback o he
s uden s, wi h hese di e en ools, Clang-Tidy, Cppcheck, ASan and UBSan, we
co e some common mis akes s uden s make wi h a lis o 53 checks.
We ha e obse ed ha a submission ha ing e o s de ec ed in he s a ic analysis
usually means ha he submission was ejec ed by he judge. Submissions wi h
ASan e o s a e accep ed only 1.74% o he imes (3/172), many imes due o a
un ime excep ion, as obse ed in igu e 3.2. Submissions wi h UBSan e o s ha e
an e en lowe pe cen age o accep ance 0.56% (23/408). E en i hey only appea
in a low amoun o submissions. On he con a y, lin e s diagnos ics appea on a
highe numbe o submissions bu a e less p one o no being accep ed, o example
many Clang-Tidy diagnos ics ha e a pe cen age o accep ance simila o he o al
submission accep ance a e, which is 42.5%. This means bo h kind o ools will
complemen hemsel es, while ins umen a ion will de ec ha she e o s ha a e
easie o de ec a un ime, s a ic analysis will de ec e o s ha a e easie o de ec
when analyzing he sou ce code o ine iciencies ha when sol ed will make o a
be e solu ion, e en i hey do no a ec he judge’s esul .
Chap e 4
In eg a ion o s a ic and un ime analysis
in DOMjudge
This chap e will ocus on he ac ual changes pe o med o he DOMjudge sou ce
code in o de o achie e ou goal. Fo his pu pose, one has o s udy and unde s and
how he judge wo ks a a mo e specialized le el in o de o see wha changes mus
be pe o med.
The e was an abundance o debugging o e o s, wi h hei co esponding in es-
iga ion, o ind wo ka ounds o issues in his sec ion. He e, DOMjudge de elope s’
insigh s and ad ice se ed as g ea pa hs o ollow o ge h ough hem, and hey
will be discussed and men ioned ac oss he co esponding segmen o he chap e .
The i s and mos impo an s ep would be o iden i y he app op ia e momen o
un he s a ic analyze s and ins umen a ion du ing he e alua ion p ocess, as well
as he loca ion in he ile sys em whe e he iles mus be s o ed. This mus no
obs uc –o do in he leas amoun possible– he judge’s asks ha mus be ca ied
ou in o de o e alua e a submission. Once an app op ia e me hod is disco e ed,
he nex logical s ep would be o ind he way o ge he s a ic and un ime analyses
ou pu s o he use .
To ca y his ask ou , Docke has been e y help ul. The ease o deploymen
o Docke con aine s allowed o a e y pe missi e es ing sui e ha could be dis-
man led and se back up as many imes as necessa y. I also allowed o e e y eam
membe o eplica e he de elopmen wo kspace in hei machines.
In sec ion 4.1, we will ocus on dissec ing how he DOMjudge wo ks and i s
s uc u e. We will how a submission is e alua ed and he p ocesses ha i has o
go h ough. Then, in sec ion 4.2, we will explain he implemen a ion decisions ha
we e made once he submission p ocess was unde s ood, and he necessa y changes
ha had o be applied in he sou ce code o make hem happen. Du ing sec ion 4.3,
we will hen explain how a message o a submission is buil based on he ou pu s
gene a ed by he s a ic analyze s and ins umen a ion ha ha e been implemen ed
and how i is p esen ed o he s uden . Finally, in sec ion 4.4, we will expose how
Docke was used in o de o achie e all o his, na iga ing he p ocess o gene a ing
a new image and wha was equi ed in o de o se up. Also, we will explain he
c ea ion o a Docke -compose ile ha o ches a es he p ocess o ini ializing all
21

22 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
he con aine s in an au oma ed manne in o de o a oid ha ing o ini ialize hem
manually.
4.1. Mo e on he DOMjudge a chi ec u e
As he i s miles one o achie e ou goal, unde s anding how he judge wo ks
is c ucial. I s componen s, equi emen s, which sec ions a e ele an o us, and
ollowing he o de o calls. The goal is se mainly on unde s anding how submissions
a e p ocessed, as o he a eas o he judge a e no ele an o his p ojec . We a e
in e es ed in knowing wha happens om he momen a s uden p esses he submi
bu on, o he momen ha he e alua ion is e u ned. Whe e does i compile? How
does i execu e? How a e he compa isons made be ween he expec ed and ac ual
ou pu ? The answe s o hese ques ions would gi e us he clues o know whe e o
s a making he necessa y modi ica ions. Since we’ e dealing wi h s a ic analyze s
and sani ize s ha ha e o be included du ing p og am compila ion, i ’s specially
ele an o us o know whe e and how his unc ion is ca ied ou .
The DOMjudge e sion used a he Compu e Science School’s labo a o ies a
he ime o conduc ing his p ojec is a a he ou da ed e sion. DOMjudge is an
open sou ce p ojec and, as such, i is cons an ly ecei ing upda es hanks o he
in ol emen o i s main de elope s and he pa icipa ion o olun ee con ibu o s.
The sou ce code is published on Gi Hub,1and he p ojec coun s wi h a Slack
wo kspace o any ype o con e sa ion ela ed o i , a mailing lis o de elopmen
discussion –which has been used o his p ojec –, and an inbox o bug epo s o
ea u e eques s h ough Gi Hub o he mailing lis i sel .
I is wo h no ing ha DOMjudge is p ima ily designed o i s use in p og am-
ming compe i ions ha ake place o e a pe iod o ime ha can ange om 5 hou s
o a weekend. As such, i s use in he class ooms is no he in ended o iginal pu -
pose, and i is impo an o unde s and his in o de o g asp he easons o he
absence o any lea ning unc ionali y in i s implemen a ion. Ou goal will be o
modi y he sou ce code o achie e ea u es ha a e no suppo ed and may igge
incompa ibili ies wi h he es o he implemen a ion. We’ll del e deepe in o he
a ious issues encoun e ed h oughou he p ojec in he sec ions ha ollow, as well
as some o DOMjudge’s implemen a ion de ails.
4.1.1. The s uc u e
DOMjudge is composed o wo main componen s, DOMse e , a da abase and
a leas one judgehos . They a e independen componen s ha communica e wi h
each o he .
DOMjudge is composed o wo main componen s: DOMSe e and a leas one
judgehos . They a e independen componen s ha communica e wi h each o he .
The di e en judgehos s a e in cha ge o compiling, unning, and checking a sub-
mission.2Aside om hese, he ac ual ins alla ion also equi es a da abase o s o e
1h ps://gi hub.com/DOMjudge/DOMjudge [9]
2h ps://www.DOMjudge.o g/docs/manual/8.2/judging.h ml# low-o -a-submission [36]
4.1. Mo e on he DOMjudge a chi ec u e 23
all he in o ma ion a ailable inside he se e . I communica es di ec ly wi h he
DOMse e componen .
DOMse e is sho o DOMjudge se e , and i is he cen al en i y ha uns
he DOMjudge web in e ace and API; which is based on he Con es Con ol Sys em
(CCS) Con es API speci ica ion. The DOMse e uses a MySQL o Ma iaDB
da abase se e o in o ma ion s o age. Fo he de elopmen o his p ojec , we
will be using a Ma iaDB da abase ins ance. The machine in which he DOMse e
is hos ed needs o be unning Linux o Unix a ian and a web se e wi h PHP 8.1
o newe .
Fo he online judge o be unc ional, i equi es one o mo e judgehos s which will
pe o m he compila ion and e alua ion o submissions. Fo ou es ing pu poses, a
single judgehos will su ice, as he DOMjudge eam ecommends one judgehos pe
wen y eams.3Judgehos s will connec o he se e and wai o a solu ion o be
submi ed by a eam. Submissions a e added o a queue om which he di e en
judgehos s will e ie e hem and hen e u n a e dic a e he equi ed p ocessing
has been done on i . The judgedaemon is in cha ge o managing his inside he
judgehos s [30].
4.1.2. Submission en i onmen
The judgedaemon compiles and execu es submissions inside a ch oo en i on-
men 4 o secu i y easons – secu i y is a main conce n o he eam, as a bi a y
code is being execu ed in he sys em. This en i onmen mus con ain all he equi e-
men s ha a e needed a compile ime o build and un he submissions. DOMjudge
also uses Linux Con ol G oups (o cg oups) and namespaces o p ocess isola ion
in he judgedaemon. Cg oups allow o alloca e esou ces such as CPU ime, mem-
o y, and ne wo k bandwid h o use -de ined g oups o asks.5This aspec limi ed
some o ou ini ial op ions, since we conside ed he use o API eques s om he
sc ip s an inside he execu ion en i onmen in o de o e ie e addi ional use ul
in o ma ion, such as he numbe o a emp s o a p oblem by a eam. Due o he
ac ha he execu ion en i onmen sees i s In e ne access limi ed, he op ion had
o be sc aped.
4.1.3. Submission p ocess
When a s uden o eam makes a submission, his submission is added o a queue
o no ye e alua ed submissions. When mul iple judgehos s exis , hey can all
wo k in pa allel o e alua e said queue, each pe o ming hei ac ions on indi idual
submissions. Each submission will be picked by one o he ac i e judgehos s and i
3h ps://www.DOMjudge.o g/docs/manual/8.1/o e iew.h ml#
equi emen s-and-con es -planning [36]
4A ch oo en i onmen can modi y he oo di ec o y o a p ocess, es ic ing
access o he es o he ile sys em. The command being un has no idea
ha any hing ou side i s en i onmen exis s. h ps://www.how ogeek.com/de ops/
wha -is-ch oo -on-linux-and-how-do-you-use-i / [15]
5h ps://access. edha .com/documen a ion/es-es/ ed_ha _en e p ise_linux/6/
h ml/ esou ce_managemen _guide/ch01 [19]
24 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
will be e alua ed by i . To do so, he code a e ses he compile, un, and compa e
phases. Execu ion is pe o med in he p e iously men ioned es ic ed en i onmen .
Res ic ions in he en i onmen will be mainly in he o m o ime and memo y usage
limi a ions. The ou pu p oduced du ing he execu ion phase is hen compa ed wi h
he expec ed ou pu o he solu ion, which is he second s ep o his p ocess, whe e
all he compa e p ocess akes place. Once comple ed, p ocessing on he judgehos
e mina es, and i e u ns he submission in o ma ion o he DOMse e , in cha ge
o displaying i o he use .
The en i e p ocess is o ches a ed by he code con ained in judgedaemon.main.php,
unning on e e y judgehos . I e alua es he submi ed assignmen and ans e s he
esul s back o he DOMSe e . The asks o compiling, unning and compa ing he
code a e delega ed o a numbe o di e en sc ip s ha gene a e he equi ed iles.
Fi s , compile.sh is execu ed o compile he submission. The espec i e execu able
pa h is passed o es case_ un.sh, whose ask is o pe o m un and compa e on
he submission. Modi ica ions pe o med o his second ile ha e been key in o de
o achie e he goals o his p ojec , as i plays a main ole du ing he e alua ion
p ocess.
Be o e con inuing, we will cla i y some e ms ha may induce con usion. The
execu ion o un phase o he submission p ocess is done by execu ing a sc ip called
un. To make i clea as o when we a e e e ing o he un phase and when
we a e alking abou he un sc ip , we will always e e o he phase explici ly.
Addi ionally, he compa e phase is pe o med by execu ing an execu able called un,
ha is ob ained when compiling he sou ce code compa e.cc. To a oid con usion
be ween bo h un iles, we will e e o he un p oduced o he compa e phase as
compa e om his poin onwa ds, and he compa e phase will always ecei e he
explici dis inc ion.
The whole p ocess ha in ol es compiling ( o ou pa icula case, his will
always ake place, since we a e using C++ code), unning and compa ing he esul s
is execu ed o e e y es case speci ied o a p oblem. This is because o he mul iple
judgehos s can exis a he same ime, and he di e en es cases can be dis ibu ed
among hem. Compila ion will only ake place once pe judgehos , as he judgehos
can iden i y i i al eady exis s a compile ou pu , as means o sa e ime. The e can
exis one o mo e han one es cases pe p oblem, and he es case_ un.sh sc ip
is un o e e y one o hem.
Figu e 4.1 isually ep esen s he o de o calls ha a e pe o med o he de-
sc ibed p ocess.
4.1.4. Mo e on iles
I has been indica ed which a e he iles in cha ge o pe o ming he di e en ac-
ions, and in his sec ion hey will desc ibe he sc ip s compile.sh, es case_ un.sh,
un, and he execu able compa e u he .
compile.sh This is an in e nal DOMjudge sc ip esponsible o compiling he
s uden ’s assignmen and is called om judgehos .main.php. Being in e nal means
ha i is no mean o be changed by he use s. I accep s as inpu he absolu e pa h
4.1. Mo e on he DOMjudge a chi ec u e 25
Figu e 4.1: Submission p ocess sequence diag am. I con ains he o de in which
he calls o he di e en sc ip s a e made.
o he compila ion sc ip o he language, he wo king di ec o y o he e alua ion
o he cu en assignmen , and he ile o iles submi ed by he s uden . I e u ns
0 in case o success, “compile -e o ” o “in e nal-e o ” i de ined, o 1 o he wise.
This is he compile w appe -sc ip called om compile.sh o compile C++ code
by de aul :
DEST="$1";shi
MEMLIMIT="$1";shi
# -x c++: Explici ly se compile language o C++ (no objec
iles o ,→
# o he languages au ode ec ed by ex ension)
# -Wall: Repo all wa nings
# -O2: Le el 2 op imiza ions (de aul o speed)
# -s a ic: S a ic link wi h all lib a ies
# -pipe: Use pipes o communica ion be ween s ages o
compila ion,→
g++ -x c++ -Wall -O2 -s a ic -pipe -o "$DEST" "$@"
exi $?
The compila ion sc ip has i s own command line in e ace, aking as inpu
he name o he execu able o be c ea ed, sa ed inside he a iable DEST, and he
maximum memo y usage bound, sa ed inside he a iable MEMLIMIT. The esul is
conside ed a compila ion e o i he ile wi h he name o he ecei ed execu able
is no c ea ed o i he compile.sh sc ip e u ns a alue o he han 0.
32 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
Figu e 4.4: Op ion enabled om DOMjudge se ings
ion o he es case un ha gene a ed i , bu we modi ied his beha io , as he
ile is cu en ly no being used by he judge. In o ma ion ega ding his ma e was
gi en o us h ough an email h ead wi h he de elope s, as o how i s ands a he
momen o de eloping his p ojec , i has no ye been included in o a main elease
o DOMjudge, hence i does no appea in i s documen a ion.
Any hing w i en du ing he compa e s age o his ile will be shown o use s
o he judge h ough he ex box. The message will con ain he pa sed ou pu
gene a ed o he s a ic analysis and ins umen a ion, i used. Essen ially, i will
display he conca ena ion o all he gene a ed eam messages. Un o una ely, his
emo es he possibili y o ha ing a message being shown pe es case un. I is a
design decision ha we would ha e p e e ed o a oid, bu we conside ha ha ing
a single message being shown is mo e con enien o his speci ic use case han
sepa a ing i pe un.
To be able o ac i a e he displaying o messages, a special op ion mus be
enabled in he DOMjudge se ings as an adminis a o . When enabling "Show
sample ou pu " unde "Gene al se ings"(see igu e 4.4), ins uc o s will be able o
use he desc ibed ea u es. Again, we would ha e p e e ed his ea u e o no be
linked o his op ion, bu he DOMjudge e sion being used de ec s when his op ion
is enabled o display he con en s o eammessage. x .
4.3.1. Ob aining he numbe o ies
In o de o make he displayed messages mo e in e es ing and in o ma i e, we
decided ha i would be a good idea o know he numbe o a emp ed submissions
o a s uden / eam o a gi en p oblem. This would allow us o c ea e a mo e gene ic
message o he i s a emp s, hin ing as o whe e he p oblem could o igina e om
i a submission is inco ec , and guide he s uden s u he as he a emp s inc ease.
We do no wan o gi e a solu ion o he p oblem s aigh away, bu a some poin
we mus guide he s uden somehow i hey canno ind a ix.
The numbe o a emp s o a p oblem can be ob ained making an API eques
o ob ain he sco eboa d, o wha he iden i ie o he con es is needed; his gen-
e a ed a se ies o p oblems in ol ing modi ica ions ha had o be applied o he
da abase schema in o de o make i accessible inside judgedaemon.main.php. The
documen a ion includes all he a ailable me hods o he API in an o ganized man-
ne . [8, 37, 13] The i s pa o ob aining he numbe o ies, in ol es making he
equi ed API eques om judgedaemon.main.php o ob ain he speci ic con es .
The s uc u e o he e u ned JSON ile o his call is as ollows:
{

4.3. P o iding eedback 33
"e en _id":"s ing",
" ime":"s ing",
"con es _ ime":"s ing",
"s a e": {
"s a ed":"s ing",
"ended":"s ing",
" ozen":"s ing",
" hawed":"s ing",
" inalized":"s ing",
"end_o _upda es":"s ing"
},
" ows": [
{
" ank":0,
" eam_id":"s ing",
"sco e": {
"num_sol ed":0,
" o al_ ime":0,
" o al_ un ime":0
},
"p oblems": [
{
"label":"s ing",
"p oblem_id":"s ing",
"num_judged":0,
"num_pending":0,
"sol ed": ue,
" ime":0,
" i s _ o_sol e": ue,
" un ime":0,
" as es _submission": ue
}
]
}
]
}
We a e in e es ed in he alue con ained as a pai o he “num_judged” ield. I
ep esen s he numbe o judged submissions o a p oblem so a , so i he cu en
a emp is he i s one, he alue e ie ed will be 0, and ice e sa. To e ie e i ,
we ha e o i e a e h ough he eams o a gi en con es o ob ain he cu en eam
and hen h ough he p oblems ha ha e been a emp ed by he eam.
Wha emains is o o wa d his in o ma ion o he es case_ un.sh sc ip ,
which makes i a ailable inside a dedica ed ile o he compa e execu able. I was
done his way o a oid modi ying wha he compa e execu ables ecei e as a gu-
men s, in case a cus omized compa e is decided o be used ollowing he de aul
DOMjudge con en ions.
34 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
This is he p ocess ha in ol es e ie ing he a emp s, and we needed he
con es iden i ie o do so; as al eady said, he API me hod ecei es a con es iden-
i ie o e u n he sco eboa d. A i s , we did no ha e di ec access o his alue,
being ou i s idea o ob ain i o e ie e he whole lis o con es s and i e a e
h ough i un il we ound he co ec one. A much be e solu ion was p o ided
o us hanks o he eam, ha is, modi ying he JudgeTask en i y class o add an
addi ional ield con aining his in o ma ion. judgedaemon.main.php includes he
JudgeTask en i y class, which is used o keep ack o he indi idual judge asks -
judgedaemon.main.php is called once pe judging ask. I uses his en i y class o
ob ain a numbe o di e en alues o make eques s o he API; he submission ID,
o ins ance.
To allow o hese changes o be applied, we made modi ica ions o he JudgeTask
en i y class in o de o add a con es _id p ope y index and column o be accessed
om judgedaemon.main.php. In o de o success ully upda e he con es iden i ie ,
we included a line in he unc ion in cha ge o adding he cu en submission iden i ie
o also add he cu en con es iden i ie o he en i y:
public unc ion ge Submi id(): ?in
{
$ his->con es idj =
$ his->submission?-%>ge Con es ()->ge Cid();,→
e u n $ his->submission?->ge Submi id();
}
This line ge s he cu en con es om he submission and assigns i s id o he
JudgeTask en y.
Nex , we had o go o he Sym ony6di ec o y and execu e he app op ia e
commands in o de o mig a e he da abase, ha is, e lec he changes made o
JudgeTask on he ac ual da abase. The base di ec o y o Sym ony is loca ed a
/webapp in he DOMse e . All da abase mig a ions a e loca ed inside he di-
ec o y webapp/mig a ions. A new ile was gene a ed in his di ec o y wi h he
co esponding change o he da abase schema.
Some debugging was equi ed du ing his p ocess o unde s and some o he
changes ha had o be made o allow his o happen. The mos ele an change
was modi ying he ile .en .local, adding an upda ed en i onmen a iable alue
ha is equi ed by Sym ony as pe hei documen a ion.7The ile now con ained
DATABASE_URL=mysql://DOMjudge:[email p o ec ed]:3306/DOMjudge?se e ⌋
Ve sion=ma iadb-10.11.6,→
This line speci ies he da abase URL, wi h he da abase use , passwo d, hos and
po numbe , as well as he da abase used and i s e sion in he ollowing o ma :
6Sym ony is an open-sou ce PHP amewo k. I is used o de elop so wa e ha simpli ies he
web design p ocessh ps://buil in.com/so wa e-enginee ing-pe spec i es/sym ony [41]
7h ps://sym ony.com/doc/cu en /doc ine.h ml [29]
4.4. P epa a ions and Se up 35
mysql://<use >:<passwo d>@<hos >:<po >/<da abase>?se e Ve sion ⌋
=< e sion>,→
All he ields can be ound in e c/dbpasswo ds.sec e excep o he e sion,
which is ob ained by execu ing mysql -V in he se e con aine . Once his was
applied, we could success ully make he desi ed mig a ion, e lec ing he changes in
he da abase.
4.3.2. Gene a ing he message
The las s ep o o wa d a comple e message is ac ually pa sing he s a ic an-
alyze s ou pu and pu ing he pieces oge he . This p ocess is ca ied ou om
he compa e sc ip , as i is now he la e s age o all he p e ious p ocesses.
I has access o he numbe o a emp s, he s a ic analyze s ou pu , and he
eammessage. x ile ha is la e used o o wa d he message o he s uden s.
To achie e his, we decided ha he bes app oach in ou case would be o euse
pa o he Py hon sc ip s ha we al eady had de eloped o he analysis in sec ion
3.2, and make he app op ia e call om he compa e sc ip o un hese pa se
sc ip s. The i s s ep was including he new sc ip s in DOMjudge, and make su e
ha hey a e included inside ou image. This was done by adding hem inside he
same di ec o y as he es case_ un.sh sc ip and adding hem o he Make ile
(judge/Make ile).
ins all-judgehos :
$(INSTALL_PROG)- $(DESTDIR)$(judgehos _libjudgedi )
compile.sh build_execu able.sh es case_ un.sh
ch oo -s a s op.sh ,→
check_di .sh e ic e sion_check.sh py hon_sc ip .py
Wha emains now is o o wa d hese sc ip s o he compa e wo king di ec o y,
which is simply achie ed om he es case_ un.sh ile as i has access o i . Each
sc ip will only be execu ed i he op ion ha selec s hei espec i e analysis ool
is indica ed in he heade o a es case. They will pa se he ou pu ile gene a ed
by he ool and ou pu a message o he eammessage. x ile. Fo ins ance, he
sc ip used o gene a e he message o ClangTidy is called pa se_clang.py, and
he ile ha i expec s as inpu is called clang idy.yaml, which is c ea ed when
Clang idy uns i s analysis on he sou ce code o he submission.
4.4. P epa a ions and Se up
We ha e explained e e y hing wi h ega d o he implemen a ion up o his poin ,
bu no how hey we e applied. All he changes we e es ed in a unc ional DOM-
judge en i onmen . Di ing a bi in o all he DOMjudge o e ings and eposi o ies,
we disco e ed ha he de elope s main ain a Docke 8image wi h all he equi e-
8Docke is an open pla o m o de eloping, shipping, and unning applica ions. I p o ides he
abili y o un an applica ion in an isola ed en i onmen called a con aine [18].
36 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
men s in o de o deploy a unc ional DOMjudge locally. I is specially in ended
o hose ha wan o collabo a e wi h he p ojec and need a es ing en i on-
men , so i wo ked pe ec ly; wi h an app op ia e use o i , i allowed us o ha e
he same wo king en i onmen eplica ed on all o ou machines. These images a e
a ailable in he public Docke eposi o y, Docke Hub, as DOMjudge/domse e and
DOMjudge/judgehos .
Be o e being able o s a he con aine s o hese images, an ins ance o a Ma i-
aDB da abase mus exis be o ehand. Fo una ely, i is e y easy o se up using
Docke , and he Docke Hub page o he DOMjudge i sel explains he s eps ha
mus be ollowed o deploy i , g an ing he commands ha mus be execu ed.
In o de o be able o un hese con aine s locally, Docke is equi ed o be
ins alled in he machines. An impo an aspec o conside is ha , as p e iously
men ioned, DOMjudge uses Linux Con ol G oups o cg oups o p ocess isola ion
in he judgedaemon. So, o enable cg oup equi emen s o DOMjudge, we ha e o
add cg oup memo y and swap accoun ing o he boo op ion. Addi ionally, mode n
Linux dis ibu ions ha e cg oup 2 enabled by de aul , and we ha e o speci y ha
we wan o boo he hos in o cg oup 19, which is he only one suppo ed by
DOMjudge. All he p e iously men ioned can be achie ed by adding he ollowing
in /e c/de aul /g ub:
GRUB_CMDLINE_LINUX_DEFAULT="quie cg oup_enable=memo y
swapaccoun =1 isolcpus=2 sys emd.uni ied_cg oup_hie a chy=0",→
The con en o his ile indica es o he sys em boo loade a sys em boo ime
wha a gumen s ha e o be passed o he sys em ke nel o con igu e he sys em.
A boo loade is a p og am which enables he selec ion o he ins alled ope a ing
sys em o ke nel o be loaded a sys em boo ime. I also allows he use o pass
a gumen s o he ke nel10. So, he code in he p e ious s ep is execu ed du ing
he boo p ocess o apply he DOMjudge equi emen s: boo ing in o cg oup 2. In
addi ion o his, cg oup memo y and swap accoun ing has also been added as pe
he ecommenda ions in he documen a ion [36].
The key GRUB_CMDLINE_LINUX_DEFAULT lis s commandline a gumen s o add
only o he de aul menu en y. [11] Commandline a gumen s a e addi ional inpu s
o wa ded o a command in o de o speci y iles o used, op ions o be enabled,
o any o he pa ame e equi ed o a command o execu e p ope ly. Many a e
op ional, and a e used o speci y a beha io o a command o he han he de aul
one. [38]
4.4.1. Ini ializing he con aine s
Ha ing his now con igu ed, we can ini ialize he di e en con aine s in o de
o ha e a unc ional DOMjudge ins alla ion. The i s s ep in he p ocess in ol es
9h ps://www.DOMjudge.o g/docs/manual/main/ins all-judgehos .h ml#
linux-con ol-g oups [36]
10h ps://access. edha .com/documen a ion/en-us/ ed_ha _en e p ise_linux/5/
h ml/ins alla ion_guide/s1-g ub-wha is [20]
4.4. P epa a ions and Se up 37
ini ializing he da abase, since he domse e equi es ha i exis s be o ehand.
This can be achie ed by unning he commands con ained in he guide ha he
DOMjudge eam has de eloped. By de aul , he con igu a ion exposes he se e on
po 13306 on he local machine.
Once he da abase is c ea ed, a domse e ins ance can be s a ed, indica ing
ha we wan o use he exis ing unning da abase o s o e in o ma ion. Ini ially, he
domse e will ha e no use aside om he adminis a o accoun , and no p oblems
o con es s a ailable.
Las ly, as many judgehos s as he ins alla ion equi es can be ini ialized. Judge-
hos s will be linked o he domse e ins ance and ecei e a di e en hos name o
each.
This is e e y hing necessa y in o de o ha e a unc ional DOMSe e an using
Docke . The web in e ace can be accessed on h p://localhos :12345/ , allowing
only o admin login a i s , as no o he accoun is p o ided by de aul . Files can be
edi ed di ec ly om he unning con aine by s a ing an in e ac i e shell session.
A main conside a ion is ha he judge a his s age lacks addi ional packages ha
could be necessa y in o de o make e ec i e modi ica ions, such as Vim as an
edi ing so wa e. They ha e o be ins alled in o de o make hem a ailable o be
used. O he packages and dependencies equi ed a e hose needed in o de o un he
s a ic analyze s, Clang-Tidy and Cppcheck, as GCC comes p eins alled. Ins alling
a ex edi o di ec ly, o example, is s aigh o wa d, as i can be di ec ly ins alled
h ough he in e ac i e shell session, along wi h any o he desi ed package. In o de
o be able o un he s a ic analyze s, we ha e o make hem accessible om he
ch oo en i onmen , as submissions a e e alua ed and execu ed he e. To do so, we
can execu e he sc ip ´bin/dj_ un_ch oo , which uns an in e ac i e shell o a
command inside he ch oo . Since ap is a ailable inside he ch oo , we can ins all
he equi ed packages using i . The name o he packages ha ha e o be ins alled
in o de o make he s a ic analyze s usable a e: clang- idy and cppcheck, and
he ollowing command ins alls hem:
ap -ge upda e && ap -ge ins all -y clang- idy cppcheck
This is no he mos con enien way o do so, and i can be con igu ed o ins all
e e y hing equi ed be o ehand, bu i will su ice o ou es ing pu poses a his
s age.
4.4.2. Building new images
Ea ly in he p ocess o applying ou changes, we ealized ha ce ain ea u es
ha we e a ailable in he cu en Gi Hub main b anch o DOMjudge we e no ye
a ailable in he la es elease. Among hese ea u es, he e was he eedback message
op ions, ha con ained all he changes ela ed o he eammessage. x ile ha we
used o ou implemen a ion. This mean ha he Docke con aine s ha we had
been using up o ha poin did no ha e his ea u e, as hey a e based on he la es
a ailable s able elease o DOMjudge. The implemen a ion o i has been discussed

38 Chap e 4. In eg a ion o s a ic and un ime analysis in DOMjudge
Figu e 4.5: Docke Hub eposi o ies
in a Gi Hub issue wi h he de elope s be o e,11 and he ea u e was inally included
on pull eques #2016.12 The e o e, we had o gene a e a new image om Gi Hub’s
sou ce code o be able o use i .
Fo una ely, he DOMjudge de elope s main ain a DOMjudge-packaging epos-
i o y on Gi Hub as pa o he p ojec . I con ains all he equi emen s o c ea e
Docke images which can be used o un DOMjudge, p o ided as pa o hei sup-
po o con ibu o s. We decided no o use i a i s because we could simply
deploy he published Docke images, and i wo ked jus ine o us a i s glance.
By de aul , he p o ided sc ip s se e e y hing up wi h main ained e sions ha do
no con ain he la es ea u es ha we need, being i he same one ha is ob ained
du ing ea lie Docke se ups. To allow o he use o cus omized e sions, we had
o make some changes o he sc ip s and download he s c code di ec ly om he
eposi o y and un ‘make dis ‘.13
An ad an age ha his app oach also p o ided is ha we could now speci y wha
packages we wan o include in he p ocess o c ea ing he image o be used, and
no ha e o ins all hem manually e e y single ime. We emo ed hem la e o he
inal e sion, as hey a e no longe equi ed.
An impo an no e o make is ha he la es Docke e sion elease has ea u es
ha a e incompa ible wi h he build p ocess, and we had o downg ade o Docke
e sion 24.0.6. A e all he changes we e made, we success ully gene a ed new
images o be used o ou es ings. We uploaded hem o a public Docke Hub
eposi o y in o de o make i accessible o all he membe s.
Apa om he eam messages inclusion, his e sion comes wi h a mo e e ined
looking UI and addi ional ea u es ha we would no use o he de elopmen o he
p ojec , bu a e ei he way nice o ha e.
4.4.3. Au oma ing he p ocess
A his s age, we decided ha i would be e y con enien o ha e an app op ia e
au oma ed me hod o deploy he en i onmen on all o ou local machines wi hou
he hus le o doing i manually. The logic pa h o ollow was he one o c ea ing a
Docke compose ile [17].
The s eps o ca y his ou in ol ed he c ea ion o a sc ip ha o ches a es
11h ps://gi hub.com/DOMjudge/DOMjudge/issues/991 [9]
12h ps://gi hub.com/DOMjudge/DOMjudge/pull/2016
13A make ile is a way o au oma ing so wa e building p ocedu e and o he complex asks wi h
dependencies. The dis ule in he gene a ed Make ile can be used o gene a e a gzipped a ile
and o he la o s o a chi e o dis ibu ion. [10] [26]
4.4. P epa a ions and Se up 39
he p ocess. The sc ip execu es he app op ia e docke -compose commands on
wo compose iles. The i s one c ea es he Ma iaDB da abase and DOMSe e
con aine s, and he second one c ea es he one o he judgehos , and hey a e
c ea ed in he men ioned o de , as he judgehos depends on he DOMSe e , which
depends on he da abase. In he end, we ob ain he same esul bu as e and in
a simple manne . Figu e 4.6 displays he gene a ed con aine s using he compose
iles. I success ully au oma es he con aine gene a ion p ocess.
Figu e 4.6: Con aine s iew om Docke Desk op
Chap e 5
Real use case scena io
Fo es ing and showcasing he pe o mance o he modi ied DOMjudge, we p e-
pa ed wo examples ha se e as a p oo o concep . Fi s ly, a es p oblem and
submission ha con ains a a ie y o ic i ious e o s o be de ec ed du ing he e al-
ua ion p ocess ha will se e as a showcase o he analysis ools ou pu s. Fo he
second example, we simula ed a ic i ious s uden submission ha a emp s o be
as ealis ic as possible.
5.1. Showcase
We ha e de eloped a minimal a i icial p oblem ha expec s he ollowing ou -
pu :
Co ec solu ion!
and a es case.cpp ile ha will be used as he submission. This submission ile
will igge se e al wa nings ha will be de ec ed by he s a ic analyze s and he
sani ize s. Fo his pu pose, we will indica e as a heade o he es case ha we
wan o pe o m a un wi h all he included ea u es. This is he con en o he es
case submission:
[...] // omi ed lines
oid *p;
in main() {
in ways[3];
o (in i= 0; i < 3; i++)
{
ways[i] =i;
}
ways[2]=ways[2];//sel Assignmen
i (ways[2]== 2);
41
48 Chap e 5. Real use case scena io
will s ill be able o know i he e is some hing o imp o e in hei codes.
5.2. Realis ic Case Scena io
Fo he ealis ic case scena io, we used a submission ha a emp s o be as close
o eali y as possible. To simula e eal beha io , e e y i e a ion o a submission will
apply a ix om he sugges ions. The submission is a emp ing o sol e a p oblem
p esen ed as ollows:
The e is a celeb a ion a a able in he shape o a ing whe e n gues s can si
inside and ano he n gues s ou side (wi h n≤10,000). The sequence o gues s
inside and he sequence o gues s ou side a e ixed, bu i is s ill possible o decide
how o align he wo sequences. Each pe son has a le el o en husiasm (an in ege
whose absolu e alue is less han 106), and he en husiasm o he celeb a ion is he
sum o he p oduc s o he en husiasms o he people who a e acing each o he a
he able. The p oblem consis s o inding he maximum possible en husiasm.
The solu ion o he p oblem is his o mula whe e 1and 2a e he en husiasm
a ays o he inside and ou side sequences o gues s:
max{
| 1|−1
X
i=0
1[i]· 2[i+k]|0≤k≤ | 1| − 1}
Two es cases a e included o he p oblem. The i s one will be lagged wi h
he heade used o apply all he ools simul aneously. Since he ou pu o a s a ic
analyze is independen o he es case inpu , we will indica e o he second es
case ha we only wan o un he ins umen a ion. This will a oid including he
same e o s and wa nings wice o he case o he s a ic analyze s.
Le us submi i s a aul y solu ion whose mo e ele an unc ion is he ollowing
sumP oduc :
using Anim =in ;
Anim sumP oduc ( ec o < Anim> 1, ec o < Anim> 2, in o se )
{
in numSea s = 1.size();
Anim o al = 0;
o (in i= 0; i <numSea s; ++i)
o al += 1[i] * 2[o se +i]; // A ay ou o bounds
access,→
e u n o al;
}
Once i is submi ed, we ge he ollowing gene a ed ou pu :

5.2. Realis ic Case Scena io 49
A heap o e low was de ec ed, his is usually due o an ou o
bounds a ay access bu can also be caused by epea ed o
big memo y alloca ions using he malloc unc ion o he new
keywo d.
,→
,→
,→
E o : signed in ege o e low
E o : signed in ege o e low
50 Chap e 5. Real use case scena io
A heap o e low was de ec ed, his is usually due o an ou o
bounds a ay access bu can also be caused by epea ed o
big memo y alloca ions using he malloc unc ion o he new
keywo d.
,→
,→
,→
con aining he e o :
A heap o e low was de ec ed, his is usually due o an ou o
bounds a ay access bu can also be caused by epea ed o
big memo y alloca ions using he malloc unc ion o he new
keywo d.
,→
,→
,→
hin ing a an a ay ou o bounds access. This clue may help us ind whe e he bug
is. In his example, he e is a missing modulo o ne e exceed he maximum numbe
o a ailable sea s. A e ixing i , we submi he code and ob ain a new esul .
Message: T y a oiding ope a ing be ween di e en ypes as i
can cause unexpec ed beha io ,→
E o : signed in ege o e low
E o : signed in ege o e low
In his second a emp , he e is no longe an ou -o -bounds access o he a ay,
bu we can s ill see he o he same message and e o s as in he p e ious a emp
since we ha e no add essed hem. No loca ion o he e o s a e being shown ye .
The same e o appea s wice since i is occu ing a wo di e en loca ions, bu we
do no know hem ye . We will make ano he submission ying o ix he issues.
Message: na owing con e sion om 's d:: ec o ::size_ ype'(aka
'unsigned long') o signed ype 'in 'is
implemen a ion-de ined, his is bes a oided as i can lead
o unexpec ed beha io
,→
,→
,→
E o : signed in ege o e low
Loca ion o he e o : Line: 15, Column: 18
E o : signed in ege o e low
Loca ion o he e o : Line: 15, Column: 9
Un o una ely, he p oblems we e no sol ed, bu we now ha e he loca ion o he
e o s and a mo e de ailed desc ip ion message. Inspec ing he “signed in ege
o e low” e o , we can see ha i is ound a line:
5.2. Realis ic Case Scena io 51
o al += 1[i] * 2[(o se +i) %numSea s];
The eason behind his e o igge is ha he alue gene a ed by he p oduc
is oo la ge o i inside he cu en 32 bi in ege a iable ha is used o s o e i
((106)2>231). Execu ion is no s opped despi e his. I is some hing ha should
be add essed, as i can lead o unde ined beha io . Obse ing he e u ned message,
we a e de e mined o make some changes in o de o achie e a be e solu ion in he
nex submission. The e o is caused because o he size o he ype used o he
alias decla a ion.
using Anim =in ;
The newly modi ied submission will ix i by speci ying ha we wan o use 64-bi
in ege s, g ea e han he 32-bi in ege used be o e.
using Anim =in 64_ ;
The gene a ed eam message is s ill shown e en when he submission is e alu-
a ed as co ec , as men ioned ea lie . This will allow o be able o de ec possible
pe o mance issues e en when eaching a igh solu ion ha compu e he co ec
esul . Wi h his in conside a ion, a new a emp is uploaded.
Message: na owing con e sion om 's d:: ec o ::size_ ype'(aka
'unsigned long') o signed ype 'in 'is
implemen a ion-de ined, his is bes a oided as i can lead
o unexpec ed beha io
,→
,→
,→
Message: na owing con e sion om ' Anim'(aka 'long') o
signed ype 'in 'is implemen a ion-de ined, his is bes
a oided as i can lead o unexpec ed beha io
,→
,→
We will a emp o change he code once mo e o y o elimina e he wa nings
and ge he co ec solu ion by changing he ype o he a iable sum used inside he
unc ion sol ecase() om in o Anim:
ec o < Anim>inne (numSea s), ou e (numSea s);
This is he e u ned message o he submission:
Message: na owing con e sion om 's d:: ec o ::size_ ype'(aka
'unsigned long') o signed ype 'in 'is
implemen a ion-de ined, his is bes a oided as i can lead
o unexpec ed beha io
,→
,→
,→
We did ge an accep ed e dic his ime. S ill, he wa nings a e displayed. I
is common o ha e some despi e w i ing p ope code, as many di e en checks a e
made. Wi h his es , we ha e es ed he co ec pe o mance o he modi ied judge
on a ealis ic s uden submission. E o s can some imes be puzzling, especially when
specialized language is used o desc ibe hem. In ou app oach, we ha e a emp ed
o ease he eadabili y o he wa nings.
Chap e 6
Conclusions and Fu u e Wo k
6.1. Discussion and conclusions
In his hesis we ha e made a s udy on wo di e en s a ic analyze s (Clang- idy
and Cppcheck), we ha e analyzed he ou pu s hese ools ga e us in a da ase o mo e
han 4000 submissions. A e he analysis, we ha e concluded which checks we e
mo e ele an . Then, we ollowed a simila p ocess o unde s and ins umen a ion,
we s udied ASan and UBSan. While hose s udies we e being made, we also s a ed
modi ying he DOMjudge pla o m, in o de o de elop a way o communica e o
he s uden he eedback we we e c ea ing. Finally, as we inished all hose asks,
we de eloped a didac ic, pe sonal eedback based on he diagnos ics issued o he
s uden submissions. This eedback is shown o he s uden ia eammessage (a
unc ionali y o DOMjudge) when hey submi hei code.
A e ixing Clang-Tidy, Cppcheck, ASan, and UBSan as a ge ools, we ana-
lyzed hei diagnos ics. The e a e no known s udies abou hese ools on s uden
submissions. So in his hesis we ha e made an exhaus i e analysis on all he diag-
nos ics ha could be aised in s uden submissions and made clea which diagnos ics
a e we a e ac ually going o communica e, excluding hose wi h li le o no ele ance
o a didac ic and use ul eedback.
We hen in eg a e he ools in o DOMjudge. To make i , we ha e eques ed aid
om he de elope s o he pla o m in o de o ix a ious issues ha a ose du ing
he pe inen modi ica ions o he pla o m ( o example o include an “ou da ed”
messaging sys em ha he pla o m p o ided “ eammessage”) and changes o he
assessmen p ocess o he online judge, speci ically, he compa e and un sc ip s
in o de o un he s a ic and dynamic analyze s o code on he submission. We
also needed o make some changes wi hin he DOMjudge PHP code and he HTML
empla es.
This p ocess concluded by gi ing ele an eedback o he s uden s abou hei
code, and aid ins uc o s on he assessmen o assignmen s (i seems impo an o
ema k ha we gi e he ins uc o 7 di e en ypes o modes in which he online
judge will un o no he s a ic and dynamic analyze s). Ou de elopmen s p o ide
an example o o he eaching ins i u ions in o de o implemen eedback based on
code analysis when needed on an online judge. Wi h he ex ension made on DOM-
53

54 Chap e 6. Conclusions and Fu u e Wo k
judge, we de ec and communica e mos e o s ega ding memo y and unde ined
beha io , and h ough lin ing and s a ic analysis we analyze o he p ope ies o he
code ha a e no de ec ed by ASan o UBSan.
6.2. Fu u e wo k
Du ing he de elopmen o his p ojec , we ha e come up wi h ideas and ea u es
ha we would ha e been deligh ed o implemen , bu exceeded he ocus o he
p ojec o could no include due o ime es ic ions.
Fi s ly, we would ha e lo ed o make he ea u es a ailable o o he languages
ou side o C++. The main eason behind no doing his was ha C++ is he main
language used in ou labo a o ies o he di e en subjec s, and adding suppo
o unused languages was no a p io i y. To do his, he judge should be able o
iden i y he language om wi hin he un sc ip o know wha s a ic analyze s and
ins umen a ion o un.
Ano he ea u e ha caugh ou in e es was including he in o ma ion o whe he
pas submissions had been accep ed o ejec ed by he judge and he eason o i .
This would ha e allowed us o de elop mo e eac i e messages and pe o m a mo e
complex message gene a ion. In he end, we did no include his ea u e due o ime
es ic ions.
As i was discussed in he s a ic analyze s conclusions, he analysis done on
s uden s submissions doesn’ di e om p oblem o p oblem, unning an analysis
speci ic o he p oblem and p o iding he main aine o he online judge an in e ace
o do his easily, would sol e his limi a ion. In [3], he au ho s p opose he CAC++
lib a y, cons uc ailo ed s a ic analysis p og ams o C/C++ assignmen s. By
in eg a ing i , we could be able o de ec he a ge ed logic e o s.
We also hough abou implemen ing he un op ions inside he es cases hem-
sel es. While unc ional, i is no he mos use - iendly app oach. An ideal solu ion
would in ol e adding op ions and bu ons in he use in e ace o selec and ac i a e
hese ea u es. I would equi e wo k ha alls ou side he scope o his p ojec , bu
would be posi i e o ha e.
Las ly, we would like o men ion ha ou u o ’s and o he p o esso s om ou
uni e si y ha e planned o use he DOMjudge in eg a ion de eloped in his hesis
on some subjec s in he compu e science school a UCM o nex yea . Indeed, a
eaching inno a ion p ojec (numbe 71 o he 2024/25 call) has been awa ded o
a g oup o people including us. To imp o e his p ojec i would be use ul o ha e
e en mo e s uden submissions o see in mo e de ail he mos common e o s among
s uden s. I will be use ul o also conside di e en kinds o p oblems and subjec s
( o example, we coded in pu pose a di ision by ze o o see i a check was aised,
bu we did no ind i aised in any o he +4000 submissions a ailable)
Wi h his hesis we ha e c ea ed a pa h o implemen many o he imp o emen s
o he eedback gi en by DOMjudge. This p ojec is scalable and modula , so
we expec ha in he close u u e we can see his hesis as he basis o a be e
au oma ed code assessmen a he Uni e sidad Complu ense de Mad id.
Chap e 7
Pe sonal Con ibu ions
In his sec ion, we will speci y he job pe o med by each o he eam membe s
inside his p ojec .
7.1. Rod igo Bu gos Sosa
Du ing he ini ial phase o he p ojec , I explo ed some o he di e en s a ic ana-
lyze s ha exis ed and he ones ha could be mo e bene icial o ou pu pose. When
looking a wha each o hem had o o e and decided o go o Clang and Cppcheck,
I ead h ough he di e en documen a ions p o ided and made su e o unde s and
how hey wo ked and he ins alla ion p ocess. Sona Qube and CodeChecke we e
o he o he op ions ha we e conside ed, he second one o i s C oss T ansla ion
Uni ea u e ha we ended up deciding no o include. Many o he a gumen s added
o he command lines o he s a ic analyze s we e chosen hanks o his p ocess.
Th ough my in es iga ion, I also ealized ha Clang-Tidy o e ed mo e alue o
us han Clang by i sel , as i an he same checks ha he la e , in addi ion o
some o i s own. I , oo, was simple o se up han Clang, and we decided o use
Clang-Tidy o hese easons.
To pe o m he ini ial execu ion o he analyze s wi h he s uden submissions,
I de eloped he sc ip s o un hem on all he submissions and ob ain he pa seable
iles in collabo a ion wi h Luis and Juan.
F om his poin onwa ds, I dedica ed my ime o he judge modi ica ions. Fi s ly,
I ead h ough hei documen a ion o unde s and how DOMjudge wo ked and he
a ailable ea u es. I will use his space o exp ess my g a i ude pe sonally o all he
DOMjudge de elope s o main aining such a comple e documen a ion. Speaking o
which, I was in cha ge o he communica ion wi h he de elope s, as he in o ma ion
hey p o ided a ec ed di ec ly my pa o he wo k. To do so, I subsc ibed o he
de elopmen mailing lis and exp essed my doub s h ough he e. The p ocess is
e y simple, one has o subsc ibe (done o coun e spam) and once he p ocess
is comple ed, I could send messages o he mailing lis . Communica ion wi h he
eam was key in o de o o e come some di icul ies o ou implemen a ion. Jaap
Elde ing and Michael Vasseu , wo o he main de elope s o he judge, answe ed
e e y hing we asked and guided some o ou hough s. I also explo ed e e y hing
55
56 Chap e 7. Pe sonal Con ibu ions
ha he p ojec had o o e , which is how we ound ou abou all he eposi o ies
specially aimed a o he de elope s a emp ing o make hei con ibu ion o he
judge.
Thanks o hese eposi o ies, I was able o build he new images ha we e used
du ing he implemen a ion p ocess. Despi e how well he p ojec is main ained, I
s ill an in o some issues when using his esou ces o se e e y hing up [42, 39]. Fo
ins ance, I had o make su e ha I enabled plugins o un when execu ing compose
as oo 1, since a a gi en poin , he op ion o plugins o un by de aul in Compose
was disabled 2. The Sym ony plugin was a ec ed by his, o ins ance. Also, some
changes had o be made o some sc ip s o add ea u es o op ions ha a e no he
de aul beha io , as men ioned elsewhe e in his hesis. A e y impo an ix ha
had o be made was he ac ha he web in e ace o ma ing a a ce ain poin was
no o ma ing p ope ly and he s yling was no loading, no only making he web
page isually wo se, bu also making he whole use expe ience a less enjoyable and
e en emo ing some in e ac ions, like he abili y o adding p oblems o a con es ,
since ce ain bu ons did no load. A message was sen o he eam and his was
a known issue ha some use s had epo ed al eady and a e wo king on. They did
sugges a possible ix, which should ha e been pa o he p ocess om he beginning
anyway, which was unning he Make dis on he code. This o una ely sol ed a
p oblem ha could ha e o he wise g ea ly de ac ed om he inal p oduc . All o
his equi ed a p ocess o in es iga ion and oubleshoo ing o ind he issues and
he solu ions; many o hem we e ound in Gi Hub and S ack O e low h eads and
discussions, and some o he s by di ing in o speci ic documen a ions. A he end,
mo e ime was spen oubleshoo ing and ixing issues ha I could ha e an icipa ed.
Once he images we e c ea ed co ec ly, I de eloped he compose iles ha we e
used la e by he eam o es ou he inal changes. Bu , hones ly, i was a ea u e
ha was mos ly used by mysel , so mo e han any hing, i was a quali y o li e
imp o emen . Ei he way, i allows anyone o easily deploy ou e sion o DOMjudge
wi h ease on hei machines.
In o de o use Docke on my machine, I downloaded Windows Subsys em o
Linux 2 (WSL2) 3since I pe sonally use Windows on i . This allowed he whole
se up o be easie and mo e con enien o use. I equi ed a bi o se ing up in
o de o make i wo k in synch oniza ion wi h Docke .
By a , he pa ha consumed mos o my ime was unde s anding how o
make he app op ia e changes inside he judge. Mos o my indings ha e al eady
been exposed h oughou his hesis, bu many o he di icul ies ha e no been
explained. One o he mos impo an miles ones was he p ocess o e ie ing
he numbe o submissions o a gi en p oblem by a s uden in o de o use i o
gene a e he eedback message. The p ocess in ol ed a lo o wo k o unde s and
and make co ec ly. Fi s ly, I had o esea ch how he API wo ked and whe e I could
1h ps://gi hub.com/api-pla o m/api-pla o m/issues/2610
2h ps://gi hub.com/compose /compose /issues/11839 [1]
3Windows Subsys em o Linux (WSL) is a ea u e o Windows ha allows o un a Linux
en i onmen on you Windows machine, wi hou he need o a sepa a e i ual machine
o dual boo ing h ps://lea n.mic oso .com/en-us/windows/wsl/abou h ps://lea n.
mic oso .com/en-us/windows/wsl/ u o ials/wsl-con aine s [24].
7.2. Juan T illo Ca e as 57
ind he equi ed in o ma ion, o hen iden i y he app op ia e place om whe e he
in o ma ion should be eques ed, landing luckily inside he judgedaemon.main.php.
Thanks o he DOMjudge de elope s, we unde s ood ha modi ying he JudgeTask
en i y would be e y bene icial, bu he change was no as s aigh o wa ds as
ini ially hough . A da abase mig a ion was, o cou se, equi ed o apply he change,
and his equi ed o unde s and a a su ace le el how o deal wi h Sym ony and
da abases. E en wi h his, he mig a ion p ocess s ill an in o issues because I had
o speci y he DATABASE_URL en i onmen a iable inside he .en .local ile. Many
hanks o he h eads ha helped me sol e he p oblems h oughou his p ocess.
Finding he mos app op ia e me hod o un he s a ic analyze s and ins umen-
a ion was no an easy ask ei he , and mul iple op ions and con igu a ions we e ied
in o de o land on he inal e sion. The p ocess was in cons an e olu ion, as a
new me hod would lead he way in o a emp ing o pe o m he same ac ion bu in
a simple o be e implemen ed manne . Figu ing ou wha speci ic changes had on
he e alua ion p ocess was ce ainly doub -inducing, and hei e ec would dic a e
he decisions aken. A some poin , compila ion o he code could be di ec ly pe -
o med wi h ins umen a ion in he compile sc ip , which mean ha code had o
be compiled only once pe es case (i necessa y), bu DOMjudge did no ole a e
wo king wi h wo di e en compila ion esul s i a es case pe o med compila ion
wi h ins umen a ion and ano he one did no , so we had o sc ap he idea. In he
end, he solu ion ound was a a less complica ed wo ka ound.
Pa s o he solu ion also in ol ed modi ying PHP and HTML code, which I
was no amilia wi h be o e beginning his p ojec . Many o he changes we e made
by obse ing sec ions o he code and ying o eplica e hem o achie e my goals.
Finally, I was in cha ge o w i ing in his hesis e e y hing ha had o do wi h
he modi ica ions made o he judge, apa o he common sec ions o he whole
eam (in oduc ion, conclusions, ...), which we e w i en by he h ee o us.
7.2. Juan T illo Ca e as
Du ing he i s phase o he p ojec , I coo dina ed he es o he pa icipan ’s
asks, while Rod igo explo ed supe icially some o he di e en s a ic analyze s and
Luis he sani ize s, I s udied ho oughly Cppcheck as i was he mos comple e and
he one I el was going o be mo e help ul o he eedback o he s uden s when
imp o ing he answe s o DOMjudge. A e ou i s explo a ion, I s a ed coding
oge he wi h Rod igo and Luis he ini ial bash o ge di e en submissions o be
examined by he di e en analyze s chosen. I made he inal modi ica ions o he
sc ip . A e comple ing he sc ip in bash we decided o make di e en Py hon
sc ip s o examine he “ aw” ou pu o he analyze s. Th ough his second phase we
had a ious changes in how o ead he ou pu o he ools as hey had di e en
op ions o he ou pu s. Fo example, a i s I w o e one sc ip o aking he
SARIF ou pu o Clang, bu a he end we decided o ake he YAML ou pu . All
he Py hon sc ip s we e going o be an by he ini ial bash sc ip , so I had o make
mo e changes o he ini ial sc ip as I p og essed wi h he o he sc ip s. I was in
cha ge o he main sc ip used o Clang and Cppcheck. A i s , we uni ed all he
64 BIBLIOGRAPHY
[15] An hony Heddings. Wha is ch oo on Linux and how do you use i ?, 2020.
u l:h ps://www.how ogeek.com/de ops/wha -is-ch oo -on-linux-
and-how-do-you-use-i /.
[16] Colin Hughes. P ojec Eule , 2001. u l:h ps://p ojec eule .ne .
[17] Docke Inc. Docke Compose o e iew | Docke docs, 2013. u l:h ps://
docs.docke .com/compose/.
[18] Docke Inc. Docke docs, 2013. u l:h ps://docs.docke .com/.
[19] Linux ke nel de elopmen communi y. Con ol g oups e sion 1, The Linux
Ke nel documen a ion, 2024. u l:h ps://www.ke nel.o g/doc/h ml/
la es /admin-guide/cg oup- 1/index.h ml.
[20] Rüdige Landmann, Jack Reed, e al. The GRUB loade . In Red Ha En e -
p ise Linux 5: Ins alla ion Guide. 2024. Chap e 9. u l:h ps://access.
edha .com/documen a ion/en-us/ ed_ha _en e p ise_linux/5/h ml/
ins alla ion_guide/ch-g ub.
[21] Kaibo Liu, Yudong Han, Jie M. Zhang, Zhenpeng Chen, Fede ica Sa o, Ma k
Ha man, Gang Huang, and Yun Ma. Who judges he judge: an empi ical s udy
on online judge es s. In René Jus and Go don F ase , edi o s, P oceedings o
he 32nd ACM SIGSOFT In e na ional Symposium on So wa e Tes ing and
Analysis, ISSTA 2023, Sea le, WA, USA, July 17-21, 2023, pages 334–346.
ACM, 2023.
[22] Anaga Mani, Di ya Venka a amani, Jo di Pe i Sil es e, and Sal ado Rou a
Fe e . Be e eedback o educa ional online judges. In P oceedings o he
6 h In e na ional Con e ence on Compu e Suppo ed Educa ion, Volume 2:
Ba celona, Spain, 1-3 Ap il, 2014, pages 176–183. SciTeP ess, 2014.
[23] Daniel Ma jamäki. Cppcheck, 2007. u l:h ps://sou ce o ge.ne /p/
cppcheck/wiki/Lis O Checks/.
[24] Mic oso . WSL documen a ion, 2016. u l:h ps://lea n.mic oso .com/
en-us/windows/wsl/.
[25] IOI o ganiza ion. In e na ional Olympiad in In o ma ics, 1989. u l:h ps:
//ioin o ma ics.o g.
[26] Sachin Pa il. Wha is a make ile and how does i wo k?, 2018. u l:h ps:
//opensou ce.com/a icle/18/8/wha -how-make ile.
[27] Jo di Pe i and Sal ado Rou a. Ju ge.o g, 2006. u l:h ps://ju ge.o g.
[28] Hen y Go don Rice. Classes o ecu si ely enume able se s and hei decision
p oblems. T ansac ions o he Ame ican Ma hema ical Socie y, 74(2):358–366,
1953.
[29] Sym ony SAS. Sym ony. High pe o mance PHP amewo k o web de elop-
men , 2005. u l:h ps://sym ony.com/.
[30] Ca los Sola Sas e. Desa ollo de una Pla a o ma O ien ada al Re ue zo en la
E aluación del Ni el de P og amación. Bachelo ’s hesis, Uni e sidad Poli éc-
nica de Mad id, 2020. u l:h ps://oa.upm.es/63124/.

BIBLIOGRAPHY 65
[31] UNICOM Sys ems. Pu i y, 1992. u l:h ps://www.unicomsi.com/p oduc s/
pu i yplus/.
[32] UNICOM sys ems. Quan i y, 1992. u l:h ps : / / www . unicomsi . com /
p oduc s/quan i y/.
[33] Clang Team. Add essSani ize — Clang 19.0.0gi documen a ion, 2012. u l:
h ps://clang.ll m.o g/docs/Add essSani ize .h ml ( isi ed on 2024).
[34] Clang Team. Clang-Tidy checks, 2024. u l:h ps : / /clang . ll m. o g /
ex a/clang- idy/checks/lis .h ml.
[35] Clang Team. Unde inedBeha io Sani ize — Clang 19.0.0gi documen a ion,
2012. u l:h ps://clang.ll m.o g/docs/Unde inedBeha io Sani ize .
h ml#ubsan-checks ( isi ed on 2024).
[36] DOMjudge eam. Domjudge manual, 2004. u l:h ps://www.DOMjudge.
o g/docs/manual/8.2/index.h ml. Manual o he DOMjudge.
[37] ICPC eam. Con es API, 2010. u l:h ps://ccs-specs.icpc.io/d a /
con es _api#in oduc ion. Con es API documen a ion.
[38] Tpoin Tech. Linux a gumen s, 2011. u l:h ps://www.ja a poin .com/
linux-a gumen s.
[39] use 3001829. How o copy iles om hos o Docke con aine ?, 2014. u l:
h ps://s acko e low.com/ques ions/22907231/how- o-copy- iles-
om-hos - o-docke -con aine .
[40] Szymon Wasik, Maciej An czak, Jan Badu a, A u Laskowski, and Tomasz
S e nal. A su ey on online judge sys ems and hei applica ions. ACM Com-
pu . Su ., 51(1):3:1–3:34, 2018. ep in : h ps:/ /a xi . o g /abs/ 1710.
05913.
[41] Alex Williams. Wha is Sym ony?, 2022. u l:h ps : / / buil in . com /
so wa e-enginee ing-pe spec i es/sym ony.
[42] Ca ch Zeng. Tool - Docke build no showing any ou pu om commands
(Docke ile RUN), 2016. u l:h ps://makeop im.com/en/ ool/docke -
build-no -ou pu /.
Appendix A
O line analysis de ails
A.1. Mos ele an Clang-Tidy diagnos ics
In Table A.1 we show he numbe o occu ences o each diagnos ic wi hin all o
he submissions, shown in he second column. Bu , in o de o a oid any kinds o bias
ela ed o diagnos ics appea ing mul iple imes wi hin a eam o a p oblem we added
wo o he columns. The hi d column coun s he numbe o eam’s submissions o a
p oblem which con ained he diagnos ic, and he ou h column con ains he numbe
o p oblems which ha e submissions con aining he diagnos ic. The las column
displays he numbe o ansi ions om no accep ed submissions o accep ed ones
when a diagnos ic is ixed. The able is o de ed by ansi ions o AC hen by
o al occu ences. I con ains he i s 40 a e emo ing checks conce ning a speci ic
p ojec , secu i y and concu ency, some speci ic checks ha a e no e y ele an
a e also emo ed, hey a e ela ed wi h e y opiniona ed s yle equi emen s ha
s uden s do no ollow.
Table A.1:
pe o mance-a oid-endl 564 263 48 18
misc-no- ecu sion 407 196 25 11
eadabili y-implici -bool-con e sion 156 69 19 8
bugp one-excep ion-escape 702 322 26 7
bugp one-na owing-con e sions 481 241 38 7
cppco eguidelines-p o- ype-membe -
ini 277 134 29 7
google-explici -cons uc o 204 118 15 7
bugp one-in ege -di ision 21 11 1 6
mode nize-use-nodisca d 314 159 21 5
Name Submission Team P oblem AC T ans
Con inued on nex page
67
68 Appendix A. O line analysis de ails
Table A.1: (Con inued)
eadabili y-make-membe - unc ion-
cons 134 85 14 5
cppco eguidelines-p e e -membe -
ini ialize 44 30 8 5
mode nize-loop-con e 184 94 25 4
mode nize-use- anspa en - unc o s 123 76 15 4
eadabili y-simpli y-boolean-exp 62 40 15 4
eadabili y- unc ion-cogni i e-
complexi y 52 16 9 4
bugp one-easily-swappable-pa ame e s 285 172 34 3
google- un ime-in 218 87 24 3
pe o mance- o - ange-copy 117 64 11 3
cppco eguidelines-a oid-cons -o - e -
da a-membe s 111 70 11 3
clang-diagnos ic-emp y-body 62 27 20 3
clang-analyze -deadcode.DeadS o es 57 32 15 3
bugp one-b anch-clone 56 25 12 3
mode nize- e u n-b aced-ini -lis 25 9 4 3
cppco eguidelines-use-de aul -
membe -ini 255 148 17 2
bugp one- ese ed-iden i ie 128 64 6 2
misc-non-p i a e-membe - a iables-in-
classes 107 52 15 2
pe o mance-unnecessa y- alue-pa am 78 56 21 2
mode nize-use-using 67 26 12 2
hicpp-use-emplace 49 32 16 2
misc-unused-pa ame e s 30 20 16 2
clang-diagnos ic-pa en heses-equali y 5 2 1 2
clang-diagnos ic-e o 91 57 18 1
clang-diagnos ic- e u n- ype 60 41 8 1
cppco eguidelines-non-p i a e-
membe - a iables-in-classes 48 34 10 1
bugp one-suspicious-semicolon 38 16 11 1
cppco eguidelines-mac o-usage 36 17 11 1
Name Submission Team P oblem AC T ans
Con inued on nex page
A.2. Cppcheck e o lis 69
Table A.1: (Con inued)
eadabili y-misleading-inden a ion 33 16 13 1
mode nize-pass-by- alue 24 16 9 1
mode nize-mac o- o-enum 21 11 11 1
Name Submission Team P oblem AC T ans
A.2. Cppcheck e o lis
He e is he lis o he mos ele an checks o Cppcheck ound and he easons
o why we chose hem:
missingRe u n (e o )
•F om 46 occu ences WA : 6.52%’, CE : 17.39%, RTE : 4.35%, TLE :
2.17%, AC : 69.57%.As he DOMjudge does only ely on ou pu o s a e
an AC, we can see how his esul is happening. Howe e his is a e y
poo coding p ac ice and could cause p oblems.
•"Found an exi pa h om unc ion wi h non- oid e u n ype
ha has missing e u n s a emen "
•I no handled will p obably cause e o s.
•Check should be communica ed al hough i may no cause any p oblems,
i aids he s uden in a didac ic way.
unini S uc Membe (e o )
•F om 28 occu ences WA : 57.14%’, CE : 21.43%, RTE : 10.71%, AC :
10.71%. P obably an unin ended e o .
•"Unini ialized s uc membe : ****"
•Pa o he s uc is no gi en ype / ini ialized by he p og amme bu
i was de ined by hemsel es his is qui e suspicious and should be com-
munica ed.
un eadVa iable (s yle)
•F om 300 occu ences WA : 36.12%’, CE : 7.02%, RTE : 11.71%,
TLE : 13.04%, AC : 32.11%.
•"Va iable ’**’ is assigned a alue ha is ne e used."
•No eaching a a iable (which means ha a a iable was no ead du ing
execu ion) is he consequence o : pa o he code is un eachable, he e is
an in ini e loop (TimeLimi Excep ion), he p og am canno be compiled
comple ely (CompileE o ) o a mix o hose easons.
•Should be communica ed in case o TLE , CE and WA.
shadowFunc ion/shadowVa iable/shadowA gumen (s yle)

70 Appendix A. O line analysis de ails
•Excep om shadowFunc ion ha has jus a 13.33% o AC he es ha e
a ela i ely high accep ance a e (a ound 40%) in compa ison.
•"Local a iable ’**’ shadows ou e unc ion/Va iable/A gumen "
•The p og amme may be inad e en ly using a name al eady used by he
p og am hinking ha hey a e equi ed o be connec ed when hey a e
no .
•Should be communica ed in case o no being AC.
knownCondi ionT ueFalse (s yle)
•F om 52 occu ences WA : 28.85%’, CE : 9.62%, RTE : 23.08%, TLE :
32.69%, AC : 5.77%.
•"Condi ion ’a bol.emp y()’ is always alse"
•Almos ne e a ises when he p og am is AC. Al hough his migh be a
conscious decision o he s uden , his can lead o a ious p oblems.
•Should be communica ed o maybe aiding he s uden in a didac ic way.
edundan Condi ion (s yle)
•F om jus 4: 3 a e RTE , and jus 1 is TLE .
•"Redundan condi ion:
j==0. ’j==0 || (j==0 AND i==-1)’ is equi alen o ’j==0’"
•Tigh ly ela ed o RTE and TLE (caused mainly by unconscious in ini e
loops).
mul iCondi ion (s yle)
•2 occu ences bo h o hem RTE .
•"Exp ession is always ue because ’else i ’ condi ion is
opposi e o p e ious condi ion a line 56."
•Tigh ly ela ed o RTE.
duplica eExp ession (s yle)
•F om 17 occu ences WA : 29.41%’, CE : 11.76%, RTE : 5.88%, TLE :
52.94%, AC :0%
•"Same exp ession on bo h sides o ’>’."
•Seems highly ela ed o TLE (In ini e loops wi h while condi ions) .
duplica eB eak (s yle)
•As one could imagine his check happens e y un equen ly in ou case
we ha e jus seen one case which was ac ually accep ed.
•"Consecu i e e u n, b eak, con inue, go o o h ow
s a emen s a e unnecessa y."
A.2. Cppcheck e o lis 71
•This is a bad coding p ac ice.
•Check should be communica ed al hough i may no cause any p oblems,
i aids he s uden in a didac ic way.
noCons uc o (s yle)
•Wi h jus 2 occu ences, one was TLE and he o he AC.
•""The class ’Supe me cado’ does no decla e a cons uc o
al hough i has p i a e membe a iables which likely
equi e ini ializa ion."
•Could cause p oblems (in ou da ase i lead o e o s).
•Should be communica ed in any case di e en om AC.
un eachableCode (s yle)
•Su p isingly his jus happens once, and i s AC, a e u he in es iga-
ion, we no iced ha his was because i was ela ed o he duplica eB eak
e o .
•"S a emen s ollowing ’ e u n’ will ne e be execu ed."
•Could cause p oblems as i can be highly associa ed wi h in ini e loops
o condi ions ha will ne e ac ually be me because o he code.
•Should be communica ed in any case di e en om AC.
knownEmp yCon aine (s yle)
•Jus one occu ence, an OLE e dic , as he example shows, i was ou -
pu ing an emp y esul wi h p obably a ixed ou pu o ma , and as i
was i e a ing o e i , i made i exceed he ou pu limi .
•"I e a ing o e con aine ’sol’ ha is always emp y."
•Will lead o p oblems, highly ela ed o OLE.
•Should be communica ed in any case di e en om AC.
cla i yCondi ion (s yle)
•Jus one occu ence in he da ase , su p isingly less han we o iginally
heo ized howe e i is ele an and he e dic was WA.
•"Boolean esul is used in bi wise ope a ion. Cla i y
exp ession wi h pa en heses."
•Will cause p oblems, highly ela ed o WA.
•Should be communica ed in any case di e en om AC.
unini a (wa ning)
•F om 14 occu ences WA : 28.57%’, RTE : 28.57%, TLE : 19.05%,
AC : 23.81%
72 Appendix A. O line analysis de ails
•"Unini ialized a iable: ***.h"
•No ini ialized a iable (i is a wa ning because as you can see is mainly
associa ed wi h heade iles w ongly in e p e ed).
sel Assignmen (wa ning)
•Jus 9 occu ences all o hem WA, as expec ed, his is no a common
e o bu i s almos always " a al" o he co ec ness o he answe .
•"Redundan assignmen o ’mane as[y]’ o i sel ."
•Highly associa ed wi h WA.
nega i eCon aine Index (wa ning)
•Jus 6 occu ences, 2 o hem caused an RTE which is no mal as i i
ac ually gi es a nega i e coin aine index i will esul in a un- ime
excep ion.
•"Ei he he condi ion ’adyacen es[i]!=an e io ’ is
edundan , o he wise he e is nega i e a ay index -1."
•Highly associa ed wi h RTE, migh cause un ime excep ions.
con aine Ou O Bounds (wa ning)
•WA : 4.76%’, CE : 23.812%, RTE : 66.67%, AC : 4.76%. These esul s
makes sense, emembe we a e ea ing wi h s a ic analysis so his e o
a ises when a a iable ies o access a pa o a a iable which can be
emp y o ou o bounds, depending on he ac ual alue o bo h a iables,
his can lead o di e en accesses esul ing majo ly in RTE.
•"Ei he he condi ion ’****[0].emp y()’ is edundan o
exp ession ’****[0]. on ()’ cause access ou o bounds."
•Highly associa ed wi h RTE, migh cause un ime excep ions.
iden icalInne Condi ion (wa ning)
•Jus one occu ence, de i ed in an AC e dic , i s de ini ely no an in-
ended e o so i mus be no i ied.
•""Iden ical inne ’i ’ condi ion is always ue."
•P obably no in ended.
•Check should be communica ed al hough i may no cause any p oblems,
i aids he s uden .
cons S a emen (wa ning)
•F om 7 occu ences WA : 14.29%’, CE : 28.57%, RTE : 28.57%, TLE :
14.29%, AC : 14.29%.I depends on he case.
•Case 1
A.2. Cppcheck e o lis 73
◦"Redundan code: Found a s a emen ha begins wi h bool con-
s an ."
◦Highly associa ed wi h WA
◦Should be communica ed as i can cause p oblems.
•Case 2
◦"Redundan code: Found unused a ay access."
◦Unused a iable
◦No ele an o eedback.
Be o e going any u he , now we a e going o analyze pe o mance e o s,
hese imply always a bad/poo coding p ac ice which means ha hey migh
no imply a judge e dic , howe e al hough no use ul o he s uden o
co ec hei answe , his kind o e o s a e e y use ul o hem o in eg a e
good coding p ac ices o hei codes.
passedByValue (pe o mance)
•F om 94 occu ences WA : 17.58%’, CE : 3.3%, RTE : 17.78%, TLE :
28.57%, AC : 30.77%. Jus a poo coding p ac ice, could deli e any
kind o e dic , bu i no he main eason o any o hem.
•"Func ion pa ame e ’ allaNecesi amos’ should be passed by
cons e e ence."
•Poo coding p ac ice.
•Check should be communica ed al hough i may no cause any p oblems,
i aids he s uden in a didac ic way.
s lFindInse (pe o mance)
•F om 22 occu ences WA : 10%’, CE : 5%, RTE : 5%, TLE :5%, AC :
75%. Poo coding p ac ice, does no imply any e dic . Howe e i could
be co ec ed.
•"Sea ching be o e inse ion is no necessa y. Ins ead o
’pe sonas[]=indPe sona’ conside using . yemplace
•Poo coding p ac ice.
•Check should be communica ed al hough i may no cause any p oblems,
i aids he s uden in a didac ic way.
i e a eByValue (pe o mance)
•F om 8 occu ences WA : 14.29%’, CE : 28.57%, TLE : 28.57%, AC :
28.57%. Poo coding p ac ice, does no imply any e dic . Howe e i
could be co ec ed.
•"Range a iable ’cancion’ should be decla ed as cons
e e ence."
80 Appendix B. Ins uc ions o Docke se up
docke un -i --p i ileged - /sys/ s/cg oup:/sys/ s/cg oup: o
--name judgehos -0 --link domse e :domse e --hos name
judgedaemon-0 -e DAEMON _ID=0 -e
JUDGEDAEMON _PASSWORD=<judgehos passwo d>
domjudge/judgehos :la es
,→
,→
,→
,→
Whe e <judgehos passwo d> mus con ain he alue e ie ed ea lie . This can
be un mul iple imes o c ea e as many ins ances o judgehos as necessa y. The
name and hos name mus be changed o allow o his o each ins ance.