Node co-activations as a means of error detection : Towards fault-tolerant neural networks

This is a sel -a chi ed e sion o an o iginal a icle. This e sion may di e om he o iginal in pagina ion and ypog aphic de ails. Au ho (s): Ti le: Yea : Ve sion: Copy igh : Righ s: Righ s u l: Please ci e he o iginal e sion: CC BY 4.0 h ps://c ea i ecommons.o g/licenses/by/4.0/ Node co-ac i a ions as a means o e o de ec ion : Towa ds aul - ole an neu al ne wo ks © 2022 he Au ho s Published e sion Myllyaho, Lalli; Nu minen, Jukka K.; Mikkonen, Tommi Myllyaho, L., Nu minen, J. K., & Mikkonen, T. (2022). Node co-ac i a ions as a means o e o de ec ion : Towa ds aul - ole an neu al ne wo ks. A ay, 15, A icle 100201. h ps://doi.o g/10.1016/j.a ay.2022.100201 2022 A ay 15 (2022) 100201 A ailable online 10 June 2022 2590-0056/© 2022 The Au ho (s). Published by Else ie Inc. This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/). Con en s lis s a ailable a ScienceDi ec A ay jou nal homepage: www.else ie .com/loca e/a ay Node co-ac i a ions as a means o e o de ec ion—Towa ds aul - ole an neu al ne wo ks Lalli Myllyaho a,∗, Jukka K. Nu minen a, Tommi Mikkonen b aUni e si y o Helsinki, Finland bUni e si y o Jy äskylä, Finland A R T I C L E I N F O Keywo ds: Machine lea ning Faul ole ance Neu al ne wo ks E o de ec ion Concep d i Dependabili y A B S T R A C T Con ex : Machine lea ning has p o ed an e icien ool, bu he sys ems need ools o mi iga e isks du ing un ime. One app oach is aul ole ance: de ec ing and handling e o s be o e hey cause ha m. Objec i e: This pape in es iga es whe he a e co-ac i a ions – pai s o usually seg ega ed nodes ac i a ing oge he – a e indica i e o p oblems in neu al ne wo ks (NN). These could be used o de ec concep d i and lagging un us wo hy p edic ions. Me hod: We ained ou NNs. Fo each, we s udied how o en each pai o nodes ac i a es oge he . In a sepa a e es se , we coun ed how many a e co-ac i a ions occu ed wi h each inpu , and g ouped he inpu s based on whe he i s classi ica ion was co ec , inco ec , o whe he i s class was absen du ing aining. Resul s: Ra e co-ac i a ions a e much mo e common in inpu s om a class ha was absen du ing aining. Inco ec ly classi ied inpu s a e aged a la ge numbe o a e co-ac i a ions han co ec ly classi ied inpu s, bu he di e ence was smalle . Conclusions: As a e co-ac i a ions a e mo e common in unp eceden ed inpu s, hey show po en ial o de ec ing concep d i . The e is also some po en ial in de ec ing single inpu s om un ained classes. The small di e ence be ween co ec ly and inco ec ly p edic ed inpu s is less p omising and needs u he esea ch. 1. In oduc ion Machine lea ning (ML) models a e s a is ical app oxima ions, whim- sical and cap icious in na u e, and o en made o en i onmen s ha e ol e o e ime. In such app oxima ions, a 99% accu a e model – some hing ha is p ac ically always co ec – is w ong 1% o he ime. Wha should be done i ha 1% happens and causes e o s in you sys em? A e he e ways o mi iga e he isk and p epa e a so wa e sys em o he ine i able ‘‘bad days’’ o you model? The us wo hi- ness o ML sys ems has been imp o ed, o example, by es ablishing pa e ns o aul ole ance, bu ools o measu ing whe he a model’s esul s a e and emain us wo hy can s ill be imp o ed [1]. Fu he - mo e, such de ec ion should ideally no only be an a e hough , bu de ec ion should occu in eal ime while he model is unning. Du ing compu a ion uns, one app oach o mi iga ing he isk and making he sys em mo e aul - ole an could be moni o ing he model’s own inne s uc u e. The inne s uc u e o a neu al ne wo k – a cu en ly common ML echnique – is some imes compa ed o he s uc u e o biological neu al ci cui y (e.g. Abiodun e al. [2]). Like biological neu al ci cui y and ∗Co esponding au ho . E-mail add esses: [email p o ec ed] (L. Myllyaho), [email p o ec ed] (J.K. Nu minen), [email p o ec ed] (T. Mikkonen). neu ons, neu al ne wo ks in compu ing consis o laye s o in e con- nec ed nodes. These nodes a e iny compu a ional uni s ha ecei e an inpu , and ei he ac i a e and pass on an ou pu o he ollowing nodes o emain do man wi h an ou pu o 0, ha ing no e ec on he compu a ions made by he ollowing nodes. We know om p e ious esea ch on neu al ne wo ks ha a e ne wo k aining, speci ic g oups o nodes end o be esponsible o speci ic ou comes and hus o en ac i a e concu en ly [3]. Fo example, in an image ecogni ion model, ce ain g oups o nodes can be expec ed o ac i a e when he image o a dog is shown, while a leas a pa ially di e en g oup should ac i a e o he image o a ca . Ac i a ions ha e been s udied in he con ex o es ing neu al ne wo ks (e.g. [3–6]), bu use in mi iga ing isks du ing un ime has been sca ce [1]. Howe e , wha i he ac i a ing nodes a e suddenly ones ha usu- ally do no ac i a e oge he and hus do no belong o a same g oup (c . Fig. 1)? Can some hing be in e ed om his? Do hese a e co- ac i a ions wi hin a neu al ne wo k indica e ha he compu a ion esul is inco ec o ha he inpu has ne e been seen be o e? I so, could a e co-ac i a ions be used o de ec e o s in neu al ne wo ks, h ps://doi.o g/10.1016/j.a ay.2022.100201 Recei ed 29 Ma ch 2022; Recei ed in e ised o m 30 May 2022; Accep ed 3 June 2022 A ay 15 (2022) 100201 2 L. Myllyaho e al. Fig. 1. Illus a ion o ac i a ion pa e ns in a simple neu al ne wo k. Colou ing indica es an ac i a ed node. Mu a ed ac i a ion pa e n on he a igh . Pic u e o he dog cou esy o Helen Lopez h ps://www.pexels.com/pho o/sho -coa ed- an-dog- 2253275/. (Fo in e p e a ion o he e e ences o colou in his igu e legend, he eade is e e ed o he web e sion o his a icle.) p e en hem om p opaga ing and causing ailu es, and, hus, inc ease he aul ole ance o ML sys ems o mi iga e inhe en isks? To es ou hypo heses, we ain ou neu al ne wo ks. Fo e e y node in a neu al ne wo k, we calcula e how o en each node ac i a es concu en ly wi h e e y o he node using he aining da a: i.e. how p obable i is ha wo nodes ac i a e concu en ly. This way, we ob ain a me ic o which nodes o en con ibu e oge he o he ne wo k ou pu and hus belong o one o mo e o he same g oups. Using a sepa a e es da a se , we coun he numbe o a e co-ac i a ions happening wi hin he neu al ne wo k o each inpu and ma k down whe he he ne wo k’s ou pu was co ec , inco ec , o i he inpu belonged o a class ha was no p esen in he aining se . Once he es s ha e been un, we de e mine whe he he numbe o a e co-ac i a ions di e s a is ically be ween he ollowing scena ios: (1) es cases o which he ou pu was co ec , (2) cases whe e he ou pu was inco ec , and (3) cases whe e he ne wo k was no ained o he inpu . Based on ou da a, we hen es ima e how well a e co-ac i a ions would i in mi iga ing h ee speci ic majo isks ha a e o en p esen in ML sys ems: d i in incoming da a, single inpu s he model canno handle, and inaccu a e p edic ions [1]. La ge numbe s o a e co-ac i a ions indica e p oblems in p edic- ions. Ra e co-ac i a ions a e, on a e age, much mo e common in inpu s om un ained classes han in inpu s he model has been ained o . Thus, a e co-ac i a ions show good po en ial in de ec ing d i in incoming da a: should he a e age numbe o a e co-ac i a ions inc ease, d i is mos likely imminen . Howe e , inpu s om ained classes con ained ou lie s wi h a high occu ence numbe o a e co- ac i a ions as well, and some un ained inpu s ha e a low numbe o occu ences. Thus, de ec ing inpu s ha he model canno handle and p e en ing hem om being used u he down in he sys em is mo e p oblema ic. Conside ing he di e ence in he a e age numbe o occu ences, i may be possible o ind sys ems and con ex s whe e using i is easible, bu he sys em should be able o deal wi h some alse posi i es and nega i es. Addi ionally, a e co-ac i a ions ended o be mo e common in inco ec ly p edic ed inpu s han in co ec ly p edic ed ones, bu he di e ence was bo h smalle and s a is ically less signi ican . Thus, de ec ing single inaccu a e p edic ions may no be easible based on he numbe o occu ences alone, bu he app oach should a leas be ine- uned o ind he mos indica i e co-ac i a ions. This pape is o ganized as ollows: Sec ion 2desc ibes key concep s o sys em dependabili y, aul ole ance, and neu al ne wo ks, along wi h p e ious wo k on ac i a ion pa e ns in neu al ne wo ks and hei u iliza ion in es ing and moni o ing he ne wo ks. Sec ion 3in oduces he no el concep s in de ail and desc ibes ou goals and esea ch ques ions. Sec ion 4desc ibes ou expe imen al se -up, how da a was Fig. 2. A simple neu al ne wo k, mo e p ecisely, a mul i-laye pe cep on. collec ed and he me hods o analysis. Resul s can be ound in Sec ion 5. Sec ion 6discusses he esul s, while Sec ion 7discusses he s udy alidi y. Sec ion 8concludes he pape . 2. Backg ound 2.1. Dependabili y and aul ole ance The dependabili y o a sys em means i s us wo hiness [7]. De- pendabili y is usually assessed by e alua ing a sys em’s eliabili y, a ailabili y, and main ainabili y. Essen ially, a dependable sys em – a he e y leas – deli e s co ec se ice consis en ly, does no su e om long pe iods o down ime, and is easily co ec ed and al e ed. Sys em dependabili y is h ea ened by ailu es,e o s, and aul s [8]. Failu es a e de ia ions om a desi ed se ice. They a e caused by p opaga ing e o s made by he sys em, i.e. inco ec unc ioning o he sys em. E o s a e caused by aul s ha a e de ec s in sys em componen s (so wa e o ha dwa e), ac i a ed by gi en inpu s in a gi en s a e. Faul ole ance is one ool o diminishing hese h ea s [8]. Faul ole ance aims o a sys em design ha can p e en occu ing e o s om p opaga ing in he sys em and causing ailu es by de ec ing he e o and handling i be o e u he damage is done. The need o aul ole ance in ML sys ems has la ely been ecognized mo e [1]. This is in no small pa due o he na u e o he ML models hemsel es. Acco ding o Myllyaho e al. [1], he p oblems in ML sys ems o en o igina e om inaccu acies ha he models hold, along wi h hei p oneness o so- called concep d i . Tha is, an ML model is ained and, ideally, i can gene alize wha i has lea ned o all da a ha a e simila o he aining da a. Howe e , he gene aliza ion a ely, i e e , is success ul enough o each a 100% p edic ion accu acy in new da a in he i s place, and he model a ely is able o handle da a ha a e as ly di e en om he aining da a. Thus, ML sys ems can be seen as inhe en ly aul y because o hei app oxima e na u e [9], and an ini ially adequa e sys em can e ode and become aul y o e ime [10]. To add insul o inju y, e oneous beha iou is o en di icul o de ec [1], which is why we ocus on he de ec ion phase o aul ole ance in his pape . Resea ch on aul ole ance in ML sys ems has mainly ocused on a ious inpu and ou pu obse e s and model edundancy [1]. This means ha , o example, changes in inpu s and ou pu s can be moni- o ed, unaccep able alues a e handled di e en ly, and he sys em may con ain mul iple models ha handle inpu s wi h some o ches a ion. The inne wo kings and s uc u es o he models, howe e , a e a ely u ilized in e o de ec ion o achie e aul ole ance. 2.2. Neu al ne wo ks The s uc u e o neu al ne wo ks consis s o node laye s [11] (see Fig. 2). The p e ious laye is connec ed o he nex one. Tha is, when a node ecei es an inpu , i ei he ac i a es and passes on an ou pu o A ay 15 (2022) 100201 3 L. Myllyaho e al. he nex laye o nodes o emains do man and, in p ac ice, ou pu s 0, hus ha ing no e ec on he ollowing compu a ions. The echnique esponsible o he ac i a ion is an ac i a ion unc- ion [12]. A ec i ied linea uni (ReLU) is a commonly used ac i a ion unc ion. ReLU e y closely ollows he philosophy o ei he ac i a ing o emaining do man . Ma hema ically ReLU is usually o mula ed as 𝑓(𝑥) = max(0, 𝑥). In p ac ice, his means ha i he inpu a node ecei es is nega i e o 0, i ac ually does no ha e an e ec on he ollowing compu a ions, bu i he inpu s a e e y s ong, he e ec he node has on he ollowing laye is also s ong. Acco ding o Sha ma e al. [12], ReLU has p o ed o be e y e ec i e and is one o he mos used ac i a ion unc ions oday. 2.3. Rela ed wo k In neu al ne wo ks, a ious g oups o nodes end o ake esponsi- bili y o di e en ou comes [4]. In hei wo k, Tian e al. showed ha di e en g oups o nodes in a neu al ne wo k o au onomous d i ing ended o ac i a e based on whe he he neu al ne wo k p oposed u ning o he le o o he igh . Xie e al. [5] also sugges ha ans o ming a es inpu oo much will lead o a de o med ac i a ion pa e n and a w ong esul , sugges ing ha he mu a ed pa e n is ela ed o he inco ec esul . The ac i a ions ha e been used in esea ch conce ning he es ing o neu al ne wo ks (e.g. [3–6]). Usually his means inding nodes ha ha e no ac i a ed du ing es ing o explo ing imp o ed me hodologies o c ea ing es cases o ind such nodes. This is oo ed in he idea ha so-called ‘‘neu on co e age’’ is ela ed o code and s a emen co e age in adi ional so wa e: i he neu on has no ac i a ed du ing es ing, he e ec o ha neu on is no known [3]. Howe e , ac i a ion- ela ed e o de ec ion measu es ha e no been used widely and consis en ly in p ac ice o achie e aul ole ance [1]. Tha is, ac i a ions ha e been moni o ed o ini ially es he model p io o deploymen bu no o con inuously alida e he sys em du ing un ime. The idea has aised some in e es in p ac i ione s, bu how he ac i a ions should ac ually be moni o ed o de ec e o s and wha conclusions should be made based on hem has emained unclea [1]. Nume ous a emp s ha e also no been made on he esea ch side, as we a e awa e o only one pape a emp ing o build aul ole - ance by speci ically u ilizing ac i a ion moni o s. In hei wo k, Cheng e al. [13] o m a pa e n om he ac i a ions o he penul ima e laye o each class. I he model p edic ion di e en ia es oo much om he p e iously es ablished pa e n, he ou pu is lagged as po en ially e oneous. This shows p omise in de ec ing some misclassi ica ions. Howe e , hey only ocus on he penul ima e laye and a ce ain subse o he nodes hey conside o be he co e nodes a ec ing he ou pu s. Thus, hey do no en e ain he idea o how ac i a ions in he ea lie laye s o ou side he co e se beha e. Also, he ocus is on immedia e e o de ec ion, and how he ac i a ions beha e ac oss a ious scena ios (i.e. whe he he ou pu was co ec , inco ec , o some hing he model was no ained o ) is no add essed. Thus, which ypes o ailu es he ac i a ion moni o s a e e ec i e agains emains unce ain, as does whe he hey could also be u ilized when moni o ing concep d i . 3. Concep s and goals In his sec ion, we in oduce he concep o a e co-ac i a ions, a no el app oach o es ima e he ypicali y o ac i a ion pa e ns in a neu al ne wo k. Ou goal is o show ha co ec ly p edic ed inpu s di - e om p oblema ic inpu s wi h ega ds o a e co-ac i a ions wi hin he ne wo k. Thus, a ypical ac i a ion pa e ns would indica e un us - wo hy p edic ions. I his is he case, moni o ing a e co-ac i a ions would show po en ial in e o de ec ion in neu al ne wo ks. Fi s , we desc ibe he concep o a e co-ac i a ions in de ail in Sec ion 3.1. Then, we discuss he mo i a ion o ou esea ch goal and p esen ou esea ch ques ions in Sec ion 3.2. 3.1. Co-ac i a ion a e & Ra e co-ac i a ions As desc ibed in Sec ion 2, nodes in neu al ne wo ks ei he ac i a e o emain do man , and hese ac i a ions o m pa e ns esponsible o ce ain ou pu s. I wo nodes belong o one o mo e o he same pa e ns, hey could be expec ed o ac i a e oge he a ai sha e o he ime. To es ima e whe he wo nodes do no belong o any o hese pa e ns, we p esen he idea o co-ac i a ion a e: how likely is i ha node 𝑚ac i a es when node 𝑛ac i a es. Mo e speci ically, o e e y node 𝑛in a neu al ne wo k, we calcula e i s co-ac i a ion a e wi h node 𝑚in a se o inpu s 𝐼as 𝑟𝑎𝑡𝑒(𝑛, 𝑚) = 𝐼 ∑ 𝑖=1 𝑛𝑖∩𝑚𝑖 𝐼 ∑ 𝑖=1 𝑛𝑖 , whe e 𝑟𝑎𝑡𝑒(𝑛, 𝑚)is he co-ac i a ion a e o node 𝑛wi h node 𝑚, and 𝑛𝑖, 𝑚𝑖= 1 i nodes 𝑛and 𝑚ac i a e wi h inpu 𝑖and o he wise 𝑛𝑖, 𝑚𝑖= 0. Algo i hmically, he co-ac i a ion a es o a se o inpu s 𝐼can be calcula ed wi h Algo i hm 1. Using he algo i hm, we will end up wi h a wo-dimensional a ay a es, om which he co-ac i a ion a es o nodes 𝑛and 𝑚can be ound as, in ac , a es[𝑛][𝑚] = 𝑟𝑎𝑡𝑒(𝑛, 𝑚). The ime complexi y o Algo i hm 1is 𝑂(𝑖𝑛2), whe e 𝑖is he numbe o inpu s in 𝐼, and 𝑛is he numbe o nodes in a neu al ne wo k 𝑁𝑁. Algo i hm 1 Co-ac i a ionRa es(𝑖𝑛𝑝𝑢𝑡𝑠,𝑁𝑁) inpu s: Se o inpu s in which he co-ac i a ion a es a e calcula ed NN: Neu al ne wo k o which he co-ac i a ion a es a e calcula ed 1: a es = [][]: an a ay in which o s o e he co-ac i a ion a es 2: o all 𝑖in 𝑖𝑛𝑝𝑢𝑡𝑠 do 3: o all node 𝑛in 𝑁𝑁 do 4: i 𝑛ac i a es wi h 𝑖 hen 5: o all node 𝑚in 𝑁𝑁 do 6: i 𝑚ac i a es wi h 𝑖 hen 7: a es[𝑛][𝑚]+= 1 8: end i 9: end o 10: end i 11: end o 12: end o 13: o all node 𝑛in 𝑁𝑁 do 14: o all node 𝑚in 𝑁𝑁 do 15: a es[𝑛][𝑚] = a es[𝑛][𝑚] / a es[𝑛][𝑛] 16: end o 17: end o 18: e u n 𝑟𝑎𝑡𝑒𝑠 To p oduce meaning ul esul s, he se o inpu s used o calcula e he co-ac i a ion a es should be chosen app op ia ely. The app oach we chose was o calcula e he co-ac i a ion a es a e he ne wo ks we e ained and use he same aining se ha was used o ain hem. In his way, he co-ac i a ion a es should ep esen he ac i a ion pa e ns o he inpu classes ha he ne wo k should be able o gene alize o. Thus, co-ac i a ion a es desc ibe he inne wo kings o he neu al ne wo ks in cases whe e he ne wo k can easonably be expec ed o handle co ec ly, whe eas cases ha ha e no ep esen a ion in he aining se may no p oduce good esul s. The ac i a ion pa e n we s udy in his pape is de i ed om co- ac i a ion a e: a a e co-ac i a ion occu s when wo nodes wi h a low co-ac i a ion a e ac i a e wi hin a neu al ne wo k du ing a p edic ion. Thus, a a e co-ac i a ion is an indica ion o such compu a ions ha no mally do no occu du ing a p edic ion. The mo e hese a e co- ac i a ions occu , he mo e disjoin ed he ac i a ion pa e n is om A ay 15 (2022) 100201 4 L. Myllyaho e al. Table 1 Models used o es hypo heses. Model Fil e ed class Numbe o ou pu s Accu acy in es se wi hou he il e ed class CNN-ankle boo 9 (ankle boo ) 10 91.8% CNN-ankle boo 9 9 9 91.7% CNN-shi 6 (shi ) 10 95.6% MLP-ankle boo 9 10 88.4% ac i a ion pa e ns ha ha e occu ed wi hin he ne wo k be o e, and mo e a ypical i is. In his s udy, we a e looking in o he connec ion be- ween he p oblema ic p edic ions and he numbe o a e co-ac i a ion occu ing when he p edic ion is made. 3.2. Resea ch goal and ques ions The goal o ou esea ch is o show ha a ypical ac i a ion pa e ns indica e un us wo hy p edic ions. To build dependable sys ems, a gene al need cu en ly exis s o aul ole ance in ML sys ems. How- e e , app oaches u ilizing node ac i a ions o de ec e o s in neu al ne wo ks ha e no been ex ensi ely s udied ega dless o hei ole in he compu a ion p ocess. The easoning we ha e he e is ha by showing ha ac i a ion pa e ns – a e co-ac i a ions in ou case – beha e di e en ly in co ec and p oblema ic p edic ions, we can a gue ha obse ing he ac i a ion pa e n has po en ial in e o de ec ion. In his pape , we s udy ac i a ions in he con ex o a classi ica ion p oblem, whe e ce ain classes a e excluded om he aining se bu emain p esen in he sepa a e es se . Mo e speci ically, we explo e how ac i a ion pa e ns beha e in he ollowing scena ios: 1. Tes cases o which he ou pu is co ec ; 2. Tes cases o which he ou pu is inco ec despi e i s class being p esen in he aining se ; 3. Tes cases whe e he inpu does no belong o any class in he aining se . Based on his, we aim o assess whe he he ac i a ion pa e ns we s udy can be used o imp o e aul ole ance, especially by de ec ing e oneous ou pu s, p oblema ic inpu s, and po en ial concep d i . Hence o h, we will add ess cases in he scena ios as co ec ly p edic ed inpu s,inco ec ly p edic ed inpu s, and un ained inpu s, espec i ely. The pa e n we s udy is a e co-ac i a ions in oduced abo e in Sec ion 3.1. As he ac i a ions end o o m pa e ns [4], i makes sense ha nodes in sha ed g oups o en ac i a e oge he . I o en ac i a ing oge he implies being in one o mo e o he same pa e ns, i may no be un easonable o hink ha he disjoin ed and a ypical pa e mani es ed in a e co-ac i a ions implies a b oken pa e n and an un us wo hy p edic ion. Thus, we y and show whe he he e is a u ilizable connec ion be ween a e co-ac i a ions and un us wo hy p edic ions. Mo e speci ically, we aim o answe he ollowing esea ch ques ions: •RQ1: Does he numbe o a e co-ac i a ions s a is ically di e in he abo e scena ios? •RQ2: Can a e co-ac i a ions be used o de ec e oneous be- ha iou when building aul - ole an ML sys ems, and how? The aim o RQ1 is o explo e whe he he idea is alid in he i s place. Only a s a is ically signi ican di e ence in he numbe o a e co-ac i a ions allows us o a gue ha ou app oach has any po en ial in building aul - ole an ML sys ems. I he dis ibu ions be ween cases whe e he neu al ne wo k made a co ec p edic ion and cases whe e he p edic ion was w ong o he inpu ne e appea ed in he aining se a e no s a is ically di e en , we canno claim ha any meaning ul conclusions can be d awn om he numbe o a e co-ac i a ions. As o RQ2, i he dis ibu ions ac ually di e in he a ious sce- na ios, we aim o de ec wha ypes o misbeha iou [1] could be add essed by abusing he a e co-ac i a ions. Di e ences in he numbe o a e co-ac i a ions in and o i sel does no mean ha he esul is use ul in e o de ec ion and aul ole ance as is. Also, as no e e y o m o aul ole ance is sui able o e e y ype o misbeha iou [1], we mus conside how he esul s could link he app oach o known misbeha iou ypes. In his case, he po en ial o ackle some o ms o misbeha iou mus be deduced om how he a e co-ac i a ions man- i es in a ious scena ios. Speci ically, we conside h ee misbeha iou ypes ha pose a majo isk o some sys ems and ha we belie e could po en ially e eal hemsel es in he a e co-ac i a ions: un us wo hy p edic ions, inpu s ha could be p oblema ic o he ne wo k, and d i in he incoming da a [1]. 4. Expe imen al se -up In his sec ion, we desc ibe how he expe imen s we e conduc ed. Fi s , Sec ion 4.1 desc ibes he neu al ne wo ks om which we ga h- e ed he da a abou he a e co-ac i a ions, along wi h how he ne - wo ks di e om each o he and why. Then, in Sec ion 4.2, we desc ibe wha kind o da a abou a e co-ac i a ions in hose ne wo ks was ga he ed and how. Finally, in Sec ion 4.3 we desc ibe how he da a was analysed o d aw conclusions and o ensu e s a is ical signi icance o ou indings. This combina ion o da a iangula ion [14] ac oss ne wo ks and s a is ical igou [15] aises con idence in ou indings. 4.1. Neu al ne wo ks To es he app oach, ou neu al ne wo ks we e buil . The ne - wo ks a e in ended o ep esen mundane neu al ne wo ks in which we obse e how he a e co-ac i a ions beha e in he h ee di e en scena ios conside ed (co ec ly p edic ed, inco ec ly p edic ed and un ained inpu s, see Sec ion 3.2). The di e ences be ween ne wo ks we e designed o ca ch di e ences o en p esen in neu al ne wo ks (see desc ip ions below) and add da a iangula ion [14]. A gene al desc ip ion o he models is p esen ed in Table 1. All ne wo ks we e ained using he Fashion-MNIST [16] da a se . Fashion-MNIST consis s o 28 ×28 size g eyscale images depic ing pieces o clo hing.1The aining se and es se o Fashion-MNIST con ain 60 000 and 10 000 images, espec i ely, bo h di ided in o 10 e enly sized classes. The classes a e – in o de o labels om 0 o 9 – T-shi / op, T ouse , Pullo e , D ess, Coa , Sandal, Shi , Sneake , Bag, and Ankle boo . The ne wo ks we e ained o 15 epochs using Ke as2 machine lea ning lib a y. To make he esul s easie o ep oduce, a ixed andom seed (3) was chosen wi h a h ow o a d20 die. To mimic Scena io 3 om Sec ion 3.2 (a class o inpu s was missing om he aining se , bu appea s a e aining), one class was excluded om he aining se , simila ly o Acke man e al. [17]. This way, in he es ing phase wi h a sepa a e es da a se , we ha e bo h inpu s ha belong o classes he ne wo k was ained o ecognize, along wi h inpu s ha he neu al ne wo k should no ha e ex ensi e knowledge o . Thus, he excluded class ep esen s si ua ions whe e all inpu s do no esemble he da a ha he neu al ne wo k was ained wi h. The ne wo ks achie ed an accu acy anging om 88.4% o 95.6% (see Table 1) in he es se , excluding he il e ed ou class. Omi ing he il e ed ou class when measu ing he accu acy gi es a be e es ima e o how well he neu al ne wo ks pe o m in asks hey should know. This, in ou opinion, be e es ima es he model’s abili y o lea n he da a se han by including he class i was no ained wi h in he i s place. We no e ha he used ne wo ks a e no ine- uned o he 1Examples o Fashion-MNIST can be ound a : h ps://gi hub.com/ zalando esea ch/ ashion-mnis . 2h ps://ke as.io/. A ay 15 (2022) 100201 5 L. Myllyaho e al. Fig. 3. A simple con olu ional neu al ne wo k. In he con olu ional laye s, e e y pa o he inpu goes h ough a con olu ion, on which he ac i a ion is applied, a e which he s onges ac i a ions o e e y small a ea a e ga he ed by a pooling laye . Con olu ional laye s a e ollowed by ully connec ed laye s. maximum no ha e hey been pushed o hei limi s h ough aining ime. This decision is wo old. Fi s , e en hough i we e possible o push a neu al ne wo k o basically label e e y inpu in Fashion-MNIST co ec ly (e.g. Kayed e al. in [18]), his would lea e us wi h e y li le da a o add ess mislabelled classes p esen in he aining da a (Scena io 2 in 3.2). This, in u n, would isk s a is ical signi icance and ou abili y o mee he esea ch goals we ha e se . Second, eal- li e ML models may no each such high le els o accu acy in hei espec i e da a se s (e.g. in [19]). Thus, no pushing he models o hei limi makes hem mo e on pa wi h hei indus ial coun e pa s, ep- esen ing hem be e . Wi h hese wo easons combined, ou ne wo ks a e, in a sense, in en ionally b oken, bu also ‘‘good enough’’. In o he wo ds, hey handle mos cases co ec ly, showing some s eng h in hei beha iou , and ye , make some e o s in o de o lea e us wi h enough da a o answe ou esea ch ques ions. This clea ly poses some h ea s o he alidi y o he s udy, which a e add essed in Sec ion 7. Nex , we go h ough all models in mo e de ail. CNN-ankle boo : As he name sugges s, CNN-ankle boo is a con o- lu ional neu al ne wo k [20] (c . Fig. 3). The ‘‘ankle boo ’’ in he name e e s o he class (9, Ankle boo ) ha is il e ed ou om he aining se o his neu al ne wo k. The ou pu node o he speci ic class is s ill p esen in he ne wo k, e en i i is il e ed ou in he aining phase. The s uc u e o CNN-ankle boo begins wi h h ee con olu ional laye s. The con olu ional laye s consis o 3 ×3 -sized il e s wi h a s ide leng h o 1 and he same padding. The h ee con olu ional laye s ha e 32, 64, and 128 il e s, espec i ely. Each con olu ional laye is accompanied wi h a ba ch no maliza ion laye [21], ReLU ac i a ion, and a 2 ×2 -sized max pooling laye [22]. The con olu ional laye s a e ollowed by wo ully connec ed laye s. The ully connec ed laye s also u ilize ReLU ac i a ion, and consis o 64 and 128 neu ons, espec i ely. Finally, he ou pu laye consis s o 10 neu ons, u ilizing he So max ac i a ion unc ion [12]. CNN-ankle boo 9: CNN-ankle boo 9 sha es mos ea u es wi h CNN-ankle boo excep o he numbe o nodes on he ou pu laye . The il e ed class is he same, along wi h he hidden laye s in he neu al ne wo k. The di e ence is ha he ou pu laye has no ese ed ou pu node o he il e ed class. This na u ally esul s in only ha ing nine nodes on he ou pu laye . The easoning behind his is ha hey ep esen wo di e en si - ua ions in aining a neu al ne wo k. In he case o CNN-ankle boo 9, he imagina y de elope s a e unawa e ha ankle boo s exis and do no ese e an ou pu node o i . Wi h CNN-ankle boo , howe e , he de elope s know ankle boo s exis , hey jus do no ha e enough da a o hem, and hey a e le unde ep esen ed in he aining se . This adds a ie y o he esul s, as CNN-ankle boo 9 wo ks ‘‘as in ended’’ by he imagina y de elope s, and begins ecei ing unexpec ed da a, whe eas CNN-ankle boo is le b oken by he aining da a and begins ecei ing app op ia e da a only a e he aining is comple e. CNN-shi : CNN-shi is s uc u ally iden ical o CNN-ankle boo . The di e ence is ha he class il e ed ou om he aining se is class 6 (Shi ) ins ead o class 9 (Ankle boo ). The pu pose o his is o assess whe he he phenomena we ind a e independen om he il e ed class o no . Shi was chosen, as i is e iden ly di e en om ankle boo s, whe eas, o example, sneake s may no be. MLP-ankle boo : MLP-ankle boo is–as he name sugges s–a mul- ilaye pe cep on [23] (c . Fig. 2 in Sec ion 2). Tha is, all he hidden laye s in he neu al ne wo k a e ully connec ed laye s, u ilizing ReLU ac i a ion and ba ch no maliza ion. The e a e wo hidden laye s wi h 64 and 128 nodes, espec i ely, 10 ou pu nodes, and he il e ed class is Ankle boo . The easoning behind he inclusion o MLP-ankle boo is wo old. Fi s , including models wi h di e en opology p o ides addi ional in o ma ion whe he he esul s a e dependen on ce ain echnologies o no . Second, MLP-ankle boo is a smalle ne wo k han he o he neu al ne wo ks. This should gi e us implica ions o he e ec size ha he size o he ne wo k has on he phenomena. 4.2. Da a collec ion Da a we e collec ed wi h an expe imen al se -up u ilizing Ke as and NumPy.3Fi s , he ne wo ks we e ained using a aining se , om which one class was en i ely excluded. Nex , co-ac i a ion a es o each node in each neu al ne wo k we e calcula ed using Algo i hm 1, in oduced in Sec ion 3.1, and he same da a ha we e used o aining he ne wo k, s ill excluding one class. Finally, he numbe o a e co- ac i a ions was compu ed and sa ed o each inpu in a sepa a e es da a se ha included all he classes. Addi ional de ails we e conside ed be o e calcula ing he co-ac i a ion a es, e.g. when should a node be coun ed as ac i a ed. An appa en choice would be when he node ou pu s a non-ze o numbe , as ha is de ac o how a ReLU ac i a ion unc ion wo ks. Howe e , a node could ou pu a e y small numbe 𝛿 > 0 ha has no ac ual e ec on he ou come o he compu a ions. I is less ob ious i such ac i a ions a e ac ually meaning ul ega ding he ou come. To assess his, we calcula e he co-ac i a ion a es using h ee h esholds ha a e coun ed as an ac i a ion: 0 and wo model-speci ic h esholds, namely, a h eshold ha is smalle han 90% o ha ne wo k’s non- ze o ac i a ions in he aining se and one ha is smalle han 99% o he non-ze o ac i a ions. Hence o h, we will add ess hese h esholds as ac i a ion h esholds. Fu he mo e, i is no ob ious which ou pu should be coun ed in he con olu ional pa s o he CNNs. Ac i a ion unc ions a e applied i s in he CNNs, a e which he s onges ac i a ions close o each o he a e ga he ed by he pooling laye , while he weakes a e il e ed ou . Thus, he e a e wo consecu i e pa s ha ha e he ou pu s o he ac i a ion unc ion as hei alues. We chose o use he ou pu s o he pooling laye , as hey a e he ones ac ually a ec ing he compu a ions o he ollowing laye s. A e he co-ac i a ion a es o e e y neu al ne wo k we e calcu- la ed, he numbe o a e co-ac i a ions was coun ed and sa ed using Algo i hm 2. Fi s , o each inpu in he Fashion-MNIST es se , we ma k down which o he h ee scena ios he inpu ep esen s: is he inpu p edic ed co ec ly, inco ec ly, o is i un ained (c . Sec ion 3.2). Nex , he numbe o a e co-ac i a ions in he neu al ne wo k 𝑁𝑁 wi h ha inpu a e coun ed and sa ed. This way, we ob ain h ee ypes o da a poin s ha , as a whole, ep esen he h ee scena ios. Finally, in p ac ice, he da a a e sa ed o a .CSV ile. On lines 14–23 o Algo i hm 2, we a emp o cap u e he elusi e keywo d a e. As we do no know how he a i y beha es in a ious ne - wo ks, i is en i ely possible ha , o example, he a e co-ac i a ions 3h ps://numpy.o g/. A ay 15 (2022) 100201 6 L. Myllyaho e al. Algo i hm 2 Coun Co-ac i a ions(𝑖𝑛𝑝𝑢𝑡𝑠,𝑁𝑁,𝑟𝑎𝑡𝑒𝑠) inpu s: A se o inpu s and hei co esponding ou pu s o which he numbe o a e co-ac i a ions in NN a e coun ed NN: Neu al ne wo k in which he co-ac i a ions a e moni o ed a es: Co-ac i a ion a es o NN 1: a eCoAc i a ions = [][]: an a ay o s o e he numbe o a e co- ac i a ions o each inpu , along wi h in o ma ion on whe he he inpu was p edic ed co ec ly, inco ec ly, o i i belongs o he un ained class 2: o all 𝑖in 𝑖𝑛𝑝𝑢𝑡𝑠 do 3: i 𝑖belongs o he un ained class hen 4: a eCoAc i a ions[i][0] = ’un ained’ 5: else i 𝑖p edic ed co ec ly by 𝑁𝑁 hen 6: a eCoAc i a ions[i][0] = ’co ec ’ 7: else 8: a eCoAc i a ions[i][0] = ’inco ec ’ 9: end i 10: o all node 𝑛in 𝑁𝑁 do 11: i 𝑛ac i a es wi h 𝑖 hen 12: o all node 𝑚in 𝑁𝑁 do 13: i 𝑚ac i a es wi h 𝑖 hen 14: i 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.05 hen 15: a eCoAc i a ions[𝑖][1] += 1 16: i 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.01 hen 17: a eCoAc i a ions[𝑖][2] += 1 18: i 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.001 hen 19: a eCoAc i a ions[𝑖][3] += 1 20: end i 21: end i 22: end i 23: end i 24: end o 25: end i 26: end o 27: end o 28: e u n a eCoAc i a ions in la ge ne wo ks a e absolu ely a e han in a smalle one. The e is, igu a i ely speaking, mo e oom o he ac i a ion pa e ns o be mos ly o comple ely seg ega ed, whe eas he pa e ns may ha e o sha e a la ge po ion o hei nodes in he smalle ne wo ks. Thus, we do no se le o one a bi a y h eshold o a i y, bu ins ead in oduce a ew o gain mo e in o ma ion on he a i y in a ious ne wo ks. Hence o h, we add ess hese h esholds as a i y h esholds. 4.3. Da a analysis Da a analysis is based on s a is ical es s. To answe RQ1 (do he h ee scena ios di e in e ms o a e co-ac i a ions), we assessed whe he o no he da a poin s in a ious g oups ac ually o igina ed om di e en dis ibu ions. Tha is, we a e no only in e es ed in whe he ou samples a e di e en om each o he , bu we also wan o gene alize he esul s o he en i e popula ions om which he samples o igina e. Using a s a is ical es , we can de e mine how ce ain we can be ha no only he samples a e di e en , bu he popula ions behind hem as well. Only a e his do desc ip i e s a is ics, such as he mean, minimum, and maximum, hold s ong ele ance when compa ing he g oups. Once he di e ence is se by es s designed o do jus ha , hese desc ip i e s a is ics e eal he na u e o he di e ence. We use he K uskal–Wallis es [15] o de e mine ha popula ions a e, in ac , di e en . The K uskal–Wallis es is an ex ension o he Mann–Whi ney U es o samples ha ha e mo e han wo g oups. As such, i is a non-pa ame ic es ha does no p esume ha samples a e no mally dis ibu ed. The ou come 𝑝o he K uskal–Wallis es should Table 2 Resul s o K uskal–Wallis es s o CNN-ankle boo . Ac i a ion h eshold Ra i y h eshold K uskal–Wallis (p) 0<5% 0.0 <1% 0.0 <0.1% 0.0 0.0156* <5% 0.0 <1% 0.0 <0.1% 0.0 0.112** <5% 0.0 <1% 0.0 <0.1% 0.0 *Ac i a ion h eshold <99% o ac i a ions. **Ac i a ion h eshold <90% o ac i a ions. be in e p e ed so, ha wi h 1 − 𝑝% o ce ain y, a leas wo o he popula ions om which he samples o igina e om a e di e en . We use he common 𝑝 < 0.05 o he signi icance le el. Thus, when he es sugges s ha , wi h mo e han 95% ce ain y, a leas wo g oups come om di e en dis ibu ions, we accep ha his is ac ually he case. To assess which g oups a e di e en when he K uskal–Wallis es is signi ican , we use Dunn’s es [24] wi h Bon e oni co ec ion. Once he K uskal–Wallis es inds a signi ican di e ence be ween he g oups and Dunn’s es has iden i ied which g oups a e di e en , we compa e he desc ip i e s a ics o hose g oups. This way, we acqui e knowledge on he na u e o he di e ence: A e a e co-ac i a ions mo e common in ce ain scena ios? Is he e a lo o o e lap? Based on hese s a is ics, along wi h in o ma ion on which g oups ac ually di e om each o he , we assess he po en ial use ulness in he con ex o aul ole ance. As desc ip i e s a is ics we use mean, median, maximum, and minimum, and – when easible – c oss- abula ion. All s a is ics we e ga he ed using SPSS.4 5. Resul s In his sec ion, we examine ou esul s ob ained using he expe i- men al se -up. The esul s a e p esen ed o e e y neu al ne wo k in hei own subsec ion. In u n, o e e y ne wo k, he esul s a e p e- sen ed o each ac i a ion h eshold and each a i y h eshold, s a ing om he lowes one. (see Sec ion 4.2 o mo e de ails). Only s a is ically signi ican esul s a e p esen ed in de ail. 5.1. CNN-ankle boo As p esen ed in Table 2, a leas wo g oups in CNN-ankle boo a e s a is ically di e en (𝑝 < 0.05) om each o he o each h eshold. Thus, we can make meaning ul in e p e a ions abou he a e co- ac i a ions be ween he g oups wi h each h eshold. Now, we examine he pai wise compa isons o g oups o each ac i a ion h eshold and a i y h eshold. Ac i a ion h eshold 0: Fo ac i a ion h eshold 0, a e co-ac i a ions a e mo e common in inco ec ly p edic ed and un- ained inpu s han in co ec ly p edic ed ones. The a e age numbe s o occu ences a e highe and he di e ences a e s a is ically signi ican . E e y pai wise compa ison be ween g oups is s a is ically signi - ican (𝑝 < 0.05) (Table 3). Thus, we can con iden ly say ha in CNN-ankle boo , co-ac i a ions ha occu ed be ween nodes wi h a co- ac i a ion a e o less han 5%, 1%, o 0.1% in he aining se , mani es di e en ly when he p edic ion is co ec , inco ec , o wi h unknown inpu s. Nex , we p esen he desc ip i e s a is ics o he g oups o compa e how he g oups di e en ia e. The compa ison is made o each a i y h eshold, as hey all hold signi icance. 4h ps://www.ibm.com/analy ics/spss-s a is ics-so wa e. A ay 15 (2022) 100201 7 L. Myllyaho e al. Table 3 Resul s o pai wise compa isons be ween g oups o CNN-ankle boo wi h ac i a ion h eshold 0. Ac i a ion h eshold Ra i y h eshold Dunn-Bon e oni (p) 0<5% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <0.1% co ec –inco ec 0.001 co ec –un ained 0.0 inco ec –un ained 0.0 Table 4 Desc ip i e s a is ics o di e en g oups in CNN-ankle boo wi h ac i a ion h eshold 0. Co ec Inco ec Un ained N 8265 735 1000 Ra i y <5% Mean 39026.32 40490.21 115802.51 Median 9380 13811 85318 Max 739477 688498 494998 Min 3 25 524 Ra i y <1% Mean 2088.31 2813.61 9445.02 Median 16 40 985.5 Max 202251 212147 96923 Min 0 0 0 Ra i y <0.1% Mean 61.34 128.01 366.59 Median 0 0 0 Max 30022 39403 19564 Min 0 0 0 Conside ing he mean and median (Table 4), he numbe o a e co- ac i a ions a e – on a e age – sligh ly mo e common when he model p edic ion is inco ec and much mo e common when he inpu is om he class ha was il e ed ou o he aining se . The ela i e di e ence in mean e en ises when lowe ing he a i y h eshold, despi e he numbe dec easing and he median in e e y scena io alling down o 0. The esul is simila when compa ing he minimum numbe . Howe e , he highes numbe o a e co-ac i a ions occu ed when he model was co ec . This applies o he highes a i y h eshold, bu he numbe emains ela i ely high wi h he lowe h esholds as well, e en i he highes maximum numbe is in he inco ec ly p edic ed ones. This sugges s ha e en co ec ou pu s ha e ou lie s wi h la ge numbe s o a e co-ac i a ions. Ac i a ion h eshold 0.0156: Nex , we aise he ac i a ion h esh- old o 0.0156. Ra e co-ac i a ions a e mo e common in inco ec ly p edic ed and un ained inpu s han in co ec ly p edic ed ones. The a e age numbe s o occu ences a e highe and he di e ences a e s a is ically signi ican . The chosen h eshold is smalle han 99% o all non-ze o ac i a ions ha occu ed in CNN-ankle boo in he aining se . As we can see om Table 5, e e y pai wise compa ison sugges s a di e ence in dis ibu ion (𝑝 < 0.05). Below, we p esen he desc ip i e s a is ics o e e y a i y h eshold. The desc ip i e s a is ics emain somewha consis en despi e he aise in ac i a ion h eshold (Table 6). Ra e co-ac i a ions in inco ec p edic ions a e sligh ly mo e common on a e age and a minimum, and much mo e common in he un ained class. Despi e his, he maximum numbe o occu ences in co ec ly p edic ed inpu s is la ge han in o he scena ios wi h he highes a i y h eshold and emains in line wi h he o he scena ios wi h he lowe h esholds as well. Median and minimum numbe s all down o 0 in all h ee scena ios when he a i y h eshold is lowe ed. Table 5 Resul s o pai wise compa isons be ween g oups o CNN-ankle boo wi h ac i a ion h eshold 0.0156. Ac i a ion h eshold Ra i y h eshold Dunn-Bon e oni (p) 0.0156* <5% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <0.1% co ec –inco ec 0.001 co ec –un ained 0.0 inco ec –un ained 0.0 *Ac i a ion h eshold <99% o ac i a ions. Table 6 Desc ip i e s a is ics o di e en g oups in CNN-ankle boo wi h ac i a ion h eshold 0.0156. Co ec Inco ec Un ained Ra i y <5% Mean 39735.17 40869.48 117150.13 Median 9629 13646 86450.5 Max 744755 692816 507542 Min 2 21 671 Ra i y <1% Mean 2141.24 2899.36 9744.11 Median 736 1646 15452.5 Max 389505 407002 214447 Min 0 0 12 Ra i y <0.1% Mean 62.64 132.6 360.68 Median 0 0 0 Max 31614 38045 19756 Min 0 0 0 Table 7 Resul s o pai wise compa isons be ween g oups o CNN-ankle boo wi h ac i a ion h eshold 0.112. Ac i a ion h eshold Ra i y h eshold Dunn-Bon e oni (p) 0.112** <5% co ec –inco ec 0.001 co ec –un ained 0.0 inco ec –un ained 0.0 <1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <0.1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 **Ac i a ion h eshold <90% o ac i a ions. Ac i a ion h eshold 0.112: Nex , we aise he ac i a ion h esh- old o 0.112, which is smalle han 90% o non-ze o ac i a ions occu - ing in he neu al ne wo k wi h he aining se . Ra e co-ac i a ions a e mo e common in inco ec ly p edic ed and un ained inpu s han in co ec ly p edic ed ones. The a e age numbe s o occu ences a e highe and he di e ences a e s a is ically signi ican . The pai wise compa isons o he scena ios (Table 7) a e s a is ically signi ican (𝑝 < 0.05) wi h e e y a i y h eshold. Below, we p esen he desc ip i e s a is ics o e e y a i y h eshold. Based on he mean and median, a e co-ac i a ions a e sligh ly mo e common in inco ec ly p edic ed inpu s o ac i a ion h eshold 0.112 and much mo e common in he non- ained inpu s (Table 8). Also, he minimum numbe o occu ences was highes in he un ained inpu s and second highes in he inco ec ly p edic ed inpu s. Howe e , he maximum numbe o occu ences was highes in he co ec ly p edic ed inpu s in all bu one a i y h eshold. O e all, his ollows he basic na a i e o he lowe ac i a ion h esholds. A ay 15 (2022) 100201 8 L. Myllyaho e al. Table 8 Desc ip i e s a is ics o di e en g oups in CNN-ankle boo wi h ac i a ion h eshold 0.112. Co ec Inco ec Un ained Ra i y <5% Mean 45173.06 46055.6 126369.11 Median 11978 16338 91225.5 Max 793519 770479 575951 Min 25 108 1269 Ra i y <1% Mean 2450.87 3126.73 10423.61 Median 33 70 1695.5 Max 223204 240410 92044 Min 0 0 0 Ra i y <0.1% Mean 74.47 179.18 359.5 Median 0 0 0 Max 37735 36194 20938 Min 0 0 0 Table 9 Resul s o he K uskal–Wallis es s o CNN-ankle boo 9. Ac i a ion h eshold Ra i y h eshold K uskal–Wallis (p) 0<5% 0.0 <1% 0.0 <0.1% 0.0 0.015* <5% 0.0 <1% 0.0 <0.1% 0.0 0.114** <5% 0.0 <1% 0.0 <0.1% 0.0 *Ac i a ion h eshold <99% o ac i a ions. **Ac i a ion h eshold <90% o ac i a ions. Table 10 Resul s o pai wise compa isons be ween g oups o CNN-ankle boo 9 wi h ac i a ion h eshold 0. Ac i a ion h eshold Ra i y h eshold Dunn-Bon e oni (p) 0<5% co ec –inco ec 0.002 co ec –un ained 0.0 inco ec –un ained 0.0 <1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <0.1% co ec –inco ec 0.012 co ec –un ained 0.0 inco ec –un ained 0.0 5.2. CNN-ankle boo 9 In his subsec ion, we go h ough he esul s o he model CNN- ankle boo 9 in a simila manne . This model is o he wise simila o and simila ly ained as CNN-ankle boo , bu does no ha e an ou pu node o he class ha was il e ed ou o he aining se . See Sec ion 4.1 o mo e de ails. As we can see om Table 9, o each ac i a ion h eshold and a i y h eshold, a leas wo g oups ep esen ing he h ee scena ios a e s a is ically di e en (𝑝 < 0.05) om each o he . Thus, we can make meaning ul in e p e a ions abou he a e co-ac i a ions be ween he g oups wi h each h eshold. Nex , we p esen he pai wise compa isons and desc ip i e s a is ics o he g oups o each ac i a ion and a i y h eshold. Ac i a ion h eshold 0: Fo ac i a ion h eshold 0, a e co-ac i a ions a e mo e common in inco ec ly p edic ed and un- ained inpu s han in co ec ly p edic ed ones. The a e age numbe s o occu ences a e highe and he di e ences a e s a is ically signi ican . Table 11 Desc ip i e s a is ics o di e en g oups in CNN-ankle boo 9 wi h ac i a ion h eshold 0. Co ec Inco ec Un ained N 8256 744 1000 Ra i y <5% Mean 26220.5 28798.58 77398.41 Median 4546.5 7165 49562.5 Max 566458 391499 384479 Min 2 11 334 Ra i y <1% Mean 1672.11 2151.78 7923.46 Median 6 15 608.5 Max 179581 139666 82259 Min 0 0 0 Ra i y <0.1% Mean 63.46 111.02 152.59 Median 0 0 0 Max 46778 27016 13429 Min 0 0 0 Table 12 Resul s o pai wise compa isons be ween g oups o CNN-ankle boo 9 wi h ac i a ion h eshold 0.015. Ac i a ion h eshold Ra i y h eshold Dunn-Bon e oni (p) 0.015* <5% co ec –inco ec 0.009 co ec –un ained 0.0 inco ec –un ained 0.0 <1% co ec –inco ec 0.0 co ec –un ained 0.0 inco ec –un ained 0.0 <0.1% co ec –inco ec 0.055 co ec –un ained 0.0 inco ec –un ained 0.0 *Ac i a ion h eshold <99% o ac i a ions. F om Table 10, we can see ha each pai wise compa ison is s a is- ically signi ican (𝑝 < 0.05) wi h ac i a ion h eshold 0. This sugges s ha he dis ibu ion o each g oup is di e en om one ano he , and compa isons be ween he g oups can be made. Below, we p esen he desc ip i e s a is ics o each a i y h eshold. The desc ip i e s a is ics o a i y h eshold seems o ollow he end in he p e ious model (Table 11). On a e age, he un ained inpu s ha e a much la ge numbe o a e co-ac i a ions han he o he wo scena ios, and a e co-ac i a ions in inco ec ly p edic ed ones a e sligh ly mo e nume ous han in he co ec ly p edic ed ones. The same goes o he minimum numbe o occu ences, be o e i alls down o 0 in all scena ios. Howe e , he maximum numbe o occu ences is sligh ly di e en han wi h he p e ious ne wo k. Unlike in he p e ious neu al ne wo k, whe e co ec ly and inco ec ly p edic ed inpu s we e qui e close o each o he wi h almos e e y h eshold, he e, he co ec ly p edic ed inpu s ha e a much la ge maximum numbe o occu ences han ei he o he o he wo scena ios. Again, his p o ides mo e e idence ha a e co-ac i a ions may occu in high numbe s in some cases, e en i he ne wo k’s p edic ion is co ec . Ac i a ion h eshold 0.015: Fo ac i a ion h eshold 0.015, a e co-ac i a ions a e mo e common in inco ec ly p edic ed and un ained inpu s han in co ec ly p edic ed ones. The a e age numbe s o oc- cu ences a e highe and he di e ences a e s a is ically signi ican o excep one. The pai wise compa ison o scena ios in CNN-ankle boo 9 wi h he aised ac i a ion h eshold o 0.015 can be ound in Table 12. The ac i a ion h eshold is smalle han 99% o he non-ze o ac i a ions in CNN-ankle boo 9 wi h he aining se . The mos no iceable di e - ences o he p e ious esul s is ha wi h he a i y h eshold <0.1%, he di e ences be ween co ec ly and inco ec ly p edic ed inpu s do A ay 15 (2022) 100201 15 L. Myllyaho e al. CRediT au ho ship con ibu ion s a emen Lalli Myllyaho: Concep ualiza ion, Me hodology, So wa e, Vali- da ion, Fo mal analysis, In es iga ion, Da a cu a ion, W i ing – o igi- nal d a , Visualiza ion. Jukka K. Nu minen: Concep ualiza ion, Re- sou ces, W i ing – e iew & edi ing, Supe ision, P ojec adminis a- ion, Funding acquisi ion. Tommi Mikkonen: Concep ualiza ion, W i - ing – e iew & edi ing, Supe ision, P ojec adminis a ion, Funding acquisi ion. Decla a ion o compe ing in e es No au ho associa ed wi h his pape has disclosed any po en ial o pe inen con lic s which may be pe cei ed o ha e impending con lic wi h his wo k. Fo ull disclosu e s a emen s e e o h ps://doi.o g/ 10.1016/j.a ay.2022.100201. Acknowledgemen s This wo k was unded by local au ho i ies (‘‘Business Finland’’) unde g an ag eemen ITEA-2019-18022-IVVES o ITEA3 p og amme and g an ag eemen ITEA-2020-20219-IML4E o ITEA4 p og amme. We acknowledge he help o An i Kleme i, Dennis Mui u i, and Juha Myllä i in implemen ing he expe imen al se -up, he help o Mikko Raa ikainen and Tomi Männis ö in e ising he manusc ip , and hank CSC – IT Cen e o Science, Finland, o compu a ional esou ces. Re e ences [1] Myllyaho L, Raa ikainen M, Männis ö T, Nu minen JK, Mikkonen T. On misbeha iou and aul ole ance in machine lea ning sys ems. J Sys So w 2022;183:111096. [2] Abiodun OI, Jan an A, Omola a AE, Dada KV, Mohamed NA, A shad H. S a e-o - he-a in a i icial neu al ne wo k applica ions: A su ey. Heliyon 2018;4(11):e00938. [3] Pei K, Cao Y, Yang J, Jana S. Deepxplo e: Au oma ed whi ebox es ing o deep lea ning sys ems. In: P oceedings o he 26 h symposium on ope a ing sys ems p inciples. 2017, p. 1–18. [4] Tian Y, Pei K, Jana S, Ray B. Deep es : Au oma ed es ing o deep-neu al- ne wo k-d i en au onomous ca s. In: P oceedings o he 40 h in e na ional con e ence on so wa e enginee ing. 2018, p. 303–14. [5] Xie X, Ma L, Jue ei-Xu F, Xue M, Chen H, Liu Y, e al. Deephun e : A co e age- guided uzz es ing amewo k o deep neu al ne wo ks. In: P oceedings o he 28 h ACM SIGSOFT in e na ional symposium on so wa e es ing and analysis. 2019, p. 146–57. [6] Ma L, Jue ei-Xu F, Zhang F, Sun J, Xue M, Li B, e al. Deepgauge: Mul i- g anula i y es ing c i e ia o deep lea ning sys ems. In: P oceedings o he 33 d ACM/IEEE in e na ional con e ence on au oma ed so wa e enginee ing. 2018, p. 120–31. [7] IEEE s anda d dic iona y o measu es o he so wa e aspec s o dependabili y. IEEE S d 982.1-2005 (Re ision o IEEE S d 982.1-1988), 2006, p. 1–41. [8] A izienis A, Lap ie J-C, Randell B, Landweh C. Basic concep s and axonomy o dependable and secu e compu ing. IEEE T ans Dependable Secu e Compu 2004;1(1):11–33. [9] Ramana han A, Pullum LL, Hussain F, Chak aba y D, Jha SK. In eg a ing symbolic and s a is ical me hods o es ing in elligen sys ems: Applica ions o machine lea ning and compu e ision. In: 2016 Design, au oma ion & es in Eu ope con e ence & exhibi ion. IEEE; 2016, p. 786–91. [10] Tsymbal A. The p oblem o concep d i : De ini ions and ela ed wo k. Compu Sci Dep, T ini y College Dublin 2004;106(2):58. [11] Wang S-C. A i icial neu al ne wo k. In: In e disciplina y compu ing in Ja a p og amming. Sp inge ; 2003, p. 81–100. [12] Sha ma S, Sha ma S, A haiya A. Ac i a ion unc ions in neu al ne wo ks. In J Eng Appl Sci Technol 2020;4(12):310–6. [13] Cheng C-H, Nüh enbe g G, Yasuoka H. Run ime moni o ing neu on ac i a ion pa e ns. In: 2019 Design, au oma ion & es in Eu ope con e ence & exhibi ion. IEEE; 2019, p. 300–3. [14] Denzin NK. Sociological me hods: A sou cebook. Rou ledge; 2017. [15] K uskal WH, Wallis WA. Use o anks in one-c i e ion a iance analysis. J Ame S a is Assoc 1952;47(260):583–621. [16] Xiao H, Rasul K, Vollg a R. Fashion-mnis : A no el image da ase o benchma king machine lea ning algo i hms. 2017, a Xi p ep in a Xi :1708. 07747. [17] Acke man S, Fa chi E, Raz O, Zalmano ici M, Dube P. De ec ion o da a d i and ou lie s a ec ing machine lea ning model pe o mance o e ime. In: JSM p oceedings. Ame ican S a is ical Associa ion; 2020, p. 144–60. [18] Kayed M, An e A, Mohamed H. Classi ica ion o ga men s om ashion MNIST da ase using CNN LeNe -5 a chi ec u e. In: 2020 In e na ional con e ence on inno a i e ends in communica ion and compu e enginee ing. IEEE; 2020, p. 238–43. [19] Gobe C, Reu zel EW, Pe ich J, Nassa AR, Phoha S. Applica ion o supe ised machine lea ning o de ec de ec ion du ing me allic powde bed usion addi i e manu ac u ing using high esolu ion imaging. Addi Manu 2018;21:517–28. [20] Albawi S, Mohammed TA, Al-Zawi S. Unde s anding o a con olu ional neu al ne wo k. In: 2017 In e na ional con e ence on enginee ing and echnology. Ieee; 2017, p. 1–6. [21] Io e S, Szegedy C. Ba ch no maliza ion: Accele a ing deep ne wo k aining by educing in e nal co a ia e shi . In: In e na ional con e ence on machine lea ning. PMLR; 2015, p. 448–56. [22] Sche e D, Mülle A, Behnke S. E alua ion o pooling ope a ions in con olu ional a chi ec u es o objec ecogni ion. In: In e na ional con e ence on a i icial neu al ne wo ks. Sp inge ; 2010, p. 92–101. [23] Jain AK, Mao J, Mohiuddin KM. A i icial neu al ne wo ks: A u o ial. Compu e 1996;29(3):31–44. [24] Dunn OJ. Mul iple compa isons using ank sums. Technome ics 1964;6(3):241– 52. [25] Shadish WR, Cook TD, Campbell DT, e al. Expe imen al and quasi-expe imen al designs o gene alized causal in e ence/William R. Shedish, Thomas D. Cook, Donald T. Campbell. Bos on: Hough on Mi lin; 2002. [26] De Cle cq D, Jalo a D, Shang R, Ni K, Zhang Z, Khan A, e al. Machine lea ning powe ed so wa e o accu a e p edic ion o biogas p oduc ion: A case s udy on indus ial-scale Chinese p oduc ion da a. J Cleane P od 2019;218:390–9.