Transformer-based fall detection in videos

Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
A ailable online 22 Janua y 2024
0952-1976/© 2024 The Au ho (s). Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-
nc-nd/4.0/).
Con en s lis s a ailable a ScienceDi ec
Enginee ing Applica ions o A i icial In elligence
jou nal homepage: www.else ie .com/loca e/engappai
Resea ch pape
T ans o me -based all de ec ion in ideos
Ad ián Núñez-Ma cos a,b,∗, Ignacio A ganda-Ca e as c,d,e,
aHiTZ Cen e - Ixa, Uni e si y o he Basque Coun y (UPV/EHU), Paseo Manuel La dizabal 1, Donos ia/San Sebas ián, 20018, Spain
bDepa men o Compu e Languages and Sys ems, Uni e si y o he Basque Coun y (UPV/EHU), Paseo Ra ael Mo eno Pi xi xi 3, Bilbao, 48013, Spain
cDepa men o Compu e Science and A i icial In elligence, Uni e si y o he Basque Coun y (UPV/EHU), Manuel La dizabal 1, Donos ia, 20008, Basque
Coun y, Spain
dDonos ia In e na ional Physics Cen e (DIPC), Manuel La dizabal 4, Donos ia, 20018, Basque Coun y, Spain
eIKERBASQUE, Basque Founda ion o Science, Plaza Euskadi 5, Bilbao, 48009, Basque Coun y, Spain
Bio isika Ins i u e (CSIC-UPV/EHU), Ba io Sa iena, Leioa, 48940, Basque Coun y, Spain
ARTICLE INFO
MSC:
68T05
68T45
Keywo ds:
Fall de ec ion
Compu e ision
T ans o me
Heal h
ABSTRACT
Falls pose a majo h ea o he elde ly as hey esul in se e e consequences o hei physical and men al
heal h o e en dea h in he wo s -case scena io. None heless, he impac o alls can be alle ia ed wi h
app op ia e echnological solu ions. Fall de ec ion is he ask o ecognising a all, i.e. de ec ing when a pe son
has allen in a ideo. Such an algo i hm can be implemen ed in ligh weigh de ices which can hen ca e o
he use s’ needs, e.g. ale ing eme gency se ices o ca egi e s. A he co e o hose sys ems, a model capable
o p omp ly ecognising alls is c ucial o educing he ime un il help comes. In his pape we p opose a all
de ec ion solu ion based on ans o me s, i.e. s a e-o - he-a neu al ne wo ks o compu e ision asks. Ou
model akes a ideo clip and decides i a all has occu ed o no . In a ideo s eam, i would be applied in a
sliding-window ashion o igge an ala m as soon as i de ec s a all. We e alua e ou all de ec ion backbone
model on he la ge UP-Fall da ase , as well as on he UR all da ase , and compa e ou esul s wi h exis ing
li e a u e using he o me da ase .
1. In oduc ion
Acco ding o he Cen e s o Disease Con ol and P e en ion,1 alls
ep esen a signi ican cause o inju y and, in some cases, e en a ali ies
o e he age o 65 in he Uni ed S a es, whe e a all occu s e e y
second, e e y day, a ec ing one ou o ou elde ly adul s each yea . In
a socie y wi h an e e -ageing popula ion, his issue no only p esen s
heal h conce ns bu also c ea es economic challenges ela ed o hei
ea men . The a e ma h o alls o en leads o a loss o independence,
impac ing elde ly adul s’ daily li e. Hence, p e en ing alls o alle i-
a ing hei impac is o pa amoun impo ance o a heal hy ageing.
Tha is why esea ch ela ed o all de ec ion is c ucial o de elop
echnologies capable o aiding he elde ly eel sa e in hei daily
ou ines.
In his pape , we ocus on ision-based app oaches ( hose including
a ision senso ) o all de ec ion due o he ad an ages hey o e
compa ed o hei wea able senso -based coun e pa (wea able senso s
like accele ome e s, excluding wea able ision senso s). Vision-based
app oaches a e less in usi e and elimina e compliance issues asso-
cia ed wi h wea ing special ga men s, pa icula ly o pa ien s wi h
∗Co esponding au ho a : Depa men o Compu e Languages and Sys ems, Uni e si y o he Basque Coun y (UPV/EHU), Paseo Ra ael Mo eno Pi xi xi 3,
Bilbao, 48013, Spain.
E-mail add ess: [email p o ec ed] (A. Núñez-Ma cos).
1h ps://www.cdc.go /inju y/ ea u es/olde -adul - alls/index.h ml
cogni i e issues such as demen ia. Mo eo e , he widesp ead p e a-
lence o came as nowadays p esen s an oppo uni y o le e age hei
ubiqui y, po en ially allowing o he scalabili y o all de ec ion models
beyond speci ic se ings like sma homes o b oade con ex s such as
public spaces. This holds especially ue o 2D came as, in con as o
3D came as (which a e capable o cap u ing dep h in o ma ion). Addi-
ionally, 2D came as p o ide a mo e cos -e ec i e solu ion compa ed
o 3D ange senso s, which a e o en mo e expensi e and may equi e
addi ional ha dwa e se up and calib a ion.
Thanks o he ad en o deep lea ning o ision-based models, he
pe o mance o ision-based me hods has signi ican ly imp o ed, clos-
ing he gap be ween senso -based and ision-based models in e ms o
pe o mance. In ac , he ans o me echnology in oduced in Vaswani
e al. (2017) has eplaced Con olu ional Neu al Ne wo ks (CNNs) in
many asks. Consequen ly, in his pape , we p opose he use o a
ans o me -based neu al ne wo k o he de ec ion o alls in ideos.
Ou objec i e is o ex ac ea u es om aw RGB ames, wi hou
he need o addi ional compu a ions such as op ical low (OF) images,
skele ons/poses and so on. To he bes o ou knowledge, we a e he
h ps://doi.o g/10.1016/j.engappai.2024.107937
Recei ed 28 Feb ua y 2023; Recei ed in e ised o m 16 Janua y 2024; Accep ed 17 Janua y 2024
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
2
A. Núñez-Ma cos and I. A ganda-Ca e as
i s o di ec ly apply ans o me s o add ess he all de ec ion ask us-
ing only RGB images, wi hou equi ing o compu e addi ional ea u es.
Gi en ha all de ec ion models a e usually deployed in ligh weigh
de ices o in e ence, i is impe a i e ha he employed models ha e
low la ency and minimal dependencies. Addi ionally, since he imely
de ec ion o alls is c i ical due o hei se e e consequences, we adhe e
o he guidelines o he da ase we p opose o e alua ion, he UP-Fall
da ase (Ma ínez-Villaseño e al.,2019), by epo ing de ec ion esul s
a 1-second in e als, o , equi alen ly, in a 16- ame ideo in e al.
The UP-Fall da ase comp ises 11 ac i i ies, making i sui able o all
de ec ion, as nea ly hal o he classes a e ela ed o alls.
We p esen wo pa allel e alua ion s a egies o acili a e a comp e-
hensi e compa ison wi h he exis ing li e a u e. The i s s a egy aligns
wi h he app oach aken by he o iginal au ho s o he UP-Fall da ase
and has also been used in he subsequen all de ec ion challenge hey
o ganised. The second s a egy ollows he app oach o Espinosa e al.
(2019), in which hey compa e hei model in he bina y classi ica ion
(by g ouping he 11 ac i i ies in o wo classes: all and no all) and
mul iclass classi ica ion se ings.
Fu he mo e, we conduc ed expe imen s o assess he model’s abil-
i y o lea n om addi ional da ase s and gene alise e ec i ely. To
e alua e his, we selec ed he UR Fall da ase (Kwolek and Kepski,
2014) and a pe o med join aining using bo h he UP-Fall and UR
Fall da ase s. Subsequen ly, we e alua ed he model’s pe o mance on
each da ase sepa a ely. I is impo an o no e ha , while ou model
demons a ed he abili y o lea n om di e se da a, we acknowledge
he limi a ion ha i s eal-wo ld applica ion would need a subs an ial
da ase , which is cu en ly una ailable o he all de ec ion ask.
None heless, we belie e ha he model has he po en ial o adap
and u he imp o e h ough addi ional da a, as demons a ed by i s
pe o mance on he UR Fall da ase .
The pape makes wo signi ican con ibu ions. Fi s ly, we p opose
he i s ision-based ans o me speci ically designed o lea n solely
om RGB da a o all de ec ion. Secondly, we p o ide a comp ehen-
si e compa ison o ou esul s wi h he exis ing li e a u e, speci ically
ocusing on wo ks ha do no ely on addi ional ea u es, he eby
ensu ing ha he model di ec ly lea ns om RGB ames using he
UP-Fall da ase . Fu he mo e, we ha e made all he expe imen al code
publicly a ailable (see Sec ion 3), enabling ellow esea che s o easily
e i y and build upon ou indings.
The emainde o he pape is o ganised as ollows: Sec ion 2
del es in o he ecen all de ec ion li e a u e, Sec ion 3in oduces ou
p oposed ans o me model and, in Sec ion 4, we p esen he UP-Fall
da ase , explain he e alua ion s a egy and compa e ou esul s wi h
he exis ing li e a u e. Finally, we gi e some concluding ema ks on
Sec ion 5.
2. Rela ed wo ks
Fall de ec ion (Alam e al.,2022) is he ask o de ec ing when a
pe son is alling so ha an ala m can be aised and call, o example, an
ambulance o wa n someone. The ypes o app oaches ollowed o his
de ec ion (depending on wha is used o de ec he all) can be di ided
be ween senso -based app oaches (Noo uddin e al.,2021) and ision-
based app oaches (Gu ié ez e al.,2021). Vision-based me hods a e,
in heo y, e y ich in in o ma ion, bu he compu a ional capaci y and
he algo i hms we e no able o co ec ly exploi i un il ecen ly. Due
o he inc easing in e es in deep lea ning ne wo ks, his esea ch opic
shi ed i s in e es o he ision-based me hods ha will be explained
in his li e a u e e iew.
Fall de ec ion canno be app oached as a egula ideo classi ica ion
ask. A po en ial all needs o be de ec ed as soon as possible (wi hin a
ideo s eam) in o de o a all de ec ion model o be use ul in a eal-
li e si ua ion. Tha is why in e media e ou pu s need o be gene a ed.
The mos common me hod, hus, is he use o a sliding window ha
akes a chunk o ames and decides whe he a all has occu ed. Fo
example, a pionee wo k which in oduced CNNs o sol e he all
de ec ion was Yu e al. (2017). The au ho s o ha wo k ex ac ed a
bina y silhoue e o he pe son appea ing in each ame and ca ied
ou a pe - ame classi ica ion o he pose, and iden i ied alls among
hei po en ial ou pu s. Ins ead o using a CNN o di ec ly classi y
images, Wang e al. (2016) ex ac ed se e al ea u es om silhoue e
images, which also included CNN ea u es among hem. Bo h me hods
equi ed o segmen people om images, which may be p one o
e o s in some cases (e.g. mul iple people, clu e ed backg ound, e c.).
Ins ead, compa ed o hose i s wo ks, we di ec ly use he RGB ames
o in e he all.
Ins ead o bina ising images, Núñez-Ma cos e al. (2017) ex ac ed
OF images om ideos o pe o m sliding-window-based all classi ica-
ion, using 10 pai s o OF images o ou pu a possible all de ec ion. The
au ho s employed a VGG16 (Simonyan and Zisse man,2014) ne wo k
(wi h he ea u e ex ac o pa ozen) and ained i o pe o m a bi-
na y classi ica ion ask. Simila ly, Espinosa e al. (2019) also ex ac ed
OF images bu ins ead o di ec ly s acking ho izon al and e ical
componen s, he magni ude o he low was compu ed. Mo eo e , he
au ho s combined hose magni udes om di e en came as and esized
hem o a small esolu ion. Thei model was a small CNN wi h a
bina y c oss en opy loss. Simila o he i s wo ks in oduced in his
sec ion, hese also equi e he compu a ion o addi ional ea u es (in
his case OF images), which can add mo e compu a ional bu den o he
all de ec ion pipeline. In ac , depending on he ligh ing condi ions,
he gene a ed OF images may no be eally help ul since he OF
algo i hm does no co ec ly ecognise he mo emen low wi h no -
con olled ligh ing condi ions. Lu e al. (2018) ained a 3DCNN and
an LSTM model in which he 3DCNN was p e- ained in he Spo s-
1M da ase (Ka pa hy e al.,2014) (no ela ed o all de ec ion) and
an LSTM was ained o all de ec ion making use o he al eady
p e- ained ea u e ex ac o . We belie e ha he T ans o me -based
ne wo k we employ in his wo k is mo e in e es ing o model he
empo al dynamics. Due o i s sel -a en ion componen , he ne wo k
can a end o all he okens.
A mul i-s eam app oach was p oposed by Ca nei o e al. (2019)
wi h a VGG16 ne wo k as a backbone ea u e ex ac o . Each s eam
p ocessed a di e en ea u e, namely: s acked OF, poses and RGB
da a. Chen e al. (2020) ex ac ed he skele on o he pe son o in e es
using OpenPose (Cao e al.,2019) and used a se o heu is ics o
decide whe he he ac i i y could be ca ego ised as a (po en ial) all.
Mo eo e , he model inco po a ed he ac i a ion o an ala m which
would be igge ed i he subjec could no s and up.
A mobile-de ice-o ien ed applica ion o all de ec ion was designed
by Han e al. (2020): a wo-s eam app oach combining a mo ion-
based ea u e ex ac ion and a ligh weigh VGG a chi ec u e called
mobileVGG. Kh aie e al. (2020) p esen ed a weigh ed neu al mul i-
s eam app oach in which he inpu modali ies we e: (i) RGB ( o
colou s and ex u es) and dep h ( o illumina ion), (ii) silhoue e a i-
a ions (in o de o de ec mo emen ), (iii) ampli ude and o ien ed low
and (i ) op ical low. The au ho s ca ied ou expe imen s on ea ly and
la e usion and also on he weigh ing o each s eam. Be lin and John
(2021) employed a Siamese ne wo k ained by dis ance-me ic-based
lea ning. The ne wo k ook pai s o di e en ideos and measu ed
hei L1 dis ance be o e applying a sigmoid unc ion o he esul . I
he ideos a e simila , hei g ound u h should be 1, o 0 o he -
wise. Gomes e al. (2022) used a YOLO 3 de ec ion ne wo k (Redmon
and Fa hadi,2018) o ex ac humans pe - ame and he Kalman il e
o he ime-awa e alignmen o ame sequences ( acking each pe son
in he scene). Each sequence was hen classi ied in o all o no all by
a 3DCNN o a 2DCNN wi h an LSTM.
Mo e ecen ly, he au ho s o Yada e al. (2022) e alua ed hei
ARFDNe model wi h he same da ase we use, i.e. he UP-Fall da ase .
ARFDNe is composed o (i) a skele on ex ac ion module, (ii) a CNN
o ex ac spa ial ea u es and (iii) a Ga ed Recu en Uni (Cho e al.,
2014) module o he spa io- empo al ea u es. The ou pu o he la e
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
3
A. Núñez-Ma cos and I. A ganda-Ca e as
Fig. 1. Ou p oposed all de ec ion model. A sequence o inpu clips (o 1 s each)
sampled om a ideo is passed h ough he Uni o me ne wo k o ex ac ea u es.
Besides, he Uni o me gene a es, o each clip, a p obabili y dis ibu ion ac oss he
possible all and no- all classes o a gi en da ase . The highes p obabili y is aken as
he p edic ed class label o each clip.
was used o he classi ica ion o ac i i ies and alls. Simila ly, Sua ez
e al. (2022) also ed hei ne wo k wi h pose in o ma ion. The o -
me was composed o 1D CNN laye s and a classi ie on op. In u i
e al. (2023) used a CNN + LSTM combina ion wi h poses as inpu
oo. Mobsi e e al. (2023) employed silhoue es as inpu o a Con L-
STM (Shi e al.,2015) model. And going e en u he , Gal ão e al.
(2022) comple ely segmen ed he pe son on each ame and ained a
Gene a i e Ad e sa ial Ne wo k (Good ellow e al.,2014) o classi y
ac i i ies o daily li ing. In his model, alls a e conside ed anomalies
and de ec ed as such. All o hese app oaches equi e a p ep ocessing
s ep o ex ac ing poses, silhoue es o segmen he pe son alling,
which adds a compu a ion o e head and can p opaga e e o s o he
nex s ep.
Ins ead o using poses, o he ea u es we e ex ac ed in he wo k
o Le e al. (2022) using wea able de ices. These ea u es, used as inpu
o a ious adi ional classi ica ion algo i hms, allowed hem o ob ain
e y high F1 me ic esul s (96.16 o alls and 99.90 o non- alls) on
he UP-Fall da ase .
In con as o mos o hese wo ks, ou model does no equi e
addi ional ea u es such as OF o dep h images o he de ec ion o alls.
This alle ia es he compu a ional o e head o compu ing mo e ea u es,
which may be pi o al o ligh weigh de ices wi h low compu a ional
esou ces (usually employed o in e ence).
3. Me hodology
A all de ec ion model add esses he bina y p oblem in which he
model mus decide, o a gi en inpu (e.g. a sequence o ames o
da a om a wea able de ice), whe he a pe son is alling o no .
Fo ha pu pose, ou all de ec ion model’s i s objec i e was o
exclusi ely use RGB ames. This means ha addi ional ea u es, e.g. OF
o dep h images, a e no equi ed, hus allowing o he de elopmen o
compu a ionally less in ensi e ne wo ks. This also educes he la ency,
which is c ucial o eal- ime all de ec ion applica ions. On he o he
hand, he second objec i e o ou model was o p ocess ideos in a
sliding-window ashion o p oduce in e media e ou pu s. Wi h his, he
model is able o de ec alls sho ly a e p ocessing a ew ames, hence
allowing he model o quickly espond o all e en s.
Mo e o mally, conside an inpu ideo 𝑋= {𝑥1, 𝑥2,…, 𝑥𝑁}com-
posed o 𝑁 ames. We ex ac se e al chunks o size 𝑊( ep esen ing
he numbe o ames wi hin each chunk) and gene a e an ou pu
𝑃= {𝑝1, 𝑝2,…, 𝑝⌈𝑁∕𝑊⌉}, whe e each elemen 𝑝𝑖= {0,1} is he ou pu
esul , indica ing whe he a all has been de ec ed in he 𝑖 h chunk
(0≤𝑖 < ⌈𝑁∕𝑊⌉). This high-le el o e iew o he model is illus a ed in
Fig. 1. In a da a s eam, ames accumula e un il 𝑊 ames a e a ailable
o c ea e a chunk and a single ou pu (indica ing whe he a all has been
de ec ed) is gene a ed. Fo he e alua ion o ou model, we will use a
s a e-o - he-a all de ec ion da ase and, hus, we will conside se s o
ideos o a ying sizes ins ead o a con inuous s eam o ames.
Ou all de ec ion model akes each o he 𝑐𝑙𝑖𝑝𝑖(0≤𝑖 < ⌈𝑁∕𝑊⌉)
chunks and passes hem h ough a ea u e ex ac ion ne wo k 𝑀. This
ne wo k decides whe he a all has ocu ed in he inpu ideo clip.
Ou chosen backbone ne wo k, 𝑀, is a Uni o me (Li e al.,2022),
which is a ision ans o me ha , as highligh ed by he au ho s, has
a good balance be ween accu acy and compu a ional e iciency. This
is desi able o applica ions looking o a good pe o mance bu wi h a
minimal la ency. Wha he au ho s o Li e al. (2022) con ibu e in hei
pape is he Uni o me block, which is composed o h ee componen s:
(i) he Dynamic Posi ion Embedding (DPE), (ii) he Mul i-Head Rela ion
Agg ega o (MHRA) and a eed o wa d ne wo k. Fig. 2 illus a es a
Uni o me block wi h i s h ee main componen s.
Conce ning each o he componen s o he Uni o me block, he i s
one, he DPE, is a ligh weigh posi ion encoding based on a dep hwise
con olu ion, adap able o a ying sequence leng hs. The MHRA is a
sel -a en ion block designed o minimise edundancy; i wo ks like a
con olu ional laye : i applies sel -a en ion on a smalle neighbou -
hood o okens ins ead o ying o apply a en ion o e all okens.
This includes a oken a ini y ma ix ha exp esses he ela ion be ween
wo okens o posi ions. In shallow laye s, oken a ini y is simply he
ela i e dis ance be ween okens. In deepe laye s, oken a ini y is
compu ed as he con en simila i y wi h he es o he okens wi hin he
neighbou hood. Ha ing aken hese h ee componen s in o accoun o
build a Uni o me block, he Uni o me ne wo k is buil s acking local
and global Uni o me blocks (i.e. s acking blocks ha apply MHRA in
shallow laye s and blocks o deepe laye s, espec i ely).
The Uni o me ne wo k is p e ained2on wo human ac ion classi i-
ca ion da ase s, Kine ics (Smai a e al.,2020) and Some hing-Some hing
(Goyal e al.,2017), a a esolu ion o 224 ×224. Since he model is
p e ained, 𝑊will be ixed o 16, i.e. 16 ames a e aken o de ec
alls. Fig. 2 illus a es he s uc u e o he model.
Each ideo clip o 𝑊 ames is au oma ically labelled aking he
majo i y o e o he pe - ame g ound- u h class labels. In o he wo ds,
wi hin a single chunk 𝑐𝑙𝑖𝑝𝑖= {𝑥𝑗, 𝑥𝑗+1,…, 𝑥𝑗+𝑊}, each ame 𝑥𝑗will
ha e i s own label 𝑦𝑗= {0,1,…, 𝐶}, whe e 𝐶 ep esen s he amoun
o classes in he da ase . The da ase comp ises se e al classes, some o
which a e ela ed o alls. Depending on he expe imen , he numbe
o classes can be educed o 2 (bina y classi ica ion) and, hence, each
ame will be classi ied as nega i e o posi i e.
We ained he model on a pe -clip basis, ea ing each clip o size
𝑊as a aining sample. We employ c oss en opy loss and he Adam
op imise o he aining. A e each epoch, an e alua ion is conduc ed
on he de elopmen se ( ha is, an e alua ion da ase ex ac ed om
he aining se and no used o aining). T aining is s opped when
a chosen me ic (F1 sco e in ou expe imen s, see Sec ion 4.2 o ou
e alua ion me ics) compu ed on he de elopmen se does no imp o e
a e a p ede ined numbe o epochs. This numbe is e e ed o as
pa ience and is shown in he expe imen ables o Sec ion 4.3. In wha
ollows, he pa ience has been se o 10 epochs.
The code o hese expe imen s can be accessed on Gi Hub.3
2h ps://hugging ace.co/Sense-X/uni o me _ ideo
3h ps://gi hub.com/Ad ianNunez/ ans o me -based- all-de ec ion
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
4
A. Núñez-Ma cos and I. A ganda-Ca e as
Fig. 2. Uni o me block ha is s acked o build he Uni o me ne wo k. I is composed o h ee main componen s: he Dynamic Posi ion Embedding (DPE), he Mul i-Head Rela ion
Agg ega o (MHRA) and a eed o wa d ne wo k (FFN). The pu pose o he MHRA is o minimise edundancy. We e e eade s o he o iginal Uni o me publica ion (Li e al.,
2022) o u he de ails abou i s a chi ec u e.
Fig. 3. UP-Fall De ec ion Da ase sample ideo ames. Example o a sequence o ames showing a all, co esponding o Subjec 1, Ac i i y 1, T ial 1.
Table 1
UP-Fall da ase ’s ac i i ies o classes. Classes 1–5
a e all- ela ed classes.
ID Desc ip ion
1 Falling o wa d using hands
2 Falling o wa d using knees
3 Falling backwa ds
4 Falling sidewa d
5 Falling si ing in emp y chai
6 Walking
7 S anding
8 Si ing
9 Picking up an objec
10 Jumping
11 Laying
4. E alua ion
4.1. Da ase s
The UP-Fall da ase (in oduced by Ma ínez-Villaseño e al. 2019)
is a la ge all de ec ion da ase composed o 11 ac i i ies (see Table 1),
each wi h 3 ials, and eco ded using 17 young adul s wi hou im-
pai men s. The da ase con ains da a om wea able senso s, ambien
senso s and ision de ices (al hough, in his pape , only he la e will
be used). Conce ning he ision de ices, wo came as a e a ailable,
each p o iding a dis inc iewpoin o he alls. Fo ou expe imen s,
we only employed he da a om came a 1 since he da a ob ained om
came a 2 was conside ed o be oo noisy. A sample sequence ( om
came a 1) o he da ase is shown in Fig. 3.
The da ase can be bina ised by me ging classes 1 o 5 in o a single
class, which we call ‘‘Class 1’’, while he es a e me ged in o ano he
one which we will e e o as ‘‘Class 0’’. Depending on he e alua ion
s a egy employed, he bina y se ing o he mul iclass se ing will be
used.
The UR Fall da ase (Kwolek and Kepski,2014) is ano he all
de ec ion da ase comp ising 70 ideos, whe e 30 o hem con ain a
all e en (see Fig. 4 o an example). Since all de ec ion da ase s a e
inhe en ly unbalanced in e ms o classes (since he e a e many mo e
non- all samples), we es ic ed he da ase o hese 30 ideos and did
include he emaining 40 ideos wi hou alls.
The da ase has been anno a ed ame by ame wi h h ee possible
labels: ‘‘ all has no occu ed’’, ‘‘ alling’’ and ‘‘on he loo ’’ (a e he
all). We bina ise he da ase so ha any ame no labelled as ‘‘ alling’’
is conside ed a ‘‘no all’’ ame. Mo eo e , he da ase also con ains
da a om accele ome e s and ano he came a iew. The o me will
no be employed in his wo k since we a e exclusi ely in e es ed in
ision-based app oaches. The addi ional came a iew p o ides a op-
down pe spec i e, which is no usual in all de ec ion da ase s. I would
be in e es ing o co e i in ano he wo k as a op- iew app oach, bu
we ha e deemed i ou o he scope o his wo k.
4.2. E alua ion me hodology
In o de o compa e ou wo k wi h he s a e o he a , we adop ed
wo e alua ion s a egies. We will simply e e o hem as he i s and
he second e alua ion s a egies.
In he i s e alua ion s a egy we will adop in his wo k, which
was o iginally p oposed in he pape o he da ase (Ma ínez-Villaseño
e al.,2019) and has been desc ibed in Sec ion 4.1, a mul iclass
classi ica ion p oblem is add essed. The au ho s also p oposed a public
all de ec ion challenge, which was p esen ed in Ponce and Ma ínez-
Villaseño (2020). This is p ecisely he i s e alua ion s a egy we
will adop in his wo k. We spli he da a in o h ee se s: aining,
de elopmen and es . The aining se is used o une he ne wo k’s
weigh s; he de elopmen se is used o e alua e he model i e a i ely
and s op he aining; and he es se is used o he inal e alua ion.
The ollowing subjec s’ da a is used o aining: 1, 3, 4, 7 and 10–14, in
o al hey comp ise 70% o he da ase . The ial 3 o subjec s 1, 3 and
4 we e chosen by us o he de elopmen se , as he o iginal challenge
does no speci y how o c ea e a de elopmen se . Fo he es ing o
e alua ion se , he challenge p oposes he da a om subjec s 15–17.
The de ec ion esul s o be e alua ed mus be gi en using windows o
1 s o du a ion, wi hou o e lapping. The label o a gi en window is
conside ed o be he mos equen one among he labels o indi idual
ames wi hin he window, as desc ibed in Sec ion 3.
The second e alua ion s a egy we employed is he one o iginally
p esen ed by Espinosa e al. (2019) in which he classes a e bina ised,
i.e. any all class is conside ed class 1 while he es o ac i i ies a e
g ouped in class 0. Fo he sake o compa ison wi h he li e a u e, we
also ob ained esul s o he mul iclass se ing. All ial 3 da a is used
o he es se while he emaining ials’ da a is used o he aining
se . Jus like in he p e ious s a egy, we c ea ed a de elopmen se
aking ial 2 da a o subjec s 1, 3 and 4.
The me ics p oposed o he e alua ion a e he accu acy and he F1
sco e (using he implemen a ion o Ped egosa e al. 2011). The o me
one is usually gi en in he s a e o he a , al hough i is no e y
use ul in all de ec ion da ase s as hey end o be skewed, i.e. he e
a e many mo e nega i e samples han posi i e samples, making he
accu acy no eliable. In ac , in asks such as all de ec ion, in which
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
5
A. Núñez-Ma cos and I. A ganda-Ca e as
Fig. 4. UR Fall De ec ion Da ase sample ideo ames (came a 0, i s sample wi h all).
Table 2
Summa y o he expe imen s pe o med wi h he i s e alua ion s a egy as p oposed by Dodge e al. (2019).
Compu ing in as uc u e N idia A100
Bes alida ion accu acy 98.21
Bes alida ion F1-sco e 91.89
T aining du a ion 5.12 h
Model implemen a ion h ps://gi hub.com/Ad ianNunez/ ans o me -based- all-de ec ion
Hype pa ame e Sea ch space Bes assignmen
Numbe o epochs {10, 50} 50
Lea ning a e {10e−4, 5e−5 10e−5, 5e−6} 1e−5
Ba ch size 16 16
Weigh decay 0.00001 0.00001
Ea ly s opping pa ience (in epochs) 10 10
O e sample classes {No, Yes} Yes
Model a ia ion {Small400, Baseline400, Small600, Baseline600} Small400
Table 3
Resul s o he i s e alua ion s a egy wi h he UP-Fall da ase .
ID Accu acy F1-sco e
Ma ínez-Villaseño e al. (2019) 94.32 (±0.31) 70.44 (±1.25)
Challenge 1s posi ion (Ponce and Ma ínez-Villaseño ,2020) – 82.47
Challenge 2nd posi ion (Ponce and Ma ínez-Villaseño ,2020) – 34.04
Challenge 3 d posi ion (Ponce and Ma ínez-Villaseño ,2020) – 31.37
Challenge hono i ic men ion (Ponce and Ma ínez-Villaseño ,2020) – 60.40
Ou s 96.67 82.24
no de ec ing a all can lead o se ious consequences, i is c ucial o
a oid alse nega i es. Gi en he small amoun o posi i e samples in
all de ec ion da ase s, he accu acy me ic can misleading, as a high
accu acy can also come wi h a ela i ely high numbe o alse nega-
i es. Al e na i ely, he F1 sco e is p oposed in he UP-Fall challenge
and is ecommended as an al e na i e o he accu acy as i akes in o
accoun he unbalanced na u e o all da ase s. Fo ou expe imen s, we
compu ed he unweigh ed mean o F1 sco es ac oss classes.
4.3. Resul s
The esul s o ou expe imen s a e compa ed wi h he s a e o he a
i he compa ison is ai , i.e. he esul s a e compa ed unde he same
e alua ion s a egy, da a spli and so on. We di ided he expe imen s
in o wo se s: hose expe imen s using he i s e alua ion s a egy
and hose using he second one. In he la e , we also di ided he
expe imen s be ween hose using a bina y classi ica ion app oach and
hose ollowing a mul iclass classi ica ion se ing.
Among he wo ks ha a e le ou o his compa ison, we ha e
Rami ez e al. (2021,2022), in which he au ho s ex ac ed skele on
poses om RGB ames. Rami ez e al. (2021) only used indi idual
ames, bu Rami ez e al. (2022) employed 1-second windows o poses
(poses o e e y ame) o classi y ins ances be ween all and no all.
Howe e , hei da a spli was andomly selec ed and, hence, i is no
di ec ly compa able wi h any o he wo s a egies p esen ed he e. Thei
bes esul s we e ob ained wi h a Random Fo es classi ie , ob aining a
99.81% o accu acy and a 99.56 o F1. A e wa ds, he same au ho s
ex ended his wo k wi h Rami ez e al. (2023). Since in hei i s wo k
hey did no ob ain good esul s using an LSTM model, in his new wo k
hey used a BERT model (De lin e al.,2018), whose inpu s we e pose
sequences. They ini ially ob ained an accu acy o 81.14% and an F1
sco e o 80.95, bu hey a gued ha he lowe esul s a e a consequence
o he class imbalance. To alle ia e his, hey a i icially augmen ed
he da ase using a GAN ne wo k called TABGAN (Ash apo ,2020).
Wi h his new da a aken in o conside a ion, he accu acy and F1 sco e
inc eased o 99.50% and 87.20, espec i ely.
Following wi h he use o poses, Tau eeque e al. (2021) ob ained
poses wi h a mul i-came a and mul i-pe son app oach. Thei app oach
also employed an LSTM ne wo k and ob ained an F1 sco e o 92.5.
Meanwhile, Gal ão e al. (2021b) employed a spa io- empo al g aph
neu al ne wo k (p e ained on a la ge ac i i y ecogni ion da ase ) as a
ea u e ex ac o . An au oencode ied o econs uc he inpu and, in
case he e o was highe han a p ede ined h eshold, an anomaly (a
all) was de ec ed. Thei p oposed me hod led hem o an accu acy o
98.62% and an F1 sco e o 93. All he wo ks men ioned he e de ec
alls in a bina y se ing (no mul iclass), bu hey do no sha e he
da a spli s o he i s and second e alua ion s a egies and, he e o e,
canno be di ec ly compa ed wi h ou expe imen s. None heless, hey
also ob ained ema kable esul s, compa ed wi h he esul s ob ained
by ou model.
4.3.1. Resul s unde he i s e alua ion s a egy
Wi h he i s e alua ion s a egy, we made he hype pa ame e
sea ch de ailed in Table 2 ollowing he guideline o p esen machine
lea ning esul s published by Dodge e al. (2019). Fou a ia ions o
he Uni o me we e used, namely, he small and baseline e sions
p e ained on Kine ics-400 and on Kine ics-600.
The esul s o he expe imen wi h his e alua ion s a egy a e
shown in Table 3 alongside o he app oaches in he li e a u e ha
ollow he same e alua ion s a egy. Ma ínez-Villaseño e al. (2019)
p esen ed he UP-Fall da ase and some baseline expe imen s using
ha da ase wi h adi ional machine lea ning algo i hms, i.e. no deep

Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
6
A. Núñez-Ma cos and I. A ganda-Ca e as
Table 4
Second e alua ion s a egy’s sea ch space and bes assignmen s.
Compu ing in as uc u e N idia A100
Bes alida ion accu acy 99.02 (bina y), 97.37 (mul iclass)
Bes alida ion F1-sco e 93.83 (bina y), 97.20 (mul iclass)
T aining du a ion 1,67 h (bina y), 12.76 h (mul iclass)
Model implemen a ion h ps://gi hub.com/Ad ianNunez/ ans o me -based- all-de ec ion
Hype pa ame e Sea ch space Bes assignmen
Numbe o epochs {10, 50} 50
Lea ning a e {10e−4, 10e−5} 1e−4
Ba ch size 16 16
Weigh decay 0.00001 0.00001
Ea ly s opping pa ience (in epochs) 10 10
Class weigh o alls {1, 2} 1
O e sample classes {No, Yes} Yes
Window size {8, 16} 16 (bina y), 8 (mul iclass)
Model a ia ion {Small400, Baseline400, Small600, Baseline600} Small400
Table 5
Resul s o e alua ion s a egy 2 wi h UP-Fall da ase
(wi h mul iclass classi ica ion).
ID Accu acy F1-sco e
Espinosa e al. (2019) 82.26 72.94
Ou s 93.17 93.39
Table 6
Resul s o e alua ion s a egy 2 wi h UP-Fall da ase
(wi h bina y classi ica ion).
ID Accu acy F1-sco e
Espinosa e al. (2019) 95.64 97.43
Ou s 99.17 94.14
lea ning algo i hm was used. The models hey applied we e Random
Fo es s, Suppo Vec o Machines, k-Nea es Neighbou s and Mul i-
laye Pe cep ons. They also explo ed a ious da a ypes and hei
combina ions: (i) in a ed senso da a, (ii) wea able IMU da a, (iii) all
wea able IMU da a and he EEG headse da a, (i ) all in a ed senso s,
all wea able IMU da a and he EEG headse da a, ( ) came a da a, ( i)
all in a ed senso s and came a da a and ( ii) all wea able IMU, EEG
headse and came a da a. Thei bes esul in e ms o accu acy and
F1-sco e, shown in Table 3, was ob ained wi h a Mul ilaye Pe cep on
and a window size o 1 s, using all wea able IMU, EEG headse and
came a da a as inpu .
A e he a o emen ioned wo k, he eam launched he challenge
p esen ed in Ponce and Ma ínez-Villaseño (2020). They p esen ed
he winne s o he challenge and one hono i ic men ion. The esul s
ob ained by hese ou pa icipan s a e shown in Table 3. The winne
employed a Random Fo es and senso da a, he second place used a 1-
laye CNN and senso da a, he hi d place made use o a bi-LSTM ( he
da a used is no men ioned) and he hono i ic men ion did no send a
sho pape and, hus, i is unknown how hey ob ained hei esul .
Wi h he i s e alua ion s a egy, we ob ained a esul simila o
he i s posi ion o he challenge p esen ed in Ponce and Ma ínez-
Villaseño (2020) only elying on ision da a, wi hou he need o
he senso da a hey employed. Besides, compa ed wi h he bes base-
line model p oposed in Ma ínez-Villaseño e al. (2019), we ha e an
imp o emen o mo e han 10 poin s in he F1 sco e.
4.3.2. Resul s unde he second e alua ion s a egy
Wi h he second e alua ion s a egy, we also made a hype pa ame-
e sea ch. The de ails ha e been w i en down in Table 4. Once again,
ou a ia ions o he Uni o me we e explo ed.
Le us begin by compa ing ou mul iclass esul (see Table 5) wi h
he one ob ained by Espinosa e al. (2019). We we e able o ob ain
a 20 poin di e ence in he F1 sco e wi h espec o hem. Fo he
bina y classi ica ion case, shown in Table 6, we a e 3 poin s below
in he F1 sco e, al hough bo h esul s a e e y high. None heless, ou
pu pose was o c ea e a model ha only akes RGB ames, wi hou any
addi ional compu a ion and, in con as , Espinosa e al. (2019) used OF
images. In ac , he ask may ge easie using OF images due o he
e ased backg ound clu e . Ou model, in con as , seems o gene alise
be e o mo e classes, maybe due o he usage o RGB ames and he
supp ession o appea ance- ela ed ea u es.
E en hough he compa ison i no ai , he wo ks p esen ed in he
in oduc ion o Sec ion 4.3, i.e. Rami ez e al. (2021,2022), Ash apo
(2020), Tau eeque e al. (2021), Gal ão e al. (2021b), also p esen ed
esul s o a bina y all de ec ion ask. We we e able o pe o m be e
han mos o hem e en hough we did no compu e skele ons.
4.3.3. Join ine- uning wi h UP-Fall and UR Fall da ase s
To assess he adap abili y o ou app oach o o he da ase s, we
conduc ed an addi ional expe imen by combining wo da ase s: UP-
Fall and UR Fall (bo h in oduced in Sec ion 4.1). Using he p e ained
ne wo k (on UP-Fall) wi hou ine- uning on he new da ase (UR Fall)
he esul s we e unsa is ac o y, as shown in he i s ow o Table 7. The
accu acy was only 43.48% and he F1 sco e was 30.30. This ou come is
a ibu ed o he ac ha he o iginal benchma k- ained model lacks
he abili y o gene alise o any all e en , as i has no been ained
wi h su icien da a om di e se sou ces. Howe e , collec ing a massi e
amoun o da a o all de ec ion is cu en ly no possible ( o he bes
o ou knowledge). To add ess his limi a ion, we p opose a ine- uning
app oach (i.e. e- aining he p e ained Uni o me om sc a ch) in
which we ain he ne wo k wi h bo h da ase s oge he (mixed in
he same aining p ocedu e) o obse e how he model adap s when
p o ided wi h mo e da a.
The aining p ocedu e o his expe imen ollowed he same ap-
p oach as in ou p e ious expe imen s (using he second e alua ion
s a egy wi h bina y classes). We used a combined de elopmen da ase
(including samples o bo h classses, equally ep esen ed) o guide he
aining. In o de o iden i y he op imal ine- uning lea ning a e, we
explo ed h ee di e en lea ning a es: 1e−4, 5e−4 and 5e−5 ( he bes
esul was ob ained wi h 5e−5). Addi ionally, we expe imen ed wi h he
use o a class weigh o 2 o he all class o add ess any class imbalance
issues ha may a ise du ing aining and we saw ha he use o his
weigh imp o ed he esul s. Fu he mo e, o ensu e a ai ep esen a-
ion o he all class in he UR Fall da ase , we pe o med o e sampling.
The all class was o e sampled o ma ch he numbe o samples in he
nega i e class wi hin he same da ase . This o e sampling echnique
allowed us o mi iga e po en ial biases and imp o e he model’s abili y
o lea n om bo h classes e ec i ely.
The esul s can be ound in Table 7. Al hough he aining is pe -
o med wi h bo h da ase s a he same ime, he e alua ion is di ided
as seen in Table 7 o assess he esul s on bo h da ase s sepa a ely. A
sligh d op in pe o mance is obse ed on he UP-Fall da ase , likely
a ibu ed o he model ha ing o lea n he appea ance o ano he
da ase . Ne e heless, e en wi h his d op, he pe o mance on bo h
da ase s emains ema kably high in e ms o F1 sco e. This ou come is
encou aging and sugges s ha he model has he po en ial o gene alise
well o di e en all scena ios.
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
7
A. Núñez-Ma cos and I. A ganda-Ca e as
Table 7
Resul s o e alua ion s a egy 2 wi h he UP-Fall and UR Fall da ase s (wi h bina y
classi ica ion) mixed oge he . The i s esul o UR Fall has been compu ed using he
bes model ine- uned wi h UP-Fall in p e ious expe imen s.
Da ase Accu acy F1-sco e
UR Fall (no ine- uned) 43.48 30.30
UP-Fall (join ly ine- uned) 99.03 92.35
UR Fall (join ly ine- uned, w/o o e sampling) 91.30 89.73
UR Fall (join ly ine- uned, w/ o e sampling) 95.45 94.76
4.3.4. Compa ison wi h wea able-based all de ec ion
Th oughou his pape , we ha e ocused on ision-based app oaches,
speci ically hose using 2D came as. Howe e , i is essen ial o ac-
knowledge ha wea able-senso -based solu ions ha e hei own se o
ad an ages and disad an ages, depending on he speci ic scena io. In
e ms o pe o mance, wea able senso s o en p o ide mo e disc imina-
i e da a o he de ec ion o alls, which can lead o a highe accu acy
in his ask compa ed o ision-based me hods. As a esul , wea able-
senso -based solu ions end o achie e be e esul s in all de ec ion
asks. In his sec ion, we p esen a compa a i e analysis, con as ing
he esul s ob ained om ou ision-based app oach wi h hose o
wea able-senso -based solu ions. By unde s anding he ade-o s and
s eng hs o each app oach, we aim o p o ide insigh s in o he ela i e
me i s o ision-based and wea able-senso -based all de ec ion models.
Table 8 p esen s a summa y o he ecen esul s om he li e a u e
o he UP-Fall da ase , speci ically ocusing on s udies using wea able-
senso in o ma ion o a combina ion o senso and RGB da a. Ou
esul s in his able a e based on he second e alua ion s a egy, as we
conduc ed expe imen s in bo h bina y and mul iclass se ings.
I is impo an o no e ha a di ec compa ison be ween he ap-
p oaches lis ed in Table 8 and he model p oposed in his pape may
no be ai , as hey may no sha e he same ain/e alua ion spli s, com-
pu e me ics di e en ly and ha e di e en clip leng hs o gene a ing
ou pu s. Mo eo e , some wo ks adop a bina y con igu a ion (i.e., all
o no all), while o he s conside all possible classes o he da ase .
Howe e , his compa ison allows us o obse e ha ou ision-based
ans o me app oach achie es esul s ha a e close o he s a e-o - he-
a solu ions in he senso -based all de ec ion ask. This inding u he
ein o ces he p omise and po en ial o ision-based me hods o all
de ec ion and highligh s he e ec i eness o ou p oposed app oach in
cap u ing ele an in o ma ion om RGB da a o iden i y all e en s
accu a ely.
I is wo h men ioning ha he goal o his compa ison is no o
es ablish supe io i y o e o he app oaches bu a he o pu in con ex
he pe o mance o ou me hod in ela ion o he exis ing body o
li e a u e. We belie e ha he di e se ange o all de ec ion echniques
showcased in Table 8 con ibu es o a comp ehensi e unde s anding o
he ad ancemen s in his ield and emphasises he signi icance o ou
con ibu ions wi hin he ision-based all de ec ion domain.
5. Conclusions
In his pape , we in oduced a ans o me -based all de ec ion
model, le e aging he Uni o me a chi ec u e. Ou RGB-only app oach,
aligned wi h UP-Fall da ase guidelines, achie ed compe i i e o im-
p o ed esul s compa ed o exis ing me hods wi hou elying on ad-
di ional ea u es o wea able-senso da a. Ou all de ec ion model
demons a es he capabili y o p omp ly emi an ala m upon de ec ing
a all e en .
Fu u e esea ch a enues include explo ing an icipa ion capabili ies,
inspi ed by ecen wo ks such as Li and Song (2023). Collabo a ing
wi h heal hca e p o essionals is also c ucial o e ining ou model’s
eal-wo ld applica ion. Thei insigh s will guide adjus men s o mee
end-use needs e ec i ely.
Table 8
Resul s o he li e a u e o all de ec ion using he UP-Fall da ase o he e alua ion
and senso da a o skele on in o ma ion as inpu . Fo ou esul s, we used he second
e alua ion s a egy.
Type Bina y? Accu acy F1-sco e
Ponce e al. (2020) Senso +RGB ✓98.72 95.77
Waheed e al. (2021) Senso ✓97.21 97.43a
Gal ão e al. (2021a) RGB+Senso ✓99.99 –
Al Nahian e al. (2021a) Senso ✓96.00 97.00a
Al Nahian e al. (2021b) Senso ✓100.00 –
Ash apo (2020) Skele on ✓99.50 87.20
Tau eeque e al. (2021) Skele on ✓– 92.5
Gal ão e al. (2021b) Skele on ✓98.62 93
Rami ez e al. (2021) Skele on ✓99.34 98.52
Rami ez e al. (2022) Skele on ✓99.81 99.56
Rami ez e al. (2023) Skele on ✓81.14 80.95
Ou s RGB ✓99.17 94.14
Type Bina y? Accu acy F1-sco e
Ma ínez-Villaseño e al. (2019) Senso ✗95.49 70.31
Chahya i and Hawa i (2020) Senso ✗– 81.40
Chahya i and Hawa i (2020) RGB+Senso ✗– 95.44
Rami ez e al. (2021) Skele on ✗99.45 92.34
Le e al. (2022) Senso ✗– 99.60
Mohan Gowda e al. (2022) RGB+Senso ✗99.2 98.4
Islam e al. (2023) RGB+Senso ✗97.90 97.88
Yan e al. (2023) Skele on+Senso ✗98.05 88.30
Ou s RGB ✗93.17 93.39
aManually compu ed based on Recall and P ecision.
Fu he mo e, o imp o e he obus ness and gene alisabili y o ou
model, a la ge , di e se all de ec ion da ase is essen ial. This ex-
pansion will acili a e aining a mo e adap able and eliable neu al
ne wo k.
In conclusion, ou wo k lays a solid ounda ion o ision-based all
de ec ion models and p esen s a p omising di ec ion o u u e esea ch.
By explo ing p oac i e all de ec ion, collabo a ing wi h heal hca e
p o essionals, and collec ing mo e comp ehensi e da ase s, we aspi e
o con inue ad ancing he ield o all de ec ion and con ibu e o
imp o ing he sa e y and well-being o indi iduals a isk o alls.
CRediT au ho ship con ibu ion s a emen
Ad ián Núñez-Ma cos: Concep ualiza ion, In es iga ion, Me hod-
ology, So wa e, Valida ion, W i ing – o iginal d a , W i ing – e iew
& edi ing. Ignacio A ganda-Ca e as: Concep ualiza ion, W i ing –
e iew & edi ing.
Decla a ion o compe ing in e es
The au ho s decla e ha hey ha e no known compe ing inan-
cial in e es s o pe sonal ela ionships ha could ha e appea ed o
in luence he wo k epo ed in his pape .
Da a a ailabili y
The code used has been publicly sha ed.
Acknowledgemen s
This wo k is suppo ed by g an PID2021-126701OB-I00 unded by
MCIN/AEI/10.13039/501100011033 and by ‘‘ERDF A way o making
Eu ope’’, and by g an GIU19/027 unded by he Uni e si y o he
Basque Coun y UPV/EHU.
Enginee ing Applica ions o A i icial In elligence 132 (2024) 107937
8
A. Núñez-Ma cos and I. A ganda-Ca e as
Re e ences
Al Nahian, M.J., Ghosh, T., Al Banna, M.H., Asee i, M.A., Uddin, M.N., Ahmed, M.R.,
Mahmud, M., Kaise , M.S., 2021a. Towa ds an accele ome e -based elde ly all
de ec ion sys em using c oss-disciplina y ime se ies ea u es. IEEE Access 9,
39413–39431.
Al Nahian, M.J., Ghosh, T., Al Banna, M.H., Uddin, M.N., Islam, M.M., Tahe , K.A.,
Kaise , M.S., 2021b. Social g oup op imized machine-lea ning based elde ly all
de ec ion app oach using in e disciplina y ime-se ies ea u es. In: 2021 In e na-
ional Con e ence on In o ma ion and Communica ion Technology o Sus ainable
De elopmen . Icic 4sd, IEEE, pp. 321–325.
Alam, E., Su ian, A., Du a, P., Leo, M., 2022. Vision-based human all de ec ion sys ems
using deep lea ning: A e iew. Compu . Biol. Med. 105626.
Ash apo , I., 2020. Tabula GANs o une en dis ibu ion. a Xi p ep in a Xi :2010.
00638.
Be lin, S.J., John, M., 2021. Vision based human all de ec ion wi h siamese
con olu ional neu al ne wo ks. J. Ambien In ell. Humaniz. Compu . 1–12.
Cao, Z., Hidalgo Ma inez, G., Simon, T., Wei, S., Sheikh, Y.A., 2019. OpenPose:
Real ime mul i-pe son 2D pose es ima ion using pa a ini y ields. IEEE T ans.
Pa e n Anal. Mach. In ell.
Ca nei o, S.A., da Sil a, G.P., Lei e, G.V., Mo eno, R., Guima aes, S.J.F., Ped ini, H.,
2019. Mul i-s eam deep con olu ional ne wo k using high-le el ea u es applied
o all de ec ion in ideo sequences. In: 2019 In e na ional Con e ence on Sys ems,
Signals and Image P ocessing. IWSSIP, IEEE, pp. 293–298.
Chahya i, D., Hawa i, R., 2020. Fall de ec ion on mul imodal da ase using con-
olu ional neu al ne wok and long sho e m memo y. In: 2020 In e na ional
Con e ence on Ad anced Compu e Science and In o ma ion Sys ems. ICACSIS,
IEEE, pp. 371–376.
Chen, W., Jiang, Z., Guo, H., Ni, X., 2020. Fall de ec ion based on key poin s o
human-skele on using openpose. Symme y 12 (5), 744.
Cho, K., Van Me iënboe , B., Gulceh e, C., Bahdanau, D., Bouga es, F., Schwenk, H.,
Bengio, Y., 2014. Lea ning ph ase ep esen a ions using RNN encode -decode o
s a is ical machine ansla ion. a Xi p ep in a Xi :1406.1078.
De lin, J., Chang, M.-W., Lee, K., Tou ano a, K., 2018. Be : P e- aining o deep
bidi ec ional ans o me s o language unde s anding. a Xi p ep in a Xi :1810.
04805.
Dodge, J., Gu u angan, S., Ca d, D., Schwa z, R., Smi h, N.A., 2019. Show you wo k:
Imp o ed epo ing o expe imen al esul s. a Xi p ep in a Xi :1909.03004.
Espinosa, R., Ponce, H., Gu ié ez, S., Ma ínez-Villaseño , L., B ie a, J., Moya-Albo , E.,
2019. A ision-based app oach o all de ec ion using mul iple came as and
con olu ional neu al ne wo ks: A case s udy using he UP- all de ec ion da ase .
Compu . Biol. Med. 115, 103520.
Gal ão, Y.M., Fe ei a, J., Albuque que, V.A., Ba os, P., Fe nandes, B.J., 2021a. A
mul imodal app oach using deep lea ning o all de ec ion. Expe Sys . Appl. 168,
114226.
Gal ão, Y.M., Po ela, L., Ba os, P., de A aújo Fagundes, R.A., Fe nandes, B.J., 2022.
OneFall-GAN: A one-class GAN amewo k applied o all de ec ion. Eng. Sci.
Technol. In . J. 35, 101227.
Gal ão, Y.M., Po ela, L., Fe ei a, J., Ba os, P., Fagundes, O.A.D.A., Fe nandes, B.J.,
2021b. A amewo k o anomaly iden i ica ion applied on all de ec ion. IEEE
Access 9, 77264–77274.
Gomes, M.E.N., Macêdo, D., Zanche in, C., de Ma os-Ne o, P.S.G., Oli ei a, A., 2022.
Mul i-human all de ec ion and localiza ion in ideos. Compu . Vis. Image Unde s .
220, 103442.
Good ellow, I.J., Pouge -Abadie, J., Mi za, M., Xu, B., Wa de-Fa ley, D., Ozai , S.,
Cou ille, A., Bengio, Y., 2014. Gene a i e ad e sa ial ne wo ks (2014), CoRR.
1406, a Xi p ep in a Xi :1406.2661.
Goyal, R., Eb ahimi Kahou, S., Michalski, V., Ma e zynska, J., Wes phal, S., Kim, H.,
Haenel, V., F uend, I., Yianilos, P., Muelle -F ei ag, M., e al., 2017. The "some hing
some hing" ideo da abase o lea ning and e alua ing isual common sense.
In: P oceedings o he IEEE In e na ional Con e ence on Compu e Vision. pp.
5842–5850.
Gu ié ez, J., Rod íguez, V., Ma in, S., 2021. Comp ehensi e e iew o ision-based
all de ec ion sys ems. Senso s 21 (3), 947.
Han, Q., Zhao, H., Min, W., Cui, H., Zhou, X., Zuo, K., Liu, R., 2020. A wo-s eam
app oach o all de ec ion wi h MobileVGG. IEEE Access 8, 17556–17566.
In u i, A.R., Manikandan, V., Ga apally, V., 2023. A no el ision-based all de ec ion
scheme using keypoin s o human skele on wi h long sho - e m memo y ne wo k.
A ab. J. Sci. Eng. 48 (2), 1143–1155.
Islam, M.M., Noo uddin, S., Ka ay, F., Muhammad, G., 2023. Mul i-le el ea u e usion
o mul imodal human ac i i y ecogni ion in in e ne o heal hca e hings. In .
Fusion 94, 17–31.
Ka pa hy, A., Tode ici, G., She y, S., Leung, T., Suk hanka , R., Fei-Fei, L., 2014. La ge-
scale ideo classi ica ion wi h con olu ional neu al ne wo ks. In: P oceedings o he
IEEE Con e ence on Compu e Vision and Pa e n Recogni ion. pp. 1725–1732.
Kh aie , C., Benza i, F., Ami i, H., 2020. Elde ly all de ec ion based on mul i-s eam
deep con olu ional ne wo ks. Mul imedia Tools Appl. 79 (27), 19537–19560.
Kwolek, B., Kepski, M., 2014. Human all de ec ion on embedded pla o m using dep h
maps and wi eless accele ome e . Compu . Me hods P og ams Biomed. 117 (3),
489–501.
Le, H.-L., Nguyen, D.-N., Nguyen, T.-H., Nguyen, H.-N., 2022. A no el ea u e se
ex ac ion based on accele ome e senso da a o imp o ing he all de ec ion
sys em. Elec onics 11 (7), 1030.
Li, S., Song, X., 2023. Fu u e ame p edic ion ne wo k o human all de ec ion in
su eillance ideos. IEEE Sens. J.
Li, K., Wang, Y., Zhang, J., Gao, P., Song, G., Liu, Y., Li, H., Qiao, Y., 2022.
Uni o me : Uni ying con olu ion and sel -a en ion o isual ecogni ion. a Xi
p ep in a Xi :2201.09450.
Lu, N., Wu, Y., Feng, L., Song, J., 2018. Deep lea ning o all de ec ion: Th ee-
dimensional CNN combined wi h LSTM on ideo kinema ic da a. IEEE J. Biomed.
Heal h In . 23 (1), 314–323.
Ma ínez-Villaseño , L., Ponce, H., B ie a, J., Moya-Albo , E., Núñez-Ma ínez, J.,
Peña o -As u iano, C., 2019. UP- all de ec ion da ase : A mul imodal app oach.
Senso s 19 (9), 1988.
Mobsi e, S., Alaoui, N., Boulmal , M., Ghogho, M., 2023. Seman ic segmen a ion-based
sys em o all de ec ion and pos - all pos u e classi ica ion. Eng. Appl. A i . In ell.
117, 105616.
Mohan Gowda, V., A ake i, M.P., Raghu Ram P asad, V., e al., 2022. Mul imodal
classi ica ion echnique o all de ec ion o alzheime ’s pa ien s by in eg a ion o
a no el piezoelec ic c ys al accele ome e and aluminum gy oscope wi h ision
da a. Ad . Ma e . Sci. Eng. 2022.
Noo uddin, S., Islam, M., Sha na, F.A., Alhe a i, H., Kabi , M.N., e al., 2021. Senso -
based all de ec ion sys ems: a e iew. J. Ambien In ell. Humaniz. Compu .
1–17.
Núñez-Ma cos, A., Azkune, G., A ganda-Ca e as, I., 2017. Vision-based all de ec ion
wi h con olu ional neu al ne wo ks. Wi el. Commun. Mobile Compu . 2017.
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O.,
Blondel, M., P e enho e , P., Weiss, R., Dubou g, V., Vande plas, J., Passos, A.,
Cou napeau, D., B uche , M., Pe o , M., Duchesnay, E., 2011. Sciki -lea n: Machine
lea ning in py hon. J. Mach. Lea n. Res. 12, 2825–2830.
Ponce, H., Ma ínez-Villaseño , L., 2020. App oaching all classi ica ion using he up-
all de ec ion da ase : Analysis and esul s om an in e na ional compe i ion. In:
Challenges and T ends in Mul imodal Fall De ec ion o Heal hca e. Sp inge , pp.
121–133.
Ponce, H., Ma ínez-Villaseño , L., Nunez-Ma inez, J., 2020. Senso loca ion analysis
and minimal deploymen o all de ec ion sys em. IEEE Access 8, 166678–166691.
Rami ez, H., Velas in, S.A., Aguayo, P., Fab egas, E., Fa ias, G., 2022. Human ac i i y
ecogni ion by sequences o skele on ea u es. Senso s 22 (11), 3991.
Rami ez, H., Velas in, S.A., Cuella , S., Fab egas, E., Fa ias, G., 2023. BERT o ac i i y
ecogni ion using sequences o skele on ea u es and da a augmen a ion wi h GAN.
Senso s 23 (3), 1400.
Rami ez, H., Velas in, S.A., Meza, I., Fab egas, E., Mak is, D., Fa ias, G., 2021. Fall
de ec ion and ac i i y ecogni ion using human skele on ea u es. IEEE Access 9,
33532–33542.
Redmon, J., Fa hadi, A., 2018. Yolo 3: An inc emen al imp o emen . a Xi p ep in
a Xi :1804.02767.
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., Woo, W.-c., 2015. Con olu ional
LSTM ne wo k: A machine lea ning app oach o p ecipi a ion nowcas ing. Ad .
Neu al In . P ocess. Sys . 28.
Simonyan, K., Zisse man, A., 2014. Ve y deep con olu ional ne wo ks o la ge-scale
image ecogni ion. a Xi p ep in a Xi :1409.1556.
Smai a, L., Ca ei a, J., Noland, E., Clancy, E., Wu, A., Zisse man, A., 2020. A sho no e
on he kine ics-700-2020 human ac ion da ase . a Xi p ep in a Xi :2010.10864.
Sua ez, J.J.P., O illaza, N., Na al, P., 2022. AFAR: a eal- ime ision-based ac i i y
moni o ing and all de ec ion amewo k using 1D con olu ional neu al ne wo ks.
In: 2022 14 h In e na ional Con e ence on Machine Lea ning and Compu ing.
ICMLC, pp. 555–559.
Tau eeque, M., Koi a, S., Spiche , N., Dese no, T.M., 2021. Mul i-came a, mul i-pe son,
and eal- ime all de ec ion using long sho e m memo y. In: Medical Imaging
2021: Imaging In o ma ics o Heal hca e, Resea ch, and Applica ions, Vol. 11601.
SPIE, pp. 35–42.
Vaswani, A., Shazee , N., Pa ma , N., Uszko ei , J., Jones, L., Gomez, A.N., Kaise , Ł.,
Polosukhin, I., 2017. A en ion is all you need. Ad . Neu al In . P ocess. Sys . 30.
Waheed, M., A zal, H., Mehmood, K., 2021. NT-FDS—A noise ole an all de ec ion
sys em using deep lea ning on wea able de ices. Senso s 21 (6), 2006.
Wang, K., Cao, G., Meng, D., Chen, W., Cao, W., 2016. Au oma ic all de ec ion
o human in ideo using combina ion o ea u es. In: 2016 IEEE In e na ional
Con e ence on Bioin o ma ics and Biomedicine. BIBM, IEEE, pp. 1228–1233.
Yada , S.K., Lu h a, A., Tiwa i, K., Pandey, H.M., Akba , S.A., 2022. ARFDNe : An
e icien ac i i y ecogni ion & all de ec ion sys em using la en ea u e pooling.
Knowl.-Based Sys . 239, 107948.
Yan, J., Wang, X., Shi, J., Hu, S., 2023. Skele on-based all de ec ion wi h mul iple
ine ial senso s using spa ial- empo al g aph con olu ional ne wo ks. Senso s 23
(4), 2153.
Yu, M., Gong, L., Kollias, S., 2017. Compu e ision based all de ec ion by a
con olu ional neu al ne wo k. In: P oceedings o he 19 h ACM In e na ional
Con e ence on Mul imodal In e ac ion. pp. 416–420.