applied
sciences
A icle
Explo ing Geome ic Fea u e Hype -Space in Da a
o Lea n Rep esen a ions o Abs ac Concep s
Rahul Sha ma *,† , Be na de e Ribei o, Alexand e Miguel Pin o †and F. Amílca Ca doso
Depa men o In o ma ics Enginee ing—Uni e si y o Coimb a, 3030-290 Coimb a, Po ugal;
[email p o ec ed] (B.R.); [email p o ec ed] (A.M.P.); [email p o ec ed] (F.A.C.)
*Co espondence: [email p o ec ed]
† These au ho s con ibu ed equally o his wo k.
Recei ed: 29 Janua y 2020; Accep ed: 10 Ma ch 2020 ; Published: 14 Ma ch 2020


Abs ac :
The e m concep has been a p ominen pa o in es iga ions in psychology and neu obiology
whe e, mos ly, i is ma hema ically o heo e ically ep esen ed. Concep s a e also s udied in
he compu a ional domain h ough hei symbolic, dis ibu ed and hyb id ep esen a ions. The majo i y
o hese app oaches ocused on add essing conc e e concep s no ion, bu he iew o he abs ac
concep is a ely explo ed. Mo eo e , mos compu a ional app oaches ha e a p ede ined s uc u e
o con igu a ions. The p oposed me hod,
Regula ed Ac i a ion Ne wo k (RAN)
, has an e ol ing
opology and lea ns ep esen a ions o abs ac concep s by exploi ing he geome ical iew o concep s,
wi hou supe ision. In he a icle, i s , a Toy-da a p oblem was used o demons a e he RANs
modeling. Secondly, we demons a e he libe y o concep iden i ie choice in RANs modeling and deep
hie a chy gene a ion using he IRIS da ase . Thi dly, da a om he IoT’s human ac i i y ecogni ion
p oblem is used o show au oma ic iden i ica ion o alike classes as abs ac concep s. The e alua ion o
RAN wi h eigh UCI benchma ks and he compa isons wi h i e Machine Lea ning models es ablishes
he RANs c edibili y as a classi ie . The classi ica ion ope a ion also p o ed he RANs hypo hesis o
abs ac concep ep esen a ion. The expe imen s demons a e he RANs abili y o simula e psychological
p ocesses (like concep c ea ion and lea ning) and ca y ou e ec i e classi ica ion i espec i e o aining
da a size.
Keywo ds:
unsupe ised machine lea ning; hie a chical lea ning; compu a ional ep esen a ion;
compu a ional cogni i e modeling; con ex ual modeling; classi ica ion; IoT da a modeling
1. In oduc ion
Concep s a e o g ea alue o humans because hey a e one o he building blocks o ou
ecogni ion p ocess. They enable us o pe o m cogni i e unc ions such as classi ica ion which
is undamen al in decision making and also capaci a e us o con ex ual comp ehension. The e m
concep has a lo o say abou i sel . Any hing can be seen as a concep , whe he i is a li ing
being, o a hing, o an idea. An indi idual concep is e e ed o as a conc e e concep (o ea u e)
whe eas a gene alized o m o a se o concep s (o ea u es) can be pe cei ed as an abs ac concep .
The denomina ion concep immedia ely coins he need o unde s and i s ep esen a ions. The e a e
se e al concep ual ep esen a ion heo e ical amewo ks [
1
] like modali y-speci ic, localis -dis ibu ed,
expe ience-dependen [
2
]. Such amewo ks no only helps us o unde s and he a ious cogni i e
p ocesses in humans bu also he psychological ones, like c ea i i y. Each heo y has a way o
ep esen conc e e concep s h ough pe cep ion (o ecogni ion), ac ion, emo ion, and in ospec ion,
bu he no ion o abs ac concep s is deba able [
1
]. Abs ac concep s a e la gely s udied in psychology,
and he e a e a emp s o s udy hem by he compu a ional linguis ics esea ch communi y o Na u al
Language P ocessing (NLP) [
3
]. Howe e , he ep esen a ion aspec o abs ac concep s is s ill
Appl. Sci. 2020,10, 1994; doi:10.3390/app10061994 www.mdpi.com/jou nal/applsci
Appl. Sci. 2020,10, 1994 2 o 28
a challenge. In his a icle, we add ess his issue o ep esen a ion o abs ac concep s compu a ionally
by simula ing and s udying he o ma ion o con ex abs ac concep s.
Compu a ional models p o ide us algo i hmic speci ici y, concep ual cla i y, and p ecision.
Besides, hey empowe us o pe o m simula ions ha can ei he be use ul o es and alida e
psychological heo ies o o gene a e new hypo heses abou how he mind wo ks— his has u ned
hem in o an indispensable ool o s udy he human b ain. The li e a u e [
4
–
6
] shows ha his
ambi ious goal is no ou o each o compu a ional cogni i e modeling. Fu he mo e, hese ypes o
compu a ional ools wi h he abili y o cap u e cogni i e phenomena also has he po en ial o simula e
and s udy some men al s a es and p ocesses such as hose linked o c ea i i y [7].
Se e al compu a ional modeling echniques (o ools) simula e cogni i e s a es and ep esen
concep s a symbolic and connec ionis le els. Symbols ep esen in o ma ion a a symbolic le el.
Rules a e de ined o manipula e symbols. Wi hin a symbolic ep esen a ion, he meaning is in e nal
o he desc ip ion i sel ; symbols ha e sense only ega ding o he symbols, and no ega ding any
eal-wo ld objec s o phenomena hey may ep esen .
Adap i e Con ol o Though -Ra ional (ACT-R)
[
8
]
is an example o symbolic app oaches, wi h con ibu ions in, almos , all ields o AI (such as language
p ocessing, pe cep ion, a en ion, decision making, e c.). A he connec ionis le el, in o ma ion is
ep esen ed by he dynamics o e densely connec ed ne wo ks o p imi i e uni s. A pa icula s eng h o
connec ionis ne wo ks is hei abili y o adap hei beha io acco ding o obse ed da a. The weigh s
among he uni s o a dis ibu ed ne wo k ep esen he lea ned beha io , hey o e limi ed explana o y
insigh s in o he p ocess, being modeled. Bioinspi ed A i icial Neu al Ne wo ks (ANN) such as
Res ic ed Bol zmann Machine (RBM)
[
9
], au oencode s [
10
], and deep neu al ne wo ks [
11
] a e some
excellen examples o connec ionis app oaches wi h a signi ican con ibu ion owa d classi ica ion,
pe cep ion, and ecogni ion.
A hi d way cons i u es a hyb id iew o connec ionis , and symbolic me hods. Connec ionis Lea ning
wi h Adap i e Rule Induc ion Online (CLARION) [
12
] is a me hodology ha is hyb id, and capable
o simula ing scena ios ela ed o cogni i e and social psychology. All hese me hodologies ei he
equi e a p ede ined s uc u e o ha e a ixed opology ha imposes a limi a ion o ha ing supe ision,
and in lexibili y while modeling he concep s. Some echniques exhibi dynamic and e ol ing
beha io while pe o ming compu a ional ope a ions, such as e ol ing neu al ne wo ks by using hei
geno ype-pheno ype mapping o cells [
13
]. The p oposed model emula es he beha io o he dynamic
c ea ion o abs ac concep s by e ol ing he compu a ional model upon iden i ying di e en g oups
in he da a.
This a icle p oposes a compu a ional me hod named Regula ed Ac i a ion Ne wo k (RAN)
which uni ies he i ues o symbolic, dis ibu ed, and spa ial ep esen a ions o ep esen concep s
(bo h conc e e and abs ac ). RAN has a g aph-based opology hence i is dis ibu ed, e e y node
in he g aph (ne wo k) iden i ies an en i y, he e o e i is symbolic, and e e y node (o en i y) is
iewed in an n-dimensional ea u e space, hence i is also spa ial. The spa ial iew o concep s as
poin s in mul idimensional geome ic ea u e space (see Figu e 1 o six-dimensional iew o concep s)
is inspi ed by he heo y o concep ual spaces [
14
]. The RAN’s modeling has an e ol ing opology
ha enables i o build a model depic ing a hie a chy o concep s. The geome ical associa ions
among concep s aid in de e mining he con ex abs ac concep s. Fu he , he ep esen a i es (nodes)
o he abs ac concep s o m a new laye dynamically, whe e each node ac s as a con ex abs ac
concep ep esen a i e o he unde lying ca ego y. Symbolically, he concep s a ( ela i ely) lowe
le els in he hie a chy a e iden i ied as conc e e concep s and he concep s a ( ela i ely) highe le els
a e seen as abs ac concep s.
Appl. Sci. 2020,10, 1994 3 o 28
Figu e 1.
A uni e se o concep s in six-dimensional ea u e hype -space. The o als in he diag am depic
indi idual concep s. Each indi idual concep is desc ibed by hei de ining six-dimensions. The clus e
o concep s shows he g oups o med by simila concep s ep esen ed by a
con ex clus e o concep s
,
and he clus e cen e s depic s he mos gene ic concep o he clus e .
The model gene a ion p ocess wi h RAN and he h ee cogni i e unc ions (i.e., concep c ea ion,
lea ning and ac i a ion p opaga ion) a e simula ed using a Toy-da a p oblem. The deep hie a chy
gene a ion, au oma ic gene ic concep modeling simula ions a e pe o med using wo Uni e si y o
Cali o nia I ine (UCI) benchma ks: IRIS da a; and IoT da a om sma phone senso s. The applica ion
o RAN as a classi ie is epo ed along wi h he p oo o concep o classi ica ion using eigh
UCI benchma k da ase s. The gene a ed models we e e alua ed using me ics p ecision, ecall,
F1-sco e, accu acy, and Recei e Ope a ing Cha ac e is ic (ROC) cu e analysis. The a icle also
epo s he RANs classi ica ion and ea u e compa ison wi h i e machine lea ning echniques,
Mul ilaye Pe cep on (MLP)
[
15
],
Logis ic Reg ession (LR)
[
16
],
K Nea es Neighbo s (K-NN)
[
17
],
S ochas ic G adien Descen (SGD)
[
18
] and
Res ic Bol zmann Machine
[
9
] pipelined wi h
Logis ic Reg ession (RBM+).
The a icle is o ganized in he ollowing o de ; Sec ion 2pu s o wa d he wo k closely ela ed o
abs ac concep ep esen a ion and models wi h e ol ing opology. Sec ion 3desc ibes he backg ound
associa ed wi h p inciples, heo ies, and mo i a ions o RAN modeling. RANs me hodology is
de ailed using Toy-da a in Sec ion 4. Sec ion 5shows he expe imen s wi h wo da ase s acqui ed
om UCI machine lea ning eposi o y o exhibi (1) lexibili y in choosing a sui able concep iden i ie ,
(2) building a deep hie a chy o abs ac concep s, (3) au oma ic associa ion o inpu -labels o hei
espec i e abs ac concep nodes. Sec ion 6p o ides RAN compa isons wi h i e classi ie s and p oo
o concep wi h eigh benchma k da ase s. A las , Sec ion 7summa izes and concludes he a icle wi h
ema ks o e ongoing and u u e wo k.
2. Rela ed Wo k
Abs ac concep s a e o immense alue because hey help in de eloping unique abili ies in
humans such as ela i e ecogni ion and e ec i e decision-making. In medical science, he e ha e
been signi ican e o s o s udy abs ac concep s wi h he help o echnology. One such example is
MRI (Magne ic Resonance Imaging), which is being used o inspec he sec ions o he b ain in ol ed
in abs ac concep iden i ica ion
[19,20]
. Resea ch in psychology has also epo ed in es iga ions o e
abs ac concep s, like p obing he ole o emo ional con en in p ocessing and ep esen ing abs ac
concep s [21].
The e has been a no able con ibu ion om cogni i e, and psycholinguis s in s udying languages
h ough abs ac concep modeling and ep esen a ions. In e nally ep esen ing abs ac concep s
Appl. Sci. 2020,10, 1994 4 o 28
ia amodal symbols like a ea u e lis , and ames [
22
,
23
] is among he p elimina y esea ch wo k
in linguis ics. The associa ion and con ex we e also es ablished, o ela ing abs ac and Conc e e
wo ds [
22
]. Some esea ch e eals ha we in e nally ecognize me apho s as
abs ac concep s
[
24
].
Besides heo e ical me hods, compu a ional app oaches a e playing a i al ole in comp ehending
and ep esen ing abs ac concep s. Resea ch in NLP add esses compu a ional lea ning, comp ehension
and p ocessing o human-unde s andable language, and i s componen s. An in e es ing a icle
published a wo k abou he ep esen a ion o abs ac , and conc e e concep s in daily w i en language
using a ex -based mul imodal a chi ec u e o NLP [
3
]. O he han NLP, seman ic ne wo ks a e also
used o s udy seman ic simila i y among abs ac , and conc e e nouns
(o G eek, and English)
[
25
]
wi h he aid o ne wo k-based Dis ibu ed Seman ic Model [26].
Though he a o emen ioned compu a ional app oaches con ibu e owa d abs ac concep
modeling and ep esen a ion, hey ha e a ixed opology (i.e., he modeling p ocess begins wi h
a ixed s uc u e and con igu a ion). In connec ionis compu a ional modeling, he e ha e been e o s
o de elop models ha e ol e. A i icial Neu al Ne wo ks Adap a ion: E olu iona y Lea ning O
Neu al Op imal Running Abili ies (ANNA ELEONORA) [
27
] demons a ed a way o g ow neu al
ne wo ks wi h he aid o pa allel gene ic algo i hms. Neu oE olu ion o Augmen ing Topologies
(NEAT) [
28
] is ano he wo k ha epo ed e ol ing neu al ne wo k modeling, showing how nodes
and weigh s a e added o he model when new ea u es eme ge as pa o he exis ing popula ion
and CoDeepNEAT [
29
] is he mos ecen membe o such e ol ing models. Ma ko B ains [
30
] also
belongs o he amily o e ol ing neu al ne wo ks which uses bina y a iables and a bi a y logic o
implemen de e minis ic o p obabilis ic ini e s a e machines. They ha e been used o in es iga e
beha io s, cha ac e ecogni ion and game heo y.
This a icle communica es an app oach which is no only hyb id bu also has an e ol ing
opology. The RANs modeling lea ns he ep esen a ion o he con ex abs ac concep s dynamically,
hence makes i an e ol ing opology. RANs app oach is a connec ionis , and each newly c ea ed node
co esponds o an abs ac concep symbolically, hus po aying i s hyb id cha ac e is ics.
3. Backg ound
This sec ion p o ides in o ma ion abou he p inciples and me hodologies ela ed o RANs
modeling. I highligh s he signi icance o each app oach, along wi h hei applicabili y in
RANs modeling.
3.1. P inciples o Regula ed Ac i a ion Ne wo ks
The ene s o RANs modeling p esen ed in [
31
], s a e ha he model should be opologically
connec ionis and in end o ep esen and simula e he dynamic cogni i e s a e o an agen . In he i s
e sion RAN [
31
] he au ho s implemen ed a single-laye e sion o he model whe e each node
had a la e al connec ion o i s same-laye companions. I had a simple lea ning and easoning
mechanisms, bu hese showed o be su icien o simula e se e al known cogni i e phenomena
such as he P iming [32], he False Memo y [33,34].
Two p inciples o Regula ed Ac i a ion Ne wo ks inspi ed ou p oposal. Fi s , he model should
be dynamic, and his is achie ed by dynamically c ea ing laye s (deep ep esen a ions) o concep s.
Second, he model mus be capable o lea ning and c ea ing an abs ac ep esen a ion o concep s.
This is ob ained by iewing associa ions among he concep s (a he same le el) in n-dimensional
geome ic space, and lea ning ela ionship be ween he newly c ea ed abs ac concep s, and inpu
le el concep s.
3.2. Concep ual Spaces
Concep ual Spaces Theo y [
14
] is one o he cogni i e app oaches ha o m he basis o RANs
modeling. This heo y iews he concep s as egions wi hin a mul i-dimensional space, wi h he da a
ea u es ep esen ing he dimensions. The simila i y among he concep s can be iden i ied based upon
Appl. Sci. 2020,10, 1994 5 o 28
he geome ical dis ance be ween he objec s. The concep ual spaces hus se e as a na u al way o ool
o cap u e he simila i y ela ionships among concep s, o objec s. Unde his se ing, one da a ins ance
co esponds o a single poin in he space. Fo mally we can say, he quali y dimensions, i.e., a se o
D
1
, .....,D
n
, o ms he concep ual space S. A poin in Sis ep esen ed by a
ec o = hd1, ....., dni
, whe e
{1,....n} a e he indexes o he dimensions. A omic concep s a e con ex egions—a con ex egion
C
ha ing poin
x
ha alls be ween poin s
x1∈C
and
x2∈C
also belongs o
C
. The quali y dimension is
he basic equi emen o concep ual spaces [
35
]. An example is a colo space wi h he dimensions
Hue, Sa u a ion, and B igh ness. Each quali y dimension has a geome ical s uc u e. Fo example,
Hue is ci cula , whe eas b igh ness and sa u a ion co espond wi h ini e linea scales (see Figu e 2).
Figu e 2. The colo space [36].
The heo y o concep ual spaces also add esses p o o ype heo y o ca ego iza ion [
37
–
39
].
The main idea o p o o ype heo y is ha wi hin a ca ego y o objec s, like hose ins an ia ing a concep ,
ce ain membe s a e judged o be mo e ep esen a i e o he g oup han o he s. Fo example, obins a e
judged o be mo e ep esen a i e o he ca ego y “bi d“ han a e a ens, penguins, and emus. I con ex
egions o concep ual space desc ibes concep s, hen p o o ype e ec is, indeed, expec ed, i.e., he mos
likely cen al posi ion o a con ex egion desc ibes an abs ac concep . Fo example, i colo concep s
in a con ex egion iden i ied as subse s o he colo space, hen he cen al poin s o hese egions
would be he mos p o o ypical examples o he colo .
Clus e ing is a sui able way o iden i ying and lea ning a omic con ex concep s in concep ual
spaces. The e a e se e al clus e ing echniques, like hie a chical clus e ing, subspace clus e ing [
40
],
pa i ioning eloca ion clus e ing, densi y-based clus e ing, g id-based clus e ing and many mo e.
Many a e equen ly used in he s a is ical and scien i ic analysis o da a [
41
,
42
], and in machine
lea ning o he iden i ica ion o concep s/ ea u es [
43
]. On he o he hand, he c ea ion o a hie a chy
o sub/supe -concep s is a way o ep esen mo e abs ac concep s and hei axonomic-like ela ions.
Deep lea ning echniques [
44
–
48
] ound in he li e a u e can also be used o c ea e deep hie a chical
ep esen a ions, bu usually do no in e p e da a as poin s in concep ual spaces. In he p oposed
app oach, he clus e ing echniques enable us o iden i y ca ego ies o concep s in a concep ual space
hus laying he ounda ion o o m a laye o abs ac ep esen a ion o concep s.
3.3. Sp eading Ac i a ion
Sp eading Ac i a ion is a heo y o memo y [
49
] based on
Collins and Quillian’s compu e model
[
50
]
which has been widely used o he cogni i e modeling o human associa i e memo y and in o he
domains such as in o ma ion e ie al [
51
]. I in ends o cap u e he in o ma ion ep esen a ion
and how i is p ocessing. Acco ding o he heo y, long- e m Memo y is ep esen ed by nodes
and associa i e links be ween hem, o ming a seman ic ne wo k o concep s. The links cha ac e ized
by a weigh deno es he associa i e o seman ic ela ion be ween he concep s. The model assumes
ac i a ing one concep implies he sp eading o ac i a ion o ela ed nodes, making hose memo y
a eas mo e a ailable o u he cogni i e p ocessing. This ac i a ion decays o e ime as i sp eads,

Appl. Sci. 2020,10, 1994 6 o 28
which can occu h ough mul iple le els [
52
], and he u he i ge s he weake i becomes. Tha is
usually modeled using a decaying ac o o ac i a ion. The me hod o sp eading ac i a ion has
been cen al in many cogni i e models due o i s ac abili y and esemblance o in e ela ed g oups
o neu ons in he human b ain [
53
]. This heo y o Sp eading Ac i a ion inspi es he ac i a ion
p opaga ion mechanism in ou p oposal o p opaga e (sp ead) ac i a ion in he upwa d di ec ion,
i.e., om he inpu - o-abs ac laye in he ne wo k. The me hod has i s signi icance, i.e., in he c ea ion
o he ne wo k, and in unde s anding he c ea ed abs ac concep s.
4. Abs ac Concep Modeling wi h RANs
The p oposed app oach models con ex abs ac concep s h ough ou co e s eps (i.e., Concep
Iden i ica ion, Concep C ea ion, In e laye Lea ning and Upwa d Ac i a ion P opaga ion), along
wi h one op ional s ep (i.e., Abs ac Concep Labeling). The RAN’s me hodology is explained using
a Toy-da a p oblem. Figu e 3shows he plo o Toy-da a displaying he Clus e Rep esen a i e Da a
Poin s (CRDPs) o all i e classes o Toy-da a ( he impo ance o CRDP is de ailed in Sec ion 4.2).
The objec i e o his expe imen is o show how RANs build a hie a chical ep esen a ion dynamically
and simula e cogni i e p ocess o concep c ea ion, lea ning, and ac i a ion p opaga ion. Fo his
expe imen , i was hypo hesized ha he c ea ed abs ac concep s symbolically ep esen s he 5 classes
o Toy-da a. Classi ica ion ope a ions we e pe o med o p o e he hypo hesis which is epo ed a
he end o his sec ion. The no a ions used o desc ibe he RAN’s me hodology a e lis ed in Table 1.
Table 1. No a ions.
No a ion Desc ip ion
WIn e -Laye weigh ma ix
AOu pu Ac i a ion
aInpu Ac i a ion
naNumbe o elemen s in inpu ec o a Laye l
nANumbe o elemen s in ou pu ec o a Laye l+1
ll’ h Laye ep esen a i e
dNo malized Euclidean dis ance
CClus e cen e o Cen oids
i,j,k
Va iables o ep esen node index o inpu -le el, abs ac -le el, and a bi a y node
index in ei he o he le els, espec i ely
I e a o a iable
(x)T ans e unc ion o ob ain simila i y ela ion
Figu e 3.
Plo o Toy-da a, a 2-D a i icially gene a ed da a. The plo shows i e classes along wi h hei
clus e cen e s.
Appl. Sci. 2020,10, 1994 7 o 28
4.1. Assump ions and Bounda ies
The necessa y bounda y ela ed o inpu da a is, da a alue should be be ween “0‘ and “1“
(bo h inclusi e)
, his limi a ion has i s inspi a ion om biological neu ons. A alue “0“ indica es
neu on (o node) is inac i e, whe eas “1“ shows he neu on is highly ac i e. The model is, by design,
applicable only o mul idimensional da a se s whe e each ea u e akes A eal alue be ween
0 and 1
—I wo ks as well o disc e e da a se s whe e he a iables ake ei he 0 o 1 alues. I he use
da a is in a di e en o ma , he use mus de ine he ans o ma ion and in e se ans o ma ion o
he da a. The ollowing a e a ew possibili ies o such al e a ions o some o he mos common kinds
o da a:
•
I a a iable in he inpu da a is ca ego ical, e.g.,
blue
;
g een
;
ed
, ans o m he da a using
One Ho Coding echnique.
•
I a a iable in he inpu da a is nume ical, bounded wi hin a minimum and a maximum alue i
can be no malized in o [0, 1], e.g., ia alue−min
max−min ;
The use mus implemen hese and he in e se ans o ma ion unc ions o in e p e he esul s
ob ained om ou model. Since ou echnique is designed o wo k wi h mul i- a ia e da a-se s,
whe e each da a alue is a poin in concep ual space, we assume ha he da a being used is compa ible
wi h he equi emen s. Though images a e a o m o mul i a ia e da a, pic u es a e no ideal candida es
o be in e p e ed as poin s in concep ual spaces, (discussed in Sec ion 3.2). Fo his eason, ou
app oach will, mos p obably, unde pe o m on image p ocessing asks agains o he models ha a e,
indi idually, designed o hese kinds o da a, such as deep ep esen a ions buil wi h Con olu ional
Ne wo ks [
47
,
54
,
55
]; ou echnique is p e e ably sui able o unde s anding and simula ing cogni i e
p ocesses like abs ac concep Iden i ica ion. The e sion o RAN in his a icle can model da a
ha consis s o con ex g oups o da a poin s, he e o e, he model does no pe o m well well wi h
he complex da a ha ing non-con ex g oups o da a poin s. Modeling non-con ex concep s is one
o he ongoing esea ch in RAN’s modeling and ou o he scope o his a icle bu eade s who a e
in e es ed in knowing mo e can e e o he published esea ch wo k [56].
To use he RANs app oach p o ide he da a o he model wi h an addi ional heade s acked
o e he da a. The size o he heade is he same as he dimension o he inpu da a ec o , and each
heade elemen holds he la ges alue o hei co esponding inpu da a a ibu e. See Appendix A.1
o elabo a ion.
4.2. S ep 1: Concep Iden i ica ion (CI) P ocess
Concep iden i ica ion is he i s s ep in RANs modeling. The objec i e o he CI p ocedu e
is o app op ia ely iden i y each ins ance wi hin he da a as a dis inguished membe o a ious
unde lying con ex g oups. This is ealized by ca ego izing he inpu da a based upon hei geome ical
ela ionship, i.e., dis ance, con o ming o he heo y o concep ual spaces (see Sec ion 3.2). He e,
we also ecognize da a poin s ha a e he mos p obable ep esen a i e o each iden i ied g oup,
complying wi h he p o o ype heo y (see Sec ion 3.2). These iden i ied da a poin s a e e med as
Clus e Rep esen a i e Da a Poin s (CRDP) and a e used in S ep 3 o lea ning he ela ionship be ween
wo adjacen laye s (see Sec ion 4.4).
The p ocess o CI ins an ia es a e p ep ocessing he inpu da a. Ini ially, an inpu laye is
o med, wi h dimension equal o he size o he inpu da a ea u e ec o . S ep 1 in Figu e 4shows
he Laye -0 wi h wo nodes which is like he magni ude o he inpu ec o o 2-Dimensional Toy-da a.
A Laye -0, clus e ing me hods a e used o de e mine geome ical ela ion among he se e al inpu
da a ins ances and iden i y he unde lying ca ego ies wi hin he da a. Thus, K-means [
57
] clus e ing
algo i hm is applied o Toy-da a o iden i y i e classes (Class-1,
. . .
, Class-5) by assigning a alue 5
o ‘K’ in K-means clus e ing algo i hm (No e: The ‘K’ alue in he K-means algo i hm is o p o ided
manually bu in unlabeled da ase s, he bes alue o ‘K’ can be de e mined using he Elbow me hod).
Figu e 3shows he plo o 2-D da a poin s ob ained a e pe o ming concep iden i ica ion ope a ion
Appl. Sci. 2020,10, 1994 8 o 28
using he K-means algo i hm. Figu e 3also displays he cen oids (C
1
,
. . .
, C
5
) o all he clus e s,
ecognized as CRDPs o all i e classes and will be used in In e -Laye Lea ning (ILL) in S ep 3
(see Sec ion 4.4).
Figu e 4. S eps in model gene a ion wi h Regula ed Ac i a ion Ne wo ks.
Any clus e ing algo i hm can ac as a concep Iden i ie in RANs modeling i i su ices wo
basic equi emen s. Fi s , he algo i hm can de e mine con ex ca ego ies based upon hei geome ic
ela ionship among he da a ins ances. Second, he algo i hm ecognizes CRDPs o all he iden i ied
clus e s. This lexibili y o choosing a sui able me hod o he concep Iden i ica ion p ocess in
RANs modeling is demons a ed by a sepa a e expe imen using A ini y p opaga ion [
58
] clus e ing
algo i hm, in Sec ion 5.1.
4.3. S ep 2: Concep C ea ion (CC) P ocess
Concep c ea ion is a cogni i e p ocess o c ea e a ep esen a ion o a newly iden i ied concep .
In RAN’s modeling, his cogni i e p ocess is simula ed by c ea ing a new laye o concep s dynamically.
Each cons i uen node in he new laye symbolically ac s as an abs ac ep esen a ion o hei espec i e
ca ego ies iden i ied in he CI p ocess. The S ep-2 in Figu e 4shows he newly c ea ed laye (Laye -1),
ha has i e nodes (N
1
,
. . .
, N
5
), co esponding o i e classes (see Figu e 3), iden i ied in CI ope a ion
wi h Toy-da a.
Besides abs ac ep esen a ion o unde lying ca ego ies, he ac i a ion o nodes in newly c ea ed
laye discloses he deg ee o con idence (DoC) (Calcula ing DoC o a node is explained in de ail wi h
upwa d ac i a ion p opaga ion ope a ion). indica ing he ce ain y o iden i ica ion o a class by
i s ep esen a i e node in he new laye ( o a gi en inpu da a ins ance). Fo example, i a node
(say N1)
ge s an ac i a ion o 0.85, i can be s a ed ha wi h a con idence o 85% he inpu da a
belongs o he ca ego y being ep esen ed by node N
1
. Thus, o all inpu da a ins ances, he ob ained
h ea u e, aluei
pai o
h
abs ac -node, Ac i a ion- alue
i
a new laye adds mo e meaning. Fo ins ance,
in Figu e 4, S ep-2, a Laye -0 inpu ec o is [0.1, 0.21] i signi ies ha he dimensions S
1
and S
2
has
ac i a ion 0.1, and 0.21 espec i ely. Fo he, a o emen ioned, inpu ec o ,
[0.13, 0.32, 0.89, 0.16, 0.05]
ec o o ac i a ion is obse ed a all nodes (N
1
,
. . .
, N
5
) espec i ely, a Laye -1. The obse ed
ac i a ion ec o i sel desc ibes ha he inpu da a belongs o Class-4 wi h a DoC o 89%.
4.4. S ep 3: In e -Laye Lea ning (ILL) P ocess
Lea ning is an impo an cogni i e p ocess i ac s as a ela ionship o associa e concep s. In RANs
modeling, lea ning is simula ed by an assignmen ope a ion. The de eloped In e -Laye Lea ning
p ocedu e also ul ills he second objec i e o RANs modeling (men ioned in Sec ion 3.1). As a o es a ed
in Sec ion 4.3 ha each node in he new laye is an abs ac ep esen a i e o ca ego ies iden i ied in
he CI p ocess, hus we lea n associa ion among he wo-laye such ha i subs an ia es he abs ac
ep esen a ion by he nodes a he new laye . Since CRDPs (see Sec ion 4.2) a e he mos appa en
Appl. Sci. 2020,10, 1994 9 o 28
choice as an abs ac ep esen a i e o a clus e (and adhe e o he inspi a ion om p o o ype heo y);
consequen ly, he CRDPs lea ned as an associa ion be ween he wo laye s.
Equa ion (1) shows he gene al lea ning in he o m o a ma ix, whe e Wis he lea ned In e -Laye
Weigh (ILW) be ween node
j
a new laye (i.e., Laye -1 in Figu e 4) and node
i
a inpu laye
(i.e., Laye -0)
. The se o ILWs, om one node
j
a new laye o all inpu nodes
i
, a e he alues o
CRDP o
j h
clus e cen e (i.e., C
j
) iden i ied in CI p ocess. Fo ins ance, clus e cen e C
1(see Figu e 3)
o ms he weigh ec o [W
1,1
, W
1,2
, W
1,3
and W
1,4
] (ILWs shown by 2 yellow lines in S ep 3 Figu e 4)
be ween he node N1a Laye -1 and all ou inpu nodes S1and S2a Laye -0.
W=







W1,1,W1,2, . . . , W1,na
. . .
Wk,1,Wk,2, . . . , Wk,na
. . .
WnA,1,WnA,2, . . . , WnA,na







=







C1
. . .
Ck
. . .
CnA







(1)
whe e j = 1, 2, . . . , nA, and i = 1, 2, . . . , na.
The dis ance be ween he lea ned weigh ec o o one node
j
(a Laye -1) and ac i a ion o all
inpu nodes S
1
and S
2
(a Laye -0), is used o de e mine how s ongly he inpu ec o ep esen s
he node N
j
a new laye . Thus, i enables us o iden i y he con ex abs ac concep s o he inpu
ins ance (elabo a ed in Sec ion 4.5).
4.5. S ep 4: Upwa ds Ac i a ion P opaga ion (UAP) P ocess
This upwa d ac i a ion p opaga ion is a geome ic easoning ope a ion, i.e., a non-linea p ojec ion
o an
i
-dimensional inpu da a ec o a
i
, in o a
j
-dimensional ou pu ec o
Aj
(see S ep 4 in Figu e 4).
The UAP ope a ion is ca ied ou in wo s ages, in he i s s age he geome ic dis ance ope a ion akes
place, and in he second s age, geome ic dis ance is ansla ed o es ablish a simila i y ela ion.
4.5.1. Geome ic Dis ance Func ion (GDF)—S age 1
In he i s phase o he UAP mechanism we de e mine he geome ical dis ance be ween
he lea ned weigh ec o s (see Equa ion (1)) and an inpu ins ance
ai
. The nume a o o Equa ion (2)
shows a unc ion o calcula e he Euclidean dis ance be ween he
j h
weigh ec o and
inpu ec o ai
.
The denomina o o Equa ion (2) shows he ela ion ha no malizes (in RANs modeling he ac i a ion
alues a e, by de ini ion, eal alues in he
[
0, 1
]
in e al – in an
n
-dimensional space he maximal
possible euclidean dis ance be ween any wo poin s is
q∑n
i=1(ai−0)2
=
√n
, whe e
ai
= 1 he calcula ed
dis ance be ween [0, 1].
dj=q∑na
i=1(Wj,i−ai)2
√na(2)
and consequen ly,
j
no malized Euclidean dis ances
dj
a e ob ained be ween all
j
weigh ec o s
and inpu ins ance ai.
4.5.2. Simila i y T ansla ion Func ion (STF)—S age 2
In he second phase he calcula ed no malized dis ance is ans o med o ob ain a simila i y
ela ion such ha ollowing equi emen s a e ul illed:
• (d=0) = 1, i.e., when dis ance is 0 simila i y is 100%.
• (d=1) = 0 i.e., when dis ance is 1 simila i y is 0%.
• (d=x)is con inuous, mono onous, and di e en iable in he [0, 1]in e al.
(x) = (1−3
√x)2(3)
Appl. Sci. 2020,10, 1994 16 o 28
classes we e mapped o one node o Laye -1. Whe eas, he labels si ing, s anding and laying aced
o he o he node in Laye -1. In e es ingly, his ou come commensu a e wi h he expec a ions om
his expe imen and shows he RANs capabili y o iden i y abs ac concep s in an unsupe ised
manne na u ally.
Figu e 9.
Model gene a ed wi h RANs app oach. Nodes N
1
and N
1
a Laye -1 ep esen s ei he o
he wo abs ac concep s, i.e., mobile and immobile. Each node a Laye -0 ep esen s indi idual
dimensions o inpu da a ec o .
The T ue-label and Tes -label ob ained h ough ACL ope a ion we e used o o m he con usion
ma ix, which is la e e e ed o calcula e p ecision, ecall, F1-sco e, and accu acy o e alua ing
he gene a ed model. Node-wise bina y labels and con idence sco es we e de e mined (as desc ibed in
Appendix A.5) o bo h abs ac nodes a Laye -1. Figu e 10 shows he A ea Unde he Cu e (AUC)
obse ed du ing he ROC cu e analysis o all 10- olds in di e en esea ch designs. Wi h bo h hese
e alua ions i is deduced ha , apa om building he ep esen a ion o abs ac concep s, he model
gene a ed wi h RANs pe o med sa is ac o ily.
Figu e 10.
A ea Unde Cu e obse ed du ing ROC cu e analysis o UCIHAR da a o de e mine
ope a ional poin s o wo abs ac concep s (i.e., Mobile and Immobile) o all nine Resea ch Designs (RD).
The RANs modeling was compa ed wi h i e di e en ypes o app oaches based upon hei
classi ica ion ope a ion. To ca y ou he compa a i e s udy i was essen ial o ans o m he six labels
in o bina y labels, because RANs modeling was iden i ying wo abs ac concep s, and i s pe o mance
was measu ed based upon hem. Thus, wi h hese i e app oaches, he Labels o he da ase we e
me ged o o m wo g oups, i.e., walking, walking
_
ups ai s, and walking
_
downs ai s in Class-1,
and si ing, s anding, and laying in Class-2. La e he modeling was pe o med ollowed by alida ion

Appl. Sci. 2020,10, 1994 17 o 28
and e alua ion. Table 5displays he compa ison o all i e app oaches wi h RANs modeling. I is
obse ed ha RANs app oach is compe en o hese i e echniques, wi h an added ad an age o being
an unsupe ised app oach, and abili y o build ep esen a ions o abs ac concep s.
Table 5. RAN’s Compa a i e S udy o UCIHAR da ase .
Model P ecision (%) Recall (%) F1-Sco e (%) Accu acy (%)
RBM 99.68 ±0.14 99.68 ±0.14 99.68 ±0.14 99.68 ±0.14
K-NN 99.96 ±0.02 99.96 ±0.02 99.96 ±0.02 99.96 ±0.02
LR 99.97 ±0.02 99.97 ±0.02 99.97 ±0.02 99.97 ±0.02
MLP 99.96 ±0.02 99.96 ±0.02 99.96 ±0.02 99.96 ±0.02
RANs 99.85 ±0.01 99.85 ±0.01 99.85 ±0.01 99.85 ±0.01
SGD 99.98 ±0.01 99.98 ±0.01 99.98 ±0.01 99.98 ±0.01
6. RANs Applicabili y and Obse a ions
This sec ion highligh s he scope o RANs modeling as a classi ie wi h espec o dis inc
domains. To suppo his ambi o RANs usabili y, expe imen al esul s a e epo ed using eigh
da ase s conce ning di e en a eas. A compa a i e s udy was also ca ied ou using hese da ase s
o ma ch RANs classi ica ion abili y wi h i e di e en classi ie s. Table A5 in Appendix A.5 shows
con igu a ions o all he models o all he expe imen s. Table A4 in Appendix A.4 p o ide he de ails
abou he all he da ase s used in his a icle.
Among he eigh da ase s (Appendix A.4 lis s he desc ip ion o all he da ase s used in he a icle),
he Mice P o ein [
62
], Mammog aphic Mass [
63
], B eas Cance 569 and 669 [
64
,
65
] da a pe ain o
he medical ield, Glass Iden i ica ion [
66
] da a ep esen ing o ensic science, C edi App o al [
67
]
ep esen s economic da a, I is [
68
] is a bo anical da a se , and Wine Recogni ion [
69
] is a da a se
o chemical composi ion analysis. The expe imen s pe o med wi h hese da ase s we e akin
o he in es iga ions done wi h Toy-da a (in Sec ion 4), and UCIHAR da a (in Sec ion 5.2), i.e.,
K-means algo i hm used as concep iden i ie , whe e ‘K’ is he numbe o class labels o each da ase ,
he hie a chy is se o ha e a dep h o wo laye s (one Inpu and one abs ac concep laye ). Fo
e e y da ase , models we e gene a ed using hi y i e a ions in nine Resea ch Designs (RDs) ( e e
he Table A3 in Appendix A.2). In e e y RD 10-Fold c oss- alida ion was applied o de e mine
he pe o mance o he models. An agg ega e o p ecision, ecall, F1-Sco e, and accu acy o all olds
in all RDs was calcula ed o all he da ase s, as shown in Figu e 11a. F om he Figu e 11a i can be
obse ed ha wi h Mice P o ein da a RANs sco es 99.99% (ca.) o all e alua ion me ic, whe eas o
I is, Glass Iden i ica ion, B eas Cance , and Wine Recogni ions he obse a ions we e con incing, i.e.,
abo e 89.00% (ca.). In all he olds o nine RDs ROC cu es we e also plo ed o each class label o
he eigh da ase s, he mean AUC o each class o he da ase s is shown in Figu e 11b. The e alua ion
me ics and ROC-AUC analysis
(Figu e 11a,b espec i ely)
display he RAN’s capabili y in machine
lea ning asks wi h di e en kind o da ase s.
The same p ocedu e was applied o ob ain a e age P ecision, Recall, F1-Sco e and Accu acy o all
he da ase s wi h i e o he classi ie s (i.e., RBM+, KNN, LR, MLP, and SGD). Table 6shows he o e all
compa ison. I is wo h no ing ha being dynamic and unsupe ised RANs modeling pe o med qui e
sa is ac o ily especially wi h Mice P o ein da a, whe e i ou pe o med SGD and RBM+, was ound
compe en wi h LR, KNN and MLP classi ie s.
Appl. Sci. 2020,10, 1994 18 o 28
(a)
(b)
Figu e 11.
RANs pe o mance wi h eigh da ase s using P ecision, Recall, F1-Sco e and Accu acy
along wi h ROC-AUC analysis wi h Eigh benchma k da ase s [Mice P o ein (MP), B eas Cance 669
(BC1), B eas Cance 569 (BC2), C edi App o al (CA), IRIS da a (ID), Mamog aphic Mass (MM), Wine
Recogni ion (WR) and Glass Iden i ica ion (GI)]. (
b
) shows he plo o pe cen age AUC o classes 1 o
8. Fo each da ase class labels o he g aph is se ially mapped as: Mice p o ein (c-CS-s [Class-1], c-CS-m
[Class-2], c-SC-s [Class-3], c-SC-m [Class-4], -CS-s [Class-5], -CS-m [Class-6], -SC-s [Class-7] and -SC-m
[Class-8]); Mammog aphic Mass (Benign [Class-1] and Malignan [Class-2]); C edi App o al (Pos i i e
[Class-1] and Nega i e [Class-2]); IRIS) (Se osa [Class-1], Ve sicola [Class-2] and Ve ginica [Class-3]); B eas
Cance 569 (Benign [Class-1] and Malignan [Class-2]); B eas Cance 669 (Benign [Class-1] and Malignan
[Class-2]), Wine Recogni ion (Class-1, Class-2 and Class-3)Glass Iden i ica ion (Window Glass [Class-1]
and Non-Window Glass [Class-2]). (
a
) RANs pe o mance wi h eigh di e en da ase s depic ing RANs
apposi eness wi h da a belonging o dis inc domains; (
b
) Obse ed A ea Unde Cu e (AUC) while
pe o ming ROC cu e analysis o RANs model gene a ed wi h eigh di e en da ase s.
Figu e 12 shows ou g aphs depic ing RANs pe o mance wi h di e en benchma k da a
se s. These g aphs display an impo an aspec o RANs modeling and i s pe o mance beha io
when e alua ed o di e en esea ch design Figu e 12. The p ecision, ecall, F1-Sco e, and accu acy
ajec o ies o Human Ac i i y Recogni ion (HAR), B eas Cance 669 (BC1), Toy-da a (TD) and Mice
P o ein (MP) Da a is almos s aigh . The e alua ion plo s o Glass Iden i ica ion (GI), Wine Recogni ion
(WR), Mammog aphic Mass (MM), B eas cance 569 (BC2) and Mice P o ein (MP) da ase s show
a minimal decline in obse a ions w. . RD-1 and RD-9 Resea ch Design. On he con a y, esul s
Appl. Sci. 2020,10, 1994 19 o 28
om IRIS Da a (ID) and C edi App o al (CA) da ase depic ed a highe alue while compa ing
he e alua ion o RD-1 wi h RD-9 Resea ch Designs o hese da a se s. P incipally, he esul s o all
ou me ics o e alua ion ob ained simila esul s (wi h ma ginal a ia ion) i espec i e o he Tes
and T ain da a a io. This is a no able obse a ion because i shows ha he RAN’s app oach ob ains
a sa is ac o y esul e en when ained wi h a small amoun o da a.
Table 6. RANs compa ison wi h eigh da ase s belonging o di e en domains.
Da a Algo P ecision (%) Recall (%) F1-Sco e (%) Accu acy (%) Da a Algo P ecision (%) Recall (%) F1-Sco e (%) Accu acy (%)
RBM+ 43.45 ±44.07 53.50 ±38.23 45.46 ±43.36 53.50 ±38.23 RBM+ 93.60 ±2.69 93.51 ±2.77 93.46 ±2.86 93.51 ±2.77
KNN 98.63 ±3.97 98.34 ±4.84 98.07 ±5.65 98.34 ±4.84 KNN 99.80 ±0.59 99.79 ±0.62 99.78 ±0.63 99.79 ±0.62
LR 98.99 ±1.94 98.28 ±3.38 98.14 ±3.71 98.28 ±3.38 LR 99.89 ±0.07 99.89 ±0.07 99.89 ±0.07 99.89 ±0.07
MLP 98.54 ±2.19 98.23 ±2.71 97.83 ±3.34 98.23 ±2.71 MLP 98.67 ±0.94 98.65 ±0.96 98.64 ±0.96 99.89 ±0.07
RAN 99.98 ±0.06 99.97 ±0.06 99.89 ±0.06 99.97 ±0.06 RAN 93.17 ±0.36 92.97 ±0.36 92.87 ±0.42 92.97 ±0.36
Mice
P o ein
SGD 99.11 ±1.84 98.84 ±2.46 98.68 ±2.81 98.84 ±2.46
B eas
Cance 569
SGD 99.87 ±0.13 99.85 ±0.18 99.83 ±0.20 99.85 ±0.18
RBM+ 95.72 ±3.62 95.34 ±4.60 95.13 ±5.16 95.34 ±4.60 RBM+ 76.44 ±12.50 75.63 ±12.98 74.04 ±14.59 75.63 ±12.98
KNN 99.46 ±0.88 99.44 ±0.93 99.43 ±0.94 99.44 ±0.93 KNN 95.48 ±0.16 95.46 ±0.17 95.46 ±0.17 95.46 ±0.17
LR 99.16 ±0.17 99.14 ±0.17 99.15 ±0.17 99.14 ±0.17 LR 95.06 ±0.38 95.04 ±0.39 95.04 ±0.39 95.04 ±0.39
MLP 98.96 ±0.76 98.95 ±0.76 98.95 ±0.77 98.95 ±0.76 MLP 98.02 ±1.32 98.00 ±1.34 97.99 ±1.34 98.00 ±1.34
RAN 95.18 ±0.25 95.15 ±0.24 95.11 ±0.25 95.15 ±0.24 RAN 80.67 ±1.37 79.58 ±1.05 79.66 ±1.13 79.58 ±1.05
B eas
Cance 669
SGD 99.88 ±0.16 99.88 ±0.16 99.18 ±0.16 99.88 ±0.16
C edi
App o al
SGD 99.77 ±0.39 99.75 ±0.40 99.75 ±0.40 99.75 ±0.40
RBM+ 82.58 ±10.29 84.19 ±4.90 80.61 ±8.42 84.19 ±4.90 RBM+ 84.85 ±16.54 85.18 ±14.98 82.42 ±20.30 85.18 ±14.98
KNN 94.08 ±12.12 95.97 ±7.32 94.82 ±10.59 95.97 ±7.32 KNN 99.65 ±0.88 99.64 ±0.89 99.64 ±0.89 99.64 ±0.89
LR 99.52 ±0.18 99.49 ±0.18 99.49 ±0.18 99.49 ±0.18 LR 99.41 ±0.30 99.40 ±0.30 99.40 ±0.30 99.40 ±0.30
MLP 93.78 ±1.40 93.28 ±1.52 92.85 ±1.64 93.28 ±1.52 MLP 98.91 ±2.11 98.79 ±2.35 98.79 ±2.35 98.79 ±2.35
RAN 90.07 ±0.43 89.18 ±1.23 89.32 ±1.10 89.18 ±1.23 RAN 80.28 ±0.18 79.20 ±0.23 79.08 ±0.24 79.20 ±0.23
Glass
Iden i ica ion
SGD 97.95 ±0.66 97.87 ±0.69 97.82 ±0.70 97.87 ±0.69
Mamog aphic
Mass
SGD 99.96 ±0.03 99.94 ±0.07 99.93 ±0.09 99.94 ±0.07
RBM+ 79.81 ±11.91 77.41 ±11.88 70.66 ±16.28 77.41 ±11.88 RBM+ 56.00 ±25.66 67.05 ±16.91 59.07 ±21.91 67.05 ±16.91
KNN 90.41 ±28.77 92.80 ±21.61 91.00 ±27.01 92.80 ±21.61 KNN 90.74 ±26.00 92.88 ±19.48 91.14 ±24.70 92.88 ±19.48
LR 97.38 ±4.15 96.64 ±5.65 96.45 ±6.12 96.64 ±5.65 LR 94.14 ±1.55 93.13 ±1.82 93.00 ±1.92 93.13 ±1.82
MLP 97.31 ±0.71 96.86 ±1.13 96.81 ±1.21 96.86 ±1.13 MLP 97.44 ±0.51 97.33 ±0.59 97.32 ±0.59 97.33 ±0.59
RAN 95.43 ±0.67 95.02 ±0.94 94.98 ±0.98 95.02 ±0.94 RAN 94.87 ±0.91 94.34 ±1.00 94.29 ±1.01 94.34 ±1.00
IRIS
SGD 94.47 ±6.40 94.46 ±5.20 93.31 ±6.78 94.46 ±5.20
Wine
Recogni ion
SGD 98.13 ±0.70 97.91 ±0.75 97.91 ±0.76 97.91 ±0.75
Besides classi ica ion compa ison, he RAN’s modeling is compa ed wi h he i e classi ie s
based upon se en ea u es: (1) Whe he he modeling in g aph-based; (2) whe he he modeling has
a dynamic opology; (3) and (4) whe he modeling can educe o expand he dimension o he da a;
(5) whe he modeling can pe o m classi ica ion; and (7) whe he modeling is biologically inspi ed
o no . Table 7de ails his compa a i e s udy. I can be obse ed om his able ha RAN is closely
ela ed o he models ha a e biologically inspi ed i.e., RBM and MLP.
Table 7. Fea u e based compa a i e s udy o RANs wi h i e modeling echniques.
Fea u es Models RBM K-NN LR MLP RANs SGD
G aph-Based Yes No No Yes Yes No
Dynamic Topology No No No No Yes No
Dimension Reduc ion Yes Yes No Yes Yes No
Dimension Expansion May be No No May be Yes No
Unisupe ised Yes No No No Yes No
Suppo s Classi ica ion Yes Yes Yes Yes Yes Yes
Bio-inspi ed Yes No No Yes Yes No
Appl. Sci. 2020,10, 1994 20 o 28
(a) (b)
(c) (d)
Figu e 12.
RANs e alua ion me ic (p ecision, ecall, F1-Sco e and accu acy) alue beha io wi h
espec o a ying es and ain da a a io o e en da ase s [Mice P o ein (MP), B eas Cance 669
(BC1), B eas Cance 569 (BC2), C edi App o al (CA), IRIS da a (ID), Mamog aphic Mass (MM),
Human Ac i i y Recogni ion (HAR), Toy-da a(TD), Wine Recogni ion (WR) and Glass Iden i ica ion
(GI)] (a) P ecision; (b) Recall; (c) F1-Sco e; (d) Accu acy.
7. Conclusions and Fu u e Wo k
To comp ehend and easoning o emo ions, ideas, e c., i is e iden o unde s and abs ac
concep s because hey a e pe cei ed di e en ly om conc e e concep s. The e ha e been no able
e o s o s udy Conc e e concep s ( ea u es like walking o ing edien s), bu p og ess in in es iga ing
abs ac concep s (gene ic ea u es such as is-mo ing o ecipe) is ela i ely less. This a icle p oposes
an unsupe ised compu a ional modeling app oach, named Regula ed Ac i a ion Ne wo ks (RANs),
ha has an e ol ing opology and lea ns a ep esen a ion o abs ac concep s. The RAN’s me hodology
was exempli ied h ough a UCI’s IRIS da ase , yielding a sa is ac o y pe o mance e alua ion o 95%
(ca.) o p ecision, ecall, F1-Sco e and accu acy me ics, along wi h an a e age AUC o 99% (ca.) o
all he h ee classes in he da ase . These e alua ion esul no only showed he classi ica ion capabili y
o RANs bu also p o ed he hypo hesis o he expe imen i.e., he h ee newly c ea ed nodes in
he Laye -1 symbolically ep esen he h ee classes o IRIS da a as abs ac concep s.
Ano he expe imen wi h IRIS da a displayed he cha ac e is ic o RAN’s deep hie a chy
gene a ion and independence in choosing he concep Iden i ie . Wi h he aid o he Concep Hie a chy
C ea ion algo i hm (p oposed in Sec ion 5.1), he e ol ing na u e o RAN’s modeling is shown using
he A ini y P opaga ion clus e ing algo i hm (as an al e na e concep Iden i ie ins ead o he K-means
algo i hm as used in modeling wi h a Toy-da a p oblem). Wi h he gene a ed model i was shown
ha he model dynamically g ew o a dep h o six laye s and pe o med wi h P ecision o 94.44% (ca.),
Recall o 93.33% (ca.), F1-Sco e o 93.26% (ca.) and Accu acy o 93.33% (ca.), along wi h an obse ed
AUC o 100% (ca.), 92% (ca.) and 94% (ca.) o he h ee classes o da a. This expe imen also highligh s
he applica ion o RANs modeling in da a dimension ans o ma ion and da a isualiza ion.
Modeling wi h UCI’s IoT based Home Ac i i y Recogni ion (UCIHAR) sma phone senso
da ase exhibi ed he RAN’s beha io o na u al iden i ica ion o gene ic concep s. The expe imen
hypo hesize ha six da a labels (ac i i y o walking, walking
_
ups ai s, walking
_
downs ai s,
Appl. Sci. 2020,10, 1994 21 o 28
si ing, s anding and laying) o he da ase a e o be iden i ied as mobile (walking, walking
_
ups ai s
and walking
_
downs ai s) and immobile (si ing, s anding and laying) abs ac concep s.
This hypo hesis was also p o en using classi ica ion ope a ion, whe e, he e alua ion o he model
shown a pe o mance o 99.85% (ca.) o all ou me ics and AUC o 99.9% (ca.) o bo h abs ac
concep s. The expe imen also demons a es how RAN can be used o model he da a om he IoT
domain in an unsupe ised manne .
The p oo o concep o RAN’s modeling as a Machine Lea ning classi ie was also p o ided wi h
eigh UCI benchma ks. I was iden i ied ha RAN’s app oach pe o med sa is ac o ily displaying
he bes ou come o 98.9% (ca.) wi h Mice P o ein da ase ( o all me ics). The compa ison o RAN’s
modeling wi h i e classi ie s subs an ia ed he e ec i eness o he p oposed me hodology. We also
obse ed ha he RAN’s pe o mance emained simila i espec i e o he size o ain da a. RAN was
also compa ed wi h he i e classi ie s based upon i s ea u es and i was obse ed ha RAN was
simila o bio-inspi ed models. The model p esen ed in his a icle is capable o modeling da a ha is
con ex which limi s he RAN’s pe o mance wi h non-con ex (o complex) da ase s. As u u e wo k,
we in end o imp o e RANs modeling ha can cap u e he non-con exi y in he da a and enhance
he pe o mance o he model wi h complex da ase s.
Au ho Con ibu ions:
R.S. pe o med s a e o he a , de eloped and implemen ed he me hodology, ca ied
ou da a selec ion and me hodology alida ion, and p epa ed he o iginal d a o he a icle. B.R. supe ised he
esea ch wo k pe o med he o mal analysis, e iew and edi ing, ook ca e o unding. A.M.P. concei ed he
s udy plan and me hodology, supe ised he in es iga ion, me hodology de elopmen and implemen a ion. F.A.C.
supe ised he esea ch wo k, pe o med o mal analysis, e iew and edi ion, managed unding. All au ho s
ha e ead and ag eed o he published e sion o he manusc ip .
Funding:
The wo k p esen ed in his pape was pa ially ca ied ou in he scope o he SOCIALITE
P ojec (PTDC/EEI-SCR/2072/2014), co- inanced by COMPETE 2020, Po ugal 2020—Ope a ional P og am
o Compe i i eness and In e na ionaliza ion (POCI), Eu opean Union’s ERDF (Eu opean Regional De elopmen
Fund), and he Po uguese Founda ion o Science and Technology (FCT).
Con lic s o In e es : The au ho s decla e no con lic o in e es .
Abb e ia ions
The abb e ia ions used in his manusc ip :
ACL Abs ac Concep Labeling
AUC A ea Unde Cu e
BC1 B eas Cance 669 Da ase
BC2 B eas Cance 569 Da ase
CA C edi App o al Da ase
CHC Concep Hie a chy C ea ion
CI Concep Iden i ica ion
CLS Cu en Laye Size
CRDP Clus e Rep esen a i e Da a Poin
DoC Deg ee o Con idence
GDF Geome ic Dis ance Func ion
GI Glass Iden i ica ion Da ase
HAR Human Ac i i y Recogni ion Da a
ID IRIS Da ase
ILL In e Laye Lea ning
ILW In e Laye Weigh s
K-NN K Nea es Neighbo

Appl. Sci. 2020,10, 1994 22 o 28
MLP Mul ilaye Pe cep on
MM Mammog aphy Mass Da ase
MP Mice P o ein Da ase
MRI Magne ic Resonance Imaging
RANs Regula ed Ac i a ion Ne wo ks
RBM Res ic ed Bol zmann Machine
RBM+ RBM pipe-lined wi h Logis ic Reg ession
ROC Recei e Ope a ing Cha ac e is ic
SGD S ochas ic G adien Descen
STF Simila i y T ansla ion Func ion
UAP Upwa d Ac i a ion P opaga ion
Appendix A
Appendix A.1. Da a and Sc ip s
This sec ion p o ides links o download he da a and py hon sc ip used o pe o m RANs
modeling expe imen s, men ioned in his a icle. The da a and sc ip olde s can be downloaded
om he web URL men ioned in Table A1. The da a olde con ains many iles and he di ec
pa h o he iles a e p o ided in he Table A1. Simila ly, he sc ip olde RAN_V2.0 also con ains
many olde s whe e Folde RAN consis o he py hon sc ip s. The olde Obse a ions is o s o ing
he ou come o he expe imen s, a he beginning o each expe imen he emp y olde in di ec o y
emp y_passes_ o _Expe imen _Obse a ions mus be copied in o he Obse a ion di ec o y. The py hon
sc ip ela ed o RANs modeling is in olde RAN, he desc ip ion is men ioned in he Table A1.
Table A1. Da a and Py hon Sc ip o RANs modeling.
Type Desc ip ion File-pa h
Download link h ps://www.d opbox.com/sh/3410oze u3o5opm/AAA24aUG US1i7xHKp9kyzRKa?dl=0
IRIS Da a da a/i is_wi h_label.cs
Mice P o ein da a da a/Da a_co ex_Nuclea /mice_wi h_class_label.cs
Glass Iden i ica ion da a da a/newDa aToExplo e/new/GlassIden i ica ionDa abase/RANs o m.cs
Wine Recogni ion da a da a/newDa aToExplo e/new/WineRecogni ionDa a/RansFo m.cs
B eas cance 669 da a da a/newDa aToExplo e/new/b eas Cance Da abases/699RansFo m.cs
B eas Cance 559 da a da a/newDa aToExplo e/new/b eas Cance Da abases/569RansFo m.cs
UCIHAR da a da a/UCI_HAR_Da ase .cs
Mamog aphic Mass da a da a/newDa aToExplo e/new/Mammog aphicMassDa a/RansFo m1
C edi App o al da a da a/newDa aToExplo e/new/C edi App o al/RansFo m.cs
Da a
Toy-da a da a da a/ oyda a5clus e sRAN.cs
Download Link h ps://www.d opbox.com/sh/ cw1cj4ce1 3zic/AAAm6wVTj2qsLZ1lbc3kn4MPa?dl=0
RANs classes and me hods RAN_V2-0/RAN/RAN_k old.py
Me hods RAN_V2-0/RAN/Laye .py
U ili ies like Labeling and plo ing RAN_V2-0/RAN/U ilsRAN.py
Sc ip
Py hon Sc ip o using RANs RAN_V2-0/RAN/RAN_inpu _T1.py
The implemen ed RANs modeling ool in py hon akes inpu da a in a speci ic o ma
(shown in Table A2)
. Besides he da a, he inpu s equi e a heade as he i s ow s acked o e
he o iginal da a. Each heade elemen , [
H−
1,
H−
2, .......,
H−n
], is he Maximum alue possible o
hei espec i e column ( ea u e, o dimension). I is assumed ha he minimum alue o he column
is ze o, i i is no hen he da a mus be ans o med be ween ze o and he maximum posi i e alue as
desc ibed in Sec ion 4.1.
Appl. Sci. 2020,10, 1994 23 o 28
Table A2. Inpu Da a Fo ma o implemen ed RANs Modeling.
Heade H-1 H-2 .............. H-n
D-1 D-2 .............. D-n
D-1 D-2 ............... D-n
Da a
Ins ances .
.
.
.
.
.
..............
...............
...............
.
.
.
D-1 D-2 .............. D-n
Appendix A.2. Model Con igu a ions and Resea ch Design
Va ious expe imen s, epo ed in his a icle, we e conduc ed wi h se e al da ase s, using
six modeling echniques including he p oposed me hodology i.e., RANs modeling. Table A5 in
Appendix A.5 shows con igu a ions o all he models o all he expe imen s. The expe imen s we e
ca ied ou using py hon p og aming language, and implemen a ions o Res ic ed Bol zmann Machine
pipelined wi h Logis ic Reg ession (RBM+), Logis ic Reg ession (LR), K-Nea es Neighbo (K-NN),
Mul ilaye Pe cep on (MLP), and S ochas ic G adien Descen (SGD) models o Sciki -lea n lib a y [
59
].
I is o be no ed ha expe imen s wi h RBM we e ca ied ou , pipelined wi h he LR algo i hm using
he de aul con igu a ion o i s implemen a ion in sciki -lea n lib a y. The Table A3 lis s he nine
Resea ch Designs (RD) used in he expe imen s o his a icle. In e e y RD he a io o he T ain
and Tes da a is a ied o cap u e he abili y o he classi ie being inspec ed.
Table A3. T ain and Tes da a dis ibu ions in nine Resea ch Designs (RDs).
RD-1 RD-2 RD-3 RD-4 RD-5
T ain Tes T ain Tes T ain Tes T ain Tes T ain Tes
90% 10% 80% 20% 70% 30% 60% 40% 50% 50%
RD-1 RD-7 RD-8 RD-9
T ain Tes T ain Tes T ain Tes T ain Tes
40% 60% 30% 70% 20% 80% 10% 90%
Appendix A.3. Abs ac Concep Labeling (ACL)
This me hod is op ional and use ul when he inpu da a is labeled. Wi h his mechanism,
we associa e an iden i ie o e e y Abs ac concep node N
j
. Ha ing gene a ed he RANs model wi h
CI, hen ough CC, ILL, inpu da a is so ed label-wise, and pe o m UAP ope a ion. The p opaga ed
da a is inspec ed class-wise, and label node N
j
wi h a class-name o which i go he maximum coun o
he highes ac i a ion. Fo example, suppose inpu da a o class-
X
has 100 ins ances, a e inspec ing
he p opaga ed da a, i is obse ed ha node N
1
ecei ed highes ac i a ion 74- imes, whe eas,
wi h emaining 26 cases o he nodes expe ienced maximum ac i a ion, he e o e, we ecognize
node N
1
as ep esen a i e o class-
X
.T ue-Labels a e iden i ied by mapping each class o he inpu
ins ance di ec ly o i s espec i e node ep esen a i e Obse ed-Labels a e ob ained by p opaga ing e e y
es -ins ance h ough UAP ope a ion, inspec ing which Abs ac node ecei ed he highes ac i a ion
o ha da a-uni , and label i wi h he class ep esen ed by ha node. T ue-Labels and Obse ed-Labels
a e used o alida e he model’s pe o mance.
Appl. Sci. 2020,10, 1994 24 o 28
Appendix A.4. Da ase Desc ip ion
Table A4. Da ase desc ip ion.
Da ase A ibu e Class Sou ce
Name Type Size Balanced Type Size # Name
Mice P o ein Mul i a ia e 1080 yes Real 82 8 UCI
B eas Cance 569 Mul i a ia e 569 yes Real 32 2 UCI
B eas Cance 669 Mul i a ia e 669 yes In ege 10 2 UCI
C edi App o al Mul i a ia e 690 yes Mixed 15 2 UCI
Glass Iden i ica ion Mul i a ia e 214 yes Real 10 7 UCI
Mammog aphic mass Mul i a ia e 961 yes In ege 6 2 UCI
IRIS Mul i a ia e 150 yes Real 4 3 UCI
Wine Recogni ion Mul i a ia e 178 yes Mixed 13 3 UCI
Human Ac i i y Recogni ion Mul i a ia e, Time-Se ies 10299 yes Real 561 6 UCI
Toy-da a Mul i a ia e 1500 yes Real 2 5 Sel
UCI- Uni e si y o Cali o nia I ine’s Machine Lea ning Reposi o y; Sel - A i icially gene a ed da ase
Appendix A.5. Mul i-Class ROC Analysis wi h RANs Modeling
This s udy is ca ied ou by wo p ocesses, i s he inpu ue-labels a e ans o med in o a sepa a e
ec o o bina y labels, indi idually o all Abs ac nodes (i.e., 1 o class c1, 0 o all o he classes),
second, calcula ing he con idence sco e o each ins ance o he inpu da a (o es -da a). Bo h p ocesses
a e desc ibed as ollows:
1Node-wise bina y ans o ma ion o T ue-Labels
: Fo example, suppose he e a e h ee classes
(c1, c2, c3) ep esen ed by h ee abs ac nodes (n1, n2, and n3) in RANs model a Laye -1,
and le ue-label be [c1, c2, c2, c1, c2, c3, c3] o 7 es ins ances, hen o node n1 label will be
[1, 0, 0, 1, 0, 0, 0] whe e 1 ep esen s class c1, and 0 depic s o he s (i.e., c2, and c3).
2Node-wise con idence-sco e calcula ion
: This is calcula ed by a e aging ac i a ion- alue
and con idence-indica o o ac i a ion o an inpu ins ance a an Abs ac node. Ac i a ion- alue
is an indi idual ac i a ion o an ac i a ion ec o ob ained by p opaga ing up he da a using
UAP mechanism o RANs whe eas, con idence-indica o is calcula ed by min-max no maliza ion
ope a ion o ac i a ion ec o . Fo example, a e UAP ope a ion each node (n1, n2, and n3)
ecei es ac i a ion [0.89, 0.34, 0.11] (a ec o o ac i a ion), and con idence-indica o is min-max
([0.89, 0.34, 0.11]) = [1.0, 0.29, 0.0]. and he con idence-sco e o nodes n1 = (0.89 + 1.0)/2.0 = 0.95,
n2 = (0.34 + 0.29)/2.0 = 0.32, and n3 = (0.11 + 0.11)/2.0 = 0.05.
Table A5. Da ase speci ic con igu a ion de ails o models.
Da a Algo Con igu a ions Da a Algo Con igu a ions
RBM +
LR
L = 0.000001, i e = 500, comp = 20
max_i e = 30, C = 70
RBM +
LR
L = 0.06, i e = 500, comp = 10
max_i e = 10, C = 1
K-NN n_neighbo s = 30 K-NN n_neighbo s = 15
LR max_i e = 10, C = 1 LR max_i e = 30, C = 1
MLP Rs = 1, hls = 10, i e = 250 MLP Rs = 1, hls = 10, i e = 400
RANs CLS = 5, Desi ed_dep h = 1 RANs CLS = 2, Desi ed_dep h = 1
Toy-da a
SGD alpha = 0.0001, n_i e = 5, epsilon = 0.25
UCIHAR
SGD alpha = 0.1, n_i e = 10, epsilon = 0.25
RBM +
LR
L = 0.1, i e = 500, comp = 20
max_i e = 30, C = 30
RBM +
LR
L = 0.006, i e = 100, comp = 10
max_i e = 30, C = 1
K-NN n_neighbo s = 15 K-NN n_neighbo s = 30
LR max_i e = 4, C = 0.00001 LR max_i e = 10, C = 0.001
MLP Rs = 1, hls = 10, i e = 300 MLP Rs = 1, hls = 10, i e = 200
RANs CLS = 8, Desi ed_dep h = 1 RANs CLS = 2, Desi ed_dep h = 1
Mice
P o ein
SGD alpha = 0.1, n_i e = 10, epsilon = 0.25
B eas
Cance 569
SGD alpha = 0.0001, n_i e = 5, epsilon = 0.25
Appl. Sci. 2020,10, 1994 25 o 28
Table A5. Con .
Da a Algo Con igu a ions Da a Algo Con igu a ions
RBM +
LR
L = 0.001, i e = 100, comp = 10
max_i e = 30, C = 1
RBM +
LR
L = 0.006, i e = 100, comp = 10
max_i e = 30, C = 1
K-NN n_neighbo s = 10 K-NN n_neighbo s = 30
LR max_i e = 10, C = 0.001 LR max_i e = 10, C = 0.001
MLP Rs = 1, hls = 10, i e = 200 MLP Rs = 1, hls = 10, i e = 200
RANs CLS = 2, Desi ed_dep h = 1 RANs CLS = 2, Desi ed_dep h = 1
B eas
Cance 669
SGD alpha = 0.0001, n_i e = 5, epsilon = 0.25
C edi
App o al
SGD alpha = 0.0001, n_i e = 5, epsilon = 0.25
RBM +
LR
L = 0.001, i e = 400, comp = 10
max_i e = 30, C = 5
RBM +
LR
L = 0.01, i e = 500, comp = 20
max_i e = 30, C = 5
K-NN n_neighbo s = 15 K-NN n_neighbo s = 30
LR max_i e = 5, C = 0.00001 LR max_i e = 5, C = 1
MLP Rs = 1, hls = 10, i e = 200 MLP Rs = 1, hls = 10, i e = 250
RANs CLS = 2, Desi ed_dep h = 1 RANs CLS = 2, Desi ed_dep h = 1
Glass
Iden i ica ion
SGD alpha = 0.01, n_i e = 10, epsilon = 0.25
Mamog aphic
Mass
SGD alpha = 0.0001, n_i e = 5, epsilon = 0.25
RBM +
LR
L = 0.01, i e = 1000, comp = 20
max_i e = 30, C = 5
RBM +
LR
L = 0.01, i e = 500, comp = 20
max_i e = 30, C = 50
K-NN n_neighbo s = 15 K-NN n_neighbo s = 15
LR max_i e = 10, C = 1 LR max_i e = 10, C = 0.01
MLP Rs = 1, hls = 10, i e = 400 MLP Rs = 1, hls = 10, i e = 300
RANs CLS = 3, Desi ed_dep h = 1 RANs CLS = 3, Desi ed_dep h = 1
IRIS
SGD alpha = 0.01, n_i e = 10, epsilon = 0.25
Wine
Recogni ion
SGD alpha = 0.01, n_i e = 10, epsilon = 0.25
LR-Lea ning Ra e; i e -I e a ions; comp-Numbe o Hidden Componen s o RBM; RS-Random
S a e; hls = Hidden Laye Sizes; CLS-Numbe o clus e s a he inpu laye o RANs.
Re e ences
1.
Kie e , M.; Pul e mülle , F. Concep ual ep esen a ions in mind and b ain: Theo e ical de elopmen s,
cu en e idence and u u e di ec ions. Co ex 2012,48, 805–825. [C ossRe ] [PubMed]
2.
Xiao, P.; Toi onen, H.; G oss, O.; Ca doso, A.; Co eia, J.A.; Machado, P.; Ma ins, P.; Oli ei a, H.G.; Sha ma, R.;
Pin o, A.M.; e al. Concep ual ep esen a ions o compu a ional concep c ea ion.
ACM Compu . Su . 2019
,52, 1–33.
[C ossRe ] [PubMed]
3.
Hill, F.; Ko honen, A. Lea ning abs ac concep embeddings om mul i-modal da a: Since you p obably
can’ see wha I mean. In P oceedings o he 2014 Con e ence on Empi ical Me hods in Na u al Language
P ocessing (EMNLP), Doha, Qa a , 25–29 Oc obe 2014; pp. 255–265. [C ossRe ]
4.
B a e , T.; Ba ch, D.; Cohen, J. Cogni ion and con ol in schizoph enia: A compu a ional model o dopamine
and p e on al unc ion. Biol. Psychia y 1999,46, 312–328.
5.
O’Reilly, R.C. Biologically based compu a ional models o high-le el cogni ion. Science
2006
,314, 91–94.
[C ossRe ]
6.
Rolls, E.; Loh, M.; Deco, G.; Win e e , G. Compu a ional models o schizoph enia and dopamine modula ion
in he p e on al co ex. Na . Re . Neu osci. 2008,9, 696–709. [C ossRe ]
7.
Kyaga, S.; Landén, M.; Boman, M.; Hul man, C.M.; Långs öm, N.; Lich ens ein, P. Men al illness, suicide
and c ea i i y: 40-Yea p ospec i e o al popula ion s udy. J. Psychia . Res. 2013,47, 83–90. [C ossRe ]
8.
Ande son, J.R.; Ma essa, M.; Lebie e, C. ACT-R: A heo y o highe le el cogni ion and i s ela ion o
isual a en ion. Hum. Compu . In e ac . 1997,12, 439–462. [C ossRe ]
9.
Hin on, G. A p ac ical guide o aining es ic ed bol zmann machines. Momen um
2010
,9, 926. [C ossRe ]
10.
Vincen , P.; La ochelle, H.; Lajoie, I.; Bengio, Y.; Manzagol, P.A. S acked denoising au oencode s:
Lea ning use ul ep esen a ions in a deep ne wo k wi h a local denoising c i e ion. J. Mach. Lea n. Res.
2010,11, 3371–3408.
11.
Collobe , R.; Wes on, J. A uni ied a chi ec u e o na u al language p ocessing: Deep neu al ne wo ks wi h
mul i ask lea ning. In P oceedings o he 25 h In e na ional Con e ence on Machine Lea ning, Helsinki,
Finland, 5–9 July 2008; ACM: New Yo k, NY, USA, 2008; pp. 160–167.