Uni e sidad Complu ense de Mad id
Facul ad de In o má ica
Final Yea P ojec
Compu e Science Enginee ing
Academic Yea 2022 / 2023
Image ecogni ion-based access secu i y /
Segu idad de acceso con econocimien o de imágenes
S uden
Pablo Cas ellanos Ga cía
Ad iso
Juan Ca los Fabe o Jiménez
Con en s
1 In oduc ion 1
1.1 Mo i a ion ........................................ 1
1.2 P ojec goals ...................................... 2
1.3 Documen s uc u e .................................. 4
2 S a e o he A 5
2.1 Simila p ojec s ..................................... 5
2.1.1 Homemade IoT p ojec s ............................ 5
2.1.2 Comme cial p oduc s .............................. 7
2.2 Tools & echnology ................................... 9
2.2.1 ESP32-CAM .................................. 9
2.2.2 Run- ime ope a ing sys em .......................... 13
2.2.3 Teleg am bo API ............................... 15
3 Sys em a chi ec u e & design 19
3.1 Fea u es ......................................... 19
3.2 Sys em a chi ec u e ................................... 22
4 Implemen a ion 25
4.1 ESP32-CAM Se e .................................. 25
4.1.1 esp32-cam-secu i y.ino ........................... 25
4.1.2 app_h pd.cpp ................................. 26
4.1.3 bo _api.cpp .................................. 32
4.2 Teleg am bo ...................................... 33
4.2.1 Bo .ja a .................................... 33
4.2.2 UDPSe e .ja a ................................ 37
4.2.3 O he iles .................................... 38
4.3 Implemen a ion p oblems ............................... 39
iii
4.3.1 Lib a y e o s .................................. 39
4.3.2 Mul i h eaded se e .............................. 40
4.3.3 Teleg am bo in ESP32 ............................ 41
5 Conclusions 43
5.1 Resul s .......................................... 43
5.2 Limi a ions ....................................... 44
5.3 Fu he imp o emen s ................................. 44
Abs ac
The In e ne o Things (IoT) and A i icial In elligence (AI) a e wo o he mos impo an
opics in Compu e Science oday, and wo o he mos comme cially success ul. The main ocus
o his p ojec is o use a combina ion o he wo—wha is known as A i icial In elligence o
Things, o AIoT— o design and build a secu i y sys em ha , h ough he use o acial ecogni ion,
is able o a) g an o deny access o a hypo he ical loca ion au onomously; b) send ale s o use s
abou e en s like de ec ed o ecognized aces; and c) unc ion as a secu i y came a.
The sys em will be implemen ed using an ESP32-CAM mic ocon olle , and he no i ica ions
o use s will be sen ia a Teleg am bo .
Resumen
El In e ne de las Cosas (IoT, po sus siglas en inglés) y la In eligencia A i icial (IA) son dos
de los ámbi os más impo an es en la In o má ica de hoy en día, los dos con g an éxi o come cial.
El obje i o p incipal de es e p oyec o es usa una combinación de ambos—la llamada In eligencia
A i icial de las Cosas o AIoT—pa a diseña y cons ui un sis ema de segu idad que, haciendo
uso de ecnología de econocimien o acial, sea capaz de a) concede o denega acceso a una
localización hipo é ica de o ma au ónoma; b) en ia ale as a usua ios sob e e en os como una
ca a de ec ada o econocida; y c) hace de cáma a de segu idad.
El sis ema se á implemen ado usando un mic ocon olado ESP32-CAM y las no i icaciones
a los usua ios se ha án a a és de un bo de Teleg am.
Keywo ds In e ne o Things, IoT, A i icial In elligence, AI, acial ecogni ion, secu i y
sys em, ESP32, Teleg am
Palab as cla e In e ne de las Cosas, IoT, In eligencia A i icial, IA, econocimien o acial,
sis ema de segu idad, ESP32, Teleg am

Chap e 1
In oduc ion
The ocus o his p ojec is on designing and building a secu i y sys em based on acial ecogni ion,
which will be able o g an o deny access o a hypo he ical loca ion (a p i a e home, o ins ance)
and send ale s o he use s.
This chap e con ains some mo i a ion o why his is an in e es ing p ojec in oday’s wo ld
and lis s he main goals o be accomplished. Finally, i con ains a b ie explana ion o he
con en s o his documen and how i is s uc u ed.
1.1 Mo i a ion
Thanks o he mass p oduc ion o inexpensi e, small and ene gy-e icien compu e chips, in he
las ew decades i has become a common p ac ice o in eg a e hem in mos e e yday objec s
( hings)—any hing om a TV se o a ligh bulb o a washing machine can now be conside ed a
‘sma ’ de ice. On op o ha , gi en he as e and as e ne wo k connec ions ha a e a ailable
o mo e and mo e people, i is only na u al o ake ad an age o his high numbe o ‘in elligen ’
de ices by connec ing hem among hemsel es—some imes, bu no necessa ily, h ough he
In e ne —and ge ing hem o collabo a e, in wha has come o be known as he In e ne o
Things (IoT).
This kind o echnology has become pe asi e in oday’s wo ld and all o he op echnological
companies ha e no iced i s po en ial. Aco ding o Amazon, “ he In e ne o Things has a wide-
anging impac on human li e and wo k. I allows machines o do mo e hea y li ing, ake o e
edious asks and make li e mo e heal hy, p oduc i e, and com o able.” 1To men ion jus a
ew examples, in 2014 Amazon announced i s Amazon Echo sma speake s ha connec wi h
sma home appliances and espond o oice commands, and wo yea s la e Google ollowed wi h
1h ps://aws.amazon.com/wha -is/io /
1
Chap e 1. In oduc ion
hei Google Nes , which allows use s o “con ol hei sma home” using only hei oice. Mo e
ecen ly, gadge s such as sma doo bells and sma secu i y came as ha e gained popula i y.
“In his hype connec ed wo ld, digi al sys ems can eco d, moni o , and adjus each in e ac ion
be ween connec ed hings. The physical wo ld mee s he digi al wo ld—and hey coope a e.” 2
Howe e , no e e y hing is as ideal when i comes o he In e ne o Things as i may seem
a i s sigh . A e all, many o hese companies ha e a epu a ion o no aking use p i acy so
se iously as many would like, and some a e conce ned [1] abou gi ing hem access o mic ophones
and came as inside ou homes. O he p oblem ha his app oach en ails is he ac ha hese
de ices oo o en ely on he manu ac u e s’ se e s, so when—ine i ably— he e is an ou age
[2], hey become useless.
Pe haps he only echnology ha has gained mo e ac ion in mains eam media han IoT in
he ecen yea s is A i icial In elligence (AI), whose p ima y goal is o sol e complex p oblems
au oma ically, elie ing people om edious o epe i i e jobs ha a e none heless impo an o
do. Typical examples o ields in which AI has been success ully applied include au oma ic
ecommende sys ems—such as he in amous “algo i hms” employed by mos s eaming se ices
o social media pla o ms o decide wha con en o show i s use s—and image ecogni ion
sys ems—which can be used o de ec o es i es om sa elli e images [3] o umo s om MRI
scans [4], o name a ew.
E en wi h all i s po en ial o good, A i icial In elligence—mos no ably ace ecogni ion
sys ems—has also been used o no -so-noble ends. Fo ins ance, i plays a big ole in he mass
su eillance p og am [5] by he Chinese go e nmen and has e en been used o supply e idence
in cou wi h desas ous consequences [6] when he sys em makes a mis ake.
A i icial In elligence and he In e ne o Things a e wi hou a doub e y in e es ing ools
and some o he mos ele an ad ances made in he las decades, bu hey can also be used
agains he in e es s o he use , as he e idence shows. This is why his p ojec will ocus on
using hese wo echnologies o build a wo king p oo o concep o an access secu i y sys em in
which he end use is in ull con ol and is as independen as possible om hi d pa y se e s,
which could po en ially ende he whole sys em useless, and which will a emp o minimize
he nega i e consequences ha could a ise om he use o a acial ecogni ion sys em, which is
inhe en ly inexac .
1.2 P ojec goals
The p ima y goal o his p ojec is o build a wo king p o o ype o an access secu i y sys em
using an ESP32 mic ocon olle . To achie e his, i will use image ecogni ion echnology o
2h ps://www.o acle.com/in e ne -o - hings/wha -is-io /
2
P ojec goals 1.2
de ec human aces on ames and acial ecogni ion o egis e he aces o au ho ized people
and assign a unique iden i ie o each o hem. A e a success ul ace de ec ion, i s main unc ion
will be o pe o m ace ecogni ion on e e y de ec ed ace and:
a) “G an ” access o use s ma ching one o he egis e ed aces, in a way ha will aim o be
as lexible as possible.
b) “Deny” access o use s no ma ching any o he egis e ed aces, op ionally sending an ale
signalling ha an in ude has ied o access he sys em.
On op o his co e unc ionali y, he sys em will also:
•O e an in e ace o acili a e he addi ion and emo al o egis e ed aces.
•O e an in e ace ia an HTTP se e ha will allow o he ansmission o s a ic images o
li e ideo eed (op ionally including a ma k a ound each de ec ed ace) and o con igu a ion
o ce ain pa ame e s o he sys em.
•Allow use s o subsc ibe o a no i ica ion lis in o de o ecei e ale s whene e an
in ude —o any use —is de ec ed.
–Fo imp o ed secu i y, he sys em will op ionally ha e a whi elis o p i ileged use s
ha a e allowed o subsc ibe o he lis .
•Fo g ea e ease o use o he end use , hese no i ica ions will be sen di ec ly o hei
sma phones o compu e s by a Teleg am bo , which will also simpli y he con igu a ion o
he sys em pa ame e s.
As o he lea ning goals, du ing his p ojec I will:
•Ge amilia wi h de eloping so wa e a ge ed a a mic ocon olle .
•Ge amilia wi h wo king wi h a ious APIs3.
•Design he so wa e a chi ec u e o a new p ojec .
•Apply, in comple ely p ac ical se ings, many o he ools augh in he Compu e Sci-
ence deg ee—mainly in e ac ing di ec ly wi h he Ope a ing Sys em ia sys em calls and
lib a ies—and building he basic in as uc u e o web se e s.
3Applica ion Public In e aces
3
Chap e 2. S a e o he A
To upload code o he ESP32 MCU (mic ocon olle uni ), i needs o be in lashing
s a e, which is de e mined by he GPIO 0 pin: he lashing s a e is ac i e whene e GPIO 0 is
connec ed o he g ound (GND) pin. I his is he case, he code can be uploaded o he MCU
om a compu e h ough a UART10 in e ace. To make his ask easie , he e exis ‘p og amme ’
shields ha a ach o he GPIO pins and allow o easily lash code on o he MCU h ough a
mic o-USB po and on-boa d bu ons o make he GPIO 0 –GND connec ion. Once he code is
lashed on he MCU, his shield also is use ul o powe ing he de ice wi h a USB cable a he
han using he GPIO pins di ec ly. Fo his p ojec I used he ESP32-CAM-MB p og amme
shield, which has a CH340G chip o enable he USB – UART connec ion.
(a) ESP32-CAM mic ocon olle (b) ESP32-CAM-MB p og amme shield
Figu e 2.3: ESP32-CAM and p og amme shield
Once he MCU is in lashing s a e and connec ed o a compu e , he e a e a ious me hods o
ac ually uploading he code, he mos ele an being he Esp essi IoT De elopmen F amewo k
(ESP-IDF) and he A duino IDE11, which is wha will be used in he p ojec . This means ha
mos o he code o he p ojec will be w i en in C++.
The came a ha has been used in his p ojec is he OV2640 model, which a aches di ec ly
o he ESP32-CAM ia a FPC cable (see igu e 2.3a). I has UXGA12 ame size (1600 ×1200
pixels), a ame bu e o 8 MB and a signal- o-noise a io o 40 dB. E en hough i is only
capable o ans e ing UXGA images a a a e o 15 ames pe second, he maximum ame a e
goes up o 30 ps o SVGA (800 ×600 pixels) and o 60 ps o CIF (400 ×296).
The whole sys em occupies a e y small space and is e y economical: a e assembly, he
ESP32-CAM i sel , he p og amme and he OV2640 came a i in a 39 ×27 ×23 mm box and
cos unde e6 when bough o he In e ne .
10Uni e sal Asynch onous Recei e -T ansmi e
11In eg a ed De elopmen En i onmen
12Ul a Ex ended G aphics A ay
10

Tools & echnology 2.2
Lib a ies
Esp essi p o ides some ools o make wo king wi h hei SoCs as simple as possible, om
ne wo king basics o h ead managemen . The o icial lib a ies designed o use wi h he A duino
IDE include all he so wa e ha is needed o de elop applica ions o he ESP32 mic ocon olle s.
They a e hos ed in a Gi Hub eposi o y [14], which con ains as well se e al examples ha
demons a e how o wo k wi h he lib a ies and a .json ile wi h all he necessa y, up- o-da e
in o ma ion abou he a ious suppo ed ESP32 boa ds o be impo ed in he A duino IDE. Mo e
speci ically, he e sion used is elease 2.0.x, which in oduces impo an ixes wi h espec o
he 1.0.x eleases.
Ano he lib a y de eloped by Esp essi is ESP-DL [15] (“a lib a y o high-pe o mance deep
lea ning esou ces dedica ed o ESP32”), which o e s an in e ace o ace de ec ion and ecog-
ni ion h ough i s Model Zoo APIs and will be c ucial o he de elopmen o he p ojec . This
lib a y is included in [14] as a Gi submodule, so ha he A duino IDE manages he ins alla ion
au oma ically.
The main code o he p ojec is somewha based on he ‘Came aWebSe e ’ example [16]
om he same Gi Hub eposi o y, as i showcases a ew o he co e unc ionali ies o he sys em;
in ac , his example shows up equen ly in he p ojec s men ioned in sec ion 2.1.1. To be mo e
speci ic, i o e s he ollowing:
•Taking and displaying s ill images.
•S eaming and displaying li e ideo eed.
•Enabling and disabling ace de ec ion / ecogni ion.
•Regis e ing (“en olling”) new aces o he sys em.
•Displaying he esul s o unning ace de ec ion / ecogni ion on a s ill pic u e o a ame
om he ideo eed.
•Changing he came a se ings (e.g. whi e balance, esolu ion, exposu e, con as , le / igh
and up/down lipping...).
Because o his, he Came aWebSe e example is a good ounda ion on which o build he
sys em, bu i does ha e a ew sho comings. Fi s o all, he use in e ace is con using because
i o e s many op ions a once, mos o which will likely ne e be used by he end use , and some
elemen s o he in e ace sugges a di e en unc ion han he one hey o e ( o ins ance, he
‘En oll Face’ bu on wo ks as a oggle, and he ‘LED In ensi y’ slide does no seem o ha e any
e ec ), see igu e 2.4. The use in e ace should be as simple as possible while main aining he
usabili y o he sys em.
Mo e impo an ly, i is lacking essen ial ea u es o a secu i y sys em, as i only displays wha
is being cap u ed by he came a in eal ime, while he secu i y sys em de eloped in his p ojec
should be able o keep some eco d o ele an e en s (such as a ace de ec ed), send no i ica ions
11
Chap e 2. S a e o he A
Figu e 2.4: Came aWebSe e web in e ace
o use s, o ha e some kind o e ec on he eal wo ld (unlocking a doo , o ins ance). Ano he
missing ea u e is he abili y o access he sys em om ou side he local ne wo k (e.g. o emo e
home su eillance o wo king as a sma doo bell).
Below is a b ie summa y o he lib a ies used in he Came aWebSe e example and which
will be used in he p ojec as well.
In his example, he ace de ec ion is ac ually ca ied ou by he in e me hod in he
HumanFaceDe ec MSR01 and HumanFaceDe ec MNP01 classes. The i s is simple and as e , and
is he e o e used when speed is a p io i y o o ob ain a i s app oxima ion. I p ecision is
p e e ed o e speed, he second can be used on he e u n alue o he i s o ge be e esul s.
Bo h me hods ecei e he inpu image as an unsigned in ege poin e and wo ypes a e suppo ed
o each o hem:
•uin 8_ * is used when he inpu image is encoded using RGB888.
•uin 16_ * is used when he inpu image is encoded using RGB565.
The pic u e aken by he came a is no encoded in any o hese o ma s; o pe o m he
a ious con e sions, he e a e se e al unc ions in he lib a ies (img_con e e s.h).
The ace ecogni ion is pe o med by he FaceRecogni ion112V1S8 class. The e is also a
FaceRecogni ion112V1S16 class a ailable, bu i is no used in he Came aWebSe e example
because i has some issues: i p oduces “ e y la ge i mwa e”, is “ e y slow”, and “ eboo s when
s eaming” [16,app_h pd.cpp:56]. I should be no ed ha ace ecogni ion akes a long ime
(∼15 seconds pe ame) on he ESP32 and ESP32-S2 chips e en using he S8 class and is
disabled by de aul [16,app_h pd.cpp:30] o all chips excep o he ESP32-S3.
The web se e i sel is c ea ed using he esp_h p_se e lib a y, so ha only he ‘handle ’
12
Tools & echnology 2.2
unc ions need o be implemen ed. These a e unc ions ha ake one HTTP eques as inpu (in
he o m o a h pd_ eq_ a iable) and implemen all he necessa y logic o p ocess he eques
and send back he esponse. The mos ele an unc ions om his lib a y a e he ollowing:
•h pd_ esp_send and h pd_ esp_send_chunk o send he esponse message.
•h pd_ esp_send_404,h pd_ esp_send_408 and h pd_ esp_send_500 o eply wi h
he co esponding e o codes (404: No Found, 408: Reques Timeou , 500: In e nal
Se e E o [17]).
•h pd_ eq_ge _u l_que y_s and h pd_ eq_ge _u l_que y_len o ge he que y s ing
om he URL13 and i s leng h.
•h pd_que y_key_ alue o pa se a que y om a URL and ob ain he (key, alue) pai .
•h pd_ esp_se _ ype and h pd_ esp_se _hd o se he esponse ype and heade , e-
spec i ely.
•h pd_ egis e _u i_handle o add a URI14 handle ( ep esen ed by a h pd_u i_
a iable) o he web handle (h pd_handle_ a iable).
•h pd_s a and h pd_s op o s a and s op an HTTP se e , espec i ely.
The unde lying implemen a ion o he TPC/IP p o ocol sui e used in he ESP32 is esp-lwip
[18], a o k o he lwIP15 s ack wi h some pa ches speci ic o he ESP-IDF amewo k. The BSD
Socke s API [19] is cu en ly he only o icially suppo ed API (and he one used in he p ojec ),
bu he Ne conn API is also enabled.
2.2.2 Run- ime ope a ing sys em
The ope a ing sys em used in he ESP32 SoC is ESP-IDF F eeRTOS [20], which is a dual-co e,
SMP16-capable RTOS de eloped in pa by Esp essi . I is buil upon he o iginal F eeRTOS [21]
( e sion 10.4.3), a eal- ime ope a ing sys em designed speci ically o mic ocon olle s. Below
a e ou lined he ea u es o F eeRTOS [22] and he ESP-IDF e sion [20] ha will be mos
ele an in he p ojec .
The main di e ence be ween a eal- ime ope a ing sys em and a non- eal- ime one is in
hei scheduling policies: as eal- ime / embedded sys ems a e designed o espond o eal-wo ld
13Uni o m Resou ce Loca o
14Uni o m Resou ce Iden i ie
15h ps://sa annah.nongnu.o g/p ojec s/lwip/
16Symme ic Mul ip ocessing: whe e a single ins ance o F eeRTOS manages asks ac oss mul iple p ocesso s
13
Chap e 2. S a e o he A
e en s which usually need a as esponse ime, a eal- ime scheduling policy has o ensu e ha
he highes p io i y ask ha is able o un is he ask ha is gi en p ocesso ime (i may be
necessa y o sha e he p ocessing ime “ ai ly” be ween asks o equal p io i y, i he e is mo e
han one eady o un). When he e is no o he ask eady o un, he idle ask — which was
c ea ed by he RTOS i sel — is gi en all he p ocessing ime. I s only unc ions a e o ee he
memo y o asks ha ha e al eady been dele ed and o un he ‘idle ask hook’, so i is impo an
o ensu e ha i is able o un in he case ha asks a e dele ed o he idle ask hook is needed;
howe e , i “can legi ima ely be s a ed o mic ocon olle ime unde all o he condi ions” 17.
In F eeRTOS, he logic o a ask is implemen ed in a oid unc ion ha akes one oid*
a gumen , and he ask i sel is c ea ed ia he xTaskC ea e o xTaskC ea eS a ic unc ions
(which also se he ask p io i y and s ack size) and dele ed ia he TaskDele e unc ion. A
a gi en ime, each ask is in one o ou possible s a es: unning, eady (able o un bu no
cu en ly execu ing because a ask wi h equal o highe p io i y is unning), blocked (wai ing
o an e en , such as a ime expi ing o a esou ce becoming a ailable; no mally includes a
‘ imeou ’ pe iod a e which he ask will be unblocked e en i he e en has no ocu ed ye )
and suspended (simila o ‘blocked’ bu wi hou a imou ; asks can only en e o lea e his
s a e when explici ly eques ed h ough he TaskSuspend and xTaskResume unc ions). In a
single-co e chip, only one ask can be in he ‘ unning’ s a e a a gi en ime. The F eeRTOS
schedule ensu es ha he ‘ unning’ ask is he one wi h he highes p io i y among all he
‘ eady’ asks; i he e a e mo e han one ask wi h he highes p io i y, he schedule uses a
ound- obin policy. I a ask en e s he ‘ eady’ s a e and i s p io i y is highe han ha o he
cu en ly unning ask, he la e is p eemp ed by he schedule o gi e he highe -p io i y ask
p ocesso ime.
Fo in e - ask communica ion, F eeRTOS p o ides he usual p imi i es (semapho es and mu-
exes), as well as queues o sending da a om one ask o ano he and i s own no i ica ions
(which only use one in e nal no i ica ion a ay o each ask and no o he objec ; his makes
hem as e han he o he op ions bu hey also equi e mo e memo y i many di e en no i ic-
a ion ypes a e needed). I also o e s suppo o he POSIX h eads API ( o bo h asks and
synch oniza ion p imi i es).
Lis ed below a e he mos ele an modi ica ions in oduced by Esp essi in o F eeRTOS in
o de o suppo dual-co e SMP.
Tasks can be pinned o a ce ain co e (i is said ha he ask has an a ini y o ha co e) when
hey a e i s c ea ed, using he xTaskC ea ePinnedToCo e o xTaskC ea eS a icPinnedToCo e
unc ions. I a ask is no pinned o any pa icula co e, i is allowed o swi ch co es du ing he
cou se o i s execu ion.
17h ps://www. ee os.o g/RTOS-idle- ask.h ml
14
Tools & echnology 2.2
Fo he mos pa , he wo co es beha e in he same way in ESP-IDF F eeRTOS, ( asks
can be pinned o any o he wo and unpinned asks can un equally on bo h o hem) bu
he e a e some excep ions when i comes o he in e nal wo kings: only CPU0 is esponsible o
inc emen ing he schedule ’s ick coun and o unblocking any ‘blocked’ asks whose ime has
un ou , while bo h co es a e esponsible o checking whe he a con ex swi ch is equi ed. As
only CPU0 is in cha ge o keeping ime, i is impo an no o p e en i om inc emen ing he
ick coun e so as no o dis up he schedule ’s ime.
ESP-IDF also p o ides a Task Wa chdog Time (TWDT) [23] o moni o ing pa icula
asks and making su e ha hey a e no s a ed o p ocesso ime. By de aul , only he idle
asks a e egis e ed o be moni o ed, bu any ask can subsc ibe o being wa ched by using he
esp_ ask_wd _add unc ion. E e y moni o ed ask mus egula ly call he esp_ ask_wd _ ese
unc ion o ese he wa chdog ime ( eed he wa chdog); i he ime is allowed o expi e, an ex-
cep ion is aised wi h message ‘ ask_wd : Task wa chdog go igge ed. The ollowing
asks did no ese he wa chdog in ime:’ indica ing ha he moni o ed ask has no
been gi en p ocessing ime since he ime was las ese .
2.2.3 Teleg am bo API
Teleg am is a popula ee ins an messaging applica ion wi h a la ge se o ea u es and o icial
clien s o he main mobile pla o ms (And oid, iPhone and iPad) and desk op ope a ing sys ems
(Windows, macOS and Linux), as well as b owse -based clien s.
One o he dis inguishing ea u es o Teleg am wi h espec o o he messaging apps is i s
suppo o bo s, special Teleg am accoun s ha use s can in e ac wi h o make que ies o ecei e
no i ica ions di ec ly o hei de ice — any hing om sea ching in Wikipedia18 o c ea ing PDF
documen s using L
A
T
EX19 o looking o public anspo schedules20;“ he possibili ies o bo s a e
endless — om simple sc ip s o complex web apps. Whe he you’ e a beginne o p o essional
p og amme , you can c ea e pe sonalized ools wi h he help o he Bo Pla o m.” [24]. All his is
accesible ia he Teleg am Bo API [25], an HTTP-based in e ace o allow de elope s o build
bo s o Teleg am.
To c ea e a Teleg am bo , he i s s ep is o ob ain a unique oken o i using he Bo Fa he
bo 21: a e choosing a use name, Bo Fa he eplies wi h he oken ha will be used o access
he HTTP API. Once he bo is al eady c ea ed, Bo Fa he can also be used o change he bo ’s
desc ip ion, name o p o ile pho o.
18h ps:// .me/wiki
19h ps:// .me/InLaTeXbo
20h ps:// .me/em madbo
21h ps:// .me/bo a he
15

Chap e 2. S a e o he A
Figu e 2.5: C ea ing a new bo
wi h Bo Fa he
The bo API suppo s all s anda d unc ions o Teleg am
accesible o use s (sending messages, pho os, polls, e c.) bu
i also allows de elope s o c ea e cus om keyboa ds (inline
keyboa ds) o ge quicke inpu om use s wi hou he need
o ill he cha wi h messages, o o de ine a “menu bu on”
wi h in o ma ion abou he suppo ed commands:
•Inline keyboa ds a e implemen ed as se s o “bu ons”
which, when p essed, cause a callback que y o be sen
back o he bo . These que ies can con ain di e en
a ibu es, bu he mos impo an in his p ojec is he
da a ield, which con ains a s ing se p e iously by he
de elope . Teleg am clien s display a p og ess ba a e
a bu on has been p essed, so each callback que y mus
be answe ed o acknowledge ha i has been p ocessed.
•The menu bu on is essen ially jus a lis o he a ail-
able commands wi h a sho desc ip i e message o ex-
plain hei unc ion.
To ac ually implemen he code o he bo he e a e se e al hi d-pa y lib a ies, some o
hem ea u ed in he o icial Teleg am websi e22, o connec wi h he Teleg am API. The chosen
lib a y o his p ojec is Teleg amBo s [26] o Ja a, which is essen ially a one- o-one mapping
om he Teleg am API [25], bu a special men ion mus be made o he Uni e sal-A duino-
Teleg am-Bo lib a y, which is “designed o be used wi h mul iple A duino a chi ec u es” [27],
including ESP boa ds.
In Teleg amBo s, all objec s in he Teleg am Bo API a e implemen ed as Ja a classes. The
mos ele an o he p ojec a e lis ed below:
•Teleg amLongPollingMode is an abs ac class o be ex ended by conc e e bo s, which mus
p o ide hei own okens. The ecei ed messages a e handled by he onUpda eRecei ed
me hod, which ecei es an objec o ype Upda e.
•Upda e objec s may con ain a mos one o se e al di e en pa ame e s depending on he
ype o he upda e. The only ypes ha a e o in e es a e Message and CallbackQue y.
•SendMessage objec s a e cons uc ed wi h he ecipien ’s ID and he message ex . To
ac ually send a message, a SendMessage objec needs o be passed o he execu e me hod.
22h ps://co e. eleg am.o g/bo s/samples
16
Tools & echnology 2.2
•SendPho o is simila ly cons uc ed wi h a use ID and an Inpu S eam which ep esen s
he pic u e o be sen . To send he pic u e, a SendPho o objec needs o be passed o he
execu e me hod.
To pe o m he ac ion ep esen ed by any o hese objec s, he execu e me hod mus be
called.
17
Chap e 3
Sys em a chi ec u e & design
This chap e con ains an in-dep h exposi ion o he ea u es o he sys em and an explana ion
o way he p ojec is o ganized.
3.1 Fea u es
All he ea u es o he designed sys em e ol e a ound he abili y o ake s ill pic u es o li e
ideo a any ime and o pe o m ace de ec ion and ecogni ion on hem. Speci ically, use s
can use he sys em o:
•Take a s ill pic u e a any ime, immedia ely ecei ing he cap u ed image, and au o-
ma ically unning ace de ec ion and ecogni ion on i i hey a e enabled. A box is shown
a ound any de ec ed ace and, i ace ecogni ion was execu ed, a ex is w i en on he
pic u e wi h he ace ID, in he case ha a known ace has been ecognized, o a message
indica ing ha an unknown ace was de ec ed o he wise.
The colo o he box indica es he esul o he acial ecogni ion: i a known ace is de ec ed,
he box is d awn in g een; i an unknown ace is de ec ed, he box is ed; and i only ace
de ec ion was un, bu no ecogni ion, he box is d awn in yellow.
•Regis e new ‘known’ o ‘au ho ized’ aces by enabling he ‘en oll new aces’ op ion and
ei he aking a pic u e o ha ing he ace appea in a ideo ame. When a new ace is
en olled i is s o ed so ha , e en i powe is los , he au ho ized aces a e p ese ed.
This op ion akes p ecedence o e acial ecogni ion, meaning ha i an en olled ace is
p esen in he ame bu he ‘en oll new aces’ op ion is u ned on, he ace will be en olled
again wi h a new ace ID.
19
Chap e 4. Implemen a ion
Then, he came a is ini ialized by se ing he con igu a ion in a came a_con ig_ a iable
and calling he esp_came a_ini () unc ion; i he came a ails o be ini ialized co ec ly, he
unc ion e u ns wi h an e o code and he p og am e mina es. The main se ings ha can be
changed a e he ollowing:
•pixel_ o ma , he o ma in which he pic u es a e aken. I is se o PIXFORMAT_JPEG
because his is he o ma ha is used o sending he pic u es and he ideo ames, which
means ha no o ma con e sion is needed jus o he image ansmission.
• ame_size, which is ini ially se o FRAMESIZE_UXGA, he highes a ailable alue. This is
se he e because i is he esp_came a_ini () unc ion ha is esponsible o alloca ing
he ame bu e s, so hey need o be big enough o hold any possible ame size. Howe e ,
aking pic u es wi h such a high esolu ion is qui e slow ( ecall om 2.2.1 ha he maximum
ame a e he came a can handle wi h he UXGA ame size is 15 ps), so he ame size is
se o a lowe alue jus a e ini ializing he came a: in he lines
senso _ *s=esp_came a_senso _ge ();
s->se _ amesize(s, FRAMESIZE_CIF);
he came a is con igu ed o use he CIF ame size o all new pic u es aken un il he
ame size is changed again.
•jpeg_quali y, ini ially se o 10, which wo ks well o si ua ions wi h good ligh ing.
A e s a ing he came a, he WiFi lib a y is ini ialized using he SSID and passwo d
p o ided in wi i_c eden ials.h and he p og am wai s un il a connec ion is es ablished. Las ly,
he web se e i sel is s a ed by calling he s a _se e ().
In addi ion o se up(), A duino iles also need a loop() unc ion, which is called con inually
once se up() has e u ned; howe e , since all he logic o he sys em is implemen ed in asks
de ined in app_h pd.cpp, his loop unc ion does no need o do any hing. Ins ead o being le
emp y (which would unnecessa ily consume CPU cicles), i only con ains a call o delay(10000)
o sleep o 10 seconds.
Besides se up() and loop(), his ile also con ains he unc ions swi ch_ lash(bool) and
swi ch_led(bool) which espec i ely u n he came a lash and he pilo LED on o o by
se ing he co esponding pins o high o low ol age.
4.1.2 app_h pd.cpp
This ile handles all he image cap u es and he web se e s, which a guably makes i he mos
impo an pa o he whole sys em. The main unc ion he e is s a _se e , which is called
26

ESP32-CAM Se e 4.1
om se up() and is in cha ge o ini ializing he ale s sys em, egis e ing he di e en URIs wi h
he co esponding HTTP handles and loading he p e iously egis e ed aces om a dedica ed
pa i ion.
URI handle s
The URI handle s a e egis e ed h ough he h pd_ egis e _u i_handle unc ion, which
ecei es as a gumen s he HTTP handle (h pd_handle_ ) and a a iable o ype h pd_u i_
ep esen ing he URI; in hese a iables, he ield me hod is always se o HTTP_GET while use _-
c x is always NULL, so only he ields u i and handle a e o in e es he e (see he de ini ion o
he ype h pd_u i_ below om [14, ile esp_h p_se e .h]). No e ha he handle unc ions
e u n an esp_e _ alue indica ing a possible e o ; i his alue is no ESP_OK (meaning ha
any kind o e o has occu ed), he unde lying socke is closed.
ypede s uc h pd_u i {
cons cha *u i; /*!< The URI o handle */
h pd_me hod_ me hod; /*!< Me hod suppo ed by he URI */
/**
* Handle o call o suppo ed eques me hod. This mus
* e u n ESP_OK, o else he unde lying socke will be closed.
*/
esp_e _ (*handle )(h pd_ eq_ * );
/**
* Poin e o use con ex da a which will be a ailable o handle
*/
oid *use _c x;
} h pd_u i_ ;
The s ing u i is one o he URIs discussed in 3.2 (/,/s a us,/con ol,/cap u e o
/s eam), while he handle ield poin s o unc ions ha manage each o hese URIs:
•index_handle only checks ha he came a is a ailable and eplies wi h he main page—
om ile index.h ml—i i is; i he came a senso is no ound, i eplies wi h a 500
(in e nal se e e o ) s a us code.
Fo e iciency easons, index.h ml is sen as a gzip-comp essed ile, which is s o ed in he
code as an a ay o by es. To make i easie o wo k wi h his ile, a Pe l sc ip (h ml_gz.pl)
27
Chap e 4. Implemen a ion
has been c ea ed o pe o m he wo-way con e sions:
–“decomp ession” (CLI op ion -d): ex ac s he ex ual ep esen a ion o he by es
om he inpu ile and w i es hem in o he ou pu ile be o e unning gzip.
–“comp ession” (CLI op ion -c): uns gzip and w i es he ex ual ep esen a ion o
he esul ing by es o a ile.
•s a us_handle sends a JSON esponse con aining he cu en sys em se ings; i is use ul
o upda e he web in e ace a he beginning wi h he ac ual con igu a ion he sys em is
using.
•cmd_handle manages he /con ol URI, which is used as an in e ace o change he
sys em se ings. I expec s ha eques s o /con ol con ain a que y wi h wo key- alue
pai s: ( a , a iable) and ( al, alue). The que y s ing is ex ac ed by a helpe unc ion
called pa se_ge , which is essen ially a w appe a ound he lib a y unc ion h pd_ eq_-
ge _u l_que y_s bu also ese es enough dynamic memo y o con ain he que y s ing
and e u ns an e o code i a que y was no ound in he URI. The que y s ing is hen
pa sed o ob ain he wo alues using he h pd_que y_key_ alue unc ion and he sys em
con igu a ion is se acco dingly. The possible alues o a iable a e:
– lash, o change he s a e o he lash LED.
–de ec , ecognize and en oll, o enable o disable ace de ec ion, ecogni ion o
en ollmen , espec i ely.
–send-ale s, o enable o disable he ‘send ale s’ mode.
–whi elis -enable, o change whe he o use he whi elis o no .
–whi elis _add and whi elis _ emo e, o add o emo e use s o he whi elis , e-
spec i ely. In bo h o hese cases, alue is in e p e ed as a s ing con aining he
use name o be added o emo ed.
– amesize, o se he ame size o he nex pic u es aken. alue should be an
in ege in he ange 0–13, in which case he ame size is se acco ding o able 4.1.
–quali y, o se he JPEG quali y ha will be used o he nex pic u es. alue should
be an in ege .
–pho os, o change whe he o send pho os o subsc ibed use s o no . This is no
expec ed o be changed di ec ly by use s (i is no exposed h ough he web in e ace
no h ough he bo in e ace), bu is used by he bo when use s subsc ibe o unsub-
sc ibe o ecei e pho os: because pho os a e se e al kiloby es in size, hey should no
be sen o he bo when he e a e no use s in e es ed in ecei ing hem.
28
ESP32-CAM Se e 4.1
Value F ame size
0 (96 ×96)
1 QQVGA (160 ×120)
2 QCIF (176 ×144)
3 HQVGA (240 ×176)
4240 ×240
5 QVGA (320 ×240)
6 CIF (400 ×296)
Value F ame size
7 HVGA (480 ×320)
8 VGA (640 ×480)
9 SVGA (800 ×600)
10 XGA (1024 ×768)
11 HD (1280 ×720)
12 SXGA (1280 ×1024)
13 UXGA (1600 ×1200)
Figu e 4.1: In e nal ep esen a ion o ame sizes [14, ile senso .h]
–cooldown, o enable o disable he cooldown ime be ween successi e ace de ec ions
when he ideo is ac i e.
–cooldown- ime, o se ing he leng h o he cooldown ime . In his case, alue is an
in ege ep esen ing he numbe o second o he ime .
– unning, o u n o he ideo. This is only use ul when alue is 0 ( alse) and he ideo
is ac ually unning; i his is no he case, he ac ion has no e ec (see s eam_handle
below).
Unless s a ed o he wise, all a iables ecei e an in ege alue ha is in e p e ed as a
boolean.
When a iable is no one o he men ioned s ings, o he e is any e o when changing he
came a se ings, he handle eplies wi h a in e nal se e e o s a us code; o he wise,
i sends an emp y esponse and exi s.
I was p e iously no ed ha some o he se ings a e dependen on one ano he , so some
ca e mus be aken when ac ually changing hem. The unc ion en o ces he ollowing
es ic ions:
–When ace de ec ion is disabled, so a e acial ecogni ion and en olling.
–When acial ecogni ion is enabled, so is ace de ec ion. When i is disabled, so is
en olling.
–When en olling is enabled, so a e ace de ec ion and ecogni ion.
These h ee ac ions se e o p ese e he ollowing in a ian in he sys em con igu a-
ion:
en olling ON ⇒ ecogni ion ON ⇒de ec ion ON
29
Chap e 4. Implemen a ion
–By design, ace de ec ion is ne e un on images wi h a highe ame size han CIF
because i would ake longe o p ocess he ame and because he esul ing image
would be oo la ge o send o he bo . Because o his, he ame size is ne e se
highe han CIF i ace de ec ion is on, and ace de ec ion is no enabled i he ame
size is oo high.
•cap u e_handle akes a pho o and pe o ms all he necessa y ac ions depending on he
sys em con igu a ion.
In he i s place, a JPG pic u e is aken ia he esp_came a_ b_ge unc ion; i he
unc ion ails o ge an image, he handle sends an in e nal se e e o esponse and
e u ns wi h an e o code.
The e is no need o do any addi ional p ocessing o he pic u e i ace de ec ion is disabled:
in his case, he pho o is sen in he esponse message and he unc ion e u ns. O he wise,
a leas ace de ec ion needs o be pe o med, which needs he pic u e o be encoded
using one o PIXFORMAT_RGB888 o PIXFORMAT_RGB565; only he o me is used in he
sys em. The JPG pic u e is con e ed o RGB888 using he m 2 gb888 unc ion and hen
de ec ion is un on he esul : i a ace is de ec ed, ale s a e sen and ecogni ion/en olling
is pe o med depending on he se ings; o he wise, he pho o is sen and he unc ion
e u ns immedia ely. The s eps aken a e he de ec ion has ended a e summed up in he
pseudo-code below:
i (de ec ed) {
i (send_ale s)
no i y ha a ace has been de ec ed;
i (is_en olling)
en oll;
i ( ecogni ion_enabled) {
un acial ecogni ion and ge ace ID;
i (send_ale s)
no i y ha ace wi h ID has been ecognized;
}
d aw ace boxes;
con e back o JPG;
30
ESP32-CAM Se e 4.1
espond o he eques wi h upda ed pho o;
i (send_ale s && send_pho os)
send he pho o o he bo o b oadcas o he subsc ibed use s;
}else {
espond o he eques wi h pho o;
}
I any e o occu s when pe o ming he image con e sions o alloca ing dynamic memo y,
he handle esponds wi h an in e nal se e e o s a us code and e u ns an e o
code indica ing ailu e. Also, an in o ma ion message is p in ed o he se ial moni o wi h
he size o he inal image and he ecognized ace ID, when applicable.
•s eam_handle manages he li e ideo, whe he i is ac ually s eamed o no . I s imple-
men a ion is e y simila o ha o cap u e_handle bu w apped inside a while( unning)
loop. I only b eaks ou o he loop when unning is se o alse ( ia con ol_handle ) o
when he e is an e o pe o ming he image con e sions o memo y alloca ions. Howe e ,
he e a e some di e ences wi h espec wi h cap u e_handle :
–The ames a e only sen back o he clien when nei he de ec ion no ecogni ion
is un, i.e. when ace en olling is disabled and he ale s a e u ned o . This means
ha , e en hough egis e ing new aces is possible in li e ideo, use s will no be able
o see he image ha was used o he en ollmen .
–I ace de ec ion is un and he ale cooldown ime is enabled, he sys em sleeps o
ale _cooldown_ ime seconds a e he ame has been p ocessed. As he TaskDelay()
unc ion akes i s inpu in numbe o icks [20], he cooldown ime in seconds needs
o be con e ed o icks di iding by he cons an po TICK_RATE_MS;
Each o hese unc ions is se as he handle o he co esponding URI (index_u i,s a us_-
u i,cmd_u i,cap u e_u i and s eam_u i). O hese, he i s ou a e egis e ed wi h he
‘main’ se e , ep esen ed by he came a_h pd handle, while he las is egis e ed wi h a sepa a e
handle o he s eam, s eam_h pd.
Helpe unc ions
The sys em elies on he ollowing auxilia y unc ions:
• gb_p in and gb_p in a e used o w i e messages in colo on he cap u es images.
They ely on he b_g x_p in lib a y unc ion.
31

Chap e 4. Implemen a ion
•d aw_ ace_boxes d aws he colo -coded boxes a ound each de ec ed ace using he b_-
g x_d awFas HLine and b_g x_d awFas VLine lib a y unc ions.
•en oll_ aces uses he en oll_id me hod om he FaceRecogni ion112V1S8 class [15]
o egis e new aces in o he sys em—s o ing hem in o lash memo y—p o ided ha he
numbe o egis e ed aces is less han he cons an FACE_ID_SAVE_NUMBER (which is se
o 7). To signal ha he sys em is busy en olling a new ace, he pilo LED is u ned on
be o e he call o en oll_id (by calling swi ch_led( ue)) and i is u ned o again a e
i has inished (swi ch_led( alse)).
• un_ ace_ ecogni ion has a simila s uc u e o en oll_ aces, using he ecognize
me hod ins ead. I he ace ID is non-nega i e, meaning ha he ace has been ecognized
as a ‘known’ ace, a message is w i en in g een on he pic u e ia gb_p in indica ing
bo h he ID and he simila i y o he ‘o iginal’ pic u e; i i is nega i e, meaning ha i
has no been ecognized as a known ace, he message “In ude Ale !” is w i en in ed.
Two-s age de ec ion
The sys em implemen s he ace de ec ion in wo s ages bacause i gi es much be e esul s o
use wi h acial ecogni ion ( ecall om 2.2.1 ha he e a e wo classes a ailable o pe o m ace
de ec ion). This is done by i s unning he in e me hod in he HumanFaceDe ec MSR01 class
o ob ain a lis o candida es, and hen using hese candida es o ob ain he inal esul s wi h
HumanFaceDe ec MNP01:
s d::lis <dl::de ec :: esul _ > &candida es =
s1.in e ((uin 8_ *)ou _bu , {(in )ou _heigh , (in )ou _wid h, 3});
s d::lis <dl::de ec :: esul _ > & esul s =
s2.in e ((uin 8_ *)ou _bu , {(in )ou _heigh , (in )ou _wid h, 3}, candida es);
Howe e , i is s ill possible o disable wo-s age ace de ec ion du ing compile ime by se ing
he TWO_STAGE p ep ocesso mac o o 0, o a as e bu less p ecise ace de ec ion.
4.1.3 bo _api.cpp
This ile con ains he necessa y unc ions o send in o ma ion o he Teleg am bo using a BSD
socke [19]. The socke desc ip o and s uc sockadd a e s o ed as global a iables inside
he ile, and a e ini ialized by he ini _ale s unc ion using he add ess and po de ined in
bo _con ig.h. The implemen ed unc ions a e:
32
Teleg am bo 4.2
•send_ale (cons cha * msg, size_ len), a gene ic unc ion o send a message o he
bo which is used by he es o he unc ions bu is no exposed ou side o he ile.
•whi elis _enable(bool b), o ell he bo whe he o no i should espec he whi elis .
•whi elis _add(cons cha * use name), o add a use o he whi elis .
•whi elis _ emo e(cons cha * use name), o emo e a use om he whi elis .
• ace_ ecognized(in ace_id), o send a ex no i ica ion abou a ecognized ace, in-
cluding i s ID.
• ace_de ec ed(), o send a ex no i ica ion abou a de ec ed ace.
• ace_ ecognized_pho o(cons cha * pho o, size_ len), o send a pho o wi h a e-
cognized ace.
• ace_de ec ed_pho o(cons cha * pho o, size_ len), o send a pho o wi h a de ec-
ed ace.
These unc ions a e explained in mo e de ail in subsec ion 4.2.2.
4.2 Teleg am bo
The Teleg am bo has been implemen ed in Ja a using he Teleg amBo s lib a y [26]. The en y
poin is in Main.ja a, whose only me hod main is in cha ge o c ea ing a new objec o he
class Bo —which will be esponsible o in e ac ing wi h he use and he Teleg am se e s—and
s a ing he UDP se e —which will ecei e he messages om he ESP32-CAM.
4.2.1 Bo .ja a
This ile de ines he Bo class, he mos impo an piece in his pa o he p ojec . To be able o
connec au oma ically wi h he Teleg am API, i needs o ex end ei he o he abs ac classes
Teleg amLongPollingBo o Teleg amWebhookBo —in his case, he o me has been chosen.
The class Bo has se en p i a e a ibu es:
•HashSe <S ing>whi elis , a se o s ings con aining he names o au ho ized use s.
•HashMap<S ing, HashSe <Long>> use s, a map con aining he in o ma ion abou sub-
sc ibed use s as
subsc ip ionLis 7→ use s subsc ibed o subsc ip ionLis
33
Chap e 4. Implemen a ion
•boolean whi elis Enabled = ue.
•boolean addingUse ToWhi elis = alse.
•boolean emo ingUse F omWhi elis = alse.
•boolean se ingCooldownTime = alse.
•boolean se ingQuali y = alse.
The e is only one cons uc o , which ecei es he bo oken gi en as a s ing and does he
ollowing:
•Call supe (bo Token);
•Ini ialize whi elis wi h he de aul whi elis ed use s (as de ined in Bo Con ig).
•Ini ialize use s by c ea ing new emp y subsc ip ion lis s a keys de ec ed, egis e ed,
in ude and pho os.
•Se he lis o suppo ed commands o p o ide au o-comple ion and a menu bu on inside
he cha .
The impo an inhe i ed me hods om Teleg amLongPollingBo ha need o be o e iden
a e ge Bo Use name and onUpda eRecei ed—ge Bo Use name only e u ns he bo use name
ha is de ined in Bo Con ig.ja a, while onUpda eRecei ed— he me hod ha implemen s mos
o he logic—is explained nex .
onUpda eRecei ed me hod
This me hod ecei es a single objec o he class Upda e; e en hough Teleg am upda es may be
o se e al di e en ypes, he implemen ed bo only handles Message o a CallbackQue y (which
a e checked by he hasMessage() and hasCallbackQue y() me hods):
•Message upda es: hese a e ecei ed when he bo ecei es a message om a use , mos
o en in he o m o a command; in his case, he message ex , he use name and he use
ID a e ex ac ed om he message and a SendMessage objec is cons uc ed o espond o
he use .
In he i s place, he me hod checks whe he he use is au ho ized o send a message
(ei he he whi elis is disabled, o he use is in he whi elis ). I his is no he case,
he esponse ex in o ms he use ha he o she is unau ho ized and exi s; o he wise, he
eques is p ocessed.
34
Teleg am bo 4.2
The commands a e handled by compa ing he message ex wi h he command names
(/s a ,/help,/subsc ibe,/unsubsc ibe,/ge _pho o,/con ig,/whi elis _add o
/whi elis _ emo e). In mos cases, o eply o he use i is only necessa y o se he
esponse ex and some imes a “ eply ma kup” as well (in he o m o an inline keyboa d);
his is done by calling he se Tex and se ReplyMa kup me hods in he SendMessage ob-
jec wi h he app opia e s ings o keyboa ds (see subsec ion 4.2.3), which a e cons uc ed
in he Cus omS ings.ja a and Cus omKeyboa ds.ja a iles.
The commands ha a e handled di e en ly a e hese:
–/ge _pho o, which needs o con ac he ESP32-CAM se e o ge a pho o and o -
wa d i o he use . This is done by c ea ing a new Inpu S eam om he /cap u e
URI and using i o c ea e a SendPho o objec .
–/whi elis _add, which in addi ion o se ing he esponse ex and eply ma kup,
also se s he addingUse ToWhi elis a ibu e o ue.
–/whi elis _ emo e, analogous o /whi elis _add bu se ing o ue he a ibu e
emo ingUse F omWhi elis .
Be o e checking whe he he message ecei ed is a command, he bo checks i ei he o he
a iables addingUse ToWhi elis o emo ingUse F omWhi elis is se o ue om a
p e ious /whi elis _add o /whi elis _ emo e command; i his is he case, he message
ex is in e p e ed as a use name o be added o emo ed o he whi elis : i is i s checked
ha he message ex is indeed a alid use name and, i i is, i is added o o emo ed
om whi elis and he co esponding boolean a iable is ese o alse. I he ex is
no a alid use name, a message is sen o he use in o ming abou he e o .
Nex , he a ibu es se ingCooldownTime and se ingQuali y a e checked. In hese
cases, he ex con ains he in ege alue ha mus be sen o he ESP32 se e ia he
/con ol URI. Because he s ing may no con ain a alid in ege , he bo needs o be
p epa ed o deal wi h an in e nal se e e o , which is done by a ca ch block ha
handles an IOExcep ion.
I a message is ecei ed when all o hese booleans a e alse and he ex does no ma ch
any o he suppo ed commands, he esponse ex in o ms he use ha he ex is no a
alid command and exi s.
•CallbackQue y upda es: hese a e ecei ed when he use clicks on a bu on in an inline
keyboa d; in his case, he que y da a is ex ac ed and handled acco ding o hese ules:
35
Chap e 4. Implemen a ion
wo ked co ec ly wi h images hos ed on o he se e s, bu i always ailed when ying o send
a pho o om he se e i sel : using an add ess such as h p://<local_add ess>/cap u e o
h p://127.0.0.1/cap u e i e u ned he ollowing JSON:
{"ok": alse,
"e o _code": 400,
"desc ip ion": "Bad Reques : w ong ile iden i ie /HTTP URL speci ied"}
Then, I ied o implemen a unc ion o send he cap u ed ame di ec ly by w i ing he
by es o a TCP clien ; his way, he e we e e o s in he memo y alloca ion:
Connec o api. eleg am.o g
[ 6055][E][ssl_clien .cpp:37] _handle_e o (): [s a _ssl_clien ():264]:
(-32512) SSL - Memo y alloca ion ailed
[ 6056][E][WiFiClien Secu e.cpp:144] connec (): s a _ssl_clien : -32512
Connec ed o api. eleg am.o g ailed.
Connec ed o api. eleg am.o g ailed.
[ 6243][E][ssl_clien .cpp:37] _handle_e o (): [s a _ssl_clien ():273]:
(-10368) X509 - Alloca ion o memo y ailed
[ 6244][E][WiFiClien Secu e.cpp:144] connec (): s a _ssl_clien : -10368
These e o s we e ixed when se ing he s ack size o he ask e en highe (a ound 24 KiB
we e needed), bu he p oblem s ill was no ixed because hen he se e ailed o es ablish a
s able connec ion: jus ha ing he bo on (e en i i was idle), he ideo li es eam s opped
andomly o ailed o s a in he i s place,
[ 31693][E][app_h pd.cpp:726] s eam_handle (): Send ame ailed
[ 41196][E][ssl_clien .cpp:129] s a _ssl_clien (): socke e o on d 54,
e no: 113, "So wa e caused connec ion abo "
[ 41197][E][WiFiClien Secu e.cpp:144] connec (): s a _ssl_clien : -1
In his case, I also ailed o ind any mo e in o ma ion abou how o sol e he p oblem, so I
decided o implemen he bo sepa a ely.
42

Chap e 5
Conclusions
This inal chap e p esen s he conclusions ha ha e been eached a e he implemen a ion o
he p oposed sys em, including he esul s ha ha e been ob ained in ela ion wi h he o iginal
p ojec goals as well as he limi a ions in he inal sys em.
5.1 Resul s
The secu i y sys em ha has been implemen ed du ing he cou se o his p ojec mee s all o
he goals s a ed in sec ion 1.2:
•I pe o ms acial ecogni ion on ames aken by he came a—ei he as s a ic pic u es o
as ideo ames—and is able o ell egis e ed use s apa om in ude s, which enables
he sys em o pe o m di e en ac ions depending on he ou come o he ecogni ion.
•I allows he use o easily egis e new aces as “known” aces.
•I o e s a web in e ace o access he came a eed and change he sys em con igu a ion.
•I is capable o sending ale s o any au ho ized use ha has subsc ibed o any subse o
he a ailable no i ica ion lis s.
•I allows au ho ized use s o change he sys em con igu a ion emo ely h ough a Teleg am
bo .
Mo eo e , he sys em o e s simila unc ionali ies o hose o many comme cial p oduc s
(see subsec ion 2.1.2) wi h wo main ad an ages: i does i a a ac ion o he cos , as all o he
needed ha dwa e can be bough o less han e6 om he In e ne , and i gi es use s ull access
o he implemen a ion, which has h ee impo an consequences:
43
Chap e 5. Conclusions
a) Use s a e comple ely in con ol o hei da a and hey can be ce ain ha hei pic u es
a e no sen o he se e s o a company hey do no us : he only ex e nal se ice used
is Teleg am, and i can be easily swi ched o o subs i u ed o o he ways o esponding
o ecognized aces.
b) Use s a e able o implemen new ea u es o disable hose hey do no wan .
c) The sys em is no dependen on a hi d pa y in a subs an ial way, so i is unlikely ha
he sys em will s op wo king om an ou side cause.
5.2 Limi a ions
The implemen ed sys em can be used in p ac ice in some si ua ions, bu i has some limi a ions
ha a e impo an o keep in mind:
1. Facial ecogni ion is e y slow on his combina ion o ha dwa e and ecogni ion so wa e
(i akes a ound 15 seconds). Apa om being po en ially us a ing o use, i poses
a secu i y p oblem as i is ela i ely easy o pe o m a Denial o Se ice (DoS) a ack
on he sys em: by p esen ing any ace o he came a when acial ecogni ion is enabled,
a acke s can ge a 15 second pe iod du ing which he sys em is essen ially ozen and is
no cap u ing any new ames. Fo his eason, he sys em could no be eliably used as a
secu i y came a wi h acial ecogni ion enabled.
This could be ixed by upg ading he ha dwa e o a ESP32-S3 mic ocon olle ( ecall om
2.2.1 ha acial ecogni ion is as e on his chip).
2. The sys em does no o e a way o see he li e ideo om ou side o he local ne wo k.
3. The web in e ace is po en ially con using o use i he se ings a e changed om a di e en
de ice, as he HTML page is no upda ed dynamically when he se e con igu a ion is
changed.
5.3 Fu he imp o emen s
Below a e lis ed some ea u es ha ha e no been implemen ed in he inal sys em, bu could be
use ul o ha e:
•Make he ideo eed a ailable om ou side he local ne wo k. Ideally, his would need
a mo e sophis ica ed solu ion han po o wa ding o secu i y easons—only au ho ized
use s should be able o wa ch he s eam. Fo his, ei he an au hen ica ion mechanism
44
Fu he imp o emen s 5.3
should be implemen ed, o he sys em should ely on ex e nal se ices o p o ide he
secu i y laye .
•Regis e new aces in o he sys em emo ely by sending a pho o o he Teleg am bo
and unning he en olling wi h ha pho o. This would need a mo e e sa ile o m o
communica ion om he bo o he se e han wha is cu en ly implemen ed o be able
o ansmi he pic u e om he bo o he ESP32 mic ocon olle .
•Dele e egis e ed aces om he sys em. As no ed in 4.1, all new en olled aces a e s o ed
pe manen ly in lash memo y, and he e is no cu en ly an easy way o emo e hem. The
lib a y p o ides a unc ion o dele e a ace by ID,
in dele e_id(in id, bool upda e_ lash = alse);
bu his ea u e has no been implemen ed in he sys em because i is no conside ed
pa icula ly use ul.
45
Bibliog aphy
[1] Sean Hollis e . Read Google’s new Nes p i acy p omise and ell me i you’ e swayed.u l:
h ps://www. he e ge.com/2019/5/7/18536184/google-nes -p i acy-commi men
( isi ed on 13/03/2023).
[2] Isabella S ege . How Amazon ou age le sma homes no so sma a e all.u l:h ps:
//www.me cu ynews.com/2021/12/08/how-amazon-ou age-le -sma -homes-no -
so-sma -a e -all ( isi ed on 13/03/2023).
[3] Rachel Me z. How AI is helping spo wild i es as e .u l:h ps://edi ion.cnn.com/
2019/12/05/ ech/ai-wild i es/index.h ml ( isi ed on 13/03/2023).
[4] Nadia Jabe . Can A i icial In elligence Help See Cance in New, and Be e , Ways? u l:
h ps://www.cance .go /news-e en s/cance -cu en s-blog/2022/a i icial-
in elligence-cance -imaging ( isi ed on 13/03/2023).
[5] Isabelle Qian e al. Fou Takeaways F om a Times In es iga ion In o China’s Expanding
Su eillance S a e.u l:h ps://www.ny imes.com/2022/06/21/wo ld/asia/china-
su eillance-in es iga ion.h ml ( isi ed on 14/03/2023).
[6] Ta e Ryan-Mosley. The new lawsui ha shows acial ecogni ion is o icially a ci il igh s
issue.u l:h ps : / / www . echnology e iew . com / 2021 / 04 / 14 / 1022676 / obe -
williams- acial- ecogni ion-lawsui -aclu-de oi -police/ ( isi ed on 14/03/2023).
[7] M. Schwa z. Wi eless Secu i y Came a wi h he A duino Yún.u l:h ps://lea n.
ada ui .com/wi eless-secu i y-came a-a duino-yun/ ( isi ed on 20/03/2023).
[8] Tom Na di. DIY ESP32 Video Doo bell Locks Ou Big B o he .u l:h ps://hackaday.
com/2020/12/22/diy-esp32- ideo-doo bell-locks-ou -big-b o he / ( isi ed on
20/03/2023).
[9] Ashish Choudha y. ESP32-CAM Face Recogni ion Doo Lock Sys em.u l:h ps : / /
ci cui diges . com / mic ocon olle - p ojec s / esp32 - cam - ace - ecogni ion -
doo -lock-sys em ( isi ed on 20/03/2023).
47

BIBLIOGRAPHY
[10] Google. Nes Cam (Ba e y) - Google S o e.u l:h ps : / / s o e . google . com / us /
p oduc /nes _cam_ba e y ( isi ed on 20/03/2023).
[11] Chloe Ma hews. Bes Sma Doo bells o 2022: Home Secu i y o Any Budge .u l:
h ps://www.gea hung y.com/bes -sma -doo bells/ ( isi ed on 20/03/2023).
[12] Jenni e Oksien. The bes ideo doo bell o 2023: i you like i should you pu a Ring
on i ? u l:h ps://www. ech ada .com/bes /bes - ideo- doo bells ( isi ed on
20/03/2023).
[13] Ai Thinke Technology. ESP32-CAM came a de elopmen boa d.u l:h ps://docs.ai-
hinke .com/en/esp32-cam ( isi ed on 16/04/2023).
[14] Esp essi Sys ems. A duino co e o he ESP32.u l:h ps://gi hub.com/esp essi /
a duino-esp32/ ( isi ed on 22/12/2022).
[15] Esp essi Sys ems. Esp essi deep-lea ning lib a y o AIoT applica ions.u l:h ps://
gi hub.com/esp essi /esp-dl ( isi ed on 17/04/2023).
[16] Esp essi Sys ems. Came aWebSe e Gi Hub eposi o y.u l:h ps : / / gi hub . com /
esp essi /a duino-esp32/ ee/23 653ad8d64e95231166e133845228b3db17e9d/lib a ies/
ESP32/examples/Came a/Came aWebSe e ( isi ed on 22/12/2022).
[17] In e ne Enginee ing Task Fo ce. RFC 9110 - HTTP Seman ics §15. S a us Codes.u l:
h ps://h pwg.o g/specs/ c9110.h ml#s a us.codes ( isi ed on 06/05/2023).
[18] Esp essi Sys ems. Fo k o lwIP wi h ESP-IDF speci ic pa ches.u l:h ps://gi hub.
com/esp essi /esp-lwip ( isi ed on 19/04/2023).
[19] The F eeBSD P ojec . Chap e 7. Socke s | F eeBSD Documen a ion Po al.u l:h ps://
docs. eebsd.o g/en/books/de elope s-handbook/socke s/ ( isi ed on 19/04/2023).
[20] Esp essi Sys ems. F eeRTOS (ESP-IDF) - ESP32.u l:h ps://docs.esp essi .com/
p ojec s/esp- id /en/la es /esp32/api- e e ence/sys em/ ee os_id .h ml
( isi ed on 19/04/2023).
[21] F eeRTOS. Ma ke leading RTOS (Real Time Ope a ing Sys em) o embedded sys ems wi h
In e ne o Things ex ensions.u l:h ps://www. ee os.o g/ ( isi ed on 19/04/2023).
[22] F eeRTOS. F eeRTOS Fea u es - F eeRTOS.u l:h ps://www. ee os.o g/ ea u es.
h ml ( isi ed on 19/04/2023).
[23] Esp essi Sys ems. Wa chdogs - ESP32.u l:h ps://docs.esp essi .com/p ojec s/
esp-id /en/la es /esp32/api- e e ence/sys em/wd s.h ml# ask-wa chdog- ime -
wd ( isi ed on 21/04/2023).
48
BIBLIOGRAPHY
[24] Teleg am Messenge Inc. Bo s: An in oduc ion o de elope s.u l:h ps : / / co e .
eleg am.o g/bo s ( isi ed on 21/04/2023).
[25] Teleg am Messenge Inc. Teleg am Bo API.u l:h ps://co e. eleg am.o g/bo s/api
( isi ed on 21/04/2023).
[26] Rubén Be múdez. Teleg am Bo Ja a Lib a y.u l:h ps://gi hub.com/ ubenlagus/
Teleg amBo s ( isi ed on 21/04/2023).
[27] B ian Lough. Uni e sal Teleg am Bo Lib a y.u l:h ps://gi hub.com/wi nessmenow/
Uni e sal-A duino-Teleg am-Bo ( isi ed on 21/04/2023).
[28] Pablo Cas ellanos. ESP32-CAM Secu i y.u l:h ps://gi lab.com/pacas ega/esp32-
cam-secu i y ( isi ed on 07/05/2023).
[29] Pablo Cas ellanos. ESP32-CAM Secu i y Bo .u l:h ps://gi lab.com/pacas ega/
esp32-cam-secu i y-bo ( isi ed on 07/05/2023).
49