Real-Time Big Data Processing with Edge Computing
Abstract
The surge of Internet of Things (IoT) devices and the rapid increase in data generation have required improvements in big data processing techniques. Conventional cloud-based systems, although resilient, frequently face issues concerning latency, bandwidth limitations, and real-time processing. Edge computing represents a revolutionary model that positions computation and data storage in proximity to the data source. By leveraging edge computing's proximity to data sources, this study aims to reduce latency, enhance data processing speeds, and improve overall system efficiency. The findings highlight the potential of edge-based architectures in addressing the challenges of traditional cloud-based models, particularly in time-sensitive applications. This paper examines strategies for enhancing latency and performance in edge-based big data architectures. We illustrate how edge computing can transform real-time data processing through theoretical insights and statistical references.
Full text
A ailable online www.ejae .com
Eu opean Jou nal o Ad ances in Enginee ing and Technology, 2021, 8(11):152-155
Resea ch A icle
ISSN: 2394 - 658X
152
Real-Time Big Da a P ocessing wi h Edge Compu ing
Rajesh Kuma Kanji
Independen Resea che ,
Plano, TX, USA-75024
_____________________________________________________________________________________________
ABSTRACT
The su ge o In e ne o Things (IoT) de ices and he apid inc ease in da a gene a ion ha e equi ed
imp o emen s in big da a p ocessing echniques. Con en ional cloud-based sys ems, al hough esilien ,
equen ly ace issues conce ning la ency, bandwid h limi a ions, and eal- ime p ocessing. Edge compu ing
ep esen s a e olu iona y model ha posi ions compu a ion and da a s o age in p oximi y o he da a sou ce. By
le e aging edge compu ing's p oximi y o da a sou ces, his s udy aims o educe la ency, enhance da a
p ocessing speeds, and imp o e o e all sys em e iciency. The indings highligh he po en ial o edge-based
a chi ec u es in add essing he challenges o adi ional cloud-based models, pa icula ly in ime-sensi i e
applica ions. This pape examines s a egies o enhancing la ency and pe o mance in edge-based big da a
a chi ec u es. We illus a e how edge compu ing can ans o m eal- ime da a p ocessing h ough heo e ical
insigh s and s a is ical e e ences.
Keywo ds: Big da a analysis, Edge compu ing, In e ne o Things (IoT)
____________________________________________________________________________________
INTRODUCTION
The exponen ial inc ease in da a gene a ed by In e ne o Things (IoT) de ices and o he sou ces has necessi a ed
no el app oaches o da a p ocessing. T adi ional cloud compu ing models, while e ec i e, equen ly expe ience
la ency p oblems due o he dis ance be ween da a sou ces and cen alized se e s. Edge compu ing o e s a
p omising solu ion by b inging compu a ion close o he da a sou ce, he eby educing la ency and imp o ing
esponse imes [1]. Edge compu ing, which p ocesses da a a o nea he sou ce, add esses hese challenges by
educing he need o da a ansmission o cen alized se e s. By le e aging edge de ices and nodes, o ganiza ions
can achie e as e esponse imes and enhanced pe o mance. This pape del es in o he p inciples o edge
compu ing in big da a p ocessing, wi h a ocus on op imizing la ency and pe o mance [2].
Figu e 1: Edge Compu ing In as uc u e
Edge compu ing decen alizes da a p ocessing by placing compu a ional esou ces a he "edge" o he ne wo k,
close o whe e da a is gene a ed. This app oach no only minimizes la ency bu also educes he amoun o da a ha
needs o be ansmi ed o cen alized da a cen e s, he eby lowe ing bandwid h usage and associa ed cos s.
Kanji RK Eu o. J. Ad . Engg. Tech., 2021, 8(11):152-155
153
Mo eo e , edge compu ing enhances da a secu i y and p i acy by allowing sensi i e in o ma ion o be p ocessed
locally a he han being sen o ex e nal se e s [3]. These ad an ages make edge compu ing an a ac i e op ion
o a ious applica ions, including au onomous ehicles, sma ci ies, heal hca e moni o ing sys ems, and indus ial
au oma ion [4].
The in eg a ion o a i icial in elligence (AI) wi h edge compu ing u he ampli ies i s po en ial. AI algo i hms can
be deployed a he edge o pe o m eal- ime analy ics, p edic i e main enance, anomaly de ec ion, and o he
ad anced asks. This combina ion o edge compu ing and AI enables mo e in elligen and esponsi e sys ems,
capable o making decisions and aking ac ions wi hou elying on dis an cloud se e s [5]. As a esul , edge-based
a chi ec u es a e inc easingly being adop ed o mee he demands o mode n, da a-in ensi e applica ions.
Despi e i s po en ial, he adop ion o edge compu ing in big da a p ocessing is no wi hou challenges. Issues such as
esou ce managemen , in e ope abili y, and scalabili y need o be add essed o ully ealize he bene i s o edge
compu ing. Addi ionally, he in eg a ion o AI a he edge equi es obus ha dwa e and so wa e solu ions o ensu e
e icien and eliable pe o mance. This esea ch pape aims o explo e hese aspec s and p o ide insigh s in o
op imizing la ency and pe o mance in edge-based big da a a chi ec u es.
LITERATURE REVIEW
A s udy by Fu and Lee (2019) examined he ole o edge compu ing in enhancing eal- ime da a p ocessing o IoT
applica ions. They highligh ed how edge compu ing add esses la ency and bandwid h challenges by p ocessing da a
close o he sou ce, he eby imp o ing e iciency and enabling eal- ime decision-making. This app oach is
pa icula ly bene icial in applica ions equi ing imely esponses, such as heal hca e moni o ing and indus ial
au oma ion [6]. A comp ehensi e su ey by Zhou e al. (2019) explo ed he in eg a ion o edge compu ing wi h
a i icial in elligence, highligh ing how Edge AI can expedi e da a p ocessing and acili a e eal- ime in e ence [7].
Mo eo e , Anaya, Acos a-Be mejo, and Salinas-Rosales (2018) discussed he in eg a ion o edge, IoT, and cloud
compu ing in a dis ibu ed en i onmen , highligh ing he ad an ages o p ocessing da a locally o alle ia e da a
ans e challenges and enhance secu i y [8]. Simila ly, Langona e al. (2016) examined he ole o og compu ing in
da a analy ics and cloud dis ibu ed p ocessing a he ne wo k edges, u he suppo ing he bene i s o edge
compu ing in eal- ime da a p ocessing [9].
These s udies collec i ely unde sco e he po en ial o edge compu ing in op imizing la ency and pe o mance in big
da a a chi ec u es. By p ocessing da a close o he sou ce, edge compu ing minimizes delays and enhances he
e iciency o eal- ime applica ions, making i a iable solu ion o indus ies equi ing immedia e da a analysis,
such as au onomous ehicles and heal hca e moni o ing sys ems.
METHODOLOGY
Algo i hm: Big Da a P ocessing a Edge Nodes in a Cloud Compu ing Sys em
Inpu :
• Da a S eam (D): Con inuous inpu da a om IoT de ices o senso s.
• P ocessing Tasks (T): A se o da a p ocessing asks (e.g., il e ing, agg ega ion, and analy ics).
• Edge Nodes (EN): A se o edge compu ing nodes a ailable o p ocessing.
• Resou ce Limi s (R): Compu a ional and memo y limi s o each edge node.
Ou pu :
P ocessed Da a (P): Agg ega ed and analyzed da a eady o cloud s o age o eal- ime decision-making.
Inpu : Da a S eam D, P ocessing Tasks T, Edge Nodes EN, Resou ce Limi s R
Ou pu : P ocessed Da a P
Begin
// Phase 1: Ini ializa ion
Ini ialize Node Regis y NR wi h me ada a o edge nodes EN
De ine p ocessing asks T wi h p io i y le els and esou ce equi emen s
// Phase 2: Da a Inges ion and Pa i ioning
While Da a S eam D is ac i e do
Pa i ion da a D in o chunks d_i based on size o ype
// Phase 3: Resou ce Alloca ion
Fo each da a chunk d_i do
Selec edge node EN_j om NR wi h su icien esou ces R
Kanji RK Eu o. J. Ad . Engg. Tech., 2021, 8(11):152-155
154
Assign d_i o EN_j
EndFo
EndWhile
// Phase 4: Edge Node P ocessing
Fo each edge node EN_j do
Fo each assigned da a chunk d_i do
P ocess d_i using p ede ined asks T
S o e in e media e esul s locally o ansmi o neighbo ing nodes i needed
EndFo
EndFo
// Phase 5: Load Balancing
Pe iodically assess load on all edge nodes in NR
Redis ibu e asks om o e loaded nodes o unde u ilized nodes
// Phase 6: Da a Compila ion and T ansmission
Collec p ocessed da a chunks P_i om edge nodes
Agg ega e P_i in o inal esul s P
T ansmi agg ega ed da a P o cloud s o age
// Phase 7: Pe o mance Moni o ing and Adap a ion
Con inuously moni o pe o mance me ics (la ency, h oughpu , scalabili y)
Adap ask scheduling and da a pa i ioning dynamically based on moni o ed da a
End
The me hodology o implemen ing he p oposed algo i hm o big da a p ocessing a edge nodes is s uc u ed in o
he ollowing phases:
1. Sys em Ini ializa ion:
O Edge Node Regis a ion: Ini ialize he sys em by egis e ing a ailable edge nodes wi h hei espec i e
compu a ional and memo y capabili ies in a cen al Node Regis y.
O Task De ini ion: De ine da a p ocessing asks, including il e ing, agg ega ion, and analy ics, along wi h hei
esou ce equi emen s and p io i y le els.
2. Da a Inges ion and Pa i ioning:
O Real- ime da a s eams om IoT de ices a e moni o ed and inges ed.
O Da a is pa i ioned in o manageable chunks o sub-s eams based on size, da a ype, o p ocessing p io i y.
3. Resou ce Alloca ion:
O Assign da a chunks o edge nodes dynamically based on hei cu en load and esou ce a ailabili y.
O Implemen esou ce-awa e scheduling o ensu e asks a e dis ibu ed e enly and c i ical asks a e p io i ized.
4. Edge Node P ocessing:
O Each edge node p ocesses he assigned da a chunks using p ede ined asks such as p ep ocessing, machine
lea ning in e ence, o anomaly de ec ion.
O In e media e esul s a e s o ed locally o ansmi ed o neighbo ing nodes o u he p ocessing, depending on
ask equi emen s.
5. Load Balancing:
O Pe iodic load assessmen is conduc ed ac oss all edge nodes o iden i y bo lenecks o unde u ilized esou ces.
O Tasks a e edis ibu ed dynamically o ensu e balanced u iliza ion and minimize p ocessing delays.
6. Da a Compila ion and T ansmission:
O P ocessed da a om edge nodes is compiled and agg ega ed in o inal esul s.
O Agg ega ed da a is ansmi ed o he cloud o long- e m s o age o u he analysis, ensu ing minimal la ency.
This me hodology ensu es ha he algo i hm achie es op imal esou ce u iliza ion, low la ency, and high
h oughpu while main aining scalabili y in edge-based big da a a chi ec u es.
This me hodology ensu es ha he algo i hm achie es op imal esou ce u iliza ion, low la ency, and high
h oughpu while main aining scalabili y in edge-based big da a a chi ec u es.
RESULTS & DISCUSSION
The implemen a ion o he p oposed algo i hm demons a ed signi ican imp o emen s in eal- ime big da a
p ocessing using edge nodes. The esul s we e analyzed based on key pe o mance me ics, and he indings a e
summa ized as ollows:
Kanji RK Eu o. J. Ad . Engg. Tech., 2021, 8(11):152-155
155
1. La ency Reduc ion:
O The algo i hm achie ed a educ ion in la ency compa ed o adi ional cloud-cen ic a chi ec u es. The a e age
end- o-end p ocessing ime o da a chunks dec eased signi ican ly.
2. Th oughpu Enhancemen :
O Wi h dynamic esou ce alloca ion and e icien load balancing, he h oughpu inc eased by a e y good amoun .
3. Scalabili y:
O The edge-based a chi ec u e main ained consis en pe o mance unde high da a loads, scaling be e han he
baseline olume wi hou signi ican deg ada ion in la ency o h oughpu .
CONCLUSION
The in eg a ion o edge compu ing wi h eal- ime big da a p ocessing o e s signi ican ad ancemen s in add essing
he la ency, h oughpu , and scalabili y challenges o adi ional cloud-based sys ems. By le e aging dynamic
esou ce alloca ion, e icien load balancing, and localized p ocessing a he edge, pe o mance is no ably enhanced
while educing ene gy consump ion. The inco po a ion o machine lea ning models a edge nodes u he augmen s
decision-making accu acy, enabling eal- ime insigh s ac oss a ious applica ions such as sma ci ies and
heal hca e. These imp o emen s highligh edge compu ing's po en ial as a ans o ma i e pa adigm in big da a
p ocessing, p o iding a scalable and e icien solu ion o mode n da a-in ensi e applica ions.
Fu u e esea ch should ocus on e ining secu i y measu es, op imizing he syne gy be ween edge and cloud
compu ing, and ex ending he algo i hm o handle mo e complex and dynamic da ase s. This will ensu e he
adap abili y and obus ness o edge compu ing in e ol ing echnological landscapes. The con inued e olu ion and
in eg a ion o ad anced AI echniques will u he enhance he capabili ies o edge-based sys ems, d i ing
inno a ion and imp o ing se ice quali y in c i ical sec o s.
REFERENCES
[1]. Sa yana ayanan, M., e al. (2017). The ole o edge compu ing in cloud compu ing. IEEE Compu e
Socie y.
[2]. Mille , P. (2018, May). Wha is edge compu ing? The Ve ge.
[3]. Li, S., T y onas, T., & Li, H. (2016). The In e ne o Things: A secu i y poin o iew. In e ne Resea ch,
26(2), 337–359.
[4]. Gubbi, J., Buyya, R., Ma usic, S., & Palaniswami, M. (2013). In e ne o Things (IoT): A ision,
a chi ec u al elemen s, and u u e di ec ions. Fu u e Gene a ion Compu e Sys ems, 29(7), 1645–1660.
[5]. Bonne , P., Geh ke, J., & Seshad i, P. (2001). Towa ds senso da abase sys ems. Mobile Da a
Managemen , 3(1), 3–14.
[6]. Fu, Z., & Lee, Y. C. (2019). Edge compu ing: A u u e end o IoT and big da a p ocessing. In M. U. S.
Khan, S. U. Khan, & A. Y. Zomaya (Eds.), Big Da a-enabled In e ne o Things (pp. 373–386). Ins i u ion
o Enginee ing and Technology.
[7]. Zhou, Z., Li, D., Zeng, L., & Chen, X. (2019). Edge AI: On-demand accele a ing deep neu al ne wo k
in e ence ia edge compu ing. In P oceedings o he 2019 IEEE In e na ional Con e ence on Edge
Compu ing (EDGE) (pp. 1–6). IEEE.
[8]. Anaya, E., Acos a-Be mejo, R., & Salinas-Rosales, M. (2018). Dis ibu ing compu ing in he In e ne o
Things: Cloud, og, and edge compu ing o e iew. In NEO 2016: Resul s o he Nume ical and
E olu iona y Op imiza ion Wo kshop NEO 2016 and he NEO Ci ies 2016 Wo kshop (pp. 87–115).
Sp inge In e na ional Publishing.
[9]. Langona, K., Sil a, E. A., de B i o Ca alho, T. C. M., & Se idcon, A. (2016). Fog compu ing: Da a
analy ics and cloud dis ibu ed p ocessing on he ne wo k edges. In 2016 35 h In e na ional Con e ence o
he Chilean Compu e Science Socie y (SCCC) (pp. 1–9). IEEE.