Database management system performance comparisons : A systematic literature review

This is a sel -a chi ed e sion o an o iginal a icle. This e sion may di e om he o iginal in pagina ion and ypog aphic de ails. Au ho (s): Ti le: Yea : Ve sion: Copy igh : Righ s: Righ s u l: Please ci e he o iginal e sion: CC BY 4.0 h ps://c ea i ecommons.o g/licenses/by/4.0/ Da abase managemen sys em pe o mance compa isons : A sys ema ic li e a u e e iew © 2023 The Au ho (s). Published by Else ie Inc. Published e sion Taipalus, Toni Taipalus, T. (2024). Da abase managemen sys em pe o mance compa isons : A sys ema ic li e a u e e iew. Jou nal o Sys ems and So wa e, 208, A icle 111872. h ps://doi.o g/10.1016/j.jss.2023.111872 2024 The Jou nal o Sys ems and So wa e 208 (2024) 111872 A ailable online 27 Oc obe 2023 0164-1212/© 2023 The Au ho (s). Published by Else ie Inc. This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/). Con en s lis s a ailable a ScienceDi ec The Jou nal o Sys ems & So wa e jou nal homepage: www.else ie .com/loca e/jss Da abase managemen sys em pe o mance compa isons: A sys ema ic li e a u e e iew✩ Toni Taipalus Facul y o In o ma ion Technology, Uni e si y o Jy äskylä, P.O. Box 35, FI-40014, Finland ARTICLE INFO Keywo ds: Da abase Pe o mance Compa ison Da abase managemen sys em Rela ional da abase NoSQL NewSQL ABSTRACT E iciency has been a pi o al aspec o he so wa e indus y since i s incep ion, as a sys em ha se es he end-use as , and he se ice p o ide cos -e icien ly bene i s all pa ies. A da abase managemen sys em (DBMS) is an in eg al pa o e ec i ely all so wa e sys ems, and he e o e i is logical ha di e en s udies ha e compa ed he pe o mance o di e en DBMSs in hopes o inding he mos e icien one. This s udy sys ema ically syn hesizes he esul s and app oaches o s udies ha compa e DBMS pe o mance and p o ides ecommenda ions o indus y and esea ch. The esul s show ha pe o mance is usually es ed in a way ha does no e lec eal-wo ld use cases, and ha es s a e ypically epo ed in insu icien de ail o eplica ion o o d awing conclusions om he s a ed esul s. 1. In oduc ion E iciency is impo an in e ec i ely all so wa e sys ems, whe he e iciency is measu ed by esponse imes, how many concu en use s he sys em can se e, o how ene gy-e icien he sys em is (To ola e al.,2018). Despi e i s impo ance, many so wa e sys ems su e om e iciency p oblems (Jin e al.,2012), as op imiza ion has been la gely ecognized as a complex ask (To ola e al.,2018;Di allah e al.,2013). The mo e a sys em holds and handles da a, he mo e he sys em’s pe o mance depends on he da abase, and he da abase is o en one o he i s suspec s when a pe o mance issue is de ec ed. The domain o da abase managemen sys ems (DBMS) saw apid ad ancemen s in pe o mance especially in he 1980s and 1990s, as benchma king com- pe i ions be ween DBMS and ha dwa e endo s led o inno a ions in DBMS echnology ha signi ican ly imp o ed DBMS pe o mance (De- Wi and Le ine,2008). Pe o mance imp o emen s a e ela ed o DBMS aspec s such as di e en suppo ing da a s uc u es (Valdu iez, 1987), and algo i hms o so ing (Es i ill-Cas o and Wood,1992; Do e al.,2022) and joining (Schneide and DeWi ,1989;Pa el and DeWi ,1996). Gi en ha DBMSs a e annually a mul i-billion dolla indus y, he pe o mance o a DBMS is one o he mos c ucial aspec s when a company chooses a DBMS o hei p oduc o se ice (Die ich e al.,1992). As di e en DBMS pe o mance compa ison s udies and DBMS endo whi e-pape s highligh he pe o mance gains o one DBMS o e ano he , i may seem emp ing o ei he conside choosing he as es DBMS o a business domain o o mig a e om one DBMS o ano he o pe o mance gains. Howe e , as we show and a gue in his ✩Edi o : D . Jacopo Soldani. E-mail add ess: [email p o ec ed]. s udy, pe o mance is ypically es ed in e y speci ic con ex s which a e no necessa ily gene alizable, and he e a e o he aspec s besides pe o mance o conside . This s udy was inspi ed by a s udy by Raas eld e al. (2018), which claimed ha ‘‘[...] we will explo e he common pi alls in da abase pe o mance benchma king ha a e p esen in a la ge numbe o scien i ic wo ks [...]’’ while consciously e aining om ci ing example s udies. While we ag ee wi h hei claim based on ou pe sonal expe iences, we wan ed o sys ema ically explo e whe he his phenomenon is com- mon among pe o mance compa isons, and whe he such s udies show pe o mance gains o one DBMS o e ano he in a se ing ha can be eplica ed. This s udy is no an a emp o c i icize s udies compa ing DBMS pe o mance, as no scien i ic s udy (ou s included) is wi hou h ea s o alidi y. Ra he , based on he su ey o he li e a u e, he p ima y goals o ou s udy a e o p opaga e in o ma ion on (i) how DBMS pe o mance has been es ed, (ii) how pe o mance has been ecommended o be es ed, (iii) how he pe o mance compa ison e- sul s should be in e p e ed, (i ) wha o he aspec s besides pe o mance should be conside ed, and ( ) wha o he a enues migh be ui ul o DBMS pe o mance es ing. Addi ionally, we p o ide ( i) a ela i ely accessible backg ound on da abase sys em pe o mance, ollowed by ( ii) a sys ema ic e iew o li e a u e on DBMS pe o mance compa - isons, ( iii) desc ibing which DBMSs and which ypes o DBMSs ha e been compa ed wi h each o he , (ix) he ou comes o he pe o mance compa isons, and (x) by which benchma ks he DBMSs ha e been compa ed. h ps://doi.o g/10.1016/j.jss.2023.111872 Recei ed 10 Ma ch 2023; Accep ed 4 Oc obe 2023 The Jou nal o Sys ems & So wa e 208 (2024) 111872 2 T. Taipalus The es o his s udy is s uc u ed as ollows. In Sec ions 2and 3, we p o ide heo e ical backg ound o unde s anding he esul s and discussion p o ided by his s udy. These backg ound sec ions a e delibe a ely p esen ed by e aining om using unnecessa y in o ma- ion echnology- ela ed e ms, ac onyms, algo i hms, o ma hema ics, o ca e o he needs o eade s om a ious backg ounds. Fo eade s mo e echnically inclined o in e es ed, we ha e p o ided u he ead- ing a he end o Sec ions 2and 3. Sec ion 4de ails how we sea ched, selec ed, and ca ego ized he DBMS pe o mance compa ison s udies, and Sec ion 5p esen s a high-le el o e iew o he esul s, which is complemen ed by he Appendix de ailing he pe o mance compa ison ou comes. In Sec ion 6, we discuss wha hese indings mean, how hey a e applicable in indus y, and p esen ou ecommenda ions o indus y and esea ch based on he indings. Sec ion 7concludes he s udy. 2. Da abase sys ems 2.1. Da abase sys em o e iew A da abase is a collec ion o in e ela ed da a, ypically s o ed acco ding o a da a model. Typically, he da abase is used by one o se e al so wa e applica ions ia a DBMS. Collec i ely, he da abase, he DBMS, and he so wa e applica ion a e e e ed o as a da abase sys em (Elmas i and Na a he,2016, p. 7)(Connolly and Begg,2015, p. 65). The sepa a ion o he da abase and he DBMS, especially in he ealm o ela ional da abases, is ypically impossible wi hou expo ing he da abase in ano he o ma . In hese si ua ions, he da abase is o en unusable by he DBMS, unless he da abase is impo ed back o a o ma unde s ood by he DBMS. Possibly due o his insepa abili y, bo h he DBMS and he unde lying da abase a e o en colloquially e e ed o simply as da abase. I is wo h no ing, hough, ha he o me is a piece o so wa e ha does, while he o he is a collec ion o da a ha is. Fig. 1 shows a simpli ied example o a sys em whe e he compo- nen s c ucial o a da abase sys em and he scope o his s udy a e emphasized. We e e o he componen s in he igu e h oughou his s udy. Se e al hings a e wo h no ing in conside ing he igu e, as we ha e aded echnical p ecision and comp ehensi eness o ease o p esen a ion by depic ing only a single end-use , a single so wa e applica ion (some pa s ypically eside on he end-use ’s de ice, while o he s eside on a sepa a e se e ), a single DBMS, single ha dwa e componen s, and a single da abase. Fu he mo e, we ha e no illus- a ed o he DBMS componen s such as access con ol, da a s uc u es such as me ada a, o ou pu s such as que y execu ion plans. The igu e also adop s he iew ha he da abase esides in pe sis en s o age — his is no always he case. Addi ionally, we ha e depic ed me ely a cen alized da abase sys em in which nei he he DBMS no he da abase has been dis ibu ed ac oss mul iple nodes. These a e will ul omissions gi en he scope o his s udy. 2.2. Da a models Da abases ollow one o se e al da a models, i.e., de ini ions o how and wha da a can be s o ed, and some imes, wha ope a ions a e a ailable o da a e ie al and manipula ion. Da a models may be concep ual, logical, o physical. Concep ual models such as he En i y- Rela ionship model (Chen,1976) do no dic a e how da a should be s o ed, bu a e a he used o desc ibe he in e ela ions and cha - ac e is ics o he da a. Logical da a models such as he ela ional model (Codd,1970) a e ela ed o how da a is s o ed and p esen ed, bu o en wi hou desc ibing how he da a is physically s o ed, e.g., which compu ing node is esponsible o s o ing he da a, whe e he da a is loca ed on a disk, and wha ypes o indices (i.e., edundan da a s uc- u es which acili a e que y pe o mance) and physical da a e ie al ope a o s a e a ailable. One DBMS is no limi ed o using a single da a model (Fo esi e al.,2022). The e a e se e al popula logical da a models, some o which a e insepa able om hei unde lying physical da a models. One o he mos p ominen logical da a models is he ela ional da a model oo ed in se heo y (Codd,1970). Rela ional DBMSs (RDBMS) ollow many o he concep s in oduced in he ela ional model. Many o he popula RDBMSs such as Pos g eSQL and O acle Da abase ha e adop ed da a s uc u es om o he logical da a models as well (Lu and Holubo á, 2019). Wha is common o e ec i ely all mode n RDBMSs is ha hey u ilize S uc u ed Que y Language (SQL) (ISO/IEC,2016a,b) o de ine da a s uc u es and o e ie e and manipula e da a. Typically, RDBMSs also implemen a s ong da a consis ency model which dic a es o allows ha da abase ope a ions g ouped in o a ansac ion mus all succeed o all ail, da a mus ollow de ined business logic, suc- cess ul ansac ions pe sis in s o age, and concu en ansac ions (c . Be ns ein and Goodman,1981) mus esul in he same da a as i he ansac ions we e se ial. A leas he las ule can o en be loosened in mode n implemen a ions o a ious deg ees. These cons ain s a e collec i ely e e ed o as he ACID consis ency model (Hae de and Reu e ,1983). NoSQL is an umb ella e m o se e al da a models, ypically de eloped o popula ized in he i s decade o he 2000s (G olinge e al.,2013). Con a y o he ela ional model, he da a models wi hin NoSQL ypically ha e no o mal de ini ions, and di e en NoSQL DBMSs implemen di e en da a models such as key– alue (e.g., Redis), documen (e.g., MongoDB), wide-column (e.g., Cassand a) and g aph (e.g., Neo4J) (Da oudian e al.,2018;Renie s e al.,2017). Fu he - mo e, hese DBMSs o en ha e a dis inc que y language de eloped o ca e o he pa icula da a s uc u es a ailable in he DBMS’s implemen a ion o a da a model. While RDBMSs ha e a o ed da a consis ency (Chaudh y and Yousa ,2020) by elimina ing edundan da a h ough logical da abase design, and h ough a s ong consis ency model, NoSQL DBMSs ha e gene ally adop ed he opposi e app oach. In se e al NoSQL da a models such as key– alue pai s and docu- men s, edundan da a a e s o ed a he cos o s o age space (Hech and Jablonski,2011). This app oach enables que y languages o be simple (Dey e al.,2014), a oiding complex and po en ially slow que ies. Fu he mo e, consis ency models a e ypically less s ic han in RDBMSs (S oneb ake ,2010), which acili a es highe pe o mance demanded by, e.g., web applica ions wi h a la ge numbe o concu en use s (Ramak ishnan,2012). Al hough NoSQL DBMSs popula ized se e al da abase- ela ed ap- p oaches such as non-s ic da abase s uc u es, da a a ailabili y o e da a consis ency, and ela i ely e o less da abase eplica ion (i.e., da a is copied o e compu ing nodes) and sha ding (i.e., da a is di ided be ween compu ing nodes) (G olinge e al.,2013), some indus y leade s such as Google deemed a s ong consis ency model and an exp essi e que y language impo an enough o design a DBMS which inco po a es ea u es om bo h RDBMSs and NoSQL DBMSs (Co be e al.,2013). These so-called NewSQL DBMSs use he ela ional model, o en wi h ex ensions, SQL as hei p ima y que y language, and a dis ibu ed da abase a chi ec u e (Pa lo and Asle ,2016). In addi ion o hese h ee main ca ego ies o RDBMS, NoSQL, and NewSQL da a models, o he s such as objec s o es (Kulsh es ha and Sachde a,2014) and GPU-in ensi e (Suh e al.,2022) sys ems a e used in speci ic con ex s. 2.3. Que y execu ion The wo d que y ypically e e s o que y language s a emen s ha e ie e some da a om he da abase. Howe e , in his s udy, we use he wo d que y o e e o any da a e ie al and manipula ion s a emen o b e i y. In imes i is necessa y o di e en ia e be ween da a e ie al and manipula ion, we use app op ia e e ms such as ead ope a ions o da a e ie al, and w i e ope a ions o da a inse ion, upda es, and dele es. In his subsec ion, we desc ibe how que ies a e The Jou nal o Sys ems & So wa e 208 (2024) 111872 3 T. Taipalus Fig. 1. A simpli ied iew o a da abase sys em and he end-use wi h he emphasis on componen s ele an o his s udy; he a ows ep esen he low o in o ma ion om he end-use ’s de ice o he da abase esiding in pe sis en s o age; he low o in o ma ion back o he so wa e applica ion is no illus a ed he e; g ay ec angles ep esen bounda ies o physical de ices. execu ed, using mainly gene al (i.e., no speci ic o a single DBMS) li e a u e om he domain o RDBMS que y execu ion. When a use — we e i a human ac o di ec ly using a e minal, a ansac ion p ocessing so wa e applica ion, o a da abase benchma k so wa e — submi s a que y o a DBMS, a mul i ude o e en s mus ake place be o e he use ecei es eedback. Illus a ed in a gene al ashion in Fig. 1, he que y pa se checks, among o he hings, ha he que y is syn ac ically alid (Helle s ein e al.,2007). I he que y passes hese (and o he ) checks, he que y is ansla ed o a lowe -le el p esen a ion and passed o he que y op imize . The op imize gene a es one o se e al que y execu ion plans. These plans consis o physical ope a o s o implemen ing, e.g., which physical da a s uc u es will be u ilized in execu ing he que y, and in RDBMSs in pa icula , how ables a e joined oge he (G ae e,1993). I se e al plans a e gene a ed, he op imize e alua es which o hese plans is he mos e ec i e in ega ds o, e.g., que y execu ion ime (Helle s ein e al.,2007). The accu acy o he op imize elies on aspec s such as da abase me ada a (Ch is odoulakis, 1984), s a is ics o p e ious que y execu ions, and he indices a ail- able (Chaudhu i,1998). Gene a ing e ec i e que y execu ion plans is a complex e o and akes ime (G ae e,1993;Chaudhu i,1998), bu once o mula ed, he plans can be e-used o a deg ee. Nex , he que y execu ion engine implemen s he que y execu ion plan, using he physical ope a o s he ein. Simpli ied, he da a objec s equi ed by he que y a e ypically i s sea ched om a memo y a ea called he bu e pool which is alloca ed and main ained by he DBMS. I some o all da a is no ound, he da a is eques ed om disk. Be o e accessing he disk, many sys ems may addi ionally u ilize o he a eas o memo y o a oid disk access (Yang and Lilja,2018). E ec i ely all da abase sys ems unc ion in an en i onmen whe e mul iple concu en end-use s use he da abase. This concu ency p esen s challenges pa icula ly when he use s execu e w i e ope a- ions on he same da abase, e.g., when wo o mo e use s wi hd aw money om he same bank accoun , concu en ly upda ing he bal- ance (Be ns ein and Goodman,1981). To gua an ee ha he w i e ope a ions do no in e e e wi h each o he in a way ha would cause he da a o no ep esen he eal wo ld, DBMSs ypically implemen concu ency con ol h ough locking o e sioning da a. E ec i ely, he simple implemen a ions o locking es ic da a objec s om be- ing accessed by o he ope a ions while he da a objec s a e being modi ied (Helle s ein e al.,2007). These locking mechanisms may be implemen ed o ensu e ha no anomalies happen, o wi h implemen a- ions ha heo e ically allow some anomalies (Be enson e al.,1995). Typically, he business domain dic a es wha ypes o anomalies a e ole a ed. Finally, as s ong consis ency models o en equi e ha ansac ions pe sis in he da abase and ha all o none o he ope a ions in a ans- ac ion succeed, locking is ypically complemen ed by ansac ion logs. These logs a e w i en be o e w i e ope a ions a e commi ed o he da abase, and can be used in e e sing ea lie w i e ope a ions i a la e w i e ope a ion in he same ansac ion ails. All hese conside a ions discussed in his sec ion play a signi ican ole om a pe o mance pe spec i e, which is discussed in he nex sec ion. Fu he eading on da abase sys ems: o eade s in e es ed in he basics o da abase sys ems, ei he he unde g adua e le el ex book by Connolly and Begg (2015), o Elmas i and Na a he (2016) a e excellen albei leng hy in oduc ions co e ing he opic om se e al poin s o iew and wi h he ocus on RDBMSs. Fo eade s in e es ed in que y p ocessing, we poin o s udies by Chaudhu y (1998), and Helle s ein, S oneb ake and Hamil on (2007). I you a e in e es ed in logical ela ional da abase design, he book by Da e (2019) is an in- dep h esou ce co e ing bo h o mal and in o mal app oaches. Fo a su ey o li e a u e on NoSQL da a models, he s udy by Da oudian e al. (2018) is an accessible s a ing poin . 3. Pe o mance 3.1. Pe o mance measu emen In gene al, pe o mance is a measu emen o how e icien ly a so wa e sys em comple es i s asks. Pe o mance is ypically measu ed in esponse ime, h oughpu (Helle s ein e al.,2007), o in some cases, u iliza ion o compu ing esou ces (Co ellessa e al.,2011, p. 4). Response ime is he ime aken o a call in he sys em o a e se o some o he pa o he sys em and back. This is also some imes called la ency (Gun he ,2011, p. 10), and in he con ex o da abase sys ems, he esponse ime may be measu ed as he esponse ime o he i s o he las esul i em (G ae e,1993). In a b oad pe spec i e desc ibed in Fig. 1, he esponse ime migh be he ime aken a e he end- use sends a eques o he so wa e applica ion (e.g., an online s o e), which passes he eques o a DBMS, which e u ns a se o da a o he so wa e applica ion, which inally p esen s he da a o he end-use ’s de ice. In da abase benchma king, howe e , esponse ime migh be measu ed by unning he benchma k on he same de ice he DBMS and he da abase eside, e ec i ely elimina ing in e -de ice-induced pe o mance d awbacks such as ne wo k la ency (Pa ounas e al.,2020; The Jou nal o Sys ems & So wa e 208 (2024) 111872 4 T. Taipalus Delis and Roussopoulos,1993) and i ewalls, and mi iga ing he e ec s o o he so wa e unning on he de ices. Al hough DBMSs pe o m o he asks besides que ying, que ying is ypically wha is measu ed in DBMS pe o mance es ing (Die ich e al.,1992). While esponse ime is pe haps he leas a duous pe o mance me ic o measu e, i is no o en enough o eliable measu emen o ansac ion p ocessing en i onmen s (Die ich e al.,1992) (o en dubbed online ansac ion p ocessing, OLTP). Tha is, esponse ime migh be a me ic be e sui ed o long- unning que ies in decision suppo en i onmen s (o - en dubbed online analy ical p ocessing, OLAP), bu as ansac ion p ocessing en i onmen s o en p ocess a la ge numbe o concu en ansac ions, esponse ime alone migh no eliably accoun o he e ec s o concu en ansac ions, unless esponse ime is measu ed as an a e age o mul iple concu en ansac ions. Pe o mance can also be measu ed by h oughpu , i.e., how many ansac ions he DBMS can execu e in a gi en ime ame. Th oughpu is o en exp essed as ansac ions pe second (Die ich e al.,1992) and equi es a mo e sophis ica ed app oach, e.g., benchma king so wa e. Again, h oughpu may be measu ed ei he locally (i.e., using only he ha dwa e he DBMS and he da abase eside on), o o e a ne wo k in case he da abase is dis ibu ed. Al e na i ely, h oughpu may be measu ed by connec ing he benchma king so wa e o he so wa e applica ion, which simula es he h oughpu o he whole da abase sys- em by accoun ing o , e.g., ne wo k and he so wa e applica ion (e.g., Kuma and G o ,2022;Sunda esan e al.,2013). Such an app oach a guably equi es signi ican ly mo e in es men , bu p o ides a holis ic pe spec i e on he pe o mance o he whole sys em, also unco e ing po en ial pe o mance issues un ela ed o he DBMS and he da abase. Finally, pe o mance may be measu ed by esou ce u iliza ion, ei he CPU ime, I/O, memo y alloca ion, o ene gy consump ion (G ae e, 1993) in sys ems s i ing o ene gy-e iciency due o, e.g., limi ed ba e y powe , o due o en i onmen al conce ns (Guo e al.,2022). In summa y, we migh conside he measu emen o h oughpu a p ocess ha ypically equi es a simula ion o some le el, and he measu emen o esponse ime as an exac o app oxima ed ma hema - ical me hod. The o me app oach equi es ela i ely high in es men s in o he de elopmen o such simula ions (Co ellessa e al.,2011, p. 142), while he la e o en elies on a se o assump ions ha do no necessa ily e lec eal-wo ld scena ios due o inaccu acies in p edic ing wha he eal-wo ld scena io ul ima ely is and how i can change. 3.2. Fac o s a ec ing pe o mance Ha dwa e: An in ui i e ac o in pe o mance is he powe o ha d- wa e (Os e hage,2013, p. 1), and while i is ue ha mos o he local esponse ime is a ibu ed o ime aken by CPU p ocessing, memo y and disk access, and so wa e wai ing o o he asks o com- ple e (Co ellessa e al.,2011, p. 5), i s in es ing in so wa e pe o - mance a he han ha dwa e pe o mance is o en mo e cos -e ec i e. Tha being said, i is gene ally accep ed ha memo y access is a leas ou o de s o magni ude as e han disk access (e.g., Gun he ,2011, p. 42). Tha is, i memo y access akes minu es (nanoseconds), disk access akes mon hs (milliseconds). These numbe s a e la gely dependen on he speed o memo y and he ype o disk, bu pain a pic u e o how zealously DBMS op imiza ion s i es o minimize disk access. Since memo y is ypically mo e expensi e han disk s o age, keeping he whole da abase in memo y is o en un easible. Addi ionally, he unde lying ha dwa e is impo an , as, e.g., some DBMSs ha e been shown o u ilize mul i-p ocesso o mul i-co e en i onmen s mo e e - ec i ely han o he s (Tu e al.,2013). In ui i ely, how well a DBMS can exploi pa allelism a ec s he pe o mance o que y execu ion (Tallen and Mello -C ummey,2009;Tözün e al.,2013). Ul ima ely, pe o - mance measu emen is abou gains o losses in pe cen ages, no in, e.g., esponse imes. Da a models: Da a models desc ibed in Sec ion 2.2 ha e indi ec e ec s on DBMS pe o mance. Rela ional da abases o en ollow design guidelines ha s i e o minimize edundancy o elimina e po en ial da a anomalies caused by edundan da a (Codd,1972,1975), and o minimize he need o s o age space, which in u n ypically causes que ies o un slowe due o a la ge numbe o able joins. In con as , di e en NoSQL da a models — especially key– alue, documen , and wide-column — ollow design guidelines acco ding o which da a s uc u es a e designed o e icien ly sa is y p ede e mined business logic que ies, wi h he elimina ion o edundan da a being a seconda y conce n (Da oudian e al.,2018). I ollows ha because many NoSQL da a s uc u es a e designed o se e que ies, que ies a e ypically sim- ple (Dey e al.,2014), and hei execu ion equi es less compu a ional esou ces han complex que ies in ela ional da abases. As discussed in Sec ion 2.3, locking da a objec s (bo h on disk and in memo y, and bo h p ima y da a s uc u es as well as indices), logging w i e ope a ions, and how memo y is managed by he DBMS all play a signi ican ole in DBMS pe o mance (Helle s ein e al.,2007;S oneb ake ,2010). Fo example, p e en ing w i e ope a ion-induced anomalies is a cos ly ac ion, and he le el o g anula i y o da abase locks p esen s signi i- can conside a ions on w i e ope a ion pe o mance, which is la gely dic a ed by he a io o ead and w i e ope a ions. Dis ibu ion: W i e ope a ions in dis ibu ed con igu a ions pose non- i ial challenges o bo h pe o mance and da a consis ency (Delis and Roussopoulos,1993). In dis ibu ed da abase sys ems, e ec i ely all ansac ions mus choose ei he da a consis ency o da a a ailabil- i y (B ewe ,2012;Gilbe and Lynch,2002). The o me gua an ees ha he da a he end-use ecei es a e no s ale, wi h he cos o pe o mance, while he la e gua an ees o a deg ee ha he end-use ecei es da a as e , bu wi h no gua an ees ha he da ase ecei ed is he mos ecen . The p e e ed app oach is la gely dic a ed by business logic. DBMS and OS pa ame e s: Mo ing om da a models and da abase sys em dis ibu ion o lowe le els o abs ac ion, ope a ing sys em (OS) and DBMS pa ame e s and hei in e ela ionships (e.g., page size) can ha e di ec o indi ec e ec s on pe o mance (Die ich e al., 1992). Addi ionally, DBMS pa ame e s such as he amoun o memo y he DBMS is allowed o use o da a p ocessing is ypically closely ela ed o he amoun o memo y a ailable. Fu he mo e, as a que y is sen o he op imize (c . Fig. 1), i depends on he DBMS in e nals how e icien ly he op imize can selec he mos e icien physical ope a ions o implemen he que y, and wha physical ope a ions a e a ailable o he op imize in he i s place (Chaudhu i,1998). Fo example, MySQL implemen ed only one physical ope a ion o able joins un il 2018,1limi ing he numbe o op ions he op imize could choose om. Rega ding que y op imiza ion, he op imize s o RDBMSs in pa icula a e ela i ely ma u e and can spo some unnecessa y complica ions in que ies, while o e looking o he s (B ass and Goldbe g, 2006). Despi e he bene i s b ough by he op imize s, some que ies a e inhe en ly slow and can only be op imized h ough que y ew i es. Physical da abase design: Las , bu de ini ely no leas , physical da abase design plays a key ole in DBMS pe o mance. I has been a gued ha pe o mance bo lenecks a e di icul o ind in la ge sys ems (Ammons e al.,2004), and ha e iciency is gained by ocusing on he i al ew a eas ins ead o he i ial many (Ju an and De Feo, 2010, p. 450). One o he mos i al a eas in da abase sys ems is physical design. In ela ional da abases, e icien physical design is la gely achie ed h ough indices, and in NoSQL da abases, ypically h ough da abase dis ibu ion o e compu ing nodes. In con as o a holis ic sys em o e iew, pe o mance bo lenecks may be easie o ind in que ies, since many DBMSs p o ide de ailed in o ma ion on que y execu ion (Fig. 2). Pos g eSQL (Fig. 2(a)) lis s he physical ope a ions used o execu e he que y, which o he ope a ions ook he mos ime uni s, and which indices, i any, we e used. Fo example, i can be seen in Fig. 2(a) ha he sequen ial scan on line 12 accoun ed o 1h ps://de .mysql.com/doc/ e man/5.6/en/explain-ou pu .h ml The Jou nal o Sys ems & So wa e 208 (2024) 111872 5 T. Taipalus Fig. 2. Que y execu ion plans illus a ing he physical ope a o s such as hash join and seq scan chosen by he op imize . app oxima ely 94% o he execu ion ime o he whole que y (178 ime uni s ou o 189 ms), p obably because he que y e ched a la ge numbe o eco ds om he da abase. The que y could be op imized by, e.g., selec ing a smalle numbe o eco ds, and showing he esul s o he end-use by paging hem, i.e., showing a subse o esul s i s , and e ching mo e la e i necessa y. In NoSQL sys ems, he que y op imize plays a smalle ole due o ypically less exp essi e que y languages (c . Fig. 2(b)). Some NoSQL sys ems such as Cassand a do no pe mi he execu ion o que ies ha do no u ilize he physical s uc u es e ec i ely. 3.3. Da abase pe o mance benchma ks The e a e se e al da abase pe o mance benchma ks a ailable, each ypically consis ing o a sample da abase and a wo kload ha simula es how he da abase could be used (Di allah e al.,2013;Qu e al., 2022b). The benchma ks usually measu e he e iciency o que ying while aking in o accoun ac o s such as concu ency bu dis ega ding o he DBMS asks such as e iciency in da a s uc u e de ini ion o bulk loading (Die ich e al.,1992). In he domain o ela ional da abases, he T ansac ion P ocess- ing Council (TPC) benchma ks (e.g., G ay,1992) a e pe haps he mos u ilized (D esele e al.,2020;Tözün e al.,2013), and es he h oughpu o he DBMS wi h a ious pa ame e s. Fo example, he TPC-A benchma k simula es a da abase o a bank wi h ou ables and wi h one ansac ion, he TPC-B benchma k a da abase o a wholesale supplie wi h nine ables and wi h i e ansac ions, and he TPC-E benchma k a b oke age da abase wi h 33 ables and 12 ansac ions. All hese benchma ks ha e he op ion o simula ing s ong consis ency, and while TPC-A and TPC-B ha e ansac ions ypical o ansac ion p ocessing, TPC-E includes also decision suppo ansac ions (Tözün e al.,2013). TPC-A simula es human end-use hinking by wai ing be ween ansac ions, as a human a guably would wai be ween clicks in an online bank. TPC-B, on he o he hand, does no wai and can be used as a p ecu so o TPC-A in adjus ing DBMS pa ame e s (Die ich e al.,1992). Al e na i ely o ansac ion p ocessing, TPC-H benchma k measu es he pe o mance o a DBMS in decision suppo (Ba a a e al., 2015;D esele e al.,2020). In he mo e gene al DBMS domain, he Yahoo! Cloud Se ing Bench- ma k (YCSB) is a amewo k o benchma king ansac ion p ocessing in sys ems wi h di e en da a models and a chi ec u es (Coope e al., 2010). Due o i s ex ensibili y, YCSB can be adap ed o di e en NoSQL da a models. YCSB con ains di e en wo kloads, each wi h a di e en a io o ead and w i e ope a ions. YCSB and i s ex ensions such as YCSB+T ypically u ilize ansac ions which consis o single ope a ions and do no en o ce s ong consis ency (Qu e al.,2022b;Dey e al., 2014). The benchma ks desc ibed abo e a e by no means an exhaus i e lis bu co e he mos popula benchma ks (c . Sec ion 2.1). O he benchma ks include LUBM (Guo e al.,2005), OLTP-Bench (Di allah e al.,2013), and JOB (Leis e al.,2015). Rega dless o he da a model and DBMS, ansac ion p ocessing benchma ks ha e ypically been he de ac o me hod o compa ing di e en DBMSs and ha dwa e (Tözün e al.,2013). Fu he eading on pe o mance: o eade s in e es ed in phys- ical da abase ope a ions and que y execu ion om a pe o mance pe spec i e, G ae e (1993) p o ides an in-dep h, DBMS-independen su ey. Fo mo e in o ma ion on physical da abase design, especially indices and how hey wo k, he book by Ligh s one e al. (2010) is a de ailed and desc ip i e sou ce. Fo a p ac ical and concise guide on SQL que y op imiza ion, we poin eade s owa ds Winand’s (2012) book. Rega ding NoSQL DBMS op imiza ion, we sugges e e ing o he manual o he DBMS o you choice, and always making su e ha he sou ce o in o ma ion is cu en , as NoSQL sys ems end o e ol e apidly. The Jou nal o Sys ems & So wa e 208 (2024) 111872 6 T. Taipalus Fig. 3. The s udy selec ion p ocess; he numbe s e e o he numbe o p ima y s udies selec ed in each s age o he p ocess. Table 1 Sea ch s ings. Da abase Sea ch s ing ACM DL [Abs ac : pe o mance] AND [Abs ac : compa ison] AND [[Abs ac : da abase] OR [Abs ac : dbms]] AND [Publica ion Da e: (01/01/2000 TO 03/31/2022)] IEEE Xplo e (‘‘Abs ac ’’:pe o mance AND ‘‘Abs ac ’’:compa ison AND (‘‘Abs ac ’’:da abase OR ‘‘Abs ac ’’:dbms)) ScienceDi ec Ti le, abs ac , keywo ds: pe o mance AND compa ison AND (da abase OR dbms) Google Schola da abase pe o mance compa ison 4. S udy selec ion 4.1. P ocess and c i e ia The DBMSs in his s udy we e selec ed based on he selec ed p ima y s udies. Tha is, we did no choose, e.g., he mos popula DBMSs o include, bu epo ed he DBMSs yielded by he p ima y s udies. The esul s he ein may be conside ed he mos popula DBMSs in e ms o benchma king epo ed in scien i ic li e a u e. Fig. 3 desc ibes he p ima y s udy selec ion p ocess s a ing om ACM Digi al Lib a y, IEEE Xplo e, and ScienceDi ec , complemen ed by subsequen Google Schola sea ches. The sea ch s ings a e de ailed in Table 1. To accoun o po en ially missing ele an s udies, we conduc ed h ee ounds o backwa d snowballing (i.e., ollowing he lis s o e e ences in selec ed s udies), un il snowballing e ealed no addi ional s udies. A o al o 117 p ima y s udies compa ing DBMS pe o mance we e selec ed. Table 2 desc ibes ou inclusion c i e ia applied in he p ima y s udy selec ion. The i s ou c i e ia a e ela ed o bibliog aphic de ails, while he las h ee c i e ia a e conce ned wi h a icle ocus and con en . Rega ding c i e ion #3, we excluded academic heses and disse a ions (e.g., Coa es,2009) due o he ac ha hey a e ypically no pee - e iewed. We also excluded whi e and g ay li e a u e o he same eason, and because hose s udies a e o en w i en o published by pa ial pa ies, e.g., DBMS endo s. We only selec ed s udies ha compa ed que y (i.e., e ie ing o modi ying da a) execu ion pe o mance, no ega ding e.g., da abase eplica ion pe o mance (Elnike y e al.,2006) o pe o mance o di - e en join ope a ions (Kim and Pa el,2010). We also excluded s udies ha compa ed a single DBMS pe o mance in di e en con igu a ions such as ha dwa e, eplica ion s a egy, da abase s uc u e, o que y language (Holzschuhe and Peinl,2013) and s udies ha compa ed a DBMS wi h di e en da a- ela ed pla o ms (Pu bo e al.,2020). S udies ha epo ed pseudonymized DBMS names we e also excluded. Finally, we only included s udies ha epo ed esul s based on a leas seemingly objec i e me ics and empi ical esul s. Tha is, s udies simply s a ing he opinions o he au ho s such as ‘‘based on ou expe iences, we belie e MySQL is as e han SQL Se e ’’ we e no conside ed. 4.2. Selec ed s udies The selec ed 117 p ima y s udies compa ed he pe o mance o a o al o 44 di e en DBMSs. We ca ego ized hese DBMSs in o h ee op-le el ypes de ined and discussed in Sec ion 2.2: RDBMSs, NoSQL sys ems, and NewSQL sys ems. Fi e DBMSs no clea ly pe aining o any o hese h ee ca ego ies we e ca ego ized unde o he sys ems (Table 3). I is wo h no ing ha hese DBMS ypes a e no always clea -cu due o he lack o speci ici y and changing na u e o he de ini ions, and should be in e p e ed as me ely means o compa men- alize he esul s o his s udy in o a mo e eadable o m. Fi e selec ed p ima y s udies did no epo esul s implying he pe o mance o one DBMS o e ano he (Padhy and Kuma an,2019;Schmid e al.,2015b; Kuma Dwi edi e al.,2012;Fa aj e al.,2014;Jing e al.,2009). Fig. 4 shows he dis ibu ion o publica ion yea s and he ypes o DBMSs discussed in he selec ed s udies. Al hough ou c i e ia allowed o s udies om he yea 2000, he i s s udies selec ed we e published in 2008. The igu e shows ha gene ally, he e is a somewha cons an numbe o DBMS pe o mance compa ison s udies each yea . I is wo h no ing ha one s udy may pe ain o se e al ypes o DBMSs. 5. Pe o mance compa ison esul s The mos popula DBMS pe o mance compa isons compa ed one o se e al RDBMSs o one o se e al NoSQL sys ems, one NoSQL sys em o ano he NoSQL sys em, o one RDBMS o ano he RDBMS, espec i ely. A o al o 48 s udies compa ed solely ead pe o mance, while 6 s udies compa ed solely w i e pe o mance. The es o he s udies compa ed bo h ead and w i e pe o mance, wi h he excep ion o wo s udies (Cheng e al.,2019;Nepaliya and Gup a,2015) which we e unclea whe he hey compa ed w i e ope a ions. All compa isons and hei esul s pe DBMS ype a e summa ized in Fig. 5. Fig. 6 p esen s an o e iew o which DBMSs and DBMS ypes he p ima y s udies compa ed. The igu e pe haps con eys how bo h o he and NewSQL sys ems a e ypically compa ed wi hin hei espec- i e DBMS ype g oups, while RDBMS and NoSQL sys ems a e bo h The Jou nal o Sys ems & So wa e 208 (2024) 111872 7 T. Taipalus Table 2 P ima y s udy selec ion c i e ia. # Inclusion c i e ion 1 A icle is w i en in English. 2 Full a icle can be accessed. 3 A icle is published in a scien i ic jou nal, o con e ence o wo kshop p oceedings. 4 A icle is published be ween 2000 and Ma ch 2022. 5 A icle ocus is on que y language s a emen execu ion pe o mance compa ison. 6 A icle ocus is on compa ing he pe o mance o wo o mo e di e en DBMSs. 7 A icle is based on a leas seemingly objec i e me ics. Table 3 DBMSs discussed in his s udy di ided in o ou ypes. DBMS ype DBMSs RDBMS Access, Azu e SQL, In e base, DB/2, H2, Hi e, Ma iaDB, MySQL Clus e , MySQL, O acle Da abase, Pos g eSQL, Pos g esXL, SQL Se e , SQLi e NoSQL A angoDB, Azu e Documen Da abase, Cassand a, Couchbase, CouchDB, Elas icsea ch, Fi ebase, HBase, Hype able, memcached, MongoDB, Neo4J, O acle NoSQL, O ien DB, Ra enDB, Redis, Re hinkDB, Riak, Scala is, Ta an ool, Voldemo NewSQL Cock oachDB, MemSQL (now known as SingleS o eDB), NuoDB, Vol DB O he BlazingSQL, Caché, Db4o, OmniSciDB, PG-S om Fig. 4. The numbe o publica ions by publica ion yea and DBMS ype; he yea 2022 was only conside ed un il Ma ch. Fig. 5. DBMS pe o mance compa isons o e iew; a di ec ed edge om node a o node b ep esen s he numbe o s udies acco ding o which a sys em o ype aou pe o med a sys em o sys ems o ype bin ( )ead and (w) i e ope a ions, e.g., a NoSQL sys em ou pe o med a NewSQL sys em in ead ope a ions in one s udy, and in w i e ope a ions in one s udy; hicke edges isualize he mos popula compa isons. The Jou nal o Sys ems & So wa e 208 (2024) 111872 8 T. Taipalus Fig. 6. An o e iew o ead ope a ion pe o mance compa isons be ween NoSQL sys ems (g een, uppe igh ), NewSQL sys ems (yellow, lowe igh ), RDBMSs ( ed, lowe le ), and o he sys ems (blue, uppe le ); a clockwise u ning edge om node a o node bdepic s node aou pe o ming node b, and he colo o he edge co esponds o he ype o he ou pe o ming node, e.g., Caché ou pe o ms Pos g eSQL acco ding o one o se e al s udies; he size o a node ep esen s ou -deg ee, i.e., la ge nodes ha e ou pe o med mo e sys ems han smalle nodes. compa ed wi hin hei espec i e g oups as well as wi h each o he . Addi ionally, he size o he nodes such as MongoDB, Redis, Cassand a, and MySQL show ha hese DBMSs ypically ou pe o m he DBMSs hey a e compa ed o. Due o hei leng h, he de ailed esul s om he p ima y s udy compa isons a e p esen ed in he Appendix, which includes ables de ailing which DBMSs ou pe o med which. Rega ding he benchma ks de ined in ea lie scien i ic li e a u e, he mos popula was YCSB, which was u ilized by 15 p ima y s udies (ap- p oxima ely 13%) (Ab amo a and Be na dino,2013;Ab amo a e al., 2014a,b;Gandini e al.,2014;Sch eine e al.,2019;Seghie and Kaza , 2021;Yassien and Desouky,2016;Abubaka e al.,2014;Kashyap e al.,2013;Swamina han and Elmas i,2016;Tang and Fan,2016; Klein e al.,2015;A aujo e al.,2021;Hendawi e al.,2018;Rabl e al., 2012). The second mos popula benchma k was he TPC-H benchma k and i s a ia ions, u ilized by i e p ima y s udies (4%) (Almeida e al., 2015;Fo ache and H uba u,2016;Oli ei a and Be na dino,2017;Suh e al.,2022;Ve shinin and Mus a ina,2021). I is wo h no ing, hough, ha wo o he s udies (Oli ei a and Be na dino,2017;Ve shinin and Mus a ina,2021) seemed o ha e execu ed he que ies o TPC-H, ins ead o unning he benchma k and accoun ing o , e.g., he e ec s o concu en ansac ions. One p ima y s udy u ilized he OLTP-Bench benchma k (Tongkaw and Tongkaw,2016), one he LUBM bench- ma k (F anke e al.,2013), and one, in addi ion o TPC-H, he JOB benchma k (Suh e al.,2022). Rega ding he benchma ks o mula ed by he p ima y s udy au ho s, 25 p ima y s udies (21%) epo ed using ad hoc que ies ins ead o ea lie de ined benchma ks o compa e he pe o mance o DBMSs. These que ies we e de ined e ba im in he p ima y s udies. In con as , 70 o he p ima y s udies (60%) compa ed DBMS pe o mance using undisclosed ad hoc que ies, likely o mula ed by he s udy au ho s. In o he wo ds, 22 p ima y s udies (19%) used some ype o ea lie de ined da abase benchma king sui e. The pe - o mance es s o hese 22 p ima y s udies and wha aspec s o he en i onmen hey epo ed a e de ailed in Table 4. The Jou nal o Sys ems & So wa e 208 (2024) 111872 15 T. Taipalus Hajjaji, Yos a, Fa ah, Imed Riadh, 2018. Pe o mance in es iga ion o selec ed NoSQL da abases o massi e emo e sensing image da a s o age. In: 2018 4 h In e na ional Con e ence on Ad anced Technologies o Signal and Image P ocessing. ATSIP, IEEE. Hassan, Mahmudul, Bansal, S i idya K., 2018. Seman ic da a que ying o e NoSQL da abases wi h Apache Spa k. In: 2018 IEEE In e na ional Con e ence on In o ma ion Reuse and In eg a ion. IRI, IEEE, Sal Lake Ci y, UT, pp. 364–371. Ilić, Miloš, Kopanja, Laza , Zla ko ić, D agan, T ajko ić, Milica, Ću guz, Dejana, 2021. Mic oso SQL Se e and O acle: Compa a i e pe o mance analysis. In: Book o P oceedings o he 7 h In e na ional Con e ence Knowledge Managemen . Jaiswal, Ga ima, 2013. Compa a i e analysis o ela ional and g aph da abases. IOSR J. Eng. 03 (08), 25–27. Jandaeng, Chananko n, 2015. Compa ison o RDBMS and documen o ien ed da abase in audi log analysis. In: 2015 7 h In e na ional Con e ence on In o ma ion Technology and Elec ical Enginee ing. ICITEE, IEEE, Chiang Mai, Thailand, pp. 332–336. Jose, Benymol, Ab aham, Sajimon, 2020. Pe o mance analysis o NoSQL and ela ional da abases wi h MongoDB and MySQL. Ma e . Today: P oc. 24, 2036–2043. Jung, Min-Gyue, Youn, Seon-A., Bae, Jayon, Choi, Yong-Lak, 2015. A s udy on da a inpu and ou pu pe o mance compa ison o MongoDB and Pos g eSQL in he big da a en i onmen . In: 2015 8 h In e na ional Con e ence on Da abase Theo y and Applica ion. DTA, IEEE, Jeju Island, Sou h Ko ea, pp. 14–17. Kabakus, Abdullah Talha, Ka a, Resul, 2017. A pe o mance e alua ion o in-memo y da abases. J. King Saud Uni . - Compu . In . Sci. 29 (4), 520–525. Kau , Ka ambi , Sachde a, Monika, 2017. Pe o mance e alua ion o NewSQL da abases. In: 2017 In e na ional Con e ence on In en i e Sys ems and Con ol. ICISC, IEEE. Khan, Wisal, Ahmad, Waqas, Luo, Bin, Ahmed, Ejaz, 2019. SQL Da abase wi h physical da abase uning echnique and NoSQL g aph da abase compa isons. In: 2019 IEEE 3 d In o ma ion Technology, Ne wo king, Elec onic and Au oma ion Con ol Con e ence. ITNEC, IEEE, Chengdu, China, pp. 110–116. Khan, Wisal, ahmed, Ejaz, Shahzad, Waseem, 2017. P edic i e pe o mance compa ison analysis o ela ional & NoSQL g aph da abases. In . J. Ad . Compu . Sci. Appl. 8 (5). Khanna, Deep i, Agga wal, V.B., Di ec o , J.I.M.S., Da e, India Meenu, 2018. Pe - o mance analysis o selec , p ojec and join ope a ions o O acle, My-SQL and mic oso access DBMSS. In . J. Compu . Eng. Technol. (IJCET). Kuma , Lokesh, Rajawa , Shalini, Joshi, K a i, 2015. Compa a i e analysis o NoSQL (MongoDB) wi h MySQL da abase. In . J. Mode n T ends Eng. Res. 2 (5), 120–127. Kuma , K.B. Sundha a, S i idya, Mohana alli, S., 2017. A pe o mance compa ison o documen o ien ed NoSQL da abases. In: 2017 In e na ional Con e ence on Compu e , Communica ion and Signal P ocessing. ICCCSP, IEEE, Chennai, India, pp. 1–6. Laksono, Dany, 2018. Tes ing spa ial da a deli e ance in SQL and NoSQL da abase using nodejs ulls ack web app. In: 2018 4 h In e na ional Con e ence on Science and Technology. ICST, IEEE, Yogyaka a, pp. 1–5. Laza ska, Malgo za a, Siedlecka-Lamch, Olga, 2019. Compa a i e s udy o ela ional and g aph da abases. In: 2019 IEEE 15 h In e na ional Scien i ic Con e ence on In o ma ics. IEEE, pp. 000363–000370. Lee, Chao-Hsien, Shih, Zhe-Wei, 2018. A compa ison o NoSQL and SQL da abases o e he hadoop and spa k cloud pla o ms using machine lea ning algo i hms. In: 2018 IEEE In e na ional Con e ence on Consume Elec onics-Taiwan. ICCE-TW, IEEE, Taichung, pp. 1–2. Li, Yishan, Manoha an, Sa hiamoo hy, 2013. A pe o mance compa ison o SQL and NoSQL da abases. In: 2013 IEEE Paci ic Rim Con e ence on Communica ions, Compu e s and Signal P ocessing. PACRIM, IEEE. Lo incz, Josip, Huljic, Vla ka, Begusic, Dinko, 2020. T ans o ming p oduc ca alog ela- ional in o g aph da abase: A pe o mance compa ison. In: 2020 43 d In e na ional Con en ion on In o ma ion, Communica ion and Elec onic Technology. MIPRO, IEEE, Opa ija, C oa ia, pp. 523–528. Magdum, Junaid, Ba ha e, Rahul, 2018. Pe o mance analysis o DML ope a ions on NoSQL da abases o s eaming da a. In: 2018 Fou h In e na ional Con e ence on Compu ing Communica ion Con ol and Au oma ion. ICCUBEA, IEEE, Pune, India, pp. 1–6. Mahmood, Khalid, O sbo n, Kjell, Risch, To e, 2019. Compa ison o NoSQL da as o es o la ge scale da a s eam log analy ics. In: 2019 IEEE In e na ional Con e ence on Sma Compu ing. SMARTCOMP, IEEE, Washing on, DC, USA, pp. 478–480. Mak is, An onios, Tse pes, Kons an inos, Spiliopoulos, Giannis, Anagnos opoulos, Di- mos henis, 2019. Pe o mance e alua ion o MongoDB and Pos g eSQL o spa io- empo al da a. In: EDBT/ICDT Wo kshops. Mak is, An onios, Tse pes, Kons an inos, Spiliopoulos, Giannis, Zissis, Dimi ios, Anag- nos opoulos, Dimos henis, 2021. MongoDB Vs Pos g eSQL: A compa a i e s udy on pe o mance aspec s. GeoIn o ma ica 25 (2), 243–268. Ma e o, Luciano, Olsowy, Ve ena, Tesone, Fe nando, Thomas, Pablo, Delia, Lisan- d o, Pesado, Pa icia, 2020. Pe o mance analysis in NoSQL da abases, ela ional da abases and NoSQL da abases as a se ice in he cloud. In: A gen ine Cong ess o Compu e Science. Sp inge , pp. 157–170. Ma ogio gos, Kons ani nos, Kiou is, A hanasios, Ma ogio gou, A gy o, Ky iazis, Di- mos henis, 2021. A compa a i e s udy o MongoDB, A angoDB and CouchDB o big da a s o age. In: 2021 5 h In e na ional Con e ence on Cloud and Big Da a Compu ing. ICCBDC, ACM, Li e pool Uni ed Kingdom, pp. 8–14. Mu azza, Muh. Ra i , Nu widyan o o, A i , 2016. Cassand a and SQL da abase compa - ison o nea eal- ime Twi e da a wa ehouse. In: 2016 In e na ional Semina on In elligen Technology and I s Applica ions. ISITIA, IEEE, Lombok, Indonesia, pp. 195–200. Nya i, Suyog S., Pawa , Shi anand, Ingle, Rajesh, 2013. Pe o mance e alua ion o uns uc u ed NoSQL da a o e dis ibu ed amewo k. In: 2013 In e na ional Con e ence on Ad ances in Compu ing, Communica ions and In o ma ics. ICACCI, IEEE, Myso e, pp. 1623–1627. Ohy e , Ma ga e ha, Moniaga, Ju ike V., Sungkawa, Iwa, Subagyo, Boni asius Edwin, Chand a, Ian A gus, 2019. The compa ison i ebase eal ime da abase and MySQL da abase pe o mance using wilcoxon signed- ank es . P ocedia Compu . Sci. 157, 396–405. Pa ke , Zacha y, Poe, Sco , V bsky, Susan V., 2013. Compa ing NoSQL MongoDB o an SQL DB. In: P oceedings o he 51s ACM Sou heas Con e ence on - ACMSE ’13. ACM P ess, Sa annah, Geo gia, p. 1. Pa il, Mayu M., Hanni, Akkamahade i, Tejeshwa , C.H., Pa il, P iyada shini, 2017. A quali a i e analysis o he pe o mance o MongoDB s MySQL da abase based on inse ion and e iewal ope a ions using a web/and oid applica ion o explo e load balancing — sha ding in MongoDB and i s ad an ages. In: 2017 In e na ional Con e ence on I-SMAC (IoT in Social, Mobile, Analy ics and Cloud). I-SMAC, IEEE. Pe ei a, Diogo Augus o, Ou ique de Mo ais, Wagne , Pigna on de F ei as, Edison, 2018. NoSQL eal- ime da abase pe o mance compa ison. In . J. Pa allel Eme gen Dis ib. Sys . 33 (2), 144–156. Poljak, R., Poscic, P., Jaksic, D., 2017. Compa a i e analysis o he selec ed ela ional da abase managemen sys ems. In: 2017 40 h In e na ional Con en ion on In o - ma ion and Communica ion Technology, Elec onics and Mic oelec onics. MIPRO, IEEE, Opa ija, C oa ia, pp. 1496–1500. Puangsaijai, Wi awa , Pun hee anu ak, Su hee a, 2017. A compa a i e s udy o e- la ional da abase and key- alue da abase o big da a applica ions. In: 2017 In e na ional Elec ical Enginee ing Cong ess. IEECON, IEEE, Pa aya, Thailand, pp. 1–4. Ra aman anan soa, Fon aine, Laha, Mahe inde o, 2018. Analysis and neu al ne wo ks modeling o web se e pe o mances using MySQL and Pos g eSQL. Commun. Ne wo k 10 (04), 142–151. Rau ma e, Sha a i, Bhale ao, D.M., 2016. MySQL and NoSQL da abase compa ison o IoT applica ion. In: 2016 IEEE In e na ional Con e ence on Ad ances in Compu e Applica ions. ICACA, IEEE, Coimba o e, pp. 235–238. Ribei o, Ja del, Hen ique, Jonas, Ribei o, Rod igo, Ne o, Rosal o, 2017. NoSQL s ela ional da abase: A compa a i e s udy abou he gene a ion o he mos equen N-g ams. In: 2017 4 h In e na ional Con e ence on Sys ems and In o ma ics. ICSAI, IEEE, Hangzhou, pp. 1568–1572. Roopak, K.E., Rao, K.S. Swa i, Ri esh, S., Chicke u , Sa yadhyan, 2013. Pe o mance compa ison o ela ional da abase wi h objec da abase (DB4o). In: 2013 5 h In e - na ional Con e ence on Compu a ional In elligence and Communica ion Ne wo ks. IEEE. Saikia, Amlanjyo i, Joy, She in, Dolma, Dhondup, Ma y R, Roseline, 2015. Compa a i e pe o mance analysis o MySQL and SQL se e ela ional da abase managemen sys ems in windows en i onmen . IJARCCE 160–164. Saman a, Ashis Kuma , Sa ka , Bidu Biman, Chaki, Nabendu, 2018. Que y pe o mance analysis o NoSQL and big da a. In: 2018 Fou h In e na ional Con e ence on Resea ch in Compu a ional In elligence and Communica ion Ne wo ks. ICRCICN, IEEE. Schmid, S ephan, Galicz, Esz e , Reinha d , Wol gang, 2015a. Pe o mance in es iga ion o selec ed SQL and NoSQL da abases. In: P oceedings o he AGILE. pp. 1–5. Seda, Pa el, Hosek, Ji i, Masek, Pa el, Poko ny, Ji i, 2018. Pe o mance es ing o NoSQL and RDBMS o s o ing big da a in e-applica ions. In: 2018 3 d In e na ional Con e ence on In elligen G een Building and Sma G id. IGBSG, IEEE. Sha ma, Monika, Sha ma, Vishal Deep, Bundele, Mahesh M., 2018. Pe o mance analysis o RDBMS and No SQL da abases: Pos g esql, MongoDB and Neo4j. In: 2018 3 d In e na ional Con e ence and Wo kshops on Recen Ad ances and Inno a ions in Enginee ing. ICRAIE, IEEE, Jaipu , India, pp. 1–5. Sholichah, Rahma ian Jayan y, Im ona, Mahmud, Alamsyah, And y, 2020. Pe o mance analysis o Neo4j and MySQL da abases using public policies decision making da a. In: 2020 7 h In e na ional Con e ence on In o ma ion Technology, Compu e , and Elec ical Enginee ing. ICITACEE, IEEE, Sema ang, Indonesia, pp. 152–157. Si ish She y, B., Akshay, Kc, 2019. Pe o mance analysis o que ies in RDBMS s NoSQL. In: 2019 2nd In e na ional Con e ence on In elligen Compu ing, Ins u- men a ion and Con ol Technologies. ICICICT, IEEE, Kannu ,Ke ala, India, pp. 1283–1286. S ancu-Ma a, So in, Baumann, Pe e , 2008. A compa a i e benchma k o la ge objec s in ela ional da abases. In: P oceedings o he 2008 In e na ional Symposium on Da abase Enginee ing & Applica ions - IDEAS ’08. ACM P ess, Coimb a, Po ugal, p. 277. T uica, Cip ian-Oc a ian, Radulescu, Flo in, Boicea, Alexand u, Bucu , Ion, 2015. Pe - o mance e alua ion o CRUD ope a ions in asynch onously eplica ed documen o ien ed da abase. In: 2015 20 h In e na ional Con e ence on Con ol Sys ems and Compu e Science. IEEE, Bucha es , Romania, pp. 191–196. an de Veen, Jan Sipke, an de Waaij, B am, Meije , Robe J., 2012. Senso da a s o age pe o mance: SQL o NoSQL, physical o i ual. In: 2012 IEEE Fi h In e na ional Con e ence on Cloud Compu ing. IEEE. The Jou nal o Sys ems & So wa e 208 (2024) 111872 16 T. Taipalus Vicknai , Chad, Macias, Michael, Zhao, Zhendong, Nan, Xiao ei, Chen, Yixin, Wilkins, Dawn, 2010. A compa ison o a g aph da abase and a ela ional da abase: A da a p o enance pe spec i e. In: P oceedings o he 48 h Annual Sou heas Regional Con e ence on - ACM SE ’10. ACM P ess, Ox o d, Mississippi, p. 1. Wei-ping, Zhu, Ming-xin, Li, Huan, Chen, 2011. Using MongoDB o implemen ex book managemen sys em ins ead o MySQL. In: 2011 IEEE 3 d In e na ional Con e ence on Communica ion So wa e and Ne wo ks. IEEE. Wiseso, Linggis Galih, Im ona, Mahmud, Alamsyah, And y, 2020. Pe o mance analysis o Neo4j, MongoDB, and Pos g eSQL on 2019 na ional elec ion big da a manage- men da abase. In: 2020 6 h In e na ional Con e ence on Science in In o ma ion Technology. ICSITech, IEEE. Xu, Wei, Zhou, Zhonghua, Zhou, Hong, Zhang, Wu, Xie, Jiang, 2014. MongoDB imp o es big da a analysis pe o mance on elec ic heal h eco d sys em. In: Communica ions in Compu e and In o ma ion Science. Sp inge Be lin Heidelbe g, pp. 350–357. Yin eng Wang, Guiquan Zhong, Lin Kun, Longxiang Wang, Huang Kai, Fuliang Guo, Chengzhe Liu, Xiaoshe Dong, 2015. The pe o mance su ey o in memo y da abase. In: 2015 IEEE 21s In e na ional Con e ence on Pa allel and Dis ibu ed Sys ems. ICPADS, IEEE, Melbou ne, VIC, pp. 815–820. Zhou, Zhonghai, Zhou, Bin, Li, Wenwen, G iglak, B ian, Caiseda, Ca men, Huang, Qun- ying, 2009. E alua ing que y pe o mance on objec - ela ional spa ial da abases. In: 2009 2nd IEEE In e na ional Con e ence on Compu e Science and In o ma ion Technology. IEEE.