This is a sel -a chi ed e sion o an o iginal a icle. This e sion
may di e om he o iginal in pagina ion and ypog aphic de ails.
Au ho (s):
Ti le:
Yea :
Ve sion:
Copy igh :
Righ s:
Righ s u l:
Please ci e he o iginal e sion:
CC BY 4.0
h ps://c ea i ecommons.o g/licenses/by/4.0/
NewSQL Da abase Managemen Sys em Compile E o s : E ec i eness and Use ulness
© 2022 The Au ho (s). Published wi h license by Taylo & F ancis G oup, LLC
Published e sion
Taipalus, Toni; G ahn, Hilkka
Taipalus, T., & G ahn, H. (2022). NewSQL Da abase Managemen Sys em Compile E o s :
E ec i eness and Use ulness. In e na ional Jou nal o Human-Compu e In e ac ion, Ea ly
online. h ps://doi.o g/10.1080/10447318.2022.2108648
2022
Full Te ms & Condi ions o access and use can be ound a
h ps://www. and online.com/ac ion/jou nalIn o ma ion?jou nalCode=hihc20
In e na ional Jou nal o Human–Compu e In e ac ion
ISSN: (P in ) (Online) Jou nal homepage: h ps://www. and online.com/loi/hihc20
NewSQL Da abase Managemen Sys em Compile
E o s: E ec i eness and Use ulness
Toni Taipalus & Hilkka G ahn
To ci e his a icle: Toni Taipalus & Hilkka G ahn (2022): NewSQL Da abase Managemen
Sys em Compile E o s: E ec i eness and Use ulness, In e na ional Jou nal o Human–Compu e
In e ac ion, DOI: 10.1080/10447318.2022.2108648
To link o his a icle: h ps://doi.o g/10.1080/10447318.2022.2108648
© 2022 The Au ho (s). Published wi h
license by Taylo & F ancis G oup, LLC
Published online: 15 Aug 2022.
Submi you a icle o his jou nal
View ela ed a icles
View C ossma k da a
NewSQL Da abase Managemen Sys em Compile E o s:
E ec i eness and Use ulness
Toni Taipalus and Hilkka G ahn
Uni e si y o Jy €
askyl€
a, Jy €
askyl€
a, Finland
ABSTRACT
Mode n da abase managemen is o en aced wi h a high numbe o concu en end-use s, and
he need o da abase dis ibu ion o ensu e aul ole ance and high h oughpu . To lexibly
add ess hese challenges, many mode n da abase managemen sys ems (DBMS) p o ide highly
au oma ed and e o less, i.e., highly usable da abase dis ibu ion, deploymen , and main enance.
Howe e , he usabili y conside a ions a e ye o ex end om he a o emen ioned DBMS ea u es
o que y language compile s. In his s udy, based on pa icipan answe s (N¼157), we compa e
he e o message quali ies o ou mode n DBMSs (Cock oachDB, SingleS o e, NuoDB, and Vol DB)
using one objec i e and h ee subjec i e me ics. Ou esul s show ha some o he DBMSs p o-
ide he use s wi h mo e use ul e o messages, e en hough many o hese e o messages io-
la e e en he mos basic usabili y guidelines. These esul s (i) a e applicable in u he de eloping
he usabili y aspec s o que y language compile s, (ii) p o ide a imely e o o b idging he gap
be ween human-compu e in e ac ion and que y language compile s, and (iii) o e sugges ions
on eaching no ices, who equi e emphasized suppo in que y o mula ion.
1. In oduc ion
E o messages a e c ucial o ixing e o s in que ies, ye
e o s a e di icul o ix because o e o messages’poo
usabili y (T a e , 2010). Se e al decades ago, schola s ha e
poin ed ou ha especially no ice use s eel “con used, dis-
mayed, and discou aged om con inuing”when encoun e ing
con using o e en agg essi e sys em messages (Shneide man,
1982). The usabili y aspec s o compile s and e o messages
ha e ecei ed ample scien i ic a en ion (Becke e al., 2019),
bu his a en ion has no been ex ended om p og amming
languages o que y languages. As he que y language is an
in eg al pa o he p ocess o e ie ing da a om a da abase,
i is c ucial ha he que y is w i en wi hou e o s.
Fu he mo e, in o ma ion e ie al om da abases is an
impo an opic in human-compu e in e ac ion (HCI)
esea ch. The inc easingly emphasized ole o da a in in o ma-
ion sys ems has led o he eme gence o nascen sub ields,
such as human–da a in e ac ion (Vic o elli e al., 2020).
A he same ime, he impo ance o da a is inc easingly
highligh ed in apidly g owing ields, such as da a mining
and machine lea ning. Addi ionally, he ise o he highly
compe i i e ma ke o web and mobile applica ions has p es-
su ed echnical da a managemen solu ions o mee demands
o he high numbe o concu en use s, high olume and
eloci y o da a, as well as high eliabili y (Ramak ishnan,
2012). Consequen ly, a la ge po ion o da a managemen
has mo ed o cloud en i onmen s, which enable apid
p o o yping, cos –e iciency, and au oma ed esou ce alloca-
ion on demand (Buyya e al., 2019). Fu he mo e, he in o -
ma ion echnology ield and ela ed skills a e becoming
mo e and mo e common, and basic so wa e de elopmen is
in oduced ea lie and ea lie as well as mo e and mo e
b oadly in o a ious cu icula (L
edeczi e al., 2021; Szabo
e al., 2019). As he ubiqui ousness o he in o ma ion ech-
nology ield is inc easing, expe sys ems, such as DBMSs
need o be accessible o no ices as well as expe s (Nicolaos
& Ka e ina, 2015; Sobiesiak e al., 2002). As such, many en-
do s behind mode n, dis ibu ed da abase managemen sys-
ems (DBMS) ha e made DBMS deploymen and da abase
dis ibu ion lexible, au oma ed, and e o less o so wa e
de elope s (Hacigumus e al., 2002).
Gi en hese conside a ions, i emains unclea whe he
usabili y ex ends om ea u es, such as au oma ed and lex-
ible da abase dis ibu ion o o he aspec s o DBMSs. To
his end, we se ou o compa e NewSQL da abase manage-
men sys ems om a scien i ically neglec ed poin o iew o
que y language compile usabili y. Speci ically, we compa e
16 e ie al que y syn ax e o messages o Cock oachDB,
SingleS o e, NuoDB, and Vol DB using e o ixing success
a e, e o eco e y con idence, and pe cei ed use ulness o
he e o message o inding and ixing he e o as pe -
o mance me ics. Ou esul s e eal mode n DBMSs wi h
many compile e o s designed agains 40 yea old HCI bes
p ac ices, as well as s a is ically signi ican di e ences in
CONTACT Toni Taipalus [email p o ec ed] Uni e si y o Jy €
askyl€
a, Jy €
askyl€
a, Finland
ß2022 The Au ho (s). Published wi h license by Taylo & F ancis G oup, LLC
This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/4.0/), which pe mi s un es ic ed use,
dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed.
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION
h ps://doi.o g/10.1080/10447318.2022.2108648
e o message use ulness be ween hese ou mode n DBMS
compile e o s.
The es o his s udy is s uc u ed as ollows. In he nex
sec ion, we discuss he heo e ical backg ound and p io
s udies on da a managemen , usabili y, and que y languages.
In Sec ion 3 we desc ibe ou esea ch se ing and da a col-
lec ion, and s a e ou hypo heses. In Sec ion 4 we p esen
ou esul s om s a is ical analyses and in Sec ion 5 he
implica ions o ou esea ch, and some ecommenda ions
o he DBMS indus y. Sec ion 6 concludes ou s udy.
2. Theo e ical backg ound
2.1. Da a managemen in he cloud
Cloud compu ing is a g owing echnology model buil a ound
p o iding a high-le el abs ac ion o dis ibu ed compu ing,
usually o e ed as a subsc ip ion-based se ice o he end-use
(Abbasi e al., 2019; Buyya e al., 2019). E ec i ely, he end-
use pays o esou ces hey u ilize (e.g., s o age space, com-
pu a ion, and ne wo k bandwid h), a he han in es ing in
ha dwa e, so wa e and in as uc u e ou igh . Consequen ly,
some o he easons o he popula i y o he cloud compu ing
model a e he speed o deploymen , scalabili y o compu ing
esou ces, and cos -e iciency (Buyya e al., 2019). Depending
on he cloud se ice p o ide , di e en se ice models a e
o e ed. These se ice models ypically dic a e which pa s o
he sys em a e p o ided and main ained by he se ice p o-
ide , and which pa s by he end-use . Fo example, he se -
ice p o ide may me ely p o ide he in as uc u e and
(o en i ual) ha dwa e, he a o emen ioned complemen ed
by an ope a ing sys em, o all he a o emen ioned comple-
men ed by a da abase managemen sys em (Somu e al.,
2017). Depending on hei equi emen s, he end-use may
choose a high le el o abs ac ion while disca ding con ol
o e low-le el con igu a ions. In con as , by choosing a low
le el o abs ac ion, he end-use e ains con ol—and espon-
sibili y—o low-le el asks, such as main aining he ope a -
ing sys em.
The ole o he ela ional da a model, SQL, and ad-
i ional ela ional DBMSs (RDBMS), such as O acle
Da abase, IBM DB/2, and Mic oso SQL Se e has been
challenged in he 2000s by new da a models and que y lan-
guages o nume ous NoSQL da a s o es (G olinge e al.,
2013). While RDBMSs ha e a o ed da a consis ency a he
cos o a ailabili y and ansac ion pe o mance (Chaudh y
& Yousa , 2020; Pa lo & Asle , 2016), many NoSQL da a
s o es ha e been designed he o he way a ound o se e,
e.g., web applica ions wi h equi emen s o low esponse
ime and a high numbe o concu en end-use s
(Ramak ishnan, 2012). In he 2010s, howe e , he indus y
leade s, such as Google deemed ansac ion suppo , da a
consis ency, and he SQL language impo an enough o
design a new DBMS o inco po a e ea u es om bo h ad-
i ional RDBMSs and NoSQL da a s o es (Co be e al.,
2013). In gene al, mode n (i.e., in his case ini ially eleased
a e 2010) online ansac ion p ocessing DBMSs ha use
he ela ional model, SQL, and dis ibu ed a chi ec u e a e
called NewSQL DBMSs. A ecen s udy (Pa lo & Asle ,
2016) u he de ines NewSQL DBMSs as sys ems buil om
he g ound up, a he han ex ensions o modi ica ions o
exis ing sys ems. The s udy concludes ha while NewSQL
sys ems do no o e new ea u es o inno a ions pe se,
hey skill ully in eg a e es ed echniques in o single sys ems.
Tha is, “NewSQL da abase sys ems a e no a adical depa -
u e om exis ing sys em a chi ec u es bu a he ep esen he
nex chap e in he con inuous de elopmen o da abase ech-
nologies”(Pa lo & Asle , 2016, p. 53). When choosing a se
o NewSQL DBMSs o his s udy, we adop he de ini ion
o NewSQL sys ems p o ided abo e (Pa lo & Asle , 2016).
The dis ibu ed a chi ec u es o NoSQL and NewSQL
sys ems a e a na u al i o cloud en i onmen s
(G olinge e al., 2013). While adi ional RDBMSs also
suppo da abase dis ibu ion and a e o e ed by cloud
se ice p o ide s, some s udies conside adi ional
RDBMS dis ibu ion di icul o a ious easons (Pa lo &
Asle , 2016;S oneb ake ,2010). In p ac ice, he new dis-
ibu ion implemen a ions p o ide au oma ic dis ibu ion
o da a, au oma ed da a balancing be ween he dis ibu ed
nodes, and wi h he e ogeneous dis ibu ion models, au o-
ma ed p ima y/seconda y elec ions du ing aul s o o he
opology modi ica ions.
2.2. Que y language usabili y
Acco ding o he e gonomics o human-sys em in e ac ion
s anda d, “Usabili y is ele an o egula ongoing use, o
enable use s o achie e hei goals e ec i ely, e icien ly and
wi h sa is ac ion; lea ning, o enable new use s o be become
e ec i e, e icien and sa is ied when s a ing o use a sys em,
p oduc o se ice”(ISO, 2018).
Usabili y is a ecu ing heme in he e olu ion o cloud
da a managemen , and one o he main easons behind
new da abase dis ibu ion implemen a ions was oo ed in
usabili y conside a ions (Shi e al., 2010; S oneb ake ,
2010). Fi s , a guably, in addi ion o pe o mance and lex-
ible scalabili y, he need o dynamic da abase schemas is
one o he de ining cha ac e is ics o many NoSQL da a
models. Dynamic schemas absol e he so wa e de elope
om de ining a s ic da abase s uc u e. Second, he need
o e u n o s ong ansac ional capabili ies wi h NewSQL
sys ems can be seen as a need o abs ac he implemen a-
ion o da abase ansac ions om he so wa e de elope
o he DBMS. Finally, he abs ac ion o compu e in a-
s uc u e, ha dwa e, and pa ial so wa e h ough cloud
se ices all se e he demand o usabili y h ough cos -
e iciency, lexibili y, and apid p o o yping. On logical
g ounds, i seems in e es ing whe he he demands o
usabili y a e also conside ed in o he aspec s o cloud da a-
base managemen sys ems, e.g., in compile e o messages,
as usabili y, in gene al, has been a gued o acili a e cos -
e iciency h ough, e.g., imp o ed p oduc i i y, educed
aining, and documen a ion cos s, lowe suppo cos s,
and compe i i e edge (Donahue, 2001).
Da a managemen solu ions in cloud en i onmen s u ilize
se e al que y languages and da a models. T adi ional RDBMSs
and NewSQL sys ems u ilize an implemen a ion o SQL, while
2 T. TAIPALUS AND H. GRAHN
NoSQL sys ems each usually ha e a dis inc que y language,
e.g., Neo4j’s Cyphe (F ancis e al., 2018), o Cassand a’sSQL-
based CQL (Wang & Tang, 2012). These p op ie a y que y lan-
guages a e some imes complemen ed by SQL. As hese NoSQL
languages a e designed o di e en da a models and ha e di -
e en le els o exp essi eness, usabili y compa ison o di e en
que y languages is a guably p oblema ic. Fu he , as some ad-
i ional RDBMSs o e implemen a ions da ing ac oss ou o
i e decades, we deemed i mo e in e es ing o ocus on he
usabili y o sys ems de eloped om he g ound up in he las
decade. As a con as ing example, O acle Da abase 8i documen-
a ion om 1998 lis ed he same SQL e o messages as O acle
Da abase 21c om 2021 (O acle Co po a ion, 2021).
SQL is a language ini ially designed o da a e ie al.
Howe e , in he decades ollowing he ini ial elease o he
SQL s anda d, he language has e ol ed o encompass da a
manipula ion, da abase s uc u e de ini ion, access con ol,
and ansac ion managemen (Chambe lin, 2012). Da a
e ie al emains he mos s udied aspec o SQL (Taipalus
& Sepp€
anen, 2020), and because o his mo e es ablished
esea ch backg ound, his s udy ocuses solely on
da a e ie al.
Possibly due o he inc easingly ubiqui ous na u e o da a,
and he ising popula i y o da a analy ics and da a science,
que y languages, SQL in pa icula , ha e ecei ed inc easing
schola ly a en ion (Taipalus & Sepp€
anen, 2020). Cu en
educa ional esea ch seems a he unanimous wi h he iew
ha lea ning SQL is di icul (Miedema e al., 2021; Shin,
2020; Taipalus & Pe €
al€
a, 2019). Usabili y conce ns in que y
o mula ion ha e been explained by human ac o s, such as
cogni i e load (Shin, 2020; Smelce , 1995), da a model and
eal-wo ld misma ch (Bo hick e al., 2001; Su cli e e al.,
2000), and di e en use cha ac e is ics (Ashkanasy e al.,
2007; Bak & Meye , 2011). Addi ionally, i has been shown
ha di e en en i onmen al aspec s, such as da abase com-
plexi y (Taipalus, 2020a) and da abase ep esen a ion (Shin,
2020; Siau e al., 2004) ha e an e ec on que y w i ing.
Finally, di e en measu es o engaging and helping he end-
use ha e been p oposed in scien i ic li e a u e, e.g., que y
isualiza ion and p e iews (Taipalus, 2019; Tanin e al.,
2000), cosme ic al e a ions (Dong & Khandwala, 2019), di -
e en na u al language in e aces (Ribei o & Mo ei a, 2003),
and he acili a ion o que y euse (Allen & Pa sons, 2010;
Too n e al., 2022). Howe e , e o message esea ch has no
ex ended om p og amming languages o que y languages,
and he la es s udies on he e ec s o SQL compile e o
messages on que y o mula ion seem o be published in he
1980s (Reisne , 1981; Wel y & S emple, 1981), un il a ecen
compa ison o SQL compile s o adi ional RDBMS in 2021
(Taipalus e al., 2021). The di e ences in he SQL s anda d
(ISO/IEC, 2016a,2016b) be ween he 1980s and 2020s, as
well as di e ences be ween SQL and impe a i e p og am-
ming languages, and he po en ial h ea s o he gene aliz-
abili y o scien i ic esul s induced he eo ha e been
highligh ed in a p e ious s udy (Taipalus & Sepp€
anen, 2020).
Rega ding usabili y, due o i s decla a i e na u e, SQL is
a guably a “blacke box” o a so wa e de elope han an
impe a i e p og amming language.
2.3. E o messages and e o eco e y
A la ge numbe o s udies ha e shown he impo ance o com-
pile e o messages o lea ning, and o so wa e de elopmen
in gene al in he con ex o p og amming languages (Becke
e al., 2016,2019; W enn & K ishnamu hi, 2017). The same
s udies ha e also a gued ha cu en compile e o messages
a e ine ec i e due o se e al easons. F om he pe spec i e o
e o messages, DBMSs ha e a que y pa se ha checks he
syn ax o he que y and ou pu s an e o message i necessa y
(Helle s ein e al., 2007). In he scope o his s udy, he que y
pa se is he componen ha sepa a es he usabili y aspec s o
di e en DBMSs om each o he . Addi ionally, some DBMSs,
such as MySQL allows pluggable s o age engines ha can be
swi ched wi h ela i e ease. The s o age engine ypically con-
ains he que y pa se , and hus he s o age engine is o en
esponsible o gene a ing he SQL e o messages. I is wo h
no ing ha while SingleS o e is a NewSQL sys em, i u ilizes
he InnoDB s o age engine also u ilized by MySQL.
When an end-use , e.g., a so wa e de elope , w i es an e o-
neous que y and submi s i o a DBMS, he DBMS ou pu s an
e o message. This is o en e e ed o as e o de ec ion ( an
de Schaa , 1995; Zap & Reason, 1994). Nex , he end–use
ies o in e p e he e o message and ind he e oneous pa
o he que y. This phase is called explaining. Finally, he end-
use a emp s o ix he e o , ypically based on he eedback
p o ided by he e o message. This p ocess o h ee phases is
called e o eco e y ( an de Schaa , 1995; Zap & Reason,
1994), and se es as a heo e ical ounda ion o ou chosen
subjec i e me ics, i.e., e o eco e y con idence,ande o mes-
sage use ulness o inding and ixing he e o .
A seminal s udy published in 1982 sugges s ha compu e
e o messages should be “b ie , posi i e, cons uc i e, speci ic,
comp ehensible”(Shneide man, 1982,p.611),posi i e e e ing
o e aining om using wo ds, such as “illegal, in alid, e o ”
in he e o message, and cons uc i e e e ing o hin s o sug-
ges ions on he causes o he e o and how o ix i .
Conside ing ha he WHERE clause is one o he mos common
SQL clauses, Figu e 1 shows an SQL que y wi h a simple ypo-
g aphical e o in he keywo d WHERE, and se en co espond-
ing e o messages om adi ional RDBMSs and NewSQL
sys ems. As so wa e de elope s, especially no ices, o en con-
side he compile he i s au ho i y in de e mining he quali y
o w i en so wa e, he e o messages in Figu e 1 succeed in
nei he communica ing why he que y is e oneous no adhe -
ing o all he sugges ions p esen ed 40 yea s ago.
3. Resea ch se ing
3.1. S udy scope
In he p e ious sec ion, we discussed he impo ance o
e ec i e e o messages in he con ex o p og amming lan-
guages, and ha p io wo ks ha e a emp ed o explain and
enhance said e o messages o acili a e mo e e ec i e so -
wa e de elopmen . We also a gued o he usabili y conce n,
which seems o be one o he d i ing ac o s behind he
popula i y o cloud en i onmen s and mo e e o less da a-
base dis ibu ion. Howe e , in ligh o p e ious scien i ic
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 3

li e a u e, and a p elimina y inspec ion o DBMS e o mes-
sages (Figu e 1), i seems ha e en SQL compile s o mode n
DBMSs do no necessa ily accoun o usabili y conce ns o
e o message design guidelines and ha he opic has no
ecei ed much scien i ic a en ion in ecen decades.
As explained in Sec ion 2, we deemed compa ing DBMSs
u ilizing SQL wi h DBMSs u ilizing some o he que y lan-
guage di icul o in e nal alidi y. On he o he hand, a
ecen s udy (Taipalus e al., 2021) compa ed SQL compile
usabili y o adi ional RDBMS. Fo hese easons, in his
s udy, we chose o ocus on NewSQL sys ems using he SQL
compile usabili y amewo k epo ed in a p e ious s udy
(Taipalus e al., 2021). We deemed i mo e in e es ing o
ocus on popula NewSQL sys ems, e en hough measu ing
popula i y is a he di icul . Based on h ee NewSQL s udies
(Kau & Sachde a, 2017; Pa lo & Asle , 2016; Sch eine
e al., 2019), we iden i ied ou popula NewSQL da abase
managemen sys ems o his s udy: Cock oachDB ( 19.2.2),
SingleS o e (7.0.10, p e iously known as MemSQL), NuoDB
(build 4.0.4-2), and Vol DB (Communi y 9.2.2). All hese
sys ems implemen ela ional o semi- ela ional da a models,
use SQL as hei que y language, and a e buil om he
g ound up in he 2010s (G olinge e al., 2013).
Addi ionally, DB-Engines
1
anks hese ou DBMSs high in
popula i y among NewSQL sys ems, when NewSQL sys ems
a e de ined as in Sec ion 2.1. In ega d o di e en ypes o
e o s, we ocus on syn ax e o s, and based on a p e iously
epo ed amewo k (Taipalus e al., 2018), we ocus on he
16 mos common syn ax e o s in SQL que ies. These p e i-
ously epo ed syn ax e o s and ou co esponding es s a e
epo ed in Table 1. These es s and que ies wi hin a e in
u n based on hose epo ed in a p e ious s udy (Taipalus
e al., 2021), bu adjus ed o accoun o he chosen ou
NewSQL sys ems. In he nex subsec ions, we de ail he da a
collec ion, hypo heses, and analyses, which a e summa ized
in Figu e 2.
3.2. Da a collec ion
To ocus on he di e ences in he selec ed DBMS usabili y
in ixing e oneous que ies, we chose no o use da abase
expe s as pa icipan s. We specula ed ha expe s migh
SELECT name, p ice_usd , b and, model
FROM p oduc
WHRE (b and LIKE S% OR b and LIKE
C% )
AND pic u e IS NULL
ORDER BY name DESC;
(a) Que y wi h a ypog aphical e o (WHRE
ins ead o WHERE)
Msg 321, Le el 15, S a e 1, Se e
q7410, Line 4
"b and" is no a ecognized able hin s
op ion.
(b) SQL Se e e o message
ERROR : syn ax e o a o nea "LIKE"
LINE 3: WHRE (b and LIKE S% OR b and
LIKE C% )
ˆ
(c) Pos g eSQL e o message
ORA-00933: SQL command no p ope ly
ended
(d) O acle Da abase e o message
in alid syn ax: s a emen igno ed: a
o nea
"like": syn ax e o
DETAIL: sou ce SQL:
SELECT name, p ice_usd , b and, model
FROM p oduc
WHRE (b and LIKE S% OR b and LIKE
C% )
ˆ
HINT: y h <SOURCE>
(e) Cock oachDB e o message
SQL e o while compiling que y: SQL
Syn ax e o in
"SELECT name, p ice_usd , b and, model
FROM p oduc
WHRE (b and LIKE S% OR b and LIKE
C% )
AND pic u e IS NULL
ORDER BY name DESC;"
unexpec ed oken: LIKE equi ed: )
( ) Vol DB e o message
ERROR 1064 ER_PARSE_ERROR: You ha e an
e o in you SQL syn ax; check he
manual ha co esponds o you
MySQL se e e sion o he igh
syn ax o use nea
(b and LIKE S% OR b and LIKE C% )
AND pic u e IS NULL
ORDER BY name DE a line 3
(g) SingleS o e (wi h InnoDB s o age engine)
e o message
E o 42000: syn ax e o on line 3
WHRE (b and LIKE S% OR b and LIKE C
% )
ˆ expec ed end o s a emen go
pa en hesis
(h) NuoDB e o message
Figu e 1. E oneous que y wi h a simple ypog aphical e o (a), and se en co esponding e o messages gene a ed by se en di e en DBMSs; h ee adi ional
RDBMSs (b–d), and ou NewSQL DBMSs (e–h).
4 T. TAIPALUS AND H. GRAHN
ha e o me expe ience on one o se e al o he DBMSs
s udied and ha hei expe ise would esul in success ul
e o ixing ega dless o he e o message, hus skewing
he esul s owa d a ceiling e ec (i.e., esul s a e no s a is-
ically signi ican ly di e en because es s we e oo easy o
selec ed pa icipan s). Fu he mo e, expe s a e a guably less
dependen on he e o messages, and mo e able o ix e o-
neous que ies ega dless o he e o message. Because we
wan ed o speci ically s udy he e ec s o di e en compile
e o messages, we ec ui ed ou s udy pa icipan s om a
da abase managemen cou se gi en a he au ho s’uni e -
si y. The pa icipan s majo ed in so wa e enginee ing o
in o ma ion sys ems science and had acqui ed basic SQL
knowledge om he cou se. The s uden s we e p omised
ex a cou se poin s o aking he su ey. Taking he su ey
was no manda o y, and i a s uden also chose o do so,
hei answe s we e anonymized and used in his s udy.
Pa icipa ing in he s udy was no equi ed o ex a cou se
poin s, and he s uden s we e shown a ull p i acy s a emen
be o e answe ing. Ou o he 188 s uden s who answe ed he
su ey, 157 (84%) chose o pa icipa e in he s udy.
Nex , a pa icipan was andomly assigned o one o he
ou da abase managemen sys em g oups—i.e., Cock oachDB
(n¼32), NuoDB (n¼44), SingleS o e (n¼39), and Vol DB
(n¼44)—and shown a se o 20 es s, one es a he ime.
The i s ou es s we e con ol ques ions measu ing pa ici-
pan skill in e o ixing, and hese ou es s we e he same
o all pa icipan s, ega dless o he g oup he pa icipan was
assigned o. Nex , he es sui e o 16 es s (c . es s T01–T16
in Table 1) was shown, es by es , and in a andomized o de
o each pa icipan . Each o he 16 es s consis ed o a
da abase s uc u e diag am, a da a demand, an e oneous SQL
que y, an e o message gene a ed by he DBMS, a ee ex
inpu box in which he pa icipan was ins uc ed o w i e he
ixed que y, and a se o i e-poin Like scale (1 ¼s ongly
disag ee, 5 ¼s ongly ag ee) ques ions pe aining o subjec i e
indica o s o he usabili y quali ies o he e o message (c .
hypo heses H
2
,H
3
and H
4
in he nex sec ion). Depending on
he g oup he pa icipan was assigned o, hey we e shown
co esponding e o messages, e.g., o pa icipan s assigned
o Vol DB g oup, Vol DB gene a ed e o messages we e
shown. Answe ing could be paused o s opped al oge he , ye
none o he pa icipan s chose o do so. The pa icipan s could
use any ma e ials o suppo du ing he es s. Fo mo e
de ails on he es s, e o messages, da abase s uc u e, and
ques ions, please e e o he supplemen a y Appendices.
A e all he pa icipan s had answe ed he es s, he i s
au ho coded he que ies submi ed by he pa icipan s as co -
ec o inco ec . A que y ha con ained a leas one syn ax
e o was conside ed inco ec .
3.3. Hypo heses
To s udy usabili y conside a ions o ou NewSQL da abase
managemen sys ems, we o mula ed wo se s o hypo heses.
Hypo heses H
1
–H
4
compa e objec i e que y ixing success
a es, as well as subjec i e e o eco e y con idence, use ul-
ness o e o inding, and use ulness o e o ixing wi h a
be ween-subjec s s udy design. Hypo heses H
5
–H
7
es co -
ela ion o e o message quali ies ega dless o he da abase
managemen sys em g oup. We chose o es he pa icula
co ela ions be ween he objec i ely measu ed a iable (i.e.,
success a e) and he subjec i ely measu ed a iables (i.e.,
pe cei ed use ulness o inding and ixing he e o , and
e o eco e y con idence) because o he na u e o how
hese a iables we e measu ed. In o he wo ds, we did no
es co ela ions be ween subjec i ely measu ed a iables.
H
1
: he medians o que y o mula ion success a es a e di -
e en o he da abase managemen sys em g oups.
H
2
: he medians o e o eco e y con idence a e di e en
o he da abase managemen sys em g oups.
H
3
: he medians o pe cei ed use ulness o e o inding
a e di e en o he da abase managemen sys em g oups.
H
4
: he medians o pe cei ed use ulness o e o ixing a e
di e en o he da abase managemen sys em g oups.
Rec ui
pa icipan s
Assign
pa icipan s o
DBMS g oups
Fix e oneous
con ol ques ion
que ies
Fix e oneous
que ies, answe
Like ques ions
Omi ou lie s
based on con ol
ques ions
Compa e he
DBMS g oups
(H1-H
4)
Check o
co ela ions
(H5 -H
7)
Figu e 2. O e iew o he da a collec ion and analysis p ocess; whi e ec angles e e o ac ions pe o med by us and g ey ec angles o ac ions pe o med by he
s udy pa icipan s.
Table 1. Tes sui e consis s o 16 mos common syn ax e o s (Taipalus
e al., 2018).
Tes Syn ax e o name
T01 Ambiguous column
T02 Omi ing quo es a ound cha ac e da a
T03 IS whe e no applicable
T04 Con using he syn ax o keywo ds
T05 Con using he logic o keywo ds
T06 Too many columns in subque y
T07 Unde ined column
T08 Misspellings
T09 Failu e o speci y column name wice
T10 Using an agg ega e unc ion ou side SELECT o HAVING
T11 G ouping e o : ex aneous g ouping column
T12 Non-s anda d ope a o s
T13 Using WHERE wice
T14 Non-s anda d keywo ds o s anda d keywo ds in w ong con ex
T15 Synonyms
T16 Cu ly, squa e, o unma ched b acke s
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 5
H
5
:q6¼ 0; he co ela ion coe icien be ween que y o mu-
la ion success a e and e o eco e y con idence is no
equal o ze o.
H
6
:q6¼ 0; he co ela ion coe icien be ween que y o mu-
la ion success a e and pe cei ed use ulness o inding he
e o is no equal o ze o.
H
7
:q6¼ 0; he co ela ion coe icien be ween que y o mu-
la ion success a e and pe cei ed use ulness o ixing he
e o is no equal o ze o.
3.4. Da a p epa a ion and mi iga ion o
con ol a iables
Due o andom pa icipan assignmen , i is possible ha pa -
icipan s wi h highe (o lowe ) que ying skills we e assigned
o he same g oup. This assignmen p esen s a h ea o
in e nal alidi y, po en ially skewing he esul s ega dless o
he quali ies o he dependen a iable (i.e., he e o mes-
sages). To mi iga e he e ec o imbalance in pa icipan
assignmen , we included ou con ol ques ions in he su ey.
Based on he con ol ques ions, we omi ed ou lie s om u -
he be ween-subjec s analyses based on que y ixing success
a e in he con ol ques ions. A e he ou lie s we e emo ed,
and because he da a we e no no mally dis ibu ed, we an a
K uskal-Wallis H es o de e mine i he e we e di e ences in
con ol ques ion sco es be ween he ou g oups o pa icipan s
using di e en da abase managemen sys ems: Cock oadDB
(n¼25), NuoDB (n¼44), SingleS o e (n¼28), and Vol DB
(n¼44). Dis ibu ions o con ol ques ion sco es we e simila
o all g oups, as assessed by isual inspec ion o a boxplo .
The e we e no signi ican di e ences in he medians o con ol
ques ion sco es be ween g oups, H(3) ¼4.987, p¼.173.
Hence, we conside ed he g oups equal in e ms o que y ixing
skills. Fo hypo heses H
5
–H
7
, which we e no conce ned wi h
be ween–subjec s compa ison, we analyzed all da a (N¼157).
SingleS o e SQL compile ole a ed syn ax e o s in es s
T09 and T11. The lack o an e o message in hese wo
es s was compensa ed in he ques ionnai es by made up
e o messages. The es esul s o es s T09 and T11 o
SingleS o e we e omi ed om he s a is ical analyses.
4. Resul s
4.1. A summa y o esul s
In he ollowing sec ions, we p esen he analyses in mo e
de ail, i.e., sys em pe sys em, and desc ibe he chosen
s a is ical es s. A signi icance le el o a¼.05 was chosen o
all he s a is ical es s. A summa y o esul s p esen ed in
Table 2 shows ha hypo heses H
3
–H
7
we e suppo ed, and
hypo heses H
1
and H
2
we e no suppo ed. Please e e o
Figu e 3 o an o e look o he DBMS compa ison.
4.2. Da abase managemen sys em g oup di e ences
Fo each o he hypo heses H
1
,H
2
,H
3
, and H
4
, we an a
K uskal-Wallis H es o de e mine i he e we e di e ences
in e o message e ec i eness (measu ed in e o ixing suc-
cess a es, H
1
), e o eco e y con idence (H
2
), and pe -
cei ed use ulness o he e o message in e ms o inding
(H
3
) and ixing (H
4
) he e o be ween ou g oups o pa -
icipan s wi h di e en da abase managemen sys ems:
Cock oachDB (n¼25), SingleS o e (n¼28), NuoDB
(n¼44), and Vol DB (n¼44). Dis ibu ions o he answe s
o all hypo heses we e simila o all g oups, as assessed by
isual inspec ion o a boxplo . Subsequen ly, pai wise com-
pa isons we e pe o med using Dunn’s(1964) p ocedu e
wi h a Bon e oni co ec ion o mul iple compa isons.
Adjus ed p- alues a e p esen ed in Table 3, and he esul s
a e isualized in Figu e 3.
4.3. Co ela ions
Fo each o he hypo heses H
5
,H
6
and H
7
, we an a ank
bise ial co ela ion o assess he ela ionship be ween e o
message e ec i eness (measu ed in success a e, H
5
) and
e o eco e y con idence; be ween que y o mula ion suc-
cess a e and pe cei ed use ulness o inding he e o (H
6
);
and be ween que y o mula ion success a e and pe cei ed
use ulness o ixing he e o (H
7
)(N¼157). Fo all h ee
hypo heses, and o indi idual da abase managemen sys-
ems, he esul s we e all s a is ically signi ican wi h a weak
posi i e co ela ion. The es s a is ics a e p esen ed in
Table 4.
5. Discussion
5.1. Implica ions o esea ch
The esul s show no s a is ically signi ican di e ences in
e o message e ec i eness be ween he DBMSs (hypo hesis
H
1
). Al hough his obse a ion implies ha none o he
DBMSs s udied has mo e e ec i e e o messages han
ano he , i is wo h no ing ha success a e may be consid-
e ed as one me ic o e ec i eness, a he han he sole
me ic. Fo example, in he con ex o p og amming
Table 2. Summa y o esul s.
Hypo hesis Sho desc ip ion Suppo ed Tes s a is ic E ec size
H
1
Di e en e ec i eness No H(3) ¼5.254, p¼.154
H
2
Di e en eco e y con idence No H(3) ¼0.157, p¼.984
H
3
Di e en use ulness o e o inding Yes H(3) ¼24.396, p<.001 g
2
¼.174
H
4
Di e en use ulness o e o ixing Yes H(3) ¼9.870, p¼.020 g
2
¼.071
H
5
E ec i eness () eco e y con idence Yes
b
(2486) ¼.283, p<.001
H
6
E ec i eness () e o inding Yes
b
(2486) ¼.238, p<.001
H
7
E ec i eness () e o ixing Yes
b
(2486) ¼.215, p<.001
6 T. TAIPALUS AND H. GRAHN
language compile e o messages, i has been sugges ed ha
he messages a ec e o eco e y ime a he han success
(Ahmed e al., 2019).
The e we e no s a is ically signi ican di e ences in e o
eco e y con idence be ween he DBMSs (hypo hesis H
2
).
This sugges s ha e o messages, al hough di e en , do no
necessa ily a ec no ice con idence in e o eco e y.
A guably, some e o messages highligh he e oneous pa
o he que y, ye ail o iden i y why he que y con ains an
e o , o may e en p o ide alse in o ma ion on why he
que y is e oneous (Figu e 1). Based on he esul s, i
emains unclea why he e we e no signi ican di e ences in
e o eco e y con idence. Simila ly, he esul s yielded by
his s udy suppo he no ion ha SQL e o eco e y con i-
dence may ha e a simila ela ionship wi h e o message
e ec i eness (hypo hesis H
5
) as con idence mo e gene ally
has wi h success (Fleming e al., 2010; Ma ino e al., 2013).
A a he unde whelming esul o a weak posi i e co el-
a ion be ween success a e and e o eco e y con idence
may indica e ha i migh be unexpec edly common ha
ei he a pa icipan was con iden in hei ixed que y, ye
he que y was inco ec , o ha a pa icipan was unsu e
Table 3. Tes s a is ics o hypo heses H
1
–H
4
;pos -hoc analyses we e pe o med only i he K uskal Wallis H es was s a is ically signi ican ; DBMS names ha e
been abb e ia ed as (Si)ngleS o e, (Co)ck oachDB, (Nu)oDB, and (Vo)l DB.
Mdn Pai wise compa ison (p- alue)
Co Si Nu Vo Co-Si Co-Nu Co-Vo Si-Nu Si-Vo Nu-Vo
E ec i eness .875 .875 .813 .813
Reco e y con idence 3.94 3.78 3.75 3.81
E o inding 4.13 3.56 4.16 3.68 .004 1 .023 <.001 1 .003
E o ixing 3.63 3.36 3.78 3.50 .256 1 .665 .048 1 .140
(a) E o message effec i eness, measu ed
in success a es
(b) Pe cei ed confidence in e o eco e y
(
c
)
Pe cei ed use ulness o e o finding
(
d
)
Pe cei ed use ulness o e o fixing
Figu e 3. Be ween–subjec s compa ison o 16 es s ega ding e o message e ec i eness, e o eco e y con idence, and e o message use ulness o e o inding
and ixing— he boxplo s ep esen in e qua ile ange, and whiske s minimum and maximum alues, excluding ou lie s. (a) E o message e ec i eness, measu ed
in success a es, (b) pe cei ed con idence in e o eco e y, (c) pe cei ed use ulness o e o inding, and (d) pe cei ed use ulness o e o ixing.
Table 4. Tes s a is ics o hypo heses H
5
–H
7
; co ela ions be ween e o message e ec i eness and e o eco e y con idence
( .c.), and pe cei ed e o message use ulness o inding and ixing he e o .
E ec i eness () .c. E ec i eness () inding E ec i eness () ixing
Cock oachDB
b
(510) ¼.315, p<.001
b
(510) ¼.306, p<.001
b
(510) ¼.316, p<.001
SingleS o e
b
(566) ¼.336, p<.001
b
(566) ¼.240, p<.001
b
(566) ¼.258, p<.001
NuoDB
b
(702) ¼.260, p<.001
b
(702) ¼.174, p<.001
b
(702) ¼.198, p<.001
Vol DB
b
(702) ¼.239, p<.001
b
(702) ¼.178, p<.001
b
(702) ¼.208, p<.001
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 7