NewSQL Database Management System Compiler Errors : Effectiveness and Usefulness

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ NewSQL Database Management System Compiler Errors : Effectiveness and Usefulness © 2022 The Author(s). Published with license by Taylor & Francis Group, LLC Published version Taipalus, Toni; Grahn, Hilkka Taipalus, T., & Grahn, H. (2022). NewSQL Database Management System Compiler Errors : Effectiveness and Usefulness. International Journal of Human-Computer Interaction, Early online. https://doi.org/10.1080/10447318.2022.2108648 2022 Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=hihc20 International Journal of Human–Computer Interaction ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/hihc20 NewSQL Database Management System Compiler Errors: Effectiveness and Usefulness Toni Taipalus & Hilkka Grahn To cite this article: Toni Taipalus & Hilkka Grahn (2022): NewSQL Database Management System Compiler Errors: Effectiveness and Usefulness, International Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2022.2108648 To link to this article: https://doi.org/10.1080/10447318.2022.2108648 © 2022 The Author(s). Published with license by Taylor & Francis Group, LLC Published online: 15 Aug 2022. Submit your article to this journal View related articles View Crossmark data NewSQL Database Management System Compiler Errors: Effectiveness and Usefulness Toni Taipalus and Hilkka Grahn University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland ABSTRACT Modern database management is often faced with a high number of concurrent end-users, and the need for database distribution to ensure fault tolerance and high throughput. To flexibly address these challenges, many modern database management systems (DBMS) provide highly automated and effortless, i.e., highly usable database distribution, deployment, and maintenance. However, the usability considerations are yet to extend from the aforementioned DBMS features to query language compilers. In this study, based on participant answers (N¼157), we compare the error message qualities of four modern DBMSs (CockroachDB, SingleStore, NuoDB, and VoltDB) using one objective and three subjective metrics. Our results show that some of the DBMSs provide the users with more useful error messages, even though many of these error messages violate even the most basic usability guidelines. These results (i) are applicable in further developing the usability aspects of query language compilers, (ii) provide a timely effort of bridging the gap between human-computer interaction and query language compilers, and (iii) offer suggestions on teaching novices, who require emphasized support in query formulation. 1. Introduction Error messages are crucial for fixing errors in queries, yet errors are difficult to fix because of error messages’poor usability (Traver, 2010). Several decades ago, scholars have pointed out that especially novice users feel “confused, dismayed, and discouraged from continuing”when encountering confusing or even aggressive system messages (Shneiderman, 1982). The usability aspects of compilers and error messages have received ample scientific attention (Becker et al., 2019), but this attention has not been extended from programming languages to query languages. As the query language is an integral part of the process of retrieving data from a database, it is crucial that the query is written without errors. Furthermore, information retrieval from databases is an important topic in human-computer interaction (HCI) research. The increasingly emphasized role of data in information systems has led to the emergence of nascent subfields, such as human–data interaction (Victorelli et al., 2020). At the same time, the importance of data is increasingly highlighted in rapidly growing fields, such as data mining and machine learning. Additionally, the rise of the highly competitive market of web and mobile applications has pressured technical data management solutions to meet demands for the high number of concurrent users, high volume and velocity of data, as well as high reliability (Ramakrishnan, 2012). Consequently, a large portion of data management has moved to cloud environments, which enable rapid prototyping, cost–efficiency, and automated resource allocation on demand (Buyya et al., 2019). Furthermore, the information technology field and related skills are becoming more and more common, and basic software development is introduced earlier and earlier as well as more and more broadly into various curricula (L edeczi et al., 2021; Szabo et al., 2019). As the ubiquitousness of the information technology field is increasing, expert systems, such as DBMSs need to be accessible for novices as well as experts (Nicolaos & Katerina, 2015; Sobiesiak et al., 2002). As such, many vendors behind modern, distributed database management systems (DBMS) have made DBMS deployment and database distribution flexible, automated, and effortless for software developers (Hacigumus et al., 2002). Given these considerations, it remains unclear whether usability extends from features, such as automated and flexible database distribution to other aspects of DBMSs. To this end, we set out to compare NewSQL database management systems from a scientifically neglected point of view of query language compiler usability. Specifically, we compare 16 retrieval query syntax error messages of CockroachDB, SingleStore, NuoDB, and VoltDB using error fixing success rate, error recovery confidence, and perceived usefulness of the error message for finding and fixing the error as performance metrics. Our results reveal modern DBMSs with many compiler errors designed against 40 year old HCI best practices, as well as statistically significant differences in CONTACT Toni Taipalus [email protected] University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland ß2022 The Author(s). Published with license by Taylor & Francis Group, LLC This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION https://doi.org/10.1080/10447318.2022.2108648 error message usefulness between these four modern DBMS compiler errors. The rest of this study is structured as follows. In the next section, we discuss the theoretical background and prior studies on data management, usability, and query languages. In Section 3 we describe our research setting and data collection, and state our hypotheses. In Section 4 we present our results from statistical analyses and in Section 5 the implications of our research, and some recommendations for the DBMS industry. Section 6 concludes our study. 2. Theoretical background 2.1. Data management in the cloud Cloud computing is a growing technology model built around providing a high-level abstraction of distributed computing, usually offered as a subscription-based service to the end-user (Abbasi et al., 2019; Buyya et al., 2019). Effectively, the enduser pays for resources they utilize (e.g., storage space, computation, and network bandwidth), rather than investing in hardware, software and infrastructure outright. Consequently, some of the reasons for the popularity of the cloud computing model are the speed of deployment, scalability of computing resources, and cost-efficiency (Buyya et al., 2019). Depending on the cloud service provider, different service models are offered. These service models typically dictate which parts of the system are provided and maintained by the service provider, and which parts by the end-user. For example, the service provider may merely provide the infrastructure and (often virtual) hardware, the aforementioned complemented by an operating system, or all the aforementioned complemented by a database management system (Somu et al., 2017). Depending on their requirements, the end-user may choose a high level of abstraction while discarding control over low-level configurations. In contrast, by choosing a low level of abstraction, the end-user retains control—and responsibility—of low-level tasks, such as maintaining the operating system. The role of the relational data model, SQL, and traditional relational DBMSs (RDBMS), such as Oracle Database, IBM DB/2, and Microsoft SQL Server has been challenged in the 2000s by new data models and query languages of numerous NoSQL data stores (Grolinger et al., 2013). While RDBMSs have favored data consistency at the cost of availability and transaction performance (Chaudhry & Yousaf, 2020; Pavlo & Aslett, 2016), many NoSQL data stores have been designed the other way around to serve, e.g., web applications with requirements for low response time and a high number of concurrent end-users (Ramakrishnan, 2012). In the 2010s, however, the industry leaders, such as Google deemed transaction support, data consistency, and the SQL language important enough to design a new DBMS to incorporate features from both traditional RDBMSs and NoSQL data stores (Corbett et al., 2013). In general, modern (i.e., in this case initially released after 2010) online transaction processing DBMSs that use the relational model, SQL, and distributed architecture are called NewSQL DBMSs. A recent study (Pavlo & Aslett, 2016) further defines NewSQL DBMSs as systems built from the ground up, rather than extensions or modifications of existing systems. The study concludes that while NewSQL systems do not offer new features or innovations per se, they skillfully integrate tested techniques into single systems. That is, “NewSQL database systems are not a radical departure from existing system architectures but rather represent the next chapter in the continuous development of database technologies”(Pavlo & Aslett, 2016, p. 53). When choosing a set of NewSQL DBMSs for this study, we adopt the definition of NewSQL systems provided above (Pavlo & Aslett, 2016). The distributed architectures of NoSQL and NewSQL systems are a natural fit for cloud environments (Grolinger et al., 2013). While traditional RDBMSs also support database distribution and are offered by cloud service providers, some studies consider traditional RDBMS distribution difficult for various reasons (Pavlo & Aslett, 2016;Stonebraker,2010). In practice, the new distribution implementations provide automatic distribution of data, automated data balancing between the distributed nodes, and with heterogeneous distribution models, automated primary/secondary elections during faults or other topology modifications. 2.2. Query language usability According to the ergonomics of human-system interaction standard, “Usability is relevant to regular ongoing use, to enable users to achieve their goals effectively, efficiently and with satisfaction; learning, to enable new users to be become effective, efficient and satisfied when starting to use a system, product or service”(ISO, 2018). Usability is a recurring theme in the evolution of cloud data management, and one of the main reasons behind new database distribution implementations was rooted in usability considerations (Shi et al., 2010; Stonebraker, 2010). First, arguably, in addition to performance and flexible scalability, the need for dynamic database schemas is one of the defining characteristics of many NoSQL data models. Dynamic schemas absolve the software developer from defining a strict database structure. Second, the need to return to strong transactional capabilities with NewSQL systems can be seen as a need to abstract the implementation of database transactions from the software developer to the DBMS. Finally, the abstraction of computer infrastructure, hardware, and partial software through cloud services all serve the demand for usability through costefficiency, flexibility, and rapid prototyping. On logical grounds, it seems interesting whether the demands for usability are also considered in other aspects of cloud database management systems, e.g., in compiler error messages, as usability, in general, has been argued to facilitate costefficiency through, e.g., improved productivity, reduced training, and documentation costs, lower support costs, and competitive edge (Donahue, 2001). Data management solutions in cloud environments utilize several query languages and data models. Traditional RDBMSs and NewSQL systems utilize an implementation of SQL, while 2 T. TAIPALUS AND H. GRAHN NoSQL systems each usually have a distinct query language, e.g., Neo4j’s Cypher (Francis et al., 2018), or Cassandra’sSQLbased CQL (Wang & Tang, 2012). These proprietary query languages are sometimes complemented by SQL. As these NoSQL languages are designed for different data models and have different levels of expressiveness, usability comparison of different query languages is arguably problematic. Further, as some traditional RDBMSs offer implementations dating across four or five decades, we deemed it more interesting to focus on the usability of systems developed from the ground up in the last decade. As a contrasting example, Oracle Database 8i documentation from 1998 listed the same SQL error messages as Oracle Database 21c from 2021 (Oracle Corporation, 2021). SQL is a language initially designed for data retrieval. However, in the decades following the initial release of the SQL standard, the language has evolved to encompass data manipulation, database structure definition, access control, and transaction management (Chamberlin, 2012). Data retrieval remains the most studied aspect of SQL (Taipalus & Sepp€ anen, 2020), and because of this more established research background, this study focuses solely on data retrieval. Possibly due to the increasingly ubiquitous nature of data, and the rising popularity of data analytics and data science, query languages, SQL in particular, have received increasing scholarly attention (Taipalus & Sepp€ anen, 2020). Current educational research seems rather unanimous with the view that learning SQL is difficult (Miedema et al., 2021; Shin, 2020; Taipalus & Per€ al€ a, 2019). Usability concerns in query formulation have been explained by human factors, such as cognitive load (Shin, 2020; Smelcer, 1995), data model and real-world mismatch (Borthick et al., 2001; Sutcliffe et al., 2000), and different user characteristics (Ashkanasy et al., 2007; Bak & Meyer, 2011). Additionally, it has been shown that different environmental aspects, such as database complexity (Taipalus, 2020a) and database representation (Shin, 2020; Siau et al., 2004) have an effect on query writing. Finally, different measures for engaging and helping the enduser have been proposed in scientific literature, e.g., query visualization and previews (Taipalus, 2019; Tanin et al., 2000), cosmetic alterations (Dong & Khandwala, 2019), different natural language interfaces (Ribeiro & Moreira, 2003), and the facilitation of query reuse (Allen & Parsons, 2010; Toorn et al., 2022). However, error message research has not extended from programming languages to query languages, and the latest studies on the effects of SQL compiler error messages on query formulation seem to be published in the 1980s (Reisner, 1981; Welty & Stemple, 1981), until a recent comparison of SQL compilers of traditional RDBMS in 2021 (Taipalus et al., 2021). The differences in the SQL standard (ISO/IEC, 2016a,2016b) between the 1980s and 2020s, as well as differences between SQL and imperative programming languages, and the potential threats to the generalizability of scientific results induced thereof have been highlighted in a previous study (Taipalus & Sepp€ anen, 2020). Regarding usability, due to its declarative nature, SQL is arguably a “blacker box”to a software developer than an imperative programming language. 2.3. Error messages and error recovery A large number of studies have shown the importance of compiler error messages for learning, and for software development in general in the context of programming languages (Becker et al., 2016,2019; Wrenn & Krishnamurthi, 2017). The same studies have also argued that current compiler error messages are ineffective due to several reasons. From the perspective of error messages, DBMSs have a query parser that checks the syntax of the query and outputs an error message if necessary (Hellerstein et al., 2007). In the scope of this study, the query parser is the component that separates the usability aspects of different DBMSs from each other. Additionally, some DBMSs, such as MySQL allows pluggable storage engines that can be switched with relative ease. The storage engine typically contains the query parser, and thus the storage engine is often responsible for generating the SQL error messages. It is worth noting that while SingleStore is a NewSQL system, it utilizes the InnoDB storage engine also utilized by MySQL. When an end-user, e.g., a software developer, writes an erroneous query and submits it to a DBMS, the DBMS outputs an error message. This is often referred to as error detection (van der Schaaf, 1995; Zapf & Reason, 1994). Next, the end–user tries to interpret the error message and find the erroneous part of the query. This phase is called explaining. Finally, the enduser attempts to fix the error, typically based on the feedback provided by the error message. This process of three phases is called error recovery (van der Schaaf, 1995; Zapf & Reason, 1994), and serves as a theoretical foundation for our chosen subjective metrics, i.e., error recovery confidence,anderror message usefulness for finding and fixing the error. A seminal study published in 1982 suggests that computer error messages should be “brief, positive, constructive, specific, comprehensible”(Shneiderman, 1982,p.611),positive referring to refraining from using words, such as “illegal, invalid, error” in the error message, and constructive referring to hints or suggestions on the causes of the error and how to fix it. Considering that the WHERE clause is one of the most common SQL clauses, Figure 1 shows an SQL query with a simple typographical error in the keyword WHERE, and seven corresponding error messages from traditional RDBMSs and NewSQL systems. As software developers, especially novices, often consider the compiler the first authority in determining the quality of written software, the error messages in Figure 1 succeed in neither communicating why the query is erroneous nor adhering to all the suggestions presented 40 years ago. 3. Research setting 3.1. Study scope In the previous section, we discussed the importance of effective error messages in the context of programming languages, and that prior works have attempted to explain and enhance said error messages to facilitate more effective software development. We also argued for the usability concern, which seems to be one of the driving factors behind the popularity of cloud environments and more effortless database distribution. However, in light of previous scientific INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 3 literature, and a preliminary inspection of DBMS error messages (Figure 1), it seems that even SQL compilers of modern DBMSs do not necessarily account for usability concerns or error message design guidelines and that the topic has not received much scientific attention in recent decades. As explained in Section 2, we deemed comparing DBMSs utilizing SQL with DBMSs utilizing some other query language difficult for internal validity. On the other hand, a recent study (Taipalus et al., 2021) compared SQL compiler usability of traditional RDBMS. For these reasons, in this study, we chose to focus on NewSQL systems using the SQL compiler usability framework reported in a previous study (Taipalus et al., 2021). We deemed it more interesting to focus on popular NewSQL systems, even though measuring popularity is rather difficult. Based on three NewSQL studies (Kaur & Sachdeva, 2017; Pavlo & Aslett, 2016; Schreiner et al., 2019), we identified four popular NewSQL database management systems for this study: CockroachDB (v19.2.2), SingleStore (7.0.10, previously known as MemSQL), NuoDB (build 4.0.4-2), and VoltDB (Community 9.2.2). All these systems implement relational or semi-relational data models, use SQL as their query language, and are built from the ground up in the 2010s (Grolinger et al., 2013). Additionally, DB-Engines 1 ranks these four DBMSs high in popularity among NewSQL systems, when NewSQL systems are defined as in Section 2.1. In regard to different types of errors, we focus on syntax errors, and based on a previously reported framework (Taipalus et al., 2018), we focus on the 16 most common syntax errors in SQL queries. These previously reported syntax errors and our corresponding tests are reported in Table 1. These tests and queries within are in turn based on those reported in a previous study (Taipalus et al., 2021), but adjusted to account for the chosen four NewSQL systems. In the next subsections, we detail the data collection, hypotheses, and analyses, which are summarized in Figure 2. 3.2. Data collection To focus on the differences in the selected DBMS usability in fixing erroneous queries, we chose not to use database experts as participants. We speculated that experts might SELECT name, price_usd , brand, model FROM product WHRE (brand LIKE S% OR brand LIKE C% ) AND picture IS NULL ORDER BY name DESC; (a) Query with a typographical error (WHRE instead of WHERE) Msg 321, Level 15, State 1, Server q7410, Line 4 "brand" is not a recognized table hints option. (b) SQL Server error message ERROR : syntax error at or near "LIKE" LINE 3: WHRE (brand LIKE S% OR brand LIKE C% ) ˆ (c) PostgreSQL error message ORA-00933: SQL command not properly ended (d) Oracle Database error message invalid syntax: statement ignored: at or near "like": syntax error DETAIL: source SQL: SELECT name, price_usd , brand, model FROM product WHRE (brand LIKE S% OR brand LIKE C% ) ˆ HINT: try \h <SOURCE> (e) CockroachDB error message SQL error while compiling query: SQL Syntax error in "SELECT name, price_usd , brand, model FROM product WHRE (brand LIKE S% OR brand LIKE C% ) AND picture IS NULL ORDER BY name DESC;" unexpected token: LIKE required: ) (f) VoltDB error message ERROR 1064 ER_PARSE_ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near (brand LIKE S% OR brand LIKE C% ) AND picture IS NULL ORDER BY name DE at line 3 (g) SingleStore (with InnoDB storage engine) error message Error 42000: syntax error on line 3 WHRE (brand LIKE S% OR brand LIKE C % ) ˆ expected end of statement got parenthesis (h) NuoDB error message Figure 1. Erroneous query with a simple typographical error (a), and seven corresponding error messages generated by seven different DBMSs; three traditional RDBMSs (b–d), and four NewSQL DBMSs (e–h). 4 T. TAIPALUS AND H. GRAHN have former experience on one or several of the DBMSs studied and that their expertise would result in successful error fixing regardless of the error message, thus skewing the results toward a ceiling effect (i.e., results are not statistically significantly different because tests were too easy for selected participants). Furthermore, experts are arguably less dependent on the error messages, and more able to fix erroneous queries regardless of the error message. Because we wanted to specifically study the effects of different compiler error messages, we recruited our study participants from a database management course given at the authors’university. The participants majored in software engineering or information systems science and had acquired basic SQL knowledge from the course. The students were promised extra course points for taking the survey. Taking the survey was not mandatory, and if a student also chose to do so, their answers were anonymized and used in this study. Participating in the study was not required for extra course points, and the students were shown a full privacy statement before answering. Out of the 188 students who answered the survey, 157 (84%) chose to participate in the study. Next, a participant was randomly assigned to one of the four database management system groups—i.e., CockroachDB (n¼32), NuoDB (n¼44), SingleStore (n¼39), and VoltDB (n¼44)—and shown a set of 20 tests, one test at the time. The first four tests were control questions measuring participant skill in error fixing, and these four tests were the same for all participants, regardless of the group the participant was assigned to. Next, the test suite of 16 tests (cf. tests T01–T16 in Table 1) was shown, test by test, and in a randomized order for each participant. Each of the 16 tests consisted of a database structure diagram, a data demand, an erroneous SQL query, an error message generated by the DBMS, a free text input box in which the participant was instructed to write the fixed query, and a set of five-point Likert scale (1 ¼strongly disagree, 5 ¼strongly agree) questions pertaining to subjective indicators of the usability qualities of the error message (cf. hypotheses H 2 ,H 3 and H 4 in the next section). Depending on the group the participant was assigned to, they were shown corresponding error messages, e.g., for participants assigned to VoltDB group, VoltDB generated error messages were shown. Answering could be paused or stopped altogether, yet none of the participants chose to do so. The participants could use any materials for support during the tests. For more details on the tests, error messages, database structure, and questions, please refer to the supplementary Appendices. After all the participants had answered the tests, the first author coded the queries submitted by the participants as correct or incorrect. A query that contained at least one syntax error was considered incorrect. 3.3. Hypotheses To study usability considerations of four NewSQL database management systems, we formulated two sets of hypotheses. Hypotheses H 1 –H 4 compare objective query fixing success rates, as well as subjective error recovery confidence, usefulness for error finding, and usefulness for error fixing with a between-subjects study design. Hypotheses H 5 –H 7 test correlation of error message qualities regardless of the database management system group. We chose to test the particular correlations between the objectively measured variable (i.e., success rate) and the subjectively measured variables (i.e., perceived usefulness for finding and fixing the error, and error recovery confidence) because of the nature of how these variables were measured. In other words, we did not test correlations between subjectively measured variables. H 1 :the medians of query formulation success rates are different for the database management system groups. H 2 :the medians of error recovery confidence are different for the database management system groups. H 3 :the medians of perceived usefulness for error finding are different for the database management system groups. H 4 :the medians of perceived usefulness for error fixing are different for the database management system groups. Recruit participants Assign participants to DBMS groups Fix erroneous control question queries Fix erroneous queries, answer Likert questions Omit outliers based on control questions Compare the DBMS groups (H1-H 4) Check for correlations (H5 -H 7) Figure 2. Overview of the data collection and analysis process; white rectangles refer to actions performed by us and grey rectangles to actions performed by the study participants. Table 1. Test suite consists of 16 most common syntax errors (Taipalus et al., 2018). Test Syntax error name T01 Ambiguous column T02 Omitting quotes around character data T03 IS where not applicable T04 Confusing the syntax of keywords T05 Confusing the logic of keywords T06 Too many columns in subquery T07 Undefined column T08 Misspellings T09 Failure to specify column name twice T10 Using an aggregate function outside SELECT or HAVING T11 Grouping error: extraneous grouping column T12 Non-standard operators T13 Using WHERE twice T14 Non-standard keywords or standard keywords in wrong context T15 Synonyms T16 Curly, square, or unmatched brackets INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 5 H 5 :q6¼ 0; the correlation coefficient between query formulation success rate and error recovery confidence is not equal to zero. H 6 :q6¼ 0; the correlation coefficient between query formulation success rate and perceived usefulness for finding the error is not equal to zero. H 7 :q6¼ 0; the correlation coefficient between query formulation success rate and perceived usefulness for fixing the error is not equal to zero. 3.4. Data preparation and mitigation of control variables Due to random participant assignment, it is possible that participants with higher (or lower) querying skills were assigned to the same group. This assignment presents a threat to internal validity, potentially skewing the results regardless of the qualities of the dependent variable (i.e., the error messages). To mitigate the effect of imbalance in participant assignment, we included four control questions in the survey. Based on the control questions, we omitted outliers from further between-subjects analyses based on query fixing success rate in the control questions. After the outliers were removed, and because the data were not normally distributed, we ran a Kruskal-Wallis H test to determine if there were differences in control question scores between the four groups of participants using different database management systems: CockroadDB (n¼25), NuoDB (n¼44), SingleStore (n¼28), and VoltDB (n¼44). Distributions of control question scores were similar for all groups, as assessed by visual inspection of a boxplot. There were no significant differences in the medians of control question scores between groups, H(3) ¼4.987, p¼.173. Hence, we considered the groups equal in terms of query fixing skills. For hypotheses H 5 –H 7 , which were not concerned with between–subjects comparison, we analyzed all data (N¼157). SingleStore SQL compiler tolerated syntax errors in tests T09 and T11. The lack of an error message in these two tests was compensated in the questionnaires by made up error messages. The test results of tests T09 and T11 for SingleStore were omitted from the statistical analyses. 4. Results 4.1. A summary of results In the following sections, we present the analyses in more detail, i.e., system per system, and describe the chosen statistical tests. A significance level of a¼.05 was chosen for all the statistical tests. A summary of results presented in Table 2 shows that hypotheses H 3 –H 7 were supported, and hypotheses H 1 and H 2 were not supported. Please refer to Figure 3 for an overlook of the DBMS comparison. 4.2. Database management system group differences For each of the hypotheses H 1 ,H 2 ,H 3 , and H 4 , we ran a Kruskal-Wallis Htest to determine if there were differences in error message effectiveness (measured in error fixing success rates, H 1 ), error recovery confidence (H 2 ), and perceived usefulness of the error message in terms of finding (H 3 ) and fixing (H 4 ) the error between four groups of participants with different database management systems: CockroachDB (n¼25), SingleStore (n¼28), NuoDB (n¼44), and VoltDB (n¼44). Distributions of the answers for all hypotheses were similar for all groups, as assessed by visual inspection of a boxplot. Subsequently, pairwise comparisons were performed using Dunn’s(1964) procedure with a Bonferroni correction for multiple comparisons. Adjusted p-values are presented in Table 3, and the results are visualized in Figure 3. 4.3. Correlations For each of the hypotheses H 5 ,H 6 and H 7 , we ran a rank biserial correlation to assess the relationship between error message effectiveness (measured in success rate, H 5 ) and error recovery confidence; between query formulation success rate and perceived usefulness for finding the error (H 6 ); and between query formulation success rate and perceived usefulness for fixing the error (H 7 )(N¼157). For all three hypotheses, and for individual database management systems, the results were all statistically significant with a weak positive correlation. The test statistics are presented in Table 4. 5. Discussion 5.1. Implications for research The results show no statistically significant differences in error message effectiveness between the DBMSs (hypothesis H 1 ). Although this observation implies that none of the DBMSs studied has more effective error messages than another, it is worth noting that success rate may be considered as one metric for effectiveness, rather than the sole metric. For example, in the context of programming Table 2. Summary of results. Hypothesis Short description Supported Test statistic Effect size H 1 Different effectiveness No H(3) ¼5.254, p¼.154 H 2 Different recovery confidence No H(3) ¼0.157, p¼.984 H 3 Different usefulness for error finding Yes H(3) ¼24.396, p<.001 g 2 ¼.174 H 4 Different usefulness for error fixing Yes H(3) ¼9.870, p¼.020 g 2 ¼.071 H 5 Effectiveness () recovery confidence Yes r rb (2486) ¼.283, p<.001 H 6 Effectiveness () error finding Yes r rb (2486) ¼.238, p<.001 H 7 Effectiveness () error fixing Yes r rb (2486) ¼.215, p<.001 6 T. TAIPALUS AND H. GRAHN language compiler error messages, it has been suggested that the messages affect error recovery time rather than success (Ahmed et al., 2019). There were no statistically significant differences in error recovery confidence between the DBMSs (hypothesis H 2 ). This suggests that error messages, although different, do not necessarily affect novice confidence in error recovery. Arguably, some error messages highlight the erroneous part of the query, yet fail to identify why the query contains an error, or may even provide false information on why the query is erroneous (Figure 1). Based on the results, it remains unclear why there were no significant differences in error recovery confidence. Similarly, the results yielded by this study support the notion that SQL error recovery confidence may have a similar relationship with error message effectiveness (hypothesis H 5 ) as confidence more generally has with success (Fleming et al., 2010; Martino et al., 2013). A rather underwhelming result of a weak positive correlation between success rate and error recovery confidence may indicate that it might be unexpectedly common that either a participant was confident in their fixed query, yet the query was incorrect, or that a participant was unsure Table 3. Test statistics for hypotheses H 1 –H 4 ;post-hoc analyses were performed only if the Kruskal Wallis Htest was statistically significant; DBMS names have been abbreviated as (Si)ngleStore, (Co)ckroachDB, (Nu)oDB, and (Vo)ltDB. Mdn Pairwise comparison (p-value) Co Si Nu Vo Co-Si Co-Nu Co-Vo Si-Nu Si-Vo Nu-Vo Effectiveness .875 .875 .813 .813 Recovery confidence 3.94 3.78 3.75 3.81 Error finding 4.13 3.56 4.16 3.68 .004 1 .023 <.001 1 .003 Error fixing 3.63 3.36 3.78 3.50 .256 1 .665 .048 1 .140 (a) Error message effectiveness, measured in success rates (b) Perceived confidence in error recovery ( c ) Perceived usefulness for error finding ( d ) Perceived usefulness for error fixing Figure 3. Between–subjects comparison of 16 tests regarding error message effectiveness, error recovery confidence, and error message usefulness for error finding and fixing—the boxplots represent interquartile range, and whiskers minimum and maximum values, excluding outliers. (a) Error message effectiveness, measured in success rates, (b) perceived confidence in error recovery, (c) perceived usefulness for error finding, and (d) perceived usefulness for error fixing. Table 4. Test statistics for hypotheses H 5 –H 7 ; correlations between error message effectiveness and error recovery confidence (r.c.), and perceived error message usefulness for finding and fixing the error. Effectiveness () r.c. Effectiveness () finding Effectiveness () fixing CockroachDB r rb (510) ¼.315, p<.001 r rb (510) ¼.306, p<.001 r rb (510) ¼.316, p<.001 SingleStore r rb (566) ¼.336, p<.001 r rb (566) ¼.240, p<.001 r rb (566) ¼.258, p<.001 NuoDB r rb (702) ¼.260, p<.001 r rb (702) ¼.174, p<.001 r rb (702) ¼.198, p<.001 VoltDB r rb (702) ¼.239, p<.001 r rb (702) ¼.178, p<.001 r rb (702) ¼.208, p<.001 INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION 7