Supplementary Materials - An empirical study on in-context annotations and notification features in dashboard development tools
Abstract
Organisations seeking to better leverage their data assets are driving the growth of data-intensive (DI) software development. DI software, which combines the development of analytics and business applications, is being deployed across an ever-growing range of domains and situations and is increasingly influencing high-stakes decisions. Developing DI software requires multiple skills, including data, software and domain skills. These skills are typically distributed across different individuals working together in multidisciplinary data-intensive software development teams (MDSTs). There is still limited research on the experiences of these teams and the development practices and tools they employ. Modern design tools that incorporate annotation features for UX design and collaborative whiteboarding have become prevalent, with their use growing substantially during and since COVID-19. We carried out a mixed-methods empirical study to explore how team members in MDSTs use data development tools, specifically dashboarding tools, when working with other team members. We investigate whether features such as in-context annotations and notifications in data development tools may be useful to support their multidisciplinary work practices, including data analysis and visualisation reviews. Our findings highlight the potential of these features to better support development practices. Key considerations include data access and security controls, dealing with alert overwhelm and alert fatigue and integration between task management, team collaboration and dashboard development tools. This file contains anonymised supplementary materials for the research paper: DB_F1 - Dashboard sharing feature video DB_F2 - Dashboard annotation feature video DB_F3 - Dashboard comment management feature video Focus Group Template Survey response data Survey response analysis - Statement Agreement Level Distributions
Full text
A_Q1 This survey focuses on 2 types of data development tools - dashboard tool and data notebooks used in a multidisciplinary team context. For this survey, we use broad definitions: Dashboard - a visual representation of data and metrics. For example Qlik, Tableau or MS PowerBI, etc. Data Notebook - a tool that supports development of queries and data analysis. For example SQL Notebook or Databricks notebook, etc. Multidisciplinary team - 2 or more team members that have different training, work or study backgrounds working collaboratively to develop dashboards and data analysis/queries. For example data analyst, domain experts, data engineer and quality assurance engineer, etc. I have experience working with a multidisciplinary data development team. o Yes (1) o No (2)
A_Q2 I have experience with the following (tick all that apply) ▢ Developing a dashboard using a dashboarding tool (for the example, PowerBI, Tableau, Qlik, etc.) (1) ▢ Reviewing and providing feedback about a dashboard or visualisation (2) ▢ Developing data queries or data analysis using a notebook (for example SQL notebook). (3) ▢ Reviewing, providing feedback about a data analysis or data query (4) ▢ None of the above (5) End of Block: Section A: Screening Questions Start of Block: Section B: Demographics B_Q1 Demographic and career experience questions In which country do you work? ▼ Afghanistan (1) ... Zimbabwe (1357) B_Q2 In what age group are you in? ▼ <18 (1) ... 71+ (12) Page 2 of 8
B_Q3 What gender do you identify as? ▢ Man (1) ▢ Woman (2) ▢ Non-binary / gender diverse (3) ▢ My gender is not listed (4) __________________________________________________ ▢ Prefer not to say (5) B_Q4 What is the highest level of education you have completed? o Less than Primary (10) o Primary (11) o Some Secondary (12) o Secondary (13) o Vocational or Similar (14) o Some University but no degree (15) o University - Bachelors Degree (16) o Graduate or professional degree (MA, MS, MBA, PhD, Law Degree, Medical Degree etc) (17) o Prefer not to say (18) Page 3 of 8
B_Q5 In what area/discipline is your highest education or qualification level (tick all that apply) ▢ General IT (1) ▢ Data Science (2) ▢ Computer Science (3) ▢ Engineering incl. Software Engineering (4) ▢ Business / Management / Economics / Accounting (5) ▢ Law (13) ▢ Health (6) ▢ Humanities (7) ▢ Social Sciences (8) ▢ Biological Sciences (9) ▢ Education (10) ▢ Other: (12) __________________________________________________ Page Break Page 4 of 8
B_Q6 This study focuses on teams that develop data analytics systems involving dashboards, data analysis and data query creation. Data systems development require specialised skills including data expertise, software engineering expertise and subject matter or business expertise. What is your role (relating to data-development) (tick all that apply)? ▢ Data Visualisation / Data Analyst (1) ▢ Data Scientist (2) ▢ Data Engineer (3) ▢ Software - Software Engineer/DevOps Engineer (4) ▢ UX (5) ▢ Quality / Testing (6) ▢ Product Manager/Product Owner (7) ▢ Scrum Master / Project Manager (8) ▢ Business Analyst (9) ▢ Subject Matter Expert / Domain expert (10) ▢ Data Governance (11) ▢ Sponsor (13) ▢ Other (12) __________________________________________________ B_Q7 How many years experience do you have in data-development related roles? ▼ <=1 (4) ... 20+ (23) End of Block: Section B: Demographics Page 5 of 8
Start of Block: Section D: Validation of dashboard collaboration features Dashboard collaboration features prototypes We now present 3 video demonstrations of dashboard development tool features designed to support team members when they review or design dashboards and would like your opinion on their usefulness and usability. We also ask you to note down any concerns or improvements that you would like us to consider about the feature. Each of the following 3 questions will present one feature video and present associated questions. D_Q1 Consider a feature in a Dashboard Development Tool that supports setting review access roles, and reviewer tagging, and notifies sharing through team collaboration software. Please watch the video and answer the questions. NOTE - see anonymised video DBF1 in supplementary materials. 0 2 4 6 8 10 I would find this feature useful when working with a multi-disciplinary data development team. () ◻ This feature could capture review feedback and design discussions. () ◻ This feature would be easy to use for data experts in general. () ◻ This feature would be easy to use for domain experts in general. () ◻ This feature would be easy to use for software experts in general. () ◻ I would find this feature easy to use. () ◻ D_Q2 What concerns or opportunities for improvement do you have about a feature that supports sharing of the dashboard for review? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ Page 6 of 8 Strongly Disagree Disagree Neutral Agree Strongly Agree NA
D_Q3 Consider a feature to visually navigate and annotate the dashboard, including prioritisation of feedback about the shared dashboard and data-model. Please watch the video below and then answer the questions. NOTE - see anonymised video DBF2 in supplementary materials. 0 2 4 6 8 10 I would find this feature useful when working with a multi-disciplinary data development team. () ◻ This feature could capture review feedback and design discussions. () ◻ This feature would be easy to use for data experts in general. () ◻ This feature would be easy to use for domain experts in general. () ◻ This feature would be easy to use for software experts in general. () ◻ I would find this feature easy to use. () ◻ D_ Q4 What concerns or opportunities for improvement do you have about a feature in a dashboard development tool to support visual annotations including prioritisation about the dashboard ? ________________________________________________________________ ________________________________________________________________ Page 7 of 8 Strongly Disagree Disagree Neutral Agree Strongly Agree NA
D_Q5 Consider a feature to track comments, and make responses in an action list format. Please watch the video below and then answer the questions. NOTE - see anonymised video DBF3 in supplementary materials. 0 2 4 6 8 10 I would find this feature useful when working with a multi-disciplinary data development team. () ◻ This feature could capture review feedback and design discussions. () ◻ This feature would be easy to use for data experts in general. () ◻ This feature would be easy to use for domain experts in general. () ◻ This feature would be easy to use for software experts in general. () ◻ I would find this feature easy to use. () ◻ D_Q6 What concerns or opportunities for improvement do you have about a feature to manage and track comments or annotations made in the dashboard? ________________________________________________________________ ________________________________________________________________ Q7 In what situations could you see dashboard collaboration features (i.e. sharing, visual navigation and annotation, comment and action management) being beneficial ? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ Page 8 of 8 Strongly Disagree Disagree Neutral Agree Strongly Agree NA