Utilizing the Capabilities Offered by Eye-Tracking to Foster Novices' Comprehension of Business Porcess Models [original]

Utilizing the Capabilities Offered by Eye-

Tracking to Foster Novices’ Comprehension of

Business Process Models

Michael Zimoch1, R¨udiger Pryss1, Georg Layher2, Heiko Neumann2, Thomas

Probst3, Winfried Schlee4, and Manfred Reichert1

1Institute of Databases and Information Systems, Ulm University, Germany

2Institute of Neural Information Processing, Ulm University, Germany

3Dep for Psychotherapy and Biopsycho Health, Danube University Krems, Austria

4Department of Psychiatry and Psychotherapy, Regensburg University, Germany

{michael.zimoch, ruediger.pryss, georg.layher, heiko.neumann,

manfred.reichert}@uni-ulm.de, [email protected],

winfried.schlee@googlemail.com

Abstract. Business process models constitute fundamental artifacts for

enterprise architectures as well as for the engineering of processes and

information systems. However, less experienced stakeholders (i.e., no-

vices) face a wide range of issues when trying to read and comprehend

these models. In particular, process model comprehension not only requi-

res knowledge on process modeling notations, but also skills to visually

and correctly interpret the models. In this context, many unresolved is-

sues concerning the factors hindering process model comprehension exist

and, hence, the identification of these factors becomes crucial. Using eye-

tracking as an instrument, this paper presents the results obtained of a

study, in which we analyzed eye-movements of novices and experts, while

comprehending process models expressed in terms of the Business Pro-

cess Model and Notation (BPMN) 2.0. Further, recorded eye-movements

are visualized as scan paths to analyze the applied comprehension stra-

tegies. We learned that experts comprehend process models more ef-

fectively than novices. In addition, we observed particular patterns for

eye-movements (e.g., back-and-forth saccade jumps) as well as different

strategies of novices and experts in comprehending process models.

Keywords: Business Process Model Comprehension, Eye-Tracking, Cog-

nition, Eye-Movement Modeling Examples, (Hidden) Markov Model

1 Introduction

Business Process Management (BPM) aims at the creation, improvement, as

well as automation of business processes and has become vital for the success of

any enterprise [1]. In this context, a process model acts as a blueprint comprising

tasks, decisions, and actors dedicated to a specific process. Usually, these pro-

cess models are expressed in terms of textual or graphical artifacts. The latter,

2 Zimoch et al.

in turn, are utilized to advance the understanding of business processes (i.e.,

process model comprehension) for all involved stakeholders [2].

Generally, process models should be created in a way such that process stake-

holders do not face any problems in comprehending them. Still, stakeholders are

encountering challenges in the comprehension of process models [3], especially

the less experienced ones (i.e., novices) [4]. In this context, the use of eye-tracking

might provide valuable insights into the cognitive processes of comprehending

process models [5]. Amongst others, assertions about the cognitive load can be

made and the process model comprehension strategies applied can be identified,

e.g., by visualizing the corresponding scan path. In detail, the scan path reflects

the chronological order of fixations (i.e., gaze over informative areas of interest)

and saccades (i.e., quick eye-movements between fixations) [6].

This paper presents the results we obtained from a process model comprehension

study relying on eye-tracking. In detail, novices and experts in the domain of

process modeling had to study three different process models expressed in terms

of the Business Process Model and Notation (BPMN) 2.0 [7], whilst their fixa-

tion and saccade patterns were recorded by an eye-tracker. The objective of the

study is to deliberately disclose the approaches applied by novices and experts

to comprehend process models. Further, we want to investigate whether or not

there are differences in comprehending process models, e.g., in which way the

applied comprehension strategies between the participants differ. The study in-

sights can be used to derive comprehension guidance, especially for novices, and

be used to augment tools with features fostering process model comprehension.

The remainder of the paper is structured as follows: Section 2 describes the con-

text and setting of the study. Study results, in turn, are analyzed and discussed

in Section 3. Related work is discussed in Section 4. Finally, a summary and an

outlook on future work are given in Section 5.

2 Study Context

In general, comprehension constitutes a cognitive process that establishes re-

lations between available information on objects and events in the long-term

memory, together with information perceived at the moment from the sensory,

working, or short term-memory [8]. In this context, process model comprehension

can be termed as the process for decoding and capturing the information docu-

mented in process models [9]. To be more precise, individuals must cope with the

complexities involved of parsing the relevant syntactic,semantic, and pragmatic

information in a process model. As a consequence, novices are frequently con-

fronted with the challenge to properly read and comprehend process models [10].

To systematically study this challenge, we conducted an eye-tracking study on

how to foster the comprehension of process models. For this purpose, we identify

the scan paths (i.e., chronological order of fixations and saccades) of both novices

and process modeling experts while comprehending process models documented

in terms of BPMN 2.0. Moreover, it is found in eye-tracking studies that experts

comprehending a stimulus (e.g., picture) are more likely to reflect a smaller num-

Eye-Tracking to Foster Novices’ Comprehension of Business Process Models 3

ber of fixations,saccades, and consequently a shorter scan path length compared

to novices [11]. Therefore, amongst others, the use of eye-tracking enables us

to measure the cognitive load as well as to reveal visual stumbling blocks in a

process model, which, in turn, might hinder overall comprehension.

2.1 Study Setting

First of all, we want to identify the strategies for reading and comprehending

process models and analyze whether or not there are differences between novices

and experts. Therefore, we invite novices (n= 17) and experts (n= 19) from

the field of process modeling to participate in the study. This categorization into

two samples (i.e., novices and experts) is accomplished by a median split, i.e.,

based on the time spent on process modeling so far, as provided through a self-

reporting. In the study, participants are asked to read and comprehend three

different BPMN 2.0 process models. These process models cover three different

scenarios, i.e., fitness training, the purchase of an item in an auction, and the

ordering of a pizza. Further, the process models reflect three different levels of

complexity, i.e., easy,medium, and hard. In detail, the easy process model only

comprises a sequence of basic modeling elements of BPMN 2.0. With increasing

level of complexity, new BPMN elements are introduced, previously not con-

tained in the process models, and the total number of elements is increased.

After comprehending a process model, participants need to answer four true-or-

false comprehension questions, solely referring on the semantics of the process

scenario. The questions are used in order to ensure that participants actually

study the process models. Moreover, relative fixations and saccade patterns are

recorded with the SMI iView X Hi-Speed system at a sampling rate of 240 Hz.

Demographic data and qualitative feedback, in turn, are gathered based on que-

stionnaires. In addition, participants are given the instruction to complete the

study as fast as possible but, on the other, as meticulous as possible.

3 Analysis of Eye-Movements

Table 1 presents mean and standard deviation (SD) for all values obtained from

the two samples, i.e., novices and experts. For each level of complexity (i.e.,

easy,medium, and hard), the process model comprehension duration (in s), the

fixation and saccade counts as well as the length of the scan path (in px) are

shown. As expected, a clear difference between the single levels of complexity is

discernible, i.e., an apparent increase in respective factors. Further, juxtaposing

the results obtained for novices and experts, novices need more time for process

model comprehension. Fixation and saccade counts reflect a higher number and

this results in a longer scan path for the comprehension of process models.

Data collected with eye-tracking during the study is analyzed and visualized

with SMI BeGaze software, which allows for an extensive analysis of recorded

eye-movements. After analyzing of the obtained data, different scan paths for

comprehending a process model (i.e., process model comprehension strategies) are

4 Zimoch et al.

Both Novices Experts

Factor Mean (SD) Mean (SD) Mean (SD)

Easy

Duration 35.37 (14.52) 38.53 (16.26) 31.78 (12.29)

Fixation 121.42 (41.57) 138.76 (39.41) 104.84 (43.39)

Saccade 120.81 (39.81) 130.47 (36.12) 111.11 (43.84)

Scan Path 24398.44 (9349.76) 26342.18 (7192.88) 22454.89 (11107.53)

Medium

Duration 54.11 (22.05) 63.11 (23.28) 46.05 (16.30)

Fixation 187.08 (57.98) 209.12 (56.37) 167.47 (57.54)

Saccade 204.36 (68.50) 220.41 (69.54) 188.32 (68.94)

Scan Path 42181.03 (29357.56) 47364.35 (38368.67) 37543.32 (17865.93)

Hard

Duration 67.14 (29.02) 73.07 (33.04) 62.16 (24.01)

Fixation 240.94 (79.16) 264.94 (99.88) 216.00 (56.04)

Saccade 233.69 (83.85) 260.06 (88.90) 207.47 (81.50)

Scan Path 45217.22 (16077.79) 49534.24 (18576.74) 40900.32 (13962.37)

Table 1: Descriptive Results (i.e., Both, Novices, and Experts)

derived for novices and experts respectively. However, both samples show alike

comprehension strategies in the first iteration, i.e., after having a first glance on

the process models. More precisely, beginning from the start element of a process

model, both samples visually consider all elements in a process model through

an element-to-element procedure. After completing the first iteration, strategies

of novices and experts respectively differ during the second iteration. Thereby,

particular eye-movement patterns become apparent, e.g., back-and-forth saccade

jumps and targeted search. Many novices reconsider the process model once more

from the start element, but with a stronger emphasis on single elements and mo-

deling constructs (e.g., parallelism, decision points before splitting control flows

into alternative paths), indicating strong variabilities in saccade patterns as well

as fixation times. Experts, in turn, focus on decisive modeling constructs (e.g.,

decision points) in the process models and, hence, their attention shifts between

these elements. As opposed to experts, the scan paths of novices reflect higher

fixation and saccade counts. Further, they require a greater duration for com-

prehending a process model (cf. Table 1). Figs. 1 and 2 illustrate two identified

scan paths, one from a novice and one from an expert.

Fig. 1: Scan Path of a Novice Fig. 2: Scan Path of an Expert

In Fig. 1, the scan path of a novice is shown. It consists of 301 fixations,254

saccades, and an overall comprehension duration of 98.17 seconds. On the con-

Eye-Tracking to Foster Novices’ Comprehension of Business Process Models 5

trary, the experts’ scan path (cf. Fig 2) comprises 100 fixations and 99 saccades.

In addition, the comprehension duration is 27.47 seconds. Comparing both scan

paths, it becomes apparent that the novice spent more time studying the elements

in the process models, whereas the expert moved quickly over the elements.

3.1 Discussion

An explanation for our study findings might be that process models reflect com-

plex business processes with high information density. Thus, the process of com-

prehending such models leads to a high cognitive load, especially regarding no-

vices [12]. Process model comprehension not only requires knowledge on process

modeling notations, but also the capability to visually interpret these models.

Usually, these capabilities evolve over time due to increasing practical experien-

ces [13]. However, existing research revealed that the visual observation capabi-

lities of experts can be efficiently conveyed to novices [14]. To evaluate this, we

are currently conducting another exploratory study, in which we explicitly show

the novices how experts visually comprehend process models. For this purpose,

we are analyzing the scan paths of modeling experts more deeply with the goal

to derive an average scan path. Furthermore, decisive modeling constructs (e.g.,

parallelism) are identified, which are essential regarding the correct comprehen-

sion of process models. Consequently, novices can be directed through a process

model by selective reading, ensuring a correct process model comprehension.

The material used in the current study is provided in terms of Eye-Movement

Modeling Examples (EMMEs), which shall serve as a basis for guiding novices

in comprehending process models [15]. EMMEs either reflect the scan paths of

experts or highlight specific modeling constructs superimposed onto a process

model. In general, the use of EMMEs allows us to focus on the various dimensi-

ons in a process model. More specifically, any process model contains information

related to different dimensions, e.g., syntactics or semantics [16]. Amongst ot-

hers, the syntactic dimension refers to compliance with modeling rules, whereas

semantics refer to proper and complete documentation of a scenario in a process

model. Therefore, based on the dimension set, the EMMEs can reflect different

conditions (cf. Figs. 3 and 4). Fig. 3 shows the Path Display Condition, which

visualizes the derived experts average scan path in a process model. The Path

Fig. 3: Path Display Condition Fig. 4: Dot Display Condition

Loading more pages...