Eye Tracking Experiments on Process Model Comprehension: Lessons Learned [original]

Eye Tracking Experiments on Process Model

Comprehension: Lessons Learned

Michael Zimoch, R¨udiger Pryss, Johannes Schobel, and Manfred Reichert

Institute of Databases and Information Systems, Ulm University, Germany

{michael.zimoch,ruediger.pryss,johannes.schobel,manfred.reichert}@

uni-ulm.de

Abstract. For documenting business processes, there exists a plethora

of process modeling languages. In this context, graphical process models

are used to enhance the process comprehensibility of the stakeholders

involved. The large number of available modeling languages, however,

aggravates process model comprehension and increases the knowledge

gap between domain and modeling experts. Upon this, one major chal-

lenge is to identify factors fostering the comprehension of process mod-

els. This paper discusses the experiences we gathered with the use of eye

tracking in experiments on process model comprehension and the lessons

learned in this context. The objective of the experiments was to study

the comprehension of process models expressed in terms of four differ-

ent modeling languages (i.e., BPMN, eGantt, EPC, and Petri Net). This

paper further provides recommendations along nine identified categories

that can foster related experiments on process model comprehension.

Keywords: Process Model Comprehension, Eye Tracking, Experiment

1 Introduction

During the last years, a lot of research was conducted to enhance our understand-

ing of working with process models. Besides their creation, particular emphasis

has been put on their reading and understanding, i.e., on process model com-

prehension. Despite extensive research in this field [5, 19, 51], there still exists

a knowledge gap between inexperienced process stakeholders and modeling ex-

perts. Usually, process models are not fully understood by all involved stakehold-

ers, who neither have experiences with process modeling nor deeper knowledge

of any specific process modeling language. This raises the challenging question to

identify the factors fostering the comprehension of process models. One promis-

ing approach for coping with this challenge is to perform experiments.

This paper contributes to the field of business process model comprehension

through experimental research. It discusses the experiences we gathered and the

lessons we learned when performing a series of experiments on process model

comprehension relying on eye tracking. In detail, the experiments conducted

dealt with process model comprehension in connection with four process model-

ing languages (i.e., BPMN,eGantt,EPC, and Petri Net). In these experiments,

2 M. Zimoch et al.

we measured the eye movements of subjects in order to assess their approaches

of comprehending process models. On one hand, we want to enable a comparison

between different process modeling languages. On the other hand, the perceived

pros and cons of respective modeling languages shall be unraveled. To the best

of our knowledge, only few approaches have considered eye tracking for such

comparison in the context of business process model comprehension so far. No-

tably, during the preparation, execution, and analysis of the experiments, several

difficulties have been encountered and various issues emerged. They constitute

valuable lessons learned that will allow for optimizations of future experiments

on process model comprehension.

As another valuable insight for researchers performing experiments on process

model comprehension, this paper introduces nine categories C1 - C9 of the lessons

learned. Process models are related to specific scenarios and, hence, familiarity

of individuals with the considered process scenario varies (C1). Following this,

the understanding (C2) and creation (C3) of process models can be juxtaposed.

Afterwards, a discussion on the structuring and layouting (C4) of the respective

process models is provided, followed by the presentation of the used process

modeling languages (C5) and their specific characteristics (i.e., basic modeling

elements (e.g., activities) (C6) and modeling constructs (e.g., gateways (C7)).

Finally, individuals (C8) as well as measurement methods (C9) are addressed.

The remainder of this paper is organized as follows: Section 2 presents the

experimental setting. The gathered experiences and the lessons learned from the

experiments are presented in Section 3. Related work is discussed in Section 4,

whereas Section 5 concludes the paper with a summary and an outlook.

2 Experimental Setting

The lessons learned refer to experiments on process model comprehension that

use eye tracking as measuring technique. Moreover, eye tracking constitutes a

cost-effective and unobtrusive method to gain deeper insights into human cogni-

tive processes [40]. Thereby, it measures eye movements in response to a visual

stimulus (e.g., picture). Most common types of evaluated eye movements are fix-

ations,saccades, and gaze paths [43]. Fixations constitute eye movements of very

low velocity at a specific point during a stimulus. Saccades, in turn, constitute

quick eye movements. Note that during saccadic eye movements, no visual in-

formation is perceived. In turn, a gaze path represents the chronological order of

fixations and saccades the eyes take while analyzing a stimulus. Furthermore, an

area of interest (AOI) constitutes a manually defined subregion in the presented

stimulus. Generally, it can be used to extract metrics, specifically for these de-

fined regions. For tracking and recording the eye movements in our experiments,

the SMI iView X Hi-Speed system1was used, which allows for accurate eye track-

ing, even over a longer time of recording. The tracking appliance was placed in

front of a monitor that presents the stimuli (i.e., process models) to the subjects;

1http://www.smivision.com/en/gaze-and-eye-tracking-systems/products/

iview-x-hi-speed.html

Experiments on Process Model Comprehension: Lessons Learned 3

eye movements were tracked at a sampling rate of 240 Hz. The eye tracking data

collected during the experiments were analyzed, visualized, and exported with

SMI BeGaze software. The latter enables behavioral and gaze analyses [39].

In the controlled eye tracking experiments, the subjects had to comprehend

12 different process models and were asked to answer several comprehension

questions related to these process models. At the same time, their eye movements

were tracked and recorded. In more detail, the process models were expressed

in terms of BPMN [44], eGantt [47], EPC [45], and Petri Net [46], respectively.

Subjects, in turn, needed to comprehend three process models for each modeling

language reflecting different levels of difficulty. More precisely, the process mod-

els were subdivided into three levels of model difficulty (i.e., easy, medium, and

hard). The simple process models solely contain basic elements (e.g., activities,

start event) of the respective modeling language. Furthermore, with rising level

of difficulty, the total number of elements was increased and new elements pro-

vided by the respective modeling language, not introduced before, were added.

After each process model had been analyzed by the subjects, the latter had to an-

swer four true-or-false comprehension questions (cf. Fig. 1). The questions solely

referred to the semantic content of the process models and were used to evaluate

whether or not subjects interpret the process models correctly. Thereby, correct

answers have been stored with ’1’ and incorrect answers with ’-1’, whereas ’0’

corresponds to ’I am uncertain’ answers. We are aware of the fact that the com-

prehension of process models without any guidance (e.g., purpose) is uncommon.

However, in the first experiments we wanted to investigate the approaches for

the pure comprehension of process models. In general, one of the objectives was

to evaluate the overall performance of subjects, when being confronted with dif-

ferent modeling languages in the context of process model comprehension. The

results obtained may serve as contributions allowing a meaningful comparison

between various process modeling languages in the future.

Q1 Q4

YES NO

Uncertain

YES NO

Uncertain

...

Fig. 1: Overall Procedure of the Experiments

Concerning the overall procedure of the experiments (cf. Fig. 1), at the be-

ginning of the experiment, for one second, a fixation cross was displayed on the

center of the monitor. The cross was used to fixate the gaze of the subjects on

a defined point on the monitor. Afterwards, the process model was presented to

subjects, who could take as much time as they wanted for model comprehen-

sion. Moreover, subjects were told that they should perform the experiments as

fast as possible, but at the same time as careful as possible. Following the model

4 M. Zimoch et al.

comprehension task, four related questions were presented of which only one was

shown on the monitor at the same time. While answering the questions, it was

not possible to reinspect the studied process model. The experimental procedure

was repeated for all considered process models.1

Regarding the considered process models and their level of difficulty, Table

1 presents the number of subjects that studied the respective process models as

well as the results (i.e., means) they delivered by showing the required time to

comprehend the models (i.e., duration time in ms). Furthermore, the response

times for answering the comprehension questions (in ms) as well as corresponding

answering scores (i.e., absolute frequency of correct answers) are illustrated.

Finally, the total number of fixations, saccades, and total gaze path lengths (in

px) are presented.

Modeling Languages

Category Item BPMN eGantt EPC Petri Net

Difficulty

Easy

Subjects Number of Subjects 29 30 30 30

Comprehension

Comprehension Duration 35270 31840 36120 36930

Response Time 6210 5710 4890 6580

Answering Score 0.66 0.58 0.92 0.78

Eye Tracking

Number of Fixations 112 105 103.93 110

Number of Saccades 101 94 89 95

Gaze Path Length 19958 14858 15169 19128

Difficulty

Medium

Subjects Number of Subjects 29 30 28 28

Comprehension

Comprehension Duration 53910 34860 53100 49170

Response Time 7640 6160 7790 8290

Answering Score 0.62 0.74 0.75 0.63

Eye Tracking

Number of Fixations 191 119 171 151

Number of Saccades 180 108 153 133

Gaze Path Length 35682 19653 26442 21562

Difficulty

Hard

Subjects Number of Subjects 28 29 27 28

Comprehension

Comprehension Duration 68940 58270 86520 76860

Response Time 9170 8360 8240 8230

Answering Score 0.27 0.53 0.54 0.23

Eye Tracking

Number of Fixations 230 169 278.93 282

Number of Saccades 215 146 252 254

Gaze Path Length 41503 20556 40602 52377

Table 1: Obtained Experimental Results

The results indicate that, with rising level of model difficulty, overall com-

prehension performance is decreasing. In particular, the duration time needed

for model comprehension increases, in this context. Furthermore, the response

times for answering comprehension questions increase as well with rising level

of model difficulty, whereas the corresponding answering scores decrease with

rising level of difficulty. Finally, the total number of fixations and saccades in-

crease, depending on the level of difficulty. Hence, the lengths of gaze paths are

increasing as well in this context.

Fig. 2 presents selected evaluation screenshots of the used SMI BeGaze soft-

ware. The evaluation provides information about fixations and saccades of the

1Sample material downloadable from:

www.dropbox.com/sh/our1qp7vkpv020i/AABr3a24DwCKjWAU_2DDCIWMa?dl=0

Experiments on Process Model Comprehension: Lessons Learned 5

difficult eGantt process model. Thereby, circles represent subjects fixations. The

size of a circle, in turn, corresponds to the subjects dwell time. Finally, the

concatenation of fixations and saccades generates the gaze path.

Accumulation of

Fixations

(a) Gaze Path of Subject 1

Prominent

Fixation Points

(b) Gaze Path of Subject 2

Fig. 2: Examples of Subjects Gaze Path

In Fig. 2 (a), several accumulations of fixations on specific areas of the process

model become visible. In Fig. 2 (b), in turn, prominent fixation points can be

identified. To be more precise, the corresponding subject spent much time at

these points. Furthermore, Fig. 3 presents the results we obtained when analyzing

specific areas of interests of a process model. It further indicates the complexity of

analyzing eye tracking data. In particular, such analysis allows for an extensive

evaluation of eye movements. For example, the number of fixations is higher

in areas of interests comprising XOR gateways compared to areas with AND

gateways. Moreover, the XOR represented by the area of interest XOR 2 contains

more fixations and highest average dwell time.

3 Lessons Learned

This section discusses the lessons learned during the eye tracking experiments

in which we compared different process modeling languages. In particular, these

lessons are grouped into nine categories C1 - C9 (cf. Fig. 4).

C1 - Familiarity with Process Scenarios

The kind of scenario considered in the context of an experiment might influ-

ence experimental outcomes as it might be easier for individuals to deal with

scenarios from an application domain they are familiar with (e.g., pizza delivery

vs. bomb defusing). Accordingly, with increasing familiarity with the scenario,

6 M. Zimoch et al.

XOR_2

Fig. 3: Defined Areas of Interest in a Process Model

the cognitive load in the working memory might become lower. By contrast,

if individuals are unfamiliar with a process scenario, they first need to get an

overview of the scenario they are confronted with. Consequently, comprehending

process models related to a scenario an individual is unfamiliar with requires a

higher cognitive load. To reduce this familiarity bias in the experiments we con-

ducted, the subjects were confronted with process models representing different

scenarios. For example, subjects needed about the same time for comprehending

a process model describing a shopping process (cf. Table 1; BPMN - Medium)

and a process dealing with the editing of a wikipedia article (cf. Table 1; EPC -

Medium); i.e., for these two scenarios, no differences could be observed. However,

if the process models are exceeding a certain level of model difficulty, in turn, the

working memory of individuals might be confronted with an information over-

load resulting in a reduction of overall understanding. For example, the more

complex BPMN model describes a pizza delivery process, i.e., a process which

can be considered as well-known. However, the subjects were facing difficulties

regarding the comprehension of respective model that might be owed to the level

of model difficulty.

C2 - Understanding of Process Models

In general, the understanding of process models (i.e., process model comprehen-

sion) is a complex matter as the information contained in these models need

to be decoded and captured by an individual. Consequently, comprehension

constitutes a cognitive process trying to establish relations between available

information on objects and events in the long term memory, together with in-

Experiments on Process Model Comprehension: Lessons Learned 7

C1 – Familiarity with Scenarios

C8 - Individuals

C9 - Measurement

Methods

IndustryHealthcare

C2 - Understanding C3 - Modeling

Education

Reading Writing

Encode information by reading

process models

Decode information by creating

process models

C5 - Process Modeling Languages

Use established process modeling

languages, like BPMN 2.0, GANTT, or EPC

BPMN 2.0 Petri NetsEPCs

C6 - Basic Modeling Elements

Atomic elements for specific process modeling

notation, like tasks, events, or data elements

Task Control Flow

Data Flow

C7 - Modeling Constructs

Combine basic elements to more complex constructs,

like gateways, loops, or communication between actors

XOR / AND Gateways

Messages for Communication

C4 - Structure & Layout

Structural characteristics of the model (e.g., block-structured vs.

hierarchical) or the layout of the control flow (e.g., left-to-right) A B C

Use of process models in the context of specific

process scenarios and related familiarity

Skill Level

Emotions

Cognitive

Biases

Eye Tracking

Smart Sensors

Heart Rate

Questionnaires

Fig. 4: Categories Describing the Experimental Setting

formation perceived at the moment from the sensory, working, or short term

memory. Concerning process model comprehension, individuals must handle the

complexities of parsing the relevant syntactic, semantic, and pragmatic informa-

tion of a process model expressed in terms of a particular language. The easier

and clearer this information is presented, the more positive will be the impact

on process model comprehension (e.g., Cognitive Load Theory (CLT)). In our

experiments, several policies for comprehending a process model could be iden-

tified. Independent from the experience a subject has with process modeling,

all policies are similar in the first comprehension iteration (i.e., after having a

first glance at the process model). More precisely, subjects visually discover all

elements of a process model in an element-to-element procedure. Usually, this

procedure begins with the start element of a process model. During the second

comprehension iteration, subjects follow different policies (e.g., jumping back

and forth between specific modeling constructs or elements). For this reason,

the comprehension questions might serve as an indicator to evaluate the effi-

ciency of the comprehension policies. The identification of concrete patterns for

process model comprehension will be addressed in future work.

C3 - Modeling of Process Models

The modeling of processes deals with the encoding of information of a process

model. This activity, in turn, involves various factors as well as specific cognitive

processes that can be neglected in process model comprehension. The conducted

experiments so far, focused on the comprehension of process models. However,

the lessons learned in this context might apply to process modeling as well.

8 M. Zimoch et al.

C4 - Structure & Layout

The processes were presented as flat (i.e., non-modularized) models to the sub-

jects; i.e., no sub-processes were used. Additionally, all models were block- struc-

tured [41], which fits well with one of the seven process modeling guidelines [16].

In general, one needs to investigate to what extent a particular structuring of

a process model influences its comprehensibility. Furthermore, all process mod-

els were created in a way to be either read from left-to-right or top-to-bottom.

However, one of the process models (i.e., Petri Net - Hard) was designed using a

very ramified structure (i.e., sequence flows running in all directions). Regarding

the results depicted in Table 1, it is unclear whether or not ramified structures

affect the comprehension of process models.

C5 - Process Modeling Languages

Table 2 presents the answering scores (i.e., means) for all considered process

models and their respective levels of difficulty.

Modeling Languages

BPMN eGantt EPC Petri Net

Difficulty

Easy 0.66 0.58 0.92 0.78

Medium 0.62 0.74 0.75 0.63

Hard 0.27 0.53 0.54 0.23

Table 2: Answering Scores Obtained in the Experiments

The results indicate that the comparison of process modeling languages with

respect to model comprehension allows for interesting insights. For example, one

might expect that the answering scores are decreasing with rising level of model

difficulty. Interestingly, in this context, the results related to eGantts constitute

a counterexample, i.e., an increase of the answering scores for the process models

with an easy and medium level of difficulty can be observed. Moreover, one might

expect that for all process modeling languages a comparable decrease can be

observed with rising level of difficulty. However, regarding the BPMN answering

scores obtained from the process models reflecting an easy and medium level of

difficulty, the results are different in orders of magnitude compared to EPC and

Petri Net.

C6 - Basic Modeling Elements

During the experiments it turned out that process models with an explicit start

and an end symbol foster process model comprehension. Initially, subjects are

trying to locate a start symbol in the process model. Usually, the start symbol

is assumed to be on the left or upper left side of the process model. However, if

subjects are unable to identify a start symbol on the assumed positions in the

process model, their gaze paths become directionless, due to the search of the

start symbol. The same effect can be observed with respect to end symbols.

Experiments on Process Model Comprehension: Lessons Learned 9

C7 - Modeling Constructs

As opposed to basic elements, more complex modeling constructs (e.g., gate-

ways) seem to be difficult for individuals. In the experiments, the main challenge

subjects were facing concerns the identification of the semantic meaning of the

presented modeling constructs (e.g., AND gateways). A common approach was

to identify the meaning of a construct by considering the described process sce-

nario in detail. Furthermore, split-and-join gateways (i.e., XOR) appear to be

particular challenging for subjects. Referring back to Fig. 3, Fig. 5 presents a

binning chart showing the proportion of fixations over the duration needed for

process model comprehension. Fig. 5 indicates that subjects spend more time

with studying the first gateway (i.e., the first gateway along the reading direc-

tion) compared to the subsequent other. The same effect can be observed in the

binning charts of other process models. In this context, it makes no differences

whether an AND or an XOR gateways appears first. As a next step, we want

to provide an extensive and direct comparison between the specific modeling

constructs (i.e., AND vs. XOR) of the respective languages as well as their effect

on model perception and interpretation of individuals.

Legend

AND_1

XOR_1

XOR_2

XOR_3

XOR_4

END_AND

20 40 60 80 100 120 140

Comprehension Duration (s)

0.2 0.4 0.6 0.8 1

Proportion of Fixations

Fig. 5: Binning Chart for Gateways in a Process Model

C8 - Individuals

The experiment revealed that the used process models, which were expressed in

different process modeling languages, can be intuitively understood by subjects,

independent from their modeling experience. In particular, the performance of

the subjects regarding the comprehension of easy process models is satisfactory

(cf. Table 1). However, with increasing level of model difficulty, the performances

of the subjects are decreasing to the same extent. Moreover, we assumed that

subjects being experienced with process modeling are more efficient regarding

10 M. Zimoch et al.

process model comprehension. Finally, subjects without any modeling experience

are facing the same challenges than experienced ones regarding process model

comprehension. The reasons for this might be manifold, ranging from personal

factors (e.g., familiarity with the provided scenario) to modeling factors (e.g.,

process model quality). The identification of those factors that might positively

or negatively influence the individuals regarding process model comprehension

will be subject of future experiments. So far, a decrease of the performance can be

observed with rising level of model difficulty (cf. Table 1). Future experiments

will particularly focus on the cognitive processes of individuals. For example,

considering cognitive psychology (e.g., Split Attention Effect), cognitive biases

(e.g., Framing), or specific emotional states (e.g., Alexithymia).

C9 - Measurement Methods

The use of eye tracking in the context of research on process model compre-

hension has led to tangible results. In the experiments, we obtained valuable

insights into how subjects understand process models that are expressed with

different modeling languages. The evaluation of fixations and saccades as well

as the gaze paths observed during process model comprehension reveal inter-

esting facts on particular policies for process model comprehension. For future

experiments, first of all, a broader distribution of the subjects based on their

experience in process modeling (i.e., novices, intermediates, and experts) needs

to be evaluated. So far, we have only investigated the differences between novices

and intermediates (i.e., individuals with moderate experience in process mod-

eling). However, the involvement of experts might reveal significant differences.

Second, the comprehension questions had to be answered after studying the re-

spective process model without the possibility to reinspect them. Therefore, the

models had to be memorized by subjects bearing the risk that given answers

were guessed due to wrong memorizations. The mental process of memorization,

in turn, raises various issues that need to be considered. We therefore will make

use of the visual Split Attention Effect, presenting the process model and corre-

sponding comprehension questions at the same time. This approach will allow

us to obtain more precise observations regarding areas of interest (cf. Fig. 3).

In addition, it will allow for statements in case the answers to related questions

correlate with a subjects’ gaze path in the defined area of interest. Finally, ba-

sic elements (e.g., activities) and constructs (e.g., AND) should be comparable

across all modeling languages. Therefore, future experiments will focus on mea-

surement methods other than eye tracking, as known from cognitive neuroscience

(e.g., smart sensors) and psychology (e.g., Construal Level Theory).

4 Related Work

This section discusses related work along the presented categories (cf. Fig. 4).

C1 - Familiarity with Process Scenarios. In a study dealing with vari-

ous process model representations (i.e., flattened vs. modularized), [6] found no

evidence that domain knowledge influences process model comprehension. De-

spite different cognitive abilities, learning styles and motives as well as policies

Experiments on Process Model Comprehension: Lessons Learned 11

of individuals, [7] could not confirm an influence of domain knowledge on process

model comprehension. Assessing the use of a particular process modeling lan-

guage, [8] shows that domain-specific modeling experience and knowledge have

no significant effect on the understanding of process models.

C2 - Understanding of Process Models. Regarding process model com-

prehension, considerable research was conducted in the last decade. Comparing

BPMN models with a textual notation (i.e., a written use-case), [14] presents a

significant increase regarding model comprehension. More precisely, when read-

ing the textual models, all individuals show an increase, whereas for BPMN

models, solely the experienced individuals show an increase. [17] investigates

whether there are significant differences in terms of understanding, depending

on the process model representation (i.e., text vs. graphical model). An extensive

discussion of a series of experiments related to process model comprehension is

presented in [34]. Finally, the SEQUAL framework provides various aspects of

process model quality fostering the comprehension of suchlike [36].

C3 - Modeling of Process Models. Common to the work related to

category C3 is its focus on the resulting process model, i.e., the product of process

modeling. [2] evaluates the process of process modeling itself. Furthermore, [15]

focuses on how process models are created. The different steps a process modeler

accomplishes during the creation of process models are discussed in [54].

C4 - Structure and Layout. [48] identifies visual features and metrics

fostering the creation of understandable process models. A set of propositions

on the effects of the notational aspects on the improvement of process model

comprehension is presented in [49]. In turn, [50, 52] discuss how modularity en-

hances process model expressiveness. Taking end user preferences into account,

[53] demonstrates the importance of structuring process models.

C5 - Process Modeling Languages. Several frameworks exist dealing with

the quality issues of different kinds of conceptual models (e.g., process models).

In this context, [4] presents frameworks for evaluating the quality of concep-

tual models. In turn, [35] investigates how different representations affect model

comprehension. In this context, it is shown that modeling languages, which allow

for concurrent activities (i.e., parallel branches) are difficult to understand. In

an empirical investigation, [38] elaborated UML Activity Diagrams as the most

versatile modeling language in the context of process model comprehension. To

be more precise, the latter outperformed the comprehension of models expressed

in terms of EPC or BPMN.

C6 - Basic Modeling Elements. [27] conducted an experiment comparing

the effects different flow directions have on model comprehension. In particular, it

was shown that readers adapt well to uncommon reading directions. In turn, [28]

investigates human understanding of process models, trying to identify influence

factors with respect to ”local” comprehensibility (e.g., activities, sequences) in

process models. A comparison of the understanding of imperative and declarative

process models is presented in [30].

C7 - Modeling Constructs. [21] demonstrates in an experiment that the

interpretation of process models benefits from gateway constructs (i.e., AND),

12 M. Zimoch et al.

due to the perceptual discriminability effect of the latter. Especially, this effect is

evident for complex process models. Specific thresholds for gateway complexity

metrics can be found in [22]. These metrics serve as guidelines for novices to

classify process models in specific level of understandability. in turn, [23] provides

a structural equation model, depicting the relationships between flow orientation

in a process model, quality of process models, and business process redesign

success. The equation model shows that flow orientation constitutes a key factor.

Finally, [24] investigates basic symbol sets of various process modeling languages,

showing that notational deficiencies concerning perceptual discriminability and

semiotic clarity have a negative impact on process model comprehension.

C8 - Individuals. [11] investigates preferences of individuals (i.e., cognitive

styles) regarding alternative process representations, which have a positive effect

on process model comprehension. In the context of business process variability,

an empirical user study shows that neither complexity nor expertise in process

modeling have a significant impact on process variant modeling [12]. Experience

in modeling, as a crucial skill of individuals having a significant impact on the

success of process modeling, is described in [13]. In turn, [25] provides evidence

that the chosen process scenario representation form as well as individuals’ char-

acteristics result in similar levels of understanding, independent from whether

subjects are confronted with familiar or unfamiliar process scenarios.

C9 - Measurement Methods. Eye tracking is increasingly used in research

related to process modeling. [32] shows that reading process models changes pupil

dilation as evidence for higher mental effort. Findings on how eye tracking might

contribute to a deeper understanding of process models can be found in [33]. In

[18], the existing research gap concerning the factors influencing process model

comprehension tasks is investigated using eye tracking. [1] identifies performance

improvement opportunities by determining the performances of individuals re-

garding different types of comprehension tasks. Finally, [9] proposes the use of

visual cues in process models to improve their overall comprehensibility.

Regarding the comparison of process modeling languages, there exists sev-

eral work. A review of process modeling languages can be found in [42]. Using

a generic meta-model as benchmark, [29] evaluates seven modeling languages

and their corresponding concepts. Further, [56] classifies existing process mod-

eling languages. Finally, a literature review of the state-of-the-art on empirical

research on process model comprehension is presented in [55].

5 Summary and Outlook

This paper gave insights into experiences we gathered in and the lessons learned

from experiments on process model comprehension using eye tracking. To obtain

these insights, process models in terms of four different modeling languages (i.e.,

BPMN, eGantt, EPC, and Petri Net) were considered in the experiments. To

structure our discussion on the lessons learned, the gained insights were grouped

into nine categories (cf. Fig. 4). Using this categorization, we are going to con-

duct a series of experiments to enhance the understanding on how the overall

Experiments on Process Model Comprehension: Lessons Learned 13

comprehension of process models can be fostered. Additionally, we will focus on

process model creation (i.e., the process of process modeling). Thereby, particu-

lar emphasis will be put on human factors, especially on the cognitive processes

involved. Altogether, using eye tracking for comparing different process model-

ing languages offers valuable insights into how process models are comprehended

by individuals.

References

1. Petrusel, R., Mendling, J., Reijers, H.A.: Task-Specific Visual Cues for Improving

Process Model Understanding. In: Inf and Soft Tech. Vol 79. (2016) 63–78

2. Martini, M., Pinggera, J., Neuratuer, M., Sachse, P., Furtner, M., Weber, B.: The

Impact of Working Memory and the Process of Process Modelling on Model Qual-

ity: Investigating Experienced versus Inexperienced Modellers. In: Scientific Re-

ports. Vol 6. (2016)

3. Kiepuszewski, B., Hofstede, A.H.M.T., Bussler, C.: On Structured Workflow Mod-

elling. In: Proc 12th Int’l Conf on Adv Inf Sys Eng. (2000) 431–445

4. Moody, D.L.: Theoretical and Practical Issues in Evaluating the Quality of Con-

ceptual Models: Current State and Future Directions. In: Data & Knowledge En-

gineering. Vol 55. (2005) 243–276

5. Mendling, J., Reijers, H.A., Cardose, J.: What Makes Process Models Understand-

able?. In: Proc 12th Int’l Conf on Business Process Management. (2007) 48–63

6. Turetken, O., Rompen, T., Vanderfeesten, I., Dikici, A., van Moll, J.: The Effect

of Modularity Representation and Presentation Medium on the Understandability

of Business Process Models in BPMN. In: Proc BPM’16. (2016) 289–307

7. Recker, J., Reijers, H.A., van de Wouw, S.G.: Process Model Comprehension: The

Effects of Cognitive Abilities, Learning Style and Strategy. In: Commun Assoc Inf

Syst. Vol 34. (2014) 199–222

8. Recker, J., Dreiling, A.: Does it Matter which Process Modelling Language we

Teach or Use? An Experimental Study on Understanding Process Modelling Lan-

guages Without Formal Education. In: Proc ACIS’07. (2007) 356–366

9. Petrusel, R., Mendling, J., Reijers, H.A.: How Visual Cognition Influences Process

Model Comprehension. In: Decision Support Systems. (2017)

10. Bera, P.: Does Cognitive Overload Matter in Understanding BPMN Models?. In:

J Computer Information Systems. Vol 52. (2012) 59–69

11. Figl, K., Recker, J.: Exploring Cognitive Style and Task-Specific Preferences for

Process Representations. In: Requir Eng. Vol 21. (2016) 63–85

12. D¨ohring, M., Reijers, H.A., Smirnov, S.: Configuration vs. Adaptation for Business

Process Variant Maintenance: An Empirical Study. In: Information Systems. Vol

39. (2014) 108–133

13. Bandara, W. Gable, G.G., Rosemann, M.: Factors and Measures of Business Pro-

cess Modelling: Model Building Through a Multiple Case Study. In: Eur J Inf Sys.

Vol 14. (2005) 347–360

14. Ottensooser, A., Fekete, A., Reijers, H.A., Mendling, J., Menictas, C.: Making

Sense of Business Process Descriptions: An Experimental Comparison of Graphical

and Textual Notations. In: J Syst Soft. Vol 85. (2012) 596–606

15. Weber, B., Pinggera, J., Neurauter, M., Zugal, S., Martini, M., Furtner, M., Sachse,

P., Schnitzer, D.: Fixation Patterns During Process Model Creation: Initial Steps

Toward Neuro-Adaptive Process Modeling Environments. In: Proc 49th Int’l Conf

on System Sciences. (2016) 600–609

14 M. Zimoch et al.

16. Mendling, J., Reijers, H.A., van der Aalst, W.M.P.: Seven Process Modeling Guide-

lines (7PMG). In: Inf and Soft Tech. Vol 52. (2010) 127–136

17. Rodrigues, R.D.A., Barros, M.D.O., Revoredo, K., Azevedo, L.G., Leopold, H.: An

Experiment on Process Model Understandability Using Textual Work Instructions

and BPMN Models. In: 29th Brazilian Symp on Soft Eng. (2015) 41–50

18. Petrusel, R., Mendling, J.: Eye-Tracking the Factors of Process Model Compre-

hension Tasks. In: Proc 25th Int’l Conf on Adv Inf Sys Eng. (2013) 224–239

19. Haisjackl, C., Barba, I., Zugal, S., Soffer, P., Hadar, I., Reichert, M., Pinggera, J.,

Weber, B.: Understanding Declare Models: Strategies, Pitfalls, Empirical Results.

In: Soft & Sys. Vol 15. (2016) 325–352

20. Riedl, R., Banker, R.D., Benbasat, I. Davis, F.D., Dennis, A.R., Dimoka, A., Gefen,

D., Gupta, A., Ischebeck, A., Kenning, P.: On the Foundations of NeuroIS: Reflec-

tions on the Gmunden Retreat 2009. In: Comm Assoc Inf Sys. Vol 27. (2012)

21. Recker, J.: Empirical Investigation of the Usefulness of Gateway Constructs in

Process Models. In: Eur J Inf Syst. Vol 22. (2013) 673–689

22. S´anachez-Gonz´alez, L., Garcia, F., Ruiz, F., Mendling, J.: Quality Indicators for

Business Process Models from a Gateway Complexity Perspective. In: Inf Softw

Tech. Vol 54. (2012) 1159–1175

23. Kock, N., Verville, J., Danesh-Pajou, A., Deluca, D.: Communication Flow Orien-

tation in Business Process Modeling and its Effect on Redesign Success: Results

from a Field Study. In: Decis Support System. Vol 45. (2009) 562–575

24. Figl, K., Mendling, J., Strembeck, M.: The Influence of Notational Deficiencies on

Process Model Comprehension. In: J Assoc Inf System. Vol 14. (2013) 312–338

25. Recker, J. Dreiling, A.: The Effects of Content Presentation Format and User Char-

acteristics on Novice Developers Understanding of Process Models. In: Commun

Assoc Inf Syst. Vol 28. (2011) 65–84

26. Figl, K., Recker, J., Mendling, J.: A Study on the Effects of Routing Symbol Design

on Process Model Comprehension. In: Decis Supp Sys. Vol 54. (2013) 1104–1118

27. Figl, K., Strembeck, M.: Findings from an Experiment on Flow Direction of Busi-

ness Process Models. In: Int’l Workshop on EMISA’15. (2015) 59–73

28. Figl, K., Laue, R.: Influence Factors for Local Comprehensibility of Process Models.

In: Int’l J Hum Comput Stud. Vol 82. (2015) 96–110

29. List, B., Korherr, B.: An Evaluation of Conceptual Business Process Modelling

Languages. In: Proc 2006 Symp Appl Comp. (2006) 1532–1539

30. Pichler, P., Weber, B., Zugal, S., Pinggera, J., Mendling, J., Reijers, H.A.: Imper-

ative versus Declarative Process Modeling Languages: An Empirical Investigation.

In: Business Process Management Workshop (LNBIP). Vol 99. (2012) 383–394

31. Melcher, J., Mendling, J., Reijers, H.A., Seese, D.: On Measuring the Understand-

ability of Process Models. In: Business Process Management Workshop (LNBIP).

Vol 43. (2010) 465–476

32. Dobesova, Z., Malcik, M.: Workflow Diagrams and Pupil Dilatation in Eye Tracking

Testing. In: Proc 13th Int’l Conf on Emerging eLearning Techniques & Applica-

tions. (2015) 59–64

33. Hogrebe, F., Gehrke, N., N¨uttgens, M.: Eye Tracking Experiments in Business Pro-

cess Modeling: Agenda Setting and Proof of Concept. In: Proc 4th Int’l Workshop

on Enterprise Modelling and Information Systems Architectures. (2011) 183–188

34. Mendling, J., Strembeck, M., Recker, J.: Factors of Process Model Comprehension

- Findings from a Series of Experiments. In: Decis Support Syst. (2012) 195–206

35. Weitlaner, D., Guettinger, A., Kohlbacher, M.: Intuitive Comprehensibility of Pro-

cess Models. In: S-BPM ONE-running Processes. Vol 360. (2013) 52–71

Experiments on Process Model Comprehension: Lessons Learned 15

36. Krogstie, J.: Model-Based Development and Evolution of Information Systems.

Springer (2012)

37. Laue, R. Gadatsch, A.: Measuring the Understandability of Business Process Mod-

els: Are we Asking the Right Questions?. In: Business Process Management Work-

shop (LNBIP). Vol 66. (2011) 37–48 ¡

38. Joˇst, G., Huber, J. Heriˇcko, M. Polanˇciˇc, G.: An Empirical Investigation of Intuitive

Understandability of Process Diagrams. In: Comp St Int. Vol 48. (2016) 90–111

39. SMI: iView X Hi-Speed, http://www.smivision.com/en/gaze-and-eye-tracking-

systems/products/iview-x-hi-speed.html, last visited on 2017/10/02 (2016)

40. Majaranta, P., Aoki, H., Donegan, M., Hansen, D.W., Hansen, J.P., Hyrskykari,

A., R¨aih¨a, K.: Gaze Interaction and Applications of Eye Tracking: Advances in

Assistive Technologies. IGI Global (2011)

41. Reichert, M., Dadam, P.: Adeptflex - Supporting Dynamic Changes of Workflows

Without Losing Control. In: J of Int Inf Sys. Vol 10. (1998) 93–129

42. Wang, W., Ding, H., Dong, J., Ren, C.: A Comparison of Business Process Model-

ing Methods. In: Int’l Conference on Service Operations and Logistics, and Infor-

matics. (2006) 1136-1141

43. Salvucci, D.D., Goldberg, J.H.: Identifying Fixations and Saccades in Eye-Tracking

Protocols. In: Proc 2000 Symp on Eye Tracking Research & App. (2000) 71-78

44. OMG: Business Process Management & Notation 2.0 www.bpmn.org, last visited

on 2016/11/10 (2016)

45. van der Aalst, W.M.P.: Formalization and Verification of Event-driven Process

Chains. In: Inf & Soft Tech. (1999) 639-650

46. Person, J.L.: Petri Net Theory and the Modeling of Systems. Prentice Hall (1981)

47. Sommer, M.: Zeitliche Darstellung und Modellierung von Prozessen mithilfe von

Gantt-Diagrammen. Bachelors Thesis, Ulm University (2012)

48. Bernstein, V., Soffer, P.: Identifying and Quantifying Visual Layout Features of

Business Process Models. In: Int’l Conf on Enterprise, Business-Process and Infor-

mation Systems Modeling. (2015) 200–213

49. Schrepfer, M., Wolf, J., Mendling, J., Reijers, H.A.: The Impact of Secondary No-

tation on Process Model Understanding. In: The Practice of Enterprise Modeling,

2nd IFIP WG8.1 Working Conf. (2009) 161–175

50. Zugal, S., Soffer, P., Haisjackl, C., Pinggera, J., Reichert, M., Weber, B.: Investi-

gating Expressiveness and Understandability of Hierarchy in Declarative Business

Process Models. In: Soft & Sys Mod. Vol 14. (2015) 1081–1103

51. van der Linden, D., Zamansky, A., Hadar, I.: How Cognitively Effective is a Visual

Notation? On the Inherent Difficulty of Operationalizing the Physics of Notations.

In: Int’l Workshop on BPMDS. (2016) 448–462

52. Reijers, H.A., Mendling, J., Dijkman, R.M.: Human and Automatic Modulariza-

tions of Process Models to Enhance their Comprehension. In: J Inf Sys. Vol 36.

(2011) 881–897

53. Koschmider, A., Reijers, H.A., Dijkman, R.: Empirical Support for the Usefulness

of Personalized Process Model Views. In: Multikonf Wirtschaftsinformatik (2012)

54. Claes, J., Vanderfessten, I., Pinggera, J., Reihers, H.A., Weber, B., Poels, G.: A

Visual Analysis of the Process of Process Modeling. In: Inf Sys and e-Business

Management. Vol 13. (2015) 147–190

55. Figl, K.: Comprehension of Procedural Visual Business Process Models. In: Busi-

ness & Information Systems Engineering. Vol 59. (2017) 41–57

56. Mili, H., Tremblay, G., Jaoude, G.B., Lefebvre, ´

E., Elabed, L., Boussaidi, G.E.:

Business Process Modeling Languages: Sorting Through the Alphabet Soup. In:

ACM Comput Surv. Vol 43. (2010) 1–56