Investigating the Effects of a Virtual Process Environment on the Comprehension of Business Process Models [original]

Universität Ulm | 89069 Ulm | Germany Faculty of

Engineering, Computer

Science and Psychology

Databases and Information

Systems Department

Investigating the Effects of a

Virtual Process Environment on the

Comprehension of Business Process

Models

Master’s thesis at Universität Ulm

Submitted by:

Carina Spitzer

[email protected]

Reviewer:

Prof. Dr. Manfred Reichert

Prof. Dr. Rüdiger Pryss

Supervisor:

Michael Winter

2020

Version from September 13, 2020

Abstract

Within the scope of Business Process Management and Modeling, gamification is used,

inter alia, to promote process model comprehension and for motivational and educational

purposes. In the context of gamification in Business Process Management, this master

thesis aims to investigate the effects of a virtual process environment on the cognitive

load a process reader perceives during the comprehension of a process model. The

comprehension of process models is essential for the proper modeling of business

processes, and vice versa. In addition to the previous research approaches in terms of

gamification regarding the management and modeling of business processes, this master

thesis takes into account concepts from cognitive research. A study with 72 participants

was conducted online. Thereby, measures of interest were the cognitive load of the

textual process description, the process model and the process model extended with

graphics extracted from the virtual process environment. Therefore, a fractorial desgin

was established as only the process model was extended with static pictures. The virtual

process environment is realized through a video based on a 3D - warehouse scenario

game. As a result, no significant difference in the perceived cognitive load of the process

reader was found between the three process variants. In summary, after experiencing a

virtual process environment, the cognitive load of the process documentations does not

differ significantly. Further analysis has shown that the process reader’s confidence in

the completeness and adequacy of the shown process documentation is associated with

the process document variant. Participants were more confident about the correctness

of the process model extended with graphics.

iii

Acknowledgment

Throughout the writing of this master thesis, I received a great deal of support and

assistance.

First of all, I would like to acknowledge Prof. Dr. Manfred Reichert and Prof. Dr. Rüdiger

Pryss for making this master thesis possible. Furthermore, I would like to thank my

supervisor Michael Winter for his tremendous assistance during the creation of the study

and the master thesis.

Finally, I would also like to thank my friends and most importantly my partner Marc Uran

for their great support and the motivation I received throughout the master thesis.

Contents

1 Introduction 1

1.1 Motivation and Problem Statement . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Fundamentals 5

2.1 Humans Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Comparison of Textual Process Description with Process Models . . . . . 9

2.5 Cognitive Load Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Study Planning and Definition 15

3.1 Context Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Goal Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3 Hypotheses Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.4 Study Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.5 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.6 Risk Analysis and Migrations . . . . . . . . . . . . . . . . . . . . . . . . . 33

4 Study Operation 37

4.1 Study Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.2 Study Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3 Data Validation and Data Set Reduction . . . . . . . . . . . . . . . . . . . 38

5 Study Analysis and Interpretation 43

5.1 Analysis of Raw Data and Descriptive Statistics . . . . . . . . . . . . . . . 43

5.2 Hypotheses Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.3 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

vii

Contents

5.5 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6 Related Work 59

7 Conclusion and Future Work 63

A Process Documentation 77

B Study 83

viii

Introduction

1.1 Motivation and Problem Statement

As the adage goes: "a picture is worth thousands words". Visualization is therefore an

integral part since it can enhance cognitive processes and compensate for cognitive

deficits because information is visually portrayed, as is the case with process models [

Process models are of utmost importance in the management of business processes

since they are utilized for documentation and reengineering purposes [

]. Rapid

reactions to market conditions are indispensable due to increasing competition in mar-

kets, and this presupposes a fundamental management of the business process [

Therefore, in the current state of Business Process Management (BPM), enterprises

concentrate primarily on the establishment of a process architecture and on major pro-

cess redesign [

]. In addition to the utilization of process models in the management

and modeling of business processes, gameful-designs and virtual environments are

an uprising approach in the research and in the practice. Information are displayed

either two- or three dimensionally. The concepts are applied in various domains, such

as for educational purposes [

], the comprehension of process models [

] and the

encouragement and the motivation of domain professionals [

]. This direction of

research is indicated as gamification [

]. As gamification is gaining attention, it is of

interest to examine its impact with respect to diverse theories such as the information

processing, the behavioral decision and the cognitive load theory [

]. Research has

been done so far primarily to investigate the impact on cognitive processes in terms of

gameful-designs [

] or with regard to process models [

]. But how do

a process reader’s cognitive processes get affected by a virtual process environment?

1 Introduction

The question addressed has not been analyzed so far and so the investigation of the

question should lead to new insights as the basic comprehensibility is not yet given.

1.2 Objective

This master thesis seeks to investigate the impact of a virtual process environment on

the process reader’s cognitive processes during the comprehension of a process model.

With respect to cognitive processes, the investigation focuses on the cognitive load

theory (CLT). It is therefore of interest to gain insights as to how a gamification approach

can have an effect on the intrinsic, extraneous and germane cognitive load. The virtual

process environment is realized through a video based on the game reported in [

]. A

study has been conducted for the investigation.

1.3 Structure of the Thesis

This master thesis is structured as follows: Section 2 sets out the fundamentals of human

visual perception, gamification and virtual environment. In addition a comparison of

the textual process description with the graphical process notation and the cognitive

load theory are outlined. The purpose of this chapter is to elucidate the knowledge

indispensable for the study of this thesis. Further, Section 3 is concerned with the

planning and definition of the study. Therefore, the context and objective of the study

is outlined in order to receive insights into the motives for the study. The research

questions and the derived hypotheses are presented in Section 3.2 and in Section

3.3. Furthermore, the study setup including the selection of materials and instruments,

participants and variables, followed by the study design, is outlined in Section 3 as well.

After describing the structure of the study, the risks of validity are discussed in Section

3.6. Section 4 presents the study operation consisting of the preparation and execution

of the respective study and the validation of the data collected. The analysis and the

interpretation of the findings are described in Section 5. Therefore, in Section 5 the

descriptive statistics of the collected data are outlined, the hypotheses are evaluated

1.3 Structure of the Thesis

and subsequently the results of the analysis are discussed and interpreted followed by

the limitations and implications of the study. In Section 6 related work is provided to

gain insights into research in this area. Finally, the master thesis is completed by the

conclusion and discussion of future work.

Fundamentals

Human visual perception in conjunction with virtual environment may be beneficial with

respect to the comprehension of process models. Hence, it is indispensable to introduce

the respective fundamentals.

Section 2 is divided into five parts. First, the fundamentals of the human visual perception

specifically the information processing of virtual environments and graphics is presented

in Section 2.1 respectively in Section 2.2. In Section 2.3 the term gamification is intro-

duced. Subsequently, an outline of the differences between textual process description

and process models will be provided in Section 2.4 and finally the cognitive load theory

will be presented in Section 2.5 as it is essential for the study.

2.1 Humans Visual Perception

The vision science compromises multiple disciplines and thus is interdisciplinary. It

includes perceptual psychology, computational vision, neuroscience and physics [

Visual perception is the key factor in how individuals gain insights and experiences

through vision [22]. In addition, vision is also among hearing, smelling, touch and taste

one of the most strongest senses regarding producing valuable information for the human

mind, its cognition and his further intervention [

]. The visual perception process can

mainly be divided into three parts: sensory recording, encoding, and interpretation [

Sensory Recording:

The prerequisite for vision is light. When the light enters the

human eye, the retina located at the back of the eye forms the incident light currents into

electrical impulses. Light impressions in the environment are projected on the retina

2 Fundamentals

as an upside-down image [

]. As shown in Figure 2.1, the retina consists of diverse

nerve cells such as ganglion cells, intermediate cells (bipolar,amacrine and horizontal

cells) and compromises more than 126 million light-sensitive photoreceptors. These

receptors can be distinguished into 120 million cones and 6 million rods. These two

types are not distributed uniformly across the retina. The cones are responsible for

human color perception (green, red, yellow and blue) while the latter is responsible for

the brightness (black, white and grey). Thus, the daylight vision of human beings is due

to the cones [

]. However, the incident light needs to pass through the ganglion

and intermediate cells in order to reach the light-sensitive photoreceptors (see Figure

2.1). The cones and rods are connected to the intermediate cells and when the light

reaches the photoreceptors the information is processed through neuronal circuits to the

ganglion cells which lead to the encoding phase [23].

Figure 2.1: View of the Structure of Retina: A Cross-Section [23]

Encoding:

The ganglion cells are responsible for transmitting the information through

the optic nerve to the visual cortex [

]. The visual cortex contains inter alia cells

which are, e.g., sensitive to orientation, color and motion. The information is coded by

various brain regions depending on the particular stimulus [22].

Interpretation:

Once the individual sees an object, it will be projected on the retina

two-dimensionally and then processed as perception back to its original form [

]. First

2.2 Virtual Environment

of all, the raw materials need to be processed in order to gain perception as an output.

For this cognitive processes are essential, e.g., for the spatial and depth perception, the

recognition, categorizing and delimitation of objects. Various theories of perception are

present in the literature [

]. They can be distinguished into top-down processing and

bottom-up processing theories. The latter is also known as data-driven processing. The

theory supposes perception begins with the incident light on the receptors transmitting

it to the visual cortex, i.e., cognition starts with the stimuli. Top-down processing relies

on the assumptions that in order to achieve the cognitive output previous experience,

knowledge, etc. is essential to process the stimuli on the retina [25].

2.2 Virtual Environment

After an introduction to the visual perception process an outline about visual perception

in the context of a virtual environment will be provided. The term virtual environment is

defined by [

] as a computer-generated environment which is embedded in the physical

environment. Inside the virtual environment users are able to communicate, interact or

collect information. Furthermore, virtual worlds are a subset of virtual environments.

Virtual worlds are self-contained and were first only used in a game-based context,

describing Massively Multiplayer Online Role-Playing Games (MMORPGs) like Elder

Scrolls Online [

]. The definition of virtual world can be referred back to [

]

initiating a discourse for a common definition. Therefore the author stated, virtual worlds

are a "synchronous, persistent network of people, represented as avatars, facilitated by

networked computers" [

]. However, it needs to be differentiated from the term virtual

reality, since virtual reality applies to mechanics (e.g., data gloves) that humans can use

in order to interact with the environment over multiple sensory channels. Thus virtual

reality can be integrated into virtual worlds [

]. Virtual reality is used, for example,

in education, entertainment, medicine or in the military [31].

Individuals first communicated through pictorial languages. It is easier to describe

objects (e.g., house, sun, car) using visualization than words. Apart from this, the human

vision system is considered to have the most powerful information processing capability.

2 Fundamentals

Graphic information processing is carried out precociously thus using graphics will ease

the human mind. Visualized information is processed more efficiently than non-visualized

information. Furthermore, visualizing information in the form of computer graphics

would also allow for human pattern recognition, as it identifies patterns in graphics

[

]. It was empirically shown that playing games enhance information processing,

i.e., foster cognitive skills [

]. Finally, visualization in the form of computer graphics

as a virtual world consists of, enables to represent a vast amount of spatial and non-

spatial information requiring only a small area. Human cognition will be eased since

those certain bits of information are offloaded to the preconscious perception system.

Thus, computer graphics containing useful information can be used as an external

memory since the vision system is a high-bandwidth channel, i.e., enabling the efficient

processing of a vast amount of visual information in parallel [

]. Besides the fact

of fast information processing, visualization also supports cognition in the form as a

deceptive tool and increased storage, as allowing interaction with graphics enabling

experience to the user and visualized information can be stored efficiently in a high

amount [1].

2.3 Gamification

The consulting- and auditing company PriceWaterhouseCoopers (PwC) developed

Multipoly Next, an online simulation game. It was used as a virtual interview application

process. The participants were placed in the role of a trainee and encountered several

scenarios (e.g., workplace scenarios and completing tests scenarios). Throughout the

game, the participants were able to gain points. Afterwards, the ranking list was used

to choose the winners who will progress to the final round [

]. In this manner, PwC

considered gamification aspects while developing its game.

Gamification connotes using game-design elements in non-game environments in order

to induce motivation and promote contribution, skills, creativity and productivity of a user

[

]. Thereby, game mechanics (e.g., badges, leader boards, points, levels) are used

to evoke certain effects on a user. The way the mechanics affect the users is known as

2.4 Comparison of Textual Process Description with Process Models

game dynamics [

]. As an example: To arouse competition among the users, ranking

lists can be used. The primary reason for the competition taking place is to achieve social

recognition. Further, implementing task or quests is seen as a challenge, promoting

cognitive stimulation [

]. Motivation can be distinguished into intrinsic and extrinsic

motivation. Intrinsic motivation emerges within humans inner selves, such as the pursuit

of being successful, social interaction and completing fulfilling tasks whereas the latter

depends on achieving extrinsic goals surrounding in the environment (e.g., monetary

rewards). Extrinsic motivation, however, is not fulfilling on the long term [

]. Game-

elements awaken intrinsic motivation and hence deliver intrinsic rewards. These rewards

lead to a long-term satisfaction [

]. According to the literature review by [

] the majority

of the reviewed empirical studies stated that the participants were positively influenced

by gamification, i.e., using gamification provided benefits and positive effects.

2.4 Comparison of Textual Process Description with Process

Models

Business Process Management focuses on identifying, reengineering, optimizing and

monitoring business processes by using concepts and techniques such as process

models and process mining. A business process consists of a set of activities being in

chronological and logical order. It produces an output which is in line with the business

goals of the company. Further, business processes require resources like staff, raw

materials and data as a prerequisite to begin [2, 3].

Business Process Modeling is an integral part of Business Process Management and its

life cycle. Identified business processes are documented in terms of process models.

Afterwards, the process model may be used as a basis for redesigning the business

process in order to achieve a better performance of the business operations [

]. Process

models visualize business processes on an abstract level, only the relevant activities and

resources are included and thereby reduces the complexity of the real-scenario [

Hence, process models are used to document the business processes of an enterprise

and are also used to improve the communication in particular among stakeholders [

2 Fundamentals

There is a broad research present comparing process models with textual processes.

A popular graphical process modeling language is Business Process Modeling and

Notation (BPMN) [

]. BPMN was developed by Object Management Group with the

objective to provide a standard that is comprehensible by business users of various

domains (i.e., technical and non-technical business users) [41].

Section 2.2 provided insights on how visualization supports cognition (e.g., parallel

processing and external memory). Besides virtual worlds, process models are as well

graphically represented using rectangles, circles and symbols (e.g., BPMN) [

]. Thus,

process models facilitate the cognitive system by offloading the visualized information to

the perception system as well (see Section 2.2). A study conducted by [

] compared

process models with text-based process descriptions. According to the results, process

models promote the comprehensibility in particular for expert users, for non-experts no

significant change has been found which implicates that both methods (process models

and text) influence the comprehension equally. Thus, knowing the notation was not

sufficient to primary understand what was modeled. However, the findings of a study

conducted by [

] comparing BPMN with written use cases indicate that using first textual

and afterwards showing the graphical notation of the business process maximizes the

comprehension regardless of the knowledge (e.g., business analyst, end-user) and the

textual aptitude. Having experience in the graphical notation facilitates the understanding

in the model whereas textual aptitude leads to a better comprehension of the textual

notation. Therefore, the result supports the assertion by [

] that notations need to be

learned in order to make a meaningful contribution from it.

In conclusion, there are various criteria that need to be considered when comparing

textual notation with graphical notation. From a cognitive perspective, visualization

processes information compared to the cognitive system more efficiently [

]. In addition,

the textual information processing is linear whereas the elements of a graph notation

(e.g., artifacts, activities, control flow) can be spatially distributed which as well leads to

an efficient information processing [46]. However, it is also important to consider which

process modeling language is being compared to textual descriptions. Therefore, textual

descriptions performed better than Petri nets with respect to the comprehension [

Consequently, process modeling languages can have different effects on comprehension.

2.5 Cognitive Load Theory

An overview of the performance of various modeling languages on cognitive processes

delivers for example [

]. Furthermore, there exist various business processes with a

multitude of complexity. Which notation should be applied, eventually depends on the

facts of the case. Loops may be better presented by graph notation whereas texts are

ideal for exceptional cases [

]. Concerning the situation in the practice, [

] for example,

evaluated which notation (textual or process model) is current predominantly present

regarding software development processes. The results of the evaluation, consisting of

multiple interviews, showed that the companies predominantly represent its processes

graphically. Textual or tabular representation is mainly used in addition to describing the

details of a process. One assumption of the authors for using graphical representation

is that it is more time-efficient since reading the whole textual process description will

consume time whereas a process model delivers an overall view of the process at a

glance. Further, textual descriptions, i.e., work instructions, can be linked to individual

activities within the process model enriching it with more details. Therefore, they can

be used in addition to validating process models especially for business users such as

domain experts, who do not feel familiar with the graphical notation but have extensive

knowledge of the visualized process [

]. However, establishing both methods

for describing a business process entails the risk of inconsistencies between them

[

]. [

] delivers an approach for detecting inconsistencies that occur between both

representations, further [

] introduces a technique transforming process models into

natural language.

2.5 Cognitive Load Theory

The roots of the cognitive load theory can be traced back to the researches of [

] and

[

] and serves as a framework for educational research with respect to the cognitive

processes and in particular to the human cognitive architecture. It explains the occurring

cognitive load of novel information in terms of learning and proposes suggestions about

how to alter the cognitive load. According to [

] knowledge can be categorized as bi-

ologically primary and biologically secondary knowledge traced back to the evolutionary

theory. An individual acquires primary knowledge automatically and without any instruc-

2 Fundamentals

tions given (e.g., learning the native language and basic social relations) due to evolution

trough a multitude of generations, whereas the secondary knowledge is acquired through

the cultural necessity of adaptation, such as reading and writing skills. These skills

are essential for interaction with society. Further, the biologically secondary knowledge

requires instructions to emerge respective secondary skills. The cognitive load theory,

therefore, considers secondary knowledge, as this theory comprises cognitive load in

terms of instructions [56].

The working memory and the long-term memory are components of the human cognitive

architecture [

]. Novel information is first absorbed by the working memory before

transmitting it to the long-term memory, thereby the working memory serves as a

mediator between the external world and the long-term memory [

]. However, the

human working memory’s inclusion of information is limited by its capacity [

] and its

duration [

] whereas the long-term memory is virtually unlimited [

]. In conclusion, if

the amount of information conveyed by an instruction exceeds the working memory’s

capacity for information processing, the unprocessed information will not be transmitted

to the long-term memory which negatively means the learning. The theory, therefore,

provides advice about how to construct instructions considering the mental effort, i.e.,

cognitive load. The amount of cognitive load contained in an instruction is also related

to the peculiarity of the types of cognitive load. Hence, the cognitive load is built upon

three major categories: the intrinsic cognitive load, the extraneous cognitive load, and

the germane cognitive load [56].

Intrinsic Cognitive Load (ICL):

The intrinsic cognitive load comprises the cognitive

load that information (i.e., instruction) contains itself. Thus, it depends heavily on the

complexity of the material. It is affected by the interaction of the information fragments,

i.e., the element-interactivity. The higher the elements are interconnected the greater is

the resulting intrinsic cognitive load and the working load. It is due to the fact that these

interconnected fragments of information cannot be progressed separately, but have to

be progressed at the same time in order to understand the entire information of the

material [

]. The intrinsic cognitive load is said to be invariable, given some expertise,

as it refers to the complexity of the information. Further, the extent of the intrinsic

cognitive load is attributable to the prior knowledge of the learner as well. Therefore,

2.5 Cognitive Load Theory

extensive prior knowledge leads to less intrinsic cognitive load as the knowledge is

stored and retained in the long-term memory as a schema enabling to treat it as a

single information fragment. Hence, a novice would perceive a higher cognitive load

in terms of the complexity than an expert. Furthermore, it needs to be differentiated

between the comprehension and the root learning of materials. A material with high

complexity may convey low intrinsic cognitive load if the information is memorized solely

as its element-interactivity is low, whereas the understanding of a material predominately

results in high element-interactivity [56].

Extraneous Cognitive Load (ECL):

The extraneous cognitive load is induced by the

way the learner is given the information. Therefore, it is affected by the design of the

instructions and thus is reduced by altering its representation. The element-interactivity

also exerts an impact on the extraneous cognitive load as the instruction may require

additional processing of various information simultaneously. This form of cognitive load

shall always be minimized as much as possible since it is not relevant to the learning

material itself [

]. [

] stated several recommendations in terms of the reduction of

the extraneous cognitive load. The respective load may, therefore, be reduced, for

example, by physically agglomerating coherent information or by presenting the text and

its respective graphics in the same area.

Germane Cognitive Load (GCL):

The germane cognitive load comprises the cognitive

load which will emerge if the learner deals intensively with the material and thus enables

productive learning [

]. Information is stored in the long-term memory as schemata.

These schemes are used to handle multiple information as one. The learning process

enhances the formation and automation of a schema [

]. Additional resources resulting

from the minimizing of the extraneous load will be used to enhance the acquisition of

schemata and automation which is subject to the germane load. In conclusion, the

extraneous load shall be reduced as far as possible in order to enable high germane

load [

]. The germane load provides positive effects as opposed to the intrinsic and

extraneous load.

The total amount of cognitive load that the learner perceives is formed by the intrinsic

and the extraneous load. These are considered as independent categories, whereas

2 Fundamentals

the germane cognitive load depends on the intrinsic cognitive load. In conclusion, if a

material’s cognitive load is greater than the working memory’s capacity, this will adversely

affect the learning process. Thus, the extraneous load should be reduced so that the

additional freed resources could be used for deep learning [53].

Study Planning and Definition

Empirical research such as studies serves the purpose of investigating the relationship

between certain observed quantities. It is of utmost importance to define the parameters

to ensure a successful experiment and afterwards to be able to draw valid conclusions.

Therefore, it is essential to carry out extensive planning in advance.

Section 3 is organized as follows: Section 3.1 defines the studies context. Section 3.2

describes the goals of the study. Afterwards, in section 3.3 the formalized hypothesis

are presented. In Section 3.4 the study setup is explained followed by the explanation

of the study design in Section 3.5. In Section 3.6 the consisting risks of the study are

evaluated.

3.1 Context Selection

Within Business Process Management gamification is spanning around in various direc-

tions. IBM Innov8 is a gamification platform developed by IBM teaching the fundamentals

of Business Process Management in a 3D environment while providing gameful experi-

ence to the users [

]. Several studies showed that implementing IBM Innov8 in education

were positively received by the students, gaining knowledge faster than traditional meth-

ods [

]. Besides the well known IBM Innov8, there exist other simulation games as

well, like imPROVE. Users are able to experience with imPROVE a 3D-real world sce-

nario, the Manchester triage system. Within the game, they are able to model processes

and afterwards simulate their processes while its effects on healthcare and resulting

costs are displayed [

]. Another study conducted by [

] considered a gameful-design

3 Study Planning and Definition

for their sail boat building game in order to increase motivation and engagement by

enabling the chance to fail and to improve throughout the game. The results showed

that the motivation and process management competencies increased. Furthermore,

[

] proposed a notation extension of the business process modeling language BPMN

2.0 considering gamification elements [

]. [

] presents a BPMS-tool combining Green

BPMN and gamification. Employees, for example, will be rewarded with badges based

on how sustainable their designed processes are. Another practical implementation is

Horus Gamification. [

] introduced its prototype. The tool considers Social BPM and

gamification. It implemented game mechanics like badges, leader boards and points

which users can gain through modeling based on process quality of their processes.

Therefore, gamification is used in Business Process Management in various ways: for

educational aspects, extensions of concepts and for practical implementations.

Broad research has been conducted to analyze the understandability of process models

considering aspects such as personal factors [

], the degree of the structured-

ness, the sequentiality, the concurrency and the size of process models [

different modeling notations [

], modularity [

], and the complexity of a

process model [

]. To ensure a high process model quality, which influences the under-

standability, various guidelines and frameworks have been developed [

Furthermore, research on the concept of gamification in Business Process Management

focuses primarily on providing gameful-experience, in particular to foster the performance

of employees and for educational reasons [

]. Only little research has been

conducted to examine the comprehension of a process domain in terms of the use of a

virtual process environment, taking into consideration the cognitive complexity a process

reader may perceive [

] and therefore whether it facilitates the user to comprehend the

respective process model. Thus, a study is being conducted to provide insights into the

stated issue.

3.2 Goal Definition

To investigate the impact of a virtual process environment on the comprehension of

different business process documentation in terms of the cognitive complexity, the

following research question (RQ) is addressed in this thesis:

RQ 1:

Does the application of a virtual environment have an effect on the cognitive load

during the comprehension of different process documentations?

As in Section 2.4 shown, the utilization of a textual process notation or graphical process

notations such as BPMN 2.0 contain diverse strengths and weaknesses. Therefore this

research question does not solely addresses the impact a virtual process environment

may have on the cognitive load of graphical process notations but also comprises the

impact on textual descriptions. Thus, it is of high interest to study whether the impact of

a virtual process environment differentiates depending on the utilized process documen-

tation. It is not uncommon that enterprises maintain graphical and textual notations to

benefit from the strengths of both [

]. Apart from this, another reason to include

the impact on textual notation is that human information processing is influenced by the

characteristics of an individual, also known as cognitive styles. Therefore there exists

inter alia humans that tend to think primary in words and humans who predominately

prefer to think in images [78]. Hence, textual descriptions should not be left out.

Furthermore, it is of high interest to analyze whether the cognitive load alters if a process

model is extended with graphics extracted from the virtual process environment. To

receive insights as to whether the impact of a virtual process environment additionally

varies in terms of included graphics, the following research question is addressed as

well:

RQ 2:

Does the application of additional graphics have an effect on the cognitive load

during the comprehension of a process model?

The study aims to obtain insights into whether the virtual process environment provides

an impact on the cognitive load a process reader perceives and whether a distinction

between business process notations exists.

3 Study Planning and Definition

3.3 Hypotheses Formulation

Ahypothesis can be characterized as a verbal proposition involving the presumptive

relationship of two or more variables. The validity has therefore not been proven and

hence the researcher attempts to confirm the stated hypotheses, either empirically or

experimentally. As a result, a hypothesis may be verified or falsified. In addition, it may

also be referred to as a statistical hypothesis and can be delineated from the terms

postulate and assumption. There exist two types of hypotheses: null hypothesis and

alternative hypothesis [79, 80]:

Null Hypothesis:

A null hypothesis serves the purpose of stating that no substantial

differences can be identified between the alternatives and in conclusion, the observed

difference is due to the fluctuation of the given sample. The null hypothesis is commonly

indicated as H0[79, 80].

Alternative Hypothesis:

A hypothesis that differs from the given null hypothesis is

referred to as an alternative hypothesis. These hypotheses are usually denoted as

If the null hypothesis is falsified the alternative hypothesis will be accepted [80, 81].

There exist various statistical tests of significance in terms of analyzing the outcome of a

study. The goal of a researcher is to falsify the null hypotheses with a high significance

[

]. The research questions addressed in Section 3.2 serve as a fundamental of the

following derived hypotheses:

Constructed Hypotheses based on RQ 1:

As outlined in Section 3.1, the cognitive load is classified in three major types and thus it

is of interest to study the impact on each type:

Intrinsic Cognitive Load:

Does the application of a virtual environment have an effect on the intrinsic cognitive

load during the comprehension of different process documentations?

3.3 Hypotheses Formulation

H0.1

: There is a significant difference between the different process notations in terms of

the intrinsic cognitive load by showing the virtual process environment beforehand.1

H0.1:µM6=µT

H1.1

: There is no significant difference between the different process notations in terms

of the intrinsic cognitive load by showing the virtual process environment beforehand.

H1.1:µM=µT

No significant difference is expected in terms of the intrinsic cognitive load, as both

process documentation (i.e, graphical and textual process notation) deliver identical

information to the process reader.

Extraneous Cognitive Load:

Does the application of a virtual environment have an effect on the extraneous cognitive

load during the comprehension of different process documentations?

H0.2

: There is no significant difference between the different process notations in terms

of the extraneous cognitive load by showing the virtual process environment beforehand.

H0.2:µM=µT

H1.2

: There is a significant difference between the different process notations in terms of

the extraneous cognitive load by showing the virtual process environment beforehand.

H1.2:µM6=µT

It is assumed that process models depicting the respective business process contribute

to a reduced extraneous cognitive load as this cognitive load reflects the instructions

design and thus a process model is presumed to be more representative (e.g., process

logic is clearer as it is represented through arcs). Furthermore, the virtual process

environment is visualized as well the respective process model and therefore it may

facilitate the information processing.

1M = Process Model, T = Textual Process Description

3 Study Planning and Definition

Germane Cognitive Load:

Does the application of a virtual environment have an effect on the germane cognitive

load during the comprehension of different process documentations??

H0.3

: There is no significant difference between the different process notations in terms

of the germane cognitive load by showing the virtual process environment beforehand.

H0.3:µM=µT

H1.3

: There is a significant difference between the different process notations in terms of

the germane cognitive load by showing the virtual process environment beforehand.

H1.3:µM6=µT

As it is presumed that a graphical process notation is positively more affected by the

virtual process environment in terms of the extraneous cognitive load than the textual

notation, it implies that the freed resources will be used for deep learning which is subject

to the germane load and thus more resources may be available for the process models.

Constructed Hypotheses based on RQ 2:

Both alternatives (process model and process model extended with graphics) will be

analyzed with respect to the different cognitive load types as well.

Intrinsic Cognitive Load:

Does the application of additional graphics have an effect on the intrinsic cognitive load

during the comprehension of a process model?

H0.4

: There is a significant difference between the different process models in terms of

the intrinsic cognitive load.2

H0.4:µM6=µMG

H1.4

: There is no significant difference between the different process notations in terms

of the intrinsic cognitive load by showing the virtual process environment beforehand.

2MG = Process Model extended with Graphics

3.3 Hypotheses Formulation

H1.4:µM=µMG

No significant difference is expected in terms of the intrinsic cognitive load as both

process models deliver identical information.

Extraneous Cognitive Load:

Does the application of additional graphics have an effect on the extraneous cognitive

load during the comprehension of a process model?

H0.5

: There is no significant difference between the different process models in terms of

the extraneous cognitive load.

H0.5:µM=µMG

H1.5

: There is a significant difference between the different process notations in terms of

the extraneous cognitive load by showing the virtual process environment beforehand.

H1.5:µM6=µMG

Despite the fact that the virtual process environment that will be displayed as a video may

enhance the internal representation [

], information is not retained from a long-term

perspective due to its transient nature [

]. Hence, the additional graphics included

in the process model could ease the human mind and less mental effort is required.

Nonetheless, as more graphics representing the same information is included, it may

possibly lead to an overload of information and consequently require to process more

information simultaneously as well (i.e., linking video sequence with respective graphics

and process elements together).

Germane Cognitive Load:

Does the application of additional graphics have an effect on the germane cognitive load

during the comprehension of a process model?

H0.6

: There is no significant difference between the different process models in terms of

the germane cognitive load.

3 Study Planning and Definition

H0.6:µM=µMG

H1.6

: There is a significant difference between the different process notations in terms of

the germane cognitive load by showing the virtual process environment beforehand.

H1.6:µM6=µMG

On the one hand, in the extended process model, the germane cognitive load could

be higher, as it will engage the motivation to comprehend the process model and raise

curiosity about how each static picture is linked to the respective process element. On

the other hand, the available resources for deep learning are reduced as it is consumed

by the additional extraneous cognitive load, due to the additional graphics that may

possibly lead to an overload of information.

3.4 Study Setup

After the goals and the objective of the study have been elucidated, it is essential to

outline the study setup to guarantee a valid and successful study. The study setup

involves the selection of the subjects, the response variables and the utilized instruments

and materials.

Selection of Participants

It would be ideal if it were possible to choose a random sample of all business practition-

ers within the scope of Business Process Management, however this is hardly to achieve.

In addition, it is difficult to gather enough domain professionals for the respective study,

and hence, for the sake of research, students are invited to participate in the study [

It is essential to gather subjects in order to conduct the study and to avoid a cause for

failure.

Primarily students, absolving the course Business Process Intelligence at the Ulm

University, were invited to participate in the study. The course is predominately designed

for master students, therefore it was expected that the students would be fluent in English

3.4 Study Setup

as the study is held in English. However, as the study was run online, everyone who had

access to the study link, had the possibility to participate. No specific restrictions have

been made.

Selection of Variables

A study investigates the relationship between two or more variables. There exists two

major types of variables that are of interest: the independent and dependent variable

(i.e., response variable) [

]. Further, a variable is "a property or characteristic on which

information is obtained" [85] in a study.

Independent Variable:

As shown in Figure 3.1, the independent variable exerts an impact on the dependent

variable and its magnitude. These variables may either be controlled or altered in order

to study the effect on the dependent variable. Further, the variables that are manipulated

over the study are indicated as factors. The attributes a factor has are also indicated as

levels, thus an independent variable is manipulated by changing its level [

]. Variables

that may exert an impact on the response variable without notice from the researcher

are referred to as confounding factors. These are linked to the threats to the validity of a

study and therefore it is essential to consider the possible confounding factors in terms

of the study’s performance [84].

Dependent Variable:

The dependent variable is the observed outcome of a study (see Figure 3.1). It is also

referred to as the response variable [

]. A study is conducted to analyze the effect that

independent variables may have on the observed response variable with respect to its

levels [84].

Applied to the study to be conducted, two independent variables are of interest: the

process documentation types and the additional usage of static pictures. The attributes

of the first independent variable are: the graphical notation and the textual process

description, whereas the second independent variable is differentiated between the

existence of static picture in the process documentation. Furthermore, information about

the gender,the prior knowledge about Business Process Modeling and BPMN 2.0 and

3 Study Planning and Definition

Figure 3.1: Variables in a Study [84] (adapted)

the gaming experience is collected and serve as control variables. As the study aims to

analyze the usage of a virtual process environment in terms of the cognitive load while

comprehending the process documentation, the dependent variables are the cognitive

load types: the intrinsic, extraneous and germane cognitive load. To reiterate, Section

2.5 provided a profound outline with respect to the cognitive load.

Selection of Instruments and Materials

Turning now to the utilized instruments and materials, the study was built upon three

surveys constructed on the platform SoGoSurvey [

]. This platform has been used

as it is practical and has a high ease of use. Each survey consisted of a demographic

questionnaire, a questionnaire about the gaming-experience, a video depicting the virtual

process environment, a process model (extended with graphics), or a textual process

description and a questionnaire in terms of the cognitive load. Considerable attention

must be paid when constructing a questionnaire in order to avoid misleading answers,

i.e., misleading results [

]. Hence, questionnaires have been used that were utilized

in other studies before. The demographic questionnaire is based on the studies of

[

] and [

] and was slightly adapted. Personal characteristics in terms of the prior

knowledge in particular of Business Process Modeling and BPMN 2.0 were collected.

The questionnaire covers in total of 12 questions, which is illustrated in Table 3.1. The

values of statements integrating a 7 - Point Likert Scale ranged from strongly disagree to

strongly agree. Further, the gaming questionnaire is based on [

] and has been slightly

extended. It summarizes in a total of three questions. One statement included a 7 - Point

Likert Scale, which ranged from strongly disagree to completely agree as well. Table 3.2

provides an outline of the gaming questionnaire.

3.4 Study Setup

Question/Statement Values

Which description matches best your current work status? Student, Academic, Professional, Other

What is your gender? Male, Female, Other

Course of studies User-Defined Text

How many courses related to Business Process Management None.

and/or Modeling have you undertaken in your study so far? One course/discipline.

Between two and four courses/discipline.

More than four courses/disciplines.

How many process models have you created or edited within None.

the last 12 months? One process model.

Between two and four process models.

More than four process models.

How many activities did all these models have on average? I have never worked with business process

models before.

Between five and fifteen activities.

Between fifteen and thirty activities.

More than thirty activities.

Do you have working experience related to Business No.

Process Management? Yes, between two and six months.

Yes, between six months and a year.

Yes, more than a year.

How long ago (months, years) did you start using I have never used BPMN 2.0 before.

Business Process Modeling and Notation 2.0 (BPMN 2.0) Between 1 and 3 months.

Between 3 and 6 months.

Between 6 and 12 months.

More than one year.

Overall I am very familiar with process modeling. 7 - Point Likert Scale

Overall, I am very familiar with the BPMN 2.0 notation. 7 - Point Likert Scale

I feel very confident in understanding BPMN 2.0 7 - Point Likert Scale

process models.

I feel very competent in using BPMN 2.0 for 7 - Point Likert Scale

process modeling.

Table 3.1: Demographic Questionnaire

Question/Statement Values

Are you familiar with video games? 7 - Point Likert Scale

Which platform do you prefer for playing video games? Computer, Nintendo, PlayStation,

Xbox, Other

What is your favorite video game genre? Action, Action Adventure

Adventure, Role-Playing, Simulation,

Strategy, Sports, Shooter,

Other

Table 3.2: Gaming Experience - Questionnaire

3 Study Planning and Definition

The results in terms of the cognitive load each participant perceived were gathered

through a cognitive load questionnaire based on [

]. Table 3.3 illustrates the cognitive

load questionnaire which consists of a total of 7 questions. Each question is based

on a 7 Likert-Scale ranging from completely wrong to absolutely right. As the original

questionnaire is intended for general use, adaptions to the query of the cognitive load

in terms of the process models and to textual descriptions were needed. The Table

3.3 summarizes both variants of the utilized cognitive load questionnaire. Words in

brackets apply to the other variant (i.e., participants received the textual description).

Participants receiving either the process model or the extended version received the

same questionnaire.

CL Type Statement Values

ICL For this process model (textual description), many things 7 - Point Likert-Scale

needed to be kept in mind simultaneously

ICL This process model (textual description) was very complex. 7 - Point Likert-Scale

GCL For this process model (textual description), I had to 7 - Point Likert-Scale

highly engage myself.

GCL For this process model (textual description), I had to 7-Point Likert-Scale

think intensively what things meant.

ECL During this process model (textual process description), 7 - Point Likert-Scala

it was exhausting to find the important information

ECL The design of this process model (textual description) 7 - Point Likert-Scala

was very inconvenient for learning.

ECL During this process model (textual process description), 7 - Point-Likert Scala

it was difficult to recognize and link the crucial information

Table 3.3: Cognitive Load Questionnaire

As in Table 3.3 shown, the cognitive load questionnaire focuses on the measurement of

each cognitive load type. Thus, two questions in terms of the intrinsic cognitive load, two

questions in terms of the germane cognitive load and three questions in terms of the

extraneous cognitive load were constructed.

Turning now to the virtual process environment which is depicted by a video. This video

is based on a 3D - game developed by [

] for a study. It was provided by the Information

System and Database Department (DBIS). As the study is conducted online, the video

is embedded on the respective page on the survey. For the embedding, it was uploaded

on Youtube in advance [

]. The copyright holder gave his consent and the uploaded

video had restricted access right. Solely viewers with a respective link had access to

3.4 Study Setup

the video. As the video was embedded on the survey, the subjects in the study also

could watch the video. The game describes the typical scenario of warehouse. It begins

with a warehouse worker taking a new order (see Figure 3.2) and ends with the loading

of the respective goods in the trailer. During the process, the demanding goods must

be collected and, inter alia, packed. Further, certain activities may be carried out in a

number of ways. For example, the goods may be picked up manually by using the forklift

(see Figure 3.3) or automatically by using the automatic picking system instead. The

video lasts about 18 minutes and the game was realized in English and therefore the

video as well.

Figure 3.2: Extract from the Video: Starting Sequence

Furthermore, to study the impact of a virtual process environment on process docu-

mentation, a process model based on the video was utilized. This process model was

provided by the Institute of Information System and Database at the Ulm University as

well. As the first research question addresses the effect of process documentation in

general, a textual process description based on the process model has been created.

To answer the second research question (see Section 3.2) each activity was extended

with a static picture extracted from the video. The provided process model included all

possible options described in detail. The video describes the possible options of the

warehouse worker, however, only the options taken is shown in detail. Therefore, the

3 Study Planning and Definition

Figure 3.3: Extract from the Video: Picking the Goods Using the Forklift

video showed merely a process instance of the resulting process model. In order to

not irritate the participants, the process model was recreated in Signavio [

] as the

tool supports BPMN 2.0 [

]. BPMN 2.0 was chosen since it is a widespread process

modeling language. The activities not shown in the process have been described at an

abstract level. Figure 3.4 and Figure 3.5 illustrate the process model of the warehouse

scenario. The textual process description and the process model extended with graphics

can be viewed in Appendix A.

3.4 Study Setup

Figure 3.4: Process Model: Warehouse Scenario - Part I

3 Study Planning and Definition

Figure 3.5: Process Model: Warehouse Scenario - Part II

3.5 Study Design

The setup of the study serves as a framework for the study, whereas the study design

determines how the respective study is conducted. Therefore considerable attention

must be paid when constructing the study design and several design concepts need to

be taken into account in order to guarantee valid results, i.e., to inhibit threats to validity.

In addition, the statistical methods used to study the results highly depend on the utilized

study design [81].

First, used termination in terms of the study design is outlined:

Randomization

: Randomization denotes the method by which subjects are randomly

assigned to existing groups or treatments [

]. Hence, treatment is referred to as one

possible factor variation [

]. Thus, one observed independent variable with a two-level

factor will result in two treatments. This design concept is essential to minimize the

effect of extraneous variables and undesired biases. Furthermore, with this method,

the likelihood of observing uniform groups in terms of the observed and the extraneous

variables is higher and serves as a fundamental for the study’s validity [79].

Factorial Design

: A factorial design is established if two or more variables are observed,

i.e. multiple factors, whereas a single-factor design refers to a design in which solely

one factor and its levels are studied [

]. When all possible alternatives to the

factors are of interest a factorial design is considered. However, as each independent

variable may consist of different values the variable may take, it will rapidly lead to an

overwhelming number of possible treatments. Thus, two observed factors with a level of

three will already lead to the establishment of nine treatments. To overcome this problem

a fractional factorial design can be established that studies the characteristics of factors

of interest [91].

Interaction Effect

: An interaction effect occurs when two or more factors (i.e., indepen-

dent variables) are analyzed in a study. Thus, it reflects the interaction of the factors.

It occurs when the magnitude of one’s factor characteristic depends on another factor

characteristic that influences the outcome of the response variable [

]. Consequently,

3 Study Planning and Definition

while constructing a factorial design attention needs to be paid in terms of the interaction

effect.

Between-Subjects

: In a between-subject design each subject undergoes solely one

treatment whereas the counterpart is the within-subject design. Subjects experience all

treatments [92].

As presented in Section 3.4, two independent variables are observed in the study in

terms of the impact exerting on the cognitive load (response variable). Therefore, a

2x2 factorial design is established. However, as solely the static picture included in the

process model is of interest, only three treatments will be constructed and analyzed.

Thus, it’s a fractional factorial design. In addition, randomization is considered to

ensure equal groups among the treatments. As the study is being conducted online,

randomization is applied in the form of a random forwarding the subjects to the respective

survey via a website button. Each subject is exposed to one survey (i.e., treatment)

and therefore a between-subject design is considered. Figure 3.6 provides a detailed

illustration of the procedure. The only aspect that varies between the treatments is the

process documentation shown, which is marked blue in Figure 3.6.

Figure 3.6: Study Design

3.6 Risk Analysis and Migrations

In this section, the threats of the study are evaluated with respect to validity. Validity

determines the trustworthiness of the results and drawn conclusions from a study. If

a study is not valid, the conclusions and results are probably biased [

]. Therefore,

to avoid low validity considerable attention must be paid while planning and designing

the study. A study’s validity is classified into external validity,internal validity,construct

validity and conclusion validity [84].

Threats to External Validity:

External Validity is given when the results of the study are

generalizable in particular with respect to the interested population. In order to ensure

the generalization of the results and of the drawn conclusions, it needs to be evaluated

whether the sample size is representative and whether the results are representative for

the selected target population [

]. As regards the study conducted in this Master’s

thesis, the subjects are primarily students absolving the course on Business Process

Intelligence. It would be ideal to choose business professionals within the scope of

Business Process Management, thus this is hardly to achieve. Therefore, students with

at least novice knowledge in Business Process Management have been chosen for

recruitment as they are easier to gather. Choosing students instead of professionals

could be a threat to external validity. However, the findings of [

] indicate that the

difference between students and domain professionals in software engineering was

not significant and hence students could be used as the sample in the study, rather

than domain professionals. In addition, randomization is considered to mitigate biases

between groups (i.e., treatments). Another aspect which could be a threat is the utilized

process complexity and process domain. The business process model and the deployed

video in the study describes a common warehouse process, which is not tied to specific

knowledge, but rather intuitive. The process model and the video are thus not complex.

The process model contains no loops or sub processes, which might make the process

complex. It is linear. However, as the process domain and the process documentation

are not complex, it is questionable whether the results can be used to draw conclusions

valid on complex process domains. Hence, this study may serve as a fundamental for

conducting multiple studies with respect to different process complexity and whether

3 Study Planning and Definition

the virtual process environment exerts different impacts depending on the complexity.

In addition, the process models were designed in BPMN 2.0. This graphical process

notation is a widespread and accepted process model language and notation. Another

aspect, which is attributable to external validity is the sample size. 74 students were

recruited for the study, which can serve as a representative sample size.

Threats to Internal Validity:

Internal Validity is related to the impact of confounding

factors affecting the response variables unknown to the researcher conducting the study.

Hence, confounding factors present potential threats to the hypothesized cause-effect

relationship [

]. As the study is conducted online, the subjects may participate in

the study in any environment they wish and thus a history threat may occur. A history

threat is concerned with events that occur during the participation which may affect the

study outcome [

]. Since it is hard to control the environment in which the participant

is during the study, events such as disturbance by other people may occur. A subject,

for example, may be disturbed while watching the 3D-Warehouse scenario and thereby

may overlook process-relevant information. Consequently, advice to watch the video

in an undisturbed area was given in advance to prevent the described history threat. It

was assured that the subjects could also pause the video at any point and the video

could be fast and forwarded. Furthermore, the characteristics of the subject may vary

and could be considered as a threat as students from different courses of studies are

able to absolve the Business Process Intelligence course and thus have prior knowledge

in particular of industrial experience and BPMN 2.0. Therefore, to mitigate this threat

several questionnaires such as the demographic questionnaire and gaming experience

questionnaire are applied to obtain personal characteristics. In Section 4.3 the obtained

data is validated. In addition, subjects were assigned randomly to the treatments in order

to obtain uniformly distributed groups. Another aspect, which could be considered as a

threat are the utilized instruments. Poorly designed instruments may negatively impact

the response variable [

]. As various questionnaires have been used as measurements,

considerable attention must be paid to the questions, as poorly formulated questions

can lead to misleading answers [

]. Therefore, questionnaires were utilized which were

applied in other studies before. The video showed every possible activity, i.e, option,

that could be taken, however only the chosen option in the video was shown in detail.

3.6 Risk Analysis and Migrations

Furthermore, the video was shown in an appropriate speed. The process model (also

extended with graphics) and the textual process description were based on the video.

Thus, not selected options were described on an abstract level to avoid irritation among

the subject. Furthermore, other factors could influence the cognitive load perceived by

the process readers. The structure and design of the process model might affect the

extraneous cognitive load as it is related to the design of the instructions (see Section

3.1). The process model is based on the video used in the study of [

] and was provided

by the Institute for Information Systems and Databases and therefore has been reviewed

in advance. The textual description was generated as neutral and structured as possible

to minimize the influence of the formulation and of the structure on the subject. However,

external validity must be given in order to establish internal validity, and vice versa [96].

Threats to Construct Validity:

Construct Validity is related to the degree to which

the operations reflect the theory behind the study [

]. One threat to the study may

be caused by the participants. As they are aware that they are participating in a

study, their behavior could be affected and therefore they would not act as they would

normally. Since the study is conducted online, each participant could choose the time

and environment to participate in the study and thereby feel more comfortable as the

experimental environment is not present. In addition, participation is anonymised, which

increases the likelihood of truthful answers, as there are no possibilities for traceability.

Furthermore, a major threat to the construct validity is also posed by the researcher.

The researcher may influence the subjects indirectly or explicitly by leading them to a

particular response using the formulation of questionnaires. Considerable attention was

therefore paid when choosing the right questionnaires for the measurements. It was

assured that the measurement applied for the cognitive load types, which serves as

the response variables, is defined in an abstract level and thus the subjects could not

suspect the response variable being examined.

Threats to Conclusion Validity:

Conclusion validity is related to the credibility of the

conclusion drawn on the relationship between the observed treatment and the outcome

[

]. There exist two types of errors which can be made by a researcher when reaching

to a conclusion. Type-I error is concerned with the acceptance of the null hypothesis

even though it is false. Choosing high statistical significance (i.e., alpha level) will

3 Study Planning and Definition

decrease the type-I error. The counterpart is a type-II error related to the issue of

the falsification of a null hypothesis which is true [

]. Therefore, an alpha level of

= .050 was chosen to minimize the threat of hypothesis validity. Furthermore, the

experiment design determines which statistical test may be used and vice versa, hence

a parametric one-way analysis of variance (ANNOVA) test was utilized which is suitable

for the established experiment design.

Study Operation

In this section, the study operation is elucidated by presenting the preparation and the

execution of the study. Before a study may be conducted, considerable preparation

must be made in order to ensure a smooth and successful study execution. Further, the

collected data is validated in Section 4.3.

4.1 Study Preparation

Before the study’s execution, the website (i.e., button) was modified for randomization

purposes with respect to the respective survey links representing the three treatments

established in the study. Thus, the website function as an intermediate step. Further-

more, a study pilot was carried out in advance in order to test aspects such as data

collection, randomization, video functionality and to debug the survey in general. Thus,

questionnaires and the study procedure were adapted based on the issues raised during

the pilot study. The process models and the textual process description were several

times reviewed in order to ensure the quality of the process documentation.

4.2 Study Execution

The study was being executed during the period from 06. June 2020 and 30. June

2020. Students absolving the course Business Process Intelligence were predominantly

recruited to participate in the study. For this, the link to the study was published in the

4 Study Operation

respective moodle course [

]. To encourage the participation, the students earned

bonus points relevant for the course.

The treatment structure is fairly similar for all treatments. First, personal characteristics

are obtained by the utilized demographic and gaming experience questionnaire. After-

wards, the subjects are forwarded to the video of the warehouse scenario. In all three

treatments the video is identical. After viewing the video, depending on the treatment,

either the initial process model (see Figure 3.4 and Figure 3.5), the textual description

(see Figure A.1 and Figure A.2) or the process model extended with graphics (see Figure

A.3 and Figure A.4) will be shown to the subjects.

In this section, the cognitive load questionnaire must be answered with respect to

the displayed process documentation. In addition, they are asked about the process

documentation’s correctness. In order to investigate aspects such as the self-assurance,

the option "Unsure" is included. The only aspect that varies between the treatments

is the process documentation shown. After completing the section, the subjects are

exposed to all the process documentation observed in this study and have to chose

the process documentation that is in their opinion best suited. Subsequently, subjects

are asked to generate their individual code as they will receive bonus points for their

participation. The study ends with the acknowledgement for the participation in the study.

The whole study can be viewed in Appendix B.

4.3 Data Validation and Data Set Reduction

Before the collected data can be analyzed, it needs to be validated. The objective is to

check whether the data collected is reasonable and whether errors need to be excluded.

An error, for example, relates to subjects who have not seriously completed the survey

[

]. A total of 74 persons have participated in the study. However, the data of two

subjects had to be excluded from further analysis on the premise that they had not fully

completed the survey. Therefore, two outliers have been identified and, thus, data from

72 subjects will be validated in the following.

4.3 Data Validation and Data Set Reduction

Participants in this study consisted primarily of students (68) of diverse courses of

studies (see Table 4.2). The majority of the subjects complete studies in Economics and

Management (in total of 46 subjects) as reported in Table 4.2. The respective data is

additionally illustrated in Figure 4.1a and Figure 4.1b.

(a) (b)

Figure 4.1: Work Status and Course of Studies

Among the subjects, 37 indicted to be male and 35 to be female (see Table 4.2). The

allocation of the gender is, therefore, fairly balanced. In addition, as shown in Table

4.2, the distribution of treatments is as follows: A total of 22 subjects participated in the

process model treatments, 27 subjects received a textual process description and 23

participants evaluated the process model extended with graphics. The distribution of

gender and treatment are additionally shown in Figure 4.2a and Figure 4.2b.

(a) (b)

Figure 4.2: Gender Distribution and Treatment Distribution

4 Study Operation

The majority of the subjects recorded having absolved one course/discipline (36 subjects)

or between two and four courses/disciplines (27 subjects) related to Business Process

Management and/or Modeling. Concerning the modeling experience, around 19 percent

of the subjects have never created a process model, and around 83 percent of subjects

have stated that they do not have any industrial experience in the field of Business

Process Modeling. The majority of the process models created by the subjects had,

on average, between five and fifteen activities which contributes to the assumption that

the knowledge of process modeling is limited to simplifying business processes. Fur-

thermore, only 13 subjects have never started to use BPMN 2.0 for modeling purposes.

With respect to the familiarity with process modeling and the familiarity with BPMN 2.0

Notation, both median (md) values are 5.000, measured on a 7-Point Likert-Scala. In

addition, the perceived competence in using BPMN 2.0 for process modeling and the

perceived confidence in the understanding of BPMN 2.0 process models provides a

median score of 5 as well (see Table 4.1).

Median

Familiarity with Process Modeling 5.000

Familiarity with BPMN 2.0 Notation 5.000

Perceived Competence in using BPMN 2.0 for process modeling 5.000

Perceived Confidence in understanding of BPMN 2.0 process models 5.000

Table 4.1: Median of Familiarity, Perceived Confidence and Perceived Competence

Turning now to the gaming experience, half of the subjects are slightly to very familiar

with video games as the median score is 5.000. According to Table 4.2, 32 participants

prefer to play video games on a computer whereas 15 participants indicated to prefer

Playstation and, inter alia, 10 subjects are playing on Nintendo.

In terms of the favorite video genre in particular on computers, subjects prefer to play

strategy or role adventure game. However, in general there is no genre that stands out.

Every genre was chosen, at least once.

4.3 Data Validation and Data Set Reduction

Independent Variable Values N

Work Status

Student 1

Academic 2

Professional 3

Course of Studies

Business Mathematics 1

Cognitive Systems 1

Computer Science 2

Economics and Management 3

Mathematical Biometry 4

Media Informatics 5

Software Engineering 6

Gender Male 1

Female 2

Platform for Video Games

Computer 1

Nintendo 3

PlayStation 4

Tablet 5

Xbox 6

Other 7

Table 4.2: Results of Frequency Distribution of Data Depending on Different Variables

Study Analysis and Interpretation

The output of the study is analyzed and interpreted in this section. First, a raw data

analysis is presented and descriptive statistics are reported. Subsequently, the derived

hypotheses from the research goal in Section 3.3 are tested for significance. After the

analysis of the collected data, a summary is presented followed by the final discussion

of the gathered output from the analysis. Finally, in Section 5.4 and in Section 5.4 the

limitations and implications of the study are outlined. Data analysis was performed by

using SPSS.

5.1 Analysis of Raw Data and Descriptive Statistics

Descriptive statistics are concerned with the analysis of the data by plotting and visu-

alizing it in order to obtain an overview thereof. Furthermore, an objective is, inter alia,

to detect invalid data points (i.e., outliers) [

]. It is essential for further analysis, in

particular for the analysis and interpretation of the data hypotheses, as it is attributable

to an understanding of the nature of the data collected.

Descriptive statistics for 72 data points are reported as 72 subjects are considered in

this study (see Section 4.3). A boxplot is utilized to analyze the relationship between

the treatments and the cognitive load types (i.e., intrinsic, germane, and extraneous).

A boxplot represents the dispersion of the data. The rectangle (i.e., the box plot)

indicates where 50 percent of the data is located which is also known as the interquartile

range. The lower box border represents the first quartile (i.e., 25 percent) and the

upper boundary indicates the third quartile (i.e., 75 percent). The whiskers from and to

5 Study Analysis and Interpretation

(a) (b) (c)

Figure 5.1: Boxplot of ICL, GCL and ECL Depending on Treatments

the rectangle display the maximum and minimum score in the data. The outliers are

presented as data points outside the whiskers and are considered atypical observations.

The value of the median is indicated by the line inside the boxplot.

Turning now to the collected data in the study, based on Figure 5.1a

it is noteworthy that

the median score for the treatment process model in terms of the intrinsic cognitive load

(md = 4.750) differs from the median score of the other treatments (md = 4.000 for both

treatments). Furthermore, an outlier is identified for the intrinsic cognitive load. However,

the respective observation has not been an outlier for the germane cognitive load and for

the extraneous cognitive load. Considering the median score in terms of the germane

cognitive load, the result just marginally differs (see Figure 5.1b). Another noticeable

aspect is that in general, when reading a process model extended with graphics, the

subjects experienced the lowest extraneous cognitive load (md = 3.333) (see Figure

5.1c). On the contrary, the subject who received solely the process model perceived the

highest extraneous cognitive load (md = 4.000) and therefore the extraneous cognitive

load was 20 percent higher. The mean and the standard deviation for the respective

cognitive load types are reported in Table 5.1 for further analysis.

In addition, the subjects also had to answer whether the process model (or the textual

process description) corresponds adequately and completely to the process shown in

1M = Process Model, T = Textual Process Description, MG = Process Model extended with Graphics

5.1 Analysis of Raw Data and Descriptive Statistics

Treatment N Mean SD

ICL

Process Model 22 4.614 1.214

Textual Description 27 4.037 1.278

Process Model with Graphics 23 4.000 1.422

Total 72 4.201 1.318

GCL

Process Model 22 3.864 1.264

Textual Description 27 3.539 1.448

Process Model with Graphics 23 3.457 1.544

Total 72 3.632 1.417

ECL

Process Model 22 3.909 1.322

Textual Description 27 3.840 1.559

Process Model with Graphics 23 3.609 1.369

Total 72 3.787 1.416

Table 5.1: Descriptives for ICL, GCL and ECL

the video (see Section 3.4). Figure 5.2 illustrates the allocation of the answers provided

by the subjects. Nearly half of the participants were confident that the displayed process

documentation corresponds adequately and completely with the warehouse scenario

seen in the video. Once each treatment is considered individually, it is evident that the

subjects were more confident about the process model extended with graphics, whereas

the textual process description was classified mostly as inadequate and incomplete.

Treatment N Yes No Unsure

Process Model 22 9 3 10

Textual Description 27 7 9 11

Process Model with Graphics 23 16 2 5

Total 72 32 14 26

Table 5.2: Allocation of Answers in terms of Correspondence

Table 5.3 represents the summary of responses to the preference of a specific process

documentation differentiated between the treatments. According to Table 5.3, the majority

chose the process model as their favorite. A small minority indicated their preference for

the textual process description.

As interaction effects are of interest, a median split was conducted to divide the sample

size into groups based on the gaming experience and on the business process modeling

experience as well. The median is, therefore, considered as a reference point for dividing

the data set into two classes. In view of the gaming experience, the median score

5 Study Analysis and Interpretation

Treatment N Process Model Textual Process Process Model

Description with Graphics

Process Model 22 11 3 8

Textual Description 27 19 1 7

Process Model with Graphics 23 14 1 8

Total 72 44 5 23

Table 5.3: Summary of Results for Preference

of the familiarity with video games was chosen to divide the groups into low gaming

experience and high gaming experience (md = 5.000). With respect to the business

modeling experience, the split was based on the median of the created and/or edited

process models over the last 12 months (md = 2.000) and was also split into high and

low groups. The values were measured on an ordinal scale ranging between 0 and 4.

Table 5.4 represents the distribution of the identified high and low groups.

Gaming Experience N Values

low gaming experience 43 ≤2.000

high gaming experience 29 >2.000

Business Process Modeling Experience

low modeling experience 39 ≤5.000

high modeling experience 33 >5.000

Table 5.4: Distribution of the Groups

Furthermore, Table 5.5 reports the descriptives for the score of the intrinsic, germane,

and extraneous cognitive load differentiated between the high and low groups considering

the gaming experience. Interestingly, there is a difference between the groups in terms of

the intrinsic cognitive load. The group with low gaming experience therefore experienced

Group N Mean SD

ICL low gaming experience 39 4.385 1.227

high gaming experience 33 3.985 1.406

GCL low gaming experience 39 3.692 1.360

high gaming experience 33 3.561 1.499

ECL low gaming experience 39 3.949 1.390

high gaming experience 33 3.596 1.442

Table 5.5: Descriptives for High and Low Gaming Experience

5.2 Hypotheses Testing

a higher intrinsic cognitive load (mean of 4.385) overall compared to the group with high

gaming experience (mean of 3.985). In addition, the high gaming experience group

experienced in general a lower extraneous cognitive load relative to subjects with low

gaming expertise. Turning now to the modeling expertise, the descriptives are presented

in Table 5.6. Generally speaking, the mean score of each cognitive load type is lower for

participants with high modeling expertise.

Group N Mean SD

ICL low modeling experience 43 4.395 1.370

high modeling experience 29 3.914 1.203

GCL low modeling experience 43 3.756 1.548

high modeling experience 29 3.448 1.198

ECL low modeling experience 43 3.915 1.427

high modeling experience 29 3.598 1.401

Table 5.6: Descriptives for High and Low Modeling Experience

Differences in the data have been identified, however they need to be tested for signifi-

cance.

5.2 Hypotheses Testing

Hypotheses Testing is concerned with determining the significance of null hypotheses.

Constructed hypotheses in Section 3.3 will therefore be tested. As mentioned before

(see Section 3.6), a significance criterion (i.e., alpha level) of

= .050 is selected. There

are various statistical tests for hypotheses. Hence, a parametric one-way ANNOVA test

is applied. The one-way ANNOVA test compares the mean of at least two samples [

]

and is therefore suitable for the experimental design in the study as three groups are

studied.

First, used termination in terms of the hypotheses testing is outlined. Observed differ-

ences are considered to be significant if the p-value is within the defined significant

criterion. As the significance level is set to .050, the p-value should therefore be lower

than the level. The p-value is also defined as p. In addition, a test’s power analysis

is related to the probability of rejecting the null hypothesis and thus finding statistical

5 Study Analysis and Interpretation

significance results when one actually exits. Power-Analysis was performed by using

G*Power [

]. The effect size reflects the magnitude of an effect exerted by independent

variables on the response variable. Thus, it describes the effect of a phenomenon. Effect

sizes are important since, for example, a finding may be significant, but its effect is

so minimal that it is considered trivial [

]. Various effect sizes are used for different

statistical tests. Effect sizes d and f are hence considered for the respective analysis.

According to [

], the effect size d of .800 and the effect size f of .400 are classified

as large and therefore the effect of .800 and .400 is used for further analysis, such as

estimating the necessary sample size for significance given the desired effect and power.

Intrinsic Cognitive Load:

In terms of the intrinsic cognitive load, the analysis did not show any significant dif-

ferences (F(1.580) = 2, p = .213) among the process documentations. Thus, the null

hypothesis H0.1and H0.4are accepted.

Extraneous Cognitive Load:

None of these differences in terms of the extraneous cognitive load were statistically

significant (F(.277) = 2, p = .759). As the p-value is greater than the significance level,

the alternative hypotheses

H1.2

and

H1.5

are rejected, and therefore

H0.2

and

H0.5

are

accepted.

Germane Cognitive Load:

There were no significant differences (F(.474) = 2, p = .625) between the treatments with

regard to the germane cognitive load. The null hypotheses

H0.3

and

H0.6

are not refuted.

As no significance was identified, no posthoc test for multiple comparisons considering

Bonferroni was applied. Table 5.7 summarizes the results of hypotheses testing.

Response Variable Differences P-Value Significance

intrinsic cognitive load between groups .213 No

extraneous cognitive load between groups .759 No

germane cognitive load between groups .625 No

Table 5.7: Results of Test of Significance (One-way ANNOVA)

5.2 Hypotheses Testing

In addition, it was also of interest to study whether the expertise in terms of gaming

and modeling experience has an impact on the perceived cognitive load regardless of

the process documentation. The parametric two-sample t-test is applied to analyze

the difference between independent variables, which compares the population mean

of two unrelated groups for significant differences [

]. The test is also known as the

independent t-test.

Gaming Expertise:

According to the descriptive statistics provided before, the means for each cognitive

load type was lower for the high gaming experience group relative to the low gaming

experience groups. As the study was an exploratory one, a two-tailed t-test was applied.

However, the analysis did not show any significant differences for the intrinsic (t(1.288) =

70, p = .202)), extraneous (t(1.054) = 70, p = .295) and germane (t(.391) = 70, p = .697)

cognitive load. Further, equal variance was assumed as the Levene’s test for equality

of variance was not significant for all three response variables (i.e., intrinsic (p = .776);

extraneous (p = .320); germane (p = .867). Therefore, no statistical differences have

been detected and thus the variations are attributable to random occurrence.

Business Process Modeling Experience:

Considering the descriptive statistics with respect to the modeling expertise, the two

groups differ in all three types of cognitive load. The high modeling experience group

experienced less cognitive load as opposed to the low modeling experience group, as

well. However, none of these differences in terms of the cognitive load types were

statistically significant (i.e., intrinsic (t(1.535) = 70, p = .129); extraneous (t(.931) = 70,

p = .355); germane (t(.902) = 70, p = .370). According to Levene ’s test, the study

showed that equivalent variance could be assumed for intrinsic (p = .718), extraneous (p

= .931) and germane cognitive load (p = .901). Table 5.8 reports the results of the test

for significance, applying the two-sample t-test for both area of expertise.

5 Study Analysis and Interpretation

Response P-Value Significance

Variable

Gaming Expertise

ICL .202 No

ECL .295 No

GCL .697 No

Modeling Expertise

ICL .129 No

ECL .931 No

GCL .901 No

Table 5.8: Results of Test for Significance (Two-Sample T-Test)

Self-Confidence and Preference for Process Documentation:

As noted before, a tendency about the self-confidence of subjects about the complete-

ness and adequateness of the process documentation were found (see Table 5.2).

Therefore, the textual process description was most rated as false whereas the process

model extended with graphics were procentually most rated as to correspond the process

warehouse scenario. It is of interest whether an association between the process docu-

mentation and the self-confidence may be concluded. To test the findings for significance,

a nonparametric likelihood-ratio chi-quare test of independence is applied. A chi-square

test analyzes the association between two categorical variables and is based on data

displayed as frequencies, thus it is suitable for the respective data set (see Table 5.2)

[100]. This test is conducted in terms of the findings of preference as well.

According to the analysis, it can be concluded that an association between the self-

confidence about the process documentation and the treatment exists (

χ2

(4) = 11.834, p

= .019). Cramer’s V effect size is considered as categorical variables with more than

two characteristics are considered in this analysis. As shown in Table 5.9, the Cramer’s

V effect size is .287 (p = .019) which can be converted into the w index of [

] resulting

in w = .406. The magnitude of the respective w is classified as a moderate effect (.300

≤

.500) [

100

]. Comparing the standardized residuals, reported in Table 5.10,

one can suggest that the confidence about the completeness and adequateness of the

process model extended with graphics shows the strongest difference. Therefore, more

subjects were confident about that process model than expected. In addition, subjects

were more confident about the incompleteness and inadequacy of the textual process

description, thus it was not false.

5.2 Hypotheses Testing

Association P-Value Cramer’s V w

effect size p-value

Self-Confidence - Treatment .019 .287 .019 .406

Preference - Treatment .524 .153 .498 .216

Table 5.9: Results of Test for Significance (Likelihood-Ratio Chi-Square Test)

Yes No Unsure

Treatment

Process Model -.200 -.600 .700

Textual Process Description -1.400 1.600 .400

Process Model extended 1.800 -1.200 -1.100

with Graphics

Table 5.10: Standardized Residuals of Self-Confidence and Treatment)

Turning now to the findings with regard to the preference, no statistical significance (

χ2

(4)

= 3.203, p = .498) could be identified. Therefore, no association between the preference

for a process documentation and treatment exists.

Interaction effects:

Furthermore, it is of interest whether there is an interaction effect between the expertise

(i.e., modeling and gaming experience) and the treatments. As explained in Section

3.5, if more than two independent variables are observed and one independent vari-

able influence on the response variable is dependent on another independent variable

attribute, an interaction effect may occur. It is therefore studied whether the impact of

certain process documentation (i.e., treatment) depended, for example, on the level of

expertise that a subject has and vice versa. A multiple linear regression model is applied

to analyze potential interaction effects. The regression model results are reported in

Table 5.112.

Model 1 and model 2 refer to the intrinsic cognitive load as the response variable, model

3 and 4 are concerned with the extraneous cognitive load and model 5 and 6 analyze

the germane cognitive load. Thereby, measures of interest are the interaction terms

Gaming Experience*Treatment and BPM Experience*Treatment. According to Table

The predictors are coded as follow: Treatment (0 = Process Model, 1 = Textual Process Description, 2

= Process Model extended with Graphics); Gaming Experience (0 = low gaming experience, 1 = high

gaming experience); BPM Experience (0 = low modeling experience, 1 = high modeling experience)

5 Study Analysis and Interpretation

Coefficients (p-value)

ICL ECL GCL

(1) (2) (3) (4) (5) (6)

Treatment -.249 -.319 -.144 -.073 -.227 -.186

(.327) (.176) (.604) (.778) (.416) (.470)

Gaming Experience -.246 -.307 -.199

(.626) (.570) (.721)

Gaming Experience * Treatment -.201 -.064 .038

(.616) (.874) (.931)

BPM Experience -.523 -.058 -.248

(.323) (.920) (.669)

BPM Experience * Treatment .034 -.261 -.064

(.936) (.574) (.890)

Table 5.11: Interaction Effects between Expertise and Treatments

5.11 no significant interaction effect for the intrinsic (p = .616), extraneous (p = .874)

and germane cognitive load (p = .931) between the gaming expertise and treatment is

identified. In addition, in terms of the interaction effect, the analysis did not show any

significant effect between the business process modeling experience and treatments

with respect to the intrinsic (p = .936), extraneous (p = .573) and germane cognitive load

(p = .890). Taken as a whole, the analyzes did not detect any significant interaction effect

and thus no dependency between the expertise and treatments exists in this experiment.

Power Analysis and Estimated Sample Size:

No significant differences were identified in the prior analysis. However, it is likely that

there is a phenomenon that could not be detected due to insufficient sample size and/or

low power. The power of the test depends on various factors, such as the size of the

sample, the variance in the size of the sample, the level of significance chosen and the

effect size [101].

Of interest are in particular the differences between the treatments with regard to the

intrinsic and extraneous cognitive load and the impact of the level of business process

modeling expertise on the cognitive loads as the subjects with a high level of expertise

experienced in general a lower cognitive load relative to the low expertise group. The

differences for each cognitive load type in terms of the process documentation utilized

were analyzed with the one-way ANNOVA test. In order to analyze whether no significant

5.2 Hypotheses Testing

results could be yield owing to low power, a retrospective power analysis is conducted.

To estimate the power of the respective test, the effect size f is required and calculated.

The effect sizes, the power and the necessary sample sizes are reported in Table 5.12.

According to Table 5.12 the power analysis for the one-way ANNOVA test in terms of the

intrinsic cognitive load reported a power of 33.655 percent with an effect size f of .214

which is classified as a small effect according to [

]. Further, the power-analysis of

the one-way ANNOVA test with regard to the extraneous load showed a power of 9.409

percent. The respective effect size f is .090, which is considered as trivial. Generally

speaking, it was unlikely that the tests would be able to detect significant results on the

basis of such low power. The essential size of the sample is hence estimated by a power

of .800 and the desired effect of .400. The minimum sample size required consists of 66

subjects. Thus, the population in this study consisted of 72 subjects and exceeded the

essential threshold, however the effect size of the test is low.

Furthermore, the effect of a phenomenon tested with a two-sample t-test is indicated by

the effect size d [

]. Hence, the statistical analysis of the impact of modeling experience

on the different cognitive loads resulted in an effect size d of .373 with a power of 33.441

percent for the intrinsic, an effect size d of .224 with a power of 15.123 percent for the

extraneous and an effect size d of .222 with a power of 14.967 percent for the germane

cognitive load. Taken as a whole, the likelihood of rejecting the hypothesis is low and all

the reported effect sizes d are considered to be small effects. The minimum sample size

given an effect size d of .800 and a power of .800 is 52. Thus, the sample size in this

study exceeded the necessary size, but the effect size is low which contributes to the

low power. The results of the power-analysis and of the estimation of the sample sizes

are reported in Table 5.12 as well.

Test Effect Power Estimated

Size Sample Size

ICL - Treatment One-way-ANNOVA .214 .337 66

ECL - Treatment One-way-ANNOVA .090 .094 66

ICL - Modeling Expertise Two-Sample T-Test .373 .334 52

ECL - Modeling Expertise Two-Sample T-Test .224 .151 52

GCL - Modeling Expertise Two-Sample T-Test .222 .150 52

Table 5.12: Results of Power-Analysis and Estimation of Sample Sizes

5 Study Analysis and Interpretation

5.3 Summary and Discussion

In summary, the findings of the study show that the type of process documentation does

not affect the cognitive load after experiencing a virtual process environment. The aim of

the study was to provide insight into the research questions raised in Section 3.2. As a

result, with respect to RQ 1, no significant differences between the process notations

could be identified in the analysis and, in addition to RQ 2, the analysis did not indicate

any significant differences between the impact of process model and process models

extended with graphics on the intrinsic, extraneous and germane cognitive load.

As assumed, no differences were identified between the process documentation with

respect to the intrinsic cognitive load, as all three process documentation deliver identical

information. Contrary to expectations, the study did not find any substantial differences

between the process documentation in terms of the extraneous load. As the extraneous

load is concerned with the design of the instruction (i.e., information), a difference was

expected to be found between the documentation of the process as all three variations

display different information presentation. The findings, however, lead to the conclusion

that a process model, a textual process description and a process model extended with

graphics generally exert the same external cognitive load after watching the related virtual

process environment. Thus, none of the process documentation increased or decreased

the cognitive load and therefore facilitated the information processing. Consequently,

this also leads to another conclusion that the additional graphics in the process model

will not have an detrimental impact on the cognitive load. One possible reason for a

negative aspect might be that the additional static pictures increase the interactivity of

the element, since the information on the graphics also needs to be considered and

linked to the video shown before. Nonetheless, one could have expected a positive

effect: when key information is again presented with graphics, it might have facilitated the

memory. Interestingly, no significant differences were found with respect to the germane

cognitive load, and thus the slight differences are attributed to random occurrence. One

assumption for the static image to raise the germane cognitive load may be that subjects

could be inspired to understand how the graphics and process elements are connected

together, thus that is not the case.

5.3 Summary and Discussion

Generally speaking, based on the findings two assumption may be suggested:

•

The video results in that the cognitive load exerted by the different process docu-

mentation is not significant different from one another.

•

The video has no effect on the cognitive load exerted by the different process

documentation and the cognitive load alone does not vary sustainably.

It is therefore of interest how the results would have been when there was no virtual

process environment considered.

Considering the expertise in business process modeling, it is surprising that there are

no notable differences in the impact on intrinsic cognitive load, as the expertise can

influence the intrinsic cognitive load as stated in [

]. Consequently, the high modeling

group with extensive prior knowledge would have stored the respective knowledge as

schemas enabling the respective knowledge to be treated as one information. The group

with high expertise would therefore have experienced less intrinsic cognitive load in

terms of the complexity of the process documentation. In addition, the analysis detected

no significant differences between the level of gaming expertise on cognitive load. This

leads to the assumption that, regardless of the gaming expertise, the virtual process

environment and subsequently the process documentation are perceived the same way.

However, given that the findings are based on a demographic questionnaires including

only three question about the gaming expertise, the results should be treated with a

considerable caution.

With regard to the stated preference for a process documentation, no association

can be concluded between the treatments and the participant’s preference for one

as no substantial differences have been analyzed. Thus, the frequencies distribution,

displayed in Table 5.3, is attributed to random occurrence. Interestingly, a significant

association has been detected in terms of self-confidence and treatments. The process

model extended with graphics was thus mostly graded as a complete and adequate

correspondence of the warehouse process in the video. Following the findings, one can

conclude the following assumptions: Self-confidence was increased, because

•

each process activity was enriched with a static picture and this lead to the

assumption the displayed process must be correct.

5 Study Analysis and Interpretation

•

wrong activities had to be enriched with false static picture, which was possibly not

expected by the participants.

In addition, the textual process description was more rated as false among all process

documentation. This result can be explained by the different degree of abstraction that

the participants have and the process formulation could have been interpreted differently.

Through qualitative research, the current results could have been expanded by asking

the respondents why they chose the respective answer. Aspects regarding further

scientific research are discussed in Section 7.

Further, interaction effects have been studied as well. It was therefore of interest if the

degree of expertise and the treatment are interdependent and whether the interaction

had an effect on the outcome. However, the study showed no significant interaction

effect, leading to the conclusion that the magnitude of the intrinsic, extraneous and

germane cognitive load imposed by the treatment is not dependent on the expertise and

vice versa.

Overall, the investigations so far have been applied to a small sample size and thus the

apparent lack of correlation can be attributed to the low sample size and to the low power

of the tests as well. Furthermore, the environment in which the study was conducted

could not be completely controlled and this could affect the validity of the results. Hence,

further studies need to be conducted and this study can serve as a fundamental.

5.4 Limitations

It is plausible that a number of limitations might have influenced the results obtained.

First, the study was conducted online. The environment could therefore not be completely

controlled and the participants could have encountered disruption during the study, which

could have influenced the cognitive load. Another potential source of error is that with

regard to the preference query, the three types of process documentation were always

shown in the same sequence. Unfortunately, it was not possible to further investigate the

significant relationship of the subject’s self-confidence and the treatments due to the fact

5.5 Implications

that the study was focused solely on the investigation of the process reader’s cognitive

load. Furthermore, the virtual process environment was limited to a video based on a

game. Therefore the gameful-experience could not be fully considered as the game

was not played by participants. Since participants just watched the video and could not

play the game, the length of the study may have affected the participants’ interest and

motivation more.

5.5 Implications

Since the results suggest that no substantial difference exists between the process

documentation in terms of cognitive load after experiencing the virtual process envi-

ronment, it can be inferred that a virtual process environment may thus be created in

practice where both the textual process notation and the graphical process notation

are used in particular. In addition, after seeing the video, process readers were more

confident of the correctness and adequacy of the process model. Thus, a virtual process

environment can be considered in particular with process models. It can be used when

testing the accuracy of a process model, because after experiencing the virtual process

environment the process readers are more confident to process models. However,

contrary to [

], the study results showed as stated before that no significant difference

was observed in the business process modeling experience in terms of the cognitive

load. Process readers with different modeling expertise experience the same cognitive

load after watching the video. As process readers are from various fields of practice,

such as process analysts and domain experts, modeling expertise differs and therefore

a virtual process environment can be considered before reading the respective process

model [3].

Related Work

Research has tended to concentrate on gamification with respect to Business Process

Modeling and Management rather than in conjunction with cognitive load theory. There-

fore, research related to the three domains and intermediated research is listed in this

section, which is also mainly referred to in this master thesis.

Gamification is used in Business Process Management in various ways: for educational

aspects, extension of concepts and for practical implementations. For example, IBM

Innov8 was developed with the aim of teaching BPMN 2.0 using a 3D virtual environment.

In the game, the player has been hired to evaluate and optimate existing processes.

Those processes represents real-life scenarios [

]. Studies have shown that students

acquired knowledge of BPMN 2.0 faster than traditional methods [

]. Thus, the in-

corporation of gamification into the management of business processes has the potential

to have positive results. However, gamification is a widespread domain. Besides the uti-

lization of a virtual environment, gamification consists of the use of game-mechanics as

well. [44] for example, proposed a BPMS-Tool known as Horus Gamification. It considers

game-mechanics such as ranking, leader boards and points which employees may col-

lect on the basis of the quality of their process models. Furthermore, [

] suggested the

expansion of BPMN 2.0, including gamificiation elements such as levels, leader boards,

points, etc. In terms of process model comprehension, Tales of a Knightly Process has

been developed to study whether the game promotes the comprehension of process

models. The findings indicated an increase of process comprehension attributable to the

use of the game. In addition, it has been shown that the complexity of the process model

has an effect on comprehension [

]. The virtual process environment utilized in this

research is based on the game reported in [

]. [

] presents a gamification approach to

6 Related Work

analyze the impact of social distance on the granularity, qualitiy and strucutre of BPMN

2.0 process models. Gamification has already been used and researched in the sense

of Business Process Management for various reasons, but the effect on cognitive load

by gamification has scarcely been considered.

Taken a step back, a 3D virtual process environment depicted as a video can be seen

as an intercept between an animation and gamification. [

] points out why it is reason-

ably to use simulation (i.e., animation) which are displayed in a 2D or 3D environment.

Accordingly, the author notes that the reason for animation is that the processing of

graphical information takes place preconciously comparable with breathing. In terms

of 3D process visualization, [

102

] presents the interactive tool 3D Flight Navigator that

analyzes and displays business process models in a 3D virtual environment while pro-

viding an heads-up display. Furthermore, gamification is cloesely linked to virtual reality,

which takes a new perspective on the use of virtual environments. The authors of [

103

]

introduced a VR-BPMN concept and developed a prototype based on immersive busi-

ness process diagrams experience. In addition, a remote collaborative process modeler

was implemented by [

104

] considering augumented technology in a collaborative virtual

environment. In terms of the cognitive processes with regard to Business Process

Modeling and Management, research has mainly concentrated on the cognitive load

a process noation exerts itself. [

] for example conducted an eye-tracking study to

examine the cognitive load of business process models using different modularization

approaches for BPMN 2.0 diagrams. [

] formally analysis various graphical process

notations (i.e., Petri Nets, BPMN 2.0, Event-driven Process Chains (EPC)). BPMN 2.0

exerts the least cognitive load, based on the authors’ study. Another study with respect

to cognitive load is conducted by [

105

], which analyzes the effect of swimlanes in BPMN

process models.

With regard to research on textual and graphical process notations, the use of textual

process descriptions and process models in practice in software engineering has been

analyzed by [33]. Therefore, textual process description is used in addtion to describe

the details of a process and should thus be considered in analysis as graphical and

textual process notation has both strength and weaknesses. [

] studied whether textual

process notation or graphical process notation is better for process comprehension.

According to their findings, no substantial differences could be detected for novice

process readers whereas the graphical notation facilitate the process understanding for

experienced process readers.

In contrast to the stated above, the master thesis analyzes the impact of gamification

on the cognitive load while understanding the process documentation. [

] and [

]

delivers an thorough overview of human visual perception which is essential for the

visual information processing. To gain a deeper insight into the cognitive load theory

[53], [56] and [52] are recommended.

Conclusion and Future Work

This master thesis has investigated the impact of a virtual process environment on the

cognitive load that the process reader perceives during the comprehension of a process

model. An online study with 72 participants was conducted to receive insights into the

research questions. Of interest was the impact on the cognitive load of a textual process

description, a process model and a process model extended with graphics which is based

on the virtual process environment. The treatments were assigned to the participants at

random. Since three variants of process documentation were analyzed, three treatments,

each representing a process documentation, were defined. First, a video was watched by

the subjects. The video is based on a game in a virtual 3D environment so it is referred

to as a virtual process environment. A process documentation of the 3D warehouse

scenario was subsequently demonstrated for the subjects followed by a questionnaire

assessing the cognitive load of the respective process documentation. In addition,

subjects were asked about the adequacy and completeness of the respective process

documentation in order to study the subjects’ self-confidence regarding the process.

Finally, the preference for a certain process documentation was obtained by displaying

all three documents to the subjects and asking which one they prefer.

In conclusion, the findings indicate that there are no significant differences between

the process documentation in terms of cognitive load after the video has been shown.

As little research has been done in this area, these findings are left to interpretations

and assumptions and further investigations are thus needed. Considering the findings,

the question arises as to whether the virtual process environment does not lead in

any differences between the process documentation or whether the respective virtual

environment does not provide an impact on the process documentation. The present

7 Conclusion and Future Work

research examined the cognitive load only after having experienced the virtual process

environment. Consequently, based on this study, the question posed cannot be answered.

Nevertheless, further findings in this study suggest an correlation between the self-

confidence about the adequacy and completeness of the process documentation and

the process documentation displayed. Process readers were more confident about the

correctness and adequacy of the process model extended with graphics whereas the

textual process description was mostly regarded as incorrect. Further work needs to be

done to establish transparency in terms of the association of a process documentation

and the process reader’s self-confidence. No notable results were found in light of

the relationship between the process reader’s preference and process documentation

(i.e., treatments). In addition to the research so far, the effect of modeling and gaming

experience on cognitive load styles has also been investigated. The respective findings

did not result in any noteworthy outcome.

With respect to potential future work, this research has given rise to many questions

in need of further examination. In terms of the cognitive load, it is of interest as stated

before how the cognitive load is effected solely by the virtual process environment.

Furthermore, since the participants have only watched the virtual process environment,

this work may serve as a fundamental for studying the impact on cognitive load after

the game has been played. In addition, the study was based solely on one process

that is intuitive and can be seen as rather simple, and thus further studies are needed

to examine the effect of a virtual process environment with regard to complexity of the

process.

Bibliography

[1]

Tory, M., Moller, T.: Human Factors In Visualization Research. IEEE Transactions

on Visualization and Computer Graphics 10 (2004) 72–84

[2]

Weske, M.: Business Process Management: Concepts, Languages, Architectures,

2nd ed. Springer, Berlin, Germany (2019)

[3]

Dumas, M., La Rosa, M., Mendling, J., Reijers, H.A.: Fundamentals of Business

Process Management. 2nd ed. Springer, Berlin, Germany (2018)

[4]

Dadam, P., Reichert, M., Rinderle-Ma, S.: Prozessmanagementsysteme.

Informatik-Spektrum 34 (2011) 364–376

[5]

Simchi-Levi, D., Kaminsky, P., Simchi-Levi, E., Shankar, R.: Designing and

Managing the Supply Chain: Concepts, Strategies and Case Studies. 3rd ed.

McGraw-Hill (2008)

[6]

Harmon, P., Garcia, J.: The State of Business Process Management 2020.

Technical report, BPTrends (2020)

[7]

Lali, A.: Play the Innov8 game to learn business process

management

(Website) Online available at

https://www.ibm.com/developerworks/

library/ws-bpm-innov8/ws-bpm-innov8-pdf.pdf

, last accessed on: 23.

April 2020.

[8]

Ribeiro, C., Fernandes, J., Lourenço, A., Borbinha, J., Pereira, J.: Using Serious

Games to teach Business Process Modeling and Simulation. In: Proceedings

of the 96th International Conference on Modeling, Simulation and Visualization

Methods (MSV), Las Vegas, USA (2012) 133–139

[9]

Winter, M., Pryss, R., Probst, T., Reichert, M.: Learning to Read by

Learning

to Write: Evaluation of a Serious Game to Foster Business Process Model

Comprehension. JMIR Serious Games 8(2020) e15374

[10]

Pflanzl, N., Vetter, A., Hussong, K., Oberweis, A., Schönthaler, F., Vossen, G.:

Horus Gamification: Ein Prototyp zum Einsatz von Gamification im Rahmen des

Bibliography

Social BPM . In: Proceedings of the 46th Annual Conference of the German

Informatics Society (INFORMATIK), Chemnitz, Germany (2017) 433–438

[11]

Mancebo, J., Garcia, F., Pedreira, O., Moraga, M.A.: BPMS-Game: Tool

for

Business

Process Gamification. In: Proceedings of the 16th International

Conference

on Business Management (BPM) Forum, Barcelona, Spain (2017)

127–140

[12]

Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From Game Design

Elements

to Gamefulness: Defining "Gamification". In: Proceedings of the

15th

International

Academic MindTrek Conference: Envisioning Future Media

Environments (Mindtrek), Tampere, Finland (2011) 9–15

[13]

Putz, L.M., Treiblmaier, H.: Creating a Theory-Based Research Agenda for

Gamification. In: Proceedings of the 21th Americas Conference of Information

Systems (AMCIS), Puerto Rico, USA (2015) 3219–3231

[14]

Mullins, J.K., Sabherwal, R.: Gamification: A cognitive-emotional view. Journal of

Business Research 106 (2020) 304–314

[15]

Lumsden, J., Edwards, E.A., Lawrence, N.S., Coyle, D., Munafò, M.R.:

Gamification

of Cognitive Assessment and Cognitive Training: A Systematic

Review of Applications and Efficacy. JMIR Serious Games 4(2016) e11

[16]

Turan, Z., Avinc, Z., Kara, K., Goktas, Y.: Gamification and Education:

Achievements

, Cognitive Loads, and Views of Students. International Journal of

Emerging Technologies in Learning (iJET) 11 (2016) 64–69

[17]

Studerus, C.: Kognition im Business Process Management - Konzeption eines

optimalen Prozessmodells unter Berücksichtigung kognitiver Fähigkeiten. Master’s

thesis, Ulm University, Germany (2018)

[18]

Tallon, M., Winter, M., Pryss, R., Rakoczy, K., Reichert, M., Greenlee, M.W., Frick,

U.: Comprehension of business process models: Insight into cognitive strategies

via eye tracking. Expert Systems with Applications 136 (2019) 145–158

[19]

Zugal, S., Pinggera, J., Weber, B.: Assessing Process Models with Cognitive

Psychology. In: Proceedings of the 4th International Workshop on Enterprise

Bibliography

Modelling and Information Systems Architectures (EMISA), Hamburg, Germany

(2011) 177–182

[20]

Zimoch, M.: Influence of Psychological Distance on Process Modeling: A

Gamification Approach. Master’s thesis, Ulm University, Germany (2015)

[21]

Thompson, W., Fleming, R., Creem-Regehr, S., Stefanucci, J.K.: Visual Perception

from a Computer Graphics Perspective. A K Peters/CRC Press (2011)

[22]

Müsseler, J.: Visuelle Informationsverarbeitung. In: Allgemeine Psychologie. 3rd

ed. Springer, Berlin, Germany (2017) 13–50

[23]

Snowden, R., Thompson, P., Tom, T.: Basic Vision: An Introduction to Visual

Perception. Rev. ed. Oxford University Press (2012)

[24]

Jobst, M.: Benutzerschnittstellen in der multimedialen 3D Kartografie–eine

Gegenüberstellung. In: Proceedings of the 9th international symposion on ICT

and planning and impacts of ICT on physical space (CORP), Vienna, Austria

(2004) 579–584

[25] Démuth, A.: Perception Theories. Towarzystwo Słowaków w Polsce (2013)

[26]

Harrison, G.W., Haruvy, E., Rutström, E.E.: Remarks on Virtual World and Virtual

Reality Experiments. Southern Economic Journal 78 (2011) 87–94

[27] Bartle, R.A.: Designing Virtual Worlds. New Riders (2004)

[28]

ZeniMax Online Studios: Elder Scrolls Online. (Website) Online available at

https://www.elderscrollsonline.com/en-gb/home

; last accessed on:

20. April 2020.

[29]

Bell, M.W.: Toward a Definition of “Virtual Worlds”. Journal For Virtual Worlds

Research 1(2008) 2–5

[30]

Brey, P.: Virtual Reality and Computer Simulation. In: The Handbook of Information

and Computer Ethics. 2nd ed. John Wiley & Sons (2008) 361–384

[31] Burdea, G.C., Coiffet, P.: Virtual Reality Technology. 2nd ed. John Wiley & Sons

(2003)

Bibliography

[32]

Rohrer, M.W.: Seeing is Believing: The Importance of Visualization in

Manufacturing

. In: Proceedings of the 2000 Winter Simulation Conference (WSC),

Orlando, USA (2000) 1211–1216

[33]

Powers, K.L., Brooks, P.J., Aldrich, N.J., Palladino, M.A., Alfieri, L.: Effects of video-

game play on information processing: A meta-analytic investigation. Psychonomic

Bulletin & Review 20 (2013) 1055–1079

[34]

Munzner, T.: Visualization. In: Fundamentals of Computer Graphics. 4th ed. A K

Peters (2016) 665–669

[35]

PriceWaterhouseCoopers: Multipoly Next. (Website) Online available at

http://

multipoly.hu/en/how-to-play/how-to-play.html

; last accessed on:

22. April 2020.

[36]

Hamari, J.: Gamification. In: The Blackwell Encyclopedia of Sociology. John

Wiley & Sons (2019)

[37]

Zichermann, G., Cunningham, C.: Gamification by Design: Implementing Game

Mechanics in Web and Mobile Apps. O’Reilly Media (2011)

[38]

Blohm, I., Leimeister, J.M.: Gamification: Design of IT-based Enhancing Services

for Motivational Support and Behavioral Change. Business & Information Systems

Engineering 5(2013) 275–278

[39]

McGonigal, J.: Reality is Broken: Why Games Make Us Better and How They

Can Change the World. Penguin Press (2011)

[40]

Hamari, J., Koivisto, J., Sarsa, H.: Does Gamification Work? — A Literature

Review of Empirical Studies on Gamification. In: Proceedings of the 47th Hawaii

International Conference on System Sciences (HICSS), Waikoloa, USA (2014)

3025–3034

[41]

Object Management Group: Business Process Model and Notation (BPMN).

(

Website

) Online available at

https://www.omg.org/spec/BPMN/2.0

; last

accessed on: 26. April 2020.

Bibliography

[42]

Ko, R.K., Lee, S.S., Lee, E.W.: Business process management (BPM) standards:

a survey. Business Process Management Journal 15 (2009) 744–791

[43]

Rodrigues, R.D.A., Barros, M.D.O., Revoredo, K., Azevedo, L.G., Leopold, H.: An

Experiment on Process Model Understandability Using Textual Work Instructions

and BPMN Models. In: Proceedings of the 29th Brazilian Symposium on Software

Engineering (SBES), Belo Horizonte, Brazil (2015) 41–50

[44]

Ottensooser, A., Fekete, A., Reijers, H.A., Mendling, J., Menictas, C.:

Making

Sense of Business Process Descriptions: An Experimental Comparison of

Graphical

and Textual Notations. Journal of Systems and Software

(2012)

596–606

[45]

Siau, K.: Informational and Computational Equivalence in Comparing Information

Modeling Methods. Journal of Database Management (JDM) 15 (2004) 73–86

[46]

Larkin, J.H., Simon, H.A.: Why a Diagram is (Sometimes) Worth Ten Thousand

Words. Cognitive Science 11 (1987) 65–100

[47]

Moher, T.G., Mak, D., Blumenthal, B., Levanthal, L.: Comparing the

comprehensibility

of textual and graphical programs. In: Proceedings of the 5th

Workshop of Empirical Studies of Programmers, Palo Alto, USA (1993) 137–161

[48]

Diebold, P., Scherr, S.A.: Software process models vs descriptions: What do

practitioners use and need? Journal of Software: Evolution and Process

(2017)

e1879

[49]

Leopold, H., Mendling, J., Polyvyanyy, A.: Supporting Process Model

Validation

through Natural Language Generation. IEEE Transactions on

Software

Engineering 40 (2014) 818–840

[50]

Baier, T., Mendling, J.: Bridging Abstraction Layers in Process Mining by

Automated

Matching of Events and Activities. In: Proceedings of the 11th

International

Conference on Business Process Management (BPM), Beijing, China

(2013) 17–32

[51]

van der Aa, H., Leopold, H., Reijers, H.A.: Detecting Inconsistencies between

Process Models and Textual Descriptions. In: Proceedings of the 13th International

Bibliography

Conference on Business Process Management (BPM), Innsbruck, Austria (2015)

90–105

[52]

Sweller, J., Chandler, P.: Evidence for Cognitive Load Theory. Cognition and

Instruction 8(1991) 351–362

[53]

Sweller, J.: Cognitive Load Theory: Recent Theoretical Advances. In: Cognitive

load theory, Cambridge University Press (2010) 29–47

[54]

Geary, D.: Educating the Evolved Mind: Conceptual Foundations for an

Evolutionary

Educational Psychology. In: Psychological Perspectives on

Contempory Educational Issues, Information Age Publishing (2007) 1–99

[55]

Geary, D.: An evolutionarily informed education science. Educational Psychologist

43 (2008) 179–195

[56]

Sweller, J.: Cognitive Load Theory. Psychology of Learning and Motivation

(2011) 37–76

[57]

Miller, G.A.: The magical number seven, plus or minus two: Some limits on our

capacity for processing information. Psychological Review 63 (1956) 81–97

[58]

Peterson, L., Peterson, M.J.: Short-term retention of individual verbal items.

Journal of Experimental Psychology 58 (1959) 193–198

[59]

Chandler, P., Sweller, J.: Cognitive Load Theory and the Format of Instruction.

Cognition and Instruction 8(1991) 293–332

[60]

Bannert, M.: Managing cognitive load—recent trends in cognitive load theory.

Learning and Instruction 12 (2002) 139–146

[61]

Boughzahla, I., Tantan, O., Lang, D.: Feedback on the Integration of a Serious

Game in the Business Process Management Learning. In: Proceedings of the

21st Americas Conference on Information Systems (AMCIS), Fajardo, Puerto Rico

(2015) 3025–3034

[62]

Pavaloiu, I.B., Neagu, A.M., Dragoi, G., Rosu, S.M.: IBM INNOV8 business

simulation games for e-learning and education. In: Proceedings of the 12th

Bibliography

International Scientific Conference eLearning and Software for Education (eLSE),

Bucharest, Romania (2016) 534–540

[63]

Ploder, C., Bernsteiner, R., Schlögl, S.: Improving Business Process Management

Competencies by Applying Gamification Aspects in Teaching Bachelor Students.

In: Proceedings of the 7th International Workshop on Learning Technology for

Education Challenges (LTEC), Žilina, Slovakia (2018) 15–23

[64]

Morschheuser, B., Hrach, C., Alt, R., Lefanczyk, C.: Gamifizierung mit BPMN.

HMD Praxis der Wirtschaftsinformatik 52 (2015) 840–850

[65]

Mendling, J., Strembeck, M.: Influence Factors of Understanding Business

Process Models. In: Proceedings of the 11th International Conference on Business

Information Systems (BIS), Innsbruck, Austria (2008) 142–153

[66]

Reijers, H.A., Mendling, J.: A Study into the Factors that Influence the

Understandability

of Business Process Models. IEEE Transactions on Systems,

Man, and Cybernetics-Part A: Systems and Humans 41 (2010) 449–462

[67]

Mendling, J.: Detection and Prediction of Errors in EPC Business Process Models.

PhD thesis, Vienna University of Economics and Business Administration (WU

Wien), Austria (2007)

[68]

Mendling, J.: Metrics for Process Models: Empirical Foundations of Verification,

Error Prediction, and Guidelines for Correctness. Springer (2008)

[69]

Mendling, J., Reijers, H.A., Cardoso, J.: What Makes Process Models

Understandable?

In: Proceedings of the 5th International Conference on Business

Process Management (BPM), Brisbane, Australia (2007) 48–63

[70]

Sarshar, K., Loos, P.: Comparing the Control-Flow of EPC and Petri Net from the

End-User Perspective. In: Proceedings of the 3rd International Conference on

Business Process Management (BPM), Nancy, France (2005) 434–439

[71]

Reijers, H., Mendling, J.: Modularity in Process Models: Review and

Effects

In: Proceedings of the 6th International Conference on Business Process

Management (BPM), Milan, Italy (2008) 20–35

Bibliography

[72]

Baß: Investigating the Effects of Modularization in Business Process Models on

the Cognitive Complexity of Humans. Master’s thesis, Ulm University, Germany

(2019)

[73]

Cardoso, J., Mendling, J., Neumann, G.: A Discourse on Complexity of Process

Models. In: Proceedings of the 4th International Workshop on the Business

Process Mangagement (BPM), Vienna, Austria (2006) 117–128

[74]

Mendling, J., Reijers, H.A., van der Aalst, W.M.: Seven process modeling

guidelines (7PMG). Information and Software Technology 52 (2010) 127–136

[75]

Becker, J., Rosemann, M., Uthmann, C.: Guidelines of Business Process Modeling.

In: Business Process Management, Springer, Berlin, Germany (2000) 127–136

[76]

Lindland, O.I., Sindre, G., Solvberg, A.: Understanding Quality in Conceptual

Modeling. IEEE Software 11 (1994) 42–49

[77]

Krogstie, J., Sindre, G., Jørgensen, H.: Process models representing knowledge

for action: a revised quality framework. European Journal of Information Systems

15 (2006) 91–102

[78]

Riding, R., Cheema, I.: Cognitive Styles—an overview and integration. Educational

Psychology 11 (1991) 193–215

[79]

Singh, Y.K.: Fundamental of Research Methodology and Statistics. New Age

International (2006)

[80]

Juristo, N., Moreno, A.M.: Basics of Software Engineering Experimentation.

Springer (2013)

[81]

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.:

Experimentation in Software Engineering. Springer (2012)

[82]

Tversky, B., Morrison, J.B., Betrancourt, M.: Animation: can it facilitate?

International Journal of Human-Computer Studies 57 (2002) 247–262

[83]

Ayres, P., Paas, F.: Making Instructional Animations More Effective: A Cognitive

Load Approach. Applied Cognitive Psychology 21 (2007) 695–700

Bibliography

[84]

Wohlin, C., Hörst, M., Henningsson, K.: Empirical Research Methods in Software

Engineering. In: Empirical Methods and Studies in Software Engineering, Springer,

Berlin, Germany (2003) 7–23

[85]

Mason, R.L., Gunst, R.F., Hess, J.L.: Statistical Design and Analysis of

Experiments

: With Applications to Engineering and Science. 2nd ed. John Wiley

& Sons (2003)

[86]

SogoSurvey: Online Survey Tool. (Website) Online available at

https://www.

sogosurvey.com, last accessed on: 01. June 2020.

[87]

Brace, I.: Questionnaire Design: How to plan, structure and write survey material

for effective market research. 4th ed. Kogan Page (2018)

[88]

Klepsch, M., Schmitz, F., Seufert, T.: Development and Validation of Two

Instruments

Measuring Intrinsic, Extraneous, and Germane Cognitive Load.

Frontiers in Psychology 8(2017) 8:1997

[89]

YouTube: Broadcast Yourself. (Website) Online available at

https://www.

youtube.com; last accessed on: 04 June 2020.

[90]

Signavio: BPM Academic Initative. (Website) Online available at

https://

academic.signavio.com/p/login; last accessed on: 04 June 2020.

[91]

Montgomery, D.C.: Design and Analysis of Experiments. 9th ed. John Wiley &

Sons (2017)

[92]

Charness, G., Gneezy, U., Kuhn, M.A.: Experimental methods: Between-subject

and within-subject design. Journal of Economic Behavior & Organization

(2012)

1–8

[93]

Runeson, P., Höst, M.: Guidelines for conducting and reporting case study

research in software engineering. Empricial Software Engineering

(2009)

131–164

[94]

Trochim, W.: Research Methods: The Essential Knowledge Base. 2nd ed.

Cengage Learning (2015)

Bibliography

[95]

Höst, M., Regnell, B., Wohlin, C.: Using Students as Subjects—A Comparative

Study of Students and Professionals in Lead-Time Impact Assessment. Empirical

Software Engineering 5(2000) 201–214

[96]

Slack, M.K., Draugalis Jr, J.R.: Establishing the internal and external validity of

experimental studies. American Journal of Health-System Pharmacy

(2001)

2173–2181

[97]

Moodle: Moodle Lernplattform Universität Ulm. (Website) Online available at

https://moodle.uni-ulm.de/login/index.php

, last accessed on: 29

June 2020.

[98]

Faul, F., Erdfelder, E., Buchner, A., Lang, A.G.: Statistical power analyses using

G* Power 3.1: Tests for correlation and regression analyses. Behavior Research

Methods 41 (2009) 1149–1160

[99]

Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Routledge

(1988)

[100]

Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures.

5th ed. CRC Press (2011)

[101]

Thomas, L., Juanes, F.: The importance of statistical power analysis: an example

from Animal Behaviour. Animal Behaviour 52 (1996) 856–859

[102]

Effinger, P.: A 3D-Navigator for Business Process Models. In: Proceedings of the

12th International Workshop on Business Process Management (BPM), Tallinn,

Estonia (2013) 737–743

[103]

Oberhauser, R., Pogolski, C., Matic, A.: VR-BPMN: Visualizing BPMN Models in

Virtual Reality. In: Proceedings of the 8th International Symposium on Business

Process Modeling and Software Design (BMSD), Vienna, Austria (2018) 83–97

[104]

Poppe, E., Brown, R.A., Recker, J.C., Johnson, D.M.: A prototype augmented

reality collaborative process modelling tool. In: Proceedings of the Demo Track

of the 9th International Conference on Business Process Management (BPM),

Clermont-Ferrand, France (2011) 1–5

Bibliography

[105]

Bera, P.: Does Cognitive Overload Matter in Understanding BPMN Models?

Journal of Computer Information Systems 52 (2012) 59–69

Process Documentation

This appendix is comprised of the two additional process documentation used in this

study. Each treatment consisted of a different way of documenting the process: subjects

received either a process model, a textual process description or a process model

extended with graphics. The latter two are presented in this appendix.

A Process Documentation

Figure A.1: Textual Process Description: Warehouse Scenario - Part I

Figure A.2: Textual Process Description: Warehouse Scenario - Part II

A Process Documentation

Figure A.3: Process Model extended with Graphics: Warehouse Scenario - Part I

Figure A.4: Process Model extended with Graphics: Warehouse Scenario - Part II

Study

In this appendix, the materials of the study are presented. As each treatment consisted

of a different process documentation, each variant is provided.

Section 1: Introduction

Figure B.1: Introduction of the Study

B Study

Section 2: Demographic Questionnaire

Figure B.2: Demographic Questionnaire - Part I

Figure B.3: Demographic Questionnaire - Part II

B Study

Section 3: Gaming Experience Questionnaire

Figure B.4: Note on the Next Step - I

Figure B.5: Gaming Experience Questionnaire

Section 4: Virtual Process Environment

Figure B.6: Note on the Next Step - II

Figure B.7: Virtual Process Environment

B Study

Section 4: Variants of the Process Documentation of the 3D-Warehouse Scenario

Treatment: Process Model

Figure B.8: Note on the Next Step (Process Model) - III

Figure B.9: Treatment: Process Model - Part I

Figure B.10: Treatment: Process Model - Part II

B Study

Figure B.11: Treatment: Process Model - Part III (Cognitive Load Questionnaire)

Treatment: Textual Process Description

Figure B.12: Note on the Next Step (Textual Process Description) - III

Figure B.13: Treatment: Textual Process Description - Part I

B Study

Figure B.14: Treatment: Textual Process Description - Part II

Figure B.15:

Treatment: Textual Process Description (Cognitive Load Questionnaire) -

Part III

B Study

Treatment: Process Model extended with Graphics

Figure B.16: Note on the Next Step (Process Model extended with Graphics) - III

Figure B.17: Treatment: Process Model extended with Graphics - Part I

Figure B.18: Treatment: Process Model extended with Graphics - Part II

B Study

Figure B.19:

Treatment: Process Model extended with Graphics - Part III (Cognitive Load

Questionnaire)

Section 5: Preference for one Process Documentation

Figure B.20: Note on the Next Step - IV

Figure B.21: Variant I: Process Model - Part I

B Study

Figure B.22: Variant I: Process Model - Part II

Figure B.23: Variant II: Process Model extended with Graphics - Part I

B Study

Figure B.24: Variant II: Process Model extended with Graphics - Part II

100

Figure B.25: Variant III: Textual Process Description - Part I

101

B Study

Figure B.26: Variant III: Textual Process Description - Part II

102

Section 6: Generation of the Individual Code

Figure B.27: Instruction for Generating the Code - Part I

103

B Study

Figure B.28: Instruction for Generating the Code - Part II

Section 7: Feedback and Acknowledgment

Figure B.29: Feedback and Acknowledgment for Participation

104

List of Figures

2.1 View of the Structure of Retina: A Cross-Section [23] . . . . . . . . . . . . 6

3.1 Variables in a Study [84] (adapted) . . . . . . . . . . . . . . . . . . . . . . 24

3.2 Extract from the Video: Starting Sequence . . . . . . . . . . . . . . . . . 27

3.3 Extract from the Video: Picking the Goods Using the Forklift . . . . . . . . 28

3.4 Process Model: Warehouse Scenario - Part I . . . . . . . . . . . . . . . . 29

3.5 Process Model: Warehouse Scenario - Part II . . . . . . . . . . . . . . . . 30

3.6 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1 Work Status and Course of Studies . . . . . . . . . . . . . . . . . . . . . . 39

4.2 Gender Distribution and Treatment Distribution . . . . . . . . . . . . . . . 39

5.1 Boxplot of ICL, GCL and ECL Depending on Treatments . . . . . . . . . . 44

A.1 Textual Process Description: Warehouse Scenario - Part I . . . . . . . . . 78

A.2 Textual Process Description: Warehouse Scenario - Part II . . . . . . . . 79

A.3 Process Model extended with Graphics: Warehouse Scenario - Part I . . 80

A.4 Process Model extended with Graphics: Warehouse Scenario - Part II . . 81

B.1 Introduction of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

B.2 Demographic Questionnaire - Part I . . . . . . . . . . . . . . . . . . . . . 84

B.3 Demographic Questionnaire - Part II . . . . . . . . . . . . . . . . . . . . . 85

B.4 Note on the Next Step - I . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

B.5 Gaming Experience Questionnaire . . . . . . . . . . . . . . . . . . . . . . 86

B.6 Note on the Next Step - II . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

B.7 Virtual Process Environment . . . . . . . . . . . . . . . . . . . . . . . . . 87

B.8 Note on the Next Step (Process Model) - III . . . . . . . . . . . . . . . . . 88

B.9 Treatment: Process Model - Part I . . . . . . . . . . . . . . . . . . . . . . 88

B.10 Treatment: Process Model - Part II . . . . . . . . . . . . . . . . . . . . . . 89

B.11 Treatment: Process Model - Part III (Cognitive Load Questionnaire) . . . . 90

B.12 Note on the Next Step (Textual Process Description) - III . . . . . . . . . . 91

105

List of Figures

B.13 Treatment: Textual Process Description - Part I . . . . . . . . . . . . . . . 91

B.14 Treatment: Textual Process Description - Part II . . . . . . . . . . . . . . . 92

B.15

Treatment: Textual Process Description (Cognitive Load Questionnaire) -

Part III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

B.16 Note on the Next Step (Process Model extended with Graphics) - III . . . 94

B.17 Treatment: Process Model extended with Graphics - Part I . . . . . . . . . 94

B.18 Treatment: Process Model extended with Graphics - Part II . . . . . . . . 95

B.19

Treatment: Process Model extended with Graphics - Part III (Cognitive

Load Questionnaire) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

B.20 Note on the Next Step - IV . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

B.21 Variant I: Process Model - Part I . . . . . . . . . . . . . . . . . . . . . . . 97

B.22 Variant I: Process Model - Part II . . . . . . . . . . . . . . . . . . . . . . . 98

B.23 Variant II: Process Model extended with Graphics - Part I . . . . . . . . . 99

B.24 Variant II: Process Model extended with Graphics - Part II . . . . . . . . . 100

B.25 Variant III: Textual Process Description - Part I . . . . . . . . . . . . . . . 101

B.26 Variant III: Textual Process Description - Part II . . . . . . . . . . . . . . . 102

B.27 Instruction for Generating the Code - Part I . . . . . . . . . . . . . . . . . 103

B.28 Instruction for Generating the Code - Part II . . . . . . . . . . . . . . . . . 104

B.29 Feedback and Acknowledgment for Participation . . . . . . . . . . . . . . 104

106

List of Tables

3.1 Demographic Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.2 Gaming Experience - Questionnaire . . . . . . . . . . . . . . . . . . . . . 25

3.3 Cognitive Load Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1 Median of Familiarity, Perceived Confidence and Perceived Competence . 40

4.2

Results of Frequency Distribution of Data Depending on Different Variables

5.1 Descriptives for ICL, GCL and ECL . . . . . . . . . . . . . . . . . . . . . . 45

5.2 Allocation of Answers in terms of Correspondence . . . . . . . . . . . . . 45

5.3 Summary of Results for Preference . . . . . . . . . . . . . . . . . . . . . . 46

5.4 Distribution of the Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.5 Descriptives for High and Low Gaming Experience . . . . . . . . . . . . . 46

5.6 Descriptives for High and Low Modeling Experience . . . . . . . . . . . . 47

5.7 Results of Test of Significance (One-way ANNOVA) . . . . . . . . . . . . . 48

5.8 Results of Test for Significance (Two-Sample T-Test) . . . . . . . . . . . . 50

5.9 Results of Test for Significance (Likelihood-Ratio Chi-Square Test) . . . . 51

5.10 Standardized Residuals of Self-Confidence and Treatment) . . . . . . . . 51

5.11 Interaction Effects between Expertise and Treatments . . . . . . . . . . . 52

5.12 Results of Power-Analysis and Estimation of Sample Sizes . . . . . . . . 53

107

Name: Carina Spitzer Matriculation number: 910620

Honesty disclaimer

I hereby affirm that I wrote this thesis independently and that I did not use any other

sources or tools than the ones specified.

Ulm,

.................... .............................................

Carina Spitzer