An Evaluation Approach for Analyzing
Workflow Management Systems from a Value-based Perspective
Bela Mutschler, Johannes Bumiller
DaimlerChrysler Research & Technology
P.O. Box 2360, 89013 Ulm, Germany
{bela.mutschler;johannes.bumiller}@daimlerchrysler.com
Manfred Reichert
University of Twente
Information Systems Group
Abstract
Workflow management systems (WfMS) have become a
widely accepted software technology, which enables the ef-
fective management, execution, and monitoring of comput-
erized processes. Though the introduction of WfMS re-
sults in high costs, corresponding investments are often jus-
tified solely by referring to assumed benefits such as im-
proved business process performance. In fact, there ex-
ists no framework to systematically evaluate WfMS from a
value-based perspective. One major reason for this draw-
back is that research about the economic impact of WfMS
has to deal with many challenges like the potential iden-
tification of impact factors, the handling of dependencies
among these impact factors, the collection of real-world
data, and the quantification of costs and intangible ben-
efits. This paper presents an evaluation methodology to
tackle these problems, which is based upon modeling and
simulation.
1. Introduction
Workflow management systems (WfMS) have become a
widely accepted software technology in practice [5]. Very
often, WfMS are embedded in middleware tools or in
process-oriented applications. In particular, WfMS enable
the effective management, execution, and monitoring of
computerized processes.
Technical issues related to WfMS have been intensively
investigated in literature [22]. By contrast, what has been
neglected so far is the systematic analysis of costs and ben-
efits related to the introduction of WfMS. In fact, projects
implying the use of WfMS are usually justified by referring
to technical feasibility and assumed benefits (e.g., improved
business process performance [14, 18]).
Picking up this issue, this paper presents an evaluation
methodology to analyze the introduction of a WfMS from
a value-based perspective. Impact factors and their depen-
dencies are considered as ”economic systems” whose struc-
ture and behavior are modeled and simulated in order to
acquire knowledge about the economic impact of WfMS
(e.g., through ”what-if” analyses). Section 2 describes our
evaluation methodology. Section 3 discusses related work.
Finally, the paper concludes with a summary and an outlook
in Section 4. The work presented in this paper is part of the
EcoPOST project [1].
2. Modeling and Simulating the Economics of
Workflow Management Systems
This section introduces our evaluation methodology to
analyze WfMS from a value-based perspective. In partic-
ular, our approach is based on the design of systems of
WfMS-specific evaluation variables. The structure and be-
havior of such systems are modeled and simulated to make
economic effects transparent.
To build our evaluation models, we use System Dynamics
[9, 15, 19] as modeling formalism (instead of developing a
new one). In particular, System Dynamics is well-suited for
our research. Suitability in our context means to increase
awareness regarding the economic impact resulting from
the introduction of WfMS. Thus, System Dynamics is suit-
able as it can be considered as a conscious-raising tool [10].
In particular, System Dynamics enables us to systemati-
cally analyze economic-driven evaluation variables as well
as their dependencies.
There are other modeling and simulation approaches
that could be used as well. In particular, Bayesian Networks
provide an interesting alternative. A Bayesian Network
is a graphical description of probability distributions that
permits probability propagation, i.e., the graph is used to
calculate probabilities. A Bayesian Network is represented
as a directed acyclic graph in which nodes reproduce vari-
ables and edges indicate causal dependencies between pairs
of variables. However, despite the structures of Bayesian
Networks and System Dynamics seem to be similar, there
is a fundamental difference between the meaning and
applicability of both approaches. In particular, Bayesian
Networks consider uncertainty, i.e., they allow for the
determination of the probability of specific events. By
contrast, System Dynamics does not model uncertainty. Its
goal is to determine the impact of certain hypotheses. Deal-
ing with uncertainty in our context, it becomes clear that
we can identify the cost and impact factors that determine
WfMS economics. We also know, for instance, that certain
costs will occur (e.g., the costs related to the adaptation
of process logic in the case of evolving processes). Thus,
uncertainty has not to be handled and we decided to use
System Dynamics. A detailed introduction to the formalism
of System Dynamics can be found in [9, 19].
We analyze the introduction of WfMS from different
viewpoints. In particular, we identify three perspectives1:
1. The first evaluation perspective (EP1) covers the im-
pact of WfMS on the efficiency and effectiveness of
business processes, e.g., regarding the reduction of
processing times due to the introduction of process au-
tomation technologies. Usually, quantifications based
on processing times and process resources allocated
constitute the evaluation baseline in this context.
2. The second evaluation perspective (EP2) analyzes the
impact of WfMS on software development. This im-
plies direct and indirect effects.
3. The third evaluation perspective (EP3) analyzes the
impact of WfMS on software maintenance. As an ex-
ample consider the need to adapt process-oriented in-
formation systems to changing business processes.
Figure 1 summarizes the major methodical steps of our
approach. Step 1 deals with the specification of evalua-
tion scenarios (Section 2.1). Step 2 identifies those fac-
tors that influence the specified evaluation scenario (Section
2.2). Step 3 concerns the design of evaluation models using
the formalism of System Dynamics (Section 2.3). Finally,
Step 4 deals with the interpretation of our evaluation models
through computer simulation (Section 2.4).
2.1 Specifying an Evaluation Scenario
The first step of our evaluation methodology concerns
the ”research problem” to be analyzed: ”Only by knowing
1Other evaluation perspectives, e.g., to evaluate the technical maturity
of a WfMS, can be introduced as well if required.
the questions to be answered, we can safely judge the per-
tinence of factors to include in or omit from the system for-
mulation” [9].
Formalism for
Evaluation Models =
System Dynamics
Definition of Evaluation
Scenario (Research
Problem to be analyzed)
Identification
of Variables relevant for
Research Problem
Design of Evaluation
Models (Modeling Sys-
tem Structure & Behavior
Interpretation of the
Evaluation Model(s)
through Simulation
1 2
34
Research
Question
Reference
Behavior
Mode
Cost
Factors
Impact
Factors
Causal
Loop
Diagrams
Flow
Graphs
Quantifi-
cation
Graphical
Chart
Tools
Major methodical step
Tools for conducting a methodical step (cf. )
x y with x and y = „major methodical step“: x enables y
Section 3.1 Section 3.2
Section 3.3Section 3.4
Section 3.x
Figure 1. Methodology Overview.
Research problems are typically described using simple
questions. For example, typical problems addressed in the
context of our research are as follows: What costs are re-
lated to the introduction of a WfMS? What benefits? How
does business process fragmentation influence the costs of
introducing a WfMS? What economic effects are caused by
explicitly describing process logic and by separating it from
application code? How does ”emotional resistance” of end
users influence the costs of introducing a WfMS?
Process
Adaptation Costs
0Business Process Fragmentation
low high
low high
Figure 2. Specifying an Assumption.
To further enhance clarity, it is usual to specify an ex-
pected answer for a research problem by means of a graphi-
cal diagram. Such a diagram illustrates the assumed course
of one or more model variables along the time axis. As
an example consider the graphical diagram shown in Figure
2. The diagram shows the assumed costs related to busi-
ness process fragmentation. Here the logic of a particular
business process is scattered over several process-oriented
applications (e.g., legacy systems and off the shelf compo-
nents). Changing the business process (i.e., its logic) may
therefore necessitate the adaptation of all applications that
deal with the changed process part. Typically, a high num-
ber of process fragments imply the need to adopt many ap-
plications when process changes occur. Consequently, the
costs of process adaptation increase.
2.2 Cost and Impact Factors
The second step of our evaluation methodology deals
with the identification of relevant evaluation variables.
Basically, our approach is cost-driven, i.e., our basic
measure and subject to evaluation are costs (e.g., costs re-
lated to the introduction of a WfMS). Direct cost factors (cf.
Fig. 3) are related to the WfMS itself. They include, for ex-
ample, software licenses or project resources. Furthermore,
we must also consider costs indirectly caused by the intro-
duction and operational use of WfMS. As an example con-
sider the efforts caused by the need to analyze and redesign
the business processes to be automated. To identify both di-
rect and indirect cost factors, we use existing methods such
as total cost of ownership.
Cost
Factors
Impact
Factors
Technology-specific Impact Factors
Organization-specific Impact Factors
Project-specific Impact Factors
Direct Cost Factors
Indirect Cost Factors
Costs adjust
Figure 3. Cost and Impact Factors.
Besides, there are many intangible impact factors that
additionally bias the costs of workflow technology. Often,
these impact factors are difficult to quantify. In particular,
we distinguish between three kinds of impact factors depen-
dent on the level of their effectiveness:
•Technology-specific impact factors deal with techni-
cal capabilities of a WfMS. Examples include the de-
gree of flexibility provided by the WfMS (e.g., regard-
ing the support of dynamic changes) or the scalability
offered.
•Organization-specific impact factors deal with organi-
zational issues that bias the economics of a WfMS. As
one example consider knowledge about organization-
specific processes. Another example concerns organi-
zational barriers (e.g., between departments) that cause
process interceptions.
•Project-specific impact factors deal with project fac-
tors that bias the economics of a WfMS. Examples
concern the domain knowledge of IT professionals
assigned to a project or the know-how of participat-
ing team members regarding the implementation of
WfMS-based information systems.
A basic set of potential impact factors following this
classification is given in Table 1. Many of them have been
collected by Parkes [16, 17]. Taking our practical expe-
riences gained in the automotive domain, we can confirm
most of these factors.
This set of impact factors represents a baseline for the
design of evaluation models with our approach. However,
it is subject to continuous review and extension. As always,
it is sometimes difficult to clearly assign one impact fac-
tor to one of the aforementioned categories. Impact factors
that could be also assigned to another category are therefore
designated with the alternative categories (cf. Table 1).
2.3 Model Formulation
The third step of our evaluation methodology concerns
the design of evaluation models. Evaluation models are
”systems of variables”. In the context of our research, vari-
ables are WfMS-specific cost and impact factors (see Sec-
tion 2.2). In particular, we focus on the identification and
modeling of causal dependencies among evaluation vari-
ables and analyze the dynamic behavior that is caused by
these dependencies.
A causal dependency between two variables is depicted
as an arrow (cf. Fig. 4). To illustrate the direction of causal-
ity, arrows can be labeled with a ”+” or ”-” [21]. A la-
bel ”+” indicates that the variable at the opposite end of
the arrow tends to move in the same direction (in the case
of a change). A label ”-” indicates a reverse relationship.
Arrows that are not labeled describe dependencies between
variables with varying directions of causal influence. Con-
sider the example shown in Figure 4. The model depicts
the causal impact of cost and benefit variables on a WfMS’s
return of investment. Benefits increase the return of invest-
ment, whereas costs decrease this ratio.
WfMS Return on
Investment (ROI) WfMS Costs
+
+
-
-
Cost
Driver
+
Value
Driver
+
WfMS Benefits
Positive Causal Influence
+
Negative Causal Influence
-
Notation:
[Text] Variable
Figure 4. Modeling Causal Effects.
Dynamic system behavior is generated by chains of
causes and effects within a system. Dynamic behavior im-
plies that a system has different states over time. To be
01: Process TransparencyII
I. Organization-specific Impact Factors
02: Knowledge about existing Business ProcessesII
03: Information about existing Business ProcessesII
04: Management StructureII
05: Management CommitmentII
11: Ability to use emerging TechnologyI
06: Need for Job RedesignII
07: End User OwnershipII
01: Motivation for the Project
II. Project-specific Impact Factors
02: Ability to reengineer ProcessesI
03: Used Workflow ModelsI
04: Modeling Perspective
05: Access to required SkillsI
06: Domain KnowledgeI
07: Resource Redistribution
08: End User ParticipationI
01. Support for Groupwork Functionality
III. Technology-specific Impact Factors
02: Required Technology InfrastructureII
03: Reuse of existing TechnologiesII
12: Automation of Decision MakingII
04: Supported Level of Change
14: Planning of changeI
05: Structural Modification
L P
L P
L P
L-
L P
L-
L P
L-
L P
L P
L-
L-
L P
L P
L-
L P
L P
L-
L-
L-
L P
L P
L-
08: CommunicationII L-
09: Changing Social Cue & Interactions L P 09: Access to Information L-
L) described in the literature [15,16]
P) confirmed/derived by practical experiences
10: Process Maturity -P
12: Business Process FragmentationI-P
06: Number of IS to be IntegratedII L P
11: End User Fears & Emotional ResistanceII -P
13: Process EvolutionIII L P
10: Degree of potential WfMS SupportIII - P
07: Flexibility regarding Process Execution -P
08: Scalability of simultaneous Processes -P
09: Support of Standards and NormsII -P
10: Implementation of Process Logic -P
11: Support of Audit Trails -P
Table 1. Impact Factors determining the Economics of Workflow Management Systems.
able to trace these states, state variables continuously ac-
cumulate the results of actions in a system. A state is a
snapshot of a system at a specific point in time. Subject to
accumulation can either be tangible or intangible variables.
State variables are graphically represented by a rectangle
that comprises the name of the variable. State variables out-
side the analyzed system are called ”sources” or ”sinks”.
They are graphically represented by cloud-like symbols.
Costs of a Software
Development Project
Inflow / Outflow
Stock External State VariableState Variable
Notation:
Figure 5. State Variables.
State variables, i.e., their accumulation, change only
through inflows and outflows. Inflows increase and outflows
decrease a state variable. Graphically, both inflows and out-
flows are depicted by twin-arrows pointing into or out of
a state variable. As an example consider the state variable
”Costs of a Software Development Project” (cf. Figure 5).
The inflow increases the accumulated project costs. The
outflow, in turn, decreases the project costs.
Inflows and outflows are controlled by rates. Thus, rates
change accumulations of state variables. Graphically, rates
are depicted by valves (cf. Fig. 6). The ”Cost Rate”, for
example, controls the increase of the state variable ”Costs
of Introducing a WfMS”. The ”Reduction Rate” controls its
decrease.
A rate is usually determined by other system variables
respectively their causal influence. However, systems con-
sist not only of state variables. There are also variables that
Costs of
Introducing
a WfMS
Cost
Rate
Reduction
Rate
Rate
Notation:
Figure 6. Controlling Inflows and Outflows.
are not subject to accumulation. These variables are called
auxiliary variables. Regarding the determination of rates,
both state and auxiliary variables can influence a rate. As
example consider Figure 7.
Costs of
Introducing
a WfMS
WfMS
License
Costs
Effective
Communication
Cost
Rate
Reduction
Rate
Fear Reduc-
tion Rate
Emotional
Resistance of
End Users
Notation: [Text] Auxiliary Variable
Figure 7. Controlling Rates.
The ”Cost Rate” is influenced by an auxiliary variable
”WfMS License Costs”. The same rate is also influenced
by the state variable ”Emotional Resistance of End Users”.
The latter is reasonable as the costs for introducing a WfMS
are significantly influenced by user resistance [12, 20].
Emotional resistance, in turn, can be decreased by an early
and effective communication of background information
regarding the introduction of a WfMS.
The correct implementation and validity of evaluation
models is typically ensured by tests. Forrester [8], for ex-
ample, describes 17 tests that can be used to verify and vali-
date a model’s structure and behavior. Richardson and Pugh
[19] recommend using test functions, to conduct hypothesis
tests, and to do sensitivity analysis.
2.4 Model Interpretation
The fourth step of our evaluation methodology deals
with the interpretation of evaluation models (i.e., with the
derivation of conclusions). In order to understand how a
given system ”works” and what policies might change its
behavior, we use computer simulation.
Figure 8. Using Vensim [2] for Simulation.
The use of simulation becomes necessary as systems of
variables tend to get complex. This complexity often mis-
guides our intuition about the system’s behavior (i.e., about
the interactions of the variables). Simulation, by contrast,
enables us to trace selected variables along a defined time
horizon (e.g., along the course of project). However, this
does not necessarily imply the exact prediction of a specific
model variable based on a given set of initial conditions.
We use the visual modeling tool Vensim [2] to con-
struct, simulate, and analyze our evaluation models (cf. Fig.
8). This tool enables us to accomplish behavioral ”experi-
ments” by temporarily changing selected model variables
in a series of simulation runs. The values of all model vari-
ables are stored for each simulation run. These data can be
further analyzed with different graphical chart tools.
Typically, at least two simulation runs are performed.
A first run (assuming a variable of interest as ”low”)
defines the baseline for further comparisons. A second
run depicts the effects when the variable of interest is
changed. The results of several simulation runs are finally
compared (based on graphical charts) to derive conclusions.
3. Related Work
We discuss related work along the three evaluation per-
spectives described in Section 2.
Several papers address the impact of WfMS on business
process performance (evaluation perspective EP1). Oba et
al. [14] analyze the introduction of WfMS and particularly
focus on the identification of factors that influence work
efficiency, processing time, and business process standard-
ization. A mathematical model is provided for predicting
the reduction rate of processing times. An extension is the
work of Reijers and van der Aalst [18]. They use process
simulation to compare pre- and post-implementations of in-
formation systems that rely on workflow management tech-
nology. Their focus is on analyzing business process per-
formance based on criteria such as lead time, waiting time,
service time, and utilization of resources. The use of work-
flow management technology has resulted in most cases in a
significant decrease of lead and service time. Choenni et al.
[7] present a model to measure the added value of WfMS to
business processes that builds upon different performance
criteria such as speed, quality, flexibility, and reliability. A
performance criterion is a parameter of a business process
that is improved or compounded by the introduction of a
WfMS. The overall economic impact of a WfMS is calcu-
lated from the costs related to these four performance cri-
teria. Aiello [4] introduces a measurement framework for
the evaluation of workflows. The framework is defined in
an abstract setting to enable generality and ensure indepen-
dence from existing WfMS.
Only few approaches deal with the economic impact of
WfMS on software development and software maintenance
(evaluation perspectives EP2 and EP3). Parkes, for exam-
ple, analyzes critical success factors for WfMS implemen-
tations based on a survey [16] and a case study [17]. Three
critical success factors are considered as being of particu-
lar importance: management commitment, communication,
and participation by end users. Empirical studies [11] also
indicate that the effort for realizing process-oriented appli-
cations can be significantly reduced when using WfMS.
There are other approaches that deal with further aspects
regarding the economics of WfMS. Becker et. al [6] have
developed a framework to identify those processes that can
be supported by WfMS in a ”profitable” way. Their frame-
work can serve as guideline for evaluating processes dur-
ing the selection and introduction of a WfMS. It contains
three groups of criteria: technical, organizational and eco-
nomic. Designed as a scoring model, their approach en-
ables users to systematically determine those business pro-
cesses that can be automated using a WfMS. A different
approach is proposed by Abate et al. [3], who introduce a
novel measurement language to evaluate the performance of
automated business processes: the ”workflow performance
query language” (WPQL). This language allows to define
and to perform measurements independent from a specific
WfMS implementation. It provides different mechanisms to
select the workflow entities that are to be measured. Con-
structs for defining metrics are also defined.
All these approaches analyze very specific facets (e.g.,
business process performance improvements). However,
the combined incorporation of all three evaluation perspec-
tives is addressed by none of them. Our approach, by con-
trast, enables practitioners as well as researchers to evaluate
WfMS from a holistic perspective as all three evaluation
perspectives can be addressed.
4. Summary and Outlook
This paper presents a methodology for evaluating costs
and benefits that arise when introducing WfMS. Doing so,
we take a value-based perspective. In particular, our ap-
proach allows for the analysis of WfMS economics based
on the design and simulation of evaluation models. These
models follow the System Dynamics notation. Cost and im-
pact factors as well as their dependencies are considered as
”economic systems” whose structure and behavior are mod-
eled and simulated to make economic effects transparent.
Next steps will include the design and analysis of more
complex evaluation models. This implies the partial em-
pirical validation of underlying assumptions based on ex-
pert interviews, controlled experiments, and case studies.
Besides, it is our goal to identify further impact factors
(and suitable metrics to quantify them). Finally, we also
want to deliberate whether we can use our approach for the
economic-driven evaluation of other process-oriented soft-
ware technologies [13] as well (e.g., business process inte-
gration technologies).
5. Acknowledgement
The authors would like to thank Stefanie Rinderle (Uni-
versity of Ulm) for her valuable contribution to this paper.
References
[1] Economic-driven Evaluations of Process-oriented Soft-
ware Technologies (EcoPOST). Project Homepage:
www.mutschler.info/ecopost, 2006.
[2] Ventana Systems. Vensim: http://www.vensim.com/, 2006.
[3] A. F. Abate, A. Esposito, N. Grieco, and G. Nota. Work-
flow Performance Evaluation through WPQL. Proc. 14th
Int’l. Conf. Software Engineering and Knowledge Engineer-
ing (SEKE ’02), pp.489-495, 2002.
[4] R. Aiello. Workflow Performance Evaluation. PhD Thesis,
University of Salerno, Italy, 2004.
[5] Y. L. Antonucci. Using Workflow Technologies to improve
Organizational Competitiveness. Int’l. Journal of Manage-
ment, 14(1), pp.117-126, 1997.
[6] J. Becker, C. v. Uthmann, M. zur Muehlen, and M. Rose-
mann. Identifying the Workflow Potential of Business Pro-
cesses. Proc. 32nd Hawaii Int’l. Conf. on System Sciences
(HICSS ’99), 1999.
[7] S. Choenni, R. Bakkera, and W. Baetsa. On the Evaluation
of Workflow Systems in Business Processes. Electronic Jour-
nal of Information Systems Evaluation (EJISE), 6(2), 2003.
[8] J. Forrester and P. Senge. Tests for Building Confidence in
S-D Models. TIMS Studies in the Management Sciences,
volume 14, pp.208-228, 1980.
[9] J. W. Forrester. Industrial Dynamics. Productivity Press,
Cambridge, London, 1961.
[10] T. Haeberlein. Understanding Software Acquisition through
Modeling and Simulation. PhD Thesis, University of Ulm,
Germany, 2005.
[11] N. Kleiner. Can Business Process Changes Be Cheaper
Implemented with Workflow-Management-Systems? Proc.
Int’l. Conf. Information Resources Management Associa-
tion, pp.529-532, 2004.
[12] P. Kueng. The Effects of Workflow Systems on Organization:
A Qualitative Study. Proc. Business Process Management,
LNCS 1806, pp. 301-316, 2000.
[13] B. Mutschler, M. Reichert, and J. Bumiller. Designing
an Economic-driven Evaluation Framework for Process-
oriented Software Technologies. Proc. 28th Int’l. Conf. on
Software Engineering (ICSE ’06), pp.885-888, Shanghai,
China, 2006.
[14] M. Oba, S. Onoda, and N. Komoda. Evaluating the Quan-
titative Effects of Workflow Systems based on Real Cases.
Proc. 33rd Hawaii Int’l. Conf. on System Sciences (HICSS
’00), 2000.
[15] K. Ogata. System Dynamics. Prentice Hall, 2003.
[16] A. Parkes. Critical Success Factors in Workflow Implemen-
tation. Proc. 6th Pacific Asia Conference on Information
Systems (PACIS ’02), pp.363-380, 2002.
[17] A. Parkes. A Case Study of Workflow Implementation Suc-
cess Factors. Proc. 15th Australasian Conf. on Information
Systems (ACIS ’04), 2004.
[18] H. A. Reijers and W. M. P. van der Aalst. The Effectiveness
of Workflow Management Systems - Predictions and Lessons
Learned. Int’l. Journal of Information Management, 25(5),
pp.457-471, 2005.
[19] G. P. Richardson and A. L. Pugh. System Dynamics - Mod-
eling with DYNAMO. Productivity Press, Portland, 1981.
[20] A. Sarmento and A. Machado. Impact Evaluation of Organ-
isational Changes Enabled by Workflow Systems. Proc. 6th
Int’l. Workshop on Groupware, pp.134-137, 2000.
[21] J. D. Sterman. Business Dynamics - Systems Thinking and
Modeling for a Complex World. McGraw-Hill, 2000.
[22] W. M. P. van der Aalst and K. van Hee. Workflow Manage-
ment. MIT Press, 2004.
