Designing an Economic-driven Evaluation Framework
for Process-oriented Software Technologies
Bela Mutschler, Johannes Bumiller
DaimlerChrysler Research and Technology
P.O. Box 2360, 89013 Ulm, Germany
bela.mutschler@daimlerchrysler.com
johannes.bumiller@daimlerchrysler.com
Manfred Reichert
University of Twente
Information Systems Group
m.u.reichert@utwente.nl
ABSTRACT
During the last decade there has been a dramatic increase
in the number of paradigms, standards and tools that can
be used to realize process-oriented information systems. A
major problem neglected in software engineering research so
far has been the systematic determination of costs, benefits,
and risks that are related to the use of these process-oriented
software engineering methods and technologies. This task
is quite difficult as the added value is influenced by many
drivers. This paper sketches an economic-driven evaluation
methodology to analyze costs, benefits, and risks of process-
oriented software technologies and corresponding projects.
We introduce an evaluation meta model and sketch a for-
malism to describe economic-driven evaluation scenarios.
Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous
General Terms
Economics, Measurement
Keywords
process-oriented software technology, economic-driven eval-
uation models, added value, costs, benefits
1. PROBLEM OVERVIEW
Enterprises crave for possibilities to enable the seamless
execution and flexible control of their business processes
and services. In the automotive industry, a broad spec-
trum of business processes has to be supported ranging from
knowledge-intense engineering processes to administrative
financial services. In recent years, there has been an ex-
plosion of paradigms (e.g., workflow management, case han-
dling), standards (e.g., BPEL4WS), and tools (e.g., business
Copyright is held by the author/owner.
ICSE’06, May 20–28, 2006, Shanghai, China.
ACM 1-59593-085-X/06/0005.
process modeling tools, workflow management systems) en-
hancing business process support.
Process-oriented software (SW) technologies operate at
two levels [7]: business process integration and business pro-
cess management (BPM). The former focuses on technical
integration of application systems. Their overall objective
is to automate business processes by connecting application
systems in a process-oriented manner. Internal and exter-
nal integration scenarios have to be distinguished. Inter-
nal integration considers integration within one enterprise
whereas external integration deals with cross-organizational
integration. BPM technologies, by contrast, focus on the op-
erational support of business processes. Relevant issues are
the design, implementation, and management of process-
oriented SW systems, and the use of BPM technology to
control process execution, to monitor and analyze real-time
process runtime data, and to allow for quick adaptations to
evolving requirements.
One problem in practice is the lack of concepts for ana-
lyzing the costs, benefits, and risks related to the use of the
above mentioned process-oriented SW technologies. First
empirical studies have indicated that the total effort for re-
alizing process-oriented SW systems can be significantly re-
duced when using BPM technology [6]. What is additionally
needed is a comprehensive evaluation framework that allows
for both qualitative and quantitative conclusions about the
use of BPM and process-oriented SW technologies.
The development of such a framework is a complex task
that poses a number of challenges. First, we must iden-
tify those value drivers that influence the costs, benefits,
and risks related to the use process-oriented SW technology.
While some value drivers can be ascribed to technical capa-
bilities (e.g., system support for process evolution and pro-
cess change), others are related to organizational issues (e.g.,
the level of process maturity). Second, we must identify de-
pendencies between these drivers. Third, we must enable
both qualitative and quantitative conclusions. While qual-
itative conclusions can be based on practical experiences,
quantitative estimations additionally require adequate met-
rics (e.g., to enable the use of financial business ratios).
Fourth, we must identify different evaluation scopes and an-
alyze their dependencies. Possible evaluation scopes are, for
example, SW development and SW maintenance. Thereby,
different evaluation scopes can result in different, perhaps
even contradictory conclusions. High costs and low benefits
for SW development can be opposed, for example, by low
costs and high benefits for SW maintenance. Finally, we
885
must compare SW engineering based on ”traditional” ap-
proaches with SW engineering relying on process-oriented
methods and technologies.
In literature (e.g., [10]), many evaluation approaches are
discussed for analyzing the costs, benefits, and risks of IT.
Cost-oriented approaches, for example, focus on the identifi-
cation and quantification of IT costs. Respective approaches
are the value analysis, the zero base budgeting, the work
value model, the break even analysis, and the cost effective
analysis. Multi-dimensional approaches (e.g., the cost ben-
efit analysis and the excess tangible cost method), in turn,
do not only consider costs, but additional evaluation dimen-
sions (e.g., benefits), too. Market-oriented approaches (e.g.,
the competitive forces model) assess the impacts of IT on
a company’s market position. Strategy-oriented approaches
evaluate the contribution of IT for enterprise strategic ob-
jectives. Examples are the approaches of Parsons and Mc-
Farlan/McKinney. Customer-oriented approaches (e.g., the
customer resource life cycle model) analyze the impact of IT
from a customer point of view.
Process-oriented approaches, like the hedonic wage model
[9], analyze IT towards its role regarding the support of
business processes. However, IT is typically evaluated from
a mere business process performance perspective looking at
efficiency measures such as lead time (also known as cy-
cle time), waiting time, and resource allocation. They ig-
nore other, more intangible facilities arising from process-
oriented SW technologies (e.g., impact on organization). Fi-
nally, controlling-oriented approaches unify concepts such as
earned value management and target costing that especially
aggregate financial data provided by classic controlling.
Besides there are frameworks that address the assessment
of IT from a broader perspective. Value-based SW engi-
neering (VBSE), for example, explicitly adds value consid-
erations to SW engineering principles and practices [4]. The
e3 value framework is a requirements-driven evaluation ap-
proach that defines a business case based on requirements
[3]. The GRAAL framework investigates the alignment of
IT to business architectures [11]. The VITAL framework
investigates the problem of aligning the services offered by
networked ICT with the requirements of networked busi-
nesses from a value-oriented perspective [2].
All approaches can be applied to evaluate various facets
regarding the (economic) effects of IT. However, none of
them can be used to adequately analyze the economic-driven
impacts of process-oriented SW technologies (e.g., a tech-
nology’s impact on organizations or its effective support of
evolving business processes).
We have defined a number of research questions that are
of particular relevance for the design of our economic-driven
evaluation framework:
1. Which value drivers determine the costs, benefits, and
risks for the use of process-oriented SW technologies?
2. Which dependencies exist between these value drivers?
How can they be made transparent?
3. How can we formally describe value drivers and their
dependencies? Formalization becomes necessary in or-
der to ensure a systematic analysis.
4. How can we derive qualitative and quantitative con-
clusions for process-oriented SW technology? What
are suitable metrics that can be used in this context?
5. What effects do process-oriented SW technologies have
on the added value of the resulting enterprise applica-
tions? This is non-trivial as an application’s added
value is determined by a quantity of different value
drivers not related to the use of process-oriented SW
technology.
Section 2 sketches preliminary results of our approach1.
Section 3 gives an outlook on future work.
2. PRELIMINARY RESULTS
Analyzing the economic impacts that are related to the
use of process-oriented SW technologies is complex. One
reason for this is the strong impact such technologies have
on the organization. The effective use of workflow manage-
ment systems, for example, requires a deep investigation of
the business processes to provide optimal process support.
Another reason is that business process support may re-
quire the customization of existing information systems and
the process-oriented SW technologies they are implemented
with. This means that process-oriented SW technologies
have to be evaluated both towards their impact on organi-
zations and towards their ability to adopt to the (evolving)
requirements of the business processes they support.
2.1 EcoPOST Framework Overview
Our EcoPOST framework is addressing this by proposing
two basic evaluation models: the domain model (DM) and
the technology impact model (TIM). The TIM represents an
economic-driven evaluation for the use of process-oriented
SW technology within a particular organization. This ap-
plication context is described by the domain model.
Both evaluation models are based on the modeling for-
malism of System Dynamics [8]. System Dynamics was
originally developed to understand the behavior of complex
systems. Its underlying assumption is that it is compar-
atively easy to observe the elementary forces and actions
of a system, but difficult to estimate the dynamic interac-
tions of its parts. System Dynamics assumes that feedback
structures are the main determinants of a system’s behav-
ior. Its basic goal is the understanding of feedback systems.
We decided to use System Dynamics as it supports basic
requirements for the modeling of economic impacts when
compared to other formalisms such as bayesian networks [5].
It allows for the modeling of different types of value flows
(e.g., cost flows, benefit flows, and risks flows), the modeling
(and structuring) of complex ”value networks” (including
both technology-driven and organization-driven evaluation
variables), and the provision of mechanisms to model both
qualitative and quantifiable impacts.
The purpose of the domain model is to provide a precise
characterization of the treated evaluation scenario. Consid-
ering organization-specific information becomes necessary as
a generic evaluation of economic effects is hardly possible.
The domain model includes a textual description of an evalu-
ation scenario (project overview, technology use cases) and
models of the supported business processes (to facilitate,
1The work presented in this paper is part of the EcoPOST
project (Economic-driven Evaluations of Process-oriented
Software Technologies) conducted at DaimlerChrysler and
the University of Twente [1].
886
Evaluation
Scenario
Technology
Impact Model
Organization-driven
Critical Impact Factors
Process-
oriented
Software
Technology
(POST)
Technology-driven
Critical Impact Factors
Knowledge
Base
Textual
Scenario Description
Business
Process Model
Technology
Characterization
Impact
Models
Meta
Model
n:1
1:1 1:1
Meta
Model
Domain
Model
used in
stored in
stored in
stored in
[Technology View] [Organization View]
includes
includes
1
2
3
1
2
Causal Evaluation
Loop Diagram
4
Figure 1: The EcoPOST Evaluation Framework.
among other things, the use of process metrics). It also
includes a collection of organization-driven critical impact
factors (CIForg). Each CIForg represents an organizational
attribute (e.g., process fragmentation, process knowledge,
process transparency, or process maturity) that significantly
influences a SW technology’s economic efficiency. It is one
goal of the EcoPOST framework to provide a basic set of
CIForg that can be identified in every organization. How-
ever, it may be also necessary to identify further CIForg
(e.g., by interviews). Finally, a causal loop diagram (known
from System Dynamics) is used to illustrate dependencies
and causal influences of the CIForg. Causal loop diagrams
are simple diagrams to illustrate that there is some kind
of economic-driven causal influence of one evaluation vari-
able on another (without further quantifying this influence).
Evaluation variables are modeled as vertices that can be con-
nected with a directed edge which may be labeled with a ”+”
or ”-” to provide information about the type of causality. A
positive edge indicates that the variable at the opposite end
of the edge tends to move in the same direction. A negative
edge indicates a reverse relationship.
Fig. 2 shows an example that illustrates the notion of
causal loop diagrams. Thereby, the variable process maturity
(e.g., specified by process maturity reference models such
as CMMi or SPICE) is assumed to have a positive impact
on both the variables process knowledge and process trans-
parency (due to a systematic process documentation that is
required for improving process maturity). This means that
an increasing process maturity results in an increasing pro-
cess knowledge and process transparency. The variable pro-
cess fragmentation negatively affects the two variables pro-
cess maturity and process transparency. Finally, the variable
”process evolution” negatively affects the variable ”process
transparency”.
The technology impact model (TIM) aggregates economic
effects described by a collection of inferior impact models
(IM). This additional structuring is helpful to further re-
duce complexity. Consistency is ensured by the basic mod-
eling rules of System Dynamics. Impact models analyze
the economic effects of a combination of technology-driven
(CIFtech) and organization-driven impact factors. Exam-
ples of CIFtech are a technology’s representation of process
logic or its ability to allow for quick process changes. Both
the technology impact model and the inferior impact models
are represented by evaluation flow graphs, which constitute
when compared to causal loop diagrams a more complex
diagram type. Evaluation flow graphs distinguish, for ex-
ample, between quantitative and qualitative edges and ex-
plicitly treat the notion of accumulating costs, benefits, and
risks. A typical scenario that can be represented by an im-
pact model is described in Section 2.3.
[Process
Maturity]
[Process
Evolution]
[Process
Knowledge]
[Process
Transparency]
+
+
[Process
Fragmentation] - -
-
Figure 2: Analyzing Organization-driven Variables.
2.2 EcoPOST Evaluation Meta Model
In this section we present an evaluation meta model that
defines the modeling elements that can be used when build-
ing evaluation flow graphs (or (technology) impact models).
The basic elements of the EcoPOST meta model are shown
in Fig. 3. Economic evaluations must consider the notion of
accumulating costs, benefits, and risks. Such accumulations
are called value levels. We distinguish between basic value
levels and auxiliary value levels.
Basic value levels represent an aggregated economic con-
clusion for one evaluation dimension. The meta model dis-
tinguishes between total costs of ownership (TCO), total
value of ownership (TVO), and risks. TCO represents a
cost-centric accumulation, whereas TVO additionally incor-
porates benefits and risks, and therefore represents a more
holistic accumulation. The risk level aggregates the effects
of various risk drivers.
Auxiliary value levels represent sub accumulations of ba-
sic value levels in order to analyze them in more detail. The
TCO level, for example, can be divided into several inferior
cost levels to separately analyze single cost drivers. The
(technology) impact models unify several auxiliary value lev-
els in order to summarize economic-driven conclusions.
Impact flows are qualitative and quantitative correlations
between different value levels. They are represented by di-
rected edges. Thus, each edge represents an impact flow (cf.
Fig. 3). Impact flows are value streams, i.e., they indicate
that there exists an economic-driven causal influence of one
variable (respectively value level) on another. In order to
enhance systematic evaluations we additionally distinguish
between different types of impact flows:
1. Negative Cost Flow: This type represents costs that
887
can be financially quantified (e.g., costs for customiz-
ing out-of-the-box SW, maintenance costs, etc.).
2. Positive Cost Flow: This type represents benefits that
can be financially quantified (e.g., optimizations of ma-
jor business processes).
3. Non-financial Benefit Flow: This type represents ben-
efits that can be quantified, but not financially (e.g.,
minor optimizations of administrative processes).
4. Intangible Benefit Flow: This type represents benefits
that cannot be quantified - neither financially nor non-
financially (e.g., improvements in data quality).
5. Risk Flow: This type represents non-quantifiable risks
(e.g., missing user acceptance).
Total
Costs of
Ownership
Total
Benefit of
Ownership
Intangible
Benefit Flow Non-financial
Benefit Flow
Positive/Negative
Cost Flow
Intangible
Benefits
Levels
Non-finan-
cial Benefits
Levels
Cost
Levels
(Technology)
Impact Model
Domain
Model
Risks
Risk
Flow
Risks
Levels
unifies
unifies
Figure 3: EcoPOST Meta Model.
Altogether, these elements will be used to develop specific
EcoPOST evaluation models.
2.3 Example of an Evaluation Scenario
An example of an evaluation scenario that can be analyzed
in detail with the EcoPOST evaluation framework is the
economical impact that is related to a process-oriented SW
technology’s capability to effectively deal with evolving busi-
ness processes. Two evaluation variables are of particular
relevance: the variable ”process evolution” (organization-
driven) and the variable ”implementation of process logic”
(technology-driven). Both variables can have different val-
ues, i.e., the first variable can have the value ”frequent pro-
cess changes” to indicate dynamic process evolution and the
second variable can have values such as ”hard-wired process
logic” or ”separation of process logic from application code”.
One assumption regarding the correlation of these two vari-
ables is described in the following. We will use the EcoPOST
framework to formally analyze and quantify this scenario.
A separated process logic is advantageous (from an eco-
nomic-driven point of view), if process logic has to be fre-
quently adapted to process changes. As the part to be
changed (i.e., the process logic) is separately available, high
costs, e.g., for adopting the application code, can be avoided.
A separated process logic is disadvantageous, by contrast, if
a business process is static and never or rarely changes. The
costs related to the use of process modeling tools and work-
flow management systems (to enable a separate handling of
process logic) may be not justified by the benefits.
However, this is just a simple example. Of course, it is
our ambition to analyze more sophisticated scenarios as well
(e.g., based on case studies and experiments).
3. FURTHER INVESTIGATIONS
This paper has illustrated our research efforts to design an
economic-driven evaluation framework to analyze the costs,
benefits and risks of process-oriented SW technology. We
have sketched basic ideas of our evaluation framework and
introduced an evaluation meta model to formally describe
evaluation scenarios. An example has illustrated the basic
notion of the used formalism (a special occurrence of System
Dynamics). The applicability of the evaluation framework
in practice is a particularly critical success factor that has
guided our research efforts. Additionally, the use of well-
elaborated methods is success critical as well.
Next steps will include the definition of more detailed
(technology) impact models as well as their validation. Sev-
eral case studies and controlled experiments will be accom-
plished. Future work will also focus on the quantification of
evaluation scenarios. This implies the definition and speci-
fication of evaluation criteria as well as of suitable metrics
to quantify these criteria.
4. REFERENCES
[1] Economic-driven Evaluations of Process-oriented
Software Technologies (EcoPOST). Project Homepage:
www.mutschler.info/ecopost, 2006.
[2] Value-based IT Alignment (VITAL). Project
Homepage: www.vital-project.org, 2006.
[3] Z. Baida, H. de Bruin, and J. Gordijin. e-Business
Cases Assessment: From Business Value to System
Feasibility. Int’l. J. of Web Engineering and
Technology, 1(1), pp.127-144, 2003.
[4] B. Boehm. Value-Based Software Engineering. ACM
SIGSOFT Software Engineering Notes, 28(2), 2003.
[5] F. V. Jensen. Bayesian Networks and Decision
Graphs. Springer, 2002.
[6] N. Kleiner. Can Business Process Changes Be Cheaper
Implemented with Workflow-Management-Systems?
Proc. Int’l. Conf. Information Resources Management
Association, pp.529-532, 2004.
[7] B. Mutschler, M. Reichert, and J. Bumiller. Towards
an Evaluation Framework for Business Process
Integration and Management. 2nd Int’l. INTEROP
Workshop, Enschede, The Netherlands, 2005.
[8] K. Ogata. System Dynamics. Prentice Hall, 2003.
[9] P. G. Sasonne. Cost-Benefit Methodology for Office
Systems. ACM Transactions on Office Information
Systems, 5(3), pp.273-289, 1987.
[10] P. G. Sasonne. Cost Benefit Analysis of Information
systems: A Survey of Methodologies. 1988.
[11] P. van Eck, H. Blanken, and R. Wieringa. Project
GRAAL - Towards Operational Architecture
Alignment. Int’l. J. of Cooperative Information
Systems, 13(13), pp.235-255, 2004.
888
