Towards Context-aware Process Guidance in Cyber-Physical Systems with Augmented Reality [original]

Towards Context-aware Process Guidance in

Cyber-Physical Systems with Augmented Reality

Klaus Kammerer1, R¨

udiger Pryss1, Kevin Sommer2, Manfred Reichert1

1Institute of Databases and Information Systems, Ulm University, Germany

Email: {klaus.kammerer, ruediger.pryss, manfred.reichert}@uni-ulm.de

2Uhlmann Pac-Systeme GmbH & Co. KG, Laupheim, Germany

Email: sommer[email protected]

Abstract—Assembly, configuration, maintenance, and repair

processes in cyber-physical systems (e.g., a press line in a plant)

comprise a multitude of complex tasks, whose execution needs

to be controlled, coordinated and monitored. Amongst others,

a process-centric guidance of users (e.g. service operators) is

required, taking the high variability in the assembly of cyber-

physical systems (e.g. press line variability) into account. More-

over, the tasks to be performed along these processes may be

related to physical components, sensors and actuators, which

need to be properly recognized, integrated and operated. In

order to digitize cyber-physical processes as well as to guide

users in a process-centric way, therefore, we suggest integrating

process management technology, sensor/actuator interfaces, and

augmented reality techniques. The paper discusses fundamental

requirements for such an integration and presents an approach

for process-centric user guidance that combines context and

process management with augmented reality enhanced tasks. For

evaluation purposes, we analyzed the cyber-physical processes of

pharmaceutical packaging machines and implemented selected

ones based on the approach. Overall, we are able to demonstrate

the usefulness of context-aware process management for the

flexible support of cyber-physical processes in the Industrial

Internet of Things.

Index Terms—Cyber-physical System, Context-aware Process

Management, Augmented Reality Enhanced Process, Cyber-

physical Process

I. INTRODUCTION

The Industrial Internet of Things (IIoT) has become increas-

ingly pervasive in industry as digitization is a business priority

for almost every enterprise. In particular, the IIoT is essential

for realizing the Industry 4.0 vision, which shall enable new

opportunities in the digital transformation of manufacturing,

integrating smart machines, advanced analytics, and people at

work [1]. In this context, the building blocks of the IIoT, the

so-called cyber-physical systems (CPS), shall allow bridging

the physical and digital world and, thus, enable innovative dig-

ital services in manufacturing as well as laying the foundation

for smart plants and digital factories.

For the assembly, configuration, maintenance, and repair

of CPS, complex procedures–denoted as cyber-physical pro-

cesses in the following–exist. The latter comprise both human

and technical tasks, whose execution needs to be controlled,

coordinated and monitored. In this context, a process-centric

interaction between users and CPS is essential for bridging

the gap between digital and physical process. Moreover, the

complex tasks of a cyber-physical process may be related

to the physical components, sensors and actuators of a CPS,

which therefore must be properly recognized, integrated and

operated. While some of the tasks need to be performed

manually, others can be fully automated (e.g., collecting sensor

data on the machine) or be enhanced with digital services (e.g.,

maintenance enhanced with augmented reality).

Although the potential of digitizing cyber-physical pro-

cesses has been recognized in literature [2], the role of business

process management (BPM) technology in the digital transfor-

mation of manufacturing has not been well understood so far.

In general, cyber-physical processes are knowledge-intensive

and their proper support plays a key role in bridging the gap

between physical systems and their digital representations.

As a particular challenge, the variability of cyber-physical

processes needs to be properly handled [3]. The latter is

caused by the variety of configurations that may exist for

the assembly of a CPS. Think of a press line, for example,

whose concrete stations, hardware components, and software

modules may depend on contextual factors like customer-

specific requirements, legal regulations, or other site-specific

factors. As a consequence, there exists a high variability of

the processes for the assembly, configuration, maintenance

or repair of CPS, i.e., the concrete tasks of a cyber-physical

process and the digital data (e.g., checklists, instructions, or

videos) provided during task execution may vary depending on

contextual factors. Note that this process variability introduces

a huge complexity for service workers, particularly when

facing CPS that run over years or decades. Finally, in certain

cases, cyber-physical processes may have to be dynamically

adapted during runtime, e.g., to react to contextual changes

(e.g., retooling of a machine) or to cope with faulty sensors

and actuators.

Another fundamental challenge in bridging the gap between

physical and digital CPS components is to enhance the sense

of workers for their physical environment with the help of

digital data [4]. In this context, the integration of cyber-

physical processes with augmented reality (AR) technologies

offers promising perspectives, i.e., user guidance for assembly,

configuration, maintenance, and repair processes of CPS will

benefit from an AR enhanced task support [5]. In consequence,

the process-centric guidance of service operators based on the

enhancement of physical components with digital data should

be a key feature of any CPS [6]. Note that the instructions

for performing the technical tasks of a cyber-physical process

are more comprehensible if they are not only available as

paper manual, but also as digital 3D drawings superimposed

upon the actual physical system, showing step-by-step the

tasks to be performed and the way how to perform them

[7], [8]. Moreover, sensor data and task guidelines might be

incorporated through the AR interface of the service operator

in real time, and data be overlaid on and registered with the

actual equipment in his field of view.

In previous work, we developed advanced methods, con-

cepts and technologies that enable full life cycle support of

digital business processes [9], [10], [11], [12]. In principle,

respective techniques can be also applied to cyber-physical

processes, but need to be enhanced in order to meet the specific

requirements of the latter. Besides sophisticated configuration

facilities and AR-enhanced process guidance, advanced tech-

niques for the proper integration of human resources, sensors

and actuators with the digital processes are needed. This paper

gives first insights into the support of cyber-physical processes

and the particular challenges imposed by them. In detail, the

contributions of the paper are as follows:

•We present a CPS maintenance process from the field of

pharmaceutical packaging, which we discovered in a case

study, and discuss its characteristics.

•Taking this real-world cyber-physical process scenario,

together with insights from a literature survey, we derive

fundamental requirements for the design, implementation

and automation of cyber-physical processes.

•We present core components (i.e., context-aware process

injection, context modeling, and AR-enhanced process

guidance) of an approach that allows for the flexible spec-

ification, configuration and enactment of cyber-physical

processes, taking contextual factors (e.g., the CPS variant

the process refers to or individual preferences of service

operators) into account as well.

We consider these contributions as an important step for the

evolution of process management technology and its broader

use in the context of CPS.

Underpinning our research, we applied the design science

research methodology [13]. In particular, our work can be cat-

egorized as a design- and development-centered approach. By

analyzing a real-world application scenario (cf. Section II) and

by evaluating existing approaches (cf. Section III), we derived

requirements and solution objectives (cf. Section IV) , and then

iteratively elaborated concepts to address the requirements (cf.

Section V). Section VI concludes the paper with a summary

and outlook.

II. CYBER-PHYSICAL PROCESSES

To reveal basic challenges of cyber-physical processes, we

conducted a case study in the field of mechanical engineering.

During this case study, we identified, discovered and analyzed

scenarios dealing with maintenance processes of a press line

for pharmaceutical packaging, i.e., a CPS used to pack drugs

(e.g., tablets) into blister packs, blister packs into boxes,

and boxes into cartons. The CPS consists of interconnected

machines like a forming station (cf. Fig. 1a) to form blisters

into an aluminum or plastic foil or a filling station (cf.

Fig. 1b) to add tablets to the blister packs. Each station, in

turn, comprises various devices like a forward feeder or a

deflection pulley (cf. Fig. 1c) to lead the foil through the

machine. Moreover, all machines are equipped with sensors

and actuators that are controlled by a programmable logic

controller (PLC). Finally, all machines may be configured and

controlled with the help of a touch-sensitive human machine

interface (cf. Fig. 1d).

Case Study: Maintenance Processes of a Press Line

We analyzed the maintenance process of the Uhlmann

B1440 Blister Packaging Press Line, which has to be

periodically executed by a service operator (cf. Fig. 2).

The process is triggered if a certain time period has

passed since the previous maintenance cycle, certain

hardware components have been used for a particular

number of machine cycles, or any error occurs.

Initially, the service operator receives the maintenance

plan of the press line as well as information on the

maintenance intervals of its various parts (cf. Fig. 2a).

Based on this information, the operator defines a task

list, which may comprise hundreds of tasks to be

processed during maintenance. For every task, in turn,

a specific checklist or handbook guiding the operator

exists. Furthermore, the service operator has to check

basic constraints that need to be obeyed to ensure a

correct execution of the maintenance process. Finally,

he informs the operating staff about the upcoming

maintenance and appoints a maintenance supervisor.

Just before starting with maintenance, the operator

checks his technical equipment, puts the press line out

of operation, removes production goods, and prepares

the operating system of the stations. Furthermore,

he cleans the press line, e.g., by removing dirt and

lubricants from screw connections, which otherwise

might contaminate other parts during maintenance.

Following these preparatory steps, the operator starts

with the processing of the task list. Note that the latter

may vary depending on contextual factors, e.g., the type

of the press line or customer-specific service agreements

(cf. Fig. 2b).

As an example of a maintenance task consider the

calibration of a forming station, which is needed to

adjust material spacing between heating station and

cover plate (cf. Fig. 2c). When processing this task, a

number of (manual) sub-tasks (e.g., removing covers

and guide plates) have to be performed. In addition, the

heating circuit has to be cooled down to avoid skin burns

(cf. Fig. 2d). Finally, during the maintenance process,

the machine is partially running in a test production

mode with reduced speed (cf. Fig. 2e).

Note that the various maintenance tasks may refer to

Fig. 1. Uhlmann Blister Press Line B1440

different stations of the press line. For most parts,

the tasks related to different stations may be executed

independently from each other. However, in certain

cases, there exist subtle task interdependencies that need

to be considered for proper process completion. For

example, the replacement of a roll mounting requires its

cleaning after test run execution (cf. Fig. 2f).

If the machine operator reports specific issues, the

respective station has to be examined in greater detail.

Every change applied to the station in this context has

to be recorded. After completing maintenance, the press

line has to be cleansed again according to a pre-specified

procedure and lubricants have to be supplied to moving

parts of the machines. Finally, the service operator

documents all executed tasks.

III. RELATED WORK

The first category of related work deals with the use of

AR technologies in the context CPS maintenance. In many

industries, the efforts for maintaining complex machines and

systems could be reduced when using AR techniques. For

example, [8] were able to show that the replacement of paper-

based assembly instructions with AR enhanced ones led to

a significant reduction of assembly times. In turn, HUDSET

[6] supported factory workers with a head-mounted display

during the process of manufacturing an aircraft; on one hand,

AR support was highly welcome by workers, on the other

its introduction raised many technical issues. KARMA, an

AR application for laser printer maintenance [7], was con-

sidered as useful with respect to maintenance processes. In

turn, ARMAR [5] investigated the effects of applying real-

time overlays to the machine components to be maintained.

Amongst others, for the field of vehicle maintenance it was

shown that ARMAR allows for an intuitive and satisfying

experience of service operators. Considerable research on

the use of AR technologies in industrial environments was

conducted by the ARVIKA consortium [14]. A particular

feature developed in the context of this initiative, was the

workflow-driven selection and presentation of information to

service operators. As opposed to our approach, however, only

hard-coded and static workflows (i.e. executable processes)

were considered. Altogether, all mentioned AR approaches

either lack process management support for manual tasks or

show severe limitations regarding the context-based support of

process variants.

The second category of related work we consider stems

from the BPM field and deals with the challenges that arise in

the context of the Internet-of-Things (IoT). In particular, [2]

presented a catalogue of challenges that need to be tackled

when combining business process management with the IoT.

For example, in cyber-physical systems, various types of tasks

require the interplay between human operators and software

modules, which should be coordinated by a BPM system.

Therefore, an appropriate mapping from process activities (i.e.

tasks) to visualizations is needed allowing operators to perform

their work in an intuitive way.

Other related work deals with the context-driven execution

and adaption of business processes. For example, [15] presents

an automated approach for the context-aware extension of

running process instances in order to flexibly cope with

process variability and to increase process flexibility. Related

approaches deal with the configuration of process models

before deploying them to any execution platform (i.e. process

engine) [16], [17], the adaptation of running process instances

in the midst of their execution [9], the late selection of sub-

processes during the execution of a superordinate process

[18], and the late composition of process-centric services

[19]. However, these approaches do not consider any context-

aware AR enhanced process guidance as in our approach.

Moreover, they neither use contextual models nor contextual

data sources (e.g., CPS sensors) for configuring and adapting

running processes.

To enable contextual awareness in business process manage-

ment, process models and contextual models need be tightly

integrated. [20] presented an approach for the modeling of

business processes that support context descriptions. A tax-

onomy was introduced, which comprises contextual elements

related to location, time, human resources, and organizational

entities in order to integrate context related knowledge into

business processes. Additionally, a concept for context-aware

Process Fragment:

AdjustFormFoilRoll

Process Fragment:

CalibrateFormingStation

Process Fragment:

ReplaceRollMounting

Process Fragment:

CleanMountingRoll

Process Fragment:

DeactivateHeatingStation

A2:

Check Equipment

A1: Receive

Maintenance Plan

and Settings

Sheet

A3:

Stop and Release

Machine

Maintenance Plan

A4:

Conduct Cleaning

Variable Part 1:

Conduct

Maintenance

A5-6:

Replace Roll

Mounting

A5-7:

Get Setting Sheet

A5-8:

Adjust Form Foil

Roll

Situation S5:

Foil Dissolves On Edges

Situation S4:

Maschine stops with

Error „FormFoilEnd“

A5-3:

Calibrate

Deflection Polley

A5-4:

Deactivate

Heating Station

A5-2:

Calibrate Forming

Station

Situation S2:

Situation 1 occurs

Situation S1:

Poor Blister Pocket Forming

A6:

Execute Test Run

A5-1:

Remove Covers

Process Fragment:

CalibratePrintDevice

A5-5:

Calibrate Printer

Device

Situation S3:

S1 occurs and Printer

Device is present

Variable Part 2:

Conduct Cleaning

A8:

Document

Machine Changes

PF..

A7-1:

Clean Roll

Mounting

Settings Sheet

Situation S6:

Situation 4 occurs

depends on depends on

Settings Sheet

Fig. 2. Excerpt of a Machine Maintenance Process Model (Simplified Views)

process instance creation was introduced. However, context-

related knowledge is evaluated prior to process instance cre-

ation rather than during process instance execution. The latter,

however, is of utmost importance to be able to adapt a run-

ning process to evolving or changing context-related knowl-

edge. Finally, [21] proposed a framework considering various

context types and their integration with business processes,

e.g., including a goal-oriented process modeling approach for

context classification, formalization, and integration.

When characterizing contextual data, constraints of a certain

data set are often described based on contextual situations.

According to [22], contextual situations can be defined as

a semantic abstraction of low-level data . Furthermore, [22]

defined constraints that need to be evaluated against low-

level data to indicate the execution of certain actions enabling

situational awareness. In turn, [23] presented a rule-based

situation management framework to facilitate the design of

situation-aware applications and to manage contextual data.

However, this work rather focused on user conditions than

business processes. Finally, [24] presented an approach char-

acterizing the context of a business process activity through

conceptual models structured by contextual, process-, and

domain-specific ontologies. However, this approach did not

describe any concepts for integrating these conceptual models

with BPM systems.

IV. REQUIREMENTS

Taking the application scenario from Section II and the

insights we gained when studying the limitations of related

work, we identified three categories of requirements for the

support of cyber-physical processes, i.e., context-aware pro-

cesses,context management, and augmented reality support

(cf. Table I).

The maintenance process of a CPS may comprise hundreds

of tasks, whose composition varies depending on contextual

factors (see Requirements C1-1 and C1-2). For example,

devices may show different wear marks depending on the

previous degree of utilization. Moreover, a CPS may have

TABLE I

REQUIREMENTS

No Title Description

Category 1: Context-aware Processes

C1-1 Variability Support Support for flexible modeling

C1-2 Execution & Adap-

tation

Run time adaptations

C1-3 Traceability Logging

Category 2: Context Management

C2-1 Modeling Representation of a context model

C2-2 Sensor Integration Integrate sensor data into the context

model

C2-3 Evaluation Evaluate contextual situations

Category 3: Augmented Reality Support

C3-1 Interaction Interaction between users and a CPS

C3-2 Attention Guidance of a user’s attention

C3-3 Mapping Mapping of physical and virtual ob-

jects

been customized to the needs of individual applications; e.g.,

press lines in pharmaceutical packaging and, hence, related

processes vary due to different kinds of drugs to be packaged.

To meet Requirements C1-1 and C1-2, we introduce context-

aware process injection (CaPI) as fundamental concept, which

allows tailoring process execution to the execution context of

the respective cyber-physical process. In certain scenarios, the

execution of cyber-physical processes needs to be traceable as

well (see Requirement C1-3). Regarding pharmaceutical pack-

aging, for example, traceability support for changes during

packaging is mandatory due to the Title 21 Code of Federal

Regulations (CFR) Part 11. The latter defines guidelines for

the use of electronic records and signatures for pharmaceu-

tical and medical devices [25]. To meet Requirement C1-3,

therefore, the proposed artifact will record manually as well

as (semi-)automatically executed process tasks.

In order to enable tailoring of cyber-physical processes

to other contextual factors, like the execution state of a

pharmaceutical machine or the current location of a technician,

we introduce the concept of context graphs. The latter allows

modeling data entities, which are semantically related (cf.

Requirement C2-1). The location of a technician, for example,

has to be mapped to the current process execution context.

Furthermore, sensor information, such as the location of a

technician and the physical structure of a machine, must be

integrated into a contextual model and mapped to modeled

contextual factors (cf. Requirement C2-2). Sensed data then

has to be evaluated in order to decide whether a given contex-

tual situation of the CaPI concept is present to be able to tailor

process execution of the considered cyber-physical process (cf.

Requirement C2-3). Note that in current practice, task-related

information for service operators is provided conventionally,

e.g., procedural instructions for machine maintenance can be

found in handbooks, whereas information about the current

state of a machine is provided on its terminal (cf. Fig. 1-

4). Finally, the sub-tasks to be performed in the context of

a particular maintenance task are summarized in paper-based

checklists.

To optimize the interaction between users and CPS (see Cat-

egory C3) and to bridge the gap between physical system and

its digital representation, the proposed approach shall enhance

the tasks of cyber-physical processes with AR-enabled user

interfaces (cf. Requirement C3-1), i.e., the various kinds of

digital information shall be presented to users in a AR view.

The latter requires precise leading of a users attention (cf.

Requirement C3-2). Finally, mapping of physical objects, i.e.

a machine part, and its virtual counterpart in the context graph

constitutes a crucial task for enabling precise interaction and

attention guidance (cf. Requirement C3-3).

V. CONCEPTS

The support of cyber-physical processes involves several

steps. Sensor data needs to be collected, processed and mapped

to a context model (cf. Fig. 3a). The latter, in turn, is integrated

into a context graph (cf. Fig. 3b), and a context-aware process

environment (CaPI, cf. Fig. 3c). Finally, AR techniques are

added to optimize the interaction between user and CPS,

e.g., by enabling gesture-based controls (cf. Fig. 3d). In the

following, we present concepts for context models, CaPI, and

augmented reality-based interaction between users and CPS.

Augmented Reality

Platform

AR Application

Context Graph

DeviceTree

Context-aware Process Environment (CaPI)

Pharma Packaging Machine

Position

Sensor

Machine Structure Exporter

Interaction

Gesture

Sensor

Fig. 3. Component Architecture

A. Context-aware Process Injection

To systematically support context-aware process variants as

well as context-driven process adaptations at run time, we

adopt the concept of context-aware process injection (CaPI)

[15]. The key objective of CaPI is to ease the modeling of

a collection of process variants at design time as well as to

automate context-driven process adaptions at run time. Taking

the current context into account, CaPI enables the late injection

(i.e., insertion) of process fragments into a base process. The

core artifact of CaPI is the context-aware process family (CPF)

(cf. Fig. 4). More precisely, a CPF comprises a base process

model (cf. Fig. 4g) with extension areas (cf. Fig. 4h), con-

textual situations (cf. Fig. 4e) characterized through process

parameters (cf. Fig. 4d), a set of process fragments (cf. Fig. 4i)

that may be injected at specific extension areas during run

time, and a set of injection specifications (cf. Fig. 4f). The

latter define process fragments to be injected at certain areas

when a given contextual situation applies.

Following the separation of concerns design principle, the

base process model solely contains decisions and activities

shared by all variants of the process. In particular, these

activities need to be known at design time, and must not

be changed during run time. By contrast, extension areas

represent the dynamic parts of the process. Accordingly,

process modelers may first model the predictable parts of the

process and then add the varying parts to the base process step

by step. Extension areas can then be used to automatically

inject process fragments into the base process during run

time, based on the respective contextual situation as well as

injection specifications. Moreover, an extension area allows

for the dynamic injection of any number of process fragments

arranged in parallel branches. In turn, contextual situations are

defined through conditions expressed in first-order logic. These

conditions can take process parameters as well as data objects

of the base process model into account. In this context, process

parameters may be linked to external factors, i.e., context

properties of a context graph (cf. Fig. 4a+b), influencing

the decisions on process injections. When injecting process

fragments, CaPI takes care of correct data flow mappings as

well, i.e., data objects of an injected process fragment (cf.

Fig. 4i) are automatically connected to existing data objects

of the base process. CaPI enables dynamic configurations and

changes of varying processes in a controlled way during run

time. By solely enabling insertions of process fragments, it

further allows process modelers to focus on the commonalities

of all variants and varying process parts instead of coping with

a complex process model covering all variants. Furthermore,

process modelers may directly integrate contextual factors into

the modeling of process variants. Thereby, external context

factors are abstracted by meaningful process parameters and

reusable contextual situations. CaPI is able to cope with

context-driven run-time changes based on the late evaluation

of contextual parameters at given extension areas. Finally, the

automated construction of a consistent data flow between the

injected process fragments and the base process mitigates the

efforts of involved users.

B. Context Modeling

The context model for the design of the proposed artifact

has been adopted from [24]. In detail, as semantic data model

we generate a context graph that comprises context entities

Context Graph

Process Fragment:

CalibrateFormingStation

Base Process

If …

Context Mapping rollMntError = BOOLEAN

Process Parameters

Mapping Rules

IF (formingStation.stopReason ==

„FormFoilEnd“)THEN rollMntError(true)

Injection Specification

If Contextual Situation FormFoilEnd is present at Extension Area 1:

Inject Process Fragment ReplaceRollMounting inline sequential

Contextual Situation FormFoilEnd

Condition: rollMntError == true

Contextual Situation RollMountingReplaced

Condition: rollMntError == true && Contextual Situation FormFoilEnd occured

Injection Specification

If Contextual Situation RollMoutingReplaced is present at Extension Area 2:

Inject Process Fragment CleanMountingRoll inline sequential

If …

User

Position

Machine

StopReason Context Factor

UserPosition

Context Factor

StopReason

Data Extraction

Sensors Position

Sensor

Station

FormSt

Process Fragment:

AdjustFormFoilRoll

Process Fragment:

ReplaceRollMounting

Process

Fragment:

CleanMountingRoll

Process Fragment:

DeactivateHeatingStation

A5-3:

Calibrate

Deflection Polley

A5-4:

Deactivate

Heating Station

A5-2:

Calibrate Forming

Station

A5-6:

Replace Roll

Mounting

A5-8:

Adjust Form Foil

Roll

A5-7:

Get Settings

Sheet

A5-1:

Remove Covers

Process Fragment:

CalibratePrintDevice

A5-5:

Calibrate Printer

Device

A7-1:

Clean Roll

Mounting

Settings Sheet

A2:

Check Equipment

A1: Receive

Maintenance Plan

and Settings

Sheet

A3:

Stop and Release

Machine

A4:

Conduct Cleaning ExtensionArea1:

Conduct

Maintenance

A6:

Execute Test Run ExtensionArea2:

Conduct

Cleaning

A8:

Document

Machine Changes

AR &

Data Extraction

Context MappingContext-aware Process Execution

Settings Sheet

Fig. 4. Example of a Context-aware Process Family

and their properties as node types. Relations between context

entities and context properties are represented as relationship

edges. Every node, in turn, is associated with one of the fol-

lowing context types: location, time, resource, or organization,

e.g., physical objects may be modeled as entities with context

type resource (cf. Fig. 5a). The current state of a context entity

is expressed by context properties. For example, the power

consumption of a forming station in a press line is a property

of a corresponding context entity (cf. Fig. 5b), i.e., the machine

or the device itself depending on how precisely the context

model is specified. A relation between context entities is

modeled through an entity relation, i.e., an edge in the context

graph that may be associated with attributes characterizing

the relation (cf. Fig. 5c). Furthermore, temporally occurring

relations between context properties can be modeled by virtual

relations, e.g., if the current actor in the application context is

the maintenance operator of a machine represented as context

entity (cf. Fig. 5d).

Regarding the considered application scenario, the structure

of the packaging press line is mapped onto context entities that

can be automatically derived from the data model of the press

line–the latter describes the CPS structure as a Device Tree

Structure. Following the context model creation, the sensor

data acquired with the Machine Structure Exporter (cf. Fig. 3a)

is mapped onto context properties. When a context property

changes, the context graph is adapted accordingly by updating

the value of a context property or changing the relation of a

context entity. The state of one or more elements of the context

graph (i.e., context entities or properties) is described by a

contextual situation (cf. Fig. 5e). The latter corresponds to a

set of predicates, i.e., context mapping rules (cf. Fig. 5f). Note,

that mappings between context nodes and contextual situations

are illustrated as situational relations (cf. Fig. 5g).

C. Supporting Interactions Between Users and CPS

The sense of users for the CPS as well as for the manual

tasks of cyber-physical processes can be enhanced by inte-

grating the latter with augmented reality (AR) technology,

i.e., the goal is to provide process-centric user guidance in

accomplishing complex tasks related to physical components

of the CPS.

In general, AR enriches the user’s view of the physical

environment with virtual (i.e. digital) objects overlaid and inte-

grated into this view, and thus it virtually enhances the user’s

perception of the CPS. Contemporary AR usually uses live

video images being digitally enriched with computer-generated

graphics. Two fundamental steps have to be accomplished

in this context [26]: First, the current state of the physical

CaPI

Context Graph

…

Printer Device

Application Context

Context Property

Context Entity

Location Context

Time Context

Resource Context

Organization Context

Entity Relation

Virtual Relation

Situational Relation

Machine Actor AR Device

Machine Location

Machine

RunTimeState

Actor Location AR Device Location

Maintenance

Operator

Forming Station

RunTimeState

MaintenanceState

PowerConsumption

Contextual

Situation

FS Maintenance

Required ReplaceRollMounting

if(formingStation.stopReason

== „FormFoilEnd“)

Context Mapping Rule

if(formingStation.maintenanceState==maintReq

|| (formingStation.runTimeState==faultStop

&& formingStation.powerConsumption > 500))

Maintenance

Running

if(actor.actorLocation==machine.machineLocation

&& machine.runTimeState==maintStop)

Operator

Punching Station

…

AdjustFormFoilRoll

if(formingStation.error

== „FoilDissolvesOnEdges“

Actor

Availabilty

Fig. 5. Context Graph Model

world and, based on this, the state to which the virtual world

(i.e., the virtual enhancements) needs to be transferred must

be determined. Second, the virtual enhancements have to be

displayed in a way, together with the physical environment,

such that users can sense it as part of this environment.

Basically, three system components are needed [26], i.e.,

sensors to collect information about the current state of the

physical world, processors to process the sensor data and to

determine the state of the virtual world, and displays to allow

users to immerse into the combined physical and virtual world.

The most common AR displays are visual displays [27], which

can be classified into optical see-through (OST) and video

see-through (VST) displays. The former allow the wearer to

perceive the physical world with their eyes with the virtual

enhancements being overlaid via a holographic optical element

or semi-silvered mirrors, whereas the latter do not directly let

provide an enhanced camera video stream to them.

To track the camera position and create an environmental

model of the CPS, easily detectable fiducial markers can be

used, which are placed in the physical environment. With

the help of computer vision techniques, these markers can

be detected based on their visual features (i.e. marker-based

tracking) [28]. On the other hand, marker-less tracking, as

for example used by Microsoft’s HoloLens1, automatically

maps the surrounding of the user (i.e. the service operator

in our case) by extracting visual features and by generating

an environmental 3D model. Additionally, markers can be

mapped to the CPS to be able to automatically identify physi-

cal parts during the execution of the cyber-physical processes.

Further, the user’s current location–in relation to the CPS–

has to be sensed based on visual marker-based detection.

The AR application reads the marker identifier and maps

1https://www.microsoft.com/en-us/hololens/hardware

the user’s current location to the one of the CPS in the

context graph. Detected parts are then integrated into the

context graph as context properties and contextual situations

can be triggered (e.g., to start new process instances or change

currently running ones).

Finally, a graphical user interface presented on a AR OST

display, in conjunction with gesture recognition, allows exe-

cuting process instances by invoking respective methods of

the process execution interface. In doing so, a user’s attention

of shown virtual objects has to be guided. For example, a

animated arrow can indicate the current point of interest of a

CPS (cf. Fig. 6).

VI. SUMMARY & OUTLOOK

This paper presented fundamental requirements for an ar-

tifact, which shall enable a context-aware guidance of cyber-

physical processes. The artifact allows creating a context graph

to manage and structure sensor data of a CPS. Further, it

enables the context-driven injection of process fragments into a

given base process (i.e., a basic template of the cyber-physical

process) during run time. Furthermore, the interaction of users

with the cyber-physical processes is supported through an

AR application, which integrates position and marker sensors

into the context graph. In particular, detected objects can be

enhanced with digital data, enabling an improved guidance

during process execution. In addition, new process instances

may be started depending on the actual contextual situation,

e.g., when a user picks a machine part with a specific marker.

Finally, the tasks to be executed may be selected based on the

current context of the user. Taking the maintenance process

of a press line as representative of a cyber-physical process,

we discussed the challenges to be tackled. Further, we showed

how the information about the physical and logical structure

of a press line can be integrated into a context graph.

Fig. 6. Augmented Reality User Interface

In summary, linking AR with CPS offers promising per-

spectives for a process-centric guidance of CPS users. Sensed

information about the CPS and the user, as well as the environ-

ment, allow for a tailored support of cyber-physical processes,

which can be adjusted to the respective contextual situations.

This way, manual tasks can be guided and monitored, enabling

a much better traceability. In future work we will consider

additional CPS application scenarios as well as the challenges

and requirements imposed by them. Moreover, we will con-

sider collaborative cyber-physical processes, involving multi-

ple users and resources, as well as AR enhanced immersive

analytics scenarios relevant for condition monitoring in CPS.

REFERENCES

[1] H. Lasi, P. Fettke, H.-G. Kemper, T. Feld, and M. Hoffmann, “Industry

4.0,” Business & Information Systems Engineering, vol. 6, no. 4, pp.

239–242, 2014.

[2] Janiesch et al., “The Internet-of-Things Meets Business Process Man-

agement: Mutual Benefits and Challenges,” Computing Research Repos-

itory, no. 709.03628, pp. 1–9, 2017.

[3] A. Hallerbach et al., “Capturing Variability in Business Process Mod-

els: The Provop Approach,” J Software Maintenance and Evolution:

Research and Practice, vol. 22, no. 6-7, pp. 519–546, November 2010.

[4] D. Gorecky, M. Schmitt, M. Loskyll, and D. Z¨

uhlke, “Human-Machine-

Interaction in the Industry 4.0 Era,” in 2014 12th IEEE International

Conference on Industrial Informatics (INDIN), 2014, pp. 289–294.

[5] S. J. Henderson and S. Feiner, “Evaluating the Benefits of Augmented

Reality for Task Localization in Maintenance of an Armored Personnel

Carrier Turret,” in Proc 8th Int Symp on Mixed and Augmented Reality,

2009, pp. 135–144.

[6] A. L. Janin et al., “Calibration of Head-Mounted Displays for Aug-

mented Reality Applications,” in Int Symp Virtual Reality, 1993, pp.

246–255.

[7] S. Feiner et al., “Knowledge-based Augmented Reality,” Communica-

tions of the ACM, vol. 36, no. 7, pp. 53–62, 1993.

[8] S. Wiedenmaier et al., “Augmented Reality for Assembly Processes.

Design and Experimental Evaluation,” HCI, vol. 16, no. 3, pp. 497–

514, 2003.

[9] M. Reichert and B. Weber, Enabling Flexibility in Process-Aware

Information Systems: Challenges, Methods, Technologies. Springer,

2012.

[10] B. Weber, M. Reichert, W. Wild, and S. Rinderle-Ma, “Providing

integrated life cycle support in process-aware information systems,”

International Journal of Cooperative Information Systems, vol. 18, no. 1,

pp. 115–165, March 2009.

[11] M. Reichert and P. Dadam, “Enabling adaptive process-aware informa-

tion systems with adept2,” in Handbook of Research on Business Process

Modeling, J. Cardoso and W. van der Aalst, Eds. Hershey, New York:

Information Science Reference, March 2009, pp. 173–203.

[12] S. Steinau, K. Andrews, and M. Reichert, “The relational process

structure,” in 30th Int’l Conference on Advanced Information Systems

Engineering (CAiSE 2018), ser. LNCS. Springer, 2018, pp. 53–67.

[13] K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A

design science research methodology for information systems research,”

Journal of management information systems, vol. 24, no. 3, pp. 45–77,

2007.

[14] W. Friedrich, D. Jahn, and L. Schmidt, “ARVIKA-Augmented Reality

for Development, Production and Service,” in International Symposium

on Mixed and Augmented Reality (ISMAR), 2002, pp. 3–4.

[15] N. Mundbrod et al., “Context-Aware Process Injection: Enhancing

Process Flexibility by Late Extension of Process Instances,” in Proc

CoopIS 2015, ser. LNCS, no. 9415. Springer, 2015, pp. 127–145.

[16] F. Gottschalk, W. M. P. Van der Aalst, M. H. Jansen-Vullers, and

M. La Rosa, “Configurable workflow models,” International Journal of

Cooperative Information Systems, vol. 17, no. 02, pp. 177–221, 2008.

[17] A. Hallerbach, T. Bauer, and M. Reichert, “Context-based configuration

of process variants,” in 3rd International Workshop on Technologies for

Context-Aware Business Process Management (TCoB 2008), June 2008,

pp. 31–40.

[18] A. Murguzur, X. De Carlos, S. Trujillo, and G. Sagardui, “Context-

Aware Staged Configuration of Process Variants@Runtime,” in Ad-

vanced Information Systems Engineering. Springer, 2014, pp. 241–255.

[19] R. Aggarwal, K. Verma, J. Miller, and W. Milnor, “Constraint driven web

service composition in meteor-s,” in IEEE International Conference on

Services Computing (SCC 2004), 2004, pp. 23–30.

[20] O. Saidani and S. Nurcan, “Towards context aware business process

modelling,” in Workshop on Business Process Modelling, Development,

and Support, 2007, p. 1.

[21] M. Rosemann, J. Recker, and C. Flender, “Contextualisation of business

processes,” International Journal of Business Process Integration and

Management, vol. 3, no. 1, pp. 47–60, 2008.

[22] C. Bettini, O. Brdiczka, K. Henricksen, J. Indulska, D. Nicklas, A. Ran-

ganathan, and D. Riboni, “A survey of context modelling and reasoning

techniques,” Pervasive and Mobile Computing, vol. 6, no. 2, pp. 161 –

180, 2010.

[23] M. G. Cimino, B. Lazzerini, F. Marcelloni, and A. Ciaramella, “An

adaptive rule-based approach for managing situation-awareness,” Expert

Systems with Applications, vol. 39, no. 12, pp. 10 796 – 10 811, 2012.

[24] da Cunha Mattos et al., “A Formal Representation for Context-aware

Business Processes,” J Comp Ind, vol. 65, no. 8, pp. 1193–1214, 2014.

[25] US Food and Drug Administration et al., “Title 21 Code of Federal Reg-

ulations (21 CFR part 11): Electronic Records, Electronic Signatures,”

URL: www.fda.gov/ora/compliance ref/part11, 2008.

[26] A. B. Craig, Understanding Augmented Reality: Concepts and Applica-

tions. Elsevier, 2013.

[27] M. Billinghurst, A. Clark, G. Lee et al., “A Survey of Augmented

Reality,” Foundations and Trends of Human–Computer Interaction,

vol. 8, no. 2-3, pp. 73–272, 2015.

[28] A. Yilmaz et al., “Object Tracking: A Survey,” ACM Computing Surveys

(CSUR), vol. 38, no. 4, p. 13, 2006.