Improving argumentation visualization of multi-stakeholder development processes

Improving Argumentation Visualization of Multi-Stakeholder

Development Processes – A Prototyping Case

Universitätsverlag der TU Berlin

Research papers in information systems management Band 18

Mathias Riechert

Improving Argumentation Visualization of Multi-Stakeholder

Development Processes – A Prototyping Case

Die Schriftenreihe Research papers in information systems management

wird herausgegeben von:

Prof. Dr. Rüdiger Zarnekow

Research papers in information systems management

| 18

Improving Argumentation Visualization of Multi-

Stakeholder Development Processes – A Prototyping Case

Mathias Riechert

Universitätsverlag der TU Berlin

Bibliografische Information der Deutschen Nationalbibliothek

Die Deutsche Nationalbibliothek verzeichnet diese Publikation in der

Deutschen Nationalbibliografie; detaillierte bibliografische Daten sind

im Internet über http://dnb.dnb.de abrufbar.

Universitätsverlag der TU Berlin, 2018

http://verlag.tu-berlin.de

Fasanenstr. 88, 10623 Berlin

Tel.: +49 (0)30 314 76131 / Fax: -76133

E-Mail: publi[email protected]-berlin.de

Diese Veröffentlichung – ausgenommen Umschlagfoto –

ist unter der CC-Lizenz CC BY lizenziert.

Lizenzvertrag: Creative Commons Namensnennung 4.0

http://creativecommons.org/licenses/by/4.0/

Umschlagfoto:

Tibor Janosi Mozes | CC0 Creative Commons

Satz/Layout: Mathias Riechert

ISBN 978-3-7983-2994-2 (online)

ISSN 2191-639X (online)

Online veröffentlicht auf dem institutionellen Repositorium

der Technischen Universität Berlin:

DOI 10.14279/depositonce-6743

http://dx.doi.org/10.14279/depositonce-6743


 
Introduction ..........................................................................................................................................8 
Case Setting .........................................................................................................................................9 
Related Work .....................................................................................................................................10 
1 Design Rationale .........................................................................................................................10 
2 Information Visualization ...........................................................................................................11 
Method ...............................................................................................................................................11 
Prototype Development .....................................................................................................................12 
1 Requirements ...............................................................................................................................12 
2 Prototype Development ...............................................................................................................13 
2.1 Information Collection, Data Modelling and Transformation ............................................... 13 
2.2 Visual Mapping......................................................................................................................................... 13 
2.3 View Creation ........................................................................................................................................... 14 
Prototype Evaluation ..........................................................................................................................20 
1 Evaluation Method ......................................................................................................................20 
2 Evaluation Results .......................................................................................................................21 
3 Discussion ...................................................................................................................................23 
3.1 R1: include functionality to reduce the visual complexity of the argumentation 
visualization ............................................................................................................................................................... 23 
3.2 R2: higher accessibility of the argumentation visualization than Compendium Web 
Maps  24 
3.3 R3: improve information presentation and coding compared to Compendium Web 
Maps  25 
3.4 R4: provide the same depth of information as in Compendium Web Maps .................... 27 
4 Limitations ..................................................................................................................................28 
Conclusion and Outlook ....................................................................................................................28 
Literature ............................................................................................................................................28 
 
  

Improving Argumentation Visualization of Multi-

Stakeholder Development Processes -

A Prototyping Case

A shared understanding of development argumentation is crucial for a wide range of development

processes (such as requirements engineering, change management, eGovernment and

eParticipation, public policy) and central to prevent the failure of IT and development projects.

Computer-Supported Argumentation Visualization (CSAV) has been used to model and represent

discourse information for about 35 years. Although modelling tools have significantly matured and

continue to evolve, the visual representation of existing tools does not scale ideally with increasing

model complexity. For large-scale argumentation models, existing visualization approaches from

argumentation visualization are reported as being too complex for target stakeholders. This

prevents them from gaining insights into the development process and may ultimately contribute

to the rejection of the development result, causing severe costs for both public and private

organizations. In this paper, we employ the ‘design science’ methodology to incrementally develop

two interactive visual representations for argumentation visualization, incorporating best

practices from information visualization research. The prototypes are implemented and evaluated

in the setting of the project “Research Core Dataset”, a nation-wide project involving all major

stakeholder groups of the German science system in order to develop harmonized definitions for

research information. In our evaluation, both of the visual representations developed are

perceived as being much better at providing insights into complex development processes with a

high number of stakeholders.

Computer-Supported Argumentation Visualization, CSAV, argumentation visualization, information

visualization, design science

1 Introduction

Complex development processes involving many stakeholders are often costly and inherently

challenging. Depending on the study and development field, 9–22 % of development projects are

failures and another 26–31 % are challenged (i.e. they do not meet time and financial constraints)

(El Emam & Koru, 2008; Eveleens & Verhoef, 2010; Glass, 2006; Jørgensen & Moløkken-Østvold,

2006), causing severe costs for public and private organizations. The vast majority of existing

studies indicate that large projects with more complex requirements and multiple stakeholders

have considerably higher failure rates of 75 % or more (Heeks, 2003; Jones, 2000; Rubinstein,

2007; Sauer & Cuthbertson, 2003; Standish, 2001). Major reasons for project failure in large-scale

development processes are a lack of involvement and acceptance among users and stakeholders

(Al Neimat, 2005; Cerpa & Verner, 2009; Conklin, 2006; Rittel & Webber, 1973), delivery decisions

being made without adequate requirements information (Cerpa & Verner, 2009) and the project

scope and objectives changing while the development is being implemented (Al Neimat, 2005;

Cerpa & Verner, 2009).

We believe that higher rates of involvement, agreement and acceptance can be achieved if

stakeholders and other concerned parties are provided with more detailed and accessible insights

into the reasoning behind the decision and development process. For example, it can be important

for users to know the reasoning behind the design decisions concerning the amount of private

data to be collected and processed in an HR system, in social media platforms, or resident

registration systems.

It has been argued that argument-centred mapping may provide a useful tool for offering such

insights (Schoder, Putzke, Metaxas, Gloor, & Fischbach, 2014). A widely used and intensively

researched approach for this task is Design Rationale (DR). It aims to present “the design

alternatives which were considered, the arguments for and against these alternatives and the

reason why final design decisions were made” (Monk, Sommerville, Pendaries, & Durin, 1995). DR

has evolved from Kunz and Rittel’s proposed Issue-Based Information System (IBIS) (Kunz &

Rittel, 1970) notation as a way to structure and document highly complex decision processes,

which is still the most widely used notation for argumentation maps (Scheuer, Loll, Pinkwart, &

McLaren, 2010). IBIS helps to elicit and clarify the arguments in an intuitive, flexible, and fast way

during a debate by documenting them in a semi-formal representation (Shum et al., 2006).

Although modelling tools in the area of DR have significantly matured and continue to evolve, the

visual representations provided by existing tools (for example Compendium NG Web Map) do not

scale well with increasing model complexity. In real-world large-scale argumentation models,

existing visualization approaches from argumentation visualization are reported as being too

complex to be of help for target stakeholders. This prevents target stakeholders from gaining

insights into the development process and may ultimately contribute to the rejection of the

development result, causing severe costs for both public and private organizations.

In this paper, we employ the ‘design science’ methodology to incrementally develop two

interactive visual representations for argumentation visualization, incorporating best practices

from information visualization research to reduce the visual complexity.

Following the design science paradigm, we address a real-world problem to generate findings for

comparable use cases. As a case study we employ the large-scale development project

“Specification of a Research Core Dataset for the German Science System”. In this project, a

national standard for information about research activities was developed in an incremental

multi-stage process involving representatives of all major stakeholder groups in the German

science system. An incremental development approach was employed in order to maximize the

applicability of the prototypes. The prototypes were presented to the stakeholders in the project.

The article is structured as follows: Section 2 provides a brief overview of the case setting and the

development context. Section 3 discusses related work in the area of DR as well as information

visualization. The methodology of the paper is outlined in Section 4. Section 5 describes the

prototypes’ development. In Section 6 the evaluation is described. Finally, Section 7 provides an

outlook about consequences for development processes and identifies potential for further

research in the fields of DR and information visualization.

2 Case Setting

The use case we analyse is the finished standard development project “Research Core Dataset”

(RCD). The project was scheduled for 24 months (October 2013 – October 2015) and was initiated

by the German Council of Science and Humanities with the aim of developing a shared set of

definitions for research information about staff, publications, third-party funding, patents, young

researchers and research awards for the German science system. More than 48 different

stakeholder groups were directly involved in the process. They included representatives of

universities, non-university research institutions, ministries, research information system vendors

and scientific societies. The specification process was organized into four groups with eight

experts each. Each group held up to six meetings lasting 1–2 days, with 8 hours discussion time

per meeting. The project group “definitions and data formats” defined research information for all

of the areas stated above. To combine internal expertise with real-world evaluation of the

proposed definitions, the procedure combined development workshops and a feedback phase

with representatives of pilot organizations, non-university research institutions, funding

organizations and research information system vendors. Another discussion phase was conducted

after the feedback phase in order to integrate the range of external feedback (more than 1800

feedback messages were incorporated into the project) into the definition specification.

3 Related Work

3.1 Design Rationale

Design Rationale (DR) aims to document “the design alternatives which were considered, the

arguments for and against these alternatives and the reason why final design decisions were

made” (Monk et al., 1995). DR has evolved from Kunz and Rittel’s proposed Issue-Based

Information System (IBIS) (Kunz & Rittel, 1970) notation as a way to address “wicked problems”.

Based on Rittel and Webber (1973), wicked problems are defined as complex design problems “for

which no single computational formulation of the problem is sufficient, for which different

stakeholders do not even agree on what the problem really is, and for which there are no right or

wrong answers, only answers that are better or worse from different points of view” (Introne,

Laubacher, Olson, & Malone, 2013, p. 45). Rittel identified ten criteria defining the nature of a

wicked problem. We argue that these criteria are met in the case of increasingly complex

development projects involving high numbers of stakeholders. We have previously analysed the

applicability of Rittel’s criteria of a wicked problem for the specification process (Riechert,

Biesenbender, Dees, & Sirtes, 2016). A formal content analysis provided a deep insight into the

dimensions of complexity. The results of the content analysis underline our interpretation of the

development process as a wicked problem according to the criteria stated. As IBIS was implicitly

designed to address wicked problems, its application to our case is presumed to offer the best fit.

Despite its long-ranging history, IBIS is still the most widely used notation for argumentation

maps (Scheuer et al., 2010). Using it helps to elicit and clarify the arguments in an intuitive,

flexible and fast way during a debate by documenting them in a semi-formal representation (Shum

et al., 2006). IBIS was later extended to graphical IBIS (gIBIS) (Conklin & Begeman, 1988), which

has found widespread adoption and application in current tools like Compendium (Selvin et al.,

2001). Newer versions of Compendium have since been published, as have numerous studies on

the influence of argumentation visualization on the working atmosphere in discussions (we refer

to Schneider et al. (2013), Scheuer et al. (2010), and Suthers (2008) for an overview).

Computer-Supported Argumentation Visualization (CSAV) widens the scientific debate on

argumentation modelling and discussion moderation by also addressing questions of how to

present the argumentation. CSAV on the basis of IBIS has been employed in the areas of

eGovernment and eParticpation with the goal of presenting argumentation information to

enhance participation (Loukis & Wimmer, 2012; Loukis, Wimmer, Charalabidis, Triantafillou, &

Gatautis, 2007; Renton, 2006). In eParticipation research, IBIS has been used both without and

with minor adoptions. The resulting maps are therefore of high complexity when it comes to real-

world argumentation structures. Although user studies found that argumentation maps have

advantages over textual representation (Loukis & Wimmer, 2012; Renton, 2006), our pre-tests

with more complex argumentation maps showed that high diagram complexity causes serious

issues for diagram presentation, comprehensibility and usability, which increases the access

barrier and may lead to total rejection of the development results. In order to improve the visual

representation of complex argumentation visualizations, we use best practices from information

visualization.

3.2 Information Visualization

Interestingly, the academic discourse on argumentation visualization has hitherto been largely

independent of the discussion in the literature on information visualization.

The term ‘Information Visualization’ was introduced by Robertson, Card and Mackinlay (1989)

and refined by Card et al. in 1999: “Information visualization is the use of computer-supported

interactive visual representations of abstract data to amplify cognition” (Card, Mackinlay, &

Shneiderman, 1999). In information visualization research, reducing the complexity of visual

information to allow for better information perception and to amplify cognition has been a central

topic since the field emerged. Based on research from psychology and perception studies

(Treisman, 1985; Ware, 2004; Wertheimer, 1922), several approaches have been discussed for the

development of information visualization. The most prominent approaches are the ‘visualization

mantra’ (Shneiderman, 1996; Shneiderman & Plaisant, 2006) and the ‘reference model for

visualization’ (Card et al., 1999), which was later extended to the ‘reference model for developing

visual representations’ (Mazza, 2009). These two approaches aim at different development levels:

While the ‘visualization mantra’ provides general guidelines and functionality for reducing

visualization complexity, the reference model aims at providing a detailed process for developing

visualizations from raw data.

To the best of our knowledge, the application of information mapping to the elements of

argumentation visualization has not yet been discussed by researchers. To reduce the high

diagram complexity of argumentation visualizations so as to provide better insights and enhance

transparency, we exploratively employ the reference model for developing visual representation

(Card et al., 1999, p. 18; Mazza, 2009), along with interactive visualization guidelines from

Shneiderman (1996).

4 Method

We employ the Design Science paradigm, which is rooted in engineering and the sciences of the

artificial (Simon, 1969) and develops knowledge about a problem domain and its solution in the

building and application of the designed artefact (A. R. Hevner, Salvatore, Park, & Ram, 2004). It

furthermore allows for the development of a rational reasoning of characteristics and functionality

of the developed artefact (Pries-Heje & Baskerville, 2006). We apply the synthesized Design

Science paradigm (Gregor & Hevner, 2013; Peffers, Tuunanen, Rothenberger, & Chatterjee, 2007)

to improve existing argumentation visualizations and develop knowledge about the applicability

of existing visualization principles for complex visualizations from information visualization.

To evaluate the prototypes, we conducted a qualitative and a quantitative evaluation phase: First,

we used the criteria found in the expert interviews during the development phase as the structure

of a semi-formal qualitative evaluation with ten interviewees, who had not seen the prototypes in

advance. Each interviewee was asked the same questions systematically and could provide

multiple arguments about why the respective criterion was more advantageous in one of the

representations. As the focus is to analyse the criteria in detail, we refer to this in the following as

the content-oriented evaluation. Details of the sampling are provided in the evaluation section

(Section 6). Secondly, we asked the interviewees which one of the representations they preferred.

In this evaluation, only one preference is counted per interviewee, which is why we call this

evaluation the preference-oriented evaluation. The qualitative evaluation results are then

compared to a quantitative evaluation phase, where the same questions were put to 105 new

interviewees after seeing two of the visual representations. Details of the sampling and analysis

method are provided in the evaluation section (Section 6).

5 Prototype Development

5.1 Requirements

We extracted the requirements from two sources: Firstly, we conducted 6 expert workshops with

8 experts each to model the rationale of the project’s results. In each workshop, all stated

alternatives and arguments were documented in Compendium NG using the IBIS notation,

resulting in an argumentation model of about 1200 interconnected nodes. Secondly, the

argumentation maps were made available to cooperation stakeholders. In total, 13 representatives

of cooperation stakeholders were included for feedback on the visualization in that phase.

Existing tools for argumentation visualization offer different forms of visual representation of

argumentations. In Compendium NG, the graphical model can be exported to bitmap or

Compendium Web Maps (an interactive browser map with each sub-map and node opening a new

browser window).

The modelling process involving increasing complexity of the argumentation model revealed that

Compendium’s argumentation maps scale poorly when the issue and argumentation structure

grows larger than about 60 nodes per sub-map. One possible complexity reduction strategy is to

introduce new levels of sub-maps so as to keep visual complexity constantly low when viewing.

With the argumentation complexity reached in our case (>1200 nodes in total), this would result

in up to nine levels of sub-maps. This was reported to be confusing to the extent that our experts

refrained from using these maps. Alternatively, all levels can be modelled using one map, resulting

in a huge network map showing all nodes. Although we introduced, documented and explained

additional structuring rules, such maps were reported to be too complex to be grasped by the

experts involved. This conflicts with Conklin’s (2003) claim, which was later reinforced by Awati

(2011), that IBIS is intuitive and understandable without prior explanation and documentation. In

those studies however, the number of nodes (and, therefore, node complexity) was much lower

than in our real-world example. Feedback from our external stakeholders showed that persons not

included in the decision process were overwhelmed by the high complexity of the diagram and

preferred discussion protocols.

However, discussion protocols are very impractical to explore, as the entire contents need to be

shown in text form (more than 150 pages in our case) without the possibility for elaborate

information aggregation, navigation or filtering.

As visual representations are potentially stronger in this regard, but the existing Compendium

Web Maps are reported to have serious drawbacks, we set visualization quality and complexity

reduction as the main requirements for any further development of a visual representation.

Further comments were concerned with visual quality, as the Compendium Web Maps are of a

relatively low quality (blurry when printed, connecting edges show single pixels). Additionally, the

space required to present maps of this size results in high requirements for printing and

documentation.

To develop a visualization prototype, two additional requirements were extracted: Firstly, our

goal is to provide the full information depth that can be provided by the Compendium’s Web

Map or the protocol form. Secondly, we set the goal of presenting the visualization on a platform

available online to minimize the access barrier for the stakeholders.

Consequently, the following requirements were extracted from the case:

R1: include functionality to reduce the visual complexity of the argumentation visualization

R2: higher accessibility of the argumentation visualization than Compendium Web Maps

R3: improve information presentation and coding compared to Compendium Web Maps

R4: provide the same depth of information as in Compendium Web Maps

5.2 Prototype Development

The development description is structured based on the ‘reference model for developing visual

representations’ (Mazza, 2009). Each step in the process is described in a subsection.

5.2.1 Information Collection, Data Modelling and Transformation

We collected the discourse knowledge in the form of meeting protocols and a discourse

argumentation tree in Compendium NG (using the IBIS notation). Compendium NG was used

because it allows for fast and intuitive diagram editing, tagging and export of argumentative

models. The meeting documentation spans over 80 hours of discussion time across 7 meetings

plus external feedback. The resulting network diagram structure comprises more than collected

1200 nodes (including 614 arguments). Data modelling was performed both during the discussion

meetings and in the subsequent review of the protocol. An example of an argumentation model is

shown in Figure 1.

Figure 1: Excerpt from a Compendium Web Map (modelled in IBIS)

The tree starts with a question node. Each question can have one or more answers.

In our case (see Section 2), only three types of questions are required: (1) “What is a suitable

definition for …?”, (2) “How can … be differentiated?”, (3) “What are possible attributes for …?”. If

more than one alternative has been discussed, they are added as two nodes on the same level.

Each node can be supported or challenged by arguments (seen on the right side).

The argumentation model is exported to XML and further computed in a custom-developed

transformation program that transforms the argumentation tree structure into a JSON tree

structure for later use in the visualization prototype. The transformation is decoupled from the

visualization prototype in order to optimize access times of the online platform.

5.2.2 Visual Mapping

After the data is transformed into JSON, the data structures are applied to visual structures. The

argumentation model is rooted in a single element, which is refined in sub-questions. The

argumentation therefore forms a hierarchical tree. Consequently, we choose two tree visualization

approaches for our prototypes: a hierarchical tree node map (HTNM) and a packed circle map

(PCM).

Hierarchical tree node maps are one of the oldest forms of graphic representation of

A working demo can be found here: HTNM and PCM (texts in German).

hierarchies and show the connection between nodes by links on a map. The most common

example of this is the Windows Explorer interface. Packed circle maps were developed by Wang et

al. (2006) to provide a clear overview of complex hierarchical structures, and have proven

beneficial in initial evaluations (2006). We use the layout algorithms implementing this concept in

the D3 JavaScript library,

the successor of Protovis (Michael Bostock & Heer, 2009). This makes it

possible to render interactive online scalable vector graphics.

5.2.3 View Creation

The view is the central access and interaction point for the visualization users. We based our

implementation and further development on existing visualization libraries in D3, as it provides

SVG (scalable vector graphics) rendering, can read JSON data and is based on JavaScript.

Therefore, it is fully functional in the browser without the need for any installation. For the packed

circle maps, the ‘circle packing layout’ (M Bostock, 2012)

was implemented. For the hierarchical

trees (HTNM) we developed a new layouting, as the existing ‘collapsible tree layout’ uses too much

vertical space, does not work well with different text lengths and leaves no space for alternatives

and arguments.

In order to reduce the visual complexity without reducing the depth of information, the

implementation was further developed by implementing functionality from Shneiderman’s ‘Task

by paper type taxonomy’ – or the ‘visualization mantra’ (Shneiderman, 1996):

Figure 2: Overview for the hierarchical tree map (HTNM – left) and packed circle map (PCM – right) with two (a) and max (b)

initial visible levels

Overview: Gain an overview of the entire argumentation: Mapping all argumentation information is

a challenging task. Overall, 1189 nodes (575 answers, questions and notes and 614 arguments) in seven

content areas are documented. We developed a visual representation that provided an initial overview

showing only 1–3 hierarchy levels (configurable) initially (see section filter) but allowed for interactive

D3 stands for Data-Driven Documents. http://d3js.org/

A minimal working example can be found at http://bl.ocks.org/mbostock/4063530.

transition into the higher hierarchy levels (see section zoom). Figure 2 shows a sample HTNM on the

left and a PCM on the right. The first row shows the initial overview. In (a), only one level below the

root element is shown. (b) shows how all levels can be presented in parallel in the PCM. When clicking,

the visible elements are rendered as shown in the second row. In the HTNM, the sub-nodes of the

clicked element are shown. Note that all other nodes on the same level as the clicked node are moved, to

ensure that sub-nodes do not overlap. To keep the visualization structured, the element labels are

rendered next to the nodes on predefined hierarchy levels. In the PCM, the sub-nodes of the clicked

element are rendered inside the element. The D3 visualization library (Michael Bostock, Ogievetsky, &

Heer, 2011) was chosen for the implementation because it renders SVG (scalable vector graphics) maps

and allows for user interaction and direct data binding.

Figure 3: PCM: After clicking on a node, the viewport zooms to the node and its sub-nodes

Zoom: Zoom in on and pan to items of interest: In contrast to Compendium Web Maps, which are

only scalable from 100 % to 25 % and become pixelated when zooming in above 100 %, it is possible to

zoom in and out of both developed visualizations without any loss of quality for rendering or printing.

Zooming can be triggered by using the standard zoom interactions (mouse wheel and gestures) or by

clicking on an element on a higher detail level.

Furthermore, the zooming design pattern is also used for focusing on the relevant part of the map in

PCM. Figure 3 shows the zoom after clicking the ‘Pa0 Patente’ node.

Figure 4: HTNM: After clicking on a node, the viewport is panned to the node and its sub-nodes

In HTNM, pan is used to navigate to the clicked element and the sub-elements displayed (see

Figure 4). In order to keep the user’s focus, all elements are moved so that the newly focused

element is at the position of the previously focused element.

It is important to note that the use of D3 allows for fluent transitions of the visible part of the

diagram (viewport) in a browser. No website reload is needed and the user is able to perceive the

zoom and pan transitions through the map.

Filter: Filter out uninteresting items: In order to reduce the items visible at the same time, only the

highest hierarchy levels (1–3) are shown initially (see overview).

For PCM, a fundamental problem surfaced after implementing the visualization like in the example in

D3. As our diagram has more than 1200 nodes, rendering all nodes results in very slow in-browser

performance. We therefore developed a concept that reduces the number of nodes rendered in parallel

without restricting navigation into the hierarchy and back. Furthermore, we wanted the user to be able to

switch branches of the tree by clicking on a parallel branch. Therefore we implemented a recursive

visibility concept (shown in Figure 5), that shows elements of higher or the same hierarchy level with

high transparency to blend out non-focused nodes. Elements on a lower hierarchy level are shown only

up to the number of configured levels below (one level in the example), to allow for intuitive navigation.

Elements on the same level but in a different branch of the tree are only shown up to the level that the

tree branches off. By only rendering elements in the same branch, the performance can be kept high,

while all relevant information about the navigation path and sub-elements (full visibility) and its

junctions (transparent) are visible. As circles are packed on top of each other, the displayed text is

always shown on the visibility level (clicked level plus configured number of levels) for elements in the

branch of the clicked element, and on the circles on the last visible (transparent) level. Texts not in the

branch of the selected element are rendered with higher transparency. Note that the viewport in Figure 5

zooms and pans as described a little earlier.

Figure 5: PCM: Visibility concept with the tree structure above and the resulting circle visualization below

For HTNM, the selected element and branch are highlighted by fading non-selected elements to a higher

transparency.

A content filter is implemented for both visual representations. Based on the element tags (which are set

in Compendium), parts of the diagrams can be filtered. Figure 6 shows the filter menu on the right with

the changes when deselecting a content group. This filter uses tags given to the diagram elements in

Compendium. Using this approach, whole areas that have a preliminary status or relevance for specific

stakeholder groups can be hidden in a flexible way. In contrast to Compendium Web Maps, we do not

require a reloaded map or new positioning for the filtering, but the original maps can dynamically fade

to the filtered mode. Using this technique, all elements stay in their original position and the navigation

history is much clearer for the user.

Figure 6: Filtering in PCM (a) and HTNM (b)

Figure 7: Details menu with navigation, details, options, filter and legend sub-menus

Details-on-demand: Select an item or group and get details when needed: The details-on-demand

functionality is implemented with three concepts.

Firstly, the node focus can be changed by clicking on a node’s sub-node or a node which is visible in

parallel (see Figure 5). This is possible in Compendium Web Maps as well, but our implementation

allows for dynamic hierarchy level transitions without the need to manage multiple windows. This is of

particular importance if there are a high number of hierarchy levels. Both HTNM and PCM show

visually which hierarchy depth is being focused on.

Secondly, detailed information about the focused element is accessible with a context menu on the right

screen side (shown in Figure 7).

Because we use clicking for changing the focus, we expect that the user wants information about the

clicked element. If detailed information is desired, then only displaying the context information once a

user has clicked on the element results in the necessity to click on each element (similar to Compendium

Web Maps). Providing detailed information when hovering over a node significantly reduces the number

of clicks required to provide information, but conflicts with the click-and-focus logic above. We

therefore developed a two-step focus concept, to integrate both advantages (Figure 8). After clicking on

an element, the focus is set on that element (state (i)). In this state, only the detailed information of the

clicked element is displayed in the detail area, even if the user hovers over other elements. If another

element is clicked on from here, the focus is set to the next clicked element in state (i). If the same

element is clicked on again, the element is set to status (ii). Now, the hovering information is activated

and detailed information about other (not clicked) elements can be obtained much faster. Using this

approach, both clicking information and hovering information can be provided with minimal user

interaction.

Figure 8: Focus concept with two states to integrate click and hover detail information

Thirdly and most importantly for argumentation maps, we fade in alternatives and arguments when an

element has been clicked on in PCM. This information is already accessible in the detailed menu, but a

visual representation in the map is more present and allows for further interaction. Figure 9 shows the

implementation of the alternatives (represented by circles around the focused element) and arguments

(represented by circle segments around each element). After the second click on the focused element,

information about the alternatives and arguments is provided by the hover information, significantly

reducing the number of clicks required (see focus concept in Figure 8). Note that the number of

arguments determines the number of circle segments. For the element ‘Be38c’ (on the left), no

arguments were documented.

Figure 9: Alternatives (circles around the focused element) and arguments (circle segments around each element) in PCM

The implementation of alternatives and arguments in the HTNM is shown in Figure 10.

Figure 10: Alternatives and arguments in HTNM

Relate: View relationships among items: As organizing multiple levels in a window for each

Compendium sub-map confused our project group members rather than enhancing the overview, we

decided that all information should be organized in a single tree. This allows all hierarchy and cross-

hierarchy dependencies to be presented without the window barriers seen in Compendium Web Maps.

Additionally, it is much easier for the user to get her bearings.

6 Prototype Evaluation

To get a multi-perspective view of the prototype’s visual quality, we split the evaluation into three

parts: First, we qualitatively analyse the prototype’s perception among a group of experts in the

project. In this part, the number of statements for or against the visual representations regarding

the evaluation criteria is analysed. This means that one interviewee can provide multiple reasons

why a representation might be of benefit regarding the same criteria. This part of the evaluation is

referred to as content-oriented evaluation in the following sections. In the second part, we

compare the final preference for a representation of those experts with a quantitative evaluation

using an online questionnaire for professors affected by the contents presented. Here, only one of

the representations can be chosen by the interviewee. Therefore, the second part of the

evaluation is preference-oriented. As in the last part, the quantitative preference results are

compared with the evaluation of the Compendium Web Map.

6.1 Evaluation Method

Content-oriented evaluation of HTNM and PCM

The evaluation of the visual representations is conducted with semi-structured questions based on

the requirements identified in Section 5.1. We examined the intuitiveness, information

presentation and coding, and information depth of the visual representations.

The sampling of the ten interviewees covers experts participating in the policy development

process (4 interviewees) and non-participating concerned parties (6). Further criteria were the

type of expertise: practical expertise (7) and theoretical visualization expertise (3). Finally, the

employment level was considered: management (6) and operational (4). None of the interviewees

had seen the prototypes before the interviews. The coding procedure included two coders

working in parallel and independently from one another. After a short introduction and

explanation of the two representations (5 interviews starting with PCM, 5 starting with HTNM), all

attendees were asked to decide which of the visual representations (PCM or HTNM) they

preferred for all the sub-questions and to explain why they chose a particular representation. The

results were transcribed and coded with MaxQDA.

Preference-oriented evaluation of HTNM and PCM

For the second part of the evaluation, we analysed which representation was chosen as the final

preference of the interviewee. This used the same transcribed interviews from the last part. Again,

the answers were coded but this time only one preference for one visual representation was

allowed per interviewee and sub-question.

To test these results against a larger user group, we had to remove some questions which were

found to be hard to explain in a standardized quantitative interview without additional

explanations. Therefore, we used the questions which were found to be understandable without

additional explanation in a standardized questionnaire phase. For that part, a random sample of

4314 German professors were invited by email to evaluate the development representations PCM,

HTNM and Compendium WebMap. Out of them, 1917 saw the PCM and HTNM combination. In

total, 52 (2.7 %) completed the questionnaire after exploring both PCM and HTNM

representations. As the representation shows discussion contents that will change the way

researchers’ output is documented, they are indirectly affected by the discussion results. The

order in which the representations were shown was randomised. Both representations were

introduced by an interactive tutorial, explaining the goal and functionality of the representations.

The average viewing time of one representation was 4.57 minutes, while the qualitative expert

interviews lasted an average of 42 minutes.

Comparison HTNM, PCM and Compendium Web Map

Out of the 4314 invitations in the quantitative part of the evaluation, 316 professors (7.3 %)

completed the questionnaire. Each interviewee was shown two randomly selected visual

representations and asked to specify their preferred representation in terms of the quantitative

criteria for visual quality. Again, the order in which the representations were shown was

randomised.

6.2 Evaluation Results

Content-oriented evaluation of HTNM and PCM

Figure 11 shows the number of statements for and against the representation for each sub-

question on the left. Those statements are aggregated on the right. The bottom aggregation sums

up the statements found for and against the visual representations.

Figure 11: Qualitative evaluation using the inductively found categories during the development process

In total, the PCM was evaluated positively in 168 statements, with 39 statements against, meaning

81 % of statements where positive in nature. Based on this evaluation, its main weakness lies in

the higher instruction requirements. It seems to be better suited to providing an overview and is

more visually appealing.

The HTNM was referred to less by the stakeholders (150 HTNM vs. 207 PCM statements).

Furthermore, 67 % (as opposed to 81 %) of the statements were positive in nature. Its main

strengths seem to lie in the structural perception, clearer navigation, and low instruction

requirements.

Preference-oriented evaluation of HTNM and PCM

Figure 12 shows the decisions of the 10 interviewees for each sub-question on the left.

Comparing the decisions of the qualitative phase (n=10) with the questionnaire phase (n=52), we

can confirm that PCM was perceived as aesthetically more pleasing in both evaluations.

Furthermore, the joy of use was perceived to be higher in the PCM in both evaluations. The

intuitiveness was perceived as being much better in the larger sample than in the small sample.

Differences in the evaluations were the clearness of the hierarchy (this is much more evenly

distributed in the larger sample), the perception of the suitability for exploring details (the larger

sample favours HTMN for this more strongly).

Figure 12: Decision evaluation with a qualitative (n=10, expert interviews) and a quantitative round (n=52, online questionnaire)

Comparison HTNM, PCM and Compendium Web Map

Finally, we compared how the HTNM and PCM were perceived in comparison to the original

representation in Compendium Web Map (CWM). The CWM shows all nodes, their connections

and arguments modelled in Compendium as a navigable, searchable and interactive interface.

Table 1 shows the number of times a representation was chosen out of the total times it was

shown. As the representations were shown randomly, they differ slightly regarding the number

shown (105 vs. 106 times). On the right the table shows the percentage of how often the

representation was shown (of the respective total). As can be seen, the PCM and the HTNM

representation were chosen much more frequently in all criteria except for details. Here CWM was

chosen slightly more often than PCM.

Table 1: Number of times and percentage the circle, tree and Compendium representation were chosen

representation

criteria

Number of times chosen

Percentage of chosen/shown

none

PCM

HTNM

CWM

%PCM

(n=105)

%HTNM

(n=105)

%CWM

(n=106)

intuitiveness

0.73

0.56

0.15

low instruction

requirements

0.56

0.55

0.16

joy of use

0.64

0.38

0.19

hierarchy is clear

0.46

0.5

0.18

suitable for gaining

an overview

0.55

0.62

0.16

suitable for

exploring details

0.27

0.6

0.33

aesthetical

pleasing

0.8

0.38

0.16

overview of the

discussion depth

0.31

0.34

0.21

discussed

alternatives clear

0.35

0.3

0.22

total average

33.3

54.6

49.4

20.7

0.52

0.47

0.20

6.3 Discussion

In the following, we discuss the findings of part 1 of the evaluation – the content-oriented

evaluation – as well as of part 2 – the preference-oriented evaluation – based on the requirements

identified in Section 5.1. As both HTNM and PCM were strongly preferred over the Compendium

representation, we focus on both of them in the following discussion.

6.3.1 R1: include functionality to reduce the visual complexity of the argumentation visualization

Both visual representations PCM and HTNM use the same techniques to reduce visual complexity.

In both representations, the argumentation network is displayed as a navigable tree structure,

with only the top levels being shown initially as an overview. Both use zoom and pan functions to

facilitate navigation. Both provide detailed information as requested by the user. Both implement

the same filter functionalities. The only difference between the representations is the way the

information is presented visually, and consequently the mode of navigation (HTNM: pan vs. PCM:

zoom).

Table 2 shows the results of the evaluation. On the left, the number of pro and contra statements is

shown. All the statements are given from the ten experts in the qualitative evaluation phase. The

cells are highlighted orange or blue if a representation tends to be evaluated better in terms of a

particular dimension. If the distribution is even or similar, the cells are not highlighted. On the

right side, the results of the two phases of the preference-oriented evaluation are shown.

Gaining an overview: The content-oriented evaluation (COE) indicates that PCM is better suited

to gaining an overview. It was stated that it is clearer “which contents belong together” and which

“information is of interest at this level”. By contrast, the preference-oriented evaluation (DOE)

indicates that it is possible to gain an overview with both representations. Although the experts

provided considerably more single statements in favour of the PCM, when looking at final

preferences decisions, the result is fairly balanced and even slightly in favour of the HTNM. As the

third evaluation (n=52) was without human interaction, this may hint at the need for more

training so that the potential benefits of the PCM will convince more experts to choose this

representation.

Table 2: Evaluation of functionality to reduce the complexity of the argumentation visualization

Evaluation phase

Evaluation type

criteria

Evaluation 1

Evaluation 2

Evaluation 3

content-oriented evaluation

(COE)

preference-oriented evaluation

(DOE)

PCM

HTNM

PCM

HTNM

PCM

HTNM

pro

con

Pro

con

n=10

n=52

gaining an overview

13+

exploring details

navigation is clear

10+

13+

Total

31+

28+

11/30

18/30

30/104

53/104

Exploring details: In the COE, both of them seem to be equally suited. In the DOE, we find a clear

preference for the tree structure, especially in the unsupervised situation. Again we expect this to

be caused by the fact that users are much more familiar with the tree structure (as it is similar to

the Windows Explorer, for example).

The navigation is slightly clearer in the hierarchical tree structure in the COE. It might be possible

to address this through more training, since two experts stated that they would “prefer the circle

layout if they were more used to it”. At the same time, experts stated that they expect the

instruction requirements to be higher in the PCM. When choosing one of the representations in the

DOE, the experts clearly favoured the tree.

Summing the evaluation of this requirement up, we found that all functions received considerably

more pro than contra statements. We therefore consider the functions to be of value for reducing

visual complexity. For all of the functions, we found the pattern that the tree structure was the

favoured representation in an unsupervised evaluation situation, although the circle structure

received more pro statements when the interviewees could ask questions directly. For the

overview in particular, there were considerably more pro statements. One viable approach for

exploiting these potential benefits, could involve addressing the familiarity gap with an improved

introduction, training and help system.

6.3.2 R2: higher accessibility of the argumentation visualization than Compendium Web Maps

Based on the inductive category building of the qualitative development, we analysed

intuitiveness, navigation, low instruction requirements, the clearness of representation sizes and

joy of use for accessibility. Table 3 shows the number of pro and contra statements for COE and

the number of decisions. The rows are highlighted in each evaluation cell in the colour of the

dominant visual representation.

Intuitiveness is perceived similarly among both representations in the content-oriented

evaluation (COE). Both the PCM and the HTNM received far more pro than contra statements. In

the PCM, the ability to see a fast visual representation of the sub-contents was reported to feel

natural. For HTNM, the possibility of going deeper into the structure without losing the overview

was reported as very intuitive. In phase two of the evaluation (DOE), the experts’ decision was

again balanced, but in the third phase we found a strong preference for the PCM among the

interviewees. This is rather surprising, as HTNM seemed to be closer to what the interviewees

were used to.

Table 3: Evaluation of the accessibility of the argumentation visualization

Evaluation phase

Evaluation type

criteria

Evaluation 1

Evaluation 2

Evaluation 3

content-oriented evaluation

(COE)

preference-oriented evaluation

(DOE)

PCM

HTNM

PCM

HTNM

PCM

HTNM

Pro

con

pro

con

n=10

n=52

Intuitiveness

navigation is clear

10+

13+

low instruction

requirements

10-

representation sizes are

clear

joy of use

13+

Total

41+

17-

31+

12-

20/50

21/50

91/156

52/156

Navigation: As stated in the discussion of R1, the navigation is slightly clearer in the hierarchical

tree structure in the COE. When choosing one of the representations in the DOE, the experts

clearly favour the tree.

Low instruction requirements: In the COE and in evaluation 1 of the DOE, the experts stated that

they expected HTNM to have lower instruction requirements. For the PCM, the navigation

structure was reported to be unusual, which results in the need for training. Additionally, the

positioning of the circle elements inside larger circles was reported as requiring explanation. In

contrast to this, the instruction requirements were evaluated similarly in the third evaluation of

the DOE. One reason for this may be that we showed a short tutorial explaining the structure for

both representations in the non-supervised setting. This may hint at the possibility of reducing

this perception substantially via this short tutorial.

Representation sizes are clear: Both in the COE and DOE, experts favoured the PCM over the

HTNM. In the PCM, experts stated that they favoured the possibility of adding additional

information in the circle size, which is not feasible in the HTNM. Additionally, the experts liked the

visual “weighting” of the sizes.

Joy of use: The experts strongly favoured the PCM in all three evaluations. The experts stated that

exploring the data was more fun when navigating inside circles than the more common tree

structure.

In total, the experts reported considerably more pro statements (PCM: 41; HTNM: 31) than contra

statements (PCM: 17; HTNM: 12). We therefore consider this requirement to be met. When

looking at the decisions, both representations were chosen equally in evaluation 2 (DOE), but PCM

clearly dominated the third evaluation with an unsupervised setting. It seems that the tutorial

provided helped people to overcome the higher perceived instruction requirements.

6.3.3 R3: improve information presentation and coding compared to Compendium Web Maps

We analysed usage of the visual space, aesthetics, text representation, and low perception

requirements for information presentation. Furthermore, we analysed hierarchy perception,

overview and detail suitability for information coding. Table 4 shows the number of pro and

contra statements for COE and the number of decisions. In each evaluation cell, the rows are

highlighted in the colour of the dominant visual representation.

Table 4: Evaluation of the information coding and information presentation of the argumentation visualization

Evaluation phase

Evaluation type

criteria

Evaluation 1

Evaluation 2

Evaluation 3

content-oriented evaluation

(COE)

preference-oriented evaluation

(DOE)

PCM

HTNM

PCM

HTNM

PCM

HTNM

pro

con

pro

con

n=10

n=52

usage of the visual space

11+

aesthetical pleasing

11+

text representation

low perception

requirements

total information coding

35+

11+

13-

21/40

8/40

40/52

7/52

hierarchy is clear

12+

19+

suitable for gaining an

overview

13+

suitable for exploring

details

total information

presentation

33+

34+

11/30

15/30

51/156

70/156

Total

68+

14-

45+

19-

32/70

23/70

91/208

77/208

Usage of the visual space is clearly perceived as better for the PCM in both evaluation 1 and 2.

The experts stated that the circles allowed more information to be displayed compared to a tree

structure.

Aesthetics is strongly perceived to be a strength of the PCM in all three evaluations.

Text representation: The experts provided slightly more statements in favour of the PCM in the

COE. In both representations, displaying very long element names was reported to be an issue.

However, the PCM deals better with displaying the text of multiple alternatives (which have to be

cut off in the HTNM). In the second evaluation (DOE), both representations were chosen equally.

Lower perception requirements: Regarding the lower perception requirements, the experts

favoured the PCM in the first evaluation (COE). It was reported that the PCM made it easier to

focus on information on the same level. By contrast, both representations were evaluated similarly

in the second evaluation (DOE).

Total information coding: In total, information coding seems to be better in the PCM. While the

PCM received significantly more pro than contra statements (35+ vs. 7-), the HTNM received more

contra than pro statements (11+ vs. 13-). When users had to choose a representation (DOE), we

also found that most experts preferred PCM in terms of information coding criteria.

Hierarchy is clear: The hierarchy was perceived more clearly with the HTNM representation in

the first (COE) and second (DOE) evaluations. By contrast, we found a higher preference for the

PCM in an unsupervised setting. This change may be influenced by the tutorial which was shown

in the third evaluation, while evaluations 1 and 2 only involved the interviewer explaining the

representations.

Suitable for gaining an overview: As stated in the discussion of requirement 1, PCM was

perceived as more suitable for providing an overview in the first (COE) and second (DOE)

evaluations. It is perceived similarly in the third evaluation (DOE).

Suitable for exploring details: As stated in the discussion of requirement 1, PCM and HTNM

were perceived similarly for exploring details in the first evaluation (COE). In the third evaluation

(DOE), the experts favoured the HTNM.

Total information presentation: Regarding the number of statements in the COE, both

representations were perceived similarly positively. However, in the DOE the experts leaned

slightly towards HTNM – in both a supervised and unsupervised setting.

In total, the PCM was viewed more positively than the HTNM (PCM: 68+ vs. 14-; HTNM: 45+ vs. 19-

) in the first evaluation (COE); as such, we consider this requirement to be met. In the second and

third evaluations (DOE), PCM also performed better according to the experts.

6.3.4 R4: provide the same depth of information as in Compendium Web Maps

We analysed the overview of the discussion depth, increase of interest, argumentation complexity

and clearness of alternatives for information depth. The number of pro and contra statements for

COE and the number of decisions are provided in Table 5. In each evaluation cell, the rows are

highlighted in the colour of the dominant visual representation.

Table 5: Evaluation of the information depth of the argumentation visualization

Evaluation phase

Evaluation type

criteria

Evaluation 1

Evaluation 2

Evaluation 3

content-oriented evaluation

(COE)

preference-oriented evaluation

(DOE)

PCM

HTNM

PCM

HTNM

PCM

HTNM

pro

con

pro

con

n=10

n=52

overview over the

discussion depth

16+

increases the interest in

contents

12+

shows argumentation

complexity

13+

discussed alternatives

are clear

18+

12+

11-

total

59+

25+

18-

22/40

13/40

26/104

Overview of the discussion depth: The experts provided substantially more statements for the

PCM in the first evaluation (COE). One main difference was that the circle does provide

information about the total number of alternatives and arguments faster. However, in the DOE we

found a slight preference for the HTNM. We believe this to be influenced by familiarity with the

interaction concept in HTNM.

Increases the interest in the contents: In the first (COE) and second (DOE) evaluations, the

experts clearly favoured the PCM. The main difference to the tree structure was reported as being

the PCM’s option of showing substructures in the circle before clicking.

Shows argumentation complexity: In the first (COE) and second (DOE) evaluations, the experts

clearly favoured the PCM. The experts stated that the form of visual representation made it clearer

that pro and contra arguments exist.

Discussed alternatives are clear: The PCM outperformed the HTNM in all evaluations (COE and

DOE). For example, the experts stated that it was easier to perceive the total number of

alternatives (as they are all shown as circles around the focused circle), while in HTNM they have

to be shortened or even hidden when the number of alternatives is too high.

Overall, the experts stated a clear preference for PCM in the first (COE) and second (DOE)

evaluations in terms of information depth. In the DOE evaluation, however, we see that experts

chose equally among the two visual representations of the questions that could be asked. This may

be skewed a little by leaving out two questions in which PCM is clearly favoured.

6.4 Limitations

The evaluation of the qualitative phases with 10 experts focused on providing detailed insights

into the reasoning for and against the visual representations. Because of the small number of

participants, the results should be interpreted with caution. For the quantitative phase, we had to

remove some questions which were found to be hard to explain in a standardized quantitative

interview without additional explanations. This means that not all dimensions of the criteria were

tested both qualitatively and quantitatively. The quantitative phase incorporated a larger number

(52) of interviewees, but because of the low participation rate (1 %) it is questionable whether the

results can be generalized to other fields without further research. The results do permit the

conclusion that both PCM and HTNM are perceived much better than the original Compendium

Web Maps in this context. To apply these results to other fields of application, further studies with

a larger quantitative sample are necessary.

7 Conclusion and Outlook

The two newly developed visual representations PCM and HTNM both outperform the existing

Compendium Web Maps in terms of all requirements analysed in our project. Our qualitative and

quantitative evaluation shows that PCM is better suited to increasing accessibility, information

presentation and coding as well as information depth. The HTNM is perceived as being slightly

stronger when it comes to reducing complexity. Regarding the specific sub-dimensions, we found

highly varying perceptions although the presented content was consistent across all

representations. These insights pave the way for a more empirical-based usage of visual

representations in development processes with a high number of stakeholders. While

argumentation modelling should still be performed using tools like Compendium, the visual

representation of results should be provided in the developed visualizations, whose focus is

customised based on the goal of the visualization.

As a further research step, we plan to expand the application of both visual representations to

other use cases than the standardization case. Furthermore, extending the evaluation to a larger

quantitative study with more participants will allow for a higher generalizability of the results.

8 Literature

Al Neimat, T. (2005). Why IT projects fail. Online in Project Perfect Project Management Software,

Available at http://www. Projectperfect. Com. Au/info_it_projects_fail. Php (Retrieved 8/2011).

Retrieved from http://sct.emu.edu.tr/courses/mtit/itec580/userfiles/files/info_it_projects_fail.pdf

Awati, K. (2011). Mapping project dialogues using IBIS: a case study and some reflections.

International Journal of Managing Projects in Business, 4(3), 498–511.

Bostock, M. (2012). Circle Packing. Retrieved April 20, 2015, from

http://bl.ocks.org/mbostock/4063530

Bostock, M., & Heer, J. (2009). Protovis: A graphical toolkit for visualization. Visualization and

Computer Graphics, IEEE Transactions on, 15(6), 1121–1128.

Bostock, M., Ogievetsky, V., & Heer, J. (2011). D3 data-driven documents. Visualization and Computer

Graphics, IEEE Transactions on, 17(12), 2301–2309.

Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in information visualization: using

vision to think. Morgan Kaufmann.

Cerpa, N., & Verner, J. M. (2009). Why Did Your Project Fail? Commun. ACM, 52(12), 130–134.

http://doi.org/10.1145/1610252.1610286

Cleveland, W. S., & McGill, R. (1984). Graphical perception: Theory, experimentation, and application

to the development of graphical methods. Journal of the American Statistical Association, 531–

554.

Conklin, J. (2003). Dialog mapping: Reflections on an industrial strength case study. In Visualizing

argumentation (pp. 117–136). Springer. Retrieved from

http://link.springer.com/chapter/10.1007/978-1-4471-0037-9_6

Conklin, J. (2006). Wicked problems & social complexity. CogNexus Institute Napa, USA.

Conklin, J., & Begeman, M. L. (1988). gIBIS: a hypertext tool for exploratory policy discussion. ACM

Transactions on Information Systems (TOIS), 6(4), 303–331.

El Emam, K., & Koru, A. G. (2008). A replicated survey of IT software project failures. Software, IEEE,

25(5), 84–90.

Eveleens, J. L., & Verhoef, C. (2010). The rise and fall of the chaos report figures. IEEE Software,

27(1), 30–36.

Glass, R. L. (2006). The standish report: does it really describe a software crisis? Communications of the

ACM, 49(8), 15–16.

Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum

impact. MIS Quarterly, 37(2), 337–356.

Heeks, R. (2003). Most eGovernment-for-development projects fail: how can risks be reduced? Institute

for Development Policy and Management, University of Manchester Manchester. Retrieved from

http://unpan1.un.org/intradoc/groups/public/documents/cafrad/unpan011226.pdf

Hevner, A., & Chatterjee, S. (2010). Design science research in information systems. Springer.

Retrieved from http://link.springer.com/chapter/10.1007/978-1-4419-5653-8_2

Hevner, A. R., Salvatore, T. M., Park, J., & Ram, S. (2004). Design science in information systems

research. MIS Quarterly, 28(1), 75–105.

Introne, J., Laubacher, R., Olson, G., & Malone, T. (2013). Solving Wicked Social Problems with Socio-

computational Systems. KI-Künstliche Intelligenz, 27(1), 45–52.

Jones, C. (2000). Software assessments, benchmarks, and best practices. Addison-Wesley Longman

Publishing Co., Inc. Retrieved from http://dl.acm.org/citation.cfm?id=335582

Jørgensen, M., & Moløkken-Østvold, K. (2006). How large are software cost overruns? A review of the

1994 CHAOS report. Information and Software Technology, 48(4), 297–301.

Kuechler, W., & Vaishnavi, V. (2008). The emergence of design research in information systems in

North America. Journal of Design Research, 7(1), 1–16.

Kunz, W., & Rittel, H. W. (1970). Issues as elements of information systems (Vol. 131). Oakland:

Institute of Urban and Regional Development, University of California.

Loukis, E., & Wimmer, M. (2012). A multi-method evaluation of different models of structured

electronic consultation on government policies. Information Systems Management, 29(4), 284–

294.

Loukis, E., Wimmer, M. A., Charalabidis, Y., Triantafillou, A., & Gatautis, R. (2007). Argumentation

systems and ontologies for enhancing public participation in the legislation process.

Mazza, R. (2009). Introduction to information visualization. Springer-Verlag New York Inc.

Monk, S., Sommerville, I., Pendaries, J. M., & Durin, B. (1995). Supporting design rationale for system

evolution. In Software Engineering—ESEC’95 (pp. 307–323). Springer. Retrieved from

http://link.springer.com/chapter/10.1007/3-540-60406-5_22

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research

methodology for information systems research. Journal of Management Information Systems,

24(3), 45–77.

Pries-Heje, J., & Baskerville, R. (2006). Designing organizational change in IT: A theory nexus. In 1st

International Conference on Design Science Research in Information Systems & Technology,

Claremont, California.

Renton, A. (2006). Seeing the point of politics: exploring the use of CSAV techniques as aids to

understanding the content of political debates in the Scottish Parliament. Artificial Intelligence

and Law, 14(4), 277–304.

Riechert, M., Biesenbender, S., Dees, W., & Sirtes, D. (2016). Developing Definitions of Research

Information Metadata as a Wicked Problem? Characterisation and Solution by Argumentation

Visualisation. Program, 50(3). http://doi.org/doi:10.1108/PROG-01-2015-0009

Rittel, H. W., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences,

4(2), 155–169.

Robertson, G., Card, S. K., & Mackinlay, J. D. (1989). The cognitive coprocessor architecture for

interactive user interfaces. In Proceedings of the 2nd annual ACM SIGGRAPH symposium on

User interface software and technology (pp. 10–18). ACM. Retrieved from

http://dl.acm.org/citation.cfm?id=73662

Rubinstein, D. (2007). Standish group report: There’s less development chaos today. Software

Development Times, 1. Retrieved from

http://projectmgnt.citsolutions.edu.au/Clients/DOGPM/documentation/Standish_Group_Chaos_

Article_2006.pdf.pdf

Sauer, C., & Cuthbertson, C. (2003). The state of IT project management in the UK 2002-2003.

Retrieved from http://www.citeulike.org/group/1694/article/896485

Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. M. (2010). Computer-supported argumentation: A

review of the state of the art. International Journal of Computer-Supported Collaborative

Learning, 5(1), 43–102.

Schneider, J., Groza, T., & Passant, A. (2013). A review of argumentation for the Social Semantic Web.

Semantic Web, 4(2), 159–218.

Schoder, P. D. D., Putzke, D. J., Metaxas, P. P. T., Gloor, P. D. P. A., & Fischbach, P. D. K. (2014).

Information Systems for “Wicked Problems.” Business & Information Systems Engineering, 1–

8. http://doi.org/10.1007/s12599-013-0303-3

Selvin, A., Buckingham Shum, S., Seirhuis, M., Conklin, J., Zimmerman, B., Palus, C., … Motta, E.

(2001). Compendium: Making meetings into knowledge events. In Proceedings of the 2001

Knowledge Technologies 2001 Conference. Austin, Texas.

Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations.

In Visual Languages, 1996. Proceedings., IEEE Symposium on (pp. 336–343).

Shneiderman, B., & Plaisant, C. (2006). Strategies for evaluating information visualization tools: multi-

dimensional in-depth long-term case studies. In Proceedings of the 2006 AVI workshop on

BEyond time and errors: novel evaluation methods for information visualization (pp. 1–7).

Shum, S. B., Selvin, A., Sierhuis, M., Conklin, J., Haley, C. B., & Nuseibeh, B. (2006). Hypermedia

support for argumentation-based rationale. In Rationale management in software engineering

(pp. 111–132). Springer.

Simon, H. A. (1969). The sciences of the artificial. Cambridge, MA.

Standish, G. (2001). Extreme CHAOS. The Standish Group International.

Suthers, D. D. (2008). Empirical studies of the value of conceptually explicit notations in collaborative

learning. In Knowledge cartography (pp. 1–23). Springer.

Treisman, A. (1985). Preattentive processing in vision. Computer Vision, Graphics, and Image

Processing, 31(2), 156–177.

Wang, W., Wang, H., Dai, G., & Wang, H. (2006). Visualization of Large Hierarchical Data by Circle

Packing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

(pp. 517–520). New York, NY, USA: ACM. http://doi.org/10.1145/1124772.1124851

Ware, C. (2004). Information visualization: perception for design (Vol. 22). Morgan Kaufmann.

Wertheimer, M. (1922). Untersuchungen zur Lehre von der Gestalt. Psychological Research, 1(1), 47–

58.

Bisher erschienene Bände der Schriftenreihe

Research Papers in Information Systems Management

ISSN 2191-639X (online)

ISSN 2196-8187 (print)

Band 01

Zarnekow, Rüdiger; Kolbe, Lutz M.; Erek, Koray; Schmidt, Nils-Holger

Studie: Nachhaltigkeit und Green IT in IT-Organisationen

ISBN (online) 978-3-7983-2263-9

Published online 2010

Band 02

Repschläger, Jonas; Zarnekow, Rüdiger

Studie: Cloud Computing in der IKT-Branche

ISBN (online) 978-3-7983-2305-6

Published online 2011

Band 03

Zarnekow, Rüdiger; Erek, Koray; Löser, Fabian; Wilkens, Marc

Referenzmodell für ein Nachhaltiges Informationsmanagement

ISBN (online) 978-3-7983-2378-0

ISBN (print) 978-3-7983-2385-8 Preis: 4,90 Euro

Published 2011

Band 04

Erek, Koray; Schmidt, Nils-Holger; Löser, Fabian; Samulat, Peter

Nachhaltigkeitsmanagement bei der Axel Springer AG. Auf dem Weg zu einer Green IT

ISBN (online) 978-3-7983-2400-8

Published online 2012

Band 05

Erek, Koray; Schmidt, Nils-Holger; Schilling, Thomas

Green IT bei Bayer Business Services

ISBN (online) 978-3-7983-2401-5

Published online 2012

Band 06

Erek, Koray; Schmidt, Nils-Holger; Glau, Thomas

Green IT im IT-Dienstleistungszentrum Berlin

ISBN (online) 978-3-7983-2402-2

Published online 2012

Band 07

Erek, Koray ; Schmidt, Nils-Holger ; Löser, Fabian

Nachhaltigkeitsorientiertes IT-Management bei einem internen IT-Dienstleister

ISBN (online) 978-3-7983-2403-9

Published online 2012

Band 08

Opitz, Nicky; Erek, Koray; Henseler, Rainer

Green IT im Bundesverwaltungsamt

ISBN (online) 978-3-7983-2485-5

Published online 2012

Band 09

Schmidt, Nils-Holger; Erek, Koray; Kusiak, Katja; Stelzer, Timo

Green IT bei der SAP AG

ISBN (online) 978-3-7983-2486-2

Published online 2012

Band 10

Schmidt, Nils-Holger; Erek, Koray; Kusiak, Katja

Green IT bei der Üstra Hannoversche Verkehrsbetriebe

ISBN (online) 978-3-7983-2487-9

Published online 2012

Band 11

Opitz, Nicky; Erek, Koray; Rekers, Jan; Dahlem, Markus

Green IT bei der Deutschen Bank AG

ISBN (online) 978-3-7983-2488-6

Published online 2012

Band 12

Repschläger, Jonas; Hahn, Christopher; Zarnekow, Rüdiger

Studie: Handlungsfelder im Cloud Computing

ISBN (online) 978-3-7983-2491-6

Published online 2012

Band 13

Repschläger, Jonas; Zarnekow, Rüdiger

Umfrage zur Anbieterauswahl & Markttransparenz im Cloud Computing

ISBN (online) 978-3-7983-2501-2

Published online 2013

Band 14

Limbach, Felix; Kübel, Hannes; Zarnekow, Rüdiger

Kooperativer Breitbandausbau in Deutschland. Eine Expertenbefragung unter

Unternehmensführern und Kooperationsverantwortlichen der deutschen

Telekommunikationsbranche

ISBN (online) 978-3-7983-2590-6

ISBN (print) 978-3-7983-2589-0 Preis: 5,90 Euro

Published 2013

Band 15

Repschläger, Jonas; Zarnekow, Rüdiger; Meinhardt, Nils; Röder, Christoph; Pröhl, Thorsten

Vertrauen in der Share Economy. Studie: Analyse von Vertrauensfaktoren für Online-Profile

ISBN (online) 978-3-7983-2775-7

Published online 2015

Band 16

Zarnekow, Rüdiger; Pröhl, Thorsten

Preisvorteile durch frei konfigurierbare Instanzen im Rahmen des Cloud Computing

ISBN (online) 978-3-7983-2839-6

Published online 2016

Band 17

Schlesinger, Daniel; Zarnekow, Rüdiger; Repschläger, Jonas

Analyse der Wohnungsbewertungen von Airbnb

ISBN (online) 978-3-7983-2844-0

Published online 2016

Universitätsverlag der TU Berlin

Improving Argumentation Visualization of Multi-Stakeholder Development –

A Prototyping Case

A shared understanding of development argumentation is crucial for a wide range of development processes

(such as requirements engineering, change management, eGovernment and eParticipation, public policy) and

central to prevent the failure of IT and development projects. Computer-Supported Argumentation Visualization

(CSAV) has been used to model and represent discourse information for about 35 years. Although modelling

tools have significantly matured and continue to evolve, the visual representation of existing tools does not scale

ideally with increasing model complexity. For large-scale argumentation models, existing visualization approa-

ches from argumentation visualization are reported as being too complex for target stakeholders. This prevents

them from gaining insights into the development process and may ultimately contribute to the rejection of the

development result, causing severe costs for both public and private organizations. In this paper, we employ the

‘design science’ methodology to incrementally develop two interactive visual representations for argumentation

visualization, incorporating best practices from information visualization research. The prototypes are imple-

mented and evaluated in the setting of the project “Research Core Dataset”, a nation-wide project involving all

major stakeholder groups of the German science system in order to develop harmonized definitions for research

information. In our evaluation, both of the visual representations developed are perceived as being much better

at providing insights into complex development processes with a high number of stakeholders.

ISBN 978-3-7983-2994-2 (online)

9 783798 329942

ISBN 978-3-7983-2994-2

http://verlag.tu-berlin.de