A comprehensive approach for the empirical investigation of success factors in product development [original]

A Comprehensive Approach for the

Empirical Investigation of Success Factors

in Product Development

vorgelegt von

Dipl.-Ing.

Ulrich W¨orz

geb. in Laupheim

von der Fakult¨at V - Verkehrs- und Maschinensysteme

der Technischen Universit¨at Berlin

zur Erlangung des akademischen Grades

Doktor der Ingenieurwissenschaften

- Dr.-Ing. -

genehmigte Dissertation

Promotionsausschuss:

Vorsitzender: Prof. Dr.-Ing. Robert Liebich

Gutachter: Prof. Dr.-Ing. Dietmar G¨ohlich

Gutachter: Prof. Dr.-Ing. Lucienne Blessing

Tag der wissenschaftlichen Aussprache: 30. September 2014

Berlin 2015

D 83

Acknowledgements

This dissertation is the result of my doctoral research work at the Department for Prod-

uct Development Methods and Mechatronics at the Technische Universit¨at Berlin, while

working as a design engineer and project manager at the Siemens Energy corporation.

It would not exist without the invaluable contribution from both colleagues in academia

and colleagues at my work place.

First and most I would like to thank my advisor Prof. Dr.-Ing. Dietmar G¨ohlich

for taking on the challenge of a long distance research relationship. His dedication and

flexibility allowed for extremely productive ’sparring sessions’ during my visits in Berlin.

Many thanks also to my second advisor Prof. Dr.-Ing. Lucienne Blessing for agreeing

to supervise this work, despite the logistical challenge of adding a third country to the

geographical mix.

I would like to extend my gratitude to the whole Department for Product Development

Methods and Mechatronics, in particular Dipl.-Ing. Felix Spangenberg, for the inspiring

discussions. Not to forget Mrs. Barbara Schmunkamp for her quick responses and support

in overcoming administrative hurdles. Things would not have run as smoothly without

her great help.

Special thanks go to my employer, the Siemens Energy corporation. Without the

permission to dig into archives of data and interview employees, this work would have

not been able to produce the results and research contribution it did. I am particularly

thankful to Dr. Alexander Beeck, for taking the time for providing insight into the

company’s development process and asking critical questions. His years of experience

with successful and less successful development projects provided invaluable input, which

greatly helped to shape this research effort.

Lastly, I would like to give my warmest thanks to my family, especially to my wife

Ad`ele, my son Edward and my daughter Helena. They sacrificed many evenings and

weekends to allow me to finish this dissertation in a timely manner. Finally I am at a

point where they can redeem all the credit of quality time with their husband and daddy

which they collected over the last four years.

Abstract

The outcome of product development projects depends on a multitude of different factors,

deriving not only from technical challenges, but also from aspects like human behavior or

complex organizational environments. Empirical research studies have become increas-

ingly important in verifying which of these Potential Success Factors contribute most to

successful product development. Many studies have focused on certain aspects of the

design process, such as the individual designer or design teams. These have contributed

greatly to a better understanding in Engineering Design Research. Comprehensive stud-

ies, viewing the product development process holistically with all its influencing factors,

are rare. However, additional information can be obtained from comprehensive investi-

gations. In addition to understanding which of the Potential Success Factors show an

actual correlation to Design Success - then considered Success Factors - it can also be

investigated how these Success Factors depend on each other. Such dependencies provide

information about causalities and subsequently lead to a set of Dominant Success Factors.

These not only influence the outcome of product development projects significantly, but

also cause other Success Factors to change, which is why they need to be given the first

and most attention by engineering management.

This dissertation illustrates the framework of an empirical approach, which supports the

comprehensive investigation of Success Factors in product development. The approach

was applied to 44 gas turbine component development projects of varying complexity.

The collected data was analyzed using statistical methods, which provided information

about the reliability of the results. Following the proposed approach with its two analy-

sis steps lead to the identification of the following four Dominant Success Factors, which

distinguished successful from less successful development projects: Priority of Project;

Skills and Experience of Project Lead; Complexity and amount of Innovation of Project;

Awareness of Lessons Learned and State of the Art Technology Knowledge. The remain-

ing factors, not identified as dominant, are considered Supplementary Success Factors, as

they can help to leverage Design Success. However, the research work clearly shows that

the Dominant Success Factors contribute most and have to be seen as the foundation

for successful product development. Engineering management can use the results of this

work for an improved project risk management, especially in the early stages of product

development.

III

Zusammenfassung

Das Ergebnis von Produktentwicklungsprojekten h¨angt von einer Vielzahl verschiedener

Faktoren ab, welche nicht nur aus technischen Herausforderungen, sondern auch aus

anderen Aspekten, wie etwa des menschlichen Verhaltens oder komplexer Organisation-

sumgebungen, bestehen. Empirische Forschungsstudien wurden im Laufe der Zeit immer

wichtiger, um zu verifizieren welche dieser Potenziellen Erfolgsfaktoren am meisten zur er-

folgreichen Produktentwicklung beitragen. Viele der durchgef¨uhrten Studien fokusierten

auf bestimmte Bereiche des Entwicklungsprozesses, zum Beispiel dem Entwickler als In-

dividuum oder Entwicklungsteams, und haben entscheidend zum besseren Verst¨andis

im Forschungsbereich Engineering Design Research beigetragen. Ganzheitliche Studien,

die den Produktentwicklungsprozess holistisch, mit all seinen Einflussfaktoren, betra-

chten, sind selten zu finden. Jedoch, k¨onnen gerade aus diesen ganzheitlichen Unter-

suchungen zus¨atzliche Information gewonnen werden. Außer dem Verst¨andnis welche der

Potenziellen Erfolgsfaktoren einen Zusammenhang zu Entwicklungserfolg zeigen - dann

als Erfolgsfaktoren identifiziert - k¨onnen zudem Abh¨angigkeiten der Erfolgsfaktoren un-

tereinander untersucht werden. Derartige Abh¨angigkeiten geben Aufschluss ¨uber kausale

Zusammenh¨ange und f¨uhren letztlich zu einer Gruppe von Dominanten Erfolgsfaktoren.

Diese beeinflussen nicht nur die Ergebnisse von Produktentwicklungsprojekten signifikant,

sondern verursachen zus¨atzlich die ¨

Anderung anderer Erfolgsfaktoren, weshalb sie die

bevorzugte und h¨ochste Aufmerksamkeit der Entwicklungsleitung erhalten m¨ussen.

Diese Dissertation beschreibt das Grundger¨ust einer empirischen Methode, welche die

ganzheitliche Untersuchung von Erfolgsfaktoren in der Produktentwicklung unterst¨utzt.

Die Methode wurde auf 44 Komponentenentwicklungsprojekte von Gasturbinen, mit vari-

ierender Komplexit¨at, angewandt. Die gesammelten Daten wurden mit Hilfe statistischer

Verfahren ausgewertet, was Auskunft ¨uber die Verl¨asslichkeit der Resultate gab. Die

Anwendung der vorgeschlagenen Methode mit ihren zwei Analyseschritten f¨uhrte letzt-

lich zu den vier folgenden Dominanten Erfolgsfaktoren, durch welche sich erfolgreiche

von weniger erfolgreichen Entwicklungsprojekten unterscheideten: Priorit¨at des Projekts;

F¨ahigkeiten und Erfahrung des Projektleiters; Complexit¨at und Innovationsanteil im Pro-

jekt; Bewusstsein ¨uber Gewonnene Erkenntnisse und aktuellen Stand der Technik. Die

verbleibenden Faktoren der Studie, die nicht als dominant indentifziert wurden, werden

als Erg¨anzende Erfolgsfaktoren betrachtet, da sie durchaus unterst¨utzend zu besserem En-

twicklungserfolg beitragen k¨onnen. Allerdings zeigen die Forschungsergebnisse deutlich,

dass die Dominanten Erfolgsfaktoren den h¨ochsten Beitrag leisten und daher als Funda-

ment f¨ur erfolgreiche Entwicklungsprojekte angesehen werden m¨ussen. F¨uhrungskr¨afte

von Entwicklungsabteilungen k¨onnen die Ergebnisse dieser Arbeit zur Verbesserung von

Projektrisikomanagement, vor allem in fr¨uhen Stadien von Entwicklungsprojekten, ver-

wenden.

Contents

1 Motivation 1

1.1 ProblemStatement.............................. 1

1.2 The Product Development Process . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Systematic Approach . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.2 Quality Function Deployment . . . . . . . . . . . . . . . . . . . . 5

1.2.3 Design for Six Sigma . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3 Factors influencing Design Success . . . . . . . . . . . . . . . . . . . . . . 9

1.4 Engineering Design Research . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.5 ResearchQuestions.............................. 12

1.6 Research Object and Observed Process . . . . . . . . . . . . . . . . . . . 12

1.7 Dissertationoutline.............................. 15

2 Literature Review and Research Needs 16

2.1 Characteristics of Empirical Industry Studies . . . . . . . . . . . . . . . . 17

2.2 Review of different Types of performed Studies . . . . . . . . . . . . . . . 22

2.2.1 Hales: Analysis of the engineering design process in an industrial

context................................. 22

2.2.2 Ehrlenspiel: Problems in Development and Design . . . . . . . . . 23

2.2.3 White and Fortune: Current practice in project management - an

empiricalstudy ............................ 25

2.2.4 Ahmed: Understanding the knowledge needs of novice designers in

the aerospace industry . . . . . . . . . . . . . . . . . . . . . . . . 27

2.3 Meta-Analyses on Success Factors . . . . . . . . . . . . . . . . . . . . . . 29

2.4 ResearchNeeds ................................ 32

2.5 Characteristics of the Comprehensive Approach . . . . . . . . . . . . . . 34

3 Framework of a Comprehensive Empirical Approach 36

3.1 Link to Design Research Methodology . . . . . . . . . . . . . . . . . . . 39

3.2 StatisticalMethods.............................. 41

3.3 Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

VII

4 Defining Design Success 45

4.1 Success Dimensions of the Comprehensive Approach . . . . . . . . . . . . 47

4.2 Primary Success Dimension . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.3 Data Collection for Primary Success Dimension . . . . . . . . . . . . . . 49

4.4 Secondary Success Dimension . . . . . . . . . . . . . . . . . . . . . . . . 50

5 Potential Success Factors 51

5.1 Reduction of Potential Success Factors . . . . . . . . . . . . . . . . . . . 51

5.2 Quantification of Potential Success Factors . . . . . . . . . . . . . . . . . 54

5.2.1 Use of Design Methodology . . . . . . . . . . . . . . . . . . . . . 54

5.2.2 Conceptual Design performed . . . . . . . . . . . . . . . . . . . . 55

5.2.3 Decision Tools utilized . . . . . . . . . . . . . . . . . . . . . . . . 55

5.2.4 Resources provided / Project Stability . . . . . . . . . . . . . . . 56

5.2.5 Priority of Project . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.2.6 TeamSize............................... 57

5.2.7 Team Members Experience / Skills . . . . . . . . . . . . . . . . . 57

5.2.8 Cultural Influence in Team . . . . . . . . . . . . . . . . . . . . . . 58

5.2.9 Experience / Skills of Project Lead . . . . . . . . . . . . . . . . . 59

5.2.10 Project Lead Personality . . . . . . . . . . . . . . . . . . . . . . . 59

5.2.11 Level of Process Compliance . . . . . . . . . . . . . . . . . . . . . 60

5.2.12 Simultaneous Development . . . . . . . . . . . . . . . . . . . . . . 61

5.2.13 Product Requirements clearly defined and stable during Development 62

5.2.14 Awareness of Lessons Learned and State of the Art Technology

Knowledge............................... 62

5.2.15 Ratio of new Technology . . . . . . . . . . . . . . . . . . . . . . . 63

5.2.16 Technical Complexity of Project . . . . . . . . . . . . . . . . . . . 64

5.2.17 Co-Location of Team Members . . . . . . . . . . . . . . . . . . . 64

5.2.18 Empowerment of Project Lead: Project and Budget Responsibility

inoneHand.............................. 65

5.3 Data Collection and Validation for Potential Success Factors . . . . . . . 65

6 Data Analysis 70

6.1 First Analysis Step: Search for Success Factors . . . . . . . . . . . . . . . 70

6.1.1 ANOVA for Hypothesis Test . . . . . . . . . . . . . . . . . . . . . 71

6.1.2 Testing for Normal Distribution . . . . . . . . . . . . . . . . . . . 74

6.1.3 The Kruskal-Wallis Method as alternative for ANOVA . . . . . . 75

6.1.4 Results of the First Analysis Step . . . . . . . . . . . . . . . . . . 76

6.2 Second Analysis Step: Search for Dependencies . . . . . . . . . . . . . . 84

6.3 Interpretation of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

VIII

6.4 Results for Secondary Success Dimension . . . . . . . . . . . . . . . . . . 92

6.5 Correlation of Findings to Literature . . . . . . . . . . . . . . . . . . . . 93

7 Applicability 96

7.1 Dominant Success Factors . . . . . . . . . . . . . . . . . . . . . . . . . . 97

7.2 Supplementary Success Factors . . . . . . . . . . . . . . . . . . . . . . . 100

7.2.1 Supplementary Success Factors detected in the first analysis step . 100

7.2.2 Supplementary Success Factors detected in the second analysis step 102

7.3 Limitations of this Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

7.4 Recommendation for Further Research . . . . . . . . . . . . . . . . . . . 105

8 Conclusion 108

Bibliography 112

Appendix 119

List of Figures

1.1 Committed costs during product development process (Ullman, 2003) . . 3

1.2 Process map of the Systematice Approach by Pahl and Beitz (2007) . . . 6

1.3 The four phases of Quality Function Deployment (Yang et al., 2003) . . . 7

1.4 Common layout of design methodologies . . . . . . . . . . . . . . . . . . 8

1.5 Role of the development process in the product life cycle . . . . . . . . . 9

1.6 The central activity of engineering design (Dixon, 1966) . . . . . . . . . . 10

1.7 Area of interest in Engineering Design Research (Horv´ath, 2004) . . . . . 11

1.8 Main components of a gas turbine (Siemens, 2013) . . . . . . . . . . . . . 13

1.9 Product Development Process in company of study . . . . . . . . . . . . 14

2.1 High level timeline of Engineering Design Research . . . . . . . . . . . . 17

2.2 Result of Ehrlenspiel’s survey study, translated from (Ehrlenspiel, 2009) . 24

2.3 Success reported in White and Fortune’s empirical study . . . . . . . . . 26

2.4 Patterns of query and responses for gathering information . . . . . . . . 28

2.5 Cross-check of patterns against topics of queries . . . . . . . . . . . . . . 28

3.1 Cause-effect relationship in product development . . . . . . . . . . . . . . 36

3.2 Transfer function of product development process . . . . . . . . . . . . . 37

3.3 Theory behind the Six Sigma approach (Brue and Launsby, 2003) . . . . 37

3.4 Concept of hypothesis test . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.5 Process map of the Comprehensive Approach . . . . . . . . . . . . . . . 39

3.6 Seven possible types of studies in the DRM framework . . . . . . . . . . 40

4.1 Examples of domains and their success dimensions . . . . . . . . . . . . . 46

4.2 Design Success of 44 investigated projects . . . . . . . . . . . . . . . . . 50

5.1 Iterative approach for reduction of data points . . . . . . . . . . . . . . . 52

5.2 Remaining Potential Success Factors after systematic reduction . . . . . 53

6.1 Example of F distribution with α-level = 0.05 . . . . . . . . . . . . . . . 73

6.2 Box plot for verification if data is normally distributed . . . . . . . . . . 74

6.3 Chi-Square distribution for df =2...................... 76

6.4 Impact of conceptual design on Design Success . . . . . . . . . . . . . . . 79

6.5 Impact of priority for engineering management on Design Success . . . . 79

6.6 Impact of project lead’s experience and skills on Design Success . . . . . 79

6.7 Impact of new technology on Design Success . . . . . . . . . . . . . . . . 80

6.8 Impact of project’s complexity on Design Success . . . . . . . . . . . . . 80

6.9 Cultural influence on Design Success . . . . . . . . . . . . . . . . . . . . 82

6.10 Impact of lessons learned and technology knowledge on Design Success . 83

6.11 Impact of co-location on Design Success . . . . . . . . . . . . . . . . . . 83

6.12 The eight Success Factor determined in the first analysis step . . . . . . . 84

6.13 P-values for dependencies between Success Factors . . . . . . . . . . . . . 86

6.14 Example of observed counts for Chi-Square Test . . . . . . . . . . . . . . 87

6.15 Dependency map, first step . . . . . . . . . . . . . . . . . . . . . . . . . 88

6.16 Dependency map, second step . . . . . . . . . . . . . . . . . . . . . . . . 89

6.17 Dependency map, third step . . . . . . . . . . . . . . . . . . . . . . . . . 90

6.18 Dependency map, fourth step . . . . . . . . . . . . . . . . . . . . . . . . 91

6.19 Completed dependency map . . . . . . . . . . . . . . . . . . . . . . . . . 92

6.20 Relation between ’Product Requirement Fullfilment’ and ’Timeline Met’ . 93

7.1 Result of analysis of all 18 Potential Success Factors . . . . . . . . . . . . 96

7.2 Square of all four Dominant Success Factors . . . . . . . . . . . . . . . . 97

7.3 Triangle of Dominant Success Factors which need to be balanced . . . . . 98

7.4 Role of Dominant and Supplementary Success Factors . . . . . . . . . . . 104

XII

XIII

List of Tables

2.1 Characteristics of empirical industry studies, adapted from (Blessing and

Chakrabarti,2009) .............................. 18

2.2 Journals represented in Mehalik’s and Schunn’s meta-analysis . . . . . . 30

2.3 Characteristics of the Comprehensive Approach . . . . . . . . . . . . . . 35

3.1 Paramteric vs. non-paramteric statistical tests (Burke, 1998) . . . . . . . 42

4.1 Rated importance of dimensions for measuring success (White and For-

tune,2002) .................................. 45

4.2 Changing success criteria with varying objective (de Wit, 1988) . . . . . 46

4.3 Quantification scale for primary dimension ’Product Requirement Fulfill-

ment’...................................... 48

5.1 Rating definitions for ’Use of Design Methodology’ . . . . . . . . . . . . 54

5.2 Rating definitions for ’Conceptual Design performed’ . . . . . . . . . . . 55

5.3 Rating definitions for ’Decision Tools utilized’ . . . . . . . . . . . . . . . 55

5.4 Rating definitions for ’Resources provided / Project Stability’ . . . . . . 56

5.5 Rating definitions for ’Priority of Project’ . . . . . . . . . . . . . . . . . 57

5.6 Rating definitions for ’Team Size’ . . . . . . . . . . . . . . . . . . . . . . 57

5.7 Rating definitions for ’Team Members Experience / Skills’ . . . . . . . . 58

5.8 Rating definitions for ’Cultural Influence in Team’ . . . . . . . . . . . . . 58

5.9 Rating definitions for ’Experience / Skills of Project Lead’ . . . . . . . . 59

5.10 Rating definitions for ’Project Lead Personality’ . . . . . . . . . . . . . . 60

5.11 Rating definitions for ’Level of Process Compliance’ . . . . . . . . . . . . 61

5.12 Rating definitions for ’Simultaneous Development’ . . . . . . . . . . . . . 61

5.13 Rating definitions for ’Product Requirements clearly defined and stable

duringDevelopment’ ............................. 62

5.14 Rating definitions for ’Awareness of Lessons Learned and State of the Art

TechnologyKnowledge’............................ 63

5.15 Rating definitions for ’Ratio of new Technology’ . . . . . . . . . . . . . . 63

5.16 Rating definitions for ’Technical Complexity of Project’ . . . . . . . . . . 64

5.17 Rating definitions for ’Co-Location of Team Members’ . . . . . . . . . . . 65

5.18 Rating definitions for ’Empowerment of Project Lead: Project and Budget

Responsibility in one Hand’ . . . . . . . . . . . . . . . . . . . . . . . . . 65

XIV

5.19 Amount of data points in each category for Potential Success Factors mea-

sured in three categories . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

5.20 Amount of data points in each category for Potential Success Factors mea-

suredintwocategories............................ 68

6.1 Significance of Potential Success Factors measured in three groups . . . . 78

6.2 Significance of Potential Success Factors measured in two groups . . . . . 81

6.3 Significance of secondary success dimension . . . . . . . . . . . . . . . . . 93

7.1 Cases for balancing the triangle of Dominant Success Factors . . . . . . . 100

XVI

List of Symbols

xiPotential Success Factor

YDesign Success

dTotal amount of data points required for an empirical study

mSample size of projects for an empirical study

H0Null-Hypothesis

µLow Mean of data sample group rated Low for a Potential Success Factor

µMedium Mean of data sample group rated Medium for a Potential Success Factor

µHigh Mean of data sample group rated High for a Potential Success Factor

SStotal Total sum of squares

SSwithin Sum of squares within the sample groups of a Potential Success Factor

SSbetween Sum of squares between the sample groups of a Potential Success Factor

yij Any data point in any data sample group

¯yiMean of any data sample group

df Degrees of freedom

dfwithin Degrees of freedom for SSwithin

dfbetween Degrees of freedom for SSbetween

MSwithin Mean of squares within the sample groups of a Potential Success Factor

MSbetween Mean of squares between the sample groups of a Potential Success Factor

FF ratio derived from variance analysis of means

pProbability value

HH ratio derived from variance analysis of ranks

niNumber of data points in the ith sample group

RiMean of the sum of the ranks in the ith sample group

RMean of the sum of the ranks of N

NTotal number of observations in all data sample groups combined

XVII

χ2Chi-Square distribution

OiObserved count for each compared sample group

EiExpected count for each compared sample group

XVIII

XIX

1 Motivation

Managing engineering development projects has evolved into an ever more complicated

task. The increased complexity of products and the global set-ups of companies challenge

designers and engineering managers to understand and stay focused on the factors that

contribute the most to successful product design. Engineering Design Research is the

field concerned with understanding what factors of the design process contribute more

to successful outcomes than others. While the subject was originally based on theories,

empirical studies have become increasingly involved (Ahmed, 2007) to help indentify these

factors by providing ’real world’ data.

Reviewing the current state of Empirical Engineering Design Research reveals that there

are areas that require more focus and provide room for improvement. These areas are

elaborated in the following Problem Statement. Througout the text, design projects and

development projects are synonymous.

1.1 Problem Statement

Many research activities about various isolated parts (Eppinger et al., 1994; Mehalik and

Schunn, 2006) of the product development process, such as the role of the individual

designer (Frankenberger et al., 1998) or human behaviour in general (Lindemann, 2003),

have taken place and produced fruitful results in Empirical Engineering Design Research.

However, investigating only partial aspects of the design process does neglect the consid-

eration of potential dependencies to other influencing factors (Ernst, 2002), which might

be more dominat. Such a scenario would mean that the studied factors were only ’symp-

toms’, which were caused to change by other factors not considered in the study. Only a

complete view on the design process, with all its influencing factors, provides this infor-

mation. Such comprehensive or holistic (Schregenberger, 1998) studies, viewing designers

from an integrated perspective within their complex organizational context, are rather

rare (Mehalik and Schunn, 2006; Schregenberger, 1998).

Another observation, which can be made on many existing empirical studies, is that a

judgement on the reliability of results cannot be made, due to limited amounts of data

and a lack of applying rigorous statistical techniques (Ernst, 2002). Oftentimes only a

handful - or even only one - development project is part of an empirical study. The

outcome of such studies does not permit for the generalization of results, which would

allow the definition of general measures for improving the success rates of product devel-

opment projects. It is the most challenging part for academia in the field of Empirical

Engineering Design Research, to obtain sufficient and reliable data from the industrial

domain (Blessing and Chakrabarti, 2009).

Lastly, it can be found that many empirical industry studies do not provide a clearly

defined theoretical framework (Horv´ath, 2004). A missing theoretical foundation results

in an increased risk for a lack of comprehensiveness and undetected causal relationships

among the influencing factors (Ernst, 2002).

In summary, the following three areas can be identified as lacking in the existing

Empirical Engineering Design Research studies:

•Comprehensiveness: Consideration of all the factors potentially influencing the out-

come of a product development project.

•Reliability of Results: Collection of sufficient data and use of appropriate statistical

techniques to identify significances.

•Theorectical Framework: A foundation to support comprehensiveness and the iden-

tification of causalities for the determination of the most significant and dominant

Success Factors.

This dissertation outlines an empirical research work in the industry, where these

three problem areas are addressed, which also marks its contribution to the current state

of the art research in the field. At first, a general approach is developed to support the

holistic investigation of Success Factors in industrial product development projects. This

approach is referred to as the Comprehensive Approach in this dissertation. A refined

set of necessary characteristics for the Comprehensive Approach is elaborated in the

Literature Review Chapter. Following the development, the Comprehensive Approach

is tested by applying it to a sample of product development projects performed in an

industrial context.

1.2 The Product Development Process

Engineering design is the process of mental creation of a product (Pahl et al., 2007),

which can then be physically realized in a subsequent manufacturing process. The need

for continuous mental creation derives from market and customer demands for ever more

user-friendly, cost-effective and environmental friendly products. Hence, product devel-

opment plays a vital role for society and its continuous advancement. Equally important,

it is essential for the economic health and competetive edge of technology firms. This be-

comes obvious when considering the fact that up to 75% percent of a product’s costs are

committed during - and especially in the early stages of - the design process, where most

of the mental creation occurs. Figure 1.1 shows this paradox of committed versus actually

incurred costs. It means that only 25% of the costs remain influenceable for reduction by

the downstream activities, such as product optimization and lean manufacturing. After

decades of focus on perfectionizing manufacturing processes, many companies have rec-

ognized the enormous potential for cost-effectiveness in the development process already.

Morgan and Liker discovered, from their investigation of Toyota’s Development Process,

that

there is only so much waste that you can squeeze out of production before

the engineering of the product and processes becomes a critical constraint.

Indeed, product and process development can have an even bigger impact on

lean enterprise than lean manufacturing. (Morgan and Liker, 2006, p.1)

Figure 1.1: Committed costs during product development process (Ullman, 2003)

Besides the ever more important focus on product costs itself, designers have to create

products that provide value for customers. Value reflects a customer’s ”desire to retain

or obtain a product” (Neap and Celik, 1999, p.181). Such a desire exists if customers

perceive that costs, usability, quality and aesthetic appeal (for consumer products) are in

an acceptable relation for them. Design teams have to consider all of these areas, combine

them and translate them into physical solutions. This broad range of boundaries and

requirements turn the product development process into a multi-faceted activity, starting

with abstract, creative stages and converging towards concrete, science-based solutions.

During this activity, design teams are confronted not only with the challenge of designing

products that provide value to customers, but also have to keep an eye on external

factors, such as stakeholders within the company, legal requirements or environmental

constraints. Due to this complexity, it has shown to be beneficial to organize this activity

systematically into subsequent steps, which became known as the product development

process. Different schools in defining this process have evolved. The overall goal, however,

has always been the same, to provide design engineers with a scheme on how to organize

the development activity in order to increase the chance of a successful outcome.

1.2.1 Systematic Approach

Pahl and Beitz were one of the first to study a systematic organization of the development

process, starting in the early second half of the 20th century. In their classic work

’Engineering Design: A Systematic Approach’ (Pahl et al., 2007), they propose splitting

the product development activity into four main phases, which they refer to as:

1. Product planning and clarifying the task

2. Conceptual design

3. Embodiment design

4. Detail design

Figure 1.2 shows a process map of the Systematice Approach. In the first phase, they

recommend clearly determining and specifying the task and the desired outcome of the

development project. For new product development, this starts with market analysis, but

also with an evaluation of the company situation, to ensure the resources and knowledge

are available. The customer and product requirements are then defined in a requirements

list, which is nowadays in most companies known as the Produt Design Specification

(PDS). The completed list, agreed on by the stakeholders of the development project,

marks the end of the first phase and starts off the second phase. Conceptual design is a

highly creative activity, where virtual requirements are translated into principle solutions.

Since this is the time in the development process where the highest impact on cost,

usability, quality and innovation can be made, the ideas that engineers produce in this

phase are an important driver for a product’s and company’s success. Being aware of this

importance, Pahl and Beitz aimed to provide a systematic that would guide designers in

this phase, without hampering their creativity. This is done by abstracting the problem

into function structures for the essential problems. Then ideas are generated to solve these

problems of sub-functions, using different types of creativity techniques. The solutions

for the sub-functions are then evaluated and combined into concept variants. At the end

of the conceptual design phase, the most promising concept variants remain, which are

subjected to a more detailed evaluation in the embodiment design phase. The third phase

starts with studying the feasibility of the concept variants. First calculations and layouts,

if necessary tests, are carried out to determine if the solutions, which were found in the

conceptual phase, are technically feasible. An economic assessment, as well as any other

external boundaries (e.g. legal requirements), needs to be part of this evaluation, so that

the development team can determine the go-forward concept variant to be developed into

a detailed design solution in the fourth phase. The objective of the detail design phase is to

finalize the design and create all the necessary documentation, allowing manufacturing to

transform the mental creation into a physical product. Manufacturing is recommended to

be involved from the embodiment design phase at the latest to initiate the manufacturing

process development in parallel, so that production can start with the completion of the

design process.

On a more detailed level, each of the four phases consists of different stages. Pahl and

Beitz recommend following these stages in sequence and provided various methods to

support the designer in these stages. In a simulated empirical study in the lab, Pahl

gave a design problem to engineers applying the systematic approach and engineers who

did not. His conclusion was that, although it takes them longer to come up with initial

solutions, engineers using the methodology were able to design a better product in an

overall shorter development cycle (Pahl, 1992).

The Systematic Approach is best known and applied in Germany and Central Europe.

1.2.2 Quality Function Deployment

Quality Function Deployment (QFD) was developed in Japan by Akao (Akao, 1990) in

the 1960’s. It found its first successful application in the Mitsubishi Heavy Industries

Kobe Shipyard in 1972 and has since been globally known and applied. The method

is an adaption of Total Quality Management Tools (TQM) (Cohen, 1995) and aims to

provide a structure for guiding the product development process from the customer voice

to the control of production. It spans wide into both areas, before the actual design

work and after. The tools used are quality tables, which are also known as Houses of

Quality. The process is organized into four main phases, each systematically elaborated

with one or more quality tables. In the first phase, the customer needs and desires are

determined and translated into specific product requirements. After that, functional

solutions are elaborated for these requirements in the second phase. In the third phase,

the functional solutions are translated into actual design components and processes are

developed. Finally, the design is transitioned into production, where the last phase is

also used to control production processes. Figure 1.3 shows the four phases of QFD.

Figure 1.2: Process map of the Systematice Approach by Pahl and Beitz (2007)

Figure 1.3: The four phases of Quality Function Deployment (Yang et al., 2003)

1.2.3 Design for Six Sigma

The Motorola Company came up with Six Sigma in the 1980’s, aiming to improve the

quality of its products significantly. Many companies, especially in North America,

adapted this methodology. The name is derived from statistics nomenclature, where

sigma stands for standard deviations of a normal distribution of data points. Being able

to control six standard deviations would mean that a process produced only 3.4 defects per

one million possibilities, which is the goal when applying Six Sigma. The core approach

of the methodology is to identify cause-effect relationships of defects quantitatively. The

central parameter is variation. If the varying parts of a process can be detected with help

of statistical tools, then measures can be implemtented to reduce the variation and in

that way the process becomes more robust. The typical Six Sigma process improvement

consists of a five step process: Define, Measure, Analyze, Improve and Control (DMAIC).

While the DMAIC process proved to be suitable for error correction and waste reduction

in processes, it was recognized that in order to sustain almost defect-free quality, focus

needed to shift upstream to the development process as well (Brue and Launsby, 2003).

This lead to the development of Design for Six Sixma (DFSS), a derivative of the Six

Sigma methodology with its own process. DFSS is not so much an invention of new

design methods, but a ’tool box’ of mostly established design and quality tools, including

parts of QFD and TQM. DFSS represents a process that provides design teams with the

guidance of using recommended methods at the right time in the design phase, similar

to the intend of Pahl’s and Beitz’ Systematic Approach. Different styles of DFSS have

evolved over time, depending on the specific design task. The most popular DFSS style

is the IDOV (Identify, Design, Optimize and Validate) process. Similar to the Systematic

Approach and QFD, it suggests splitting the design process into four phases:

1. Identify: Analyze the market and determine customers CTQs (Critical To Quality

requirements).

2. Design: Translate CTQs into alternative concept solutions.

3. Optimize: Use statistical and simulation tools to predict performance of concepts.

Select most robust design solution that meets CTQs.

4. Validate: Finalize design and verify that completed design meets customers ex-

pectations.

The aforementioned three design methodologies are examples of established approaches

which have been developed in Germany, Japan and North America, respectively. More

schools have evolved over the last half century. To mention is Pugh’s (1991) approach

Total Design, or Ullman’s (2003) book The Mechanical Design Process. Ehrlenspiel

propagates an intergrated approach (Ehrlenspiel, 2009), emphasizing the importance of

collaboration between engineering, manufacturing, sales, procurement and controlling

throughout the development process. His approach includes elements from Simultaneous

Engineering, Quality Function Deployment and Target Costing.

Although the authors use different terminologies and propose varying tools for certain

tasks in the development process, they all have in common that they recommend splitting

the design process into four phases. Starting with a clear definition of the product require-

ments, it is demanded to maximize the solution space by generating as many alternative

solution concepts as possible in the next steps. These concepts are then evaluated and

systematically down-selected, so that in the last phase, the most suitable design solution

remains and is finalized. Figure 1.4 shows this concept graphically using the terminology

from the Systematic Approach.

Figure 1.4: Common layout of design methodologies

For most companies, the completion of a new product development does not end the

engineering process. After successful completion of the development, products need to be

monitored and maintained, if necessary upgraded over time, until their discontinuation.

For this reason, the product development process is oftentimes embedded into the product

life cycle process, which spans from the idea of a new product until its disposal. Figure 1.5

shows this relationship.

Figure 1.5: Role of the development process in the product life cycle

1.3 Factors influencing Design Success

The central function of the product development process in the life cycle of a product

and the above mentioned complexity of the development activity itself, give an indication

of the multifold of factors to be considered that potentially influence the outcome.

Dixon locates the work of design teams in the center of two intersecting streams (Fig-

ure 1.6), one being technical, the other cultural. On the technical side, design engineers

need a broad knowledge of the natural sciences, engineering sciences and manufacturing

technologies. Their solid technical background and experience are important factors in

accomplishing their work successfully. While these influencing factors are more related to

and learnable by the individual within the engineering domain, the cultural factors result

mainly from external or less controllable influences. An artistic and creative mindset is

likely to be more of a personality trait than learnable. The economic situation might be

dictated by customers, competitors or management targets and support. Sociological fac-

tors derive from the central position the engineering department has within a company,

but also from the interaction of globally distributed design teams, where many engineers

ﬁnd themselves these days. Political factors might be speciﬁc to the product and its

exposure to society, but also refer to legal requirements, e.g. environmental or customs

regulations.

All these cultural and technical factors have an impact on the successful outcome of a

development project. But they do not do that independently. At least some of them are

related, likely through causal dependencies.

Figure 1.6: The central activity of engineering design (Dixon, 1966)

1.4 Engineering Design Research

While the classic engineering sciences - mechanics, dynamics, materials, thermodynamics

etc. - have a history back to the 18th century, Engineering Design Research is a fairly

young scientiﬁc ﬁeld, which developed from the middle of the 20th century. With the in-

creased complexity of engineering products and organizational structures, it evolved as a

science with its own body of language, ”related but not identical to other sciences” (Bless-

ing and Chakrabarti, 2009, p.3). Unlike most other sciences, it is rather diﬃcult to draw

the boundaries to other research ﬁelds. The aforementioned, multifolded inﬂuencing fac-

tors, result in a fuzzy area of investigation (Horv´ath, 2004). Figure 1.7 shows the current

scope of interest in Engineering Design Research, according to Horv´arth.

It is the ultimate goal of Engineering Design Research, to gain a better understanding of

the design process in all its complexity and to develop and validate methods to improve

the current situation in design (Blessing, 2002).

Since the beneﬁciaries of study results are companies in the private sector, Engineer-

ing Desing Research eﬀorts rely on collaboration with the industry. This circumstance,

paired with the broad scope of inﬂuencing factors, from the arts to the ’hard’ sciences,

leads to challenges which are very unique to this research ﬁeld. Eckert et al. describe

these challenges as:

subject to tensions between conﬂicting needs and goals:

•between the need for valid, well-grounded research results, and the need

for industry supported research to have immediate practical applications;

•between the academic need to produce reportable results quickly from

projects with limited resources, and the industrial need for powerful,

reliable, validated tools and techniques;

•between the need for large research groups to exploit their resources to

make major advances, and the need to allow isolated researchers to make

eﬀective contributions; item between the need for students to achieve

intellectual independence in their own research, and research leaders to

achieve larger-scale, longer-term results;

•between the need for students to develop skills in diﬀerent aspects of

applied research and their need to focus to achieve results in a reasonable

time. (Eckert et al., 2003)

Figure 1.7: Area of interest in Engineering Design Research (Horv´ath, 2004)

Aurisicchio and Wallace investigated the competencies and conditions required for

conducting empirical design research successfully. They concluded

that for a research project to be successful, a researcher has to have knowledge

of the design domain under investigation, the topic of study, the research

process and the collaborating company. (Aurisicchio and Wallace, 2007, p.397)

The author of this dissertation spent five years as a design engineer and technical

project lead in the collaborating company’s engineering department, prior to conducting

this research.

It has to be distinguished between different types of Engineering Design Research

studies. The difference is defined by the characteristics researchers have selected. Some

characteristics can be freely chosen, while others are forced upon the researcher by con-

straints, such as available object to be observed, amount of samples to be studied, or

time frame available for the research project. The characteristics of Empirical Engineer-

ing Design Research studies are elaborated in detail in the Literature Review Chapter.

1.5 Research Questions

The motivation for the research effort described in this dissertation can be summarized

in the following three research questions:

•Research Question I: How do the characteristics of an Empirical Engineering Design

Research study need to be set to obtain comprehensiveness?

•Research Question II: Which of all the influencing factors can be proven to have a

significant and dominant influence on Design Success?

•Research Question III: What can engineering management do to increase success

rates of development projects?

The answer to question one provides the framework for the Comprehensive Approach.

Both, the first and the second answer are contributions to the state of the art in Engineer-

ing Design Research. They provide proposals for complementing current practices with

a theorectical framework, which allows for a more rigorous focus on comprehensivenes

and reliability of results. From the answer to the second question, recommendations for

future research activities are derived. The answer to the third question will be particu-

larly beneficial to engineering managers working in the industry, by providing concrete

measures on how to increase the successful outcome of product development projects.

1.6 Research Object and Observed Process

For the empirical study, the research object was defined by the company and the product

under investigation. The company is a global player in the field of energy systems, spe-

cialized in large scale power generation equipment. The product studied are Heavy Duty

Gas Turbines, which are utilized in power plants. The gas turbine business employs close

to 4000 people, about a third of these are employed in engineering functions.

Different models of gas turbines are in the company’s product portfolio, covering a power

output range between 5-375MW. Besides the casing and the rotor, gas turbines consist

of three major components: compressor, combustion system and turbine (Figure 1.8).

Stationary gas turbines have become a significant part of the energy mix in many coun-

tries over the last 20 years. Their low emission levels, high efficiency, fuel and opera-

tional flexibility have been recognized as beneficial; especially in the developed countries,

where power companies have to fulfill ever more stringent emissions requirements, face

higher fuel costs and increasinlgy inbalanced power grids, caused by a growing portion

of renewable energy sources. This demand lead to enormous research efforts to improve

efficiencies, reduce emission levels and extend the life times of components. While the

largest gas turbine in the world had a power output of around 100MW only 25 years

ago, it is now close to 400MW. This rapid and ongoing increase is only possible by con-

tinuously increasing turbine inlet temperatures, which require advancements, especially

in materials, cooling and combustion technologies. For this reason, the combustion and

turbine component are in the focus of research and development activities and experi-

ence a high rate of innovation and increasing complexity. The scope of this empirical

study was on development projects within these two components. In total 50 projects of

varying complexity and innovative content were analyzed, where 44 remained at the end,

providing valid data for the quantitative analysis. Out of these 44, 21 were combustion

system, 23 turbine component development projects.

Figure 1.8: Main components of a gas turbine (Siemens, 2013)

The observed process was deﬁned by the portion of the product life cycle (see Fig-

ure 1.5) which was investigated. The company has developed its own product develope-

ment process. Figure 1.9 shows the process, with a total of nine phases (green arrow

shaped boxes) and eleven review gates (R0-R10). The phases are organized in ﬁve clus-

ters (grey boxes). Due to the complexity of the products and the associated stretched

testing and validation intervals, it extends far beyond the actual design process itself.

This speciﬁc product development process could almost be seen as covering the complete

product life cycle. However, this is not the case, as eventually the developed and sold

products go into ownership of the company’s service business, which then oﬀers mainte-

nance and upgrades for several more decades.

Although organized and grouped a little diﬀerent between the clusters and particular

phases, the four phases which most design methodologies have in common can still be

recognized. They are referred to as: Strategic Product Planing (ﬁrst cluster), Concep-

tual Design (ﬁrst phase in second cluster), Basic Design (second phase in second cluster),

Final Design (ﬁrst phase in fourth cluster). These are complemented by separate Sales

Preparation (after the Basic Design Phase), Procurement, Manufacturing & Assembly

and Validation Phases, after the design has been ﬁnalized. The observed process of this

study spans from after the Product Planning Phase (past R1 Review) until completion

of commissioning and trial operation. Strategic Product Planing is excluded, as it is

driven by the marketing and strategy department. At the end of this cluster, a Product

Development Speciﬁcation (PDS) has been developed for a new generation gas turbine.

Speciﬁc product requirements for the components have to be derived from this PDS in

the Conceptual Design Phase of each component development project.

Figure 1.9: Product Development Process in company of study

Collected Data

The data for the influencing factors was collected for the time between the beginning of

Conceptual Design and completion of Final Design (R5 Review) for each project. The

success measurement for each development project was collected at the time of completion

of the Erection, Installation, Commissioning & Trial Operation Phase. In that way each

developed component in the study has gone through the manufacturing process and

the initial testing period. This end of the observed process was chosen to allow equal

comparison between each of the 44 projects, as some had just passed this phase (R8

Review), while other developed components already had years of operating experience.

Variation in Process

The company specific product development process provides a guideline that does not

apply to each project with all its phases. While the development of a new product

generation demands that each process step is followed, it is not mandatory for component

development projects, which happen more frequently as part of product upgrades over

their life cycle. This flexibility is intentionally provided to reduce administrative effort

and hampering of design teams. Design teams may chose to pass through all design

review gates (R2+R3+R5), only two of them (R2+R5 or R3+R5), or hold only the last

and always mandatory design review (R5 Review), weighing risk against justified effort.

The resulting variability in the process is considered in the study as being one of the

influencing factors.

1.7 Dissertation outline

After a brief historic introduction of Engineering Design Research, past research efforts

are analyzed and characteristics of comprehensive studies elaborated in Chapter 2. The

theoretical framework of the Comprehensive Approach is illustrated in Chapter 3. Def-

inition and measurement of Design Success are explained in Chapter 4, including the

collection of the data. This is similarly shown for the Potential Success Factors in Chap-

ter 5, including the validation of the collected data. Chapter 6 is concerned with the

introduction of the analysis methods and data anlysis itself, where the results are inter-

preted in this chapter as well. The interpreted results are reviewed in Chapter 7, where

their applicability is discussed and recommendations to engineering management, as well

as for further research activities, are provided. Chapter 8 summarizes the work of this

dissertation.

2 Literature Review and Research

Needs

Designing products has a long history in the different fields of engineering. But it was

not until ”well into the second half of the 20th century” (Blessing and Chakrabarti, 2009,

p.2) that researchers would become interested in systematically investigating the prod-

uct development process itself. At first, phases of experiential and intellectual design

research dominated the research activities. In the experiential phase, senior developers

documented their experiences in product design. At that time, there was no link to a

theoretical framework or methodology, which later became the interest of researchers in

the intellectual phase. In the 1980s, the focus shifted slowly to empirical studies (Wallace

and Blessing, 2000). In the following decade, a series of experiments were performed -

mainly in the laboratory with small sample sizes of professional designers or engineer-

ing students - with the aim to understand the role of the designer in the development

process. Ullmann, Dietterich and Stauffer (1988) developed a model of the mechanical

design process by analyzing audio and video protocols of the design work of five me-

chanical designers. Dylla (1991) investigated design strategies of individual designers in

the early development stages. Pahl (1992) studied the traits of good problem solvers

by observing designers of different skill sets and educational backgrounds. Fricke (1996)

observed designers and analyzed their work to find the factors for best success in the

design process.

Many more of these types of studies were performed in this period, at the end of 20th

century. Their results contributed greatly in understanding the product development

process, especially the role of the individual designer. However, as these studies were

conducted in the laboratory under simulated conditions, they missed to view the engi-

neers wihtin their organizational context. Impacts of influencing factors resulting from

the environment in an industrial company, where engineers actually work, could not be

verified. Subrahmanian described the sparseness and need of studies on a higher level, as

”design very seldom takes place in actual practice only at individual levels” (Subrahma-

nian, 1992, p.4). Schregenberger pointed out, that in order to understand the complete

product development process, comprehensive or holistic (Schregenberger, 1998) studies,

would have to be carried out within an industrial environment. Indeed, since the 1980s,

an increased interest in empirical studies with ’real world’ data can be observed (Ahmed,

2007). Cantamessa (2003) found that industrial involvement is benefical to empirical

design research, but also indicated a lack of research methodology, which can be found

quite frequently in empirical projects. Figure 2.1 shows a high level timeline of the phases

in Engineering Design Research. It has to be noted that the phases cannot be clearly

separated, but that they rather overlap chronologically. As aforementionted, the em-

pirical studies can be divided into two main categories, referring to their environments:

laboratory versus industrial. Due to their reflection of ’real world’ situations and data,

industrial studies - being the most complex kind - have the potential to deliver the most

reliable results on success factors of product development projects (Ahmed, 2007; Auris-

icchio and Wallace, 2007; Schregenberger, 1998; Subrahmanian, 1992). The scope of this

literature review and the described research work focuses on this category.

In the following, an overview of Empirical Engineering Design Research studies performed

in the industry, is given. They are referred to as empirical industry studies. The dif-

ferent types of empirical industry studies with their characteristics are introduced. It

is illustrated, what results were obtained by past studies. The research approaches and

methodological frameworks of these efforts are discussed, revealing the gaps and areas of

improvement, which were addressed in this research work.

Intellectual Phase

Experiential

Phase Empirical Phase

Laboratory Industrial

1960 1980 2000

Figure 2.1: High level timeline of Engineering Design Research

2.1 Characteristics of Empirical Industry Studies

Different types of empirical industry studies have to be distinguished. The distinction de-

rives from the characteristics that are selected for the type of study. Some characteristics

are given to the researcher by constraints such as: access to data, available resources or

timeline of the study. Table 2.1 gives an overview of the most significant characteristics,

which distinguish the different types of empirical industry studies from each other. The

table is adapted from Blessing and Chakrabarti (2009), in that only the characteristics

significant for empirical industry studies are listed. Characteristics needed for simulated

studies (role of reseearcher, time constraint, duration, setting, findings and notes) are not

shown in the table and are not part of the later discussuion.

Table 2.1: Characteristics of empirical industry studies, adapted from (Blessing and

Chakrabarti, 2009)

Characteristic Options

Aim, research question, hypothesis The aim of the research project and of the study,

main research question and hypothesis, Success

Criteria and/or Measured Success Criteria.

Study process (nature of study) Controlled versus natural.

Theoretical basis Paradigms, methodologies, theories, views, as-

sumptions that guided the researcher.

Unit(s) of analysis The element(s) for which findings are reported

and about which to draw conclusions that are

intended to be generalized.

Data collection method The method(s) used, such as direct observation

using video, participant observation, diary keep-

ing, archival research, questionnaire, interview.

Observed process Starting point and required deliverables of the

observed process: e.g., specification as starting

point, layout drawing, prototype or product as

deliverable.

Task Type and complexity of task.

Number of cases Number of data sets collected.

Case size Number of persons, product elements, employ-

ees, etc., within each case.

Participants Level and type of experience, background, size of

organisation, etc.

Object Description of the design object, company,

project or documents analyzed.

Coding and analysis method(s) Methods used to process, code and analyze the

data, e.g., use of pre-determined coding schemes

or not, and statistics applied.

Verification method(s) Methods used to verify the results.

Blessing and Chakrabarti refer to characteristics also as dimensions. Since the char-

acteristics of studies are mainly independent, the terminology ’characteristics’ is used

throughout this dissertation instead of dimensions. This is also to avoid confusion with

the dimensions of success defined later on. Aurisicchio and Wallace (2007) also refer to

the characteristics defined by Blessing et al. (1998), but propose - instead of what is

defined as nature of study by Blessing and Chakrabarti (2009) - to add the characteristic

process, which can be either controlled or natural. It is referred to as study process in the

following, to avoid confusion with the characteristic observed process.

Understanding the characteristics supports the identification of areas of improvement on

existing empirical industry studies. Each characteristic is briefly discussed in terms of

how their selection impacts the outcome of a study.

Aim, research question, hypothesis: The aim of the study is the first characteristic

to be defined. It reflects the motivation and need for the research effort. By formulating

the research question, some of the other characteristics are determined and the type of

study is being narrowed down. The domain of intererst - which is usually the develop-

ment department in Engineering Design Research - is known with the aim, as well as the

unit of analysis within the domain.

Study process (nature of study): The two options are controlled or natural. A study

process is controlled when one ore more variables are held constant. Are all variables al-

lowed to vary freely, it is natural (Aurisicchio and Wallace, 2007). In a controlled study,

researchers adjust the natural situation in order to be able to focus on certain aspects.

As a result of the created constraints, they likely influence the results - consciously or

subconsciously. Examples for such studies are researchers attending a certain stage in

the development process and asking questions, or video taping the work. The influence

from the adjustment of the situation does not exist in a natural study. Here the re-

searchers stay in the background and only observe and investigate ’real world’ events.

Examples for natural studies are document research or interviewing engineers after they

completed a design project. Natural empirical industry studies are the most complex

kind of study (Aurisicchio and Wallace, 2007). The data collection and validation pro-

cess can turn into an extensive enterprise. But if valid data in sufficient quantity can be

obtained, high quality results - reflecting reality in engineering design - can be expected.

As discussed later in section 4.3 and 5.3, valid data exists if the sample size shows a

representative distribution of the population and if sufficient data points exist for the

categories of each influencing factor, in order to be able to apply statistical methods.

Theoretical basis: A theoretical framework is necessary for any kind of research work.

It provides guidance during the elaborating process and helps to ensure that the steps

of a study converge towards the desired aim, which has been defined at the beginning.

With a defined theoretical framework, it becomes obvious what kind of data has to be

obtained. Defining the theoretical framework includes deciding which methods should

be utilized, but also assumptions that need to be made. Assumptions might have to be

redefined as a study progresses.

Unit(s) of analysis: The unit of analysis directly derives from the research question.

It can vary from a small, detailed scale to a large scale. Examples of a unit of analysis

(listed with increasing scale and complexity) are: behaviour of the individual designer,

project teams, design phases, product development process. Defining the unit of analysis

is a condition to begin with and already determines the options for the method of data

collection. Many studies found in the literature focus on units of analysis of a smaller

scale, such as the behaviour of individual designers (Frankenberger et al., 1998; Linde-

mann, 2003; Pahl, 1992) or design teams (Frankenberger and Auer, 1997). They have

contributed greatly in understanding these units. The high frequency of studies on these

smaller scale units of analysis can be explained by their ability to compromise on con-

straints, such as limited research windows or access to industry data. Empirical industry

studies, focusing on the overall product development process in a holistic manner, can

rarerly be found (Mehalik and Schunn, 2006; Schregenberger, 1998).

Data collection method: Many methods of data collection are possible. Some exam-

ples are: observation, interview, questionnaires, archive or product analysis. By having

defined the characteristics research aim, study process, theoretical framework and unit

of analysis, the usable methods for data collection are already narrowed down. The goal

of a study needs to be to gather as much data as possible to allow for a generalization of

results. On the other hand, bias needs to be kept to a minimum. The latter criteria calls

for a data collection method where the researcher does not interact with the participants

of the observed process, as can be done by researching archives or completed projects.

However, significant amounts of data can best be obtained from direct interaction with

participants, e.g. through questionnaires. In general, a trade off between the amount

of data and bias has to be found. Some studies use a combination of data collection

methods.

Observed process: Although not declared specifically in all studies found in literature,

this characteristic is important to know. It indicates the validity of the results with re-

spect to the domain and the aim defined at the beginning. An observed proccess can be

as detailed as a single stage in the development process, for instance idea generation in

the conceptual design phase. But it can also be from the product requirement specifica-

tion until completion of validation or even consider the complete product life cycle. The

researcher has to clearly state what the observed process of the study is, in order to bring

the results into perspective.

Task: The task being studied has an influence on the generality to be interpreted from

the results. If the task is specific to a certain industry, the results might only be represen-

tative for this specific area. The complexity of the task, but also the innovative content

of the task are important to know. A task might be complex, but have little problem

solving or unknown technology involved. The outcome might still be successful, even if

some success factors were not met. Ideally, design tasks with varying complexity and

amount of new technology are investigated for understanding how these variables relate

to the outcome, but also how they depend on other factors influencing Design Success.

Therefor, including a sufficient number of cases into the scope of a study is essential.

Number of cases: An important characteristic which determines the potential for gen-

eralization and the ability to verify the results. The more cases can be investigated, the

better applicable become quantitative methods, such as descriptive statistics (used to

describe the sample size being studied), or ideally inferential statistics (used to extend

the studied sample size to make universal projections on the population). Here is where

many studies are limited by a lack of access to industry data.

Case size: The amount of employees, departments or interfaces involved in the task. As

with the Task, this characteristic should be evaluated for the relationship to other factors

influencing the design process. In order to do so, a sufficient number of cases is necessary.

Participants: Depending on the research question and the unit of analysis, information

about the participants needs to be known. This information can be objectively measur-

able data, such as years of experience or educational background. But it can also be more

complex, on an abstract level, especially when it comes to traits related to personality.

Again, in an ideal study, sufficient and varying data about the participants is collected

and analyzed. The participants of the study (e.g. the person answering a questionnaire)

might be different than the participants of the observed process.

Object: The object brings the unit of analysis, observed process and task into perspec-

tive. Knowing the product, company or area of industry, supports the determination

of how general the results can be considered. Depending on the research question, the

validity of the results depend upon this characteristic. For instance, if the aim of a study

is the understanding of the design process in a global environment, the size and global

set-up of the company studied has to be known.

Coding and analysis method(s): In order to remove as much bias as possible from

a study, quantitative data should be the preferred type of data. Some factors are easier

to quantify than others. For instance, project durations, budgets, team sizes etc., can be

obtained from archives in a straightforward, objective manner. Others, especially when

it comes to investigating the human side, are more difficult to define objectively. In these

areas, researchers can utilize methods for quantification from other academic fields, for

instance the social sciences, where such tools have been developed due to the need.

Verification method(s): The use of advanced (inferential) statistical methods provides

tools for the verification of results, such as significance levels and probabilities of repeata-

bility. The determination of dependencies between factors of significance can serve as an

additional method for verification of results. In general, verification should not be re-

duced to analysis of results, but also considered in earlier steps. After all, statistical

analysis methods strictly follow the rule of ’garbage in, garbage out’. The quality of

the data input needs to be assured upstream, from the very beginning of the research

work. Verification means having sound methods for research set-up, research execution

and post-processing.

2.2 Review of different Types of performed Studies

A multifold of empirical industry studies can be found in the Engineering Design Research

literature within the last three decades. The majority of them were conducted through

questionnaires and surveys, due to the ability of obtaining large amounts of data in a

fairly straightforward manner. Giving a complete overview within this literature review

is impossible. Hence, four different studies are introduced, representing common types of

studies with varying characteristics. The pros and cons, in terms of quality and reliability

of the results, are discussed for each. Through that, preferable characteristics for a

comprehensive empirical industry study are elaborated.

2.2.1 Hales: Analysis of the engineering design process in an

industrial context

Hales (1987) conducted one of the first empirical studies in the industry over a longer

term. The object of this study was the development of a coal gasification system, with up

to 37 people being involved into the development process. The observed process spanned

over 36 month, from the initial planning stage until near completion of the project, when

the company decided to stop the project. Aim of this single case study was the under-

standing of factors influencing the design process, in particular with respect to the use of

a systematic design approach, which was introduced for this development project. The

Systematic Approach (Pahl et al., 2007) provided the theoretical framework, but also

determined the units of analysis, which were the stages of the design process. Due to

this intervention - controlling the design methodology to be utilized - into the work of

the designers, the study process was controlled. The data was collected on a weekly basis

through observation, audio taping, diary notes and design reports. Main findings of the

study were: a list of factors likely to influence the design process; the design process

follows overlapping phases, which, if not forecasted and managed properly, will lead to

cost and time overrun; and less than a quarter of the overall engineering time was spent

on the use of methodological techniques.

Pros of this study type: The adavantage and contribution of Hale’s study is the observa-

tion of a large development project in a ’real’ environment over almost all stages of the

development process. The different types of data, collected on a weekly basis from many

participants, provided a large amount of information to be evaluated. In general, bias

can be reduced by collecting data form many participants rather than a few or even one.

Cons of this study type: The fact that this research was based on a single case, raises

the question on how general these results can be recognized. From a statistical point of

view, a sample size of one does not allow for determining the confidence of the result. It

cannot be known, if all the factors identified to influence the design process do so in a

general manner and could hence repeatedly be observed on other development projects.

The results might just be specific to the studied object and people involved. In addition,

the observed process did not include the building and testing of the product. The ques-

tion, if the use of a systematic approach lead to an adequate product, which fulfilled the

expected product requirements, could not be answered.

2.2.2 Ehrlenspiel: Problems in Development and Design

This survey-based study was performed by Ehrlenspiel (2009) in 1991/92. In total, 300

participants with a design or engineering management background were asked for their

feedback. The participants all worked in engineering companies with 500-2000 employees,

so the object of the study were medium sized companies. Aim of the study was to

understand the role of issues in product development and their resulting negative impact

on the business as a whole. A set of potential problem factors - grouped in three problem

areas: organizational; process related; technical-economical - was provided in the survey.

The participants were asked to rank them on a scale from 0 to 4. The results of the study

are shown in Figure 2.2. Only the mean value is reported for each pre-defined, potential

problem factor.

This type of study is a natural study in that participants were not influenced in their

actual design work. The theoretical basis goes back to assumptions on what problem

factors of the development process can potentially be. The unit of analysis was the

complete product life cycle. The task and complexity of the cases is not known.

From the representation of the results, the author identified cost issues, time constraints,

unclear definition of objectives and the control of the design process as the most significant

problem factors. What is interesting about the factors cost issues and time constraints,

is, that these are effects rather than causes. There must have been specific reasons,

which must have existed before and eventually lead to time and cost issues. One can

think of a lack of simultaneous development, or low management support, just to give

two examples of what can be found in literature, to cause missed deadlines and budget

overruns in development projects.

Figure 2.2: Result of Ehrlenspiel’s survey study, translated from (Ehrlenspiel, 2009)

Pros of this study type: A sufficient amount of data was collected in this study (300

data sets from participants of different engineering companies), allowing for the use of

quantitative analysis methods. Due to that, the results can be considered as being general

in nature and representative, at least for medium sized companies.

Cons of this study type: Only a most basic statistical evaluation - the mean value for each

problem factor - is presented in the result summary (Figure 2.2) of the study. With only

the mean value being known, the confidence of the results cannot be predicted. A high

variance in distribution in the answers could still exist for each individual problem factor,

which could, for instance, go back to the different industries where the participants were

from. Illustrating the data distribution in addition to the mean, as an example, would

give a much clearer picture about the reliability of the survey result. Another source of

inconsistency is in the way the survey was designed. The problem areas and problem

factors were given to the participants. It is not explained how these were determined and

where they came from. There might be other influencing factors on the design process,

which were not asked, hence the comprehensiveness of the study is to be questioned. It

can also be recognized, that some of the asked for problem factors are causes while others

are effects, resulting from causes. As discussed before, cost and time issues are more likely

to be symptoms, triggered by other factors. These circumstances raise questions about

the soundness of the theoretical framework and the assumptions made for the study,

which can frequently be observed in such types of studies (Cantamessa, 2003).

2.2.3 White and Fortune: Current practice in project manage-

ment - an empirical study

White and Fortune (2002) conducted a survey for determining influencing factors on

project management. Aim of the study was not just the determination of the factors that

are most critical to project success, but also what methods and tools are most successful

in supporting the project management process. The questionnaire was designed in a way

in that potential success factors were given for choice, but also in that the particiants were

asked to add factors they thought are most important for project success. In addition,

the participants were asked to list disturbances and side-effects they experienced during

project work. The mixture of questions resulted in data that allowed for answering several

research questions. The unit of analysis of this study was the project management process

from kick-off to project completion. 236 valid data sets were collected. Different industries

were represented in the study. Two thirds of the participants worked for companies with

1000 employees or more, the rest for companies with 100-1000 employees. The study

process was natural in that the participants were not controlled in their actual project

management activity. A theoretical framework existed in the methodological way the

questionnaire was designed to allow for answering different research questions. In the first

step the participants were asked to list the three most important dimensions to measure

success. As was to be expected, these were answered as: meet client’s requirements,

completed within schedule, completed within budget. What is interesting in this study,

is that 41% of the studies were reported as being finished completely successfully. 85% of

the projects were ranked with a six or seven on a 1-7 scale, as shown in Figure 2.3. This

unsually high success rate - in contrast to findings of many surveys in literature (White

and Fortune, 2002) - is likely related to bias, as 82% of the participants reported to have

managed the projects themselves.

The three factors found by White and Fortune to have the most significant impact on

the project success were:

1. Clear goals/objectives

2. Support from senior management

3. Adequate funds/resources

Figure 2.3: Success reported in White and Fortune’s empirical study

Pros of this study type: As with many survey studies, enough data was collected to

allow for the use of quantitative evaluation methods. The questionnaire did not only pro-

vide pre-determined questions, but also open questions. Open questions were, when the

participants were asked to describe any other influecing factors they can think of. This

helped in identifiying factors the researchers might have not thought of when preparing

the survey.

Cons of this study type: The success rate was reported very high for almost all of the

projects. These positive results are likely to be linked to the fact that 82% of the partici-

pants were the project managers themselves. Bias apparently plays a significant role with

this method of data collection. Another problem, with only highly successful projects

being reported, is, that it cannot be proven what really distinguishes successful from less

successful projects. The success factors identified through the survey were believed to

have contributed to the project success. However, if less successful projects lacked these

exact factors is not known, as little is known about projects with low success from the

survey results. Mehalik and Schunn (2006) have found in a meta-analysis of 40 empirical

studies that, although factors influencing success are extensively researched, few studies

have examined what impacts the design process negatively. What is also not shown, is

how the factors found are related to each other. For instance, it would be of interest to

know how and in what direction the success factor ’clear goals/objectives’ changes, when

the factor ’support from senior management’ changes. Knowing such dependencies can

help identifying causal relationships of factors.

2.2.4 Ahmed: Understanding the knowledge needs of novice

designers in the aerospace industry

In this study Ahmed and Wallace (2004) investigated how unexperienced designers gain

knowledge about the design process from their experienced peers in an industrial environ-

ment. Aim was to understand the knowledge needs and awareness of their own knowledge

of novice designers. The research consisted of two parts, with the second part being this

study. It built up on the first part in which Ahmed, Wallace and Blessing (2003) observed

experienced and unexperienced designers to study the difference in how they approach

design tasks. Unit of analysis were novice designers, which were defined as being young

engineers with less than 2.5 years of work experience. Experienced designers were defined

as having more than 10 years of work experience in the engineering field. The study pro-

cess was controlled, as the participants had a pre-defined scope and were aware that the

conversations were recorded. The data was collected through audio-recorded interview

sessions, which the novice designers held with the experienced designers. Using discourse

analysis, the recorded qualitative data was then analyzed. Discourses were the interac-

tions between the novice and experienced designers. Discourse analysis is a central tool

in the social sciences (Potter, 2004), utilized to conceptualize language with the goal of

finding explaining patterns. In the case of this study, for example, five different patterns

of query and response were defined through which the novice designers gathered informa-

tion from the experienced designers, as shown in Figure 2.4. The number of queries and

responses fitting in one of these patterns were then counted from the recorded data. This

step meant a conversion of qualitative data into quantitative data, allowing for the use of

descriptive quantitative methods. In the next step each pattern was cross-checked against

different query topics the novice designers had asked about and which were counted as

well. Figure 2.5 shows an example for the pattern ’rephrased or irrelevant queries’ checked

against the topics of queries. The experience of the teams - each consisting of two novice

designers, except of Team C, where one of the two members had 8 years of work experi-

ence - increased from A to C. In this example from the study, the researchers concluded

that the less experience designers have, the more they are interested in how the product

or technology works. However, the more experience they gain in their professional career,

the more their focus shifts, e.g. to how processes in the company work.

Pros of this study type: The researchers attempted to translate the qualitative recorded

data into quantitative data. They did so by using discourse analysis, a method from the

social sciences, where translating highly qualitative data into quantitative data is a com-

mon research challenge. Hence, supportive tools were devloped over the years. Having

quantitative data to work with, conclusions can be made more objectively, e.g. by using

basic descriptive statistics. In this case a bar diagram was used to illustrate frequencies

for query/response patterns against different topics the novice designers asked about.

Figure 2.4: Patterns of query and responses for gathering information

Figure 2.5: Cross-check of patterns against topics of queries

Cons of this study type: Again, as in Hales’ study, a small sample size was investigated in

the study. The question remains how representative the data of the three teams (with two

participants each) is. The definition of what a novice and what an experienced designer

is appeared to be broad and neglected the consideration of individual traits, preferences

and skills, factors that might strongly influence the results. Another important aspect

to recognize is, that although the data was quantified for use of descriptive methods, no

criteria for distinction were defined. The results were still fairly freely interpreted from

the visualized bar charts. A beforehand defined criteria, on when a factor was considered

as distinctive, would have been desirable in order to avoid this room for interpretation

in the results. Using more advanced statistics provides such criteria through significance

levels and probabilities of repeatability. Therefore, sufficient sample sizes are essential.

2.3 Meta-Analyses on Success Factors

Beyond individual empirical research projects, studies reviewing and summarizing the

results of empirical studies have been performed. The results of these meta-analyses pro-

vide a valuable overview on the reoccurrence and frequency of success factors found in

individual empirical research efforts. Such reviewing studies can be found in Engineering

Design Research, but also in related higher level fields, like the NPD (New Product Devel-

opment), Innovation and Management Science. Three meta-analyses, with considerable

sample sizes, are discussed in the following.

Ernst: Success factors of new product development: a review of the empirical

literature

In this review of the empirical literature, Ernst (2002) summarized the findings of em-

pirical studies on success factors of NPD. The studies were based on questionnaires to

industrial companies and provided significant sample sizes (n = 18-1400) in most cases.

A total of 52 studies where reviewed, all performed in the time between 1974 and 1999.

The author developed five categories and assumed the determined success factors would

fit in any of these. Ernst summarized his findings as follows:

•NPD process: Presence of a formal or informal new product development process

to support a clear technical and commercial planning, and provide ongoing control

of the project progress.

•Organization: Dedicated, multi-disciplinary project teams. Skills and know-how of

Project Lead plays significant role.

•Culture: No significant results found, topic ”has not been adequately researched to

date” (Ernst, 2002, p.32).

•Role and commitment of senior management: Recognition of the value of new

product developments has positive effect on product success.

•Strategy: Only weak evidence, suggesting that long-term, strategic NPD planning

yields higher success.

The selection of the categories relates to the domain of interest, which was the man-

agement of NPD in this meta-analysis. A more limited view on the product development

department as domain of interest would have likely suggested a different categorization.

The measurements of success of the investigated studies varied between the dimensions:

commercial success, technical succes and on time delivery.

Mehalik and Schunn: What Constitutes Good Design? A Review of Empirical

Studies of Design Processes

Mehalik and Schunn (2006) reviewed 40 journal-published, empirical studies. The authors

were looking for elements which are reported to be associated with good and effective

design pratice. The reviewed studies where performed between 1986-2003, most of them

in the years around the turn of the century. The publications on success factors in product

design were found in a variety of journals from different areas (see Table 2.2).

Table 2.2: Journals represented in Mehalik’s and Schunn’s meta-analysis

International Journal of Human-Computer Interaction 10

Design Studies 8

Cognitive Science 3

International Journal of Man-Machine Studies 3

Behaviour & Information Technology 2

International Journal of Human-Computer Studies 2

International Journal of Technology and Design Journal of Technology and Design

Education

Journal of Engineering Design 2

Applied Ergonomics 1

Ergonomics 1

Human-Computer Interaction 1

International Journal of Intelligent Systems 1

Journal of Applied Psychology 1

Learning and Instruction 1

Proc Instn Mech Engers 1

Thinking and Reasoning 1

Total 40

Around half of the studies consisted of design tasks in the engineering fields (mechan-

ical, electrical, structural, civil, automotive, and other types of engineering). Software

development projects comprised about 40% of the studies, the rest were studies from non-

engineering fields, such as architecture or marketing. In 60% of the studies, the subjects

were engaged in real design tasks. The remaining studies were based on artificial studies

or a combination of aritificial and real. The focus of the majority of the studies was on

the earlier design stages, which explains why the findings show more detailed factors than

the higher level studies. The three most significant success factors detected were:

•Explore problem representation (refers to how designers go about identifying and

defining the design task or problem)

•Use interactive/iterative design methodology

•Search the space (explore alternatives)

Schimmoeller: Success Factors of new Product Development Processes

Schimmoeller (2010) reviewed eleven publications on success factors of new product de-

velopment projects from in between 1975 and 2003. Among the reviewed publications

where not solely empirical studies, but also articles on the topic, mostly with a view from

innovation management. The review scope was limited, in that the aim was to verify,

if successful development projects have three commonly suggested (Schimmoeller, 2010)

key success factors in common:

•Cross-functional teams

•Support of upper-management

•Organizational structure

By organizational structure, the author was referring to an integrated product devel-

opment process, supporting the development steps, communication within the company,

documentation and tracking of project progress. The dimensions for measurement of

success were defined as: product performance, speed to market and development costs.

The reviewed publications provided some evidence that all three factors play a role. The

factor of most significance, with the most frequent citation, was found in the management

support, particularly with respect to speed to market.

In total the three meta-analyses provide findings of 103 empirical studies and publica-

tions on success factors of the product development process. It has to be noted that this

sample size represents an accumulation of different types of studies with varying charac-

teristics. In addition, variance existed in the domains of interest and observed processes.

Schimmoeller’s and Ernst’s reviews focused on a higher organizational level and found as

common success factors:

1. Need for a product development process

2. Multi-disciplinary teams

3. Skill set and know-how of project lead

4. Support of (upper-)management

Mehalik and Schunn reviewed studies of a more detailed level, mainly focusing on the

early design stages as unit of analysis. Their findings add three success factors on a design

work level to the list:

5. Define the task/problem

6. Use design methodology

7. Search for alternative solutions

These three are all elements which can frequently be found recommended in the clas-

sic engineering design literature (Ehrlenspiel, 2009; French, 1999; Pahl et al., 2007; Pugh,

1991; Ullman, 2003).

From the introduced meta-analyses, one can recognize that the Comprehensive Approach

needs to take Potential Success Factors of different organizational levels into considera-

tion, from management down to detailed design tasks. The seven factors, together with

the findings of the introduced empirical industry studies, serve as input for the later

process of determining all the Potential Success Factors and are complemented by the

factors found from a review of the product development literature.

2.4 Research Needs

The four research works introduced in section 2.2 provide a representative cross-section

through existing types of empirical industry studies with varying charactersitics. Their

pros and cons revealed the consequences on the results, that come with the way certain

characteristics are chosen. From the cons, the research needs for comprehensive empirical

industry studies can be derived. These can be summarized as follows:

•Study process (nature of study): Primarily controlled studies can be found. They

were characterzied by the researcher directly being involved into the research or

at least controlling some parameters. Controlled studies promote the chance for

bias and are also likely to alter the behaviour and habits of participants. Natural

studies avoid any interaction between researchers and the observed process with its

participants and do in that way produce the closest to ’real world’ results.

•Theoretical basis: The theoretical framework of studies is in many cases missing

or not obvious. A missing framework results in two effects. First, the comprehen-

siveness of the studies are to be questioned. This case can be observed when a

limited amount of factors are investigated and it is not explained how they were

derived and what assumptions were made. Second, the causality of the results is to

be questioned. Factors considered in a study and assumed to contribute to success

are oftentimes linked in relationships of cause and effect. Some factors are actual

causes, others might just be symptoms which result from other causing factors.

Without a sound framework to distinguish between cause and effect, results might

be misleading with respect to causality. Blessing and Chakrabarti (2009) defined

a Design Research Methodology that provides engineering design researchers with

a process framework, which partially addresses comprehensiveness and causality.

They aknowledge that industrial data is difficult to obtain for academic researchers

and developed their methodology to be used for small sample sizes and support the

making of necessary assumptions on dependencies.

•Data collection method: The way the data is collected should support the quan-

tification to allow for the best possible objective analysis. It should also support

the collection of sufficient data. The influence of participants and chance for bias

should be considered as well, when making this choice.

•Observed process: The exact observed procees is not always known for studies.

Especially the end part, where the outcome of the design process is validated,

e.g. through a commissioning or testing phase, can be found to be missing quite

frequently. A representative study should take the outcome of the observed process

into consideration.

•Number of cases: A small number of cases is not suitable if study results are aimed

to be generalized, which research efforts usually aim for. A sufficient amount of

cases is essential in order to verify the confidence of the results and draw general

conclusions.

•Participants: This is the primary source of bias and the researcher needs to be

aware of it. Ideally the participants of the observed process are not influenced

during the study. This can be done, for instance by analyzing projects after they

have been completed. The participants of the study might be different from the

participants of the observed process, which is desired in order to reduce bias. Study

participants asked to rate the outcome of a project they were not involved, are likely

to give a more objective answer, than would be the case if participants are asked

to rate projects they managed or were involved themsleves. In addition to the

participants, researchers themselves can be a source of bias, especially when they

have to interpret data or convert qualitative into quantitative data themselves. And

similar to the participants, if they were involved in the studied projects themselves.

To minize the researchers bias, they ideally only collect already quantified data (e.g.

documented team sizes, project time lines, budgets) and gather data that needs to

be converted from neutral sources, e.g. by interviewing stakeholders that have or

had no personal bond to the investigated projects. In addition to that, researchers

can increase or reduce bias by the way they collect data, e.g. through the types

of questions or quantification scales (Blessing and Chakrabarti, 2009; Landy and

Barnes, 1979). For that reason, researchers should run trial runs with their defined

questions and quantification scales to get an understanding of how comfortable and

neutral participants are likely going to be with their answers.

•Coding and analysis method(s): Quantitative methods are desirable. Most studies

in literature show the conversion to and use of quantitative data. However, the

methods used to analyze the data cannot always be found to be suitable. If only

basic descriptive statstics are applied, hard criteria are missing, which support

the researcher in deciding which factors are significant and which are not. Ernst

reported in the findings of his meta-analysis of empirical studies that

studies frequently do not give reliability coefficients. Because these data

are missing, it is not possible to make a judgement on the reliability of

the constructs. Here one must encourage scholars to apply more rigor-

ous statistical techniques in empirical studies and one should introduce

minimum reporting standards in publications. (Ernst, 2002, p.33)

The use of advanced (inferential) statstics can help to overcome this shortcoming,

as clear criteria are provided by significance level and probabilities. In fact, a more

rigorous application of advanced statistical methods can be observed within the last

few years, e.g. in Olechowski et al. (2012) or Welo et al. (2013). In order to be

able to apply methods of inferential statistics, the right kind of data needs to be

available and collected, as discussed in section 3.2.

•Verification methods: Utilizing advanced statistics provide objective indicators,

such as confidence intervals and probability of repeatability. In addition, the veri-

fication of dependencies (e.g. through multiple variable regression) and their direc-

tion of change, can support the determination of causal relationships. Ernst also

noted in his findings that

Only in the past few years have some authors begun to conduct empirical

research of success factors on the basis of reliable measurement for the

dependent and the independent variables. (Ernst, 2002, p.33)

2.5 Characteristics of the Comprehensive Approach

The discussed research needs lead to a set of desired characteristics, as shown in Table 2.3.

These characteristics are regarded as best suited for a comprehensive empirical industry

study. It is the aim in this research, to be aligned with this ’ideal’ type in the best possible

manner. This attempt is referred to as the Comprehensive Approach. As can be seen in

Table 2.3, some of the characteristics are assumed to be influencing factors (Task, Case

Size and Participants). These have to be evaluated for their cause-effect relationships

together with the population of influencing factors, which are systematically determined.

In the next chapter, the theoretical framework is explained. Background on framework

related characteristics, such as data anlysis and data collection methods, are presented

in this chapter as well.

Table 2.3: Characteristics of the Comprehensive Approach

Characteristic Options

Aim, research question, hypothe-

sis

Find factors which distinguish successful from less

successful development projects.

Study process (nature of study) Natural.

Theoretical basis Framework supporting identification of cause-effect

relationships. Systematic consideration of all influ-

encing factors to ensure comprehensiveness.

Unit(s) of analysis Product development process, holistic view.

Data collection method Investigation of completed projects. Collect and

quantify data from project reports and interviews.

Success to be rated by participants not involved in

the design projects themselves.

Observed process Product development process from planning phase

to product validation.

Task Varying cases of complexity to verify cause-effect

relationships.

Number of cases As many as possible.

Case size Varying case sizes to verify cause-effect relationship.

Participants Varying experience, skills, personalities to verify

cause-effect relationships.

Object Global industrial companies.

Coding and analysis method(s) Advanced (inferential) statistics.

Verification method(s) 1.) Derives from use of advanced statistics (proba-

bility of repeatability). 2.) Determine dependencies

between found Success Factors.

3 Framework of the Comprehensive

Approach

A sound methodological framework is the basic condition for any research activity. Dif-

ferent viewpoints and approaches can be found in any scientiﬁc ﬁeld. However, in En-

gineering Design Research ”a rather fragmented, if not a chaotic, picture” (Horv´ath,

2004, p.155) can be found, certainly attributed to the complexitiy of engineering design,

consisting of a multifaceted combination of science and human behaviour. Dixon de-

scribes the objective of engineering design research to be the search for ”theories that

can be tested by formal methods of hypothesis testing” (Dixon, 1987, p.145). Antonsson

pointed out that ”EDR [Engineering Design Research] often lacks a clear hypothesis and

testing method” (Antonsson, 1987, p.153). It was the aim in this research to develop a

framework, representing a clear theory, which can be veriﬁed with methods of hypothesis

testing. The underlying assumption of this theoretical framework is that the product

development process can be viewed as a matter of cause and eﬀect, as shown in the

Cause-Eﬀect-Diagram (Ishikawa-Diagram) in Figure 3.1.

Figure 3.1: Cause-eﬀect relationship in product development

The nature of any process is that it transforms certain input parameters (xi)into

a desired output (Y). In theory, by knowing all the input parameters (causes) and by

knowing the transfer function f(x1...xn) of the process, it is possible to predict the output

(Y), as shown in Figure 3.2. However, the transfer function is not known and determining

it is precisely the objective of Engieering Design Research. In other words, Engineering

Design Research is interested in knowing which of the input parameters correlate to the

output, and knowing dependencies between input parameters.

Figure 3.2: Transfer function of product development process

The transfer function can be developed if the input and output parameters are known.

Sample data for both these factors can be obtained from real world development projects.

Then the transfer function can be developed by performing hypothesis testing with meth-

ods of inferential statistics. This approach of converting a practical problem into a statis-

tical problem, finding the transfer function from collected actual input and output data,

and converting the statistical solution back into a practical solution; is in analogy to the

Six Sigma theory, as shown in Figure 3.3.

Figure 3.3: Theory behind the Six Sigma approach (Brue and Launsby, 2003)

The concept of the hypothesis test is, to separate the independent variables (causes

= Potential Success Factors) from the dependent variable (effect = Design Success), then

test each independent variable for a significant correlation to the dependent variable

(Figure 3.4), using methods of inferential statistics. The significant input parameters

identified can then be considered Success Factors. However, this analysis step does not

provide the full picture. It is possible that some of the Success Factors are dependent

on each other. For instance, Success Factor A could be caused to change by Success

Factor B. The first analysis step would show a relationship to Design Success for both

factors, because Success Factor B causes Success Factor A to change whenever it changes

itself. In such a case Success Factor B would be dominant over Success Factor A, but

it would not be obvious. In order to reveal such kinds of dependencies, a subsequent,

second analysis step is required. This second analysis step can be performed again, with

statistical methods for hypothesis testing. Its sole aim is to show dependencies between

the Success Factors that were determined in the first analysis step. The methods which

were used for the two analysis steps in this study are discussed in the Data Analysis

Chapter. Only after both these steps were performed, could the results be interpreted

and the Dominant Success Factors be filtered out. Ernst pointed out that this second

and essential analysis step can seldomly be found in empirical studies (Ernst, 2002). The

process map in Figure 3.5 illustrates the theoretical framework of the Comprehensive

Approach. The first step is the definition of the research objective (What is the effect of

interest?), which in the case of this study was Design Success. After that, the cause-effect

relationship needs to be established. In the next step, the effect (Design Success) needs

to be defined and quantified. As explained in more detail in the Defining Design Success

Chapter, domain and dimensions of the effect need to be known to allow for a clearly

defined scope. Similarly to the effect, all the causes (Potential Success Factors) need

to be determined, grouped and quantified. It is shown in the Potential Success Factor

Chapter that these steps are necessary to limit the amount of data to a manageable size,

without sacrificing comprehensiveness. With the effect and the causes quantified, the two

analysis steps can be performed, which results in the Dominat Success Factors. Knowing

these, allows for the introduction of suitable measures for increased Design Success.

Figure 3.4: Concept of hypothesis test

Figure 3.5: Process map of the Comprehensive Approach

The most critical requirements for the successful implementation of this framework

are the following:

•Clear definition of what Design Success (Y) is and how it is quantitatively measured.

•Determination and quantification of all Potential Success Factors (xi).

•Use of robust statistical methods, which provide a measure for confidence of the

results.

•Collection of sufficient ’real world’ sample data to allow for the use of these statis-

tical methods for hypothesis testing.

3.1 Link to Design Research Methodology

Recognizing the aforementioned lack of theoretical foundations in Engineering Design

Research, Blessing and Chakrabarti (2009) defined an overall framework for research in

engineering design in their book DRM, a Design Research Methodology. They recommend

to split the research process into four stages:

•Research Clarification (RC): Formulate Research Goal. Define what part of the

engineering design process shall be improved.

•Descriptive Study I (DSI): Obtain understanding what factors influence the area of

interest, e.g. by conducting literature reviews (Review-based) or empirical studies

(Comprehensive).

•Prescriptive Sudy (PS): Develop measures to improve the situation, based on the

findings in DSI.

•Descriptive Study II (DSII): Verify effectiveness of the developed measures, e.g. by

conducting empirical studies.

The framework provides flexibility in that not each stage has to be followed for specific

research projects, depending on the research question and scope of the project. In total,

the DRM framework results in seven possible research types (Figure 3.6). The approach

and empirical study outlined in this dissertation falls under the second type. The first two

stages (RC+DSI) are addressed in their full entirety, while the third stage (PS) is only

addressed in an initial state, which are recommendations for engineering management on

what factors need more focus in development projects. The Comprehensive Approach

complements the DRM framework in that it provides a more in depth and rigorous process

for conducting a comprehensive study in the DSI stage.

Figure 3.6: Seven possible types of studies in the DRM framework

3.2 Statistical Methods

In this section, only a broad overview of statistics and the terminology related to this

research is given. For a more detailed view on the subject, it is referred to the extensive

literature available.

Statistics can be divided into two main fields: Descriptive Statistics and Inferential Statis-

tics. The first field is concerned with describing the nature of an existing set of data, by

using for instance graphical illustrations, such as Histograms or Box Plots. The latter

field is concerned with extrapolating the pattern of data samples of limited size to a

population and, in addition, providing levels of confidence for this extrapolation. The

confidence is a measure for the probability, an observed pattern could repeatedly be seen,

when drawing different samples from a population. The multifold scenarios in all areas

of science, where extensive - or even infinite - amounts of population data exist, made it

necessary to come up with this ’mathematical shortcut’. Many different kinds of statis-

tical methods exist. Selecting the most suitable method for a problem is dependent on

two main factors:

1. Types of data available

2. Distribution of the available data

Data Types in Statistics

Four different types of data have to be distinguished:

•Nominal: Data divided into categories which are neither in a certain order, nor

have any relationship. An example would be male versus female, or parts separated

into categories by color. Changing the order of the categories does not change the

nature of the data.

•Ordinal: Data points which have ranks on a scale. The ranks might be represented

by numbers (usually integer), or descriptions (e.g. negative, neutral, positive). The

distance between the ranks might not be equal and can usually not be determined.

•Interval: Variables which are measured on an interval scale are uniform on the

scale. Mathematical operations can be applied, but ratios are not meaningful. A

typical example for interval data is the Celcius scale. Temperatures can be added

or subtracted and the results are valid. However, it cannot be claimed that 40C is

twice as hot as 20C, as 0C is not an absolut zero point.

•Ratio: Similar variables than interval data, but with meaningful ratios. A typical

example is the Kelvin temperature scale. Unlike on the Celcius scale, it is valid to

claim that a temperature of 600K is twice as hot as a temperature of 300K, since

an absolute zero point exists at 0K.

The first two types (nominal and ordinal) are common whenever qualitative data is

translated into quantitative data, e.g. data obtained through surveys. Suitable statistical

methods for these data types have especially been developed in the social sciences. These

are the types of data usually seen in Engineering Design Research. Interval and ratio

data on the other hand, are more commonly found in the areas where ’hard data’ can be

obtained, for example in the natural sciences or economics.

Distribution of Data

The second important criterion for the selection of a statistical method is the distribution

of the sample data. Parametric tests are used under the assumption that the data follows

a normal distribution. In case it is found that the data is not normally distributed, non-

parametric methods have to be used. Table 3.1 gives an overview of suitable tests for

different kinds of comparisons.

Table 3.1: Paramteric vs. non-paramteric statistical tests (Burke, 1998)

3.3 Data Collection Methods

Varying methods for data collection are available in the social sciences and can be applied

for problems in Engineering Design Research. They can be divided into the main groups:

observation, experiments, case studies, analyzing documents, questionnaires, interviews

(adapted from Blessing and Chakrabarti (2009)). Each of these groups have subsidies,

sometimes a combination of methods is applied to research problems.

The criteria of comprehensiveness and high reliability of results for the Comprehensive

Approach, narrow the options for suitable data collection methods. The requirement

for ’real world’ data from development projects in an actual industrial setting prevents

the use of experiments or case studies. Observational studies are time consuming and

only allow for the collection of one or very few cases. Blessing and Chakrabarti called

it ”impossible or very difficult” (Blessing and Chakrabarti, 2009, p.255), to collect data

from multiple cases in industrial studies. However, the use of statistical methods for data

analysis demands data from multiple industrial studies. The following three methods

remain, in permitting the collection of data from multiple industrial cases:

•Interviewing

•Questionnaires

•Analyzing documents

The quality of the data is an important aspect which needs to be considered care-

fully. Especially when information is directly received from humans, or even participants

of the design projects investigated, bias is unavoidable (Donaldson and Grant-Vallone,

2002). The goal in empirical research needs to be, the reduction of chances for bias to

a minimum. The way in which the data is collected is an essential part of this. Al-

though questionnaires provide the most amount of data with the least amount of effort,

they bear a lot of uncertainty, as oftentimes little is known about the participants, their

background and involvement in the investigated projects. Interviewing and analyzing

documents gives researchers more control on this factor and are hence the better quality

sources for data collection with respect to bias.

Still, researchers need to be aware of two aspects of bias. At first, they need to prevent

self-report bias as much as possible, as participants - as well as researchers, if they were

involved in study objects themselves - tend to answer in ”socially desirable ways” (Don-

aldson and Grant-Vallone, 2002, p. 247). This is because participants usually want to

look as good as possible, or they sense the possibility that their employer could have ac-

cess to the data they provided (Donaldson and Grant-Vallone, 2002). White and Fortune

experienced such optimistic self-reporting (see Figure 2.3) in a study where 82% of the

participants were directly involved in the projects they were asked about. To avoid this

aspect of bias, data can be collected from sources that know the project well enough,

but do not have a direct involvement. These could be stakeholders of studied projects or

managers of the participants.

Secondly, researchers need to chose and prepare the data collection in a way that lim-

its bias from a rating error perspective. They can do so by formulating questions in a

neutral manner that do not direct participants towards a desired answer (Blessing and

Chakrabarti, 2009). For data quantified on a scale, it helps to provide definitions along

the quantification scale, giving the participants anchor points that are more informative

for decision making than simpley numbers (Landy and Barnes, 1979). In order to get an

understanding of the potential for bias, researchers should run test trials of the defined

questions and quantification scales with one or a few participants. This will give them

the chance for correction, if they sense that trial run participants struggle with making

solid judgements and to test the fitness of the purpose (Aurisicchio and Wallace, 2007).

4 Defining Design Success

Success in product development cannnot be defined in an one-dimensional measure (Grif-

fin and Page, 1993; Prabhakar, 2008; Suomala and Jokioinen, 2003). In Engineering De-

sign and Innovation Management Research, the triad (Schimmoeller, 2010), also referred

to as the ’iron triangle’ (Gericke, 2011), is oftentimes used:

•Development of products that fulfill defined Product Requirements

•Meeting Development Timeline

•Stay within Development Budget

These three dimensions for measuring Design Success are widely accepted as the best

practice (Kerzner, 2010). White and Fortune (2002) found, in a survey to 995 project

managers in different companies, that these dimensions are recognized significantly more

important than any others (Table 4.1).

Table 4.1: Rated importance of dimensions for measuring success (White and Fortune,

2002)

Criteria Sum of re-coded

ranking

Sums

ranked

Meets client’s requirements 970 1

Completed within schedule 850 2

Completed within budget 766 3

Meets organisational objectives 188 4

Yields business and other benefits 86 5

Causes minimal business disruption 71 6

Meets quality/safety standards 48 7

Other criteria 20 8

Conversely, it can be found that the triad is critized as being ”too simplistic” (de Wit,

1988, p.166). De Wit recognizes the three dimensions, but considers these dependent on

the objective of the development project. Objectives vary by types of projects, throughout

the life cycle, view from management hierarchy and stakeholders involved. He demon-

strates different success criteria during the life cycle of a oil field development project,

due to varying objectives in different project phases (Table 4.2).

Table 4.2: Changing success criteria with varying objective (de Wit, 1988)

Phases Primary objective

Exploration Find oil in large enough quantity for devel-

opment

Development Develop the oil field in the most economic

manner

Production Maximize daily production and optimize to-

tal oil recovery

In order to account for the objective, it is necessary to define the domain, in addition

to the dimensions of success. By domain, it can be referred to the different departments

in an organization. Depending on the department under consideration, the dimensions

of success vary. For instance, the marketing or service departments have different success

criteria than the engineering department. Having defined the domain of interest for

a research project, the dimensions result from the metric this department is measured

against in the organization (Figure 4.1). In the case of Engineering Design Research,

the domain of interest is the development department, which leads to the three success

dimensions: product requirements, timeline and budget.

Figure 4.1: Examples of domains and their success dimensions

4.1 Success Dimensions of the Comprehensive Ap-

proach

For the empirical study, it had to be decided which of the three success dimensions is

of most interest and serves as the primary dimension. Defining a primary dimension is

inevitable, as pursuing all three dimensions in one study with one set of data leads to

one of the following two issues:

1. In order to determine the factors for design success statistically, a good distribu-

tion of data from highly successful projects, to projects of low success, has to be

considered (ideally normally distributed to allow for the use of parametric statis-

tical methods). If only successful or only unsuccessful projects are investigated,

the findings cannot be seen as verified, as the opposite was not proven. Finding

a good distribution of projects on the success scale with respect to one dimension

of interest is doable. However, finding a sample size of projects that includes a

representative sample distribution in three different dimensions is very unlikely. At

least it would require extensive amounts of data, more than is possbile in the course

of a doctoral research.

2. The three dimensions are not necessarily on the effect-side, but can also switch to

the cause-side. This is especially true for products which have development cycles

of several years. Development budgets in companies are typically determined and

distributed on a yearly basis. A development project which is planned for several

years can be impacted by funding cuts during that duration. In this case, the third

dimension ’Stay within Development Budget’, which was thus far considered to be

an effect, would now become a cause that impacts the first two dimensions.

The primary dimension was chosen to be ’Product Requirement Fulfillment’. ’Time-

line Met’ was chosen as secondary dimension. The correlation of Success Factors to this

secondary dimension is investigated indirectly through the first dimension, as shown in

the Data Analysis Chapter. This verification is necessary, as otherwise the question, if

any project can be accomplished successfully if there was just enough time spent and re-

sources invested, remains open. The third dimension ’Stay within Development Budget’

turned out to be a cause and is investigated under the Potential Success Factor ’Resources

provided / Project Stability’.

4.2 Primary Success Dimension

An additional advantage of chosing the fulfillment of product requirements as primary

dimension is that this factor can be easily measured on at least an ordinal scale. For this

research it was decided to measure success on a scale from one to ten. This would even

allow a direct translation to success rates between zero and a hundred per cent.

For the attempt of quantifying originally qualitative data (perception of success), it was

important to define anchors on the rating scale using clear descriptions. This helped

to recognize the reasons behind the decision, guiding interviewees towards a rational

judgement rather than an unspecified rating anywhere on the scale (Landy and Barnes,

1979). Table 4.3 shows the success quantification scale for the primary dimension ’Product

Requirement Fulfillment’, with descriptions on two point intervals. In this case, Product

Requirement represented a combined measure, not only including functional and quality

aspects, but also legal and cost considerations. The interviewees were asked to use the

end customer’s perception of and reaction to the developed product, with respect to these

dimensions, as a reference for their overall judgement.

Table 4.3: Quantification scale for primary dimension ’Product Requirement Fulfillment’

10 Product fulfilled all product requirements as specified and worked ”out of the

box”.

Minor problems with functionality/quality and fulfillment of product require-

ments; minor rework/adjustments required during commissioning phase; minor

costs occured for customer and/or company from rework.

Noticeable problems with functionality/quality and fulfillment of product

requirements; moderate rework/adjustments required during commissioning

phase; moderate costs occured for customer and/or company from rework.

Large problems with functionality/quality and fulfillment of product require-

ments; large amount of rework/adjustments required during commisioning

phase; high costs occured for customer and/or company from rework.

Very large problems with functionality/quality and fulfillment of product re-

quirements; product only worked after fundamental changes/redesign; exces-

sive costs occured for customer and/or company.

0Product failed to function; repair not possible or with extensive efforts; devel-

opment program terminated before completion.

The other source of bias in success measurement is self-report bias (Donaldson and

Grant-Vallone, 2002). This occurs when participants, involved in the projects themselves,

are asked to rate success (Blessing and Chakrabarti, 2009; White and Fortune, 2002), as

described in section 3.3. To avoid this ’pitfall’, the success ratings for the projects in this

study were provided by the stakeholders of the project. In the case of this study, these

were internal customers, such as the Product Line Management or Service Department.

On the cause side (Potential Success Factors), managers and team members (if managers

didn’t have the knowledge) were interviewed for missing information, or information

which could not be retrieved directly from the engineering database, such as the average

experience of project teams. The clear definition of success on a scale and the separation

of success rating from persons involved in the projects themselves, is expected to have

greatly helped in minimizing the chance for bias in the data. However, it has to be

recognized that a certain amount of bias remained, which is inherent to any translation

of qualitative into quantitative data.

4.3 Data Collection for Primary Success Dimension

Collecting the data for success from the stakeholders of the projects meant interviewing

actual persons in the organization. The stakeholders were managers in the Product Line

Management and Service Departments. These departments served as customers of the

investigated development projects in that they provided the funding to the engineering

department for execution and were the recepients of the final product. Hence, they had a

special interest in the on time delivery of a specified product, within a provided budget.

In total, nine managers were asked to judge the success of the projects using the rating

scale shown in Table 4.3. A trial run was performed with one manager at first to allow

for a ’fine tuning’ of the descriptions on the scale. This trial run showed that using a

descritpion anchor on every second step on the scale was sufficient. In general, this held

true for all participants during the actual data collection. In cases they were indecisive

about two descritptions, they chose a rating value in between.

In the end, 44 projects remained which provided valid data (see section 5.3) and were

used for the study. After collecting the success data for all these projects, the next step

was to understand the distribution of the data. Figure 4.2 shows the distribution of the

44 projects over the sucess scale from one to ten. It can be seen that the distribution

is not even. For the projects rated between four and ten, the data is close to normally

distributed. In addition, four projects of zero success are found on the scale. These are

projects which were terminated before completion. No projects were found between the

scale points zero to four. This pattern suggests that the organization had measures in

place to terminate projects, before they lead to ’catastrophic’ results. The resulting gap

in the data meant that the desired even, or normal distribution of data points on the

scale was non-existent. However, as elaborated in the Data Analysis Chapter, suitable

statistical tools exist for such uneven distributions of data; and are used for the analysis.

Figure 4.2: Design Success of 44 investigated projects

4.4 Secondary Success Dimension

Defining success in terms of the project timeline is different than for the product require-

ments. On the one hand, the data is already in quantitative form, as the completion dates

of the projects can be retrieved from the engineering database. On the other hand, the

question on how these dates could be translated into a success scale, provided different

possible answers. For instance, if a project met the timeline, it should have been consid-

ered as high success. But what about projects which were completed beyond the target

date? Due to this unclarity with respect to a reference, it was decided that the best way

to define this success dimension is with categories for ’timeline met’ and ’timeline not

met’. As the investigated projects have typical cycle times of several years, a tolerance

band of plus and minus one month was accepted as timeline met. For the later evaluation

against the primary dimension, the four projects with zero success were left out, as these

projects had been terminated before completion. A third category ’timeline exceeded’

could have possibly been defined, but it was already known from the data collection pro-

cess, that none of the 44 projects were completed ahead of the target date.

The data for meeting or missing the completion dates was directly retrieved from the en-

gineering database, where target and actual completion dates were recorded. The anlysis

against the primary success dimension is shown in the Data Analysis Chapter.

5 Potential Success Factors

The causes that ultimately lead to the effect in product development, sum up to an

extensive amount of data. As shown in Figure 3.1, the Potential Success Factors can be

found to fit into one of the six categories: Information, Environment, People, Method,

Process and Management. A comprehensive review of the product development literature,

journals and meta-anlyses on succes factors in engineering desing, as well as feedback

from the interviewed project leads and engineering managers, lead to a total of 63 causes

(xn) in the six categories, for the domain development department. The Comprehensive

Approach demands the quantification of all of the causes in order to perform a hypotheses

test against the effect (Design Success). This means that for a sample size of 44 projects

(m) and two (primary and secondary) success dimensions (yk), the resulting amount of

required data points (d) is

(n+k)∗m=d(5.1)

(63 + 2) ∗44 = 2860.

This number illustrates a major reason why comprehensive studies are so difficult and

are rarely conducted. The effort for collecting and evaluating data of that many causes

would be extensive and beyond the time frame of typical studies, such as a doctoral

research. A more manageable amount of data would result, if there was a way to reduce

the amount of Potential Success Factors systematically.

5.1 Reduction of Potential Success Factors

A systemtatic reduction of the data size on the cause-side demands that the compre-

hensiveness of the approach must not be sacrificed. With respect to this demand, a

simplification can be justified (W¨orz and G¨ohlich, 2012) using the following two consid-

erations:

1. The results of an Empirical Engineering Design Research study are intended to be

of practical use for engineering managers and development project leads working

in the industry. Keeping this in mind, the data set can be reduced to the Poten-

tial Success Factors that can be influenced and measured, for instance Team Size

or Team Composition. Managers are certainly interested in results that suggest

measures of how to form productive teams. Conversely, certain Potential Success

Factors, for instance creativity, are difficult to controll or be influenced by engi-

neering management or project leads, when setting up a development project. In

addition, it needs to be assured that there is a way to quantitatively measure such

a cause. In psychology, there is not yet a clear common understanding or way of

measuring creativity (Gausemeier et al., 2000). While it is oftentimes acknowledged

as a personal trait, some researchers consider it a result of situational circumstances

and environments (Amabile, 1996). Therefore, the first reduction in the data relates

to the ’practicality’ of the Potential Success Factors.

2. Instead of quantifying each cause, it is reasonable to combine certain causes into

groups, which can then be quantified as one measure. For instance, in the litera-

ture, aspects of performing conceptual design are oftentimes mentioned as factors

leading to Design Success, such as: breaking problems into sub-functions, use of

creativity techniques for solution finding, finding many alternative solutions, etc.

A higher level category or cause, where all these single causes fit, can be defined

as ’Conceptual Design performed’. By defining one quantitative scale for such a

higher level cause, it is possible to evaluate if and to what extent it contributes to

successful product design. Figure 5.1 shows the concept of this iterative approach.

Figure 5.1: Iterative approach for reduction of data points

Applying these two criteria leads to a reduction in the determined Potential Success

Factors from origianlly 63 to 18. This means that, applying formula 5.1 again, the number

of data points has been reduced from 2860 to

(18 + 2) ∗44 = 880.

A reduction to less than a third of the originally required data points was achieved,

by retaining the comprehensiveness of the study - with respect to information necessary

for engineering management. Applying the iterative approach might result in the need

for a second iteration, as shown in Figure 5.1. Still, the overall required data points are

reduced, as some groups of higher level causes can be expected to be eliminated in the

first iteration. An additional advantage of this approach is that the extensive research

effort of a comprehensive investigation of Success Factors in product development can be

split clearly into separatable stages. The time constraint of this research work permitted

only to focus on the first iteration. However, the results presented in this dissertation can

provide the ground work for future researchers, to focus in more detail on the determined

Success Factors, especially if they are groups of higher level causes.

Figure 5.2 shows the 18 remaining Potential Success Factors in the Ishikawa Diagram,

after the systematic reduction. The six categories (Methodology, Management, People,

Process, Information and Environment) were derived from categorizing the 18 causes with

help of the Affinity Diagram Procedure (Cohen, 1995).

Figure 5.2: Remaining Potential Success Factors after systematic reduction

5.2 Quantification of Potential Success Factors

In order to allow for the collection of quantitative data (as described in 5.3), the 18

remaining Potential Success Factors had to be clearly defined on rating scales. The

factors were defined as either ordinal data in three groups, or nominal data in two groups.

The suitable use of these scales resulted from trial runs with some participants in the

company. For the factors measured in three groups it was originally intended to apply a

five point Likert type scale, with the categories: Very Low, Low, Moderate, High, Very

High. However, it was recognized that this scale provided difficulties in deciding between

Very Low or Low and High or Very High. The concern for too much noise in the data

using the five categories lead to the decision to only use three: Low, Medium and High.

It was confirmed during the data collection that the interviewees were able to respond

comfortably to these three categories. The same ’trial and error’ approach was applied

to the factors which ended up to be measured in two categories.

In the following sections, the 18 Potential Success Factors and their respective rating

descriptions are explained.

5.2.1 Use of Design Methodology

The rating scale definitions for this Potential Success Factor are shown in Table 5.1.

Applying design methodologies is widely found to be an important factor for success in

the product development domain (Krause et al., 2006; Lindemann, 2009; Mehalik and

Schunn, 2006; Schregenberger, 1998). Pahl and Beitz define methodology as ”concrete

course of action...to achieve general and specific goals” (Pahl et al., 2007, p.10). Different

methodological movements evolved in the industrialized countries in the scond half of the

20th century. These methodologies might appear different on a detailed level, but they

all have in common that they regard the product development process to consist of four

main phases (with varying terminology): Definition of Task; Conceptual Design; Basic

Design; Detail Design. The methods themselves vary within these phases, but the aim

is always the same: to provide design engineers with the appropriate tools to master the

particular challenges of each respective phase.

Table 5.1: Rating definitions for ’Use of Design Methodology’

High Design methodology used completely or mostly throughout the development

process

Medium Design methodology used occasionally or at different stages of the development

process

Low Design methodology used rarerly or not at all during the development process

5.2.2 Conceptual Design performed

Being usually - but not necessarily - part of an overall methodology, the Conceptual

Design Phase is emphasized by some authors as crucial for successful product develop-

ment (French, 1999; Pahl et al., 2007; Ullman, 2003). In this phase, a series of stages

are proposed to be performed. The most common of these are: breaking the system or

problem into sub-systems; generate multiple solution ideas for the sub-systems using dif-

ferent (creativity) techniques; systematically down-select to the most suitable solutions;

evaluate solutions on a system level before moving on to basic design. The definitions for

rating to what extent coneptual design was utilized in the investigated projects, is shown

in Table 5.2.

Table 5.2: Rating definitions for ’Conceptual Design performed’

High Conceptual design with all or most of its stages performed

Medium Partial stages of conceptual design performed

Low Very little or no conceptual design performed

5.2.3 Decision Tools utilized

Table 5.3 shows the rating scale definitions for this Potential Success Factors. Similar

to conceptual design, the systematic application of decision tools can be, but is not

necessarily applied as part of an overall design methodology. Instances can be found

where the decision making process almost solely determines the progress of a development

project. Morgan and Liker found that one of Toyota’s cornerstones of success in the design

process, is the use of engineering checklists throughout the development cycle to support

decision making. This can lead to cases were decisions have to be postponed due to

incomplete information in the decision checklists. As a result, Morgan and Liker claim

that ”delaying decisions at Toyota leads to faster overall product development” (Morgan

and Liker, 2006, p.65).

Table 5.3: Rating definitions for ’Decision Tools utilized’

High Decision tools utilized consequently in each or most stages of the development

process

Medium Decision tools utilized in some stages of the development process

Low Decision tools rarely or not at all utilized during the development process

5.2.4 Resources provided / Project Stability

This Potential Success Factor includes internal, as well as external influences, meaning

it varies based on decisions made outside of the engineering department. Although en-

gineering management distributes resources within its domain, upper-level management

decisions can change the course of action in a way - e.g. by changing the product strategy

or cutting the budget - that engineering management has no other choice than to real-

locate resources. Especially on products with development cycles of several years, this

impact on product stability can occur frequently as research and development budgets

usually are evaluated and redistributed at least once a year. By setting a strategy and

vision that supports the stable execution of development projects over the whole cycle,

upper-management is believed to have a significant impact on the outcome of development

projects (Ehrlenspiel, 2009; Gausemeier et al., 2009; Schimmoeller, 2010).

Table 5.4: Rating definitions for ’Resources provided / Project Stability’

High Throughout project: no or very minor changes in budget or core team member

commitment to project

Medium Throughout project: moderate changes in budget or core team member com-

mitment to project

Low Throughout project: large or complete changes in budget or core team member

commitment to project

5.2.5 Priority of Project

Table 5.5 shows the rating scale definitions for this Potential Success Factors. The

project’s priority refers to the status the develoment project has within the engineering

domain. Due to the amount of tasks being executed in a global engineering organization,

engineering management has to set priorities on which projects to give more attention

than others. The priority setting derives from the need to balance different stakeholders

interests, such as upper-managment directive, internal or external customers. Fricke and

Shenbar point out that

Division and assignment of resources, prioritization, and customized manage-

ment style, which have little relevance in relation to single projects, are shown

to play a major role in the success of multiproject management. (Fricke and

Shenbar, 2000, p.258)

Table 5.5: Rating definitions for ’Priority of Project’

High Project had high priority in the engineering department

Medium Project had medium priority in the engineering department

Low Project had low priority in the engineering department

5.2.6 Team Size

One would assume that the size of a development team is a function of other factors,

such as the complexity of the project. Such relationships would have to be verified if

the Team Size showed to be related to the successful outcome of projects. In the first

step, however, it was only of interest if smaller teams caused a different effect than

larger teams did. The impact of teams on project performance can most commonly

be found in the literature under the collective term ’Team Composition’ (Ernst, 2002;

Schimmoeller, 2010). However, bundling this factor into a one-dimensionsal measure

appeared to be difficult. It was actually an allocated projection in the Potential Success

Factors: ’Team Size’, ’Team Members Experience / Skills’, ’Cultural Influence in Team’

and ’Simultaneous Development’. Team sizes vary with the industries and products. The

size definitions for small, medium and large project teams were derived from typical team

size conventions (rule of thumb) used for gas turbine component development projects.

Table 5.6: Rating definitions for ’Team Size’

High Team consisted of 13 or more core team members

Medium Team consisted of 6-12 core team members

Low Team consisted of 1-5 core team members

5.2.7 Team Members Experience / Skills

Similar to Team Size, the requirements for experience level and skill set of the team mem-

bers varies, depending on the industry and product. Engineering management might

accept teams with less average years of experience for the development of software or

consumer goods - this might even be desired, for staying connected to the trends of the

market - than for complex physical products, such as power plants or airplanes, which

require very experienced teams (Lindemann, 2009). The development of gas turbines

demands highly specialized skill sets, which explains the relatively ’stretched’ rating defi-

nition intervals chosen for the three categories, shown in Table 5.7. The skill set is related

to work experience in this case, as close to all engineering team members enter the gas

turbine development domain possessing related advanced engineering degrees.

Table 5.7: Rating definitions for ’Team Members Experience / Skills’

High Average team experience of 15 or more years in gas turbine development

Medium Average team experience of 8-14 years in gas turbine development

Low Average team experience of 7 or less years in gas turbine development

5.2.8 Cultural Influence in Team

It showed to make sense to measure this Potential Success Factor binominal, dividing

teams with members of one culture from teams with members of more than one culture,

as shown in Table 5.8. Defining more categories on an ordinal scale, e.g. measuring

amounts of cultures involved, wouldn’t have provided valuable information, as a change

in group dynamic usually occurs between culturally homogeneous and culturally hetero-

geneous teams (Thomas, 1999). It would have also meant that the distances between

the categories was highly unproportional, which would have made the use of quantita-

tive methods difficult. Dividing between no cultural influence and any cultural influence

provides the highest level evaluation, suitable for a first analysis step. If the factor was

found to be of significance, it could be investigated in more detail in a subsequent step.

Baumg¨artner and Blessing (1999) studied project-related differences in the design practice

of two different cultures (Italy vs. Germany). They found that the differences lie often-

times in soft factors which are not always obvious. The research proposed that although

things are done in a different way, they can still lead to the same success. Finally, the

authors recongized a trend to more common practices in executing development projects

between cultures, as more and more companies are forced into international collaborations

or take-overs.

Table 5.8: Rating definitions for ’Cultural Influence in Team’

High Core team members from more than one culture

Low Core team members from one culture

5.2.9 Experience / Skills of Project Lead

Project leads and their personalities are believed to play an important role in successsful

design work (Ehrlenspiel, 2009; Pahl et al., 2007). As it showed impossible to combine

the complexity of this influence into one factor, it was divided into two Potential Success

Factors, one measuring the experience and skills, the other measuring personality traits.

Table 5.9 shows the definitions for categories on experience and skill levels of project

leads.

Table 5.9: Rating definitions for ’Experience / Skills of Project Lead’

High High degree of technical and project lead experience; has lead several similar

development projects in respective technical area and environment before

Medium Moderate degree of technical and project lead experience; has lead few similar

development projects in respective technical area and environment before

Low Low degree of technical and project lead experience; has not lead similar de-

velopment projects in respective technical area and environment before

5.2.10 Project Lead Personality

Measuring personality types has traditionally been an area of interest in psychology.

Different models have been developed over time. The Myers-Briggs personality test has

been established as the most used evaluation (Quenk, 2009). Founded on Jung’s model

of typology (Jung, 1971), the test is a standard instrument in assessment and recruiting

centers. The model suggests that a humans personality can be defined by four basic

categories with pairs of opposite traits: Introverted vs. Extroverted; Sensing vs. Intuitive;

Thinking vs. Feeling; Judging vs. Perceiving. While this model might not directly be

applicable to traits of successful project leads, three - to these personality dimensions

related - traits can be found in the literature (Ehrlenspiel, 2009; Pahl, 1992; Pahl et al.,

2007):

a) Communication

b) Decision Making

c) Risk Awareness

The opposites were defined for the rating, as shown in Table 5.10. Although no person

can be located clearly on one or the other end of these scales for the traits, the interviewed

managers were, in general, able to make a judgement using these definitions.

Table 5.10: Rating definitions for ’Project Lead Personality’

10a: Project Lead Personality - Communication

High Strong/active communicator

Low Subtle communicator

10b: Project Lead Personality - Decision Making

High Quick decision maker

Low Takes time to make decisions

10c: Project Lead Personality - Risk Awareness

High Continuously/often evaluates risks and corrects course of project if necessary

Low Rarerly/never evaluates risks and corrects course of project if necessary

5.2.11 Level of Process Compliance

Processes are essential in large organizations (Morgan and Liker, 2006) to define clear

responsibilities, coordinate activities and ensure the adequate flow of information. The

product development process in the company of the study consists of different review

and quality gates over the complete lifecycle of a product (Figure 1.9), starting with a

Product Requirement Specification Review and ending with a Field Validation Review

after several years of operation of the product. In the area of engineering design, three

process steps are required to be completed with a review when developing a new gas

turbine:

1. Conceptual Design Review: review of usually two to three alternative solutions,

which were elaborated in this phase on a high level and shall be pursued in the

basic design phase.

2. Basic Design Review: review of the basic design work, including analysis, feasibility

studies and prototype tests. The design team presents the proposal for a go-forward

solution and the respective technical justification.

3. Final Design Review: review of the detailed and completed design work, including

product definitions (drawings and bill of materials). A successful Final Design

Review results in the product release for production.

For component development projects (e.g. combustion system upgrades) however,

it is not a requirement to follow all these three process and review steps. This is done

conciously to allow for flexibility in the process. The design team itself is allowed to make

the call on how many reviews are required, weighing complexity and risk against justified

process effort at the beginning of each project. All component development projects have

to have a final review conducted upon completion of the Final Design Phase. The rating

scale definitions shown in Table 5.11 measure the level of process compliance in how many

of these reviews were conducted.

Table 5.11: Rating definitions for ’Level of Process Compliance’

High All three design process steps (Conceptual, Basic and Final Design) followed

and completed with official reviews

Medium Two out of three process steps followed and completed with official reviews

Low Only final process step (Final Design) completed with official review

5.2.12 Simultaneous Development

The rating definitions for this Potential Success Factor were aligned with the design phases

(Conceptual, Basic and Final Design) of the product development process (Table 5.12).

Simultaneous development is the concept of shortening development cycles by performing

as many design tasks in parallel as possible, rather than sequential (Krause et al., 2006).

This means developing and evaluating concepts in parallel, building prototypes early and

- most importantly - involving downstream stakeholders (manufacturing and suppliers)

from an early stage on. In an ideal case, good coordination does not only lead to shorter

development time, but also improves quality by supporting the detection of ’downstream

issues’ early on. Evaluating this factor against the success dimension ’Timeline Met’ it

was found that projects where simultaneous engineering was utilized from the beginning

had a higher chance of being completed on time.

Table 5.12: Rating definitions for ’Simultaneous Development’

High Integration of manufacturing and suppliers from the Conceptual Design Phase

Medium Integration of manufacturing and suppliers from the Basic Design Phase on

Low Integration of manufacturing and suppliers in the Final Design Phase

5.2.13 Product Requirements clearly defined and stable during

Development

A clear understanding and definition of product (customer) requirements is one of the

most commonly found conditions for success in product development. Any of the classic

literature sources about engineering design refer to it with emphasis (Akao, 1990; Ehrlen-

spiel, 2009; Pahl et al., 2007; Pugh, 1991; Ullman, 2003). An additional potential influnce

on success, which was found from interviews with engineering managers and project leads,

is the ’transient’ change of product requirements over the course of a project. It was re-

ported that issues occured, in cases where the requirements were not continuously checked

and if necessary updated.

This factor is somewhat specific to the object of the study. Power companies (customer)

usually hire purchasing engineers with many years of design experience in power gener-

ation products. Unlike on consumer products, the chance for significantly missing the

customers needs is low. The customer’s purchasing engineers know and define almost ex-

actly what they need, which determines the product requirements of a new gas turbine.

The component development team has to translate these overall requirements into prod-

uct requirements for the components. Researching different projects in the engineering

database showed that some projects had the requirements clearly defined in the project

charters, while others didn’t. For the projects that showed clear definitions, it was deter-

mined through interviews if there were any transient changes in the product requirements

during the project, which were not recorded. If both the conditions - clear definition and

stability of requirements - were fulfilled, the Potential Success Factor was rated High. If

one of them was not fulfilled, it was rated Low. Table 5.13 shows the rating definitions

for the specification and stability of product requirements.

Table 5.13: Rating definitions for ’Product Requirements clearly defined and stable during

Development’

High Product requirements clearly defined, continuously checked and updated dur-

ing development project

Low Product requirements not clearly defined, or continuously checked and updated

during development project

5.2.14 Awareness of Lessons Learned and State of the Art Tech-

nology Knowledge

Poor knowledge transfer, from internal or external (competitors or the scientific com-

munity), can be found to have a negative impact on success in new product develop-

ment (Knudsen, 2007; Lindemann, 2009). Many different things have oftentimes been

tried out over the years within a company and valuable lessons have been learned. These

might not be well documented. Especially when teams of younger engineers are in charge

of projects, they might face issues which could have been prevented, if it was known what

was done in the past. The same is true for knowledge about what has been done in the

technical field external to the company, which can be referred to as ’State of the Art

Technology’. The rating definitions are shown in Table 5.14 for this Potential Success

Factors.

Table 5.14: Rating definitions for ’Awareness of Lessons Learned and State of the Art

Technology Knowledge’

High No issues occurred due to lack of awareness of lessons learned or state of the

art technology knowledge

Low Issues occured due to lack of awareness of lessons learned or state of the art

technology knowledge

5.2.15 Ratio of new Technology

Unlike the Potential Success Factor ’Awareness of Lessons Learned and State of the Art

Technology Knowledge’, this factor measures the amount of new technology that has

not been used or developed before. It is basically a measure of how much innovation is

involved in the development project. Some projects require a low amount of innovation,

for instance routine upgrades of existing products with plenty of operating experience.

Others require the use of completely new design solutions and technologies, such as

when the company wants to take a leap by developing a product with the aim to excel

current competitor’s products significantly. Table 5.15 shows the rating definitions for

this Potential Success Factor.

Table 5.15: Rating definitions for ’Ratio of new Technology’

High Development of product with high amount of new technology compared to

other development projects executed in department

Medium Development of product with medium amount of new technology compared to

other development projects executed in department

Low Development of product with low amount of new technology compared to other

development projects executed in department

5.2.16 Technical Complexity of Project

The technical complexity is a measure for the size of the project, the amount of people

involved and the interfaces that have to be considered. Complex projects can be expected

to have a higher need for state of the art technology knowledge or innovation. But this

does not necessarily have to be the case. A project might involve a lot of resources

due to its size, but it might in its nature be more of a routine task, which has been

done before and is mostly known. This does not exclude problem solving activities, e.g.

finding solutions to work around geometric contrainst, but it won’t require completely

new technologies. For this reason, this factor needs to be measured separately from the

two formerly introduced Potential Success Factors ’Awareness of Lessons Learned and

State of the Art Technology Knowledge’ and ’Ratio of new Technology’. In order to

provide a basis for judgement, the population of component development projects in the

gas turbine development department served as reference for defining what low, medium

or high complexity meant. Having a reference was necessary, as otherwise interviewees

would have had trouble to decide how the projects should be rated. Table 5.16 shows the

definitions for the measurement of a project’s complexity.

Table 5.16: Rating definitions for ’Technical Complexity of Project’

High

High technical complexity in comparison to other projects in department. High

amount of: changes and developments that involve problem solving; interfaces

to disciplines within and outside of component

Medium

Moderate technical complexity in comparison to other projects in department.

Moderate amount of: changes and developments that involve problem solving;

interfaces to disciplines within and outside of component

Low

Low technical complexity in comparison to other projects in department. Low

amount of: changes and developments that involve problem solving; interfaces

to disciplines within and outside of component

5.2.17 Co-Location of Team Members

A rapid increase in the use of global teams can be observed in technology firms (Mc-

Donough et al., 2001). Van den Bulte and Moenaert (1998), as well as Kahn and Mc-

Donough (1997), found evidence of improved R&D performance in their investigation of

the effect of co-location. The company’s engineering department in this study was dis-

tributed over six locations on three continents (Europe, North-America, Asia). Due to

that, many project teams consist of core members being in different locations. The differ-

ence to the Potential Success Factor ’Cultural Influence in Team’ is, that team members

from different cultures can still be co-located. On the other hand, teams consisting of

members from one culture could potentially be distributed over more than one location.

In other words, co-location measures the geographical separation of project teams, not

the composition. The rating definitions are shown in Table 5.17.

Table 5.17: Rating definitions for ’Co-Location of Team Members’

High Core Team Members all in one location

Low Core Team Members distributed over two or more locations

5.2.18 Empowerment of Project Lead: Project and Budget Re-

sponsibility in one Hand

Project leads in engineering are responsible for delivering a specified product at a certain

time, within the allocated budget. While they always own technical responsibility for

the project outcome, budget ownership might be in other hands in the company, e.g.

Product Line Management or Marketing. Empowerment means that project leads have

the technical responsiblity, but are also awarded with the authority to manage the de-

velopment budget, without having to get permissions for any project related expenses

from other institutions in the company. Ehrlenspiel (2009) considers this kind of project

lead empowerment as crucial for Design Success. Table 5.18 shows the rating definitions

for the Potential Success Factor ’Empowerment of Project Lead: Project and Budget

Responsibility in one Hand’.

Table 5.18: Rating definitions for ’Empowerment of Project Lead: Project and Budget

Responsibility in one Hand’

High Project and budget responsibility in one hand (project lead)

Low Project and budget responsibility in different hands

5.3 Data Collection and Validation for Potential Suc-

cess Factors

The data points for the Potential Success Factors were collected from the engineering re-

view database in the first step. The project charters and the archived review documents

provided information about: timeline, budget, resources provided, methodologies, design

tools used, team sizes, location of core team members, level of process compliance and def-

inition of product requirements. Missing information in the database was gathered from

interviews with engineering managers, project leads or team members of the respective

projects. Information about team members and project leads skills/experiences, project

lead personalities, priorities, complexity, amount of innovation and awareness of lessons

learned were directly obtained by interviewing engineering or project managers. Most

of these data points were collected from single sources, due to the limited availability of

the interviewees. Only a few cases, where interviewees expressed to be uncertain about a

rating, required an inter-rater reliablility check. This was done by interviewing a second

source with sufficient project knowledge. The ratings coincided for these few instances.

From the original 50 projects considered for investigation, 44 provided the complete set of

data. For the other six projects, it was not possible to obtain all data points. The reason

was insufficient documentation in the review database, or missing sources for interviews.

It was decided to remove these incomplete data sets from the study and continue with

the remaining 44 projects. The alternative would have been to interpolate the missing

data. However, this technique bears risks (Kitchenham and Pfleeger, 2003), especially

when cross-checking of dependencies among the causes is desired to be performed.

Before the actual analysis, the data matrix of the 44 projects and 18 Potential Success

Factors was screened for the distribution of the data. In order to be able to identify singi-

ficant influences on Design Success, each Potential Success Factor had to show variance in

the categories (Low, Medium or High) for the 44 projects. Potential Success Factor that

did not show this variance had to be considered constant and were not suitable for the

analysis. In order to determine when a Potential Success Factor was considered constant,

a criteria was needed. As explained in the later Data Analysis Chapter, the analysis

method (Kruskal-Wallis Method) used for the first analysis step requires a minimum of

five data points per category. The five data points were used as the treshold. For the

Potential Success Factors which were measured in three categories (Table 5.19), it was

required that at least two of the categories showed more than five data points. If this

condition was fulfilled, the data was used for the analysis. The second Potential Success

Factor ’Conceptual Design performed’ is such a case. The 44 projects showed only a

low or medium level of conceptual design performed. However, both categories showed

sufficient projects (low = 27, medium = 17). This data could be used to determine if

there was a significant difference in impact on the succesful outcome between projects

where a low versus a medium level of conceptual design was applied.

Of the Potential Success Factors measured in three categories, ’Use of Design Methodol-

ogy’ and ’Resources Provided / Project Stability’ harmed the criterion of more than five

data points in at least two categories. These had to be considered constant and their

significance could not be determined. Although these two factors were not involved in

the further analysis, it was still possible to draw some conclusions.

First constant Potential Success Factor: Use of Design Methodology

Almost all of the projects showed only a low application of design methodologies. It could

not be determined if this factor belonged to the Dominant Success Factors. However, it

was known that this factor was not the only dominant factor. If such had been the case,

there wouldn’t have been variation in the success of the 44 projects as it is seen (Fig-

ure 4.2). The variance in the success data indicated that there are other (at least one)

Table 5.19: Amount of data points in each category for Potential Success Factors mea-

sured in three categories

Potential Success Factors Low [n]Medium

[n]

High

[n]

1 Use of Design Methodology?* 40 4 -

2 Conceptual Design performed? 27 17 -

3 Decision Tools utilized? 15 22 7

4 Resources Provided / Project Stability?* 1 4 39

5 Priority of Project? 9 18 17

6 Team Size? 7 27 10

7 Team Members Experience / Skills? 6 26 12

9 Experience / Skills of Project Lead? 1 20 23

11 Level of Process Compliance? 20 14 10

12 Simultaneous Development? 9 20 15

15 Ratio of new Technology? 13 14 17

16 Technical Complexity of Project? 13 13 18

*to be considered as constant

Dominant Success Factors. Although it could not be proven if it is a Dominant Success

Factor, it has to be concluded that more use of design methodology could have elevated

the success rate of projects, as the literature commonly suggests this positive relation-

ship (Krause et al., 2006; Lindemann, 2009; Mehalik and Schunn, 2006; Schregenberger,

1998).

Second constant Potential Success Factor: Resources Provided / Project Sta-

bility

The fact that 39 out of 44 projects had good project stability showed that a lack of

resources was not an issue in the company. In addition, it showed that upper-level and

engineering management made it a priority to keep projects stable and frequent reallo-

cation of resources low. Similar to the use of design methodology, it was not possible to

determine if this factor belonged to the Dominant Success Factors. But, contrary to the

Table 5.20: Amount of data points in each category for Potential Success Factors mea-

sured in two categories

Potential Success Factors Low / No

[n]

High / Yes

[n]

8 Cultural Influence in Team? 19 25

10a Project Lead Personality - Communica-

tion? 30 14

10b Project Lead Personality - Decision

Making? 30 14

10c Project Lead Personality - Risk Aware-

ness? 28 16

13 Product/Customer Requirements clearly

defined and stable during Development? 13 31

14 Awareness of Lessons Learned and State

of the Art Technology Knowledge? 7 37

17 Co-Location of Team Members? 7 37

18 Empowerment of Project Lead: Project

and Budget Responsibility in one Hand? 7 37

first constant factor, it had to be concluded that Design Success would have likely been

lower, if this factor was not constantly high.

The Potential Success Factors which were measured in two categories did not show any

data cell with less than five data points (Table 5.20). The last three factors (14, 17 and

18) had each had seven projects in the Low categoy and 37 projects in the High category.

This pattern suggested that these factors were directly related, meaning that the seven

projects in the Low catogory represented the same projects for all three Potential Success

Factors, and the 37 projects in the High category the same projects for all three Potential

Success Factors, respectively. Looking at the data revealed that there was a difference in

projects for these three factors (Appendix A) and the similar data points in the categories

were just coincidence.

6 Data Analysis

The objective of analyzing the data quantitatively was at first to understand which of

the Potential Success Factors showed a relationship to Design Success - then identified

as Success Factors. Secondly, it was of interest to understand if and how the Success

Factors identified depended on each other. The dependencies, if any existed, would be

beneficial for determining causal relationships. These two objectives were addressed in

two subsequent analysis steps using suitable statistitical techniques for each.

6.1 First Analysis Step: Search for Success Factors

The first analysis step was aimed at detecting relationships between successful product

design and each of the Potential Success Factors. One statistical method for testing

the dependency of multiple independent variables on a dependent variable is multiple

linear regression. However, this method demands the dependent variable (in this case

Design Success) be normally distributed (Garson, 2012). As Figure 4.2 shows, this is

not the case for the collected data. Alternatively, a hypothesis test can be applied for

each independent variable (the Potential Success Factors) against the dependent variable

(Design Success). The statistical methods used for hypothesis testing are introduced in

this chapter.

The Null-Hypothesis states that there is no significant difference between the sample

groups of each Potential Success Factor:

•Null-Hypothesis for two sample groups: H0:µLow =µHigh

•Null-Hypothesis for three sample groups: H0:µLow =µMedium =µHigh

The finding of a significant difference would necessarily lead to a rejection of the Null-

Hypothesis. This case would mean that there was enough evidence in the data to believe,

that what was seen can be repeated on a different sample from the same population,

although with a certain chance of error. The significance level as a criteria, and the re-

lated chance for error, is provided by methods of inferential statistics. Using methods of

inferential statistics demands caution when it comes to types and distribution of the data.

6.1.1 ANOVA for Hypothesis Test

It was intended to perform the hypothesis test using the ANOVA (ANalysis Of VAriance)

method. The ANOVA is a technique, particularly friendly to experimental data, for com-

paring the means of two or more sample groups (Sirkin, 2006). The sample groups are the

categories (Low, Medium or High) for each Potential Success Factor, where the means for

each group results from the independent variable, the rating of success. Due to the eval-

uation of one independent variable, this test is also referred to as the One-Way ANOVA.

Using the ANOVA, the goal would be to understand if there were a significant difference

between the sample groups, considering the means and their variances. The measure of

significance is the p-value (from probability value). The p-value is a dimensionless factor

in the range of 0 to 1. It represents the probability that a difference in sample groups is

due to chance. The p-value can be derived if the F distribution - which results from the

F ratio - is known. The F ratio is determined by the variance within the sample groups

in relation to their degrees of freedom, and the variance between the sample and their

degrees of freedom. The variances can be expressed by the sum of squares, where the

total sum of squares (SStotal) is the sum of squares within (SSwithin) the sample groups

and the sum of squares between (SSbetween) the sample groups:

SStotal =SSwithin +SSbetween (6.1)

Assuming ksample groups and ndata points within each sample group, SSwithin can

be written as

SSwithin =

i=1

j=1

(yij −¯yi)2(6.2)

with yij being any of the ndata points in any of the ksample groups, and ¯yibeing

the mean of each of the ksample groups. Knowing the overall mean of all sample points

¯y,SSbetween can be written as

SSbetween =

i=1

n(¯yi−¯y)2(6.3)

which leads to

SStotal =

i=1

j=1

(yij −¯yi)2+

i=1

n(¯yi−¯y)2.(6.4)

With 6.2 and 6.3 and the degrees of freedom (df) for SSwithin and SSbetween

dfwithin =n−k(6.5)

and

dfbetween =k−1 (6.6)

the variances can be expressed as the mean of squares within the groups

MSwithin =SSwithin

dfwithin

(6.7)

and the mean of squares between the groups

MSbetween =SSbetween

dfbetween

.(6.8)

The F ratio results as

F=MSbetween

MSwithin

.(6.9)

The F distribution is a non-symmetric, right skewed curve. Its shape depends on

the degrees of freedom dfbetween and dfwithin. The total area under the curve is always

1. Looking at 6.9, it becomes obvious that the significance in difference between sample

groups increases with an increasing MSbetween and a decreasing MSwithin. Hence, larger

F ratios signify a higher probability that the sample groups do not come from the same

population and a higher likelihood that the Null-Hypothesis has to be rejected. At what

point it has to be rejected, depends on the chosen α-level. The α-level is the remaining

chance for error. If the F ratio lies within the α-level (grey area under the curve in

Figure 6.1), the Null-Hypothesis has to be rejected. Every value of the F ratio has a

p-value associated with it. The p-value stands for the probability of chance in the data

and can directly be evaluated against the chosen α-level. In case the p-value is smaller

than the α-level, the Null-Hypothesis has to be rejected. Figure 6.1 shows an example

of a F distribution for α=0.05, with the critical value for the F ratio and the associated

p-value for dfbetween=2 and dfwithin=12. In this case a F ratio of less than 3.89 would

suggest that the sample groups come from the same population and that the data had

to be considered random. On the other hand, for a F ratio of larger than this value, a

significant difference in the sample groups would be the conclusion, as the probability of

chance was less than the chosen significance level.

It is common practice in scientific studies to regard differences in sample groups as signif-

icant, if the p-value is <0.05. This would mean that the chance of error is less than 5%.

Or, in other words, one would have a 95% confidence to reject the Null-Hypothesis and

would only expect a different result in one out of 20 tests, if repeatedly testing different

samples from the same population. The significance level might be chosen more (e.g.

0.01) or less (e.g. 0.1) stringently, which is dependent on the sensitivity and potential

impact of the results. For empirical studies in the medical or pharmaceutical area, signif-

icant levels that are usually chosen are more conservative (smaller), as severe harm could

result from a wrongly rejected Null-Hypothesis. In this research, a significance level of

equal or smaller than 0.1 was chosen. The reasons for this limit are elaborated during

the discussion of the analysis results.

Figure 6.1: Example of F distribution with α-level = 0.05

Since the ANOVA technique is based on mean values and their relative variances,

the sample groups must show normal distribution in their data, with similar variances.

Due to this underlying requirement, the ANOVA belongs to the parametric tests. Before

the ANOVA could be applied to the data, it needed to be verified if the data fulfilled

the conditions of normal distribution and similar variances. If this was not the case, the

ANOVA could not be used and a non-parametric alternative method had to be found

instead.

6.1.2 Testing for Normal Distribution

Different methods are available for the verification if data points in the sample groups are

normally distributed. A box plot is a descriptive graph that provides a visual illustration

of the distribution patterns, showing the four quartiles of each group. Figure 6.2 illus-

trates a box plot for the three sample groups (Low, Medium and High) of the Potential

Success Factor ’Simultaneous Development’, which measures the degree of concurrent

design and manufacturing activities. The horizontal line in each grey box represents the

median, which should be similar (or close to similar) to the mean value, if the data were

normally distributed. The grey rectangles below and above the mean, and the vertical

lines below and above the box, represent each a quartile, which is a quarter of the sample

group’s data points. By looking at the shape of the boxes, it can already be detected

if the data sample groups have similar variances. If groups had similar variances, the

boxes should be of similar size. In addition, if the data was normally distributed, the

horizontal line in the grey box should be approximately centered in the box. In the case

of ’Simultaneous Development’, it was clear that this is not the case. From looking at

the distribution of the sample population (Figure 4.2), one would have already expected

that not all sample groups showed similar distribution patterns.

Figure 6.2: Box plot for verification if data is normally distributed

As can be concluded from Figure 6.2, the parametric ANOVA method could not

be used for the first analysis step. Instead, a non-parametric method was required for

determining the significance of relationships between the Potential Success Factors and

Design Success. The Kruskal-Wallis method was found as an alternative to the ANOVA.

6.1.3 The Kruskal-Wallis Method as alternative for ANOVA

The Kruskal-Wallis Method is a non-parametric analog to the parametric One-Way

ANOVA analysis, which can be used for two or more sample groups (Kruskal and Wallis,

1952). Instead of the means, it is based on the ranks of the data. Using ranks instead

of means has advantages, which make this method very useful for research problems,

especially common in the social and behavioral sciences. It is suitable for data measured

on an ordinal scale and it allows for sample groups of different sizes, with a required

minimum of five data points in a sample group. Most importantly, it does not assume

that the population or the sample groups are normally distributed. Rather it makes no or

only a very general assumption about the distribution of the data (Chan and Walmsley,

1997). This was a helpful attribute in the case of this research, as it has been shown

in 6.1.2 that the collected data cannot be considered normally distributed for all sample

groups.

While the ANOVA is based on the F statistic, determined from the means, the Kruskal-

Wallis Method is based on the H statistic, determined from the ranks. The concept,

however, is the same. If the H value exceeds a certain critical value (similar to the F

ratio in the ANOVA as shown in Figure 6.1), the Null Hypothesis is to be rejected. The

higher the value for H, the more likely it is that the sample groups, or at least one of

them, come from different populations. The H statistic is defined as:

H=12

N(N+ 1)

i=1

−3(N+ 1).(6.10)

A different formulation of 6.10 is

H=Pk

i=1 ni(¯

Ri−¯

R)2

N(N+1)

(6.11)

where

k= the number of sample groups

ni= the number of data points in the ith sample group

N=Pnithe number of observations in all samples combined

Ri= the mean of the sum of the ranks in the ith sample group

R= the mean of the sum of the ranks of N.

This Formulation (6.11) illustrates the analogy of the Kruskal-Wallis Method to the

One-Way ANOVA. Just as in the F ratio (6.9), the expression in the numerator is the

sum of squares between the sample groups. The only difference is that the One-Way

ANOVA uses the means while the Kruskal-Wallis Method uses the ranks of the data.

The denominator is expressed by the mean of the Nranks, which is N(N+1)

12 .

The distribution of H is a close approximation of the Chi-Square distribution for df =

k−1, which is especially true for sample sizes of at least five in each group (Kruskal and

Wallis, 1952). Figure 6.3 shows a Chi-Square (χ2) distribution for a sample data with two

degrees of freedom (df = 2). Similar to the F ratio, the higher the value of H (χ2), the

more likely it falls within the chosen α-level and the Null Hypothesis has to be rejected.

Figure 6.3: Chi-Square distribution for df = 2

6.1.4 Results of the First Analysis Step

The results of the analysis for relationships between the Potential Success Factors and

Design Success are shown in Table 6.1 for the Potential Success Factors measured in three

groups. Table 6.2 shows the results for the Potential Success Factors measured in two

groups.

Potential Success Factors measured in three groups

The second, third and fourth column in Table 6.1 show the means of the success mea-

surement for the three categories - Low, Medium and High. The two factors identified as

constant were excluded from the analysis. Although the probabilities were determined

with the H statistic from the ranks, the mean values are shown here, as they help to il-

lustrate the trends better. As can be seen in Table 5.19, there is obvious variation in the

sample sizes of the different groups. It was important that the analysis method provided

robustness against varying group sizes, which is what made the Kruskal-Wallis Method

suitable for the kind of data collected in this study. The fifth column shows the p-value

for each Potential Success Factor.

Assuming a α-level = 0.1, there are five factors which show a significant result and allow

for rejection of the Null-Hypothesis. These factors are:

•Conceptual Design performed

•Priority of Project

•Experience / Skills of Project Lead

•Ratio of new Technology

•Technical Complexity of Project

The limit of equal or smaller of 0.1 was chosen because of the gap to the next higher

p-value of 0.36. The data showed quite a leap between the five factors having a p-value

equal or smaller of 0.1 and the other factors. This pattern was seen as justification for

chosing this limit. Although not as stringent, it still means that out of ten different

samples drawn from the same population, nine can be expected to show a similar result.

Whenever the Kruskall-Wallis Test indicates significance, it means that at least one

out of the three groups is significantly different than and not from the same population

as the other groups. It does not tell which of the groups is significantly different. In a

worst case scenario, the Medium group would be significantly different, but the Low and

High group would be on a similar level. Such a scenario wouldn’t allow for the conclusion

of either a positive or negative relationship between the Potential Success Factor and the

success of the design project. What is of interest is a significantly increasing or decreasing

trend over the three groups, respectively.

The box plot of the Success Factor ’Conceptual Design performed’ is shown in Figure 6.4.

As only data for two groups existed (Low and Medium) for this factor, the significance is

unambiguous. The data shows that there is an inverse relationship between conceptual

design and Design Success. It is shown in section 6.3 that this unexpected result can be

rationalized with help of the second analysis step. Figure 6.5 shows the box plot of the

determined Success Factor ’Priority of Project’, measured against the Design Success.

The increasing trend of the three groups is obvious. As the projects get higher priorities

by engineering management, the output becomes better. It can be noticed that the

Low group shows a high variance, which is due to the projects ranked with zero success

(projects terminated before completion). Since the analysis method is rank based, this

gap in the data distribution does not matter. After all, it could have been decided to rate

terminated projects with a three instead of a zero. Due to the conversion to ranks, the

resulting p-value would still be the same, but the variance in the graph would be lower,

Table 6.1: Significance of Potential Success Factors measured in three groups

Potential Success Factors Low [¯yi]Medium

[¯yi]

High

[¯yi]p-value

1 Use of Design Methodology? constant -

2 Conceptual Design performed? 7.4 5.9 - 0.1

3 Decision Tools utilized? 7.1 6.8 6.1 0.65

4 Resources Provided / Project Stability? constant -

5 Priority of Project? 4.9 6.7 8 0.01

6 Team Size? 8.1 6.7 6.1 0.36

7 Team Members Experience / Skills? 7.2 6.3 7.8 0.51

9 Experience / Skills of Project Lead? 5 6.1 7.5 0.04

11 Level of Process Compliance? 6.8 6.5 7.4 0.63

12 Simultaneous Development? 5.9 7.2 6.9 0.72

15 Ratio of new Technology? 7.6 7.5 5.6 0.09

16 Technical Complexity of Project? 7.5 7.9 5.5 0.02

more similar to the Medium and High group.

The graphical illustration of the found Success Factor ’Experience / Skills of Project

Lead’ against the success, is shown in Figure 6.6. The Low group consists of only one

data point and can be neglected. The reason for only one data point is specific to the

product of this study. Gas turbines are technically sophisticated, high cost products,

which are being built and sold in low volumes. Design errors can lead to most serious

safety issues in power plants and enormeous economic damage to the company or the

customer. These circumstances promote the ’habit’ of putting generally experienced

engineers into project lead positions. The Medium and High groups consisted of similar

amounts of data points. The graph shows clearly the higher success rates of projects

which have highly experienced and skilled project leads.

Figure 6.4: Impact of conceptual design on Design Success

Figure 6.5: Impact of priority for engineering management on Design Success

Figure 6.6: Impact of project lead’s experience and skills on Design Success

Figure 6.7 shows the box plot for the Success Factor ’Ratio of new Technology’, which

basically measured the degree of innovation. The box plot of the fifth Success Factor in

the three group category - the technical complexity of the project - is shown in Figure 6.8.

It can be seen that for both these Success Factors, the Low and Medium categories were

on about the same level. The High category was ranked significantly lower. Although

the two graphs don’t show a steadily decreasing relationship over the three groups, this

factor was regarded as significant, as the High group was significantly lower (two success

scale points in the mean) than the other two groups. The Low and Medium groups were

treated as one group being evaluated against the High group. With these two Success

Factors having similar patterns, it was expected that a dependency existed between them.

The second analysis step would provide the verification for this assumption.

Figure 6.7: Impact of new technology on Design Success

Figure 6.8: Impact of project’s complexity on Design Success

Potential Success Factors measured in two groups

The second and the third columns of Table 6.2 illustrate the Low versus the High group.

For the Potential Success Factors answered with Yes or No, No is referred to in the Low

column, Yes is referred to in the High column. The p-value is shown in the fourth column.

Table 6.2: Significance of Potential Success Factors measured in two groups

Potential Success Factors Low / No

[¯yi]

High / Yes

[¯yi]p-value

8 Cultural Influence in Team (No

Team Members from one culture, Yes

Team Members from more than one

culture)?

7.7 6.1 0.08

10a Project Lead Personality - Communica-

tion? 6.6 7.2 0.48

10b Project Lead Personality - Decision

Making? 6.5 7.5 0.53

10c Project Lead Personality - Risk Aware-

ness? 6.7 7 0.6

13 Product/Customer Requirements clearly

defined and stable during Development? 6.8 6.8 0.95

14 Awareness of Lessons Learned and

State of the Art Technology Knowl-

edge?

2.4 7.6 0.01

17 Co-Location of Team Members (No

TMs in different locations, Yes TMs

in same location)?

5.3 7.1 0.03

18 Empowerment of Project Lead: Project

and Budget Responsibility in one Hand? 6.9 6.8 0.52

Three Success Factors - with p-values equal to or smaller than 0.1 - were identified for

the Potential Success Factors measured in two groups, which are:

•Cultural influence in Team

•Awareness of Lessons Learned and State of the Art Technology Knowl-

edge

•Co-Location of Team Members

The box plot of the Success Factor ’Cultural Influence in Team’ is shwon in Fig-

ure 6.9. The analysis of this factor compares the performance of teams with members

from one culture against the performance of teams with members from different cultures.

The result suggested an inverse relationship, meaning that teams with members from the

same culture performed better. Again, the second analysis step was needed to verify and

correctly interpret this finding. Figure 6.10 shows the box plot of the determined Suc-

cess Factor ’Awareness of Lessons Learned and State of the Art Technology Knowledge’.

Figure 6.11 shows the box plot of the third determined Success Factor ’Co-Location of

Team Members’. It can be seen in Table 5.20 that the sample groups for the awareness of

lessons learned and the co-location factor had quite varying sample sizes. For these, the

robustness of the analysis method was tested by drawing 7 data points randomly from the

37 data points in the High/Yes group. Comparing the reduced High/Yes sample group to

the Low/No group showed a similar significance. This test was repeated three times with

similar results, which proved that the Kruskal Wallis Method was indeed robust against

varying sample sizes.

As with the Success Factors found from the Potential Success Factors measured in three

groups, the question about the causality and dependency of other Success Factors re-

mained. Knowing these helped to provide answers on how design teams can improve

their performance, especially in an international environment. Analyzing dependencies

between Success Factors, required a different statistical method for hypothesis testing,

as the data of these factors was of nominal nature. As discussed in section 6.2, the

Chi-Square Test was found to be suitable for this second analysis step.

Figure 6.9: Cultural influence on Design Success

Figure 6.10: Impact of lessons learned and technology knowledge on Design Success

Figure 6.11: Impact of co-location on Design Success

Success Factors after first analysis step

In total, eight Success Factors with a significant relationship to successful product design

were determined in the first analysis step. Figure 6.12 shows the relationships graphically.

The plus and minus signs indicate the nature of the correlation, which can be verified

from the medians in the box plots or the means in Table 6.1 and Table 6.2. For the factors

indicated with pluses, an increase in such lead to an increase in Design Success. For the

other four, indicated by minus signs, the correlations worked in the opposite direction.

After knowing the eight Success Factors, some questions remained when it came to defin-

ing measures for engineering management for better prediction of Design Success. For

instance, which Success Factors can be considered independently and which have to be

viewed in the context of others? Are there Success Factors which cause others to change,

but are not caused by other Success Factors to change? This nature of dependency is

referred to as the direction of causality. If in the end, out of the eight Success Factors

found in analysis step one, the ones were identified, which not only impacted the outcome

of a design project, but also influenced other Success Factors to change, they had to be

considered the Dominant Success Factors. Only the results of the first analysis step and

the results of the second analysis step combined, provided the necessary information to

make this distinction.

Figure 6.12: The eight Success Factor determined in the first analysis step

6.2 Second Analysis Step: Search for Dependencies

The first analysis step revealed the eight Success Factors (Figure 6.12), which showed a

significant impact on Design Success. The findings suggest that these influencing factors

should get special attention during the set-up and execution of development projects.

However, the results of analysis step one did not give any indication of whether these

Success Factors were dependent on each other. Knowing dependencies, if any existed,

could provide more information and allow conclusions to be drawn about causal rela-

tionships. For instance, a dependency check on the Success Factors ’Priority of Project’

and ’Experience / Skills of Project Lead’ could lead to two scenarios, which would allow

for different conclusions. In the first scenario, the dependency check wouldn’t show any

significance. The conclusion would be that management could have increased the chance

for success, by selecting more highly experienced and skilled project leads for their high-

est priority projects. The second scenario could be, that a dependency existed between

the two Success Factors. This would be an indication for a causal relationship between

the factors. The question would then be, in which direction the causality pointed. In

the case of this example, it would be obvious that the priority was the causal root, as

management determines the priority of projects and also selects the project leads. The

elaboration of causalities and their possible explanations are part of the result interpre-

tation in section 6.3. The Potential Success Factors that did not show significance in

analysis step one were not considered for further analysis. It was excluded that any of

these were dependent on any of the eight Success Factors, as otherwise a relationship to

Design Success would have been seen in the first analysis step.

The input for the second analysis step were the eight Success Factors, which were quan-

tified in either three ordinal or two categorial groups. Alternatively, these data types can

be considered as categorial, with three or two groups respectively. A suitable method

for determining dependencies between factors measured in categories is the Chi-Square

Test (Pyzdek and Keller, 2009). This test is one of the most basic and versatile statistical

tests. The concept of the test is to compare expected counts against observed counts.

The Chi-Square (χ2) Test, also known as Pearson’s Chi-Square Test, is defined as:

χ2=

i=1

(Oi−Ei)2

(6.12)

where

n= number of compared groups

Oi= observed count for each compared group

Ei= expected count for each compared group

Similar to the H statistic, the p-value can be determined by comparing the calculated

value to the χ2-distribution with the respective degrees of freedom df =n−1.

The comparison matrix in Figure 6.13 shows the p-values for the dependencies between

the eight Success Factors found from the Chi-Square Test in analysis step two. Significant

dependencies are indicated by the bold font. As in the first analysis step, a p-value equal

to or smaller than 0.1 was used as significance limit. A guideline for reliable results of the

Chi-Square Test is that ”No more than 20% of the expected counts are less than 5 and

all individual expected counts are 1 or greater” (Yates et al., 1999, p.734). The p-values

marked with an asterix do not comply with this guideline, which is why caution was

required in interpreting these. Looking at the distributions of the observed counts for

Figure 6.13: P-values for dependencies between Success Factors

each pair helped to verify if a significant dependency really existed. Figure 6.14 shows an

example of a table of observed counts for the Chi-Square Test between the Success Factors

’Conceptual Design performed’ and ’Co-location of Team Members’. It can be seen that

the project teams, which utilized a low level of conceptual design, were almost all co-

located. This ratio changed however, for the projects with a medium level of conceptual

design. Significantly less of these project teams were co-located compared to the total of

the category. Knowing these patterns provided the information to determine the direction

of causalities for all Success Factors that showed dependencies. In this example, it can

be concluded that the direction of causality went from co-location to conceptual design,

meaning that a less in co-location lead to a more in conceptual design. This would make

sense. After all, co-location was the factor established before the design work was started.

Obviously, it would mean that co-location caused the effect of a different behaviour in

terms of the use of conceptual design. Before that final conclusion could be drawn, the

question had to be answered if co-location by itself was a real cause, or if it was caused

by another Success Factor to change. It did not seem logical that the co-location was

the root cause factor that would influence the use of conceptual design. The step by

step interpretation of the dependencies, described in the following, established this full

picture, which eventually lead to the Dominant Success Factors. The tables of observed

counts for each pair of Success Factors are shown in Appendix B.

Figure 6.14: Example of observed counts for Chi-Square Test

6.3 Interpretation of Results

With the Success Factors and their dependencies known, it was possible to map the

causalities. A dependency map was created step by step, interpreting each Success Factor

and its direct dependencies, and connecting them subsequently.

Conceptual Design performed

The first Success Factor showed a significant dependency to ’Awareness of Lessons Learned

and State of the Art Technology Knowledge’ and ’Co-location of Team Members’. These

three factors were mapped, as shown in Figure 6.15. The nature of the dependencies

were verified from the tables of the observed counts and are indicated by the plus and

minus signs in the dependency map. It can be seen that an increase in the awareness of

lessons learned and an increase in co-location had the inverse effect of lowering the level

of conceptual design applied. The direction of causality is indicated by the direction of

the arrow head. As mentioned above, it was not known at this moment if co-location was

the root cause for a different level of conceptual design. It had to be assumed at that

point, that co-location itself was caused to change by another, more dominant factor,

which is why no arrow heads are drawn yet. The relation between conceptual design and

the awareness of lessons learned was inverse in nature as well. It was concluded that the

latter is the cause which lead to a change in level of conceptual design. A reasonable

explanation is, that the projects where technology knowledge was high, were more of

routine tasks and the teams apparently felt less need for the use of methods.

After the first step of mapping, it became obvious that ’Conceptual Design performed’

was not a Dominant Success Factor, because it was caused by other Success Factors to

change, indicated by an arrow pointing towards it or being connected to another Success

Factor with a line without an arrow head.

Figure 6.15: Dependency map, first step

Priority of Project

This Success Factor did not show any significant dependency to any other Success Factor.

It had to be considered independent, which was a surprising finding. One would have

assumed that the priority which management gives to projects, would be related to their

complexity and the amount of new technology. However, it was not found to be the

case, which can be seen as an indication that too many development projects were in the

engineering pipeline. The apparent effect was that engineering management was not able

to give the highly complex and innovative projects the priority, which they likely needed

for better success. In most of the observed cases, highly complex and innovative projects

were completed successfully, whenever the management priority was high. As this Success

Factor had no dependencies to other Success Factors, it was not caused by any of these to

change. This means that it directly influenced Design Success as determined in analysis

step one and can be considered as Dominant Success Factor. The relationship to Design

Success was mapped as shown in Figure 6.16.

Experience / Skills of Project Lead

Similar to the management priority of the project, this Success Factor did not show

any significant dependency on other Success Factors. Again, one would have expected

that a significant dependency existed with highly complex and innovative projects. But

apparently not all of these projects had a highly experienced and skilled project lead.

As with ’Priority of Project’, for most projects which were lead by a highly experienced

and skilled project lead, even high complexity and innovative content did not prevent

them from being successful. Looking at the data matrix in Appendix A, it can be seen

that there were projects with a high ratio of new technology and a high complexity that

were completed with high success. What these had in common was that the management

priority and experience of the project lead were high. At most, one of the two Success

Factors was at a medium level. Lower success resulted, whenever none of the two Success

Factors was high, or when one of the two Success Factors was low. The projects on the

lower end of the success scale apparently suffered from this mismatch. It likely means that

the engineering department exceeded its capacity limit, which would have been the point

where it could give all its highly complex and innovative projects a high management

priority and a highly experienced project lead. At least one of the two factors had to be

high, the other medium. At the point where this condition cannot be met anymore, due to

a lack of experienced and skilled project leads or focus on priority, management should be

aware that the chance for success is greatly reduced. Since no other dependencies existed,

’Experience / Skills of Project Lead’ was mapped to Design Success (Figure 6.16) and

considered a Dominant Success Factor.

Figure 6.16: Dependency map, second step

Ratio of new Technology and Technical Complexity of Project

A very strong dependency existed between these two Success Factors. As can be seen

in the data matrix in Appendix A, the two Success Factors had identical categories in

almost all of the projects, which means that complex development projects had a high

level of innovation as well. It suggested to combine the two Success Factors into one,

as dependencies to other Success Factors were expected to be similar. The p-values

in Figure 6.13 confirm that both Success Factors had similar dependencies and shared

the same significant dependencies, which were to ’Cultural Influence in Team’ and ’Co-

location of Team Members’. The dependency between the combined Success Factor and

the cultural influence changed in similar direction. Projects with lower innovation and

complexity had lower cultural influence. This can be explained by the circumstance that

as projects get more complex, more resources are required, which results in project teams

distributed more globally. Technical complexity and innovation was the dominant factor

which caused a more in cultural influence. Since more complex projects had a lower

chance for success, the factor of cultural influence related to lower success rates as well.

Equally, this causal relationship existed between the complexity and the co-location fac-

tor. Lesser complexity lead to more co-location. Converesly, more complexity lead to

project teams which were less co-located and distributed over the the different global

engineering locations.

Figure 6.17 shows the Success Factors added in this third mapping step together with

their dependent factors. Since the combined Success Factor - measuring the technical

complexity and the level of innovation - did only cause other Success Factors to change,

but was not caused by other factors to change, it was mapped directly to Design Success

Figure 6.17: Dependency map, third step

and was identified as a Dominant Success Factor.

Cultural Influence in Team

Interpreting the dependency to the combined factor of innovation and complexity re-

vealed, that the negative impact of cultural influence on Design Success was a symptom,

that was triggered by higher complexity. This could be proven by looking at projects

of similar complexity, where it was seen that cultural influence had no influence on the

project outcome. The similar but inverse dependency of the co-location to the complex-

ity factor, explains why cultural influence and co-location showed a singificant inverse

dependency as well. However, none of the two caused the other to change, as both were

caused to change by the combined and dominat factor of complexity and innovation.

Since no causal relationship existed, no arrow heads are shown between cultural influence

and co-location in Figure 6.18.

Awareness of Lessons Learned and State of the Art Technology Knowledge

Only one dependency to conceptual design existed. Since no other dependency existed

and the awareness of lessons learned was determined to cause a change in the amount

of conceptual design applied, it meant that it had to be considered a Dominant Success

Factor. Figure 6.18 shows the added dependency to Design Success.

Figure 6.18: Dependency map, fourth step

Figure 6.18 still contains an inconsistency with respect to the Success Factor ’Con-

ceptual Design performed’. The factor did not show a significant dependency to the

dominant innovation and technical complexity factor. However, this dependency must

exist, as the co-location factor, which changed similar to conceptual design, was caused

to change by this dominant factor. Looking at the Chi-Square Matrix in Figure 6.13,

it can be seen that the dependency between conceptual design and innovation (ratio of

new technology) had a significance of p = 0.12, which is just slightly above the limit of

0.1. Similarly, the dependency between conceptual design and the cultural influence was

with p = 0.14 not too much above the singficance limit either. This dependency must

exist as well, although without any causal relationship. Although the defined statistical

treshold was exceeded, it was decided to consider these two dependencies significant as

well, which then lead to a conclusive dependency map. The tables of the observed counts

confirm that a trend for these dependencies existed in the data. The higher p-values have

to be explained with variation in the data.

Figure 6.19 shows the completed depenceny map, with the two dependencies above the

significance limit being indicated by the dashed lines.

Figure 6.19: Completed dependency map

6.4 Results for Secondary Success Dimension

As explained in section 4.1, the secondary success dimension ’Timeline met’ had to be

analyzed indirectly by evaluating it against the primary success dimension ’Product Re-

quirement Fullfilment’. Using the same analysis method (Kruskal-Wallis Method) as in

the first analysis step of the primary success dimension, the secondary success dimension

served as independent variable. The four ’zero projects’ were not included in this analy-

sis, as they had no completion date to be measured, due to their termination. The box

plot in Figure 6.20 graphically illustrates that there is a relationship between projects

which delivered a better fulfillment of product requirements and their completion in time.

With a p-value = 0.03 (shown in Table 6.3), it can clearly be stated that this relation

is significant. From that it has to be concluded that Success Factors which have a posi-

tive impact on Design Success in terms of product requirements, similarly cause a better

chance for completion of projects in time.

Figure 6.20: Relation between ’Product Requirement Fullfilment’ and ’Timeline Met’

The Potential Success Factor ’Simultaneous Development’ was not found to be a

significant Success Factor to product requirement fulfillment. However, the data showed a

positive relationship to the secondary success dimension, which proved that Simultaneous

Engineering helps to shorten development cycles.

Table 6.3: Significance of secondary success dimension

Secondary success dimension No [¯yi](n) Yes [¯yi](n) p-value

Timeline Met? 6.5 (10) 7.8 (30) 0.03

6.5 Correlation of Findings to Literature

The four Dominant Success Factors (factors with an arrow pointing to Design Success

in Figure 6.19) identified in this chapter have been investigated by other researchers and

can be found referenced in the Engineering Design literature as follows:

Priority of Project: The positive impact of management giving priority to develop-

ment efforts is frequently cited, as Schmimoeller (2010) found in his meta-analysis about

success factors in product development. Many authors refer to this factor and some

aspects of it, like providing the necessary resources and attention over other projects,

but also defining goals and sharing a vision that emphasizes the priority and importance

of development projects (Ehrlenspiel, 2009; Gausemeier et al., 2009; Morgan and Liker,

2006; Pillkahn, 2007; Schimmoeller, 2010). Fricke and Shenbar (2000) found, that, while

the prioritization is not of great relevance in a single project environment, it plays ”a

major role in the success of multiproject management” (Fricke and Shenbar, 2000, p.258).

Experience / Skills of Project Lead: Level of experience and the related skills

of project leads that evolve from it, have been recognized in the literature to contribute

greatly to development success (Ehrlenspiel, 2009; Lindemann, 2009; Pahl et al., 2007).

Ehrlenspiel (2009) refers to the Heuristic Competency as an important trait of successful

design engineers and project leads. This multi-dimensional trait is described as the ability

to solve problems that extend the available base of knowledge at a certain time (Ehrlen-

spiel, 2009). The results of this research confirm, that having the experience from having

been in similar situations before, contributes to this trait. Other researchers indicate

another aspect of a project lead’s experience, which is the political capital one obtains

over time in an organization. Chollet et al. (2012) concluded, that experienced project

leads, who have led successful development projects before, can make more powerful use

of their internal network in order to get things done.

Technical Complexity and Ratio of new Technology: The negative impact

of complexity and innovation on project outcomes has been discussed in the literature.

Ehrlenspiel (2009) points out the challenges and risks of having to coordinate tasks of

a larger scope with more interfaces. In addition to the increased coordination effort,

authors refer also to the uncertainty related to higher complexity and innovation. The

uncertainty oftentimes results in an application of inappropriate measures for managing

the projects, resulting in lower success (Chalupnik et al., 2009; de Weck et al., 2007).

Awareness of Lessons Learned and State of the Art Technology Knowl-

edge: Ullman describes this awareness as leading to a ”knowledge of fact” (Ullman,

2003, p.52) over a knowledge of possibilities during the design process. This includes

learning internally from past mistakes (Krause et al., 2006), but also from external part-

ners or competitors. Knudsen (2007) found the importance of learning from partners and

competitors on a broader range, extending the area of a companies own core technology.

Lindemann (2009) put additional emphazise on the importance of creating an environ-

ment, where this knowledge is shared and known within the organization. Research efforts

addressing the significance of knowledge sharing, can be found in the fields of Knowledge

Management (Goh, 2002) and Organizational Behaviour (Argotea and Ingramb, 2000).

The research approach described in this dissertation is based on a hypothesis test

of Potential Success Factors, which were found in the Engineering Design literature.

That means that all factors - whether they showed significance in the applied empirical

study or not - show a correlation to the literature about successful product development.

However, as concluded from this chapter and described in more detail in the next chapter

(Applicability), the four Dominant Success Factor have a primary impact. Only if these

factors are existent in the right balance at first, can the other factors contribute as well.

7 Applicability

Figure 7.1 illustrates the results of analyzing the 18 Potential Success Factors as causes

for Design Success. The two step analysis approach of the collected data revealed that

out of the 18 Potential Success Factors, eight showed a significant relationship to Design

Success. Out of these eight Success Factors, five were detected to be Dominant Success

Factors. Complexity and the amount of new technology were combined into one fac-

tor ’Technical Complexity and Innovation’, due to similarity in ratings for each project,

so that four Dominant Success Factors remained. The three Success Factors that were

caused to change are considered Supplementary Success Factors. Their detected relation-

ship to Design Success was only an indirect effect of the dependencies to the Dominant

Figure 7.1: Result of analysis of all 18 Potential Success Factors

Success Factors, hence their impact is much weaker. Still it has to be recognized that

they can help to improve Design Success in a supplemental manner. The same is valid

for the ten Potential Success Factors, which did not show a significant correlation to the

success. These are considered Supplementary Success Factors as well.

In this chapter the applicability of the research results to engineering companies is

discussed. It is proposed how engineering management should focus on the Dominant

Success Factors. The role of the Supplementary Success Factors in the design process

is discussed as well. The last two sections illustrate the limitations of this study and

recommendations for further research.

7.1 Dominant Success Factors

The four Dominant Success Factors, which were determined in section 6.3, are character-

ized by two criteria:

1. Show a significant relationship to Design Success.

2. Are not caused by any other Success Factor to change.

Whenever these criteria were fulfilled, it had to be concluded that a Potential Suc-

cess Factor was a direct cause of successful product design. The four Dominant Success

Factors are shown in Figure 7.2. This square of Dominant Success Factors represents

a condensed overview of the factors recommended to engineering management to focus

most. The factor ’Awareness of Lessons Learned and State of the Art Technology Knowl-

Figure 7.2: Square of all four Dominant Success Factors

edge’ should always be fulfilled. Measures should be in place, allowing the company to

utilize the available knowledge in the best possible way. It also means that a good sys-

tem for knowledge transfer from external resources - conferences, competitor data and

the latest academic research - should be established.

The other three factors however, have to be viewed in context. They allow to be adjusted

and do not necessarily have to be on the highest level for each specific project. Rather

they form a triangle (Figure 7.3) which needs to be in balance in order to achieve high

success. Knowing this, can help the organization to decide how much of a ’leap’ it possi-

bly can take with respect to technical complexity and level of innovation.

Figure 7.3: Triangle of Dominant Success Factors which need to be balanced

The balance of the triangle gives engineering management some flexibility in how prior-

ities and resources are to be distributed. But it also provides an indication about the

maximum capacity of the engineering department and the risk level for projects. Ta-

ble 7.1 shows the possible combinations between the priority and project lead experience

for a high level of complexity and innovation. The combinations of the first three cases

are desired. It requires at least one of the two Dominant Success Factors to be high and

the other to be medium. The data showed that it did not matter which one was medium

and which one was high. However, as soon as both the priority and the experience of the

project lead, were only medium, the chance for successful product design declined singif-

icantly. Similarly, this was the case when one of the two factors was low, independent of

what the other was.

Engineering management is advised to evaluate the balance of the triangle of Dominant

Success Factors (Figure 7.3), whenever a new development project is intiated. The fol-

lowing two questions need to be asked for any project with an expected high complexity

and innovation level:

1. How much priority and attention can management give the project?

2. What’s the experience and skill level of the available project lead(s)?

The case where the requirements as they are shown in Table 7.1 cannot be met,

would be an indication that the maximum capacity level of the engineering department

is reached. In order to bring the triangle back into balance in such a scenario, engineering

management has the following options:

•Adjust complexity: The complexity and innovation level of the project might allow

for adjustment. It is not always a question of the end goal, which needs to be

met by the solution, but the way taken to achieve the goal. An example in gas

turbine technology is the control of emission levels. Achieving low emission levels

in the combustion process itself requires highly complex and innovative technologies.

An alternative solution with lower complexity could be the treatment of emissions

external to the gas turbine, simply by adding a catalyzer downstream of the turbine

exhaust duct. This could be a solution to meet more stringent emission levels by

using an established, less complex technology. Medium to low complexity projects

were less sensitive to the other two Dominant Success Factors of the triangle and

yielded significantly higher success rates, as can be seen in Table 6.1.

•Re-prioritize: Management can evalaute the balance on all the other development

projects ongoing in parallel. There might be projects with medium or low complex-

ity, where the priority and attention level can be reduced, so that management can

shift the focus to a new, highly complex project.

•Change project lead: Similar to the re-prioritization, all other, ongoing projects can

be evaluted for their balance. It might be possible that a lower complexity project,

which is lead by a highly experienced engineer, is handed to a less experienced

project lead.

If none of these alternative options are possible, engineering management should be

warned that the maximum of projects is reached, where successful outcome can still be

expected. It marks the point where the organization should consider reducing the amount

of projects. This could either mean not starting any new development projects, until some

of the ones in the pipeline are completed, or stopping ongoing projects in order to be able

to start new ones.

Table 7.1: Cases for balancing the triangle of Dominant Success Factors

Priority of

Project

Experience /

Skills of Project

Lead

Technical

Complexity and

Innovation

Design Success

High High High %

Medium High High %

High Medium High %

Medium Medium High &

High Low High &

Low High High &

7.2 Supplementary Success Factors

All Potential Success Factors which were not identified as Dominant Success Factors are

considered Supplentary Success Factors. Altough the Dominant Success Factors have a

much stronger impact on success, it still has to be expected that these factors have an

additional effect to some extent. In total 13 Supplementary Success Factors remained.

Ten of them were detected in the first analysis step as they did not show a significant

relationship to Design Success. Three were detected in the second analysis step, where it

was revealed that these were caused to change by the Dominant Success Factors.

7.2.1 Supplementary Success Factors detected in the first anal-

ysis step

The role of these factors in the design process can be interpreted as:

•Use of Design Methodology: Almost all projects investigated showed a low

utilization of design methodologies, which is why the factor had to be considered

constant. The absence of formal methodologies could support the finding of Jensen

and Andreason, that the best suitable methods are ”outcomes of local interactions

100

and sense-making processes in the companies” (Jensen and Andreasen, 2010, p.28),

rather than general, pre-defined schemes. This would suggest that methodologies

were used as part of the daily business in the company, but in a more subtle, less

detectable way. Nevertheless, the results suggest that - similar to conceptual design

- a more in use of design methodology could have helped to increase Design Success,

provided that the Dominant Success Factors were in place and in the right balance.

•Decision Tools utilized: The outcome of the 44 development projects was mini-

mally influenced by the extent to which decision tools were used during the product

development process. In general, a rather low utilization of design methodologies

and related tools was found in the projects investigated. Again, it suggests that the

overall success rate for all projects could have been elevated, had more systematic

design been applied, in addition to having the Dominant Success Factors in the

focus.

•Resources Provided / Project Stability: Almost all projects showed that this

factor was not an issue, which is an indication that the company had sufficient

resources available. The bottle-neck was apparently not the overall amount of

resources, but the amount of experienced and skilled project leads.

•Team Size: No significant influence on Design Success derived from the size of a

project team. Even projects with high complexity showed varying team sizes. Some

of these projects with smaller team sizes were still completed successfully. All these

had in common that the Dominant Success Factors were sufficiently in place. The

conlusion to draw from this pattern is, that team size is not as important. A highly

complex project can still be tackled successfully, as long as it gets the right priority

and an experienced project lead.

•Team Member Experience / Skills: Similar to the team size, no significant

pattern beetween success and the experience of the teams was found. It suggests

that when an experienced project lead is in place and/or the project has sufficient

priority, successful teams are assembled, regardless of their experience.

•Project Lead Personality: Communication, Decision Making, Risk Aware-

ness: None of the three dimensions of this Potential Success Factor showed sig-

nificance. This finding appeared unreasonable at first, as the experience and skill

of the project lead shows a strong correlation to Design Success. However, it does

support the importance of the Heuristic Competency. The Heuristic Competency is

a multi-dimensional, personality independent trait, resulting in the ability to solve

problems which extend the available base of knowledge at the time (Ehrlenspiel,

2009). The findings suggest that the experience and skills of the project lead in

101

the technical field, rather than personality traits, is the significant aspect of this

competency.

•Simultaneous Development: In terms of product requirement fulfillment, no

difference was found between the projects where manufacturing was involved very

early in the process, and the ones where it happened later. However, it was found

that the projects where a high degree of simultaneous development was applied, had

a higher chance to be completed in time. Apparently, the utilization of Simultaneous

Engineering does support shorter development cycles.

•Level of Process Compliance: All projects went through the minimum re-

quired process step (Detail Design Review after design completion). Some projects

went through additional, not mandatory steps. No difference was seen between

the projects which went through all PDP process steps and the ones which only

went through the required minimum. This suggests that process steps should be

kept to a minimum. It does not suggest that a process is completely irrelevant, as

a minimum was always there. A minimum amount of process is always required,

especially in global organizations, to ensure proper flow and documentation of infor-

mation. However, as with most other Supplementary Success Factors, with enough

management priority and an experienced project lead, a low level of this factor can

be overcome.

•Product / Customer Requirements clearly defined and stable during De-

velopment: No indication of a different outcome was found between the projects

that had clearly defined customer and product requirements in their documentation

and the ones which did not. Again, this suggests that other factors, such as the

project lead’s experience and priority given by management, are able to overcome

a lack of this factor.

•Empowerment of Project Lead: Project and Budget Responsibility in

one Hand: Very low significance, with a similar explanation as for the former

Potential Success Factor.

7.2.2 Supplementary Success Factors detected in the second

analysis step

The role of these factors in the design process can be interpreted as:

•Conceptual Design performed: More conceptual design was performed on the

projects with higher complexity and innovation, which is a sign that the project

teams were aware that methods can be used to overcome more complex problems.

102

The negative relationship to Design Success resulted from the circumstance that

more complex projects had a lower success rate. It cannot be concluded that the

use of conceptual design lead to less success. Rather the opposite has to be as-

sumed, since at most, only a medium level of conceptual design was applied, even

on the highly complex and innovative projects. Using a high amount of conceptual

design could have increased the chance for success on these. However, it has to

be recognized, that this effect can only occur if the Dominant Success Factors are

in place with the right balance. Only if the project teams are aware of lessons

learned and the latest technology trends, and the triangle of priority, project lead

experience and complexity is in balance, can conceptual design help additionally to

improve Design Success.

•Cultural Influence in Team: The effect of projects with higher complexity was

that the amount of global involvement increased, due to the need to utilize more

resources. This caused an increase in cultural influence in teams, which is why this

factor showed an indirect relationship to Design Success as well. The results suggest

that if the Dominant Success Factors exist, cultural differences can be overcome.

•Co-location of Team Members: Similar to the cultural influence, this factor

was a symptom resulting from higher complexity and can be overcome when the

Dominant Success Factors are sufficiently existent.

As described for most of the factors above, showing statistical insignificance or only

an indirect relationship to Design Success must not be interpreted that these factors do

not contribute to successful product development. It is certainly to be believed that

techniques like design methodologies or the clear definition of product requirements help

to improve development projects. However, the results of this study show that there

are more dominant factors which need to be fulfilled first. Project teams need to be

aware of the latest technology trends and the lessons which have been learned in the

own company. For highly complex projects, experienced and skilled project leads are

essential. In addition, projects need to be given sufficient priority by management. Once

these factors are established, the Supplementary Success Factors can help design teams

in achieving their goals better. But they are only able to do so, if the Dominant Success

Factors are established with the right balance in the first place.

The Pareto Chart in Figure 7.4 shows in a qualitative manner the weight of the Dominant

Success Factors and the additional impact the Supplmentary Success Factors can have,

as it was found in this study.

103

Figure 7.4: Role of Dominant and Supplementary Success Factors

7.3 Limitations of this Study

It was the aim of this study to comply with the characteristics as they were elaborated in

the Literature Review Chapter and deﬁned in Table 2.3. This Comprehensive Approach

was seen as to providing results of high reliability and practicality, which can directly

be used by engineering management in the industry. The complexity of the task and

the limited time frame of this research left some of these ’ideal’ characteristics unfulﬁlled.

These unfulﬁlled characterisitics from Table 2.3, which mark the limitations of this study,

are:

•Unit(s) of analysis: A holistic view on the product development process was in-

tended, in order to ensure all factors, potentially inﬂuencing the outcome of design

projects, were considered. While the developed framework (Figure 3.5) supported

the determination of all inﬂuencing factors, it was also required that all factors

showed suﬃcient variance in success over the 44 projects. Only with this variance

was it possible to apply the statistical methods, which helped to detect signiﬁcances

of relationships. Two of the 18 Potential Success Factors did not show this variance

and had to be considered constant. These factors were the use of design method-

ology, as well as the factor measuring provided resources and project stability. As

four other Dominant Success Factors existed, it was known that these two could

not be the only Dominant Success Factors. The low use of design methodologies

suggests that a more in such could have yielded better results, but this remains a

hypothesis. The other constant Potential Success Factor - provided resoureces and

project stability - showed a high existence on almost all the investigated projects.

104

It suggests that a lower average success resulted, had this factor been lower. But

again the lack of variance in the data did not allow for a quantitative verification.

•Number of cases: The 44 projects of the study provided a sample size which allowed

for the use of inferential statistics for most of the cases. However, the fact that

two of the Potential Success Factors turned out to be constant and some of the

dependency tests did not comply with the guidelines for reliable Chi-Square Test

results (indicated by the asterix in Figure 6.13), is an indication that more projects

would have been required to obtain complete comprehensiveness.

•Object: The projects of the study were all performed in the same global organi-

zation. Although the Dominant Success Factors are general in their nature and

are hence believed to apply to all product development projects executed in similar

organizational environments, it needs to be pointed out that influences specific to

the company or the product could not be filtered out.

•Coding and analysis method(s): While it was the goal in this research to rely solely

on statistical significances, it was not possible to achieve in the second analysis

step. The multifold of possible combinations for the dependency tests showed that

44 projects were not sufficient to provide results with the desired confidence and

reliability. On two dependencies, personal judgements from studying the trends in

the distribution of the counts could not be avoided.

The described limitations of this study in combination with the determined results,

lead to recommendations for further research in Engineering Design Research.

7.4 Recommendation for Further Research

The four Dominant Success Factors identified in this study require more focus in upcom-

ing research activities in Engineering Design Research. As explained in section 5.1, an

iterative approach was chosen in this research in order to be able to reduce the amount

of data to a manageable size. As a result, some of the eventual 18 Potential Success

Factors, represented groups of closely related factors. An example are elements of con-

ceptual design, such as breaking the problem into sub-functions or the use of creativity

techniques, which were grouped under the Potential Success Factor ’Conceptual Design

performed’. This approach allowed - considering the limited time frame of the research -

to achieve the detection of the most dominant influences, by maintaining a holistic view

and without compromising on comprehensiveness. For the Dominant Success Factors

found, it means that more refined iterations can help to understand these factors and

their impact in more detail.

105

Priority of Project

The role and importance of management can be found mentioned in the product devel-

opment literature. Authors refer to the support in terms of providing resources, defin-

ing goals and sharing a vision (Ehrlenspiel, 2009; Gausemeier et al., 2009; Morgan and

Liker, 2006; Pillkahn, 2007; Schimmoeller, 2010). However, there is still a lack in under-

standing, especially how engineering management can contribute to successful product

development, beyond the task of allocating resources. Further research is recommended

to identify specific solutions which can help engineering management to set its priorities

and focus correctly. It is referred to the management sciences, like NPD and Innova-

tion Management, where many publications with focus on the role of management and

their relation to Design Success can be found (Bonner et al., 2002; Schimmoeller, 2010;

Thamhain, 2003; Zirger and Maidique, 1990).

Experience / Skills of Project Lead

Project leads of engineering development projects play an essential role in the outcome of

development projects. The results of this research suggest that personality traits that are

related to the role of a project lead do not matter. It does not make a difference if a person

is a strong or subtle communicator, quick or slow decision maker, or shows a passive rather

than an active sense of risk awareness. What does matter are the skills these persons

posses in the technical field and the experience they have gained from leading similar

projects formerly. The exact causes why experienced project leads are more successful

need to be understood, which is why more specific research is recommended. It might be

what Ehrlenspiel refers to as Heuristic Competency (Ehrlenspiel, 2009), but could also

be other factors, such as the fact that due to their seniority in the company, experienced

project leads can make better use of social capital (Chollet et al., 2012).

Technical Complexity and Ratio of new Technology

Although the complexity of a project and the level of innovation is oftentimes given by

external factors, like competitors products, some flexibility exists. Alternative solutions

which have been proven for a longer time might be available. A company needs to

be aware of how much of a leap it can take in terms of complexity and innovation.

Evaluating the balance with the other Dominant Success Factors, especially the project

lead’s experience and management priority, is a helpful indication as shown in section 7.1.

More research is recommended in order to provide companies with intergrated approaches

on how to tie the technological planning - e.g. by technology road mapping - to the human

side.

106

Awareness of Lessons Learned and State of the Art Technology Knowledge

Good knowledge transfer and transparency is important for companies to prevent avoid-

able and costly mistakes. Companies need methods helping them to ensure knowledge

from the past and present is known and accessible, for both knowledge from internal and

external sources. Technology can contribute partially to the solution for this challenging

task, e.g. through smart databases for storing information. However, the human side

needs to be considered as well in that collaborative and transparent environments are

created. Specific research in this area has been performed in the fields of Knowledge

Management (Goh, 2002) and Organizational Behaviour (Argotea and Ingramb, 2000),

which Engineering Design Research can build upon.

In the field of Engineering Design Research, it is recommended to follow a more

rigorous methodology, aspiring towards comprehensiveness and the use of hard criteria.

Therefore, the Comprehensive Approach developed in this research can serve as a frame-

work. As shown in Section 3.1, this framework complements DRM (Design Research

Methodology). To verify the robustness of the approach, studies with larger populations

across different companies or industries are desireable. More data would lead to more

reliable results, and also help to verify the results of this study. The extended study ob-

ject (different companies and different industries) would overcome the limitation of this

study, with projects from only one company. This would allow for even more generalized

results. It would mean an extensive effort and require active collaboration with industry

partners, but provide extremely valuable information to the Engineering Design Research

community.

It has to be mentioned again, that the Supplementary Success Factors are not insignif-

icant and can help to increase Design Success. Further investigation of these is desired

as well. However, researchers have to be aware - especially when researcheing them in

an indepedent manner - that there are factors that are more dominant over these, when

they act in a holistic context. Researchers can neutralize this dependency by consider-

ing sample projects for a study that share the same, constant rating for the Dominant

Success Factors.

107

8 Conclusion

This dissertation comprises an empirical investigation on what distinguishes succesful

from less successful product development projects in engineering design. Studying the

state of the art literature and results of past research in the field of Engineering Design

Research revealed the need for further work. The specific needs were formulated in three

research questions at the beginning of this dissertation. The answers to these questions

mark the contribution to the academic research, as well as to the beneficiaries of the

results, which are engineering and technology firms.

Research Question I: How do the characteristics of an Empirical Engineering

Design Research study need to be set to obtain comprehensiveness?

The goal of the study was to achieve comprehensiveness, which demanded a holistic

view of the product development process, with all its factors potentially influencing the

outcome. In addition, it was aimed for a high level of reliability in the results. From

these boundary conditions, a set of required characteristics was derived (summarized in

Table 2.3). A theoretical framework was developed in order to cast these characteristics

into the research process. This developed research process (Figure 3.5) is referred to

as the Comprehensive Approach. It was shown that it is compatible to overall design

research frameworks like Design Research Methodology.

A systematic elaboration and grouping of all Potential Success Factors showed to be

essential in order to reduce the amount of data to a manageable size, without sacrificing

comprehensiveness. The clear definition of domain and dimensions of Design Success

showed to be of equal importance. The need for quantification of mostly qualitative data

into quantitative data required a scheme of reasonable descriptions on rating scales to

reduce bias in the data. An additional measure to reduce bias was the strict separation of

data collection between cause and effect. While data on the cause side (Potential Success

Factors) was collected from documentation and interviews of engineering managers or

project participants, data on the cause side (Design Success) was gathered from the

company’s internal customers of the products, outside of the engineering department.

Applying advanced statistical techniques to analyze the collected data, lead to results

which are expected to be of high reliability and minimal bias. The concept of regarding the

108

product development process as a strict matter of cause and effect served the purpose of

identifying causal relationships. Adding a second analysis step for analyzing dependencies

made it possible to draw a conclusive map of relationships and dependencies. Only after

this step, was it possible to identify the Dominant Success Factors.

Research Question II: Which of all the influencing factors can be proven to

have a significant and dominant influence on Design Success?

The analysis and interpretation of results revealed that the originally identified 18 Po-

tential Success Factors ended up in two groups of Success Factors.

Supplementary Success Factors are the Potential Success Factors which either did not

show any significant relationship to the Design Success in the first analysis step, or

showed to be caused to change by other Success Factors in the second analysis step. It

cannot be concluded that these factors do not contribute to successful product develop-

ment, however, it can be concluded that these factors can only contribute if other more

dominant factors are existent and in the right balance.

Dominant Success Factors are the factors which directly influence Design Success and

cause other Success Factors to change, but are not caused by other Success Factors to

change. The following four factors were identified to be dominant:

•Priority of Project

•Experience and Skills of Project Lead

•Technical Complexity and Ratio of new Technology

•Awareness of Lessons Learned and State of the Art Technology Knowledge

These are the factors which should provide the best gauges for engineering manage-

ment to increase the chance for success in development projects. Only if these factors

are sufficiently established in the first place, can other factors, which did not show the

same level of significance, help to further increase the successful outcome of development

projects.

Question III: What can engineering management do to increase the chance of

Design Success?

The findings of this study suggest that engineering management can elevate success in

product development, by focusing on the four Dominant Success Factors first and most.

While the awareness of lessons learned and technology knowledge should always be a

focus, the other three dominant factors do not necessarily have to be maximized, but in

the right balance. Projects of high complexity and innovation demand for a high level of

109

management priority and project lead experience. At least one of the two factors needs

to be at a high level, the other at a medium level. As soon as none of these two factors

is high, or any of them is low, a significant decrease in success has to be expected. For

projects of lower complexity, these factors were found to be of less importance. The tri-

angle and its balance between the three Dominant Success Factors management priority,

project lead experience and complexity, is primarily of use in the pre-planning and early

stages of development projects. It can support management in the best utilization of

the engineering departments resources. It can also serve as an indicator for reaching the

maximum capacity of development projects, which can be executed in parallel, but still

completed successfully.

110

111

Bibliography

Ahmed, S. (2007), “Empirical research in engineering practice”, Journal of Design Re-

search , Vol. 6, pp. 359–380.

Ahmed, S. and Wallace, K. M. (2004), “Understanding the knowledge needs of novice

designers in the aerospace industry”, Design Studies , Vol. 25, pp. 155–173.

Ahmed, S., Wallace, K. M. and Blessing, L. T. M. (2003), “Understanding the differences

between how novice and experienced designers approach design tasks”, Research in

Engineering Design , Vol. 14, pp. 1–11.

Akao, Y. (1990), QFD: Quality Function Deployment - Intergrating Customer Require-

ments into Product Design, translated by Mazur, G., Productivity Press, New York,

NY.

Amabile, T. (1996), Creativity in Context, Westview Press, Boulder, CO.

Antonsson, E. K. (1987), “Development and Testing of Hypotheses in Engineering Design

Research”, Journal of Mechanisms, Transmissions and Automation in Design , Vol.

109, pp. 153–154.

Argotea, L. and Ingramb, P. (2000), “Knowledge Transfer: A Basis for Competitive

Advantage in Firms”, Organizational Behavior and Human Decision Processes , Vol. 82,

pp. 150–169.

Aurisicchio, M. and Wallace, K. M. (2007), “Competencies required to undertake empir-

ical engineering design research”, Journal of Design Research , Vol. 6, pp. 381–399.

Baumg¨artner, C. and Blessing, L. T. M. (1999), “A Comparison of Project Planning in

an Italian and a German Company”, Proceedings of 12th International Conference on

Engineering Design (ICED’99), 24-26 August 1999, Munich, Germany .

Blessing, L. T. M. (2002), “What is this thing called design research?”, Annals of the

2002 International CIRP Design Seminar. CIRP, Hong Kong , pp. 1–6.

Blessing, L. T. M. and Chakrabarti, A. (2009), DRM, a Design Research Methodology,

Springer, London.

112

Blessing, L. T. M., Chakrabarti, A. and Wallace, K. M. (1998), An Overview of Descrip-

tive Studies in Relation to a General Design Research Methodology, in E. Franken-

berger, P. Badke-Schaub and H. Birkhofer, eds, ‘Designers - the Key to Successful

Product Development’, Springer, London, pp. 42–56.

Bonner, J. M., Ruekert, R. W. and Walker, Jr., O. C. (2002), “Upper management

control of new product development projects and project performance”, Journal of

Product Innovation Management , Vol. 19, pp. 233–245.

Brue, G. and Launsby, R. G. (2003), Design for Six Sigma, McGraw-Hill, New York, NY.

Burke, S. (1998), “Missing Values, Outliers, Robust Statistics & Non-

parametric Methods”, Europe Online Supplement, , retrieved 29-August-2013,

https://www.webdepot.umontreal.ca/Usagers/sauves/MonDepotPublic/CHM%203103/

LCGC%20Eur%20Burke%202001%20-%204%20de%204.pdf .

Cantamessa, M. (2003), “An empirical perspective upon design research”, Journal of

Engineering Design , Vol. 14, pp. 1–15.

Chalupnik, M. J., Wynn, D. C. and Clarkson, P. J. (2009), “Approaches to mitigate the

impact of uncertainty in development processes”, International Conference on Engi-

neering Design, ICED’09, Stanford, CA .

Chan, Y. and Walmsley, R. P. (1997), “Learning and Understanding the Kruskal-Wallis

One-Way Analysis-of-Variance-by-Ranks Test for Differences Among Three or More

Independent Groups”, Journal of the American Physical Therapy Association , Vol. 77,

pp. 1755–1761.

Chollet, B., Brion, S., Chauvet, V., Mothe, C. and G´eraudel, M. (2012), “NPD Projects

in Search of Top Management Support: The Role of Team Leader Social Capital”,

M@n@gement , Vol. 15, pp. 43–75.

Cohen, L. (1995), Quality Function Deployment: How to Make QFD Work for You,

Addison Wesley, Reading, MA.

de Weck, O., Eckert, C. and Clarkson, J. (2007), “A classification of uncertainty for

early product and system design”, International Conference on Engineering Design,

ICED07, Paris, France .

de Wit, A. (1988), “Measurement of project success”, International Journal of Project

Management , Vol. 6, pp. 164–170.

Dixon, J. (1966), Design Engineering: Inventiveness, Analysis, and Decision Making,

McGraw-Hill, New York, NY.

113

Dixon, J. R. (1987), “On research methodology towards a scientific theory of engineering

design”, Artificial Intelligence for Engineering, Design, Analysis and Manufacturing ,

Vol. 1, pp. 145–157.

Donaldson, S. I. and Grant-Vallone, E. J. (2002), “Understanding Self-Report Bias in

Organizational Behavior Research”, Journal of Business and Psychology , Vol. 17,

pp. 245–260.

Dylla, N. (1991), Denk- und Handlungsabl¨aufe beim Konstruieren, Hanser, Munich.

Eckert, C. M., Stacey, M. K. and Clarkson, P. J. (2003), “The spiral of applied research:

A methodological view on integrated design research”, Proceedings of the 14th Inter-

national Conference on Engineering Design (ICED03), 19-21 August 2003, Stockholm,

Sweden .

Ehrlenspiel, K. (2009), Integrierte Produktentwicklung, 4rd edn, Hanser, Munich.

Eppinger, S. D., Whitney, D. E., Smith, R. P. and Gebala, D. A. (1994), “A model-based

method for organizing tasks in product development”, Research in Engineering Design

, Vol. 6, pp. 1–13.

Ernst, H. (2002), “Success factors of new product development: a review of the empirical

literature”, International Journal of Management Reviews , Vol. 4, pp. 1–40.

Frankenberger, E. and Auer, P. (1997), “Standardized observation of team-work in de-

sign”, Research in Engineering Design , Vol. 9, pp. 1–9.

Frankenberger, E., Badke-Schaub, P. and Birkhofer, H., eds (1998), Designers - The Key

to Successful Product Development, Springer, London.

French, M. (1999), Conceptual Design for Engineers, 3 edn, Springer, London.

Fricke, G. (1996), “Successful Individual Approaches in Engineering Design”, Research

in Engineering Design , Vol. 8, pp. 151–165.

Fricke, S. E. and Shenbar, A. J. (2000), “Managing multiple engineering projects in a

manufacturing support environment”, Engineering Management , Vol. 47, pp. 258–268.

Garson, G. D. (2012), Multiple Regression (Statistical Associates Blue Book Series), 1

edn, Statistical Associates Publishers.

Gausemeier, J., Lindemann, U., Reinhart, G. and Wiendahl, H.-P. (2000), “Kooperatives

Produktengineering: Ein neues Selbstverst¨andnis des ingenieurm¨aßen Wirkens”, HNI

Verlagsschriftenreihe, Paderborn, Germany , Vol. 79, pp. 50–51.

114

Gausemeier, J., Plass, C. and Wenzelmann, C. (2009), Zukunftsorientierte Un-

ternehmensgestaltung, Hanser, Munich.

Gericke, K. (2011), Enhancing Project Robustness: A Risk Management Perspective,

PhD thesis, Technische Universit¨at, Berlin, Germany.

Goh, S. C. (2002), “Managing effective knowledge transfer: an integrative framework and

some practice implications”, Journal of Knowledge Management , Vol. 6, pp. 23–30.

Griffin, A. and Page, A. L. (1993), “An Interim Report on Measuring Product Devel-

opment Success and Failure”, Journal of Product Innovation Management , Vol. 10,

pp. 291–308.

Hales, C. (1987), Analysis of the Engineering Design Process in an Industrial Context,

PhD thesis, Cambridge University, Cambridge.

Horv´ath, I. (2004), “A treatise on order in engineering design research”, Research in

Engineering Design , Vol. 15, pp. 155–181.

Jensen, T. E. and Andreasen, M. M. (2010), Design Methods in Practice - beyond the

’Systematic Approach’ of Pahl & Beitz, in ‘Proceedings of the 10th International Design

Conference DESIGN 2010’, pp. 21–28.

Jung, C. G. (1971), Psychological Types, Princenton University Press, Princenton, NJ.

Kahn, K. B. and McDonough, III, E. F. (1997), “An Empirical Study of the Relationships

among Co-location, Integration, Performance, and Satisfaction”, Journal of Product

Innovation Management , Vol. 14, pp. 161–178.

Kerzner, H. (2010), Project Management: Best Practice, Achieving Global Excellence, 2

edn, John Wiley & Sons, Inc., Hoboken, NJ.

Kitchenham, B. and Pfleeger, S. (2003), “Principles of Survey Research Part 6: Data

Analysis”, Software Engineering Notes , Vol. 28, pp. 24–27.

Knudsen, M. P. (2007), “The Relative Importance of Interfirm Relationships and Knowl-

edge Transfer for New Product Development Success”, Journal of Product Innovation

Management , Vol. 24, pp. 117–138.

Krause, F.-L., Franke, H.-J. and Gausemeier, J. (2006), Innovationspotentiale in der

Produktentwicklung, Hanser, Munich.

Kruskal, W. H. and Wallis, W. A. (1952), “Use of Ranks in One-Criterion Variance

Analysis”, Journal of the American Statistical Association , Vol. 47, pp. 583–621.

115

Landy, F. J. and Barnes, J. L. (1979), “Scaling Behavioral Anchors”, Applied Psycholog-

ical Measurement , Vol. 3, pp. 193–200.

Lindemann, U. (2009), Methodische Entwicklung technischer Produkte: Methoden flexibel

und situationsgerecht anwenden, 3 edn, Springer, Berlin.

Lindemann, U., ed. (2003), Human Behaviour in Design, Springer, Berlin.

McDonough, III, E. F., Kahn, K. B. and Barczaka, G. (2001), “An investigation of the use

of global, virtual, and colocated new product development teams”, Journal of Product

Innovation Management , Vol. 18, pp. 110–120.

Mehalik, M. and Schunn, C. (2006), “What Constitutes Good Design? A Review of Em-

pirical Studies of Design Processes”, International Journal of Engineering Education ,

Vol. 22, pp. 519–532.

Morgan, J. M. and Liker, J. K. (2006), The Toyota Product Development System: Inte-

grating People, Process and Technology, Productivity Press, New York, NY.

Neap, H. S. and Celik, T. (1999), “Value of a Product: A Definition”, International

Journal of Value-Based Management , Vol. 12, pp. 181–191.

Olechowski, A., Oehmen, J., Seering, W. and Ben-Daya, M. (2012), Characteristics of

successful risk management in product design, in ‘Proceedings of the 12th International

Design Conference DESIGN 2012’, pp. 269–278.

Pahl, G. (1992), Merkmale guter Probleml¨oser beim Konstruieren, in ‘VDI Berichte’,

number 953, pp. 187–201.

Pahl, G., Beitz, W., Schulz, H.-J. and Grote, K.-H. (2007), Engineering Design - A

Systematic Approach, 3rd edn, Springer, London.

Pillkahn, U. (2007), Trends und Szenarien als Werkzeuge zur Strategieentwicklung. Der

Weg in die unternehmerische Zukunft, Publicis Publishing, Erlangen, Germany.

Potter, J. (2004), Discourse analysis, in M. A. Hardy and A. Bryman, eds, ‘Handbook of

Data Analysis’, Sage, London, pp. 607–624.

Prabhakar, G. P. (2008), “What is Project Success: A Literature Review”, International

Journal of Business and Management , Vol. 3, pp. 3–10.

Pugh, S. (1991), Total Design: Integrated Methods for Successful Product Design, Addison

Wesley, Harlow, UK.

116

Pyzdek, T. and Keller, P. A. (2009), The Six Sigma Handbook, A Complete Guide for

Green Belts, Black Belts and Managers at All Levels, 3 edn, McGraw-Hill, New York,

NY.

Quenk, N. L. (2009), Essentials of Myers-Briggs Type Indicator Assessment, 2 edn, John

Wiley & Sons, Inc., Hoboken, NJ.

Schimmoeller, L. (2010), “Success Factors of new Product Development Processes”, Ad-

vances in Production Engineering & Management , Vol. 5, pp. 25–32.

Schregenberger, J. (1998), The Further Development of Design Methodologies, in

E. Frankenberger, P. Badke-Schaub and H. Birkhofer, eds, ‘Designers - the Key to

Successful Product Development’, Springer, London, pp. 57–67.

Siemens (2013), “Main components of a gas turbine”, retrieved 14-Nov-2013, from

http://www.energy.siemens.com/hq/en/fossil-power-generation/gas-turbines/sgt5-

4000f.htm .

Sirkin, R. M. (2006), Statistics for the Social Sciences, 3 edn, Sage Publications, Thou-

sand Oaks, CA.

Subrahmanian, E. (1992), Notes on Empirical Studies of Engineering Tasks and Environ-

ments, in ‘NSF Workshop on Information Capture and Access in Engineering Design

Environments, Ithaca, NY’, pp. 567–578.

Suomala, P. and Jokioinen, I. (2003), “The patterns of success in product development:

a case study”, European Journal of Innovation Management , Vol. 6, pp. 213–227.

Thamhain, H. J. (2003), “Managing innovative R&D teams”, R&D Management , Vol. 33,

pp. 297–311.

Thomas, D. C. (1999), “Cultural Diversity and Work Group Effectiveness: An Experi-

mental Study”, Journal of Cross-Cultural Psychology , Vol. 30, pp. 242–263.

Ullman, D. (2003), The Mechanical Design Process, McGraw-Hill, New York, NY.

Ullman, D., Dietterich, T. and Stauffer, L. (1988), “A Model of the Mechanical Design

Process. Based on Empirical Data”, Academic Press Limited , Vol. 2, pp. 33–52.

Van den Bulte, C. and Moenaert, R. K. (1998), “The Effects of R&D Team Co-location on

Communication Patterns among R&D, Marketing, and Manufacturing”, Management

Science , Vol. 44, pp. 1–18.

Wallace, K. and Blessing, L. (2000), “Observations on Some German Contributions to

Engineering Design. In Memory of Professor Wolfgang Beitz”, Research in Engineering

Design , Vol. 12, pp. 2–7.

117

Welo, T., Aschehoug, S. H. and Ringen, G. (2013), “Assessing the Relationship between

New Product Development Practices and Performances in the Norwegian Manufac-

turing Industry”, Smart Product Engineering, Lecture Notes in Product Engineering ,

pp. 895–904.

White, D. and Fortune, J. (2002), “Current practice in project management - an empirical

study”, International Journal of Project Management , Vol. 20, pp. 1–11.

W¨orz, U. and G¨ohlich, D. (2012), A Comprehensive Empirical Approach for Determina-

tion of Success Factors of Product Development Projects, in ‘Proceedings of the 12th

International Design Conference DESIGN 2012’, pp. 133–140.

Yang, Y. Q., Wang, S. Q., Dulaimi, M. and Low, S. P. (2003), “A fuzzy quality function

deployment system for buildable design decision-makings”, Automation in Construc-

tion , Vol. 12, pp. 381–393.

Yates, D. S., Moore, D. S. and McCabe, G. P. (1999), The Practice of Statistics, W.H.

Freeman, New York, NY.

Zirger, B. J. and Maidique, M. A. (1990), “A Model of New Product Development: An

Empirical Test”, Management Science , Vol. 36, pp. 867–883.

118

Appendix

119

Appendix A: Data matrix

120

Appendix B: Tables of observed counts for Chi-Square

Test

121

122

123

124