sustainability
Article
Objectives for Stakeholder Engagement in Global
Environmental Assessments
Jennifer Garard 1,2,* and Martin Kowarsch 1
1Mercator Research Institute on Global Commons and Climate Change, EUREF Campus 19,
2Faculty VI, Technical University of Berlin, Straße des 17 Juni 135, 10623 Berlin, Germany
*Correspondence: [email protected]; Tel.: +49-(0)-30-338-5537-245
Received: 8 August 2017; Accepted: 31 August 2017; Published: 4 September 2017
Abstract:
Global environmental assessments (GEAs) are among the most large-scale, formalized
processes for synthesizing knowledge at the science–policy–society interface. The successful
engagement of diverse stakeholders in GEAs is often described as a crucial mechanism for increasing
their legitimacy, salience and credibility. However, the diversity of perspectives on the more precise
objectives for stakeholder engagement remains largely unclear. The aims of this study are to categorize
and characterize the diversity of perspectives on objectives for stakeholder engagement in GEAs;
to explore differences in perspectives within and between different stakeholder groups and categories;
and to test whether the more practical prioritization and selection of objectives in GEAs can be linked
to deliberative policy learning as a higher-level rationale for stakeholder engagement. For these
purposes, we conduct a grounded theory analysis and a keyword analysis of interview material and
official GEA documents relating to two GEAs: UN Environment’s Fifth Global Environment Outlook
and the Working Group III contribution to the Intergovernmental Panel on Climate Change Fifth
Assessment Report. Based on the analysis, we identify six categories of objectives and present as
hypotheses promising ways forward for prioritizing and characterizing objectives for stakeholder
engagement in GEAs, as well as potential reasons for the differences between perspectives on
objectives. This study draws attention to the need for future GEA processes to have more explicit
discussions on the objectives for stakeholder engagement, as well as the importance of moving
towards increasingly deliberative and inclusive assessment processes more broadly.
Keywords:
stakeholder engagement objectives; participation; global environmental assessment;
environmental governance; deliberative policy learning
1. Introduction
Navigating the interface between scientific expertise, policy-making and society more broadly is
a crucial concern for environmental governance. Doing so can present significant challenges in terms
of legitimacy, credibility and policy-relevance, in particular when addressing wicked sustainability
problems, which are transboundary, long-term and uncertain in nature, and where facts and values are
inherently intermingled [
1
–
4
]. Perhaps the most prominent, elaborate and legitimate mechanism
currently in place for navigating this interface is the collaborative synthesis of policy-relevant
knowledge in global environmental assessments (GEAs).
GEAs are large-scale, formalized processes through which scientific and other types of knowledge
is assessed and synthesized in order to inform policy-making processes at multiple scales. The major
goals of GEA processes are to frame and analyze the drivers, impacts and potential solution pathways
associated with global environmental problems in a legitimate, credible and salient manner, and
to communicate these findings to their target audiences [
1
,
2
,
5
]. More than 140 GEAs have been
initiated to date, covering topics such as climate change, agricultural production, ozone depletion and
Sustainability 2017,9, 1571; doi:10.3390/su9091571 www.mdpi.com/journal/sustainability
Sustainability 2017,9, 1571 2 of 21
biodiversity loss [
6
]. GEAs are often mandated by intergovernmental or international bodies [
1
,
3
,
7
].
When well-designed, these highly interdisciplinary and collaborative processes have been shown to
exert an influence on policy discourses, lead to shifts in thinking in science, and have contributed
to developments in international environmental governance [
1
,
7
–
9
]. Among many other elements,
GEA processes require the involvement of a large number of stakeholders who participate in a many
different roles, and who often bring into the process a variety of different perspectives on the issues
being assessed [10].
While there is no unified definition of the term stakeholder, this study employs the term very
broadly, considering any individual involved in a GEA process in some capacity as a stakeholder, as
well as non-involved target audience members (similar to other studies, for example [
10
–
12
]). In this
sense, the term stakeholder can refer to government representatives, expert scientists from diverse
disciplinary and geographic backgrounds, as well as representatives of international organizations,
non-governmental organizations, business and industry, among others. Stakeholders have been
engaged in GEAs dating back to their origins in the 1970s [
6
]. Since then, stakeholder engagement
has been increasingly recognized as an integral component to the success of GEAs, as evidenced for
example in the case of the Millennium Ecosystem Assessment in the early 2000s [
5
,
9
] and more recently
in the Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES) [13–15].
Stakeholder engagement activities in GEAs have the potential to significantly strengthen
environmental governance, for example by building bridges between different scientific disciplines and
approaches, between political and administrative barriers, scales and boundaries, as well as between
conflicting interests and value beliefs. However, recent research on GEAs has found that, in order to
continue playing a valuable role in informing decision-making at multiple scales, their approaches to
stakeholder engagement must be systematically adapted to the evolving environmental governance
landscape [
3
,
6
,
10
,
16
]. This evolving landscape has been characterized by: (1) the proliferation of
agreed environmental goals, such as the Paris Agreement and the Sustainable Development Goals,
without clearly suitable policies for their achievement; (2) a related shift towards analyzing solutions,
and in particular policies, in environmental assessments; and, in turn, (3) the emergence of newly
active groups of stakeholders (including, for example, the direct engagement of indigenous and
local knowledge holders through multiscale processes or the increasing prominence of sub-national
scale environmental advocacy groups). This emergence of new actor groups has occurred in part
because focusing on solutions in a legitimate manner forces assessments to explicitly address multiple
conflicting interests, knowledge and value systems, geographic contexts, socioeconomic contexts and
policy interdependencies, as well as issues related to uneven institutional capacity [1,6,16,17].
As the stakes and stakeholder groups relevant to solution-oriented GEA processes have
diversified, so too have the perspectives on the objectives for stakeholder engagement. A first step
towards adapting the approaches to stakeholder engagement in GEAs to the evolving environmental
governance landscape is to establish a better understanding of the current, often underestimated
diversity of objectives. Objectives can include broader or more specific rationales for engagement,
are sometimes only implicit, and can be understood from different perspectives and suit different
purposes and motivations (societal, group or individual). Those who mandate and produce GEA
processes often represent the official, sanctioned objectives for stakeholder engagement, though they
may have their own personal objectives as well. Authors and other actors who take part in GEA
processes also have in mind certain objectives for their own engagement, which may or may not align
with the official objectives. Less often addressed are the objectives for stakeholder engagement held
by target audiences, who have comparatively little influence in a GEA process but nonetheless have
expectations for engagement which can influence their use of GEA outputs and outcomes. Improving
our understanding of the objectives for stakeholder engagement in GEAs is fundamentally important
to facilitate systematic evaluation and improvement of engagement activities along the lines proposed
by Garard and Kowarsch [
10
]. Characterizing the diversity can serve to challenge the predominant
tendency to focus only on very few objectives, encouraging more nuanced deliberations over the
Sustainability 2017,9, 1571 3 of 21
relative merits of different options. Understanding the diversity of objectives can also help provide
better incentives for effective engagement by more clearly and explicitly framing and communicating
engagement objectives [
10
], as well as helping to identify divergent viewpoints (and potential conflicts)
early on regarding these objectives.
The importance of understanding the diversity of objectives for engagement has been reflected
for example in the literature on co-production [
18
] and in theories of co-creation or co-design [
19
],
and the topic of stakeholder engagement at the global scale has been addressed in the literature as
well [
2
,
5
,
10
,
17
,
20
–
22
]. However, the peer-reviewed literature and official documents guiding GEA
practice usually analyze or propose only a limited number of objectives (if at all) in isolation, failing
to do justice to the increasing diversity of perspectives on objectives and how these relate to one
another [
10
,
23
]. Thus, there is a gap in the literature and GEA documents regarding an explicit and
comprehensive analysis of the diversity of more precise objectives for stakeholder engagement in
GEAs and the relationships between different perspectives and objectives.
In light of this, the more specific aims of this particular study are to characterize and categorize
the diversity of perspectives on objectives for stakeholder engagement in GEAs, to explore differences
in perspectives within and between different stakeholder groups and categories, and finally to test
whether different objectives in GEAs can contribute to deliberative policy learning as a normatively
significant, higher-level rationale for stakeholder engagement. Based on these analyses, we aim to
suggest as hypotheses practical ways in which stakeholder engagement in GEAs can be strengthened.
The GEAs selected and the four stakeholder engagement activities analyzed are described in
Section 2.1. The research methodology is presented in Section 2.2. The results are presented in Section 3,
including the categories and characteristics of objectives from multiple perspectives (Section 3.1) and
the results of a keyword analysis linking categories of objectives to the higher-level goal of fostering
deliberative policy learning (Section 3.2). We then discuss the practical implications of the analysis and
present as hypotheses promising ways forward for stakeholder engagement in GEAs, focusing on the
selection of objectives (Section 4.1), the use of indicators (Section 4.2) and addressing the trade-offs
between objectives (Section 4.3).
2. Materials and Methods
2.1. Cases: GEAs and Methods of Stakeholder Engagement
The first step in the study was selecting cases in which to analyze the objectives for stakeholder
engagement in GEAs, which was done based on four criteria. The first criterion was that the GEA
process be recurring to maximize the relevance of our findings for future iterations of the assessments.
The second criterion was that the GEA strongly emphasize the exploration of solutions, important
in light of the evolving governance landscape described in the Introduction. The third criterion
was that the GEA process aims to address multiple aspects of environmental problems and solution
options, requiring the engagement of a broader diversity of stakeholders as opposed to processes
examining a more limited number of perspectives. Finally, the fourth criterion was that information on
stakeholder engagement in the selected GEAs be relatively accessible to the researchers.
These criteria led to the selection of two GEAs. The first GEA is UN Environment’s Fifth Global
Environment Outlook (GEO-5), published in 2012. GEO-5 placed considerable focus on exploring
concrete solutions, for example conducting an appraisal of specific policies and policy clusters at
the regional scale in Chapters 9–14 and analyzing tools and strategies relevant at the global scale
in Chapter 17. GEO-5 also covered a very broad diversity of topics from multiple perspectives.
The second GEA selected is the Working Group III contribution to the Fifth Assessment Report of
the Intergovernmental Panel on Climate Change (IPCC WGIII AR5). The IPCC in general is very
comprehensive in terms of covering issues related to climate change from as many perspectives as
possible, and WGIII focuses on potential solutions for mitigating climate change. The collaboration
Sustainability 2017,9, 1571 4 of 21
with UN Environment staff and IPCC WG III practitioners (see Acknowledgements) provided us with
excellent access to both cases.
Different methods for engaging with stakeholders take place at different points during a GEA
process and target different objectives. This study focuses on four prominent methods employed in
GEAs to engage with stakeholders which span the entire GEA process and cover all major stages at
which interactions occur: (1) the scoping meeting at the outset of the GEA to determine its outline
and guiding questions; (2) the regional consultations which take place partway through a GEA and
contribute to content development; (3) the review process which takes place towards the end of the
process and opens up draft text to comments from a very wide diversity of actors; and (4) the Summary
for Policy Makers (SPM) negotiation meeting, where the contents of the summary document are
discussed and negotiated line-by-line by government representatives from around the world (Table 1).
Table 1. Descriptions of four methods for stakeholder engagement in GEAs [24,25].
Method of
Engagement GEAs Goals Participating Groups Additional Information
Scoping
meeting
GEO-5, IPCC
WGIII AR5
Determine general scope
and guiding questions,
as well as precise chapter
outlines for the assessment.
Government representatives,
assessment producers, research or
academic institutions, civil society
organizations, business
organizations, intergovernmental
organizations, and others.
3-day (GEO-5) or 5-day
(IPCC WGIII AR5)
meeting before
assessment begins.
Regional
consultations GEO-5
Determine 3–5 priority
issues by region, associated
multilateral environmental
agreements, and discuss
promising solution options.
Government representatives,
assessment producers, research or
academic institutions, civil society
organizations, business
organizations, intergovernmental
organizations, and others.
2 days of meetings
in each of 6 UN
world regions.
Review
process
GEO-5, IPCC
WGIII AR5
Rigorously review contents
of assessment report prior
to publication (in a process
similar to peer-review).
Government representatives,
assessment producers, research or
academic institutions, civil society
organizations, business
organizations, intergovernmental
organizations, and others.
Two rounds of expert
and governmental
review, requirement to
respond to all comments.
SPM
negotiation
GEO-5, IPCC
WGIII AR5
Governments negotiate
line-by-line the content of
SPM draft (requiring
consensus on final SPM)
with input from authors.
Government representatives,
assessment producers, selected
authors, and others
(as observers only).
3-day (GEO-5) or 5-day
(IPCC WGIII AR5)
meeting at the end of the
assessment process.
Source: Own elaboration.
2.2. Research Approach and Methods
2.2.1. Data Collection and Grounded Theory Analysis
Grounded theory was chosen to structure the study for two main reasons. Firstly, this
methodology provided a robust structure for the research design, where concurrent data collection
and analysis allowed for improvement of the interview protocol over time. Secondly, grounded
theory is an appropriate method for analyzing material where the goal of the exercise is not only
to summarize the findings and develop categories, but also to understand the relationship between
different perspectives within and between categories [26].
Collecting data on the different perspectives on objectives for stakeholder engagement in GEAs
is difficult, since it is dispersed across many different sources which are not always easily accessible.
First, to capture the official objectives from the perspective of the organizations producing the two
selected GEAs, GEA documents were analyzed. After reviewing a number of official documents,
including official mandates, meeting reports, fact sheets, and procedural guidelines, we determined
that two documents in particular contained all of the objectives for stakeholder engagement in these
two GEAs which could be found elsewhere. The first document is the Statement on the Objectives,
Scope and Process of the fifth Global Environmental Outlook by the Global Intergovernmental and
Sustainability 2017,9, 1571 5 of 21
Multi-stakeholder Consultation for GEO-5 [
24
], which was proposed by UN Environment actors and
agreed on by a wide diversity of stakeholders at the first GEO-5 consultation in March 2010. The second
document is the Appendix A to the Principles Governing IPCC Work: Procedures for the Preparation,
Review, Acceptance, Adoption, Approval and Publication of IPCC Reports for IPCC WGIII AR5 [25],
which was agreed on by the Intergovernmental Panel members (government representatives) and
most recently amended in October 2013.
To capture the perspectives of as many stakeholders as possible and go beyond the official
objectives, 99 semi-structured interviews were conducted with authors and other scientific experts,
with government representatives involved in one of the two GEAs, as well as with non-involved
target audience members (Table 2). Priority was given to individuals who were either involved with
GEAs or whose work aligned closely with them, because relatively little is commonly known about
stakeholder engagement in GEAs outside of these circles, let alone on the topic of objectives. Speaking
to individuals with some experience is thus a reasonable starting point for the given research aims.
Within the three broad groups of interviewees, concerted efforts were made to achieve a balance
amongst interviewees along multiple criteria; however, this did not end up being the case due to
a lower response rate amongst some groups. The reasons for this lower response rate are now known.
It is interesting to note, though, that the lack of perfect balance amongst interviewees mirrors, to some
extent, the imbalance which has been observed in practice and written about elsewhere, for example
regarding diversity in geographic origins [
27
,
28
] and disciplinary background [
29
,
30
]. The relatively
higher number of authors and other scientific experts involved in GEO-5 who were interviewed comes
as a result of the theoretical sampling employed as part of the grounded theory approach. Through
these interviews, the first conducted, it became more apparent the types of additional perspectives
missing from the analysis. Thus, subsequent interview invitations were more targeted, and the
additional perspectives were collected from a relatively smaller number of interviews.
Table 2.
Information on interviews and interviewees, organized into three different stakeholder groups.
Group of Interviewees Description (and Examples) Number of
Interviews Rate of Response
Authors and other involved
scientific experts
Authors, producers, expert members of
oversight bodies and panels 76 31.5%
GEO-5 GEO-5 High Level Panel 68 29.9%
IPCC WGIII AR5 Technical Support Unit 8 66.6%
Involved government
representatives
Members of national delegations
officially representing their country 13 19.4%
GEO-5 Member of Council of Permanent
Representatives 5 15.6%
IPCC WGIII AR5 IPCC national focal point 8 22.9%
Target audience (not involved) Civil society organizations,
intergovernmental organizations, etc. 10 12.7%
Total 99 25.6%
Source: Own elaboration.
All interviews were conducted in English between July 2013 and April 2015 via Skype or telephone,
lasting on average 55 min. Prior to each interview, written informed consent was obtained. eighty-eight
interviewees gave explicit permission for the interviews to be recorded and transcribed, and for quotes
to be shared as long as no identifying information was conveyed. Interviewees who did not want to
be recorded gave permission for extensive interview notes to be prepared, which were subsequently
analyzed in lieu of a transcript. Due to the sensitive nature of the issues discussed, the full transcripts
of interviews are not available to the public, but a list of all interviewees who consented to having their
names released as well as a longer portion of transcript for all quotes employed will be made available
upon reasonable request. Each interviewee was asked questions along five themes: stakeholder
engagement objectives and methods; GEA impact and influence; treatment of divergent viewpoints
Sustainability 2017,9, 1571 6 of 21
and conflict; policy analysis methodology; and a historical overview of GEAs. More precise follow-up
questions were asked in cases where the interviewee showed interest or had direct experience. On the
theme of stakeholder engagement, the theme most relevant to this study, these follow-up questions
included queries about personal objectives, institutional objectives, and perceptions of other actors’
objectives, as well as what aspects of different methods of stakeholder engagement were deemed to be
most successful and why. The specific wording of the questions was tailored to the different groups of
interviewees in order to promote more open discussion.
Concurrently with data collection, the interview material along with official GEA documents
containing the official objectives for the engagement methods were iteratively coded using grounded
theory analysis, following the guidance of Anselm Strauss and Juliet Corbin [
26
], in Max QDA.
The material was coded in four iterations, first identifying segments of text related to stakeholder
engagement activities at all, then identifying any mention of objectives, including personal, institutional
and official objectives for engagement (either held by those involved, or claimed by observers), and
different interpretations of official objectives or the objectives of other stakeholders. In the third stage
of coding, the objectives were grouped into broad categories. In the fourth stage, the categories were
refined through constant comparison of different perspectives on objectives, both within and between
interviewee groups and categories. All categories and category names were derived exclusively from
the interview material and official GEA documents analyzed, and remain grounded in the data. The
results of the iterative coding analysis were visualized using the Code Relations Browser in Max QDA,
a tool which displays the intersections between selected coded segments (in this case, depicting when
descriptions of the different categories of objectives overlapped with descriptions of the methods
of engagement).
For each of the six categories of objectives distilled from this analysis, a brief review of selected
peer-reviewed literature was conducted in order to highlight additional relevant insights, drawing on
the limited knowledge specifically on stakeholder engagement in GEAs, as well as on environmental
management literature more broadly. This served to complement and enrich the empirical material on
objectives, to contextualize the findings within a broader environmental governance framework and to
strengthen the understanding of different perspectives.
2.2.2. Keyword Analysis
Next, to explore the potential contribution of different categories of objectives to the realization
of deliberative policy learning, which has been identified in the literature as plausible normative
rationale for engagement in GEAs [
1
,
31
], an exploratory keyword analysis was conducted. A list of
keywords was developed associated with four major building blocks for deliberative policy learning
based on the literature (see Table 3). In Max QDA, the coded material from interviews and GEA
documents associated with a category of objectives was selected one category at a time. Within the text
selected for each category, a keyword search was performed using the root words in Table 3. The text
surrounding each statement containing both a reference to a category of objectives and a keyword was
then carefully studied. This was done in order to confirm that the keyword was indeed being used
in a manner associated with deliberative policy learning, and also to ensure that the statement could
be interpreted to imply that the respective objective could contribute positively towards realizing
the building block. For each category of objectives, the number of times a category of objectives was
mentioned in conjunction with a keyword associated with deliberative policy learning was averaged
in order to take into account the number of times objectives from this category were mentioned
overall. This step avoids frequency bias, where the most often-mentioned category of objectives
would automatically be the one that seems to contribute most strongly towards the building blocks of
deliberative policy learning. Interviewees were not explicitly asked about linkages between objectives
and deliberative policy learning to avoid distorting their responses and to get more spontaneous,
unbiased viewpoints.
Sustainability 2017,9, 1571 7 of 21
Table 3. Description of four major building blocks of deliberative policy learning and root keywords.
Building Block Description Normative Rationale (e.g.) Root Keywords References
Representation
To the extent possible,
represent the perspectives of
diverse individuals who might
be affected by process or might
affect its outcomes. Ensure
that representation is fair
and balanced.
It is normatively right and
procedurally fair to include
the perspectives of those
affected. It can improve the
quality of decisions to
involve perspectives of those
who can affect the process or
its outcomes.
represent*, divers*,
inclu*, behalf,
perspective, view*,
belief, vision,
account*
[1,21,32–34]
Empowerment
Ensure all actors are able to
engage on even footing,
and different perspectives
are given equal weight.
Break down participation
barriers in particular for
marginalized groups.
It is normatively right and
procedurally fair to strive to
overcome power imbalances,
and to broaden the
knowledge base contributing
to the GEA. Empowerment
can support trust and mutual
respect between actor groups.
power, equal*,
marginaliz*, strong,
weak, peer,
even, level
[1,21,31,35]
Building capacity
Enable effective and
meaningful participation and
inter-disciplinary collaboration
(between different experts and
non-experts alike). Consider
differential capacity needs (and
capacity-building mechanisms)
for different actors.
It is important to strengthen
the quality of debates and the
ability of different actors to
contribute together in the
context of an inter-
disciplinary and multi-scale
process. This can foster
mutual understanding
between groups.
capacity, learn*,
understand,
grow*, know*,
comprehen*, ability,
competenc*
[1,21,36,37]
Spaces for
deliberation
Provide physical space for true
face-to-face (or, online)
deliberation and trust-building
(or, online space). True
deliberation can include, e.g.,
backing up opinions and
perspectives with rational
justifications based on
underlying worldviews.
It is important to build trust,
increase willingness to work
collaboratively, and foster
learning, which can be to
which deliberation can
contribute. This in turn can
improve understanding of
different perspectives on
potential solution options.
space, deliberat*,
convene, bring,
dialogue, place,
discuss*, face*,
person, location
[1,31,38–40]
Source: Own elaboration.
3. Results: Existing Objectives for Stakeholder Engagement
3.1. Categories of Objectives
Six categories of objectives were distilled from the Grounded Theory Analysis. These categories
comprise all objectives mentioned in the interviews and in the selected GEA documents. The objectives
mentioned in interviews represent the perspectives of different stakeholders, while the GEA documents
represent the official objectives. Given their diverse backgrounds, the interviewees described particular
objectives in very different ways. For example, different interviewees discussed their own personal
objectives or motivations for engaging with a GEA process, their personal interpretations of the official
objectives or the objectives of others, or the perspective of the institution, organization or government
they were representing. Moreover, in both interviews and GEA documents, objectives were discussed
somewhere along a spectrum between being targeted to a specific meeting and being overarching
objectives transcending the stakeholder engagement activities to confer benefits on the GEA process as
a whole or even on society more broadly. This often coincided with the extent to which an objective
was described in a pragmatic sense, using practical and precise wording and reflecting directly on
outcomes, as opposed to a more conceptual or theoretical description. These characteristics are used to
describe the Results below and to highlight the different viewpoints within categories of objectives.
The category of objectives mentioned most often was the importance of providing a source
of information (discussed by 86 interviewees and found in both GEA documents, for a total of
88 mentions). The second most often mentioned category was fostering a dialogue (76). These are
followed in frequency by the category of objectives to improve communication and understanding (71),
and the category to create a sense of ownership (67) respectively. Lastly, the categories of objectives
Sustainability 2017,9, 1571 8 of 21
to exert control over the process (52) and to facilitate learning (47) were mentioned the least often by
interviewees and in GEA documents. These six categories are presented in the following subsections
(Sections 3.1.1–3.1.6, respectively), each of which includes specific examples to highlight the diversity
of perspectives on objectives which underlie the categories. For each subsection, a brief overview of
selected literature is included in order to bring in pertinent ideas which complement or go beyond the
empirical material.
The extent to which each category of objectives was mentioned in interviews or in GEA documents
in conjunction with a specific method of stakeholder engagement is shown in Figure 1, where the four
lines represent the four methods detailed in Table 1. Figure 1shows the absolute number of sources
(interviews or GEA documents) which mentioned a method and category of objectives together at
least once.
Sustainability 2017, 9, 1571 8 of 22
organization or government they were representing. Moreover, in both interviews and GEA
documents, objectives were discussed somewhere along a spectrum between being targeted to a
specific meeting and being overarching objectives transcending the stakeholder engagement
activities to confer benefits on the GEA process as a whole or even on society more broadly. This
often coincided with the extent to which an objective was described in a pragmatic sense, using
practical and precise wording and reflecting directly on outcomes, as opposed to a more conceptual
or theoretical description. These characteristics are used to describe the Results below and to
highlight the different viewpoints within categories of objectives.
The category of objectives mentioned most often was the importance of providing a source of
information (discussed by 86 interviewees and found in both GEA documents, for a total of 88
mentions). The second most often mentioned category was fostering a dialogue (76). These are
followed in frequency by the category of objectives to improve communication and understanding
(71), and the category to create a sense of ownership (67) respectively. Lastly, the categories of
objectives to exert control over the process (52) and to facilitate learning (47) were mentioned the least
often by interviewees and in GEA documents. These six categories are presented in the following
subsections (Sections 3.1.1–3.1.6, respectively), each of which includes specific examples to highlight
the diversity of perspectives on objectives which underlie the categories. For each subsection, a brief
overview of selected literature is included in order to bring in pertinent ideas which complement or
go beyond the empirical material.
The extent to which each category of objectives was mentioned in interviews or in GEA
documents in conjunction with a specific method of stakeholder engagement is shown in Figure 1,
where the four lines represent the four methods detailed in Table 1. Figure 1 shows the absolute
number of sources (interviews or GEA documents) which mentioned a method and category of
objectives together at least once.
Figure 1. Number of interviewees and global environmental assessment (GEA) documents who
mentioned a category of objectives in conjunction with a method of stakeholder engagement at least
once (own elaboration).
3.1.1. Source of Information
The most prominently mentioned category of objectives in interviews and in GEA documents
alike was that stakeholders should be engaged with in order to provide a source of information to
the assessment. There were two overarching ways of describing this category of objectives in
interviews, which were mentioned evenly across all groups and which were sometimes both
0
10
20
30
40
Source of
information
Dialogue
Communication &
understanding
Ownership
Exerting control
Learning
Scoping Meeting
Regional-scale Consultations
Review Process
SPM Negotiations
Figure 1.
Number of interviewees and global environmental assessment (GEA) documents who
mentioned a category of objectives in conjunction with a method of stakeholder engagement at least
once (own elaboration).
3.1.1. Source of Information
The most prominently mentioned category of objectives in interviews and in GEA documents
alike was that stakeholders should be engaged with in order to provide a source of information to the
assessment. There were two overarching ways of describing this category of objectives in interviews,
which were mentioned evenly across all groups and which were sometimes both mentioned by one
interviewee. The first and most commonly mentioned standpoint is pragmatic in the sense that it
refers to actually writing the report, emphasizing primarily the role of stakeholders as authors and
other scientific experts. One government representative involved in the IPCC stressed the need to
include information from “a mix of disciplines involving social science, natural science, economics,
technology, and others” to make the findings more policy-relevant. A scientific expert who participated
in GEO-5 stated that “you just can’t do the assessments without access to stakeholders’ information”.
One target audience member not involved in either assessment also took this perspective, explaining
that you need authors with “different science and research background to present for understanding,
for discussion, different interpretations of science findings”.
The other standpoint mentioned prominently within this category of objectives is that GEAs
should aim to diversify the sources of information to the process beyond scientific information alone.
Two interviewees involved in GEO-5 scientific expert panels described the need to engage stakeholders
from different “cultural backgrounds”, and representatives of “private sector, civil society, [and]
holders of traditional knowledge” for example. A government representative involved in IPCC WGIII
Sustainability 2017,9, 1571 9 of 21
AR5 stressed that assessments should explore a “plurality of perspectives of the problems”. A target
audience member eloquently described the need to move “beyond a view that the only kind of
expertise which is valid to bring in to assessments is from formal science and accredited peer-review
publication towards opening up to a wider range of stakeholders and a wider range of forms of
knowledge [
. . .
] to contribute to the kinds of solutions that might emerge”. One interviewee with
experience as an author in both GEO and IPCC assessments explained the need for diversifying sources
of information “as a validation of whether solutions are actually feasible”, important in light of the
shift towards solution-orientation. However, one interviewee involved in IPCC WGIII AR5 warned
that “for any assessment [it would be] very, very risky to include people when in the end you rely on
information that are not peer reviewed”, suggesting that “if you want to have broader stakeholder
involvement then you have to set up a research project”.
In GEA documents, this category of objectives was linked directly to all four of the engagement
activities described in Table 1, and in particular focused on diversifying the sources of information.
For example, the IPCC WGIII AR5 selected authors and participants for the scoping meeting based
on “the following criteria: Scientific, technical and socio-economic expertise, including the range
of views; geographical representation; a mixture of experts with and without previous experience
in IPCC; gender balance; experts with a background from relevant stakeholder and user groups,
including governments” [
25
]. In GEO-5 regional consultations, an official objective was to engage
“geographically representative and gender balanced regional groups” [24].
In the literature, this category of objectives has also often been discussed with regards to
diversifying the sources of information to an assessment. Similarly to what was found in GEA
documents, this includes the engagement of authors from diverse disciplines, institutional affiliations,
geographic locations, and with a variety of areas of topical expertise [
9
,
14
,
41
]. However, stakeholders
can also provide information in other roles stemming from a broader diversity of worldviews,
beliefs and knowledge systems [
7
,
42
–
46
]. For example, non-scientific stakeholders could contribute
information based on cultural beliefs or experiences to the problem framing and to analyzing the
potential success or failure of solution options, including co-benefits and side effects [47,48].
3.1.2. Dialogue
The category of objectives to foster a dialogue was common across interviews and was the
second most commonly mentioned category in the material analyzed overall, but was actually not
mentioned explicitly in GEA documents. Within this category, interviewees focused on two points in
particular. Firstly, some interviewees characterized this category of objectives in an overarching sense,
as an objective which applies to stakeholder engagement over the whole course of the GEA process
and has benefits transcending the process itself. This standpoint was mainly described by government
representatives and target audience members. For example, one target audience member stated bluntly
that “we lose if we do not discuss, we win if we discuss”, while another advocated taking a “more
deliberative approach which explicitly acknowledges that there are alternative views on what the
problem is and why it matters”. Some interviewees focusing on the overarching nature of this category
of objectives described dialogue as leading to a broader, desirable outcome. For example, a government
representative involved in IPCC WGIII AR5 described how dialogue processes within a GEA could
help “bring new actors into the broader discussion, reset priorities, re-frame the agenda”, while
a government representative from GEO-5 argued this could help “deal with lack of trust from some
countries”. Another government representative for IPCC WGIII AR5 described how regional-scale
dialogues in particular could “build up a regional common understanding on issues, and then move
forward to a global process” while a target audience member explained that “when you have the
chance to discuss [
. . .
] this interaction is a positive thing towards collaboration at the regional level”.
A second standpoint within this category of objectives had to do with describing dialogue
as an objective with more practical implications for reaching the targeted goals of a specific
meeting or engagement activity. This perspective came up primarily during interviews with
Sustainability 2017,9, 1571 10 of 21
authors and other involved scientific experts. For example, a GEO-5 author saw a high value in
promoting “interdisciplinary discussions to clarify the scientific findings and methodology” in regional
consultations. Regarding the SPM negotiations, an expert involved in the publication of GEO-5 felt
that striving to foster dialogue would help to overcome “misunderstandings between policy makers
and scientists”. An author involved in the IPCC WGIII AR5 explained that in some cases which come
up during the SPM negotiations “there is no scientific way of saying that this way of representing
data to carry this political message is better than that way—you can say it’s marginally better but you
can’t say it’s right or wrong”. In these situations, this interviewee argued, a broader dialogue must be
fostered since science alone cannot justify the choice of presentation.
The main emphasis in the literature when addressing dialogues at the science–policy interface
focuses on an open, back-and-forth flow of information between different actors [
49
]. This can often be
observed in practice in GEAs [
10
,
17
,
20
,
21
], even though the official GEA documents do not explicitly
state that fostering a dialogue is an objective. Similar to interviewees, some researchers have argued
that dialogue can build trust and mutual understanding, and clarify representations of problems,
impacts and potential solutions [
9
,
50
]. Others have emphasized in particular the importance of
fostering dialogue in situations not necessarily requiring consensus as an end point [
31
,
51
]. Fostering
dialogue, and ensuring that stakeholders feel they are not only listened to but have a more meaningful
role in an ongoing discussion, can contribute to strengthening the legitimacy and salience of the
assessment report, important criteria for success in GEAs [2,46].
3.1.3. Communication and Understanding
The third category of objectives for engaging with stakeholders in GEAs, mentioned most often
by government representatives, is to improve the communication and understanding of the main
messages and findings of the assessment. There were three general ways of describing this objective
in the interview material, all of which have a relatively pragmatic basis. Firstly, and most commonly,
this category of objectives was described with regards to how different findings were worded and
presented. From this standpoint, the pragmatic end goal is to ensure that information shared with
target audiences is clear and comprehensible to them. One IPCC WGIII AR5 author described “the act
of translation to a policy context” as a crucial objective of the SPM negotiation, further explaining that
“policy makers can help shape that language in a way that communicates the ideas better”. A GEO-5
author stated that “we do not all speak the same language, we need to translate the researchers
language into the stakeholders language”. In an unrecorded interview, a government representative
involved in IPCC WGIII AR5 explained that some of the technical language often used by scientists in
the GEA process can be difficult to understand for many policy makers, in particular in less developed
countries. However, some interviewees felt that too much emphasis on communication could be
detrimental to scientific credibility, in particular when negotiating the SPM document. For example,
many interviewees from all groups felt that the findings in the SPM were “watered down” as a result
of the negotiation process. However, as one GEO-5 author pointed out, this is a “trade-off—by going
through with that [negotiation] the report gets more visibility, but some information gets lost”.
A second line of thinking focuses on the types of outputs produced. From this standpoint,
a major aspect of engaging with stakeholders with the objective of improving communication and
understanding is to determine which types of products are most appropriate and useful for different
audiences. Here, the practical goal is to produce these appropriate and useful products in order
to increase the likelihood that they will actually be used. This perspective was mentioned most
prominently by authors. For example, one GEO author suggested that “if they want to have an impact
on teaching, they should provide PowerPoint slides”, which could be best designed together with
experienced teachers to be used more easily in classrooms. An IPCC WGIII AR5 author described how
the GEA should tailor-make “presentations in some developing countries, so that policy makers can
understand better the issues”.
Sustainability 2017,9, 1571 11 of 21
Finally, a third line of thinking has to do with the strategic dissemination of GEAs’ findings in
order to exert an influence on global governance more broadly. This was most often mentioned by
government representatives with regards to engaging with stakeholders to secure a place for the GEA
in milestone global environmental governance events. For example, one government representative
in GEO-5 described the role of the assessment with regards to the United Nations Conference on
Sustainable Development (Rio+20), stating that “the more [GEO-5] findings are shared the more a wide
spectrum of the stakeholders gain the support for consequential actions in the right directions to
reach agreements at the international stage”. An unrecorded interview with an IPCC government
representative involved since the first assessment report stressed the connection between the IPCC
reports and the United Nations Framework Convention on Climate Change (UNFCCC) Conference of
the Party meetings, a major target audience for the IPCC and a means of exerting influence.
In GEA documents, this category of objectives was mentioned most prominently with regards
to the SPM negotiations. In GEO-5 this was done quite directly. For example, an important objective
of the GEO-5 SPM was to effectively “communicate the findings of the GEO-5 assessment and
maximize accessibility of GEO-5 information [to target audiences], including in terms of format
and languages” [
22
] (p. 8). This is very much in line with the first standpoint focusing on the act of
translating scientific findings. GEO-5 also strove to “make use of a range of multi-media and tools
and innovative approaches” to reach a diverse target audience [
22
] (p. 8), more in line with the second
standpoint on the types of outputs envisaged. In IPCC WGIII AR5 this objective was mentioned more
indirectly. For example, government representatives involved in the IPCC review process and SPM
negotiation must provide “integrated comments on the accuracy and completeness of the scientific
and/or technical content and the overall scientific and/or technical balance of the drafts” [23] (p. 16).
This would in part involve ensuring that the wording and framing are appropriate for the target
audience (in this case, for governments).
In the literature, discussions about communication and understanding primarily relate to the
efficient and effective transmission of information to target audiences. The literature stresses that while
scientists involved in GEAs are experts in their fields, they are not necessarily experts in communication,
requiring input from a broader group in order to improve understanding. This includes in particular
ensuring that GEA messages fulfill a demand from end-users [
5
,
20
] and are framed appropriately [
42
,
49
].
Another crucial aspect is the role of stakeholders in interpreting information, making often complex and
technical findings more usable [
52
], a highly pragmatic rationale akin to those which came up during
interviews regarding this category of objectives.
3.1.4. Ownership
The fourth category of objectives is to build up a sense of ownership over the GEA process or
its products amongst target audience members. While not mentioned in official GEA documents,
interviewees who discussed ownership saw this category of objectives as very important. Interviewees
often use the term ownership interchangeably with buy-in. Ownership was chosen as the heading for
this category because, as some interviewees described it, ownership can be a means of encouraging
stakeholders to buy into (or believe) a GEA process or products, whereas buy-in on its own does not
necessarily require engagement.
There were two general ways of describing this category of objectives in the interview material,
both mentioned primarily by authors and other scientific experts as well as by government
representatives to some extent, and both referring primarily to building a sense of ownership amongst
target audiences and in particular governmental actors. Firstly, interviewees talked about the objective
to build up a sense of ownership as a unique way that GEAs in particular can increase impact. One
government representative involved in IPCC WGIII AR5 described how “the degree of authority
and ownership is something that other good reports, like World Bank or other similar global reports,
don’t quite have”. An author involved in GEO-5 described how “people who are part of the process
become ambassadors for the results, and that’s how the assessment would increase its impact”. One
Sustainability 2017,9, 1571 12 of 21
scientific expert involved in the production of IPCC WGIII AR5 stated that “government buy-in is the
critical part of the whole process [
. . .
] that’s what gives the whole thing weight and allows it to have
credibility above and beyond normal scientific work”. Many interviewees linked this in particular to
the SPM negotiation (see Figure 1). One scientist with many years of experience working with GEO
assessments stated directly that “by having these really negotiated versions, governments feel a sense
of ownership [ . . . ] and do something with the report”.
The second way of describing this category of objectives was a more pragmatic perspective
regarding how exactly the objective of fostering ownership amongst different stakeholders might take
place in a GEA. One author involved in GEO-5 felt that this was a central concern for future GEAs,
asking “how do we download it to the country level how do we get buy-in from policy makers and
politicians?” One government representative involved in IPCC WGIII AR5 felt that “the larger buy-in
is there when all countries can participate”. A producer involved in GEO-5 explained how particular
strategies can help in building ownership, but come with trade-offs: “focusing on policy successes
probably did some good in achieving some buy-in from policy makers, but maybe the trade-off was
that problems most in need of policy attention were downplayed or not featured that strongly because
they didn’t have success stories in them”.
In the literature, building a sense of ownership is often linked to the GEA having a pragmatic
impact or influence along the lines of the first perspective above, with the assumption being that
stakeholders who are involved are more likely to feel ownership and in turn are more likely to actually
use the end products [
2
,
5
,
44
,
52
,
53
]. The most prominent example is the SPM negotiation, where
the engagement of government representatives is thought to increase their sense of ownership and
ultimately increase the influence of the final document on policy decisions [54].
3.1.5. Exerting Control
The fifth category of objectives involves different actors exerting some form of control over the
GEA process or its products. This category of objectives was never explicitly mentioned in official
GEA documents, but still came up in interviews to some extent. Very few interviewees described
themselves as holding this objective (rather discussing their observation that others exerted control),
and all descriptions referred to a specific instance of a stakeholder or group exerting control.
By far the most common examples given by interviewees had to do with government stakeholders
exerting control over the SPM document. Both scientific experts as well as government representatives
themselves discussed this standpoint. For example, one scientific expert who participated in the
IPCC WGIII AR5 SPM negotiation meeting felt that governments were “trying to skew the science
in their favour”. A government representative involved in that same meeting noted that many other
governments “were exerting a certain influence which was based on the political priorities of their
countries”. A high-level scientific expert involved in GEO-5 explained that “political influence had
too much influence on the Summary for Policy Makers”. One author wrote in a public letter after the
IPCC WGIII AR5 SPM negotiation meeting that “nearly all delegates in the meeting demonstrated
the same perspective and approach, namely that any text that was considered inconsistent with their
interests and positions in multilateral negotiations was treated as unacceptable” [
55
]. More broadly,
one government representative involved in GEO-5 observed that there has been a push for “much
more governmental control of all the sessions”, while a GEO-5 producer reflected that “you see
governments wanting more and more control over the assessment process”, implying that government
control may actually be increasing. As explained briefly in Section 3.1.3 above, government control
is often seen as negative, for example when the final SPM becomes too “watered down” as a result
of their involvement, sacrificing scientific credibility. Thus, increasing government control would
not likely be welcomed by many expert scientists. However, many scientists do still see the value of
relinquishing some control in particular in order to foster more ownership or buy-in. For example,
one highly experienced individual who has participated as an author and producer in both GEO and
IPCC assessments explained that while “the negotiation process eliminates a lot of the stuff which the
Sustainability 2017,9, 1571 13 of 21
scientists consider important, at least it’s some common point that has the buy-in from some policy
makers”, also indicating a trade-off with the objective of building a sense of ownership.
In addition, there were a few cases where scientists exerted control. One author involved in
GEO-5 described how they observed cases where “authors seemed to want to push what went into
these final chapters as a function of anticipated funding needs”. Another expert with a long history
of engaging with GEAs including all GEO assessments recalled numerous cases where “people [are]
coming in with their pet projects and favourite subjects”, insisting these are included in the assessment.
However, at least one scientific expert who has been involved in many GEA processes made the case
that scientific control did not have to be a negative thing. This interviewee stated that “a lot of time
you really have to sneak stuff in that you feel is a greater good”.
In the literature, control has long been linked with power relations, for example as used as
leverage during debates, in particular where competing values play a strong role [
56
]. Control
has been described as being exerted both between and within stakeholder groups. For example,
some individual governments or negotiating blocks may be more powerful than others [
57
], or some
scientific disciplines may be taken more seriously in debates [
28
]. In general, non-scientific stakeholders
exert comparatively less control over GEA processes, at least in part due to continuing emphasis on
quantitative and science-based knowledge in place of other perspectives [45,58].
3.1.6. Learning
The final category of objectives identified is learning. Learning via stakeholder engagement,
though not mentioned explicitly in GEA documents, was mentioned to some extent during interviews,
in particular by target audience members who had not actually been involved in GEO-5 or IPCC
WGIII AR5. These interviewees spoke primarily about the high potential for learning during GEAs,
how learning could or should be an objective. One target audience interviewee stated bluntly that
“knowledge exchange and learning should become one of the main objectives and one of the main
results of this kind of exercises”. Another described how GEAs are unique in bringing together such
a diversity of actors at the global scale, stating that “people can learn a lot from each other, across
different scales, different government levels, but also among government levels”. Another still focused
on how learning could take place by building connections to other groups, stressing that “the focus
should be on the communities that are built, you know, the social networks that are cultivated”.
Expert authors also mentioned learning as an objective of stakeholder engagement, though to
a lesser extent. Scientists had a different overall perspective on learning, focusing more on learning
by doing. For example, one author involved in GEO-5 explained how “working with people from
different backgrounds, disciplines, countries—I think we all learned something and gained capacities
through the process”. Other authors confirmed this statement and described, for example, learning
to work in an interdisciplinary environment, learning how to think collectively, or learning about
problems and potential solutions from entirely different perspectives. Another GEO-5 author described
the value added of learning through GEAs as opposed to other channels, explaining how “in those
[stakeholder] meetings, you could gather a lot of information and feelings and ideas that are not in
the research papers”. A highly experienced IPCC WGIII AR5 author stressed that GEAs provided
an important opportunity for learning amongst early career researchers, who often “have gone on to
become scholars in their own right, writing fabulous stuff—think about what a great training ground
it is”.
Government representatives also mentioned learning to some extent, but focused more on what
they needed to learn, emphasizing the demand-side of this category of objectives. One government
representatives involved in IPCC WGIII AR5 described how governments wanted to learn “what’s
new since the last report”. A government representative involved in GEO-5 described how “we learn
from China, Japan and others in the Pacific Region where they have similar problems and much better
approaches”.
Sustainability 2017,9, 1571 14 of 21
However, while learning was generally described positively, there were still reservations. One
government representative involved in IPCC WGIII AR5 in particular described how “governments
have better and better understood their respective constraints and what is at stake for each nation
and where they are situated globally and the things they can do, they have learned and learned and
learned”. This interviewee went on to explain that despite the clear evidence that learning has taken
place over many years of IPCC reports, there was still a lack of concrete policies in place, implying that
learning may simply not be sufficient on its own.
Described in the literature in a pragmatic sense, learning occurs through an uptake of relevant and
comprehensible information from the products of a GEA, and can be facilitated by participation [
43
,
49
,
53
].
Stakeholder engagement can engender social learning and behavioral change at multiple levels and can
also translate upwards to organizational learning, in GEAs and in other processes at the science-policy
interface [59–61].
3.2. Linkages between Objectives and Broader Deliberative Goals
The keyword analysis described in Section 2.2 yielded a series of text segments from interview
transcripts and GEA documents which described objectives for stakeholder engagement in GEAs
while also describing how these might contribute positively towards the concepts represented by the
keywords associated with deliberative policy learning. We found a total of 234 such overlaps between
categories of objectives and building blocks: 93 with representation, 71 with capacity building, 59 with
spaces for deliberation, and 11 with empowerment. Table 4shows the number of overlaps per source
(per interview or per GEA document) averaged by number of sources which mention each category of
objectives overall. This is representative of the percentage of sources which have described a category
of objectives which also explicitly mention a keyword associated with one of the building blocks of
deliberative policy learning.
Table 4.
The percentage of sources (interview material and GEA documents) who described a particular
category of objectives and also mentioned a keyword associated with a building block of deliberative
policy learning.
Communication Exerting Control Learning Source of
Information Ownership Dialogue
Representation 0.19 0.23 0.11 0.41 0.25 0.14
Empowerment 0.00 0.04 0.00 0.02 0.06 0.04
Capacity building 0.22 0.06 0.38 0.17 0.09 0.18
Spaces for deliberation 0.12 0.10 0.06 0.17 0.10 0.28
Source: Own elaboration.
The two strongest linkages found through this analysis are between the category of objectives
to foster learning outcomes and the “capacity building” building block, and between the category of
objectives to provide a source of information and the “representation” building block. The strongest
contribution to the building block of providing “spaces for deliberation” comes from the category
of objectives to foster dialogue. The two building blocks mentioned most explicitly in official GEA
documents [
24
,
25
], those of ensuring balanced representation and of contributing towards capacity
building, are the two which are most strongly linked with the highest number of categories of objectives.
GEA documents nearly always linked the building blocks “representation” and “capacity building”
with the category of objectives to provide a source of information. Only in one case did the GEO-5
document explicitly link the category of objectives to improve communication to the building block
“capacity building”.
As mentioned in Section 3.1.5, not all interviewees painted the objective of exerting control in
a negative light, with many implicitly connecting this objective with the building block “representation”
in the sense that the control exerted by some stakeholders (where interviewees referred primarily to
relatively more powerful government representatives) could be balanced by others exerting their own
Sustainability 2017,9, 1571 15 of 21
control (here, interviewees referred to either relatively weaker governments or scientists). Overall, the
“empowerment” building block was nearly never mentioned during interviews in connection with any
of the objectives. The keyword analysis further highlights that within a single category of objectives
there are many different perspectives regarding what exactly is meant by a specific objective. This has
been demonstrated by the fact that different interviewees connected certain categories of objectives with
certain aspects of deliberative policy learning whereas others did not. This can in turn have a strong
impact on the extent to which broader goals such as deliberative policy learning can realistically be
envisaged by targeting specific objectives from the different perspectives of the interviewees.
4. Discussion
Based on the diversity of (sometimes potentially incompatible) objectives described in Section 3.1
and the links between objectives and deliberative policy learning in Section 3.2, objectives for
stakeholder engagement in GEAs must be evaluated and prioritized in future studies with the
end goal of improving stakeholder engagement in GEAs. Going beyond the scope of this paper,
this presupposes a careful analysis of the direct and indirect (desirable and undesirable) effects of
implementing these objectives in GEA processes [
10
]. Among other things, such an evaluation will
require more attention to the feasibility of implementing objectives (Section 4.1); to metrics of success
associated with engagement objectives (Section 4.2); and to trade-offs between objectives (Section 4.3).
4.1. Feasibility and Selection of Objectives
One issue that connects all the findings in the Results Section is the extent to which different
objectives for stakeholder engagement or combinations of objectives are actually practically feasible in
a GEA. In order to avoid frustrations and unmet expectations at a later stage of the GEA process, it is
very important to ensure that the selected objectives can actually be achieved given available resources
and other contextual factors. An effective alignment of engagement objectives, methods and resources
has been shown to be absolutely crucial [10].
A first step towards addressing this issue would be to have open and explicit discussions of
objectives early in a GEA process, based on the results presented in Section 3. Such discussions could
help to identify potentially over-ambitious combinations of objectives and to ensure that the objectives
actually selected are clearly defined. This could paint a more accurate picture of the feasibility of the
objectives for engagement, including the required number and diversity of stakeholders, given the
limitless pool of potential stakeholders. For example, while the categories of objectives in Section 3
do not necessarily imply clear criteria for stakeholder selection, more specific objectives such as
diversifying sources of information can have severe resource implications regarding stakeholder
engagement activities. Clarifying the objectives selected and explicitly acknowledging the diversity of
potential objectives can help individuals distinguish and find a balance between their own personal
perspective on objectives, the perspective of the institution they represent, and the officially sanctioned
objectives for engagement.
Selecting the objectives for stakeholder engagement in GEAs can be facilitated and more clearly
justified by linking the objectives to a higher-level rationale. A rationale such as deliberative policy
learning can be easier to agree on, and can then help to ground discussions over which objectives to
prioritize and ultimately select for stakeholder engagement activities.
4.2. Indicators for Success
The development of (context-dependent) metrics of success for stakeholder engagement in
GEAs, for example in the form of indicators, is crucial to strengthen evaluation and foster iterative
improvement, especially with a view towards future decisions about objectives for engagement and
related funding requirements. Our interview results demonstrate that such indicators can and should
be developed in parallel to early discussions about objectives with input from a broad diversity of actors.
The characteristics and perspectives associated with the categories of objectives in Section 3.1 already
Sustainability 2017,9, 1571 16 of 21
often point towards potential indicators for success, in particular when interviewees specifically discuss
whether or not the objectives were achieved. This can be seen, for example, when interviewees describe
the mix of participants as sufficiently (or insufficiently) diverse based on institutional affiliation or
country of origin. Thus, tracking the diversity of actors based on criteria such as these could be
seen as one indicator of success for the objective to provide a source of information, in particular
with regards to the standpoint that the objective should serve to diversify the sources of information.
Another example can be highlighted within the category of objectives to improve communication
and understanding, specifically with the focus on tailoring wording and presentation to the target
audience. Here, an indicator could be that, from the perspective of government representatives as
the target audience, the messages are clearer and more useful after stakeholder engagement activities
such as the SPM negotiation. Linking categories of objectives to higher-level rationales for stakeholder
engagement, such as deliberative policy learning, can also serve to specify what types of indicators
could be appropriate. For example, stating that providing a source of information is an important
objective might not necessarily imply a specific indicator for success, and could be interpreted in many
different ways. However, explicitly linking this category of objectives to the underlying rationale of
ensuring that different viewpoints are represented in a fair and balanced manner, a building block of
deliberative policy learning, points more clearly towards specific indicators. These indicators could
include, for example, measures of diversity or representation, and could be measured, for example,
using stakeholder analysis or mapping techniques.
There have been many examples of successful stakeholder engagement in GEAs or similar global
mechanisms with regards to different criteria. For example, the Millennium Ecosystem Assessment
was largely praised for its broad representation of multiple actor groups. Engagement activities during
this assessment process, in multi-stakeholder meetings for example, but also in developing scenarios
and in other instances, explicitly sought information from a broad diversity of knowledge systems
relevant to decision-makers at multiple scales and to many different scientific disciplines [
5
,
9
]. The
UNFCCC has, beginning in 2013, conducted four Structured Expert Dialogue sessions closely linked
with IPCC AR5, which successfully brought together scientific experts from multiple disciplinary
backgrounds and national government representatives in open discussions away from the pressures of
a negotiation to foster learning [
1
,
62
]. A third example of success with regards to the indicator of high
participants diversity can be found in the case of IPBES, which places more emphasis on indigenous
and local knowledge than any other GEA to date, and already includes actors coming from a diversity
of cultural backgrounds and with different types of knowledge in meetings such as the pre-plenary
Stakeholder Days [10,14,15].
A process of determining indicators for success would ideally involve the diversity of relevant
stakeholders from the outset. However, this involves practical challenges. For example, even though
a large number of interviews were conducted for this paper (99), more marginalized stakeholders
were not reached. The lack of marginalized or less empowered viewpoints in the analysis could be
one reason that the building block of deliberative policy learning “empowerment” was nearly never
mentioned in conjunction with any of the categories of objectives. The method of linking objectives
to higher-level rationales, such as deliberative policy learning (Section 3.2) could be improved by
adapting it to actual practice in GEAs. For instance, GEA producing organizations could widely
circulate a well-designed survey to gather even more information on different perspectives (including
more marginalized viewpoints) on indicators for success of stakeholder engagement.
4.3. Trade-Offs between Objectives
As mentioned, in addition to selecting objectives based on their individual characteristics, it is also
important to understand how they might play out in different combinations. To this end, understanding
potential trade-offs and synergies between objectives, and how these relate to the different perspectives,
can be very important. Some of the different viewpoints on objectives which imply potential trade-offs
have been highlighted in the Results, including different perspectives on the extent to which the
Sustainability 2017,9, 1571 17 of 21
sources of information should be diversified, on how much emphasis to put on communication in
light of the potential trade-offs with scientific credibility, or on how desirable control is as an objective
and in which cases it might be acceptable. We can also consider a hypothetical example where the
objective for a particular stakeholder engagement method within a GEA is to foster dialogue, a sense
of ownership or learning exclusively with regard to national governments rather than amongst a more
diverse group. On the one hand, this may, under certain conditions, reduce funding needs and perhaps
even reduce some tensions within GEA processes by limiting the number of potential stakeholders
to engage with. However, this would create significant trade-offs with other possible engagement
objectives, particularly with the ambition to ensure the inclusion in the GEA process of a diverse and
balanced set of viewpoints stemming from multiple perspectives and to give diverse actor groups the
chance to exert some form of control over GEA processes.
The most prominent example of trade-offs between objectives to come out of the analysis is
between securing ownership amongst governments on the one hand and allowing governments some
form of control on the other. Many scientists explicitly pursue governmental buy-in in order to ensure
that their research and their work on a GEA have an influence on policy. In exchange for this, many are
willing to relinquish some degree of control over the process and the final products. Governments often
seek to exert control inter alia to maintain a say over the information which will become crucial inputs
to international negotiations (for example the UNFCCC negotiations, where the IPCC SPM documents
are generally taken as agreed text, as highlighted by the interviewees and in the literature [
63
]). Based
on our interview results, it seems clear that there is an implicit deal struck at least between parts of
these groups, where each accepts the position of the other to achieve their own objectives. While some
interviewees merely alluded to this deal, some (and in particular highly experienced contributors to
GEAs) mentioned it explicitly (Sections 3.1.4 and 3.1.5).
This deal is one way to visualize the very important social dynamics at play between expert
scientists and government representatives which occur at the very center of the GEA enterprise. The
engagement of these two groups in particular is absolutely crucial to the continued production of GEAs,
since after all they are often the ones who mandate, provide an audience for, and actually write the GEA
products among many other roles. In light of this, it becomes highly important to consider what might
happen to GEAs if this deal were to break down, as may be the case according to some interviewees
and other claims that, for example, the “watering down” of the SPM has gone too far (Sections 3.1.3
and 3.1.5) [
10
,
55
]. One way forward would be to explicitly avoid prioritizing the engagement of these
two particular groups above others and to purposefully open up the entire governance structures of
GEAs, providing balance to the inputs from governments and scientific authors by including a diversity
of perspectives. Governance of GEAs along the lines of the Pragmatic Enlightened Model [
64
], already
attempted in part in IPCC WGIII AR5, would even go a step further. In addition to involving a diversity
of stakeholders, this model advocates for highly inclusive exploration of policy alternatives in GEAs,
giving scientists the opportunity to make significant contributions to policy debates by freely exploring
policy pathways, while also ensuring that the experts are not prescribing a particular policy pathway,
which many governments would appreciate. This model would also in general limit the control exerted
by both governments and scientists by including many more stakeholder perspectives for example
when determining guiding questions or frameworks, or when evaluating different policy pathways.
Such a model could effectively pave the way for an updated deal to guide GEAs and their engagement
of stakeholders into the future.
5. Conclusions
This study has shed light on the large variety of perspectives on objectives for stakeholder
engagement in GEAs, which are crucial tools in international environmental governance. Engaging
with a diversity of stakeholders in these processes is important inter alia to create robustness and
resilience in environmental governance [
16
,
65
]. Understanding the diversity of existing objectives for
stakeholder engagement in GEAs can help to improve these activities. Categorizing and characterizing
Sustainability 2017,9, 1571 18 of 21
the objectives for stakeholder engagement in GEAs (Section 3.1) provides a crucial first step towards
grounding normative discussions and clarifying the objectives for engagement in future GEAs. This is
in turn is a prerequisite for justifying stakeholder engagement, especially in light of the high costs and
challenges involved, and improving the methods for engagement, including selecting stakeholders
based on an improved understanding of the actor constellations and social dynamics central to
GEAs. Improvements to engagement activities can then take place through an evolutionary process,
encouraging systematic learning from success and failures (based on objectives and explicitly taking
into account trade-offs between objectives) (Section 4), and by explicitly adapting engagement to
the new environmental governance contexts (Section 1). To our knowledge, this study presents the
most comprehensive systematic exploration of current objectives for stakeholder engagement in GEAs
to date.
Successful stakeholder engagement in GEAs can increase the effectiveness and adaptiveness
of GEAs more broadly, particularly when moving towards more deliberative and inclusive
assessments—going beyond the traditional objectives for stakeholder engagement focusing in
particular on the dissemination of results and governmental buy-in. The Paris Agreement, for example,
has made the stakes much more concrete when it comes to solutions to climate change. In light of
this, the IPCC, and in particular WGIII focusing on mitigation options, must adapt its processes and
procedures in order to more explicitly take into account these increasingly high and diverse stakes, and
must govern its assessment processes in a more inclusive and interactive manner regarding non-state
actors [
16
]. As argued in Section 4, this presupposes not only selecting feasible, clear and well-justified
objectives, but also indicators of success and appropriate methods for engagement.
Future research may focus on: (1) the motivations and rationales, but also on the external factors
underlying diverging perspectives on objectives for stakeholder engagement; (2) normative evaluation
of objectives along the lines outlined in Section 4; and (3) more systematic research on specific
stakeholder engagement methods and outcomes in GEAs and beyond, including a more in-depth
exploration of the trade-offs between objectives as well as implications of objectives for the selection of
stakeholders, for the timing of engagement, and for the required resources and procedures. In addition,
understanding and strengthening the relationships between the individuals involved in GEA processes
is of central importance for future research and practice.
Improving our understanding of the diversity of objectives for stakeholder engagement in
GEAs can help support appropriate evaluation, can motivate engagement, and can strengthen the
relationships which make GEAs function effectively and efficiently. This is a crucial point to address
for future GEAs in order to adapt to the evolving environmental governance landscape.
Acknowledgments:
This research was conducted as part of a larger research initiative on “The Future of Global
Environmental Assessment Making” (2013–2016) funded by MCC and UN Environment (see https://www.
mcc-berlin.net/en/research/cooperation/unep.html). The research design was aided greatly by insightful
conversations with the director of MCC, Ottmar Edenhofer, and many other colleagues who participated in the
IPCC WGIII AR5 process.
Author Contributions:
Jennifer Garard developed the research framing and analyzed all the empirical data.
Martin Kowarsch, contributed to the development of the storyline and conceptual framework of the article.
Jennifer Garard wrote the paper.
Conflicts of Interest:
The authors declare no conflict of interest. The founding sponsors had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the
decision to publish the results.
References
1.
Kowarsch, M.; Garard, J.; Riousset, P.; Lenzi, D.; Dorsch, M.J.; Knopf, B.; Harrs, J.-A.; Edenhofer, O. Scientific
Assessments to Facilitate Deliberative Policy Learning. Palgrave Commun. 2016,2. [CrossRef]
2.
Clark, W.C.; Mitchell, R.B.; Cash, D.W. Evaluating the influence of global environmental assessments.
In Global Environmental Assessments: Information and Influence; Mitchell, R.B., Clark, W.C., Cash, D.W.,
Dickson, N.M., Eds.; MIT Press: Cambridge, MA, USA, 2006; pp. 1–28.
Sustainability 2017,9, 1571 19 of 21
3.
Beck, S.; Borie, M.; Chilvers, J.; Esguerra, A.; Heubach, K.; Hulme, M.; Lidskog, R.; Lövbrand, E.; Marquard, E.;
Miller, C.; et al. Towards a Reflexive Turn in the Governance of Global Environmental Expertise. The Cases
of the IPCC and the IPBES. GAIA—Ecol. Perspect. Sci. Soc. 2014,23, 80–87. [CrossRef]
4.
Kowarsch, M. Fact value conflation and the danger of the traditional models. In A Pragmatist Orientation
for the Social Sciences in Climate Change: How to Make Integrated Economic Assessments Serve Society; Boston
Studies in the Philosophy and History of Science; Springer International Publishing: Basel, Switzerland,
2016; pp. 101–132.
5.
Reid, W.V.; Berkes, F.; Wilbanks, T.J.; Capistrano, D. Introduction. In Bridging Scales and Knowledge Systems:
Concepts and Applications in Ecosystem Assessment; Reid, W.V., Berkes, F., Wilbanks, T.J., Capistrano, D., Eds.;
World Resources Institute: Washington, DC, USA, 2006; pp. 1–17.
6.
Jabbour, J.; Flachsland, C. 40 years of global environmental assessments: A retrospective analysis.
Environ. Sci. Policy 2017. [CrossRef]
7.
Watson, R.T. Turning science into policy: Challenges and experiences from the science–policy interface.
Philos. Trans. R. Soc. Lond. B 2005,360, 471–477. [CrossRef] [PubMed]
8.
Riousset, P.; Flachsland, C.; Kowarsch, M. Global environmental assessments: Impact mechanisms.
Environ. Sci. Policy 2017. [CrossRef]
9.
Norgaard, R.B. Finding hope in the millennium ecosystem assessment. Conserv. Biol.
2008
,22, 862–869.
[CrossRef] [PubMed]
10.
Garard, J.; Kowarsch, M. If at first you don’t succeed: Evaluating stakeholder engagement in global
environmental assessments. Environ. Sci. Policy 2017. [CrossRef]
11.
Reed, M.S.; Graves, A.; Dandy, N.; Posthumus, H.; Hubacek, K.; Morris, J.; Prell, C.; Quinn, C.H.; Stringer, L.C.
Who’s in and why? A typology of stakeholder analysis methods for natural resource management.
J. Environ. Manag. 2009,90, 1933–1949. [CrossRef] [PubMed]
12.
Colvin, R.M.; Witt, G.B.; Lacey, J. Approaches to identifying stakeholders in environmental management:
Insights from practitioners to go beyond the ‘usual suspects’. Land Use Policy 2016,52, 266–276. [CrossRef]
13.
Koetz, T.; Farrell, K.N.; Bridgewater, P. Building better science-policy interfaces for international
environmental governance: Assessing potential within the Intergovernmental Platform for Biodiversity and
Ecosystem Services. Int. Environ. Agreem. 2012,12, 1–21. [CrossRef]
14.
Díaz, S.; Demissew, S.; Carabias, J.; Joly, C.; Lonsdale, M.; Ash, N.; Larigauderie, A.; Adhikari, J.R.;
Arico, S.; Báldi, A.; et al. The IPBES Conceptual Framework—Connecting nature and people. Curr. Opin.
Environ. Sustain. 2015,14, 1–16. [CrossRef]
15.
Pascual, U.; Balvanera, P.; Díaz, S.; Pataki, G.; Roth, E.; Stenseke, M.; Watson, R.T.; Ba¸sak Dessane, E.; Islar, M.;
Kelemen, E.; et al. Valuing nature’s contributions to people: The IPBES approach. Curr. Opin. Environ. Sustain.
2017,26, 7–16. [CrossRef]
16.
Kowarsch, M.; Jabbour, J.; Flachsland, C.; Kok, M.T.J.; Watson, R.; Haas, P.M.; Minx, J.C.; Alcamo, J.; Garard, J.;
Riousset, P.; et al. A road map for global environmental assessments. Nat. Clim. Chang.
2017
,7, 379–382.
[CrossRef]
17.
Esguerra, A.; Beck, S.; Lidskog, R. Stakeholder Engagement in the Making: IPBES Legitimization Politics.
Glob. Environ. Politics 2016,17, 59–76. [CrossRef]
18.
Jasanoff, S. Ordering knowledge, ordering society. In States of Knowledge: The Co-Production of Science and
Social Order; Routledge: London, UK, 2004; pp. 13–44.
19.
Sanders, E.B.-N.; Stappers, P.J. Co-creation and the new landscapes of design. Co-Design
2008
,4, 5–18.
[CrossRef]
20.
Andonova, L.B. Structure and influence of international assessments: Lessons from central and eastern
Europe. In Global Environmental Assessments: Information and Influence; Mitchell, R.B., Clark, W.C., Cash, D.W.,
Dickson, N.M., Eds.; MIT Press: Cambridge, MA, USA, 2006; pp. 151–173.
21.
Miller, C.; Erickson, P. The politics of bridging scales and epistemologies: Science and democracy in global
environmental governance. In Bridging Scales and Knowledge Systems: Concepts and Applications in Ecosystem
Assessment; Reid, W.V., Berkes, F., Wilbanks, T.J., Capistrano, D., Eds.; World Resources Institute: Washington,
DC, USA, 2006; pp. 297–314.
22. Van der Hel, S. New science for global sustainability? The institutionalisation of knowledge co-production
in Future Earth. Environ. Sci. Policy 2016,61, 165–175. [CrossRef]
Sustainability 2017,9, 1571 20 of 21
23.
Wynne, B. Public Engagement as a Means of Restoring Public Trust in Science—Hitting the Notes, but
Missing the Music? Public Health Genom. 2006,9, 211–220. [CrossRef] [PubMed]
24.
United Nations Environment Programme (UNEP). Statement by the Global Intergovernmental and
Multi-Stakeholder Consultation on the Fifth Global Environment Outlook; UNEP: Nairobi, Kenya, 2010; p. 9.
25.
Intergovernmental Panel on Climate Change (IPCC). Procedures for the Preparation, Review, Acceptance,
Adoption, Approval and Publication of IPCC Reports; IPCC: Geneva, Switzerland, 2008.
26. Strauss, A.; Corbin, J. The Basics of Qualitative Research; Sage Publications: Thousand Oaks, CA, USA, 1998.
27.
Yamineva, Y. Lessons from the Intergovernmental Panel on Climate Change on inclusiveness across
geographies and stakeholders. Environ. Sci. Policy 2017. [CrossRef]
28.
Corbera, E.; Calvet-Mir, L.; Hughes, H.; Paterson, M. Patterns of authorship in the IPCC Working Group III
report. Nat. Clim. Chang. 2016,6, 94–99. [CrossRef]
29.
Victor, D. Climate change: Embed the social sciences in climate policy. Nature
2015
,520, 27–29. [CrossRef]
[PubMed]
30.
Palsson, G.; Szerszynski, B.; Sörlin, S.; Marks, J.; Avril, B.; Crumley, C.; Hackmann, H.; Holm, P.; Ingram, J.;
Kirman, A.; et al. Reconceptualizing the ‘Anthropos’ in the Anthropocene: Integrating the social sciences
and humanities in global environmental change research. Environ. Sci. Policy 2013,28, 3–13. [CrossRef]
31.
Wiklund, H. In search of arenas for democratic deliberation: A Habermasian review of environmental
assessment. Impact Assess. Proj. Apprais. 2005,23, 281–292. [CrossRef]
32.
Berkes, F.; Reid, W.V.; Wilbanks, T.J.; Capistrano, D. Conclusion: Bridging scales and knowledge systems.
In Bridging Scales and Knowledge Systems: Concepts and Applications in Ecosystem Assessment; Reid, W.V.,
Berkes, F., Wilbanks, T.J., Capistrano, D., Eds.; World Resources Institute: Washington, DC, USA, 2006;
pp. 315–331.
33.
Goodin, R.E. Who counts? In Innovating Democracy: Democratic Theory and Practice after the Deliberative Turn;
Oxford University Press: Oxford, UK, 2008; pp. 127–154.
34. Chambers, S. Deliberative Democratic Theory. Annu. Rev. Political Sci. 2003,6, 307–326. [CrossRef]
35.
Miller, C. Resisting empire: Globalism, relocalization, and the politics of knowledge. In Earthly Politics: Local
and Global in Environmental Governance; Jasanoff, S., Martello, M.L., Eds.; MIT Press: Cambridge, MA, USA,
2004; pp. 81–102.
36.
Cohen, J. Deliberation and democratic legitimacy. In Deliberative Democracy: Essays on Reason and Politics;
Bohman, J., Rehg, W., Eds.; MIT Press: Cambridge, MA, USA, 1997; pp. 67–91.
37.
Integrated Environmental Assessment: Training Manual. Available online: http://www.iisd.org/sites/
default/files/publications/iea_brochure.pdf (accessed on 25 May 2017).
38.
Strandberg, K.; Grönlund, K. Online deliberation: Theory and practice in virtual mini-publics. In Deliberative
Mini-Publics: Involving Citizens in the Democratic Process; Grönlund, K., Bächtiger, A., Setälä, M., Eds.;
ECPR Press: Colchester, UK, 2014; pp. 93–113.
39.
Cole, D. Advantages of a Polycentric Approach to Climate Change Policy. Nat. Clim. Chang.
2015
,5, 114–118.
[CrossRef]
40.
Lemos, M.C.; Morehouse, B.J. The co-production of science and policy in integrated climate assessments.
Glob. Environ. Chang. 2005,15, 57–68. [CrossRef]
41.
Vohland, K.; Mlambo, M.C.; Horta, L.D.; Jonsson, B.; Paulsch, A.; Martinez, S.I. How to ensure a credible
and efficient IPBES? Environ. Sci. Policy 2011,14, 1188–1194. [CrossRef]
42.
Engels, A. Anthropogenic climate change: How to understand the weak links between scientific evidence,
public perception, and low-carbon practices. Energy Emiss. Control Technol. 2016,4, 17–26. [CrossRef]
43.
Fabricius, C.; Scholes, R.; Cundill, G. Mobilizing knowledge for integrated ecosystem assessments. In Bridging
Scales and Knowledge Systems: Concepts and Applications in Ecosystem Assessment; Reid, W.V., Berkes, F.,
Wilbanks, T.J., Capistrano, D., Eds.; World Resources Institute: Washington, DC, USA, 2006; pp. 165–182.
44.
Mitchell, R.B.; Clark, W.C.; Cash, D.W.; Dickson, N.M. Global Environmental Assessments: Information and
Influence; MIT Press: Cambridge, MA, USA, 2006.
45.
Ford, J.D.; Vanderbilt, W.; Berrang-Ford, L. Authorship in IPCC AR5 and its implications for content: Climate
change and Indigenous populations in WGII. Clim. Chang. 2012,113, 201–213. [CrossRef] [PubMed]
46.
Dietz, T. Bringing values and deliberation to science communication. Proc. Natl. Acad. Sci. USA
2013
,110,
14081–14087. [CrossRef] [PubMed]
Sustainability 2017,9, 1571 21 of 21
47.
Cornell, S.; Berkhout, F.; Tuinstra, W.; Tàbara, J.D.; Jäger, J.; Chabay, I.; de Wit, B.; Langlais, R.; Mills, D.;
Moll, P.; et al. Opening up knowledge systems for better responses to global environmental change.
Environ. Sci. Policy 2013,28, 60–70. [CrossRef]
48.
Hage, M.; Leroy, P.; Petersen, A.C. Stakeholder participation in environmental knowledge production.
Futures 2010,42, 254–264. [CrossRef]
49.
Reed, M.S. Stakeholder participation for environmental management: A literature review. Biol. Conserv.
2008,141, 2417–2431. [CrossRef]
50.
Stirling, A. “Opening Up” and “Closing Down”: Power, Participation, and Pluralism in the Social Appraisal
of Technology. Sci. Technol. Hum. Values 2008,33, 262–294. [CrossRef]
51.
Stevenson, H.; Dryzek, J.S. The legitimacy of multilateral climate governance: A deliberative democratic
approach. Crit. Policy Stud. 2012,6, 1–18. [CrossRef]
52.
Field, C.B.; Barros, V.R. Added value from IPCC approval sessions. Science
2015
,350, 36. [CrossRef]
[PubMed]
53.
Leemans, R. Personal experiences with the governance of the policy-relevant IPCC and Millennium
Ecosystem Assessments. Glob. Environ. Chang. Hum. Policy Dimens. 2008,18, 12–17. [CrossRef]
54.
Agrawala, S. Structural and Process History of the Intergovernmental Panel on Climate Change. Clim. Chang.
1998,39, 621–642. [CrossRef]
55.
Stavins, R. Is the IPCC Government Approval Process Broken? 2014. Available online: http://
www.robertstavinsblog.org/2014/04/25/is-the-ipcc-government-approval-process-broken-2/ (accessed on
25 May 2017).
56.
Deitelhoff, N. Is fair enough? Legitimation internationeln Regierens durch deliberative Verfahren.
In Transnationale Gerechtigkeit und Demokratie; Niesen, P., Ed.; Campus: Frankfurt, Germany, 2012; pp. 103–130.
57.
Siebenhüner, B. The changing role of nation states in international environmental assessments—The case of
the IPCC. Glob. Environ. Chang. 2003,13, 113–123. [CrossRef]
58.
Klenk, N.L.; Meehan, K.; Pinel, S.L.; Mendez, F.; Lima, P.T.; Kammen, D.M. Stakeholders in climate science:
Beyond lip service? Science 2015,350, 743–744. [CrossRef] [PubMed]
59. Fiol, C.M.; Lyles, M.A. Organizational Learning. Acad. Manag. Rev. 1985,10, 803–813. [CrossRef]
60.
Siebenhüner, B. Social Learning in the Field of Climate Change; Carl von Ossietzky University of
Oldenburg: Oldenburg, Germany, 2006; Available online: https://pdfs.semanticscholar.org/da05/
2d23aa7b9a4541c9514a667d8b89a542b887.pdf (accessed on 25 May 2017).
61.
Gerlak, A.K.; Heikkila, T.; Smolinski, S.L.; Huitema, D.; Armitage, D. Learning our way out of environmental
policy problems: A review of the scholarship. Policy Sci. 2017, 1–37. [CrossRef]
62.
United Nations Framework Convention on Climate Change (UNFCCC). The Structured Expert Dialogue—The
2013–2015 Review; UNFCCC: New York, NY, USA, 2015.
63.
Carraro, C.; Edenhofer, O.; Flachsland, C.; Kolstad, C.; Stavins, R.; Stowe, R. The IPCC at a crossroads:
Opportunities for reform. Science 2015,350, 34–35. [CrossRef] [PubMed]
64.
Edenhofer, O.; Kowarsch, M. Cartography of pathways: A new model for environmental policy assessments.
Environ. Sci. Policy 2015,51, 56–64. [CrossRef]
65.
Lemos, M.C.; Agrawal, A. Environmental Governance. Annu. Rev. Environ. Resour.
2006
,31, 297–325.
[CrossRef]
©
2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).
