CO-SUSTAIN: Impact Assessment Framework

Impact Assessment Framework (IAF) Deliverable 1.3 Pathways for CO-creation between local authorities and collective actions for a SUSTAINable transition Grant Agreement n° 101132467 Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Ref. Ares(2025)2587507 - 31/03/2025 CO – SUSTAIN - Deliverable 1.3 2 / 36 Start date of the project 01/01/2024 Duration of the project 36 months Project’s website https://co-sustain.eu/ Deliverable n° & name D1.3 - Data Management Plan Version 1 Work Package n° 1 Due date of the deliverable 31/03/2025 Beneficiary responsible University of Tartu (UT) Main author(s) Lilian Pungas, Aet Annist Contributors Bianka Plüschke-Altof, Anneli Kährik, Dagmar Narusson, Helina Tamm (UT) Fiona Lilith Medea de Fontana (BOKU) Davide Grasso (UNITO) Matti Kojo, Maare Kais (LUT) Marina Frolova (UGR) Joana Munó Olivé (Ecoserveis) Nature of the deliverable R Document, report (excluding the periodic and final reports) X DEM Demonstrator, pilot, prototype, plan designs DEC Websites, patents filing, press & media actions, videos, etc. DATA Data sets, microdata, etc. DMP Data management plan ETHICS Deliverables related to ethics issues SECURITY Deliverables related to security issues OTHER Software, technical diagram, algorithms, models, etc. Dissemination level PU Public, fully open, e.g. web (Deliverables flagged as public will be automatically published in CORDIS project’s page) X SEN Sensitive, limited under the conditions of the Grant Agreement CO – SUSTAIN - Deliverable 1.3 3 / 36 Quality procedure Date Version Reviewer Action 18/03/2025 V1 Alessandro Sciullo (UNITO) Matti Kojo (LUT) First review 31/03/2025 V2 Alessandro Sciullo (UNITO) Matti Kojo (LUT) FInal review and submission CO – SUSTAIN - Deliverable 1.3 4 / 36 TABLE OF CONTENTS Project abstract ............................................................................................................................................. 5 Executive summary ....................................................................................................................................... 5 Abbreviations ................................................................................................................................................ 6 1 Introduction: the importance of assessing deliberative and participatory processes .......................... 7 2 Impact Assessment Framework: approaches, objects, tools and methodologies ................................ 8 2.1 Approaches .................................................................................................................................... 8 2.1.1 Theory of Change (ToC) ......................................................................................................... 8 2.1.2 Participatory System Mapping (PSM) .................................................................................... 8 2.2 Objects of assessment ................................................................................................................... 9 2.2.1 Case studies ........................................................................................................................... 9 2.2.2 Deliberation process ............................................................................................................ 13 2.2.3 Design of the Impact Assessment Framework .................................................................... 14 2.3 Tools and Methodologies ............................................................................................................ 17 3 Conclusion ........................................................................................................................................... 19 4 References ........................................................................................................................................... 20 5 Annexes ............................................................................................................................................... 22 5.1 Pre/Post-panel individual survey questionnaire preliminary version (to be finalised in parallel with emerging Case Study data) .............................................................................................................. 22 5.2 Pre/Post-panel individual semi-structured interview guide ....................................................... 25 5.3 Expert survey (internal use) ........................................................................................................ 27 5.4 Deliberation observation (internal use) ...................................................................................... 28 5.5 Serious games .............................................................................................................................. 30 5.6 Impact stories .............................................................................................................................. 34 CO – SUSTAIN - Deliverable 1.3 5 / 36 PROJECT ABSTRACT In Western democracies, traditional institutional participation is in decline while non-institutional participation has been increasing. Non-institutional participation for the climate transition often relies on a prefigurative approach, thus creating spaces to incubate alternative ideas and novel forms of political participation (niches). Empowering these forms of political participation to encourage niche innovations will provoke the radical yet necessary changes in the socio-technical system for a climate transition. The CO-SUSTAIN project seeks to address this opportunity for a democratic climate transition, by defining and testing new democratic pathways enabling local policymakers to support various novel forms of political participation and empowering citizens to act for a sustainable transition. To develop a better understanding of political participation linked to environmental, political and societal imperatives, CO-SUSTAIN will study 18 historic examples in 6 different European countries for each of the latent and manifest forms of political participation underlined by Ekman: involvement, civic engagement, formal political participation, and activism. It will use institutional ethnography and system mapping to understand the dynamics of the participation and its management, thus delivering best practices to stimulate and support political participation around these imperatives. These best practices will serve to define interventions for solution co-creation in six case studies that represent different forms of political participation: involvement through Spanish energy communities, civic engagement through Food Solidarity in Turin (IT), manifest political participation through participatory processes promoted by the government in Northern Europe (EE, FI) and activism through the Lobau Bleibt social movement (AT). The outputs and outcomes of the deliberations will be assessed to draw conclusions for more democratic climate policymaking across Europe. This is what the following report is focused on. EXECUTIVE SUMMARY The Impact Assessment Framework (IAF) provides a roadmap for the impact assessment of deliberation panels, including indicators and goals. IAF’ s goal is to gain a comprehensive understanding of the impact of the interventions co-created with the case study groups. For this, IAF defines approaches and tools to evaluate changes at both the individual level (through individual interviews and surveys) and the group level (using impact stories and participatory observation of deliberative processes). As such, the IAF enables qualitative and quantitative assessment of the changes brought about by project interventions in terms of social cohesion, empowerment, inclusion/exclusion, and capacity and community building.. CO – SUSTAIN - Deliverable 1.3 6 / 36 ABBREVIATIONS Abbreviation Definition CO-SUSTAIN Pathways for CO-creation between local authorities and collective actions for a SUSTAINable transition CLD Causal Loop Diagram DMP Data Management Plan DOI Digital Object Identifier EOSC European Open Science Cloud EU European Union FAIR FAIR data principles: Findability, Accessibility, Interoperability and Reusability GDPR General Data Protection Regulation IPR Intellectual Property Rights IAF Impact Assessment Framework NGO Non-governmental organisation OAIS Open Archival Information System PI Project Investigator PSM Participatory System Mapping ToC Theory of Change Table 1. List of abbreviations CO – SUSTAIN - Deliverable 1.3 7 / 36 1 INTRODUCTION: THE IMPORTANCE OF ASSESSING DELIBERATIVE AND PARTICIPATORY PROCESSES To respond to growing calls for improved representation and more opportunities for public involvement in decision-making, governments at all levels have increasingly turned to citizens' assemblies, citizens' juries, and other forms of representative deliberative processes to address complex policy issues (OECD, 2020; Papadopoulus & Warin, 2007; Dryzek, 2000). Beginning in the 1980s, this trend has not only persisted but has gained momentum in recent years (Curato et al. 2022). The COVID-19 pandemic brought unprecedented challenges to governments and had a direct impact on citizen participation, civic spaces, transparency, and access to information. The crisis underscored the need for even greater citizen involvement and deliberation to support democratic systems during such exceptional times, spurring innovation in this area. However, the growing reliance on representative deliberative processes has also highlighted a significant gap: the lack of clear guidelines for evaluating the implementation and the impact of these processes. To address this need, various organisations have developed guidelines for policymakers and practitioners who wish to assess the representative deliberative processes they initiate, commission, or implement (OECD 2021a, 2021b; Demski et al., 2024). These guidelines outline minimum standards and criteria for evaluating such processes. Based on the combination of OECD evaluation guidelines, a critical and partial application of the Logical Framework Approach (Chambers & Pettit, 2004) as well as various other approaches we have developed an Impact Assessment Framework (IAF) for CO-SUSTAIN to assess the quality of our deliberative panels throughout the chain of effects: from project outputs to outcomes and their potential impact. CO-SUSTAIN’s main objective is to define and test new democratic pathways enabling local policymakers to support various and novel forms of political participation, as well as to empower citizens to act for a sustainable transition. As such, the following IAF will firstly deliver an overarching coherent framework for the CO-SUSTAIN project partners as well as the general public to evaluate the outputs and outcomes in analysing, initiating and implementing democratic deliberative processes for sustainable transition. In the second step, this assessment of deliberative governance will provide crucial and comparative insights for the policymakers to better understand the potential, benefits and management of such novel forms of political participation. The broader goal of the IAF is to encourage public authorities, policymakers and other relevant stakeholders to learn from the deliberative processes to further foster and enhance processes of co-creation, citizen empowerment and civic participation in the future. Additionally, a shared evaluation framework such as IAF can also help to improve the design of the future deliberative processes and generate data for comparative analysis. This framework is neither prescriptive nor exhaustive. Rather, it provides a set of evaluation criteria for the CO-SUSTAIN project to assess the impact of the project’s planned interventions within the 6 different case studies that will be investigated (see section 2.2.1) and contributes to a coherent and comparable approach across all case studies. However, being an overarching conceptual and methodological framework, IAF will be expanded, refined and revised with additional case study-specific and context- / stakeholder-dependent criteria as established necessary in the case study design process that will be finalized by June 2025. CO – SUSTAIN - Deliverable 1.3 8 / 36 2 IMPACT ASSESSMENT FRAMEWORK: APPROACHES, OBJECTS, TOOLS AND METHODOLOGIES 2.1 Approaches The Impact Assessment Framework (and the whole CO-SUSTAIN project) is inspired by, and designed in line with approaches such as the Theory of Change (ToC) and Participatory System Mapping (PSM). Both approaches feed theoretically into this Impact Assessment Framework by conceptualizing CO-SUSTAIN project’s understanding and overall objective of encouraging novel forms of political participation for the sake of niche innovations that would provoke the radical yet necessary changes in the socio-technical system for a climate transition. Moreover, they are also the conceptual foundation of the deliberation panels. As such, these approaches support our understanding of the change processes based on which we carry out the panels (Wilkinson et al., 2021). In a nutshell, ToC aims for answering the question how the process of change takes place and PSM explains why – by making the processes, stakeholders and power relations explicit. 2.1.1 Theory of Change (ToC) ToC will provide an analysis on the strategies, actions, processes, conditions and resources that facilitate change and achieve outcomes. Its 'explanatory power' (Reinholz & Andrews, 2020) enables a thorough understanding of why particular activities or actions will lead to particular outcomes - and why others don’t. As a process of analysis and reflection it is used to develop a theory. It is a continuous action-learning cycle with phases that outline a program logic, which is systematically verified as new actors join, contextual changes happen, and learning occurs. However, it is also a mindset that encourages participatory and critical thinking about change. It explores implicit assumptions or uncertainties held by those involved, addresses diverse perspectives, and fosters shared understanding and common knowledge to guide action planning. Within the CO-SUSTAIN project, ToC is applied to identify preconditions and define innovative logical pathways to change in all different case studies. As such, we regard it as a process of analysis and reflection, guiding participants to identify change preconditions and levers, while also communicating how and why goals can be achieved. This critical and systemic thinking enhances democratic governance, mobilizing citizens to address climate issues. ToC helps participants understand contextual limits (e.g. context) and macro-theoretical boundaries (e.g. behavioural, technological), and once common ground is established, it analyzes problems and identifies long-term outcomes and the pathway to achieve them. In combination with Participatory System Mapping, ToC enables stakeholders to gain a deeper understanding of the complexities surrounding shortand long-term outcomes, as well as interim initiatives, projects, and programs. 2.1.2 Participatory System Mapping (PSM) PSM makes implicit assumptions and beliefs about causal links and loops within the case studies explicit and visible. To provide a thorough explanation of what can be observed and how it can be preserved or changed in the future, it is essential to understand not just what is seen, but also why it is happening. This is precisely where PSM’s role becomes visible: it demonstrates how events and patterns (the “what”) arise from systems’ structures and their deeper goals (the “why”). In social systems, those goals are defined by the underlying values and worldviews. Derived from the field of System Dynamics, PSM uses the so-called Causal Loop Diagrams (CLDs) - understood here as qualitative mapping or modeling tools that reveal the CO – SUSTAIN - Deliverable 1.3 9 / 36 feedback structure of a system - to map the dynamics within and related to the case studies in COSUSTAIN. CLDs, as outlined here, can be created by the researcher using literature and other empirical data, such as interviews, and/or through a participatory process (PSM) where various stakeholders contribute to the development of system maps. Conducting PSM enables the integration of diverse expertise, improving the understanding of existing system structures (Videira et al., 2021). Transition theories have been criticized for their limitations and their descriptive nature, which makes it difficult to identify general transition dynamics (Geels, 2022). (Participatory) System mapping can address this issue by providing a deeper layer of analysis. It helps uncover the underlying structures that drive specific dynamics at the regime and niche levels, focusing on the “why” behind the development of latent and manifest political participation and its influence. Moreover, all transitions involve a broad network of stakeholders, and a PSM process can integrate their expertise and perspectives to foster a shared understanding of various transition dynamics. By enabling the participants of the panels to examine the dynamics around the intended change, the feedback structures and power relations, system mapping can reveal interventions to preserve or change specific system characteristics. Focusing on the regime and niche levels during mapping enables the analysis of how top-down and bottom-up dynamics are connected. 2.2 Objects of assessment CO-SUSTAIN project’s Impact Assessment Framework will use a mixed-methods approach to assess the different aspects and parts of the whole deliberation process in 6 case studies. This section provides a brief description of the case studies and an overview of the deliberation process. 2.2.1 Case studies CO-SUSTAIN investigates conditions, dynamics and results of political participation through the analysis of 6 Case Studies from 5 countries, each belonging to one of the four categories of political participation by Ekman and Amnå (2012) (table 2 below) and which are further differentiated between individual / collective as well as latent/manifest forms of political participation. Below a brief description of all four forms of political participation is given. Involvement encompasses personal interest in politics and attentiveness to political issues, which are often also perceived through the sense of belonging to a group or collective which follows certain distinct political agenda (e.g. environmental protection) or lifestyle related politics (e.g. specific aspects of identity politics). Civic Engagement refers to all sorts of informal and formal social activities such as voluntary work and organisational membership that are based on personal interest and values around political and societal issues. In general, it aims at improving the social conditions of some specific groups or objects, be it local community, charity for others or sustainability oriented values. Formal Political participation strives for (an increased) electoral participation and contact activities and often implies membership in political parties, trade unions and other political organizations. Activism involves various extra-parliamentary forms of participation to make one’s voice heard (demonstrations, petitions, strikes, protests etc.). Activism can be both legal as well as illegal. CO – SUSTAIN - Deliverable 1.3 16 / 36 separately in a later phase. All panels are followed by a 5) post-panel data extraction (post-panel surveys and interviews). The figure demonstrates how the design of the deliberation process aims at implementing the assessment exercise throughout the deliberation process development from the preparatory phase to the completion of the third deliberation panels (see section 2 for details). It also displays a rough overview of the assessment methods consisting of both individual methods (grey circles with qualitative interviews and quantitative surveys) as well as the group-level methods (pink circles with gamification elements, serious games and impact stories). Figure 3. The linear logic of the Logframe (Logical Framework Approach). As mentioned above, this IAF focuses on assessing the output and outcome of the CO-SUSTAIN project’s interventions. As deliberation panels are project’s direct activities, the IAF will assess the quality of the deliberation panels as project’s output (green triangle i)) and their respective impact within the case studies as project’s outcome (green triangle ii)). The overall and long-term impact will be beyond the scope of this project to assess, however, within the case studies, the aim is to reflect upon potential long-term impacts based on the qualitative individual-level interviews and group-level impact stories. This logic is partially inspired by the Logframe Matrix, as seen above (Figure 3). In sum, the Impact Assessment Framework will assess two dimensions respectively, as briefly sketched below. Output dimension (for internal assessment within the case studies) Evaluation by a neutral expert selected by each case study team will follow through ● an expert survey (based on the “3-step-evaluation-cycle”, as demonstrated in table 6 in Appendix) ● a deliberation observation during the panels (based on the “observational protocol template”, see respective section in the appendix) The neutrality of the experts would be ensured by providing them with training and materials such as the OECD (2021a,b), and internal discussions sharing know-how and understanding of the value of neutrality. Outcome dimension CO – SUSTAIN - Deliverable 1.3 17 / 36 To fully assess the “outcomes” of our “outputs” (i.e., the deliberation panels and serious games), we will explore the prefigurative potential of these interventions, co-created with the case study groups. This will involve evaluating both - individual-level changes (through preand post-panel interviews and surveys) and - group-level changes (through impact stories which contextualize the deliberative processes and participatory observations of serious games). 2.3 Tools and Methodologies The following table (3) offers an overview of the goals and tools of the COSUSTAIN project’s Impact Assessment Framework with regard to the respective phase of the deliberation process. This table follows the logic of What-Why-How and describes all different goals and tools of the respective phase of the deliberation process and its specific part of the IAF. WHAT Phase of the Deliberation process WHY Goal HOW Tool 1 Preparatory Phase To guarantee that all potentially vulnerable groups have access to and feel comfortable being part of the deliberation panels Survey (or some other method designed by the case study teams during stakeholder analysis and panel preparation) to assess access of different groups to the deliberation panels with regard to availability in terms of timing, language requirements, technology and more. 2 Preliminary Phase – Gamification To foster early engagement and ensure shared understanding of the project’s aims and approach. A group feedback or some other method designed by the case study teams during stakeholder analysis and panel preparation to assess the level of understanding after the introduction of the COSUSTAIN project CO – SUSTAIN - Deliverable 1.3 18 / 36 3 Before Deliberation Panels 1 and 2 To assess the attitudes, understanding and perceived challenges towards novel forms of political participation before the deliberation process Appendix 8.1. Pre-panel individual survey questionnaire to all stakeholders of the panel Quantitative individual-level data extraction on the knowledge and openness before the panels 4 Before Deliberation Panels 1 and 2 To assess in-depth the attitudes, understanding and perceived challenges towards novel forms of political participation before the deliberation process Appendix 8.2. Pre-panel individual semi-structured interview to 2-3 stakeholders of the panel Qualitative individual-level data extraction on the understanding, attitudes and openness before the panels 5 Deliberation Panels 1, 2,3 Internal assessment of the deliberation panels’ quality to critically examine, compare and ensure the deliberative panel quality with regard to the panel design integrity and stakeholders’ experience Appendix 8.3. Expert survey questionnaire by a neutral expert observer to assess the quality of the deliberative process during the panels (OECD 3-step-evaluation-cycle) Deliberation Panels 1, 2,3 Internal assessment of the deliberation panels’ quality To reflect upon and ensure the deliberative panel quality with regard to the panel design integrity and stakeholders’ experience Appendix 8.4. Deliberation observation by a neutral expert with an observational protocol template 6 After Deliberation Panel 3 To assess the potential increase of (or: change in) the understanding, motivation and challenges towards novel forms of political participation Appendix 8.1. Post-panel individual survey questionnaire to all stakeholders of the panel Quantitative individual-level data extraction on the knowledge and openness after the panel 7 After Deliberation Panel 3 To assess in-depth the potential increase of (or: change in) the understanding, motivation and challenges towards novel forms of political participation Appendix 8.2. Post-panel individual semi-structured interview with 2-3 stakeholders of the panel Qualitative individual-level data extraction on the understanding, attitudes and openness after the panels 8 Deliberation Panel 3 To improve understanding of the complexity of democratic processes, to simulate real-world decision-making, test policy decisions, and evaluate long-term impacts Co-creation of solutions for climat change and roadmaps and collaborative decisionAppendix 8.5. Serious games through the PSM approach assessed through participatory observation and a survey which measure changes in perceptions and experiences after the game CO – SUSTAIN - Deliverable 1.3 19 / 36 making through the employment of game 9 Deliberation Panels 1,2,3 (Data collection throughout the panels, completed after Deliberation Panel 3) To understand the collective forms of change in community building capacity and practices, including social cohesion, empowerment and inclusion/exclusion; deliberative practices and dealing with conflicts and change; and capacity for/focus on sustainability transitions on different levels (from individual and group level to system-level). Impact stories provide qualitative insights into how initiatives evolve, engage communities, and overcome barriers. Appendix 8.6. Impact stories assessed through participatory observation by one of the case study members (based on the interviews, panels and participatory observation of the whole collective process), as well as aggregated interview data. Together, the different methods tracking change would provide an institutional ethnography of change that the deliberative process is contributing to or part of. Table 3. Overview of all the tools employed in the COSUSTAIN case studies’ deliberation processes. The CO-SUSTAIN project’s Impact Assessment Framework will assess both, 1) the outputs (interventions - deliberation panels) (via 5 in the table) as well as 2) their outcomes (the interventions’ impact onto the case studies) (via 3, 4, 6, 7, 8, 9 in the table) within the project. As seen above, the main focus is on assessing the impact of the deliberation panels onto and within the case studies, for which surveys and semi-structured interviews (preand post) as well as impact stories and participatory observation of the serious games play the crucial role. The Appendix includes the preliminary templates and methodology for the tools employed throughout the COSUSTAIN case study deliberation processes (see the overview in table 3. These preliminary versions represent a first release that will be validated, expanded and refined along with the case study design process which will be finalized by June 2025. While keeping the theoretical background, the approaches and the goals of the final version adopted in the fieldwork might be slightly different from the current ones. 3 CONCLUSION This Impact Assessment Framework (IAF) has set an aim to provide a preliminary yet thorough roadmap for the impact assessment of deliberation panels through a complex mixed-methods approach including qualitative and quantitative data on individual and collective-level changes. The general objective of the IAF is a comprehensive understanding of the impact of the interventions (deliberation panels) which are co-created within the case study groups. In particular, the IAF will assess changes in social cohesion, empowerment, inclusion/exclusion, as well as capacity and community building. CO – SUSTAIN - Deliverable 1.3 20 / 36 4 REFERENCES Chambers, R., & Pettit, J. (2013). Shifting Power to Make a Difference 1. In: “Inclusive Aid : Changing power and relationships in international development" by Leslie Groves and Rachel Hinton, (2004) Earthscan, London Curato, N., Sass, J., Ercan, S. A., & Niemeyer, S. (2022). Deliberative democracy in the age of serial crisis. International Political Science Review, 43(1), 55-66. Daré, W., Hassenforder, E., Dray, A. (2020). Observation manual for collective serious games. Montpellier: CIRAD-ComMod Association Demski, C., & Capstick, S. & Hessellund Beanland, M. (2024). Impact evaluation framework for climate assemblies. KNOCA (Knowledge Network on Climate Assemblies), December 2024. https://cdn.prod.websitefiles.com/65b77644e6021e9021de8916/675ff37b91b3f53c964415ad_Updated_IEF_Dec_2024.pdf Dryzek, J. S. (2002). Deliberative democracy and beyond: Liberals, critics, contestations. OUP Oxford. Ebrahim, Alnoor & Rangan, V. (2014). What Impact? A Framework for Measuring the Scale and Scope of Social Performance. California Management Review. 56. 118-141. 10.1525/cmr.2014.56.3.118. Ekman, Joakim & Amnå, Erik (2012). Political participation and civic engagement: Towards a new typology. Human Affairs 22 (3):283-300 Gattenhof, S., Hancox, D., Klaebe, H., Mackay, S. (2021). Nothing About Us Without Us: Co-creation with Communities for Impact Assessment. In: The Social Impact of Creative Arts in Australian Communities. Springer, Singapore, pp. 45-60. Geels, F. W. (2022). Causality and explanation in socio-technical transitions research: Mobilising epistemological insights from the wider social sciences. Research policy, 51(6), 104537. Marquardt, Jens & Delina, Laurence. (2019). Reimagining energy futures: Contributions from community sustainable energy transitions in Thailand and the Philippines. Energy Research & Social Science. 49. 91102. 10.1016/j.erss.2018.10.028. CO – SUSTAIN - Deliverable 1.3 21 / 36 Mosse, D. (2005). Cultivating development: An ethnography of aid policy and practice. Pluto Press. Olivier de Sardan, J.-P. (2005). Anthropology and development: Understanding contemporary social change. Zed Books. OECD. (2020). Innovative citizen participation and new democratic institutions: Catching the deliberative wave. Paris: OECD Publishing. DOI: https://doi. org/10.1787/339306da-en OECD (2021a) (Organisation for Economic Co-operation and Development), Evaluation Guidelines for Representative Deliberative Processes, OECD Publishing, Paris, https://doi.org/10.1787/10ccbfcb-en. https://www.oecd.org/content/dam/oecd/en/publications/reports/2021/11/evaluation-guidelines-forrepresentative-deliberative-processes_10b0cea1/10ccbfcb-en.pdf OECD (2021b) (Organisation for Economic Co-operation and Development), Eight ways to institutionalise deliberative democracy. OECD Publishing. https://www.oecd.org/content/dam/oecd/en/publications/reports/2021/12/eight-ways-toinstitutionalise-deliberative-democracy_e1f898a0/4fcf1da5-en.pdf Papadopoulos, Y., & Warin, P. (2007). Are innovative, participatory and deliberative procedures in policy making democratic and effective?. European journal of political research, 46(4), 445-472. Reinholz, D. L., & Andrews, T. C. (2020). Change theory and theory of change: what’s the difference anyway?. International Journal of STEM Education, 7, 1-12. Smith, D. E. (2005). Institutional ethnography: A sociology for people. AltaMira Press. Tourais, P., & Videira, N. (2021). A participatory systems mapping approach for sustainability transitions: Insights from an experience in the tourism sector in Portugal. Environmental Innovation and Societal Transitions, 38, 153-168. Wilkinson, H., Hills, D., Penn, A., & Barbrook-Johnson, P. (2021). Building a system-based theory of change using participatory systems mapping. Evaluation, 27(1), 80-101. CO – SUSTAIN - Deliverable 1.3 22 / 36 5 ANNEXES 5.1 Pre/Post-panel individual survey questionnaire preliminary version (to be finalised in parallel with emerging Case Study data) 25% socio-demographic 25% CO-SUSTAIN-relevant 25% case-study relevant 25% panel/stakeholders/CS-relevant PRE-PANEL SURVEY Question Format SOCIODEMOGRAPHIC 1 Age 2 Gender 3 Education 4 Employment 5 Income Ethnicity Political affiliation Religion CO – SUSTAIN - Deliverable 1.3 23 / 36 CO-SUSTAIN relevant questions 6 6a) How important do you consider climate change amongst issues of our time? 6b) How possible do you consider sustainable transition to be? a) Aa distraction from important issues b) Not particularly important c) Neither important nor unimportant d) Quite important e) Most important question of our time f) Not sure a) Entirely possible b) Somewhat possible c) Neither very likely nor impossible d) Somewhat doubtful e) Completely impossible f) Not sure 7 7a) Do you think the type of activities here (deliberative panels etc) are useful for democratic transition towards sustainability? 7b) Which form of political participation do you consider effective for achieving sustainable transition to a greener society? (please list in order of effectiveness in your opinion) Scale 1 (I am doubtful) - 5 (definitely useful) (List: Democratic national elections Democratic local elections Petitions Donations by individuals Demonstrations and marches People’s/citizen’s assemblies Strikes Non-violent civil disobedience and protests Protests and action against property Other, please write…. ) 8 Personal forms of political participation: Which forms of political participation are you yourself involved personally? List: CO – SUSTAIN - Deliverable 1.3 24 / 36 Same list as above 9 How do you assess your knowledge with novel political participation forms? Scale: 1 – No knowledge / interest to 5 – Experienced 10 Personal experience with participatory processes: In which contexts do you find yourself in participatory, collaborative, cocreation, deliberative processes? Describe the most recent or the most vivid example. Descriptive What are the obstacles to participating in political life for you? What are the obstacles to participating for others? Who do you think are the worst affected by such obstacles? - CASE STUDY QUESTIONS 1-5 11 Perceived Relevance of the Initiative: How important do you consider the [case study] for sustainable transition? Scale: 1 (The case study has no potential to bring about or contribute to change) to 5 (The case study has contributed substantially to changes towards sustainable transition) 12 Please describe the decision making process in your initiative.. What, if anything, would you do differently if you could? Open answer 13 How important do you consider involving other groups in the activities of your initiative and why? Open answer 14 Has the project facilitated new collaborations or strengthened existing relationships within the initiative (e.g., between residents, policymakers, businesses, and researchers or others)? Open answer 15 Describe the tools and methods you have used or plan to use to achieve your long term vision. Why do you consider these valuable? Open question PANEL/stakeholders group questions 1-5 CO – SUSTAIN - Deliverable 1.3 25 / 36 16 Describe how you have contributed to the initiative. Is there anything more you would like to do? Open answer 17 Please describe the decision making process that your institution/…/ usually utilises.. What, if anything, would you do differently if you could? Open answer 18 What do you think of the current local or regional policy-makers approaches to this CAI and its goals? What, if anything, would you change in these? Open answer 19 What do you think about the current way of working with different stakeholders related to this [case study]? What, if anything, would you change about this approach? Open answer 20 Do you use or plan to use any participatory methods in your work? If yes, what and how? If not, why? Open answer Table 4. Overview of the survey questionnaire questions and their composition with regard to their specific topic (in four colours). 5.2 Pre/Post-panel individual semi-structured interview guide Dear Mrs/Mr. Firstly we would like to express our appreciation and gratitude for taking time both for this interview as well as the subsequent deliberation panel today. Thank you so much for joining us today and for taking the time to be a part of this deliberation panel. Your insights and opinions are truly appreciated, and we’re excited to hear your thoughts. Therefore, we have some more specific questions aiming to go in-depth with regard to your experiences and opinions with regard to novel forms of political participation (such as citizen assemblies and …) and citizen empowerment. We would like to record this interview for later transcription. Is this okay for you? This data will be only used anonymously for later analysis within the CO-SUSTAIN project. Question Probing Sub-questions - There are different ways to be (socially and) politically active. How would you describe your role as a citizen? - Have you ever been active in any of these ways? If so, can you give some concrete examples across the years? CO – SUSTAIN - Deliverable 1.3 32 / 36 Game rounds Engagement with system dynamics How effectively did participants engage with the system mapping and Theory of Change aspects? Were they considering the interconnections between different components (e.g., energy, land use, policies)? Very effectively / Somewhat effectively / Not effectively Decision-making and system Impact Did participants make decisions considering the long-term systemic effects (e.g., energy transition, social equity, environmental sustainability)? Yes / No / Somewhat Incorporation of ToC concepts How well did participants incorporate ToC concepts into their decision-making process? Did they reference long-term outcomes or steps needed to achieve them? Very well / Somewhat / Not well Role of system mapping Did participants use the SM to understand trade-offs and see the interconnections between their decisions and different sectors? Yes / No / Somewhat Collaborative dynamics in system understanding How did participants collaborate to improve their understanding of the system dynamics (e.g., discussing feedback loops, actor relationships, etc.)? Strong collaboration / Moderate collaboration / Poor collaboration Debriefing phase Reflection on systemic thinking Did the group reflect on the systemic dynamics discussed in the game? Did they recognize how their decisions impacted different components of the Yes / Somewhat / No CO – SUSTAIN - Deliverable 1.3 33 / 36 system (e.g., land, energy, social sectors)? Discussion on Theory of Change Did the group discuss how their decisions aligned with or deviated from the Theory of Change objectives (e.g., achieving long-term sustainability, social equity, etc.)? Yes, fully discussed / Somewhat discussed / Not discussed Identifying areas for improvement Did participants identify areas where policy strategies could be improved based on their understanding of the system dynamics and ToC? Yes, fully identified / Somewhat identified / Not identified Co-creation of solutions Did participants identify cocreated solutions based on their collective understanding of the system map and ToC dynamics? Yes, new solutions proposed / Some solutions proposed / No solutions proposed Balancing climate emergency and social/economic considerations How did participants navigate the trade-offs between the climate imperative (e.g., emissions reduction, sustainability) and social/economic considerations (e.g., job creation, economic growth)? Strong balance achieved / Moderate balance / Focused more on climate / Focused more on social/economic aspects CO – SUSTAIN - Deliverable 1.3 34 / 36 Democratic decision-making and local adaptation Did participants consider the local context and community needs (e.g., local climate risks, social equity) when making decisions in the game? How did they ensure democratic decision-making while addressing the climate imperative? Fully considered local context and democratic values / Somewhat considered local context, less focus on democratic process / Did not prioritize local context or democratic values 5.6 Impact stories Impact stories are in essence collections of extended case materials that enable narrativizing change across the timeline of the initiative from its inception to the period after the Deliberation Panels, aiming to assess shifts in (1) community building capacity and practices, including social cohesion, empowerment and inclusion/exclusion; (2) deliberative practices and dealing with conflicts and change; and (3) capacity for/focus on sustainability transitions on different levels (from individual and group level to system-level). Impact stories provide qualitative insights into how initiatives evolve, engage communities, and overcome barriers. This approach aligns with institutional ethnography (Smith, 2005), which examines how institutional structures shape social practices and experiences. Here, this approach offers a lens to analyze how deliberative panels interact with broader policy and governance systems as well as the CAIs internal discourses and practices. Furthermore, the following protocol draws from the anthropology of development, particularly how institutional logics shape local interventions (Mosse, 2005; Olivier de Sardan, 2005). Such angle directs the researcher to trace changes shaped by features of niches as institutions, or by features of the regime as the institutional setting within which the CAI has to operate, but also the interactions within and between niche-level institutions (e.g grassroots climate initiatives) and regime-level institutional structures (such as governance frameworks, funding mechanisms, or dominant policy discourses). The case materials comprise the data collected before, during and after the Deliberation Panels (field notes, interviews, media, etc). If it is feasible to include additional data gathering, focus group discussions and/or reflections from participants after the panels would be the most appropriate for complementing data on group level impact. Whilst analysis of post-panel interviews would be the central starting point for developing the group level impact stories, f the materials collected on the initiative (A) before the panel (as part of the CS research), which would form a vital background material for the impact stories, the following could warrant particular attention – however, these are for guidance and inspiration only and would depend on the particular cases. Please keep in mind the topics above where shifts would be assessed, and adjust the following as needed: CO – SUSTAIN - Deliverable 1.3 35 / 36 (1) The motivations for the CAI (especially with regard to community-building and empowerment – did the initiators aim for creating communities and empower these vis-à-vis the regime level?); what were the challenges in terms of practical obstacles, especially in terms of access to power, but also of relations on different levels (internal, with regime, etc)? What kinds of attitudes were evident towards inclusiveness or what evidence of exclusions were observable, and to what degree was this an institutional feature as opposed to personal preference? What institutional constraints or supports shaped the CAI’s trajectory? What do the CAI representatives consider its successes/failures and why? (2) Any evidence of events/situations/experiences of conflict, and how these were solved and/or did unsolved issue(s) remain a part of the CAI? What kinds of patterns of decision making characterised the CAI? Were these issues (conflict, solutions, decision making) addressed with awareness (institutional) or appeared mostly random? Did conflict resolution strategies reflect institutional norms, alternative approaches to governance, novel thinking from CAI or smth else? (3) What kinds of engagement with versions of theories of, or thinking about change, as well as understanding systems could be observed in CAI interactions within the group, or when engaging with the surrounding society? Which positions do climate/sustainability have in this thinking/acting regarding change/systems? What kind of involvement in climate and sustainability topics could be observed? On what level (individual, niche, regime/societal, landscape?) is transition envisioned, discussed, practiced, pushed for? Are there actual experiences of success regarding this? Was there an institutionalized understanding of sustainability, or were interpretations diverse and contested? Please keep in mind and point out where these features were individual/random/unplanned versus institutional/systemic/planned. During the panels (B), data would be collected from observations made during the serious games, but also during the rest of the intervention, from the preliminary meetings to the end of the panel run. The following would be worth paying particular attention to )in addition to or as part of the other deliberation panel observations): (1) How were inclusivity, empowerment, and institutional access negotiated during different phases of the panels? How do the members of the CAI present in the context of the period of deliberation panel (from preparations to the end) – as a uniform group, as a group of equal/unequal individuals etc? Are any power relations evident? What efforts are observable in inclusion (or exclusion) of others, who/which groups in particular? What other group cohesion and power-related observations can be made during the panel process? Are there changes during the panel period (from start to finish), or notable moments of rapid change (“aha” moments in relation to inclusiveness, empowerment, group cohesion)? (2) How do the CAI representatives relate to the deliberative practices/systems thinking and conflict? What do they struggle with, what new approaches might they bring to the thinking/solutions as individuals and/or as a group? Are institutionalized ways of problem-solving (which ones?) reinforced, resisted, or transformed? CO – SUSTAIN - Deliverable 1.3 36 / 36 (3) How do the CAI representatives engage with theories of change and systems thinking in particular and how do they relate these approaches to their own initiative? How do they link their initiative and the activities during the deliberation panels to climate/sustainability approaches? What aspects of their initiative as an institution are revealed in this process? Do they adapt their language or strategies in response to institutional expectations/panel process? Finally, data from the period (C) after the panels would primarily come from collated individual interviews. Potentially, focus group discussions with CAI members could be added. The following questions would offer data on the presence of shifts, and by extension, impacts: (1) Has the CAI’s understanding of community, inclusiveness, or power relations changed and if so, how? Has the sense of empowerment changed and if so why/how? Are institutional barriers now seen differently? Is there a change in how challenges are tackled? Has the CAI’s approach to engaging with regime-level institutions evolved? Do the members attribute any of those changes/sense of change to the interventions? Has the sense of what is success/failure for CAI changed, how and why? (2) Has there been a change and/or sense of change in how the CAI approaches/discusses/handles conflict? Do the members sense a change in patterns of decision making characterising the CAI and is there any evidence of it? Do members attribute any changes to their experience with deliberation panels? (3) Have the CAI representatives applied theories of change/systems thinking in their activities/practices since the panels? In what realms (within CAI, when engaging with similar groups, when engaging with the surrounding society or the regime level)? Have new governance strategies or networks emerged? Has there been a change/perception of change in the role of climate/sustainability in thinking about change and the systems of power, and in this being part of the discourse or actions of the CAI? Has there been a change regarding the envisioning, discussion, practicing or aiming for transition, and on what level (individual, niche, regime/societal, landscape?)? Has there been a change in terms of actual experiences of success regarding this, or is there a sense of failure in relation to the CAIs ability to action change? Is there evidence of institutional adaptation on the niche or regime level (e.g., practices, new policies, partnerships, funding shifts, interventions), and are those attributed to the interventions in any way by any of the participants? (D) Follow-up Once the group level impact stories have been teased out of the material above, it is advisable to check with the participants whether they agree, would like to add detail or change anything. Whilst on the one hand, this is a further co-creative layer on this research (Gattenhof et al 2022), it would likely also add further material, as contestations and disagreements with the group level version would both correct any misunderstandings as well as reveal what the group considers important or challenging. Discussions on contestations or disagreements over institutional interpretation would be particularly revealing in understanding how CAIs navigate institutional discourse and resist or adopt dominant logics.