Sowing & harvesting, participatory evaluation handbook

Sowing Harvesting & Participatory Evaluation Handbook Esteban Tapella - Pablo Rodríguez Bilella - Juan Carlos Sanz Jorge Chavez-Tafur - Julia Espinosa Fajardo Esteban Tapella - Pablo Rodríguez Bilella - Juan Carlos Sanz Jorge Chavez-Tafur - Julia Espinosa Fajardo Sowing Harvesting & Participatory Evaluation Handbook SOWING & HARVESTING. PARTICIPATORY EVALUATION HANDBOOK Authors: Esteban Tapella, Pablo Rodríguez Bilella, Juan Carlos Sanz, Jorge Chavez-Tafur, Julia Espinosa Fajardo Graphic design: María Clara Graffigna Illustrations and book trailer: Ana Clara Bustelo Translation into English: Amy Bell This handbook is an outcome of the project, “Strengthening Participatory Evaluation theory and practice in Latin America and the Caribbean: generating a community of practice and learning”, developed by the Social and Environmental Labour Studies Programme (PETAS) at the Social and Economic Research Institute, Faculty of Social Sciences, National University of San Juan (Argentina) together with the German Institute for Development Evaluation’s (DEval) Focelac Evaluation Capacity Development and Networking project in Latin America. Tapella, Esteban Sowing & Harvesting. Participatory Evaluation Handbook / Esteban Tapella; Pablo Rodríguez Bilella; Juan Carlos Sanz; Jorge Chavez-Tafur; Julia Espinosa Fajardo. - 1st ed. – Bonn, Germany: DEval, 2021. 224p.; 21 x 23.7 cm. Includes multimedia hyperlinks, contents and bibliographic references. ISBN 978-3-96126-143-7 1. Evaluation. 2. Social participation. I. Rodríguez Bilella, Pablo D. II. Sanz, Juan Carlos. III. Chavez-Tafur, Jorge. IV. Espinosa Fajardo, Julia. V. Title. This publication received support from: With funding support from: © Esteban Tapella / Pablo Rodríguez Bilella / Juan Carlos Sanz / Jorge Chavez-Tafur / Julia Espinosa Fajardo © German Institute for Development Evaluation (DEval). Fritz-Schäffer Strasse 26, 53113 Bonn, Germany. Tel: +49 (0) 228 33 69 07-0. Email: inf[email protected] www.DEval.org ISBN 978-3-96126-143-7 (PDF/electronic) The reproduction of part or all of the contents is permitted, provided the source is properly cited. Tell me and I forget, teach me and I may remember, involve me and I learn. Benjamin Franklin 7 6 By the end of 2018, the current EvalParticipativa coordinator team members had facilitated various participatory evaluation processes with multiple public and private organisations in several Latin American countries. We had also developed many different capacity training sessions; published on the topic in academic journals; and supported young evaluators, in particular, with methodologies and tools for conducting participatory evaluations. We were pleased with what we had achieved but we also had the feeling that there was much more for us to discover. And so, in early 2019, we launched Eval- Participativa, a community of practice and learning for participatory evaluation for Latin America and the Caribbean. Energised by joint efforts and collaboration, the initiative brought together two entities: the Social and Environmental Labour Studies Programme (PETAS) at the National University of San Juan (Argentina), and Focelac+, the evaluation capacity development and networking project run by the German Institute for Development Evaluation (DEval) in Latin America. Confident in the potential of peer collaboration as a mechanism to deepen knowledge and experience, we set out with the key objective to reinforce the inclusive involvement of civil society in Foreword evaluation processes. To do this, we decided to create a space for both virtual and face-to-face interaction so that those interested in participatory evaluation could reflect on and improve their practice, support each other’s efforts, and create products together (methodologies and tools) to help professionalise this type of evaluation practice. Two years have passed since EvalParticipativa opened its doors and we believe it is a living testimony to the growing interest in, and development of, participatory evaluation experiences in the region. It also provides ample opportunity to make connections, value the knowledge that we have gathered and keep learning. Today, various like-minded organisations have joined us and there has been a continual increase in the number of people that participate in our forum, follow our posts and interact with topics on our social networks. We finished this book at the end of 2020, a year that nobody would have wished for. A global health crisis never before experienced with economic and social impacts that are still difficult to estimate. This past 2020 will be remembered for the collective efforts invested to find ways and reasons to press on, isolating ourselves to protect our elders, multiplying our contributions to solidarity, finding new life-work balances, and making 9 8 ourselves available and present even when we could not do so physically. At the same time, COVID-19 has strengthened us by forcing us to recover a sense of community. Despite the circumstances, EvalParticipativa, a space for practice and learning, has consolidated itself, and even outperformed its most optimistic expectations. This handbook is just one of the outcomes of these collaborative efforts. Illustrated with a multimedia format, we have attempted to synthesise the thoughts, feelings and collective learning gathered from a large and diverse group of colleagues that work in the field. Special recognition goes to the colleagues who participated in the First Latin American and Caribbean Gathering of Participatory Evaluation Experiences, held in Qui- to (Ecuador) at the end of 2019. Their valuable contributions enriched our reflections and some of their thoughts have been recorded in the testimonial videos that accompany this handbook. They are: Andrés Nicolás Peregalli Magnus Kossman Dagny Karin Skarwan Matthias Edouart Casasco Ana Tumi Guzmán David Olmos Joaquín Navas Olga Niremberg Ericka Vanessa Valerio Mena María Eugenia Brisson Marcia Itzel Checa Gutiérrez Karla María Salazar Sánchez María Virginia Vintimilla Suárez Erika Abril Torres Vargas Morella Miraballes Baz Valerie Fragnaud Mallorquín Fernanda Massiel Arriaza López Carmen Luz Sánchez Bretón Daniela Beatriz Miranda Prado In the process of developing the content of this handbook, we have appreciated extremely valuable collaboration with Jorge Chavez-Tafur and Julia Espinosa Fajardo, and we would like to thank them once again for their contributions and enthusiasm. We trust that this handbook will be a useful tool that will help facilitate high quality participatory evaluations in the region. Esteban Tapella Pablo Rodríguez Bilella Juan Carlos Sanz EvalParticipativa Coordinator Team CHAPTER 1 Contents The meaning and purpose of participatory evaluation today… and this handbook. -page 14- -page 36- CHAPTER 2 Participatory evaluation. The "what" and "who" -page 116- CHAPTER 4 How to facilitate participatory evaluation. -page 156- CHAPTER 5 Tools for participatory evaluation. Bibliography Authors -page 76- CHAPTER 3 How to conduct a participatory evaluation. -page 202- -page 212- Supplementary information to the main text, such as a complementary bibliography or practice-based suggestion. The chapter colours help navigate the reader around the handbook. They are used in the page numbers, highlighted phrases and spotlights for each section. Next to the main text in the handbook, you will find the following: CHAPTER COLOURS CHAPTER 1 CHAPTER 2 CHAPTER 3 CHAPTER 4 CHAPTER 5 LINK TO TESTIMONIAL VIDEO BASED ON PARTICIPATORY EVALUATION EXPERIENCES IN LATIN AMERICA AND THE CARIBBEAN. Examples to illustrate the core concepts using experiences and case studies. SPOTLIGHTS How to use this handbook 14 Chapter 1 The meaning and purpose of participatory evaluation today… and this handbook 29 28 tin American scene. Moving on from this, we advance in participatory evaluation’s conceptual delimitation and present its key principles, enabling us to reflect on its potential to promote rights, inclusion and equity. In chapter three, we will go on to describe, step by step, the participatory evaluation process. We will discuss general aspects that characterise how this kind of evaluation is carried out as well as specific elements that feature in each of the specific steps. We also highlight some tools and techniques that help ensure that a participatory evaluation is conducted in line with the quality standards and methodological rigour expected in any evaluation. Chapter four presents and analyses the role of the person who facilitates a participatory evaluation. Here, we begin with the idea that the one who facilitates this type of evaluation assumes a different role to those involved in conventional evaluation processes. In the chapter we present a general overview of the notion of facilitation and then take time to specifically consider what this means in It is within this framework, and with the contributions from EvalParticipativa community members, that we present this handbook for participatory evaluation. This handbook aims to: • develop basic conceptual and methodological considerations concerning participatory evaluation that emphasize the “what” and “who” of the process, the role of facilitation, and the choice and use of appropriate tools for this approach, and • review the theory and practice of participatory evaluation in Latin America in order to contribute ideas/tips/recommendations on how to conduct a high quality participatory evaluation. In the same vein, in chapter two, we explore the development of the relationship between evaluation and participation. To do this, we present varied proposals that have been created all around the world. We pay special attention to connections between this style of practice and other participatory initiatives that have deep roots in the La- 31 30 4. HOW TO READ THIS HANDBOOK terms of a participatory evaluation: the guide in the different stages and phases of the process, the relevance of contributing to the development of specific capacities and the need to monitor the level of progress and show results. Finally, chapter five focuses on tools for participatory evaluation. We present some key conceptual considerations on methodology and tools as well as the advantages and some limitations to the latter. We will then proceed to classify the tools in terms of their methods and the different purposes they can be used for. Through examples and practical recommendations, we propose a list of seven criteria that should be taken into account when selecting and using these instruments. EvalParticipativa is a community of practice and learning, focused on learning from, and enhancing the scope of, successful participatory evaluation experiences in the region of Latin America and the Caribbean. As we stated in this chapter’s introduction, this handbook has its origins in the First Gathering of Participatory Evaluation Experiences in Latin America and the Caribbean, held in Quito (Ecuador) towards the end of 2019. During five intense days together with around twenty participants, we shared lessons learned and new ideas that emerged from our respective practices. The structure of the gathering has been replicated to a certain extent in this handbook, with the content highlighted in the previous section. We believe that there is not only one way to access and make use of this handbook’s content. A linear reading through the chapters will take you on a journey that starts with a general overview of participatory evaluation’s roots, links and principles (Chapter 2). From there, you will go on to discover the process used to develop this type of evaluation and its particularities (Chapter 3), reflect on the key role played by the facilitator in the participatory evaluation (Chapter 4) and finally learn about different participatory tools and how they should be used in the context of this approach (Chapter 5). It is also possible to read the chapters individually, in line with the reader’s particular aims and interests. Each chapter follows on from, and is connected to, the others; yet at the same time, they are complete in themselves and can be read separately. Each chapter, in addition to its main content, includes dialogue boxes with practical examples, bibliographic suggestions to deepen understanding, and links to tools and experiences. Also, when it is merited, we have included links to testimonies from members of the EvalParticipativa community of practice, via short audiovisuals that reproduce concepts or highlight personal or organisational experiences. An additional note on the use of non-se- xist language in this handbook. Language is formed in a conflictive space and arena where tensions are expressed between institutional regulations and processes of change and transformation. In editing this handbook, we have sought to use language that helps us name the reality as it is and therefore we make use of linguistic resources available in order to not discriminate against anyone. Therefore, we make it a priority to be clear and inclusive in our communication without clashing with grammatical rules. We use generic, collective or impersonal terms and avoid gender-exclusive pronouns. There is not only one way to access and make use of this handbook’s content. Both a linear reading through the chapters and a focus on a single chapter are valid in line with specific purposes and interests. 5. OUR EXPECTATIONS During the process of developing this handbook, the COVID-19 pandemic unleashed an unprecedented global emergency that seriously compromised the 2030 Agenda. Evaluation should also respond to this enormous challenge and make itself available to development stakeholders as a vital instrument for 33 32 As with other approaches, participatory evaluation has the key function of providing evidence to guide decisions on project continuity and improvement. good governance. As with other evaluation approaches, participatory evaluation has the key function of providing evidence to guide decisions on programme continuity and improvement. Further value is added by the way participatory evaluation is adapted to the 2030 Agenda principles and fosters cohesion among stakeholder groups involved in development processes. The EvalParticipativa community members are aware of the seriousness of the situation and we believe that a significant contribution can be made from presenting this evaluation approach and making it available to others. We consider that EvalParticipativa has the potential to: • build capacity in these areas, exchange experiences and document good practice that allows lessons to be shared and new initiatives to be fostered; • develop skills in the public sector, civil society organisations and the community of participatory evaluation practitioners; • support and work alongside projects, programmes and interventions that have a participatory approach in their methodology to any extent or type; • construct and share theoretical and practical knowledge based on experiences developed in different countries, by placing value on the wide range of existing material and generating new material; • strengthen alliances both within and between regions, institutions and organisations including academia and evaluation training programmes; and • promote the creation of conditions that are necessary so that participatory evaluation processes become regular practice in the region. Some of these conditions include raising awareness among the main stakeholders, encouraging participants, and demonstrating its benefits and the need to institutionalise this practice. We trust that this handbook will enable us to analyse these ideas, refashion them and move forward to consolidate a new evaluation paradigm with social participation at its core. 34 Chapter 2 Participatory evaluation. The "what" and "who" 39 In this chapter, we explore the origins, developments, principles and groups involved in participatory evaluation. In the first section, we describe this evaluation approach’s roots and connections. We start by examining evaluation’s early relationship with participation, before looking at distinct versions developed in different places around the world and highlighting the connections between this style of practice and other participatory initiatives deeply rooted in Latin America. After situating participatory evaluation within this broad overview, we move on, in the second section of the chapter, to define its conceptual boundaries and key principles. In the third section, we reflect on the wide range of interpretations and practices linked to the concept of “participation”, and the potential of participatory evaluation to promote rights, inclusion and equity. In the conclusion, we reiterate the chapter’s central aspects and frame participatory evaluation as an instrument that helps ensure that nobody is left behind. 1. PARTICIPATORY EVALUATION ROOTS AND CONNECTIONS The ever-increasing presence of participatory evaluations, together with their respective reflections, training sessions and conceptualisations, could lead one to think that a new phenomenon, if not a fad, had emerged. Yet, the connection between evaluation and participation has a long history rooted in different contexts and scenarios, whether explicitly in the field of evaluation or more implicitly when a participatory emphasis is integrated into an initiative that is in some way related to the field. 41 40 discuss, the term “participation” itself has been stretched somewhat and participatory evaluation practices vary greatly. Various ways evaluation and participation have been linked Since the start of the 1990s, several evaluation approaches have recognised the importance of involving a plurality of stakeholders in the evaluation process. In fact, the inclusion of voices from civil society is now considered a central element, not only in order to respond to evaluation needs, but also in order to achieve high quality, credible, viable and evidence-ba- sed evaluations (Tapella and Sanz 2019). In contrast to conventional evaluation (which puts an emphasis on the need to use scientific methods, controlled experiments and modelisation), these approaches place the social actors at the centre, view evaluation as a process of pluralist negotiation (Weiss, 1998; Monnier, 1995) and place the focus on how the outcomes will be used. More specifically, they look at the context and advocate that the com- Evaluation and participation: a long-term relationship The relationship between evaluation and participation started to develop in the 1970s, in response to criticism against classic evaluation models, characterised by rational technical approaches, with positivist and quantitative natures, and clear top-down perspectives (Cousins, Whitmore, and Shulha, 2013; Plottu and Plottu, 2011; Guba and Lincoln, 1989; Everitt and Hardiker, 1996). Until that point, evaluation was considered to be purely technical in nature, supported by measurement tools and the external eye of an expert evaluation team. It was thought to be able to extract objective conclusions from programme and project performance and results. However, new contributions in the field of evaluation recognise the political nature of evaluation processes as well as the centrality of reflexivity and intersubjectivity (Chelimsky, 1998). Influenced by interpretivism and constructivism stemming from social sciences, evaluation understands the importance of including qualitative assessments to better understand the complexity of social programmes and social change processes. It therefore questions whether evaluation is free from value judgements and recognises the need to negotiate between the different values represented. Thus, for the first time, the evaluation discipline considers that the diversity of social actors involved, and their different perspectives and priorities, are a fundamental element to consider in public policy programme inquiry and assessment processes. However, it was not until halfway through the 1990s, within an international framework focused on promoting “human development”, that we observed a flourish of reflection and theoretical and methodological debate on participation and evaluation. Since the mid-1990s there have been numerous articles, books and handbooks discussing, albeit under different names, evaluation practices that involve key stakeholders in a dynamic and ongoing manner (King, Cousins, and Whitmore, 2007). Many of these versions of evaluation were generated within the scope of development cooperation and have gained in popularity as awareness has grown concerning the complexity of the evaluation context, programmes, policies, and the organisations themselves (Cousins and Chouinard, 2012). Since that time, there have been many evaluation approaches that have incorporated the involvement of stakeholders as a relevant and key component, displaying a particular sensitivity to their needs and perspectives, as well as a clear interest in getting them involved in the evaluation process (Chouinard and Milley, 2018). However, as we will go on to Since the mid-1990s, a wide range of evaluation approaches have incorporated the involvement of stakeholders as a relevant and key component. YOU CAN WATCH THE VIDEO "PARTICIPATORY EVALUATION GLOBAL ROOTS AND CONNECTIONS" HERE 43 42 munity involved should be more implicated and the evaluation team play a more active and differential role (Patton 2018, 2010; Shulha et al., 2016). There are other versions that coincide in a significant part of participatory evaluation's general focus. However, they differ with regard to the purpose of the evaluation, the type of social actor involved and the role of the evaluation team. We will now introduce four of the approaches that we consider particularly significant: transformative evaluation, empowerment evaluation, the collaborative approach in evaluation, and developmental evaluation. Transformative evaluation Transformative evaluation emphasizes that this process is not only technical in nature, but also highly political. Its aim is to recognise situations of discrimination and social exclusion (linked to different forms of diversity) and transform them as an act of social justice. For Mertens (2009), an important aspect of the transformative paradigm is the conscious inclusion of a wide range of people that are usually excluded in society. In line with this approach, knowledge (including that generated in evaluations) is influenced by the different interests and power relationships at play. In this sense, the evaluation process enables the least powerful voices to be included, builds capacity in the participants, and helps societies become more equal and fair. In order that the evaluation process itself implies transformation (Nirenberg, 2013), the evaluation should constitute an opportunity to rethink programmes and drive actions for change. In the same way, the evaluation team should not be considered simply methodologists, analysts or facilitators, but also, and most importantly, as agents of TO FIND OUT MORE ABOUT TRANSFORMATIVE EVALUATION Mertens, D. M. 2009. Transformative research and evaluation: the intersection of applied social research and program evaluation. New York: Guilford Press. Nirenberg, O. 2013. Formulación y evaluación de intervenciones sociales. Buenos Aires: Noveduc. 45 44 In this type of evaluation, the evaluation team works alongside the different key stakeholders (organisation staff, programme participants and the general community) to develop the evaluation and becomes a “critical friend” or “coach” that helps develop a rigorous and appropriate process (Fetterman 1994; Fetterman et al., 2014). When this approach is used, there is a distinction between practical empowerment evaluation and transformative empowerment evaluation. The first more closely resembles formative evaluation and aims to improve the programme with participation from the different stakeholders. The second encourages people to take control of their own lives and available resources in order to change predefined roles and structures (Fetterman, 2015). Collaborative Approach in Evaluation The Collaborative Approach in Evaluation (CAE) aims to foster learning and impro- ve programmes and policies run by organisations and institutions. It focuses on TO FIND OUT MORE ABOUT EMPOWERMENT EVALUATION Go to the Better Evaluation page: Empowerment Evaluation Developmental Evaluation Developmental Evaluation focuses on informing and supporting one or several agents of change that implement innovative approaches in complex situations (Wilson-Grau and Britt, 2013). The evaluation process begins in the usual way of forming evaluation questions and, as in all evaluations, gathers and analyses information throughout the innovation process. The approach supports the development of innovative interventions with the aim of guiding their adaptation to emerging and dynamic realities in complex situations (Patton 2010, 2012). Inchange that take on an activist role that seeks to advance toward greater social justice (Ligero Lasa et al., 2014). Empowerment Evaluation Empowerment Evaluation has the end goal of promoting self-determination among groups as a way of contributing to improving programmes and making them more likely to achieve their results. Self-determination is linked to mechanisms and behaviours that help people take control over their lives and the resources in their area. In this sense, this type of evaluation puts the focus on helping communities assess their own activity and results by implementing participatory processes focused on strengthening stakeholder capacities to plan, implement and evaluate. TO FIND OUT MORE ABOUT THE COLLABORATIVE APPROACH IN EVALUATION Shulha, L. M. et al. 2016. "Introducing Evidence-Based Principles to Guide Collaborative Approaches to Evaluation", American Journal of Evaluation, 37(2), pp. 193-215. understanding how processes of change work so that social interventions can be guided better, always responding with the specific context in mind. At the same time, it is used to improve organisational capacities with regard to planning, implementation and also evaluation. To do this, the diverse interests held by the different stakeholders and the context-specific characteristics are used as starting points. The evaluation team facilitates the negotiation and evaluative reasoning process (Shulha et al., 2016). Moreover, CAE defines eight principles that are understood to be factors that facilitate this type of practice. These principles are: (1) explain the motivation to collaborate; (2) boost significant interprofessional relationships; (3) develop a shared understanding of the programme; (4) promote suitable participation processes; (5) supervise and react to resource availability; (6) supervise the development and quality of the evaluation; (7) strengthen evaluation thinking; and (8) monitor to ensure findings are put to effective use (Shulha et al., 2016). 47 46 example of an evaluative innovation from within this approach is Outcome Harvesting, developed by Latin American, Ricardo Wilson-Grau (2019). It considers that a highly participatory process is indispensable to achieve a successful evaluation route and product. The versions briefly described here coincide in presenting evaluation as an exercise which can, and should, give voice to diverse stakeholders, and their different interests, in order to understand a multiple, complex and intangible reality. The goal is to increase understanding of the processes and/or their results as an indispensable platform from which to apply changes. It is worth mentioning that these evaluation perspectives have been created in the English speaking world and in the context of the Global North, with their theoretical and methodological roots in the action-research approach of Kurt Lewin (1946) or, more recently, in methods drawn from Participatory Rural Appraisal / Participatory Learning and Action (Chambers, 1992). While these influences have had some influence in Latin America, it is the significant prior novation and adaptation are considered integral parts of the intervention and as such are located in the design and information gathering phases so that support for the programme and/or organisational development is viable with opportune and swift feedback (ideally, in real time). The evaluation team collaborates closely with social innovation professionals to conceptualise, design and test new approaches in an ongoing process of adaptation, intentional change and long term development. The team’s main functions aim to clarify the innovation and adaptation processes, tracking the implications and results to facilitate continual decision-making in real time, based on data that emerges from the development process (Patton, 2012). One TO FIND OUT MORE ABOUT DEVELOPMENTAL EVALUATION Patton, M. Q. 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press. 61 60 3. Institutional representatives work with local stakeholders in the design, implementation and interpretation of the evaluation findings In participatory evaluation, evaluators work alongside social actors or stakeholders in an intervention in order to generate evaluation knowledge for that intervention. These social actors include those responsible for the programme while examples of local stakeholders include grassroots organisations, user groups and technical staff that work on programmes at local level. Within the framework of participatory evaluation, they are the ones to define what will be evaluated, who will participate, when it will be carried out, what data collection and analysis methods will be used and how the results will be communicated (Coupal, 2000). All stages in the process are relevant but the scope of the evaluation focus and its characteristics, negotiated between the different stakeholders, is what sets the tone for the rest of the activities. This prompts the generation of participation moments that are effective for both clearly delimiting the desired participation objective and structuring and channelling this participation during the evaluation process (Aubel, 2000). 4. The use of didactic tools and materials facilitates dialogue by generating spaces and procedures for gathering, analysing and using information There are challenges involved in developing effective participation in diverse and culturally complex contexts. These challenges are related to real inequalities that exist between the different stakeholders with regard to their power, voice and capacities. Participatory evaluation has tackled this challenge by suggesting and creating innovative participatory didactic materials and tools that recover and keep present the perspective of everyone involved in the evaluation process, independent of their location, position, gender, etc. (Chouinard and Milley, 2018). 63 62 Before offering a definition of participatory evaluation to structure and formalise the conceptualisation, we will first present seven key principles that are specific to the Latin American context and that also make explicit its transformative nature. When the tools are used, the rational and emotive dimensions come together and this favours more real ownership of the reflections and their findings. In addition to this, evaluation becomes a way of fostering the development of critical thinking in the people involved in the process. The tools, workshops and games prompt exchanges that deepen individual and collective knowledge; return centre stage to the stakeholders; facilitate debates, trust and innovative forms of learning and reflecting; and affect the desired transformative direction. 5. The participants or stakeholders take ownership of the evaluation process and results The notion of “participation” is used in multiple ways and this means that it can become vague and unclear, diluting its ability to democratise and transform (Cornwall, 2008). In the evaluation context, social participation can be understood and interpreted in diverse ways (Chouinard and Milley, 2018), but multiple efforts have been made in Latin America to overcome its instrumentalist use, in a bid to stop local stakeholders being used as informants for the purpose of legitimising an evaluation process that has actually been externally and vertically defined, i.e. a topdown focus. This is made possible through a horizontal distribution of power in the different decision-making processes required in the evaluation. 6. The evaluation process strengthens local skills for participatory planning and decision making The empowerment perspective in participatory evaluation reflects Freire’s contributions regarding the creation of power in people and the central role in organising and mobilising them for this purpose (Torres, 1987). Participation requires the involvement of local stakeholders to capture the specificities of the place and guarantee that their perspectives are included. 65 64 participation is not limited to those who have a more structured discourse equipped with more skills and power. In the same way, if participation is not managed properly, frustrations can be generated between the people involved; a lack of motivation for effective participation can creep in; and the evaluation’s empowerment potential can be reduced. Within this framework, the role of the evaluation teams as facilitators of the process becomes increasingly relevant as they are a key central part of participatory evaluation at every stage. The evaluation team “has to be respectful and able to manage group construction processes as they decide what should be evaluated and how this should be carried out, and define the evaluation results, conclusions and recommendations” (Espinosa Fajardo, 2019). Thus, in addition to integrating other points of view and helping form a shared interpretation of the programme under evaluation, participatory evaluation also generates a process through which local stakeholders increase their decision-ma- king capacities for their own lives and their environment. As a result, they are able to handle mechanisms that can improve their position in society. 7. The evaluation teams act as facilitators in the evaluation process Participation in evaluation can empower, contribute to learning and improve the democratic culture of organisations and social groups. However, as we have shown, it can also be used in a purely symbolic way and reproduce the same asymmetries of power that exist in the real world if we are not intentional in ensuring that Valuing the perspective of multiple stakeholders implies recognising their knowledge, constructed from social, cultural and historical practice. Participatory evaluation generates a process through which local stakeholders grow in their capacity to make decisions regarding their own lives and surroundings. 67 66 Levels of participation INFORMATION SUPPLIERS Our creation, inspired by Arnstein (1969). PARTICIPATION THROUGH CONSULTATION PARTICIPATION FOR INCENTIVES PASSIVITY FUNCTIONAL PARTICIPATION INTERACTIVE PARTICIPATION INDEPENDENT DEVELOPMENT The stakeholders provide information generally through questionnaires, but do not have influence in the evaluation. The stakeholders are only informed about the evaluation. The local stakeholders are listened to but not included in decisionmaking concerning the evaluation, its results, conclusions and recommendations. The local stakeholders participate in groups in order to respond to the evaluation’s goals. They do not have an influence on the way it is formulated but they do impact the way it is developed. The local stakeholders participate in all phases of the evaluation in an active manner. Local stakeholders initiate and develop the evaluation without waiting for external interventions. The stakeholders participate by providing information in exchange for incentives. 69 68 3. PARTICIPATORY EVALUATION IN ACTION: AN OPPORTUNITY TO ADVANCE HUMAN RIGHTS, INCLUSION AND EQUITY One of participatory evaluation’s key characteristics is the way it gives voice to the different social actors. However, the participation of these stakeholders in evaluation does not always lead to the same type of practice and, as we have shown, there are even times when exercises are not deemed to be strictly participatory. To that effect, the multifaceted and kaleidoscopic nature of the concept of “participation” is not anything new and has already been explored by Sherry Arnstein (1969) in her “ladder” or levels of participation. In line with this author, there are different types of participation that form a continuum that goes from the manipulated use of participation at one end, to participation exercised and managed by citizens at the other. In reality, when you go beyond the discourse in support of participatory processes, participatory evaluation does not always reflect a truly participatory “vocation” (Guijt, 2014), and the notion of participation has acquired different connotations. In fact, often stakeholder participation only equates to the role of information providers. In this sense, participatory evaluation is reduced to moments of mere consultation (passive participation) without offering the local stakeholders the possibility of influencing all the decisions that are made during an evaluation process (Tapella and Sanz, 2019). All this reveals practices of exclusion and privilege as it sheds light on core issues in evaluation such as voice, power and politics (Chouinard and Milley, 2018). More specifically, as Chambers (2003) highlights, evaluation runs the risk of becoming a mere symbolic simulation as it does not confront the status quo and redistribute power. The same drive to establish participatory evaluation has the potential to generate location-specific knowledge as a result of bringing together local stakeholder knowledge and that of external practitioners. This dialogue between different knowledge sources developed in the evaluation process includes deliberative exchanges between the evaluator team and a host of stakeholders, establishing an atmosphere of democratic debate (House and Howe, 2000) at the heart of participatory evaluation. As participatory evaluation processes contribute more in terms of flexibility and adaptability to the context, and given that they address the needs and worries of local stakeholders to a greater extent, stakeholders are more likely to take ownership of both the evaluation results and the evaluation mechanisms and logic (the methodological design, fieldwork, data analysis and presentation of findings and recommendations). This final point has a direct and positive impact on how effectively evaluation is used as it allows people to take ownership of the intervention process and establish themselves as bearers of rights. In a process of this type, time-bound and developed collaboratively between different stakeholders, participatory evaluation has the potential to strengthen local skills that can be activated beyond the project under evaluation. This can be seen in the diverse attitudes and behaviours that can be activated through this type of project and which reflect stakeholder empowerment: flexibility, critical reflection, negotiation, seeking consensus, creativity, etc. In addition to fostering greater knowledge and voice among individuals participating in processes and decisions that affect their lives, ensuring that nobody is left behind, participation in the evaluation context can strengthen the internal capacity of organisations, their learning and generation of learning with regard to the way the interventions perform and ways they can be improved. The ability to contextualise evaluations is one of the clearest benefits of bringing together participation and evaluation. Various participatory evaluation experiences analysed in the Latin American context showcase its potential for centring evaluation around people, by increasing the impact felt by the various stakeholders. It is within the framework of collaborative and horizontal work that the reciprocal attention of plural perspectives, contributed by the different stakeholders, can be appreciated when it comes to assessing the programme’s success. 71 70 Participatory evaluation has the advantage of being more adaptable to specific contexts, focusing on people, aligning with rights-based approaches and enabling advances in inclusion and equity. One particular aim in Latin America is to separate participatory evaluation from “tool-centrist” visions, in other words, those that tend to equate the whole approach and its purpose to a set of tools that encourage participation. Their absence could even be positive if they are only being used as “fun” alternative ways to present content and direct evaluation efforts whilst pretending that they are gaining the perspectives of local stakeholders. The tools, workshops and games are, however, central and highly relevant parts of the participatory processes. Through them, learning comes from the collective task of “doing together” and “throwing your whole self in” (and not just your thoughts). What’s more, this “throwing one’s whole self in” is a strongly Latin American trait, not often valued in evaluation models based on other rationales. On the other hand, participation in evaluation practice also reveals that it is not always possible or relevant to carry out the strictest versions of participation. Although this may be a long term goal, the participation level that can be achieved often depends on the programme, the specific context and the evaluator team, among other factors. In this regard, it is always important to remain aware of who is participating, how they are participating, in what activity, what the real participation potential is in each specific case and how to keep promoting it. In the balance between the desired participation and possible participation, the experience in Latin America, as we have commented in the previous paragraph, underlines the importance given to ensuring that social actors participate as bearers of rights. In this sense, participatory evaluation views the acti- ve role of the stakeholders as the natural extension of their rights as citizens. In addition to this, it understands evaluation to be an opportune and perfect moment to encourage this commitment from public institutions. This tendency within participatory evaluation in the region overlaps with the drive over the last few years to incorporate Human Rights into the very core of evaluation. In line with evaluations that include a Human Rights based approach, several different versions have been developed to help evaluation become an instrument to “promote, respect and 73 72 guarantee the real enjoyment of human rights with special attention awarded to the most vulnerable groups” (Ligero Lasa et al., 2014, p.19). In this case, the research process focuses on valuing how the programme contributes to reducing discrimination, developing skills in stakeholders and encouraging their participation, demands and action (Ligero Lasa et al. 2014). With regard to the connection between participation and rights, participatory evaluation carried out in the region is also an opportunity to advance in inclusion and equity. Participatory approaches have been criticised for being naive at times as they have not always considered the inequalities existing within the social groups involved in the programme and evaluation (Guijt and Shah, 1998). In recent years, Latin American practice YOU CAN WATCH THE VIDEO “AN EVALUATION WITHOUT EVALUATORS” HERE has started to recognise existing discriminations and has suggested the need to work to transform this. It is generally recognised that among relevant stakeholders, there are diverse needs, interests, knowledge and knowhow. Likewise, existing power relationships between participants are being taken into consideration more, as are the different situations of discrimination and inequality that may surface. However, no social actor type is one and the same and so a stakeholder's situation and position cannot be explained just by considering one of the categories that they fit into, for example, their socio-economic status. Attention must be paid, in each specific evaluation, to the different social characteristics at play among the stakeholders and how they overlap in each case. This requires an approach that understands that the different systems and power structures are interwoven and can change over time (Hankivsky and Jordan-Zachery, 2019). It is necessary to understand how different forms of inequality, such as socio-econo- mic status, gender, ethnic origin, age, etc, intersect in each specific case. Furthermore, philosophical, political and practical considerations that mediate and reproduce certain types of discrimination and exclusion, affect not only evaluated programmes, but the very evaluation itself (Cousins and Chouinard, 2012). With regard to this point, it seems highly relevant to bring to light and analyse the social position, roles, values and assumptions existing in the programme under evaluation, the evaluation itself and the context that surrounds it. Thus, evaluation presents an opportunity to highlight and change beliefs that are classist, sexist, racist, homophobic etc. Likewise, participatory evaluation practice in Latin America has initiated dialogue and efforts to establish connections with evaluation proposals that intentionally incorporate a gender and intercultural focus with the aim of questioning the structures and logics of power and discrimination (Faúndez Meléndez and Weinstein, 2013). It places an emphasis on giving voice to those people and groups often silenced, protecting the rights of all those who participate in the evaluation, and recognising and valuing diversity. It also pinpoints more specifically where others are coming from. For example, it can be observed that the programmes are not neutral in terms of socially constructed gender ideologies and if they do not adopt relevant measures, they will reproduce the same type of structural inequality. In the same way, some of the region’s practices recognise and value the cultural diversities found in Latin American societies and promote the rights and voices of indigenous people and afro-descendants. It is therefore important to properly identify the key stakeholders and their diversity from the start of the evaluation and throughout their involvement in the process by breaking with vertical or top-down approaches that are often the norm. In finding a balance between the desired participation and possible participation, the experience in Latin America underlines how central it is for social stakeholders to participate as full subjects. 75 74 has shared influences and synergy with experiences such as Popular Education and the Systematisation of Experiences. Furthermore, participatory evaluation in the region shares a set of specific principles that consider stakeholders to be full subjects with rights, recognise and value local knowledge, promote horizontal didactic tools and materials, encourage stakeholders to take ownership of the processes and results, and strengthen local skills and the facilitator role of the evaluator team. This chapter has highlighted how the relationship between evaluation and participation has an extensive history. Several versions seeking to involve diverse social actors in evaluation processes have been described. As we have seen, participatory evaluation is connected to transformative evaluation, empowerment evaluation and the collaborative approach in evaluation, among others. In the Latin American context, its development Yet, participation does not always result in the same type of practice. Even though there are times when participation is used as a “label”, and does not take advantage of its transformative power; at other times, it can open up space to, little-by-little and according to each context, provide opportunities that enable advances in rights, inclusion and equity. In this regard, participatory evaluation can constitute a tool with wide-ranging potential to help promote participation among diverse stakeholders, actively include their voices, needs, interests and knowledge and, in line with the 2030 Agenda, not leave anyone behind. In the next section, we will explore how this type of evaluation activity is put into practice, how to facilitate participation in evaluation, and what tools we can use for this purpose. 4. CONCLUSIONS 77 76 Chapter 3 How to conduct a participatory evaluation 79 In this chapter, we describe how to carry out a participatory evaluation step by step. We will pay special attention to the features that differentiate this type of evaluation from others: both those features that affect the evaluation process in general, and those that are specific to one of the steps in particular. We will also highlight specific tools and techniques that ensure that participatory evaluations are carried out with the same methodological rigour and quality standards expected in any evaluation. 1. INTRODUCTION The steps taken in a participatory evaluation are not too different from those in a conventional evaluation. There are obviously certain elements that are different or steps that require special attention when you want more stakeholders to participate, but all evaluations, participatory or not, follow a very similar pattern. They begin by formulating questions about the project under evaluation and then search for the information needed to be able to respond to them. The process ends when these responses translate into improvements in the project and its impact. This sequence of events is normally known as the “evaluation process”, and the three activities described above are referred to as the programming, implementation and use “phases” of evaluation respectively. The steps that have to be taken within each of these phases are commonly known as “stages”. It is at this more detailed level of the evaluation process (stages) that the features that differentiate participatory evaluation from other approaches are found. Therefore, we will review these stages in detail in this chapter. 93 92 An evaluation can propose aims that are not directly related to the specific evaluation questions (for example, increase coordination between the participant stakeholders in the programme, or give an account to the entity that finances the project). On the other hand, if the evaluation matrix includes a question that is not related to any concrete aim, the evaluation team should consider if it is really necessary to respond to this question before going ahead with the evaluation planning. A common problem in evaluation (and also in project planning is that the aims and questions formulated are too broad and so can be interpreted (and answered) in very different ways. The risk for the evaluation is that the responses generated do not correspond to the evidence that was sought when the questions were formulated. Using the indicators (sixth column of the evaluation matrix) is often helpful for avoiding this problem and being more concise in formulating evaluation aims and questions. The indicators express what should be achieved or measured, and they are very useful for detailing clearly what the evaluation is going to analyse. For example, if one of the results that we wanted to achieve with the evaluation was to improve the coverage of a child vaccination programme against polio, it would be useful to establish with indicators what we are referring to when we talk of coverage: the number of health centres that can administer the vaccine, the number of health staff, the percentage of vaccines for births of children at risk of poverty and extreme poverty, etc. Given that the indicators are what finally establish the evaluation questions it is important that all relevant collectives in a participatory evaluation are involved in debating what should be selected. During the whole evaluation process, but especially in this first phase, it is essential to not lose sight of the evaluation’s purpose. Every objective or question will demand time and resources. For this reason, in this first stage, it is important to discern and agree on the essential questions, those that will produce useful responses or added value to a project and/or its impact. The evaluation matrix that we suggest in this handbook includes a third column dedicated to “the use that will be given to the evidence” generated by the evaluation. We believe this resource should be consulted at all moments during an evaluation. Stage 3: Identify information sources and tools to gather information During this stage, sources will be identified that can provide us with the information necessary to respond to the evaluation questions. The responses provided by the evaluation contribute significant added value as they are the result of applying social research tools and techniques, adding scientific rigour, and making it possible to contrast them easily. For this reason, it is usually said that evaluations provide “evidence”. It is worth consulting various information sources, even several on the same topic or going back to the same source and using different tools. For example, you can ask one of the project’s practitioners to complete a questionnaire and, on another occasion, interview them in order to go into some topics in more depth. This is known as “triangulation” and often results in richer information, removing bias or subjective perceptions. Triangulation adds rigour and credibility to evaluation. The sources are classified as primary when referring to information directly gathered by the evaluation team (for example, when interviews are conducted) or secondary, when information has already been gathered by other people (for example, when a project report is consulted). The primary sources of an evaluation are usually people that have been linked to the project either in beneficiary groups or as managers, planners or funders. However, there are other groups that, although not directly implicated in the project, can provide highly valuable information for the evaluation and should also be considered including community leaders, local or sectoral authorities, The indicators express what should be achieved or measured, and they are very useful for detailing clearly what the evaluation is going to analyse. 95 94 A notable advantage to participatory evaluation teams is that they have members who are in close contact with the programme and so it is easier to identify and access the groups that can provide relevant information for the evaluation. scientific staff or researchers linked to subjects related to the project, etc. An advantage to participatory evaluation teams is that they have members who are in close contact with the programme and so it is easier to identify and access the groups that can provide relevant information for the evaluation. It is even likely that much of the information needed can be provided by the evaluation team members themselves. Classic tools for gathering information from primary sources are surveys, interviews and group meetings. They have the advantage of being flexible tools and enable information required for the evaluation to be obtained in a more precise way. However, gathering primary information is usually considerably more expensive and, furthermore, requires the implication of people who represent the consulted groups. For this reason, the evaluation team must carefully prepare the tools and plan in detail the sessions where the stakeholders will be consulted. An additional challenge faced by participatory evaluation teams, is that the tools for information gathering have been designed to be used by evaluation professionals or at the very least, social science practitioners. As this is not the case for the participatory evaluation team members, the tools should be adapted twice: firstly, to the sources that will be consulted and secondly, to the people who are going to put them into use. How tools are adapted is reviewed in more detail in the pages devoted to describing the evaluation implementation phase. Furthermore, the final chapter deals with the topic of tools that are used to gather primary information, and provides some good ideas and experiences to illustrate how they can be adapted for use in participatory evaluation. The most relevant secondary information source for evaluation is, or should be, the evaluated project’s monitoring system. The main function of a monitoring system is to provide information that guides project management on an ongoing basis. But the monitoring system should also be designed in such a way as to provide relevant information for the evaluation. 97 96 Unfortunately, this ideal does not exist in most cases. Monitoring systems are usually designed in a project’s planning phase or during the first steps of implementation. Unless there are people in the planning teams with ample experience or knowledge on evaluation, it is difficult to foresee at this point what type of information could be required in a future evaluation. However, the participation of these people in the evaluations (especially in participatory evaluation teams) offers them the opportunity to adjust their monitoring systems to ensure that in the future, they gather information that is also relevant for evaluation. Other relevant secondary sources for evaluation include official statistics, academic or information-sharing publications on topics related to the project and other evaluation reports that have been published on similar projects. In contrast to the primary sources, identifying these information sources is usually more complicated for participatory evaluation teams that do not have people with research experience. In this situation, the facilitator’s guidance on the sources and documents to consult, and even a first examination of the information and data to make it more easily understandable for the evaluator team, can be a highly valuable contribution. BY WAY OF RECOMMENDATION Many evaluation teams (including professional ones) discover that the monitoring reports are incomplete, to the extent that not only are they not useful for the evaluation, but they also do not respond to the project management requirements. For this reason, the evaluation teams are obliged to gather very basic data during the evaluation: how many people they helped, what activities they carried out, how many teams were sent, etc. It is necessary to do this because without this data, the evaluation cannot continue, but it should be clearly fed back to the people responsible for the programme that waiting for the evaluation before gathering this data is an error; in part because this is not the function of the evaluation, but even more so because regular monitoring of this data is fundamental for the project management. Stage 4: Plan costs and timeframe for the evaluation Once stage three has been completed, especially if we have integrated all the participant group interests, it is highly likely that our evaluation matrix is full of aims, questions, information sources and proposed data collection instruments. This is the perfect moment to estimate financial costs and, perhaps even more importantly, the time that will be spent on the evaluation. It is also the moment when the stakeholders should negotiate which questions should be prioritised and concentrate evaluation efforts on the questions that can be answered with the resources available and that are going to be the most useful for the project and/or their impact. In this stage, it is important to remain aware that the evaluation process does not conclude when the evidence generated from the evaluation is presented. Instead, this evidence should be communicated to the stakeholders who will then be responsible for making corresponding decisions and monitoring them to ensure that they achieve the improvements expected for the project. Keeping aside resources that will be needed to ensure that the results are used properly will allow us to approach this final and important phase in the evaluation process with certain guarantees of success. Therefore, in line with the evaluation’s instrumental character (with an emphasis on the importance of putting its findings to use), this programming phase must also designate activity planning, time and resources to the evaluation process’ final phase which is dedicated precisely to the way this evidence will be put to use to bring about improvements in the project. Cost planning In conventional evaluations, most financial resources are used to contract professional evaluation teams. The participatory evaluation team members usually collaborate through their connection to HERE YOU CAN WATCH THE VIDEO “THE CHALLENGE OF USING TECHNIQUES THAT INCLUDE DIVERSE STAKEHOLDERS" 99 98 other functions that they carry out in the project or as part of their voluntary commitment. Therefore, the funds reserved for a participatory evaluation are used for concepts such as paying the facilitator, if contracted externally, and covering the cost of food, transport and necessary materials for the evaluation team work sessions. There are also costs involved in programming meetings for gathering information and the sessions for presenting and monitoring the way evaluation results are used. In all cases, these are costs that the organisations are usually used to calculating as they are comparable to the projects’ many other activities. Planning timeframes Calculating the time required to implement a participatory evaluation is usually far more complex and limiting than calculating the financial costs. Two elements make it essential to carefully plan a participatory evaluation timeframe: the timeliness of the evaluation results and the need to organise moments for tool adaptation and capacity building for the participants. Timeliness is an important consideration when it comes to planning an evaluation’s timeframes. No matter how good an evaluation and its results are, they are not worthwhile if they arrive late or after decisions have already been made. The only way to avoid these situations is to define the timeframes within which the results will be useful and keep them in mind during evaluation planning. It is very possible, for example, that consulting all involved stakeholders in a project would take up much more time and resources than are available, and so it would be necessary to consult a representative sample. As explained at the start of this chapter, the need to build capacity among evaluation team members is a characteristic trait of participatory evaluation. It does not expect to convert the members into professional evaluators, but it does seek to convey the logic of the evaluation process and help them understand the different evaluation tools in enough depth that they are able to use them. In this sense, the role of the facilitator is once again highly important. Their function is to not only identify the capacity 101 100 training needs at each moment, but also, and especially, to meet requirements that cannot be covered through the training sessions. If capacity building is to take place as and when participants need it at different points in the evaluation process, the timeframes need to take this into account. At times, building capacity so that a specific step of the evaluation can be carried out may take more time than the actual step itself, but it is important that the rhythm of the participatory evaluation is tailored at all times to the learning of the participants and not the other way round. If anyone is left behind in one of the steps, they will not be able to follow the next steps and will no longer be authentic protagonists in the process and the participatory essence of the evaluation will be diluted. If the problems to be solved demand quick results, it is worth considering if the participatory approach is the ideal option or not for this evaluation. Once the planning is complete, the evaluation implementation phase begins. Here, activities prioritised in the previous phase are put into action: the information is gathered and analysed in order to respond to the evaluation questions. Moving on to their use, these responses translate into recommendations for improving the evaluated project. It is also in this phase that formats or “products” are created to communicate the evaluation results to the people that will put them into use. In conventional evaluation, this phase coincides with the activity carried out by the evaluation teams. In participatory evaluation, it is not quite the same. As the evaluation teams were formed at the start of the participatory process, in this phase they dedicate themselves to implementing the work plans that they themselves drafted. Stage 5: Gather and record information Participatory evaluation teams are broad and diverse. Furthermore, they know the evaluated project and context intimately. For this reason, the evaluation team members themselves can contribute a significant part of the information necessary to carry out the evaluation. However, this does not mean that they BY WAY OF RECOMMENDATION Conducting interviews in an evaluation is not as easy as presumed at first sight. It requires skills, knowledge and practice to obtain the information that is being sought, without pressuring or conditioning the answers given by the interviewee. In participatory evaluation, it is very important that the teams have several rehearsals before conducting the definitive interview, especially if they are going to broach controversial or sensitive topics. should not also consult other sources to broaden or contrast their knowledge. As we already mentioned in the pages dedicated to evaluation planning, at first the participatory evaluation teams find it difficult to use the data collection tools as they are not familiar with them. Even a supposedly simple tool, such as carrying out an interview, can become complex if it has never been done before, if certain results need to be achieved, or if it is necessary to research with a critical approach whilst maintaining a constructive tone with the interviewee. However, the success of any evaluation depends largely on all the necessary information being collected during this stage. For this reason, participatory evaluation relies on three mechanisms that can help teams with less experience: • adapt the information gathering techniques and tools so that they can be used by people with no evaluation experience yet still fulfil their function; • include capacity building on the use of tools in the participatory evaluation work plans; and • carry out mock interviews that the other participants can observe and correct while also anticipating problems that could arise during the real interview. The figure of the facilitator takes on a special relevance in this stage. They not only guide the evaluation team in the data collection, but also organise these preparatory activities to ensure that they are subsequently used successfully. 103 102 In any evaluation it is usual to collect a large quantity of information in very little time. If this data is not organised as it is collected, it can become very difficult to analyse later. In participatory evaluations, it is common for different people to be involved in different stages and so the challenge to keep the information ordered is even greater. This information organising process is known as systematisation and it is highly important in a good evaluation. A good systematisation system helps to: • during the data collection stage: know at any given moment if there is enough information to be able to respond to the evaluation questions or if more time is needed for this stage; • during the information analysis stage: review and reflect in order to translate information collected into answers to the evaluation questions; and • during the evaluation results use phase: add credibility and methodological rigour to the evaluation results as it enables third persons to know where the evaluation’s subsequent conclusions have come from. Recording or systematising the information that has been gathered can be a complex process. And it normally differs according to the tools used. For example, a personal interview is often recorded by taking notes or by audio recording and the main content is transferred to a table. For group work, notes could be used but it is usually much more practical to record the main ideas on cards that are then ordered on a wall or board and then take a photo. Generally speaking, we are not ready to use a data collection tool until we have decided how we are going to record and order the data that we get from it. In chapter five of this handbook we provide some ideas, adapted to each tool, on how to record and systematise information. But the group’s creativity can also The need to build capacity among evaluation team members is a characteristic trait of participatory evaluation. It does not expect to convert the members into professional evaluators, but it does seek to convey the logic of the evaluation process and help them understand the different evaluation tools in enough depth that they are able to use them. HERE YOU CAN WATCH THE VIDEO “SELECTING AND USING THE CORRECT TOOL” 105 104 be exploited to put forward new solutions that are tailored to each situation. Once again, the support of the person in charge of the facilitation will be extremely helpful for this stage. Finally, it is worth mentioning that, no matter how well this step is planned, changes are likely to be needed regarding which information sources are to be consulted or which tools we expect to use. It is very common to find that the information we have been looking for is lacking or incomplete or that it is more difficult to access than initially expected. This also happens in conventional evaluations: they face the additional complication that the changes concerning planning require new agreements between the evaluation teams and those that commissioned the evaluation. One advantage of participatory evaluation is that the teams are more flexible to respond to unexpected situations. Stage 6: Analyse information Once this data is collected, (or as it is being collected), it is time to analyse it, review the available information and reflect on the responses it offers, always taking as a reference the evaluation questions, the aims and how the responses that come from the evaluation are going to be put to use. We insist again that the aim of an evaluation should not remain at the level of responding to questions, but rather in turning these responses into actions to improve projects or the impact that they generate. For this reason, the analysis results for the information gathered in an evaluation are usually presented in two ways. The first is known as “conclusions” and they should be linked directly to the evaluation questions, demonstrating a direct relationship with the data collected during the evaluation. The second is that of “recommendations”, and the objective here is to contribute to the project through improvement measures that come from the evaluation’s proposals. With regard to formulating the recommendations, the conclusions are taken as a reference but other elements are also considered. For example, whether it is possible or not to put them into practice and the costs involved. In this stage, the essence of the evaluation results is captured. An evaluation that delivers good recommendations is highly likely to be successful and bring many benefits to the project and its associated stakeholders. The formulation of recommendations is usually the part of the evaluation process that is most questioned. This is not only because of their impact on the project, but also because this is the point where the subjectivity of the evaluation team also comes into play. Conserving the connections between the information gathered, the conclusions derived from its analysis and guidance from the recommendations in response to the questions and aims set for the evaluation, will not diminish the pressure felt at this stage, but it will enable the evaluation teams to respond with scientific rigour to any potential questioning. This is what evaluation professionals refer to as the “traceability” of the recommendations as it allows us to track the path that has led to these recommendations so that they are better understood and accepted by third persons. Stage 7: Transfer the results This stage completes the second phase of the evaluation process and marks the transition between the evaluation implementation and use phases. Even though the evaluation implementation phase ends with the results presentation, this does not mean that the evaluation process ends here. On the contrary, this stage leads onto the final phase of use, which is what gives purpose to any evaluation. We make this clarification because many people get confused and think that the evaluation ends when the report is presented. This is because, from this moment on, the evaluator team take on a less central role. In the following phase (phase of use), the focus is put on the teams that are going to use the evaluation recommendations to plan and implement improvement measures in the evaluated projects. For this reason, you could say that the aim in transferring results is to ensure a good One advantage of participatory evaluation is that the teams are more flexible to respond to unexpected situations. 107 106 prefer to have all the information and read a complete evaluation report. For other groups, it will be more useful to have a summary of what is recommended and why. The broad representation featured in the participatory evaluation teams will hugely facilitate the decision on what format(s) to use to transfer the evaluation results to each group. Aside from being attractive, a good evaluation results presentation must be in the sense that it should enable the target groups to correctly interpret the recommendations that are being proposed and turn them into improvement measures for the programmes. This will be easier with those groups that have been represented in the evaluation team. For the rest of the groups, the recommendation is to leave open the option of clarifying doubts directly with the evaluation team. To do this, results can be shared in writing ahead of time and/or in results presentation meetings that can include moments for clarifying doubts held by those who are to implement the recommendations. When the people responsible for making the decisions are not directly contransition between the people who have carried out the evaluation and those who will put the results into use (evaluation recommendations) in improvement measures in the evaluated project. A big advantage for participatory evaluations is that it is highly likely that a large part of the recommendation’s target groups have already been included in the evaluation teams. Each group’s representative will need to share the results with the rest of the collective, but this transfer will be much easier if the proposals come “from within”. In order to ensure a good transfer of the results of their work, the evaluation teams must ensure that: (1) results are presented in an attractive way in order to captivate the interest of each target group; and (2) results are correctly interpreted by the groups that are going to use them so that they can be easily turned into improvement measures. In order to make the presentation of evaluation results attractive, it is important to choose the most appropriate presentation format(s) for each group. There will be groups who will respond best to a face-to-face presentation. Others will 109 108 nected to the evaluation teams, it will be important that the evaluation recommendations are convincing and well argued. This is when the traceability of the recommendations gains in importance (in reality it is as simple as showing the journey the evaluation teams have taken to arrive at each of the conclusions) and the technical rigour applied because it will confer credibility to the conclusions and decisions made by the evaluation teams. No matter what format is chosen to present the results, traceability and technical rigour should always be easily identified as they will determine, to a large extent, whether or not the evaluations translate into improvements in the projects and/or their impact. A good way of organising results that will be presented to various target groups is to take the evaluation matrix as a reference. By reviewing the columns from right to left, you can get back to the questions that led to each evaluation recommendation (result), identify who formulated them and discover the intended use for the resources. This list of evaluation target groups will help us plan what results will be presented to each group and what is the best way or format to do so. Conventional evaluation recommendations are usually directed at a single entity; the one that commissioned the evaluation or, at best, they are separated out for the different units of this same entity. On the contrary, it is common for participatory evaluation recommendations to respond to different entities and/or collectives. This is due to the fact that, as several groups are implicated right from the beginning, the evaluation teams can more legitimately propose recommendations to all the represented groups. This opens up vast possibilities for participatory evaluation as it enables recommendations to be proposed that are based on combined efforts that reinforce each other. Evaluation reports are the classic format for presenting results in conventional evaluations. They have the advantage of including a broad description of the evaluation process: background, aims, methodology applied, etc. Furthermore, they have a standardised structure that makes it easier to compare them with other evaluation reports or for other people to consult them even if they are not connected to the evaluation. A disadvantage to evaluation reports is that their results presentation format is quite rigid. Unless the people who review them are very familiar with evaluation, the norm is that they will have to make a considerable effort to interpret the recommendations correctly, making their subsequent use notoriously difficult. For this reason, almost all reports incorporate a summary (“executive summary”) that includes only the most important information that guides actions for change proposed by the evaluation. Face-to-face sessions are also becoming increasingly common where the evaluation teams present their work’s results and interact with the people who attend in order to help them interpret it properly. Writing the evaluation report is a complex task even for professional evaluation teams. A good report should be both simple and concise so that both the recommendations given and the research journey that informs them can be easily interpreted. And the task becomes even more complex if there are various authors as it is necessary to ensure that the argument and style is maintained throughout, that there are no repetitions, incomplete information, etc. A good resource for writing a participatory evaluation report is the facilitator. This person can prepare a draft of the report beforehand so that it can be reviewed and discussed in detail by everyone in the evaluation team. If the person who drafts the report uses the information that has been recorded in the evaluation matrix as a reference, and given that the facilitator has accompanied the evaluation process from the beginning, it is highly probable that the report draft will accurately represent the general feeling shared by the evaluation team. However, reviewing and discussing the content proposed in detail will help the group enrich and/or consolidate the result of each of the stages covered during the evaluation process. Finally, it should be noted that each format and meeting to present the results in this stage will require time and resources and this should have been considered earlier in the evaluation planning (stage 4). The process includes a final phase, which relates to the use of the evaluation results to improve the intervention 111 110 under evaluation. As we saw in the previous stage, significant support is required to ensure that stakeholders accurately interpret the recommendations for their specific action areas. At this point, we will see what the stakeholders can do to ensure that the recommendations (a) are progressively incorporated into practice and (b) produce the desired improvements. Stage 8: Put findings into action The evaluation recommendations are guidelines for adopting improvement measures. While they are initially formulated by the evaluation teams, the final design and planning of actions corresponds to the groups that receive each recommendation. During this stage, each of the groups will draft an action plan based on the evaluation recommendations, taking into account knowledge they have regarding the programme, its context and the resources available. The wide representation in participatory evaluation teams provides yet another advantage during this stage because if the target groups for the recommendations are implicated in the evaluation, it will be easier for them to understand the logic of the recommendations and put them into actions that produce the desired improvements. As a minimum, for each recommendation, the following should be planned or transferred to an action plan: (1) the activity or activities that result from each recommendation; (2) the person or people in charge of putting it into practice; (3) the resources necessary for its implementation; and (4) a timeframe for this. As has already been mentioned, it is common for participatory evaluation recommendations to have various target groups. This means that during this stage, various action plans are created, one for each group. An interesting exercise (and one that is highly recommended for the successful use of evaluation) is to The broad representation featured in the participatory evaluation teams will hugely facilitate the decision on what format(s) to use to communicate the evaluation results to each group. 125 124 implemented. But, another two factors may be just as, if not more, important than these. Firstly, the person who facilitates should have a clear interest in the process that is unfolding and in the results that will hopefully be achieved and should thus transmit and “infect” all the team members with their enthusiasm. At the same time, it is important that those who facilitate the process are properly prepared. This implies seeking out information on the potential participants: what do they know?, what do they want to know?, what problems do they have? They should also research what tools can be used and how. It is also essential that they successfully seek to learn about the other “co-facilitators” that they will share responsibilities with, and the evaluation team members too. And, above all, that they are interested in learning about themselves, the activities that are put into action and the results. As we have shown in previous pages, facilitators are, to a certain extent, the process “managers”. The facilitators plan the process and ensure that the team achieves the aims set out. This means that they: • set the process in motion: they articulate the terms of reference, aims, timeframes and resources; • build a team and ensure that different people are part of the process; • identify a set of activities and manage time so that they are implemented; • share information with the other participants at specific moments, presenting the plans that they have drawn up, the methodology that will be followed or the way that the people or team members should interact with each other; • guarantee that the whole process follows its course, that the main focus is not lost and that it reaches conclusions as planned; • ensure that the team understands how an evaluation process is developed, is aware of appropriate tools and how to use them correctly; and • provide the team with necessary support to enable access to required institutional support and the right tools for the process. Within the framework of a participatory evaluation process, the main role of facilitators is to ensure participation for everyone involved in the whole process. Facilitators help incorporate different perspectives by gathering different opinions from everyone involved. They also ensure that those who can (or should) contribute most, do it actively, pushing the process to achieve better results. The following section describes each of these points in more detail. 2. GUIDING THE PROCESS we can say that good facilitators should: • have social skills: they should be able to handle a group of people with diverse backgrounds and different interests; • be organised: able to plan a process that lasts over time, build a logical sequence of activities so that each of these activities is completed and the desired results are achieved; • be able to identify problems or difficulties, to see the doubts that participants can have, and respond to them; and • be able to manage time: they should design and follow a programme of activities that covers a period of time and meet the proposed deadlines. Just as we discuss in other chapters in this handbook, facilitators need to know how to carry out an evaluation process, know which steps to take and how to use some of the tools that are relevant to each step. In addition to this, facilitators are often required to know about the topic under evaluation, the area where the work takes place, the local customs and the general context in which the project or programme under evaluation is being LEARN AS YOU GO ALONG How can you be a better facilitator? These days it is possible to find many guides or handbooks on the internet with many specific recommendations. However, all facilitators agree that the best way to facilitate a process is precisely by doing it: facilitating a process and seeing how it is done. Monitor it, ask for feedback... and repeat. 127 126 Building a team A facilitator’s main responsibility is to ensure that the process is truly participatory. This means that all members participate actively. The first step consists, therefore, in analysing who is going to participate and in carefully selecting who is going to be invited to join in the process. So, who should be invited to join the team?, what can or should be expected of each person? Logically, the first thing to take into account is that they should be people who are familiar with the activities to be evaluated. They are likely to have been directly involved in the project or programme that is under evaluation, and therefore have information and a clear opinion about what has been done and what has been achieved. But, in addition to this, the participants should show an interest in getting involved in the whole process. For this reason it is necessary to select those who are going to want to give their opinion, who are going to want to debate with others and who recognise that this will be beneficial. They should know that not everyone has the same interest in participating in an evaluation process and that nobody should feel obliged to do so. Even when a group of people has been involved in a project in a similar way, they are likely to be interested in discovering different results. Therefore, we should start by recognising the different expectations that could be present: what do they hope to discover?, what is going to make them participate in an active way? In some cases it is worth inviting a large number of people who have shown an interest in the process that will soon begin (for example, to respond to a questionnaire), and then invite those that express the most interesting opinions to get involved in other activities. The facilitator could also think of incentives to make people want to get involved. In addition to mentioning everything that they are going to learn in the process, some type of recognition could be considered (for example, awarding a certificate). It could also be important to cover all the costs involved in participating in different stages of the process. 129 128 The facilitator should also consider how these participants come together as a team, whether in terms of the distribution of gender or age, and should ensure that the group of participants is the most heterogeneous possible. It is also important to consider people who have had a different role within the project or programme that is being evaluated: some will have been technical team members, others beneficiaries. With regard to heterogeneity once again, the more diverse the group, the better the results are likely to be. Putting the process into motion Even if the work is carried out with all team members, the person who facilitates is the one responsible for planning NON-BENEFICIARIES A good recommendation could be to invite those that have not been beneficiaries of the project under evaluation, or who have not been involved. These people can have an “outsider” view or can highlight some of the project’s weak points. • determine how long will be dedicated to each of the activities mentioned abo- ve; and • determine the amount of resources that will be required and see if these resources are actually available and if they can be used. It is also necessary to think about the logistical aspects. If one of the programmed meetings will be held online, it is important to find a platform that everyone can access that works effectively. On the other hand, if a face-to-face meeting is going to be organised, a place needs to be found that meets all necessary conditions. Furthermore, travel tickets will need to be purchased and accommodation considered for all the participants. It is often said that this is not the responsibility of the facilitator and that these are tasks that should be taken on by the secretary. But it could also be argued that it is, in fact, the facilitator’s responsibility as these factors are going to affect the participation of those that we hope will participate actively. Ultimately, using a platform that does not permit free access impedes participation in the same way that a bad seating layout does in a meeting room. In both face-to-face and online meetings, an important role for the facilitator at this point is that of managing the expectations of all members. What does each person expect?, what has motivated each person to join the group, and what do they hope to see at the end of the process? It is necessary to show what is realistically possible to achieve in the time available and what things will need to be left for a subsequent session or even a new process. the process and then checks that its implementation goes according to the plan. This is activated through a brief baseline analysis that contrasts “what is desired” with “what is possible”. As part of this process, it is necessary to see if this is the right moment for the evaluation or if it would be better to wait until there are more results to analyse; if there is a particular interest in participating and discovering lessons and recommendations; if there is sufficient information; and if the necessary resources are available (or if they could be found). As a starting point for the whole process, the main planning task is to prepare an action plan. This begins with definitions of the aims that will guide the evaluation and key questions. What are we going to look for? Following on from this, the next task is to determine the data collection, systematisation and analysis activities for the evaluation. Some of these activities could be carried out online and others need one or several face-to-face meetings. In this case, it is necessary to: • decide when they will take place and where; A SHARED VISION As we have seen, it is important for facilitators to have a clear idea of what their role is and comply with the participants’ expectations of them. But, it is just as important that the participants also understand this so that nobody is disappointed when facilitators do not present themselves as experts or the ones who are going to solve all the team’s problems. Everyone needs to know what the facilitator is not going to do and what the team’s responsibilities are. 131 130 Creating conditions for collaboration There are different factors that influence whether or not team members are willing to work together and how effectively they are able to do it. The starting point is usually a shared interest that they agree on: the general aims of the evaluation process should be in line with each individual’s vision and vice versa. But it is also important for team members to feel free to express themselves, give their opinion and feel listened to. An important aspect of the work of facilitation is to get all participants to feel comfortable and maintain that feeling throughout the whole process. A key part in the whole evaluation process is to define the core features that will guide the process. The facilitator should ensure that participants reach agreements on the importance of the evaluation and can ask specific questions regarding the intervention's relevance, efficiency, effectiveness, impact and sustainability as well as other aspects they choose to prioritise. The person facilitating a participatory evaluation process will have the important role of managing the expectations of everyone involved. What does each person expect?, what has motivated each person to join the group, and what do they hope to see at the end of the process? A REPOSITORY Working virtually will require documents to be shared and also stored so that they are available to others. In many cases, it will be necessary to edit them, without duplicating efforts and without losing previous versions. Consider a platform or system that is easy to use, that you can go into quickly and simply and does not require a subscription to paid programmes. Likewise, it is important to consider that participatory evaluation is a process that takes time: we need enough time to present an idea, analyse it in detail and give an opinion on it (if you take into consideration that all team members will also be busy with their own day-to-day activities). If we want a considerable amount of people to take part, it is logical to think that it will not be possible to bring everyone together at all times. For this reason, it is important to think of tools that allow us to stay in touch and share information from a distance. The growth of the internet and development of software programmes has had a 133 132 very positive effect in this regard. Today it is possible to communicate with someone practically anywhere in the world with no problem or expense and we do it almost without thinking about it (just think of the amount of time we dedicate to reading and writing emails). We can now use many tools that are easy to use, free and do not require additional inputs or equipment. But, at the same time, working virtually is not always easy. Some offices do not have the necessary equipment yet; some people find it difficult to express themselves in front of a camera; and misunderstandings seem to be more frequent. Despite the number of people that use platforms like Facebook or Twitter, many of us find it difficult to give our opinion in public spaces when we know it will be recorded and on record forever. Facilitators should exploit the available advantages to achieve greater participation and at the same time try to minimise the difficulties. They should plan these processes with a set of specific criteria in mind. Forming the work groups • The initial suggestion is that, as far as possible, work should be carried out by small teams (no more than ten people at a time) simultaneously. Just as in workshops or events, it is better to build homogeneous teams with regard to interest, and heterogeneous with regard to experience. It is necessary to assign specific roles and responsibilities, not just talk about end goals or aims. Online meetings • In order for online meetings to run effectively, provide clear guidelines for each meeting that set out the “rules of the game”. To do this, you should define beforehand how long a meeting will last and what is expected from each participant. • Plan short meetings and invite the participants to introduce themselves, assigning enough time for informal conversation. • Ensure that the participants are not distracted or busy with other things at the same time (“multitasking”). Virtual collaboration requires everyone to be focused. It is also good to take regular breaks for informal conversation. A medium and long-term plan • Consider implementing online meetings as part of the process (preferably after a face-to-face meeting). The team members will feel more comfortable in a virtual discussion if they already know each other. • Follow a pre-established plan, with regular meetings (that are not impromptu). • Consider a regular monitoring process, which should demonstrate what has been achieved to date and what still needs to happen in the future. Technology • In addition to the “live” discussions, think also about the steps that should be taken before and after, and what you will do so that these discussions are more effective. Here we are referring, for example, to those systems that allow us to share documents or work on the same document from different places. • Consider using good quality tools, equipment and programmes that are not necessarily the most recent (as these will probably not be available everywhere). COMMUNICATIONS As there are many ways to send and receive messages, it is best to just use one or two in order to avoid information being lost along the way or not reaching many of the participants. What system should you choose? Consider: • if the participants mind combining “work” messages with private ones; • if everyone has access to it; • the need to use passwords or enter a platform, which can make its immediate use more difficult; • the fear of feeling “bombarded” and, with that, the possibility of only receiving daily or weekly summaries; and • the possibility for messages to be stored, and available for a period after. HERE YOU CAN ACCESS THE VIDEO “HOW TO GIVE CENTRAL ROLES TO ALL VOICES" 135 134 • Choose one or two tools to communicate with others, but do not abuse the frequency that messages are sent. The participants will get confused if the messages come and go through different programmes or platforms or if they do not know what tool or programme to use for each situation. Be aware that many people connect through their mobile phones and so compatible programmes or platforms should be used. At the end of every meeting, conduct a small evaluation. What went well? What do we still need to do? It is important to take notes and share them with all the participants as well as with those who were not able to participate this time, but who will participate in the following meeting. Ensure participation throughout the process Even though we generally associate facilitation with an event or session, it is necessary to highlight that the facilitator ensures involvement from all participants throughout the whole process and not only during the meetings. It is worth emphasizing that participatory evaluation does not end when a workshop or meeting finishes but rather that it is a long process that requires ongoing discussions and a continual exchange of opinions. Depending on the size of the group, the facilitator can consider forming subgroups during the process, whether homogeneous or heterogeneous. In some cases, there will be people that will feel more comfortable talking with their peers. At other times, forming heterogeneous groups will allow for more interesting results. It is also necessary to see who is going to participate at each given moment (or who is expected to). The ideal is that everyone participates in all phases, but this may not be possible (or may not interest them). We should recognise that there will be moments when participation will be lower, when the facilitator will have a bigger role or when one group/subgroup will be more involved than others, and so you could use a table like the one below to show how involvement is distributed through the process. This will help us plan different activities in order to obtain better results. Throughout the different phases and stages in the process, the facilitator will seek to create the right conditions so that participation will continue. If by “participation”, we understand an exchange of ideas and opinions, or joint HERE YOU CAN WATCH THE VIDEO "PUTTING STAKEHOLDERS AT THE CENTRE OF THE WHOLE PROCESS IS KEY". COLLEAGUES OTHER INTERESTED PARTIESBENEFICIARIES DONORS OTHERS ANALYSIS PREPARING BASELINE PREPARING REPORT PLANNING ACTIVITIES DISSEMINATING AND USING INFORMATION GATHERING INFORMATION INSTITUTIONALISING Activities according to participants 137 136 work between different people, it makes sense to hope that this will continue beyond the evaluation process. Creating the necessary conditions means ensuring that: • participation is not seen as an additional load that distracts people from their work, but rather that it is part of their daily responsibilities; • the necessary resources are available; • it is possible to use tools and platforms (for example, they do not require a subscription); and • there is critical mass, in other words, a sufficient number of participants that enables greater collaboration and exchange. The most important aspect, as will be seen later, will be to regularly show that this participation is beneficial and that it is worth playing an active role in the process. As we saw at the beginning of this chapter, facilitators are not the main evaluators, nor necessarily evaluation team leaders. However, they fulfil a key and non-delegable function within the participatory evaluation framework: supporting and building capacity in the team conducting the evaluation. One of the purposes of a participatory evaluation, in contrast to a conventional evaluation process, is that the participants develop new skills or capacities, and this enables them to participate more and plan and implement a better evaluation process in the future. Rather than create a capacity-building process, one of the facilitator roles is to generate the conditions so that participants can “learn as they go”. Team members develop skills that will enable them to participate better and enable this and other evaluations to produce better results. And their responsibility is that everyone learns (including themselves). It makes sense to begin with the evaluation process aims: • what do you want to do?, what do you want to do better?, what do we need to learn?; and • the participants: what do they already know?, what opinions come to mind when they think about a participatory evaluation process?, what experience do they have?, and how do they act when they see themselves as evaluators? While it is not necessary to prepare a guide on developing capacities, it is a good idea to conduct a small diagnosis, as a baseline, and a brief action plan that considers different strategies that could be put into motion throughout the process. We will now present some different ideas for the facilitator to keep in mind. Different roles and responsibilities As has been highlighted, one recommendation frequently given to facilitators is to build a heterogeneous team with regard to gender, age, background and experience; and distribute roles according to specific skills shown by each person (in other words, the person that conducts good interviews, for example, should be the one to take this on). But, alongside this, the facilitator should invite different people to take on specific roles at different moments with the idea of “learning as you go”. The facilitation role involves creating conditions and enabling team members to try and do what needs to be done; feeling confident that they will 3. DEVELOPING CAPACITIES not be criticised, but rather supported by those who have more experience and who will be able to give suggestions and recommendations at any time. Furthermore, and depending on the group, the facilitator could invite someone with more experience to guide another person, as a kind of tutor, so that they work together over a period of time. After this period, the person supported is then invited to accompany others, thus multiplying the learning experience. Organise small rehearsals Another recommendation is to test out specific activities, tailoring them to the EXPERIMENT WITH THE TOOLS In order to learn how to use data collection tools, you need to experiment with them. The evaluation team can test them within the group before carrying out the activities with the key informants. Once the tools are designed, for example the interview guide or the instructions for a focus group, you can act out how it will be used. This will allow for modifications and those who are going to use the tool will gain confidence. 139 138 context, the team’s specific skills (or those of specific team members), or to the needs of the process. This is followed by an analysis of what has been done and achieved, and lesson sharing. One of the best times to test this is, for example, when gathering information. Learning as you go is easier if we try new things gradually. Even though everyone gathered at an event can give their opinion and present data and information, in many cases it will be necessary to seek additional information. One of the facilitator roles is to organise this search and ensure that all team members can access the information needed to draw out lessons and create recommendations. How is this going to be achieved? Where will we need to go and who do we need to consult? And then when organising interviews or focus groups, how is the information going to be recorded? When we observe the context and amount of information already available, the facilitator can set up some experiments and invite the members of a group to see if they work. Thus, in some cases, you could distribute a questionnaire and invite a large number of people to fill it in. In other situations, it will be better to organise a set of interviews and invite key people to give their opinion. It is also possible to form focus groups in which homogeneous groups of people have a discussion and share their opinions. The idea is to not only gather information, but see what method works best. These days it is very easy to gather testimonies and opinions in the field by THE SPOKEN TEXT As many people prefer speaking rather than writing, many software programmes enable dictation and recognise practically everything that we say. This gives us a first draft which is then easy to edit, allowing us to expand on what has been said and add new ideas. This requires ideas to be prioritised and organised before speaking, as in a speech. You also have to make sure the system is working (for example, if it recognises the language); and edit it, adding commas and full stops, so that it has narrative logic and makes sense to those who have not listened to the audio. 141 140 The best evaluation recommendations are those that are taken directly from the analysis. Here, the facilitator should try to centre discussion around areas that will be within the users’ remit and present different options, avoiding recipes that should be strictly followed and considering the costs and risks involved for each proposal. recording them with our mobile phones. But what remains difficult, despite technological advances, is the need to sort the information that is gathered in such a way that it can be easily used and then shared with others. This is another example of a moment when discussing the results seen in these “small rehearsals” could be useful. The person who facilitates can invite the team to use worksheets, check boxes or tables, and systematise the information (show what has been found, and especially what has not been found and what still needs to be found). Each of these opinions can help build a text without it necessarily being complicated. Testing out different options allows us to see what works best in each situation and why. Analyse every step The step that follows the “experiments” is linked to the monitoring and evaluation process described later. It is necessary to continually review all steps taken, specifically what was planned for each stage and the results that are seen. A detailed analysis of each stage will consider: • what happened and how does this compare to what was expected; • the results achieved and how they compare to what was expected; and • the factors behind the results: what factors had a positive effect? At this point it is important to (a) try not to confuse causes with consequences, and, at the same time (b) think especially about the role taken on by each person: what did this or that team member do? Once enough information is gathered, perhaps the most interesting part of the process will be the group discussion and the analysis that goes with it. The role of the facilitator will be to guide these discussions so that they generate specific lessons and recommendations. For this, various tools can be used, such as those identified in chapter five of this handbook. Organise exchanges Just like their role in capacity-building courses, here facilitators will need to organise a process that enables a group of people to go and see what others are doing, and then, go back to their own situations and try to put what they have learned into practice. This shadowing could be organised between teams from one or more organisations. But, the exchanges should not only take place when people travel or visit another area, nor only between members of different teams. Another option is to organise an exchange system within the same team, in the style of a peer review, that enables people to see what others HERE YOU CAN WATCH THE VIDEO “THE SOCIAL COLLECTIVE AS UNIT OF ANALYSIS” are doing, offer commentaries and at the same time receive them. There are specific times when this is especially useful, for example, when drafting a document. It is practically impossible to teach how to write a good document in a course, but it is still very important in every evaluation, especially if you hope to share the results with others in a written report. What should a facilitator do other than draft the process report or send it to someone to draft? Their role is to invite the participants to also get involved in this stage: • make sure there is enough time for everyone to express themselves in writing and comment on what others have written; • provide guides or templates to help with drafting a document (with a “table of contents” to be included), invite all participants to respond to questions or fill in check boxes; • organise oral sessions that can be recorded and transcribed; and • offer commentaries and check not only that the texts include the different points mentioned during the discussions or 143 142 Share resources The final step is probably the one that is used the most: recommend guides, handbooks or articles that are of interest to all the participants; for example, those that are found on the EvalParticipativa website. Basically, this means: • finding new publications and reviewing their content; • suggesting that team members read them, depending on the moment or phase in the evaluation process; and • inviting the participants to write short summaries and organise discussions in which the content is discussed. As we have seen, the role of the facilitator involves ensuring that the process is well planned, that it is implemented and successfully completed, and that lessons are generated that can be developed in the future and that will improve practice. It is just as important to generate lessons that help improve the evaluation process itself; both the process that the participants are currently involved in, and the different processes they will be involved with in the near future. This is directly meetings, but also that they present this information in a clear and concise way. As different people will prepare different texts, the facilitator can build a mutual review system and invite everyone to comment on each others’ work. To do this, it is necessary to gather the texts and distribute them (you can share the name of the author or keep it anonymous) and invite everyone to pay attention to: • the content: is it complete?, is there too much information?, does it present data or information that distracts the reader?; • the organisation of the text: does it follow a clear logical progression?, are the ideas ordered or would it be preferable to reorder them?, does the text invite the reader to keep reading?; • the interpretation and analysis: is what it says correct?, does it present enough proof?; • the general design: is it attractive?, is the language used simple and easy to understand?; and • the final version: are there spelling mistakes?, does it need more charts or images? 157 156 Chapter 5 Tools for participatory evaluation 159 Those of us who have facilitated participatory processes, have at some point wondered what we need to do to achieve the highest degree of involvement possible, from as many stakeholders as possible, in the activities that we plan. One of the many challenges that surface when carrying out participatory evaluation is how to create spaces for real participation where multiple stakeholders can be true protagonists in the evaluation agenda. We know that this is not achieved only by understanding in depth participatory evaluation and the methodological steps, it is also necessary to identify and be able to handle appropriate tools for each social and cultural context where the evaluation is carried out. The use of participatory tools is increasingly valued in the field of evaluation whether to analyse the reality, facilitate communication, build shared viewpoints, stimulate creativity, facilitate decision making or even decrease the volume of some voices in order to make space for quieter voices. A simple look at the tools section on the EvalParticipativa website will suffice to acknowledge that there is a rich selection of tools and activities. Yet, although we have a valuable set of tools at our disposal, we do not always know how to get the best out of them. And while there seems to be an instrument for every potential situation or purpose; we will, however, need to keep recreating them or designing new tools that are specifically tailored to new contexts. But, what are the participatory tools and how can we understand their use in an evaluation process?, what is their potential?, what are their limitations?, what do we need to take into account when we choose them?, and how do we use 161 160 A person does not become a mechanic only because they receive a set of screwdrivers. And yet, even the most experienced mechanics cannot do their job properly without them. Participatory evaluation tools can only fulfil their purpose when we have an indepth understanding of what it means to incorporate social participation. 163 162 them? These and other similar questions usually emerge when evaluators offer to design and guide a participatory evaluation. In this chapter we will try and provide answers to these concerns. First, we will present some conceptual considerations regarding participatory methodology and tools, their advantages and limitations. Then, we will generate a generic classification of methods used, indicating which senses are most activated in an activity and what purpose it is used for. Finally, we will propose a list of seven criteria to take into account when choosing and using these instruments. As we develop the topics, we will include practical examples and recommendations. As we have seen in chapter 2, participatory evaluation in our region is heir to a rich tradition that includes Popular Education, the Systematisation of Experiences and Participatory Action Research. While these approaches have their respective nuances and differences, they also converge at various points and share the same key core feature: they adhere to a liberating and transformative perspective of reality. This is the core foundation that sustains and legitimises these initiatives. They are unique in that they make use of tools that facilitate equal participation among social actors linked to the intervention. Often “methodology” and “tool” are treated as synonyms but we prefer to make the distinction. We understand “tool” as an instrument or technique that helps us carry out a specific job; and “methodology” as the way that we carry out this job, the underlying assumptions and the conception of reality from which we use any given instrument. This implies, as Leis (1989) and Jara (1987) highlight, breaking radically with the tendency to perceive methodological issues to be merely instrumental problems. Expressed in other terms, we believe that we need to be clear about the methodological principles that underpin the elaboration, adaptation and use of a tool or instrument. This implies having conside- 1. PARTICIPATORY TOOLS IN EVALUATION red and chosen a different way of perceiving our evaluation practice and the way that we will relate to the multiple stakeholders during the process. It also implies (de)constructing the role attributed to the evaluator in conventional approaches (see chapter 4 on facilitation). Thus, the same instrument (tool) can be used with different methodological conceptions. For example, someone could use a video documentary to uncritically persuade, convince and condition behaviour; thus constituting an authoritarian practice. But we can also use the same video to generate a space for discussion and debate that is helpful for prompting multiple perspectives on the documentary’s message, and thus constituting a more open and democratic practice. Therefore, the methodological features refer to the set of criteria and principles that unify and give coherence to the adopted evaluation approach. The tools are instruments and techniques that we adopt and their use has to be coherent with these criteria and principles. Therefore, we could use a focus group to gather a smaller number of stakeholders to deal with a topic in depth, or choose a simulation game to create an engaging space where everyone feels free to express what they feel. We could also construct a transect so that, through a collective diagram, we can take advantage of the knowledge of those who do not know how to write or are fearful of expressing themselves verbally. No matter what tool we choose, it should respond to the adopted methodological conception and be consistent with Participatory traditions in Latin America adhere to a liberating and transformative perspective of reality, with one of its unique features being the tools that it makes use of to facilitate equal participation among social actors linked to the interventions. HERE YOU CAN ACCESS THE VIDEO “PARTICIPATORY TOOLS IN EVALUATION" 165 164 the principles that underpin a participatory evaluation and the type of reflexive processes that it intends to foster. Chapter 1 elaborates on these principles and concepts that underpin the participatory evaluation methodology. The tools used are not what make the difference, but rather the open and respectful attitude that supports an evaluation process. This participatory conception of an evaluation assumes that everyone involved in an intervention has the right and possibility, even if at times they require specific training, to be protagonists in evaluating the programme or project they have been involved in. With this methodological conception in mind, the use of participatory tools should be understood as a way to break down inhibitions and fears among participants so that a diversity of opinions can be captured in the most faithful way possible for each intervention wherever possible. Whatever tools or instruments we use under this methodological conception, the evaluation should allow us to: • share one’s personal knowledge on the intervention, enrich it and boost collecti- ve knowledge concerning it; • develop reflection on a shared point, in which participants contribute their particular experiences, broadening collective experience; and • construct potential recommendations or solutions together for the problems identified, enabling everyone to participate in its design. In this joint effort, different perceptions are shared on topics of interest from the intervention and on the assessment of its processes, results and impacts. It also hopes that the people involved take ownership of the methods and tools as well as the logic of the process so that the experience can be multiplied in their groups and communities. This converts the methodology into a process of empowerment for the different social actors involved and their organisations. racteristics and experiences of the participants, the needs of the evaluation, the moment it is implemented, and how the work, objectives and expectations of the participants are established. It is important to grasp that the tools are a “means” and not an “end”. They have an instrumental role to serve a deep and rigorous evaluation process. We are not suggesting that a participatory evaluation facilitator becomes a “tools expert” or “activity guru”, and forgets that these are simply instruments to achieve an aim, that is sustained by ideas of participation and democracy. Whether they are used to facilitate informative, consultation or decision-making processes, they are tools at the service of evaluation. In different forms, they seek to reflectively analyse the practice, synthesis and consensus of ideas, share individual The tools as participatory evaluation instruments There are many different definitions and concepts of tools. In line with what we have said regarding general methodological criteria and principles, we understand participatory tools as those specific instruments that make each step of the evaluation process viable, guaranteeing the highest levels of protagonism possible among the people involved (Slocum, 2003). The tools can be made up of a combination of activities such as group exercises, sociodramas, popular game adaptations, puppets, drawings or puzzles, and any additional way to help us achieve quality participation from the multiple stakeholders involved in the evaluation process. We can make use of a wide variety of tools whenever they are adapted to the topic, the group’s characteristics and proposed objectives including collective interviews, round tables, conferences, or the use of a flip chart with illustrations, a simulation game, a group challenge or a video debate. These tools can be used in a flexible and varied way, meaning that they can be adapted to the context, cha- Participatory tools are those instruments or techniques that make each step of the evaluation process viable, guaranteeing the highest possible levels of participation among the people involved. The elaboration, adaptation and use of a tool or instrument requires clarity on the methodological principles that underpin our work. 167 166 Advantages and limitations The use of participatory techniques contributes many advantages to the evaluation process. As we have said, the use of these tools allows evaluation processes to be developed that are enriched with the contributions of each and every person who participates. This contributes to the collective construction of knowledge on the intervention, its achievements, difficulties, advances and disappointments. These processes also strengthen individual identity and self-esteem and contribute to the consolidation of organisations as they feel recognised and valued. The exchange that occurs through a game or group activity stimulates creativity and the search for solutions to shared problems, enables the objectification of the issue itself and helps different initiatives meet and connect. Furthermore, when recommendations that arise from participatory and inclusive evaluation processes are owned, they are much more likely to be adopted. knowledge, and value various perceptions and feelings about a specific intervention. The use of these tools is supported by the notion that critical knowledge is constructed through a set of intellectual processes and traction which imply forming associations, relationships, and abstractions, and formulating conclusions, analysis or synthesis, in an active and conscious way (Nuñez, 1989). This set of tools, used at different times and in different situations, enables us to treat a topic in a different way, encourage discussions without causing offence, arrive at conclusions that are representative of the whole group after a democratic debate, communicate the result of a deliberation, gather and share relevant information, etc. It is important to grasp that the tools are a “means” and not an “end”. They have an instrumental role to serve a deep and rigorous evaluation process. 169 168 On the other hand, the participatory tools also introduce risks and limitations. Even when there exist a large number of tools in specific books, with a wide range of activities, games and instruments, we will not always find a technique or tool that fits exactly with our needs. In many cases, it may not be possible to use the ideal tool in a specific situation because of time constraints, or a lack of resources or capacity among those who facilitate the process. We therefore see ourselves obliged to redesign the tool, adapt it or combine it with others. Furthermore, no matter how well thought through and designed the activities are, these do not guarantee results and it is possible that some participants prefer not to be involved in the process. It is also important to note that some activities and tools can generate conflicts or set off emotional processes that we are not equipped to deal with and we will not know how to handle them. It is important to be aware of these limitations and recognise that the tools “cannot do it all”. In the second section of this chapter, we will offer some suggestions with regard to the selection and use of the tools as a way of mitigating these risks and difficulties. Transcending recreation and motivation As we have said, the tools and games are not, in and of themselves, the heart and soul of participatory evaluation. However, we consider them to be a key and highly relevant part of the participatory process because they enable us to “do together”. While these tools are being used increasingly in educational processes and also in programme and project management and evaluation, in many cases we have observed that they are only being used to motivate the group, “break the ice”, introduce each other, have relaxing moments between work sessions, etc. We believe that the use of participatory tools in evaluation, especially the games, should transcend recreational and The use of participatory techniques contributes many advantages to the evaluation process. On the other hand, they also introduce risks and limitations. motivational purposes, going beyond entertainment or the idea of “hanging out”. Likewise, these instruments should not be used in such a way that they require each participant to take on a role that is far from their own reality, thus evading their own situation to submerge themselves in a world of fantasy. To understand the participatory tools in a different way, we have to ask ourselves how to use evaluation or “content” to deal with a complex topic that requires both background knowledge and professional participation and requests personal opinions and options in a participatory and horizontal way that is not domineering. Most participatory tools that we could use in evaluation require participants to “throw their whole self in”. In other words, they have to act, discuss, argue and defend a posture with respect to an intervention or situation that affects them. To do this, the participatory tools that we use should try to reproduce, as faithfully as possible, the conditions of each situation, generating exchanges that can deepen personal and collective knowledge and facilitate trust and shared learning. Said in another way, the tools that we design have to incorporate the elements that take, or have taken, place in the real lives of the people and the intervention under evaluation. This way, the participants represent their own role, “living the game” or “playing their life”; or in other words “playing seriously”. Understood this way, the tools also help us to distance ourselves somewhat from the object of study as it allows different perspectives to surface as we go If when we use tools and techniques, we are looking for “fun” alternative ways to impart content and guide the evaluation effort while pretending to include the perspectives of local stakeholders, we could easily underestimate their use. By doing this, we would be manipulating the participants by introducing content in a “fun” and “friendly” way and conclusions from external needs and logic. In these cases, it perverts the profound sense of participation implied in a participatory evaluation (Tapella & Rodríguez Bilella, EvalParticipativa, 2019) 171 170 The use of participatory techniques should help us to develop a sense of “us”, teach us to think actively and critically, stimulate collaboration, responsibility, independence and creativity. It also allows us to overcome fears, inhibitions and tensions and create an atmosphere where we can express ourselves and collectively construct. For this, it is necessary to choose tools that facilitate communication and trust, and will stimulate participation and active cooperation. 173 172 beyond common sense and question reality. Freire once said: “The best way for someone to get closer is to get further away from the knowledge theory view”. And if this way of overcoming common sense is facilitated via tools that contribute space for humour, irony, drama, surprise and collective analysis, they will be contributing to the production of knowledge in the evaluation context. The tools that we use have to be capable of creating conditions for us to communicate, express our feelings, experiences and knowledge, our ideas and expectations as well as learn and find out about different topics and situations in a non-hierarchical way. It should be possible to use these tools to relive situations from our lives, our work and our organisations. Thus, the rules of play facilitate exchange and generate a space for conversing about complex topics that we would not otherwise address. The dynamics of play involved invite the participants “to switch channels” during the game in order to be able to deal with difficult topics, ensuring that everyone's perceptions are heard. It is also important to think about the mechanisms that make a process more dynamic or that constitute the “key” that opens up this space for communication and collective construction. We are referring here to a sort of scheme or ploy involved in games; in other words, the tool includes a dynamic of achieving a shared aim and this makes the participants “get involved”. This is simply a participatory and democratic way of dealing with issues that would normally be difficult to broach. Furthermore, many of the games that we invent or adapt use the logic of “competition” (horse race, lottery/bingo, card/dice games, etc.). This should only be a way to get people motivated and not constitute the end goal. The games should stimulate “cooperation” and the idea that “we all win” more than competition. As we have mentioned, there exists a wide variety of participatory tools. In the fields of popular education and social development, the use of participatory tools, games and activities has increased exponentially in programme and project planning and evaluation over the last twenty years. It would be impossible to include all the existing instruments that are useful for a participatory evaluation in one single handbook. Just by looking at the EvalParticipativa website, we can consult a repository with dozens of handbooks and guides with tools that we could use in an evaluation that seeks to put multiple stakeholders at the centre. With the aim of illustrating the plurality of participatory tools in an organised way, we will use this section to classify the tools by their predominant method and the main purpose they will be used for in diverse situations and contexts. In the first classification, the tools are organised according to the method or senses most activated in the activity proposed. In the second classification, the tools are ordered according to the main purpose they will be used for in diverse situations, needs and contexts. These classifications are not rigid or static as many tools combine the method with a set of human aptitudes to express and communicate ideas. At the same time, the same tool could be used for different purposes. Tools according to method We can classify the tools into four groups according to the method and main senses involved: (1) audiovisual tools, (2) graphic and textual tools, (3) narrative tools and (4) experiential and group tools. The audiovisual tools correspond to the combination of images (illustrations, photos or video) with sound (audio and music) which is predominant in their nature. The most common examples of this tool type are audiovisuals (with slides or video), song analysis or audio debate, or The tools that we use have to be capable of creating conditions for us to communicate, express our feelings, experiences and knowledge, our ideas and expectations as well as learn and find out about different topics and situations in a non-hierarchical way. 2. CLASSIFICATION OF PARTICIPATORY TOOLS 175 174 TOOLS ACCORDING TO METHOD Classification of participatory tools according to method and purpose TOOLS ACCORDING TO PURPOSE AUDIOVISUALS PRESENTATION AND MOTIVATION DATA COLLECTION ANALYSIS, REFLECTION AND EXAMINATION COMMUNICATING AND SHARING RESULTS Source: tools identified in the guides available in the EvalParticipativa repository. This is not an exhaustive list but rather an (illustrated) example of how tools can be perceived according to “method” and “purpose”. You can find out about all the tools and methods through the following link. DIAGRAMS & TEXT NARRATIVES GROUP & EXPERIENTIAL • My photo, my story. • The project in 5 photos. • Proverbs and sayings. • Puzzles. • My life’s cart. • Introducing in pairs. • A man of principles. • Live story. • The spider’s web. • Mail. • Streets and avenues. • Giraffe and elephant. • Talking photo. • Photographic record and/or video recording. • Transects. • Communal map. • Collective drawing. • In-depth interview. • Focus group. • Life story. • Innovation story. • Brainstorming. • Polar opposites. • Dramatised story. • Audio forum. • Video debate. • Venn diagram. • Timeline. • Transect diagram. • Collective interview. • Focus group. • The most significant change. • Simulation game. • Sociodrama. • Pantomime. • Documentary video. • Short testimonies. • Illustrated posters. • Information sharing cards. • Public presentations. • Protagonist testimonies. • Roleplay. • Puppets. • Public testimony. 189 188 Define the social actors that will participate in the activity. Get to know the tools, their potential and limitations. 3 Adapt and redesign tools to fit the reality. 4 Test the tool. 5 Assess our level of preparation to “handle"the situation. 6 Facilitate the process: introduction, development and outcome. 7 What should we have in mind when we choose and use participatory tools? The most important thing is the participatory vocation. This implies being tolerant to error, and willing to explain again, willing to go over agreements and redesign work plans. Define the specific topic and aim that the tool will be used for. 1 2 also very important to identify the social actors involved as well as the use that we want to give to the data/information that emerges. Define which social actors will participate in the activity As we have said, in a participatory evaluation it is important to involve the highest level of diversity possible among the stakeholders: specialists in the topic, directors and mid-level managers in the intervention, programme or service users, local residents, high-ranking public officials, NGO representatives and civil society organisations, among others. How much a person will participate will depend on the topic chosen at each moment and the scope of the evaluation. Even if we wanted them to, not everyone involved or related to the intervention under evaluation is able to participate. When it comes to designing each activity it is necessary to first identify and define the group of stakeholders that will take part. This decision will also affect the choice of tool. According to the group or sector’s characteristics, the tool or game selected for each activity will differ or require certain adaptations. Designing an activity in which programme users will participate is not the same as designing one that will involve public officials and decision-makers. In addition to considering the different roles in the intervention evaluated, you also need to consider sex; age; the cultural, economic and political context where the activity will be carried out; the background and previous experience of the participant group; their level of schooling, etc. You have to keep in mind that the people that will be involved have different skills, motivations and time availabilities. This will also condition the type of tools to be used. We also need to know our role as facilitators, to motivate without pushing, reflecting with the group without conditioning conclusions, suggesting ideas without imposing our own and asking questions without suggesting answers. HERE YOU CAN WATCH THE VIDEO "TOOLS THAT FACILITATE MULTISTAKEHOLDER DECISION MAKING" 191 190 Get to know what tools we have, determine their potential and limitations As we know, there is a wide range of tools that can be used for democratic and participatory evaluations today. It is important to look at what tools are available to us and know which will encourage significant contributions from as many people as possible and not just from the most extroverted. Furthermore, it is necessary to recognise that the tools have limitations which vary according to the group, context, resources and time availability among other aspects. It is important to explore this properly in order to be informed on the particular characteristics of each tool, its potential and limitations. Often, especially with regard to the experiential activities, we abuse symbolic elements and, as a result, force conclusions or comparisons beyond what were generated during the group process. It is recommended that you do not expect more from the use of a tool than what it can provide. If not, as facilitators of the process, we could end up presenting our imaginings as if they were the group’s conclusions, rather than its own reflections and findings. Moreover, you have to avoid becoming a “fan” of a single tool that you know well. It is common to acquire a certain level of skill to “handle” certain tools, either because we know them well or because we use them effectively. It is important to understand that we cannot use the same tool in all circumstances and we must search for the most adequate for each case. Exploring new options, trying different ways or combinations will make us better when it comes to facilitating participatory and creative processes. The diversity of tools also enriches the process. Adapting and recreating the tools to fit the reality As we indicated in the previous point, the tools have limitations. Using them always in their “purest” form, following the book or handbook to the letter, without combining them with others or adapting them, will almost certainly make them less effective. This means that for each specific case, we need to redesign them according to the specific circumstance or situation that we wish to address, as well as the characteristics of the group or sector. It is important to use the toolkit creatively. The same tool can have multiple versions and procedures when used with different participants or in different situations or with different objectives. Knowing how to adapt them to a specific reality or context is the best way to use them. In the same way, combining them is recomended as this enables an ordered and systematic process of examination. As is true in most situations, with practice, we acquire skills to know what aspects of a tool we should change so that “it works” well. “It is quite a challenge to find tools that allow groups, people and beneficiaries to not only have an opinion, but also make decisions based on their own analysis. I think you have to be pretty sincere and not so romantic to think that everyone is going to participate and that we are going to hear all voices. It’s not like that! So it is important that the main stakeholders participate, I mean those that should gain things from the intervention. Because there too, from a rights-based approach, you have to examine unequal relationships, the lack of information and often, also, the lack of participation in the planning process for projects and programmes. So, it is very important to make use of an inclusive methodology right from the planning stages of an intervention”. Dagny Skarwan, participant in the participatory evaluation gathering in Quito. The participatory tools and techniques need to be redesigned according to the specific circumstance or situation that we are addressing, as well as the characteristics of the group or sector. HERE YOU CAN WATCH THE VIDEO "HOW TO ADAPT AND REDESIGN THE TOOLS" 193 192 Testing the tool, even if it is in a simulation of a real situation with the evaluator team, will allow us to tweak the instructions, clarify terms according to local language, estimate more precisely the time that will be spent on the exercise and know to what extent the tools captivate the participants and have the impact we had hoped for. It will also help us foresee undesirable situations that the tool could generate. Test the tool While it is not always possible, it is definitely recommendable to test the tool that we are going to use with a similar group and context. As we have said, we will not always find a tool that responds to the specific need and so we find ourselves obliged to adapt and redesign tools in line with the aims, topics and group that we are going to work with. BY WAY OF EXAMPLE At EvalParticipativa we have developed a participatory evaluation capacity training process. When it comes to addressing the conceptual issue in a participatory manner, we have designed two tools that, depending on participant experience and training, can be used independently or together. In the latter, they can provide a step-by-step approach to impart understanding on the topic. The tool “Defining Participatory Evaluation” is aimed at sociocultural activity coordinators and professionals that work in the field of managing and evaluating programmes and projects. The aim is for the participants to construct a definition of participatory evaluation by offering a set of words which may or may not be relevant to this concept. The debate based on various words facilitates reflection and prioritises the most relevant, resulting in a first point of contact with the “what?” of this evaluation approach. The tool “The Participatory Evaluation Playing Card” is aimed at going deeper into the topic. The purpose is to reflect on the main characteristics of this evaluation type and the role of the evaluator using a rights-based approach that is sensitive to the gender component and committed to citizen participation. A card game is used to prioritise, rule out and create slogans from a list of key characteristics and collectively construct a set of participatory evaluation principles that are taken as a reference for future evaluation practices. If the activity is going to be carried out with different groups, it is necessary to keep to one procedure or protocol. This will enable us to ensure that the information can be compared and the process is rigorous and credible. In these situations, it is necessary to design and test the tool for different scenarios and possible contexts. Assess how prepared we are to "handle" situations The capacity and experience of the person who will facilitate a specific activity during the evaluation process, will often limit the use of certain types of tools and oblige them to use others. This is because their lack of credible knowledge on a tool's structure, dynamic and possibilities will reduce its possibilities of success to such an extent that it could even be counterproductive to use it. It is not enough to just know about a tool and understand its procedure, duration and target group. You also need to know if you have the conditions to “manage” situations that the activity can generate, taking care of and respecting the sensitivity of the participants. We cannot forget that in the group activity coordination, just as in a participatory evaluation, we are dealing with people with diverse sensitivities, cultures, habits and preferences. We often use experiential activities that awaken emotional processes, conflicts or other situations that we are not equipped to handle. Tools such as in-depth interviews, life stories and the most significant change, require more than just openness and a willingness to listen. In certain groups, role play can, unintentionally, open some people up to ridicule or generate conflicts that we lack the skills to deal with. We should be aware that when people “throw their whole selves in”, they hand It is not enough to just know about a tool and understand how to use it, its duration and target group. We also need to know if we have the right conditions to “manage” situations that the activity can generate, taking care of and respecting the sensitivities of the participants. 195 194 There are topics that may require more than one tool. At these times, we should consider using a wide range of resources that can lead on one from the other rather than limit ourselves to rigid or prescribed ways of using them. 197 196 on and what procedures are to be followed. Clear, easy-to-follow guidelines or instructions can help the group work. The development. At this point, the group or subgroups carry out the activity mentioned, following the steps set out in a guide or set of instructions. It is important to accompany the process, respond to doubts and difficulties and ensure that most participants are getting involved in the activity proposed by the tool. There are times, depending on the tool selected, when it is useful to delegate people to monitor the work in each group to ensure that the established procedure is being followed and to keep a record of the main conclusions that they come to. It all depends on the type of gathering, the size of the group, how long the activity lasts and the human resources available. This moment of peer discussion and debate with regard to their ideas and perceptions on the intervention under evaluation, is usually the richest part of the process. It is very important to observe, document and be aware of what emerges from the evaluation’s key stakeholders. over a part of their life to the collective process. In this situation, we have to be extremely respectful and careful with what we generate, especially if we do not have the professional means and resources to handle and contain difficult situations. If we do not feel up to the task, it is better to choose a simpler tool even if the result and depth of analysis will not be as rich. Facilitating the process: introduction, development and outcome Except for when tools and activities are used for the purposes of presentation, group division or motivation, with the aim of achieving an appropriate atmosphere for the task, organising group work or maintaining participant interest, we suggest following these steps when using the tools: introduction, development and outcome. The introduction. It is important to introduce the topic to the participants and encourage them to participate fully in the process. The facilitator should ensure that all participants know what is going This consists in a first collective sorting of the information (quantitative or qualitative) that emerges from the group. We generate questions to explain what has happened. It is an exercise in abstracting findings and making sense of them. Using the data and opinions that come from the activity, we try to construct useful recommendations at different levels. This will vary according to the case and tool used. We try to identify the main conclusions on the topic and think of a way to document and communicate them. 1. RECOVERING THE EXPERIENCE 2. INFORMATION ANALYSIS 3. ANALYSIS OF IMPLICATIONS 4. ENDING THE ACTIVITY Auditory tools Audiovisual tools Experiential tools Written or graphic tools What did we hear? What did we see? What did we hear? What happened? What did we feel? What did we read? What did we present? What do we think about the things we have seen, said or experienced? Why do we think this way? Are there different perspectives or is there a consensus? What are the implications of what we think about the things we have seen, said or experienced? Is it possible to generate recommendations based on these implications? What have we learned with this activity? What new knowledge can we offer from it? What can we do so that the conclusions from this topic and the recommendations generated are useful for the programme and other people? Recovering conclusions, agreements and bringing the activity to an end. 199 198 set of criteria and recommendations to bear in mind when we select and use different instruments. In this final section we want to conclude with a set of challenges that those of us who desire to facilitate participatory evaluation face. First, we should acknowledge that the instruments cannot do it all. Just as important as the tools, (if not more important), is the participatory vocation that is expressed in a tolerance for error, the willingness to repeat explanations, go over agreements and redesign work plans, etc. In this sense, it is important to be clear about our role as external agents, even if we are also participants, in the whole process. This means recognising that our role is to motivate without pushing, reflect with the group without conditioning conclusions, suggest ideas without imposing our own and ask questions without suggesting answers. Secondly, we believe it is necessary to be imaginative and creative when adapting or designing instruments. Organisations will benefit infinitely and take more ownership of tools when these instruments bility. As with any other tool, you have to know what it can be used for and how and when it should be used. The most important thing is that the facilitator has an open and receptive attitude. Going beyond the tool, it is essential to keep in mind the deep significance that we place on evaluations being participatory, based on equal relationships and having spaces for dialogue and knowledge exchange. And, although we might not want to admit it, we have to accept that “there are no magic solutions”. Coordinating sociocultural group activity and our experiences in the field of participatory evaluation reveal that intuition often plays a key role. The outcome. This refers to the moment when conclusions and agreements are agreed upon, the activity is brought to a close and products that we hoped to achieve come to fruition. As we have seen, one tool can be used for different purposes: to analyse a an intervention, reflect on achievements and difficulties, identify lessons learned, achieve agreements and consensus, identify points of disagreement and be aware of the diversity of opinions, etc. In some situations, this is the moment to bring the topic to a close. There are different ways of doing this depending on the way each tool works, but it generally involves a plenary moment that all the participants attend. Once the group work is finalised (tool development), we can move onto the next steps and questions featured in the table on the next page. By way of conclusion, we could say that a tool is not good or bad in itself, as it all depends on the specific topic that we want to address, the objective that we propose, the context and the participant characteristics. We believe that choosing, replicating and adapting participatory tools brings with it great responsi- 4. CONCLUSIONS are created with specific participants, challenges, spaces and contexts in mind, as well as heeding evaluation requirements. Thirdly, we should recognise that there is a credible and rigorous process behind every evaluation that is successful in producing results and recommendations that are put to use. We should study and find out more about tools that can be used, test them out in pilot schemes and be sure that their use and results will be the most robust possible. It is important to guarantee spaces for genuine and equal participation, with transparent selection processes and participants that are representative of the whole. Likewise, we have to be explicit with data analysis and systematisation criteria. Monitoring the process, requesting and receiving supervision from a mentor, sharing and validating partial results to legitimise findings and conclusions are aspects that we should not neglect if we want to conduct a high quality evaluation. Finally, when it comes to adopting and using participatory tools, we should be In this chapter, we have developed some basic notions around participatory evaluation methodology and tools. We have constructed a generic classification of the tools available that brings together the tool’s method with the purpose we can use it for. Finally, we have shared a 201 200 aware of different problems that could arise. This means being willing to reduce tensions, offer additional information, dialogue with all parties, encourage the participation of those with less experience, create spaces for dialogue and promote consensus building. 202 Bibliography 205 Arnstein, Sherry R. 1969. "A ladder of citizen participation". Journal of the American Institute of Planners 35(4): 216–224. Aubel, Judi. 2000. Manual de Evaluación Participativa del Programa. Involucrando a los participantes del programa en el proceso de evaluación. Madrid: Nueva Imprenta. Bamberger, Michael, Marco Segone & Florencia Tateossian. 2017. "Evaluando los Objetivos de Desarrollo Sostenible con el enfoque 'Que Nadie se Quede Atrás' a través de evaluaciones centradas en la equidad y con perspectiva de género". New York: UN Women, IOCE & EvalGender. Cendales, Lola & Alfonso Torres. 2014. "La sistematización como experiencia investigativa y formativa". Revista Arcadia, no. 23: 29–38. Chambers, Robert. 1992. "Diagnósticos rurales participativos: pasado, presente y futuro". Bósques, Árboles y Comunidades Rurales 15/16: 4–9. Chambers, Robert. 2003. Whose Reality Counts? Putting the first last (Revised Edition). London: Intermediate Technology. Chavez-Tafur, Jorge. 2006. Aprender de la experiencia. Una metodología para la sistematización. Lima: Fundación ILEIA / Asociación ETC Andes. Chelimsky, Eleanor. 1998. "The role of experience in formulating theories of evaluation practices". American Journal of Evaluation 19(1): 35–55. 207 206 Chouinard, Jill Anne & Peter Milley. 2018. "Uncovering the mysteries of inclusion: Empirical and methodological possibilities in participatory evaluation in an international context". Evaluation and program planning, no. 67: 70–78. Clark, Mari & Rolf Sartorius. 2004. Seguimiento y evaluación: instrumentos, métodos y enfoques. Washington D.C.: World Bank. Cohen, Ernesto & Rolando Franco. 1988. Evaluación de proyectos sociales. Santiago de Chile: ILPES/ONU & CIDES/OEA. Cornwall, Andrea. 2008. "Unpacking “Participation”. Models, meanings and practices". Community Development Journal 43(3): 269–283. Coupal, Françoise. 2000. Seguimiento y evaluación participativos basados en los resultados. Mosaic.net International, Inc. Cousins, J. Bradley & Jill Anne Chouinard (Eds). 2012. Participatory evaluation up close: an integration of research-based knowledge. Charlotte, NC: Information Age Publishing. Cousins, J. Bradley, Elizabeth Whitmore & Lyn M. Shulha. 2013. "Arguments for a common set of principles for collaborative inquiry in evaluation". American Journal of Evaluation 34(1): 7–22. Daigneault, P.-M., & Jacob, S. 2009. Toward Accurate Measurement of Participation: Rethinking the Conceptualization and Operationalization of Participatory Evaluation. American Journal of Evaluation 30(3), 330-348. Eggens, Laura & Jorge Chavez-Tafur. 2019. Facilitating experience capitalization: a guidebook. Wageningen: Technical Centre for Agricultural and Rural Cooperation, CTA. Eizaguirre, Marlen, Gorka Urrutia & Carlos Askunze. 2004. La sistematización. Una nueva mirada a nuestras prácticas. Bilbao: Hegoa, Alboa. Instituto de derechos humanos Pedro Arrupe. Espinosa Fajardo, Julia. 2019. "La evaluación de programas y políticas de infancia: aproximaciones clásicas y emergentes". Infancia y bienestar. Una apuesta política por las capacidades y los cuidados, edited by Lina Gálvez-Muñoz & Lucía del Moral-Espín. Seville: Editorial Deculturas. Everitt, Angela & Pauline Hardiker. 1996. Evaluating for good practice. London: Macmillan. Fals Borda, Orlando. 2009. "Experiencias teórico-prácticas". In Una sociología sentipensante para América Latina, 303–365. Bogotá: CLACSO. Faúndez Meléndez, Alejandra & Marisa Weinstein. 2013. "Mapeo de Sistemas Nacionales de Evaluación en América Latina y la inclusión del enfoque de igualdad de género". Panama: UN Women. Fetterman, David M. 1994. "Empowerment evaluation". American Journal of Evaluation 15(1): 1–15. Fetterman, David M. 2015. "Empowerment Evaluation and action research: a convergence of values, principles, and purpose". The SAGE Handbook of Action Research, 83–89. London: SAGE. Fetterman, David, Liliana Rodríguez-Campos, Abraham Wandersman & Rita Goldfarb O’Sullivan. 2014. "Collaborative, Participatory, and Empowerment Evaluation". American Journal of Evaluation 35(1): 144–148. Freire, Paulo. 1972. Pedagogía del oprimido. Buenos Aires: Tierra Nueva y Siglo XXI. Gadotti, Moacir. 2017. "The global impact of Freire’s pedagogy". New Directions for Evaluation 2017 (155): 17–30. García Sánchez, Bárbara Yadira & Javier Guerrero Barón. 2012. Núcleos de Educación Social-NES. Bogotá: Universidad Distrital Francisco José de Caldas. Geilfus, Frans. 2002. 80 Herramientas para el Desarrollo Participativo. San José de Costa Rica: IICA. González Gómez, Lara. 2005. La evaluación en la gestión de proyectos y programas de desarrollo. Una propuesta integradora en agentes, modelos y herramientas. Vitoria-Gasteiz: Servicio Central de Publicaciones del Gobierno Vasco. Gordijn, Femke, Jan Helder & Natalia Eernstman. 2012. Reflection methods: tools to make learning more explicit. Wageningen: Centre for Development Innovation. 2020 will forever be marked in our memories as the year that the COVID-19 pandemic irrupted and, in turn, disrupted our lives. We experienced the depth of our global connection and interdependence, and the way that our realities are deeply interwoven. This participatory evaluation handbook was birthed and developed in this context as a message from the field of evaluation to express the idea that we can only immunise ourselves from the limits of self-sufficiency and fragility by recovering a plurality of participant voices and perspectives. To this effect, we explore the reality and potential of participatory evaluation in Latin America, the phases of its development in practice, the key role of those who facilitate these processes, and the purpose and place of participatory tools and instruments in this evaluation approach. Guided by a transformative way of perceiving reality, this handbook presents participatory evaluation as a community of dialogue that collectively harvests the fruits of a project or programme and then sows these lessons back into present and future realities. ISBN 978-3-96126-143-7 9 783961 261437 BOOK-TRAILER Sowing Harvesting &