Service-learning in the Spanish university system. A study focused on course evaluation

Service-learning in the Spanish university system. A study focused on course evaluation María del Mar Lorenzo Moledo Universidad de Santiago de Compostela, España mail: mdelmar.lor[email protected] ORCID: https://orcid.org/0000-0002-2768-3329 Covadonga Ruíz de Miguel Universidad Complutense de Madrid, España mail: co[email protected] ORCID: https://orcid.org/0000-0002-6941-5402 Elena Arbués Radigales Universidad de Navarra, España mail: [email protected] ORCID: https://orcid.org/0000-0001-5872-830X María Jesús Martínez Usarralde Universidad de Valencia, España mail: [email protected] ORCID: https://orcid.org/0000-0001-6777-3399 Mariana Buenestado Fernández Universidad de Córdoba mail: [email protected] ORCID: https://orcid.org/0000-0002-3242-5332 Ígor Mella Núñez Universidad de Santiago de Compostela, España mail: igor[email protected] ORCID: https://orcid.org/0000-0002-3242-5332 ABSTRACT In this article, 41 Service-Learning (SL) courses have been evaluated in 5 Spanish universities (4 public and 1 private), in order to determine whether they are SL initiatives or other type of educational practices. These experiences, which were implemented during the 2015/2016 academic year, covered degree programs from the five knowledge areas. Most of them involved Bachelor’s degree students and only one of them Master’s degree students. A qualitative methodology has been used, focused on content analysis and based on evaluations made by four experts. The analysis of the experiences points out an encouraging course of action for this methodology in the Spanish university system, showing a certain change in the teaching culture, as they all meet the basic requirements which, according to international literature, must be present in service-learning projects and are also considered as criteria for assessing quality in such projects. Finally, a three-way action is presented, to open new prospects of educational innovation and social responsibility through Service-Learning possibilities in Higher Education. Keywords: service-learning, higher education, courses, evaluation, teaching culture. Aprendizaje-servicio en el sistema universitario español. Un estudio enfocado en la evaluación de los proyectos RESUMEN En este artículo se han evaluado 41 proyectos de aprendizaje-servicio (ApS) en 5 universidades españolas (4 públicas y 1 privada), con el objetivo de delimitar si se trata de experiencias de ApS o de otro tipo de prácticas educativas. Estas experiencias, que fueron implementadas en el curso académico 2015/2016, se distribuyen en titulaciones de las cinco áreas de conocimiento. Mayoritariamente implican a alumnado de Grado y únicamente en una participan también los de Máster. Se hace uso de una metodología cualitativa, enfocada en el análisis de contenido y basada en las evaluaciones realizadas por cuatro expertos. El análisis de las experiencias señala un curso de acción alentador para esta metodología en el sistema universitario español, que muestra un cierto cambio en la cultura docente, pues todas ellas cumplen con los requisitos básicos que, según la literatura internacional, han de estar presentes en los proyectos de aprendizaje-servicio y que, además, son considerados como criterios para medir la calidad en los mismos. Finalmente, se presenta una triple acción para abrir nuevas perspectivas de innovación educativa y responsabilidad social a través de las posibilidades del aprendizaje-servicio en la educación superior. Palabras clave: aprendizaje-servicio, educación superior, proyectos, evaluación, cultura docente. ISSN: 0210-2773 DOI: https://doi.org/10.17811/rifie.49.4.2020.353-362 Volumen 49, número 4, octubre-diciembre, 2020/págs. 353-362 María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 354 Introduction Nowadays, the strengthening of student-centered learning approaches has substantially modified the role played by university teaching staff, who abandon their role as transmitters of information and disciplinary knowledge in order to act as mediators and facilitators of the autonomous learning of their students (Gargallo, Sahuquillo, Verde, & Almerich, 2018). In this regard, university professors, concerned with innovation in their classrooms and with their students, and considering the changes that occur in the university context, look for new methodologies or design experiences which, to a great extent, are based on intuition or teaching experience rather than on a strong pedagogical background. At other times, assisted by university institutions, professors use methodologies which seem to become widespread, but in many cases, are at risk of becoming some transient fashion. Teaching staff have become a key element in any attempt at pedagogical innovation, regardless of the reason for this renewal process, and the support they receive is essential, especially with respect to the training which contributes to the acquisition and development of teaching skills (Lorenzo, Ferraces, Pérez, & Naval, 2019). One of these methodologies is service-learning, which should not be confused with university volunteering or field education (Furco, 1996). Over the last few decades, service-learning has increased its presence in higher education institutions as a result of the advent of a scholarship of engagement, understood as a way to “link theory and practice, cognitive and affective learning, and colleges with communities” (Butin, 2006, p. 473). In this regard, this scholarship of engagement demands institutional changes that strengthen links between mission and practice, in which case pedagogies like service-learning will be institutionalized (Santos Rego & Lorenzo, 2018). In the Spanish university system, there are professors who carry out experiences that could be conceptualized as SL, even without being aware of it. Following this line of research, Puig, Batlle, Bosch, & Palos (2007) suggested that beginning with similar experiences and making the necessary modifications represented one of the procedures by which this methodology can be introduced into higher education. In other cases, a quick evaluation and reorganization would allow for certain initiatives to become SL courses, especially if they are designed through different principles of good practice for service-learning (Santos Rego & Lorenzo, 2018). There are also activities defined in such a way that one cannot genuinely interpret them as SL, although no one is questioning their pedagogical value. The most important thing is to clarify what distinguishes SL from other methodologies (Furco, 1996). Thus, service-learning “involves the integration of academic material, relevant service activities, and critical reflection and is built on reciprocal partnerships that engage students, faculty/staff, and community members, to achieve academic, civic, and personal learning objectives as well as to advance public purposes” (Bringle & Clayton, 2012, p. 105). It is important, therefore, to identify what is, what is not, or what appears to be, SL in the university context. This is the objective that has led us to evaluate different experiences that are under development in 5 Spanish universities, in order to clarify whether they can be considered SL courses, or rather practices which may become good SL projects, as certain aspects of their design are elaborated on. In this regard, our analysis has focused first on the elements that define an SL course, next on the experiences selected, while paying attention to the conceptualization of service and learning in each of the steps. To this end, the level of development of different aspects related to learning (linked learning activities, linked curriculum contents, results, and assessment) and service (need, recipients, objectives, service definition, results, and assessment) is evaluated. Without these elements, there would be no pedagogical proposal anticipated by their connection (Rubio, Puig, Martín, & Palos, 2015). More specifically, this article is aimed at analyzing the degree to which the experiences identified as SL meet the main pedagogical requirements of this methodology, which differentiates it from other activities such as volunteering or field education, with which it has been confused on numerous occasions (Furco, 1996). This is the first step to guarantee an adequate institutional progress of this educational strategy in our universities (Santos Rego & Lorenzo, 2018). Theoretical Overview Learning in the context of service-learning In service-learning courses, the pedagogical-didactic purpose should not be omitted, since one could make the mistake of giving priority to service, in which case we would be discussing other types of practices. In other words, it would keep us away from a practice clearly linked to service-learning and would bring us closer to what is traditionally known as volunteering (Sotelino, Santos Rego & Lorenzo, 2016). The learning derived from SL can be grouped into dimensions such as civic skills (Bringle & Clayton, 2012), academic performance (Jameson, Clayton, & Ash, 2013) or skills for employability (Naval & Arbués, 2016). This is useful when promoting learning of basic knowledge and skills related to the contents of academic subjects, and it will be even more useful when it is understood, designed and implemented in order to achieve higher levels of learning (Jameson et al., 2013). It is not surprising that educators use it as a suggestive tool in their work, insofar as they can see SL as a means for the development of disciplinary learning, social skills and values, structured in a single project (Santos Rego, Sotelino & Lorenzo, 2015). It could, therefore, be understood that service-learning promotes generative learning in a great number of students (Wittrock, 1974), by adding new behaviors, knowledge and skills to the educational development, whereas students are involved in their own learning and take an active role in the process, raising more questions, contributing to group discussions and integrating ideas from different sources. By getting involved in this way, students interpret the incoming information and provide meaning to it, besides strengthening their self-efficacy beliefs (Anderman, 2010). This can be explained by the generative learning processes, which involve metacognitive processes, because they inevitably imply reflection on action. It is well-known that the typology of learning developed within SL is, primarily social-constructivistic in its orientation, as students have to establish functional connections with previously acquired formal knowledge in their social and training dynamics (Eyler & Giles, 1999). The effectiveness of the metacognitive process will go hand in hand with the students’ involvement and critical reflection, without disregarding their role as professionals undergoing training (Cooks & Scharrer, 2006). What is service in the context of service-learning? The idea of ‘service’ in this methodology is simply the opportunity given to students to strengthen their own learning by implementing their theoretical knowledge and applying it María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 355 to real situations. Evidence shows that service is also linked to the construction of civic virtues that, in the public sphere of civil society, favor the participation and active involvement of young people in tasks that are beneficial to the community (Bringle & Clayton, 2012).This is aimed at strengthening an educational path towards greater commitment and social responsibility of the university (Torrego, Martínez, & Sonlleva, 2018). Moreover, it is known that service presents itself as an opportunity in the context of personal experience, whose greatest challenge is none other than that of knowing oneself capable (motivation of self-efficacy) of putting into practice their theoretical cognitive knowledge. Since service is part of a concerted plan, it goes without saying that its dynamics of involvement must also include cooperation and shared leadership qualities. With a well-designed service, correctly embedded in social performance and/or professional partnership, and responding to an objectifiable need, the question remains as to whether there are signals within competences and skills which could be transferable or, simply, could clear the channels of employability. The reader must bear in mind that many civic skills developed through service-learning (communication, cooperative interaction, critical thinking, diversity management, problem solving) are also valued on the labor market, as shown by different employers’ reports on a wide range of business activities. Implementing a service-learning course requires the design of strategies that allow students to clarify the reciprocal links between academic learning and community service through a continuous reflection. If the purpose is to optimize the benefits of this educational practice, the service must be characterized by a high level of quality, which, according to Wilczenski and Coomey (2007), should respond to a current and socially acknowledged need, an adaptation to students’ characteristics and demands, as well as a planning aimed at achieving significant results for both students and the community. What is reflection in the context of service-learning? It has always been accepted that reflection is a key component of service-learning. However, solid justification should be provided, since this is not a cursory deliberation to contend with. This obviously refers to a genuinely critical reflection that creates solid links between service and learning. In a pedagogical context, one can note Dewey’s influence and his pragmatic reflection-on-action discourse, a synthesis of an epistemic loop, in whose dynamics learning is represented as a holistic process, assuming students’ continuous interaction with their world. It is not surprising that reflection is understood as the possibility of turning experience into learning (Boud, 2001). Without reflection as a part of critical thinking, it would be difficult for any university student to make progress towards a type of metacognitive empowerment, which is synonymous with genuine intellectual development in Higher Education (Ross & Gautreaux, 2018). It is worth mentioning that the study developed by Eyler and Giles (1999), already a classic in the field, found that with more rigorous reflection in SL, greater learning in general is achieved and, more specifically the academic results allow for a deeper understanding of a subject, better analysis and problem solving, as well as openness to new ideas and critical thinking skills. Therefore, in addition to the benefits associated with critical incidents, as a result of the evaluative functionality they provide for the development of service-learning in a given context of academic and social work, one should also consider the usefulness of journals which, precisely because they need a reflexive structure, may be considered products that strengthen the learning processes in this experiential pedagogy. As expressed by Deeley (2015), journal writing can follow the same pattern as an experiential learning cycle, which requires reflecting on past events and re-evaluating them for learning that will serve as a basis for future action. However, facing a central component of this philosophy of educational action should not prevent a quality reflection from being a significant challenge in the SL context. This, according to Ash, Clayton, and Atkinson (2005), is due, in part, to the difficulty in developing and implementing effective structures to guide it, and significant strategies to evaluate the results of their associated learning. Planning in the context of service-learning Students’ motivation is more likely to materialize when, under normal conditions, there is involvement between teachers and students in an SL course (Simons & Clearly, 2006). However, what is essential is that the results also have an impact on a greater integration between academic life and socio-professional reality. Proper course planning is important for such integration to occur, avoiding at any cost pronounced imbalances between service and learning (Lim & Bloomquist, 2015). In this sense, service-learning projects have to be planned with a strong conviction and a clear focus on learning, integrated with an equally clear focus on community (Bringle & Clayton, 2012). The first step is to present to the working group a sufficiently attractive proposal for a plan of action. Puig et al. (2007) recommended, based on real interests, taking into account previous experiences, which would trigger the intervention of people from outside the university, directly motivating group leaders, as well as allowing for the sharing of experiences with other partners, thereby making the most of situations in which a topic gains in relevance, or even using audiovisual resources. Once this phase has been completed, it is appropriate to define the course and to analyze it in the environment. This is the moment to clarify ideas through the classic interrogative technique: what, who (or for whom), how (organization, resources, time, costs…), when (timing), where (specific scope) and why. Answering these questions will define our objectives and how to achieve them. Individual responsibilities are relevant in this case, as well as commissions or teams responsible for specific tasks (Santos Rego et al., 2015). Before putting it into practice, it would be useful to make an evaluation of the learning activities that have already been developed, including the values that have emerged thereof. In this way the project could be re-adapted, if necessary, by improving its instruments. In order to avoid common mistakes, the planning of an SL course has to systematize the expected learning activities, not only the potential answers to the perceived needs. This suggests the creation of areas in which a deep and contextualized reflection on the meaning and scope of the experience should be promoted. Method Sample One of the most difficult aspects about the study was identifying and having access to faculty who are working with SL methodology or developing activities that could be conceptualized as such, since universities generally lack a record of good María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 356 practices subject to consultation. This was the reason why accidental or casual sampling had to be used. In order to obtain the sample of service-learning experiences that were being implemented in five of the Spanish universities participating in the research supporting this study, several mechanisms were used: - Semi-structured interviews. These interviews were administered to university deans (42 in total), with the aim of detecting the presence of SL practices. - Information from university centers, units or offices involved in teacher training. - Analysis of the answers to an item on professors’ use of SL, which is included in the Survey on Faculty’s Teaching Practice and Attitude towards Innovation (CUPAIN). - Analysis of faculty’s participation in the conferences on university SL in Spain, and the membership in the University Network of service-learning – (U)SL. - Direct knowledge of some experiences of research team members. Although the Experience Report was sent to 61 teachers (invited sample), a total of 44 teachers responded (accepting sample) indicating that they were implementing experiences/programs defined as service-learning. However, after a review and filtering process, three of the received experiences were eliminated, as they did not meet the requirements of our study. These three projects were discarded for being volunteer rather than service-learning activities, as acknowledged by the teachers who coordinated them (Santos Rego et al., 2015). The final sample consisting of 41 experiences (accepting sample) was distributed to participating universities, as shown in Table 1. Table 1 Sample of experiences University Experiences Complutense University of Madrid 1 University of A Coruña 5 University of Navarra 14 University of Santiago de Compostela 11 University of Valencia 10 The experiences that were the subject of close scrutiny are being developed in programs in the area of Social and Legal Sciences (63.4%), followed at a certain distance by those in Health Sciences (17.1%), Experimental Sciences (9.8%) and Technology (7.3%). Arts and Humanities programs barely reached 3%. All SL courses were developed within Bachelor’s degree programs, except for one, in which Master’s students were involved. Professors showed a clear preference for introducing the methodology in compulsory subjects (79.5%), during the final two years (48.8%) of the degree program. These were experiences whose implementation has already been underway, inferred from the fact that 34.4% have done it for three or more academic years in a row. Measuring instrument The reports used to record and evaluate the service-learning experiences implemented during the 2015-16 academic year in these universities were elaborated from an adaptation of the instrument designed by Villa (2013) to analyze the way in which SL experiences were developed in Latin American universities (Santos Rego & Lorenzo, 2018). The instrument was implemented in 2016, at the end of the academic year of reference. In the report, professors were asked for information regarding the SL courses which they were developing. The report consisted of 28 closed-ended and open-ended questions grouped around six dimensions: data from the university institution (2 items); identification of the course/subject in which the program/experience was being developed (4 items); program/experience identification (8 items); community partners (4 items); evaluation (6 items); and program/experience projection (4 items). The validation process of the instrument was carried out based on expert judgment, through a two-stage approach: an internal phase and an external phase. The initial version of the instrument was presented to the teams of two of the universities involved in the research project. Once the relevant considerations were included, this version was evaluated by the teams of two other universities. With all the contributions, a new version of the report was prepared, which was sent, along with a correction template, to three experts in the implementation of service-learning programs at university level. In the template, they were asked about their opinion on different aspects for each of the dimensions making up the instrument: items to be added or removed, recommendation about their location and general observations about the dimension; in addition, they were required a general assessment of the report. Based on these estimates, the final instrument was developed. Procedure Once professors responsible of SL courses were identified, the report was sent to them (if there were several professors in the same subject, it would be sent to the coordinator) via email, and after filling it out, it was returned the same way. This form of distribution was chosen because professors pointed to a lack of time to meet with us. Our team is obviously aware that a semi-structured interview would provide more details on the matter. Data analysis For the analysis of the reports, and given the nature of the data, the IBM-SPSS.20 statistical package was used. In addition, the team designed, based on the information collected in the reports, a self-evaluation rubric of SL, taking into account other already existing rubrics (Campo, 2015; Rubio et al., 2015; Shumer, 2003). Our motivation was the possibility of performing a content analysis (Krippendorff, 2004) in order to assess the development level of each of the pedagogical elements of SL. The score did not matter as much as distinguishing between what works correctly and what needs to be analyzed in greater depth in order to improve the project. The section was structured around five blocks that identify a complete SL course: Identification of experience; Service performed; Learning activities achieved; Evaluation; and Overall evaluation of the experience. All the blocks are made up of a series of sub-dimensions that explain SL. Each of them is assigned a value according to the level or degree in which the participants adopt and apply the SL philosophy. Four levels were established: Advanced (value 4); In development (value 3); Incipient (value 2); and Disaggregated (value 1). In short, the section was initially designed to allow different lines of reading and interpretation which should be integrated María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 357 and interpreted together to obtain the information that allows us to draw conclusions about the analyzed experiences and to establish relevant lines of action. The 41 experiences, along with the section, were distributed to four (4) experts, who were asked to examine them individually. Subsequently, they met to analyze their evaluations and discuss discrepancies, calculating an agreement/disagreement index. In order to evaluate the degree of agreement among the four evaluators, Krippendorff’s (2004) alpha reliability coefficient was employed, which is very useful when the choices are dichotomous (1=presence and 0=absence). This coefficient, expressed in percentages of agreement, is reflected in the analysis of consistency between items where there is agreement. Before discussing the findings, it should be noted that the consulted sources are all in agreement (Hopkins & Hermann, 1977; Krippendorff, 2004) on the interpretation of different levels of reliability. They all explained that only the cases exceeding .80 should be considered, although, for provisional and cautious conclusions, the reliability should be ranging from .67 to .80. Table 2 shows the reliability percentages, in terms of agreement, for each of the participating universities in the four factors that characterize the Curriculum dimension (learning, content, results and assessment), and in the six factors referring to the Service dimension (need, recipients, objectives, definition of service, results, assessment). Table 2 Reliability of the data gathered on factors of the Curriculum and Service dimensions for each of the participating universities Number of experiences UCM UV UDC UNAV USC 1 10 5 14 11 Curriculum dimension Learning 1.00 1.00 .80 1.00 .87 Content 1.00 .85 .80 1.00 .87 Results 1.00 .95 .90 .97 .96 Assessment 1.00 1.00 1.00 1.00 1.00 Service Dimension Need 1.00 1.00 .90 .90 .79 Recipient 1.00 1.00 .70 .97 .87 Objectives 1.00 .80 .80 .90 .97 Definition of service 1.00 .90 .40 1.00 .92 Results 1.00 1.00 1.00 1.00 .95 Assessment 1.00 1.00 .80 1.00 1.00 With regard to the Curriculum dimension, appropriate indices were found in all categories analyzed by the four evaluators. In only two cases, the percentage lies at the lower limit (.80). These are the Learning and Content dimensions, both in the same university (UDC). In the first case, the lack of agreement occurs in experiences 1 and 3. Specifically, in experience 1, three of the experts stated that the linked learning activities were superficially detailed and explained (in development), but for the forth expert they were simply listed, with no explanation (incipient), whereas in experience 3, most thought that it was incipient, and one of the experts considered it in development. In the factor referring to the Content dimension, the discrepancy is found in experience 5, as experts agreed that the curriculum contents were listed, but not really explained. In the Service dimension, other divergences were detected. In the factor referring to Need, one of the universities, specifically that of Santiago de Compostela (USC), presented a non-significant agreement index (.79). The problem was detected in experience 11, where two of the experts observed that professors explained the initial need for the service, but they reached it through a unilateral analysis; the other two experts noted that only the need was justified, but the manner in which it was discovered was not specified. Similarly, in another case (UDC), a non-significant index emerged in the factor referring to Recipients, which affected experience 4. The assessments were divided between those interpreting that recipients were accurately and clearly described, and those which were considered as merely general; in the same experience, the obtained index was the lowest (.40) of those recorded in the factor Description of service (two of the experts considered the description to be correct, but not detailed enough, and for the other two the description was inaccurate or incomprehensible). In the remaining factors, the agreement was within appropriate limits of significance, as shown in the Table 2. Results Evaluation of learning in the context of service-learning First, it should be noted that professors seemed to be quite aware of the learning activities to be linked to the SL experience. Despite the fact that only two of them adequately detailed and explained the learning activities they were trying to pursue, in 32 cases they were clearly set out, either by mentioning them (22), or by explaining them superficially (10), as shown in the Table 3. When these projects pointed to learning, they referred to the acquisition and development of knowledge, skills, abilities, behaviors and values through practice (Rubio et al., 2015). Specifically, professors referred to the specific skills of the course they taught, but also to generic skills, especially interpersonal and systemic, required by the labor market (Naval & Arbués, 2016). However, professors did not identify the contents of the academic curriculum associated with the experience, although many of them (17) reported positive learning outcomes. The reasons professors found it difficult to identify the contents linked to the experience may be due to the fact that they had worked on competences, rather than on contents. The fact that the courses had been planned for the entire academic year and they were taught for a significant part of the semester should also not be ruled out. They are aware that students’ knowledge is reflected in the service development (knowledge is necessary for their effective development), even if in return, they do not prioritize knowledge as much as competences, skills or abilities. This is what Rubio et al. (2015) called ‘useful learning’. In general, the four experts clearly stated that they would not describe any of the experiences as advanced, but as incipient (24) or in development (10). Seven (7) of them are described seamlessly, as disaggregated, which suggests that they may not really be SL experiences. María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 358 Table 3 Assessment of the Learning dimension Curriculum dimension Level of development Experts’ total agreement Disagree Linked learning activities Advanced 2 4 In development 10 Incipient 22 Disaggregated 3 Linked curriculum contents Advanced – 5 In development 4 Incipient 9 Disaggregated 23 Results Advanced 4 4 In development 13 Incipient 2 Disaggregated 17 General assessment Advanced – – In development 10 Incipient 24 Disaggregated 7 In any case, the data resulting for this part of the learning process can be described as positive, although it is clear that professors need further work to systematize and be able to link the contents of the curriculum with their SL course. The evaluation of service in the context of service-learning A number of issues are addressed together in the service provided, although civic education (58.5%), justice and social equity (41.5%), and social inclusion (41.5%) are the most common. The topics covered by the service are closely related to the activity of the community partners. It is well known that for the development of any SL course, it is essential to find partners whose role is vital if teachers aspire to carry out quality experiences. At university level, the collaboration pillar consisted of different community organizations (85.4%), 28.6% stating that they worked with public sector organizations, and a similar percentage (21.4%) with private sector organizations (multiplechoice answer). Within the organizations, the public entities such as hospitals, educational institutions and foundations stand out. When analyzing the service as a whole, the experts reached a number of conclusions (Table 4). The most important thing in terms of service refers to the identification of the need or needs to be responded to. Basically, the evaluators agreed that teachers working with this methodology do not spend time analyzing or reflecting on the needs that drive the (disaggregated) service. In 8 cases the need was justified, but there was no specification on how it was discovered (incipient). Three of the experiences explained the initial need, although its identification was the result of a unilateral analysis performed by the teacher. The highest level of assessment was applied to only one of the courses, as it did not detailed the need only, but its identification was possible due to the participation of several actors. The fact that teachers neither spend a lot of time, nor do they involve students in this work –which determines the motivation and commitment to the course– may have to do with two fundamental variables: the academic calendar or, in other words, the short duration of the quarter; the lack of incitement to reflection, and the detection of needs by teachers. Given their link with different social organizations, they obviously get a first-hand view of the situation and this would influence the low level of processing conditions previously mentioned (Lorenzo et al., 2019). The service has certainly a number of recipients. In this case, the agreement between experts when assessing this factor should also be noted. Teachers described in a very diffuse manner who were the recipients of the service (16 cases); in twelve of them, the experiences were described, but in a generic fashion (in development), whereas in seven of them (7) the description was precise and clear (advanced). The large number of professors who were not clear on who were the recipients of the service can be explained because, sometimes, there are services that do not require a direct relationship with the people targeted. They keep in mind their partners, but their recipients, not as much. Regarding the objectives sought to be achieved through the service, in 5 experiences they were clearly set out, and in 12 experiences it was done partially; however, in 11 cases they were not clearly presented, and in six (6) of them, they were overlooked. It is not surprising that teachers hardly mentioned this aspect when they were required to present the results of the service. This is an incentive to reflect on the meaning and scope of the service, since it seems to be conceptualized as a means that serves the student’s learning as a teleologically unique activity. In other words, it gives the impression that there is no reciprocity in the relationship, a fact that agrees with situations in which the partner takes a more instrumental role, rather than a properly active role in the pedagogical framework of the course. This is a topic that obviously needs more attention and further study. The most clearly defined element of the service is the description of the work performed by students, since in 16 experiences, the service was explained correctly, but not detailed enough, and in 17 the description was inaccurate. In any case, the experts considered, following the levels proposed by Rubio et al. (2015), that the service performed was complex, especially in view of the long duration of the service. Most of the experiences were developed throughout the semester, which allowed students to acquire experience and skills in the performance of tasks with a certain level of difficulty, and whose implementation required a great engagement and involvement. Finally, when making a general assessment of this dimension, the four experts almost unanimously stated that they would not describe any of the experiences as advanced, but as incipient (23) or in development (5); moreover, the fact that five of them were defined as disaggregated suggests again that they were not really SL experiences. María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 359 Table 4 Assessment of the Service dimension Service dimension Level of development Experts’ total agreement Disagree Need Advanced 1 5 In development 3 Incipient 8 Disaggregated 24 Recipients Advanced 7 4 In development 12 Incipient 16 Disaggregated 2 Objectives Advanced 5 7 In development 12 Incipient 11 Disaggregated 6 Service Advanced 1 6 In development 16 Incipient 17 Disaggregated 1 Results Advanced – 1 In development 2 Incipient – Disaggregated 38 General assessment Advanced – 1 In development 13 Incipient 23 Disaggregated 4 It is also important to analyze, quantitatively, other issues included in the instrument in order to understand the real importance of the service in these experiences. Although 97.6% of the professors reported that the course had moments of reflection with the students (a vital aspect to bring service and learning together), and despite the fact that this had been done before, during and after the service, when asking those with whom the reflection was conducted, the preferred option was the course and the teacher (72.5%), or the teacher with the work group of the SL experience (67.5%), the involvement of the collaborating entity (50%), or of the community which received the service (10%), being limited. The number of those who stated that their evaluation system covered the community institution is less than half (41%). For the most part, there was no collaboration agreement (58.5%), and in only 42.5% of the cases, the answer was positive. This may be due to the fact that the relationship with the partner was circumstantial or temporary in 63.4% of the courses and only 36.6% were standard and long-term practice. This is one of the issues that concerns teachers the most, since it involves, among other things, the search for community partners, the establishment of guarantees for students, or the academic recognition based on European Credit Transfer and Accumulation System (ECTS) credits (see http://www.usc.es/ apsuni/; www.uv.es/aps). Data show that these experiences should clarify the social needs, and those which the course aims to address, as well as the way to identify such needs, due to the importance of students’ involvement. This would allow defining the objectives of the service and evaluate the obtained results. One could say, at this point, that experiences are what Rubio et al. (2015) defined as a ‘sense of service’, facing the questions ‘why’ and ‘what for’ the participants’ action arises in a given situation. What is more important, service or learning? Different authors claim that, in addition to this analysis, when assessing an SL course, the relationships between the elements or components should also be considered, understanding the entirety of the experience, visualizing its complexity, and optimizing it (Rubio et al., 2015). That is why, the analysis performed on service and learning was cross-sectional, in view of the experiences in which the general assessment coincides in both dimensions, and those in which the majority of the burden falls on the learning activities or, on the contrary, on the service. As shown in Figure 1, in 16 experiences (39%), the assessment of service and that of learning concur. Consequently, both SL fundamental dimensions are balanced. Most were assessed as incipient (12), which would be at an intermediate level of the implementation process of an activity, which should be growing progressively. In three of them, the experts observed more progress, thus they were ‘in development’; only in one case, they reported that the experience had lacked a well-defined service and the learning process had not been explained. It was concluded that this was not an SL course, despite having been developed throughout three or more academic years, which does not prevent us from recognizing it as a good pedagogical practice. In a smaller number of experiences (10), learning is assessed more positively than service: in 7 of them, learning is considered ‘in development’ but service is ‘incipient’; in other 3 experiences, the sights were lowered for both learning (incipient) and service (non-existent). The assessment of service stands out above that of learning in 15 (36.58%) of the experiences analyzed. There are 9 cases in which service is defined as ‘in development’, and learning is evaluated as ‘incipient’; in 2 experiences, service has the same assessment, but no learning was mentioned; and finally, in 4 of them, service is described as ‘incipient’, and learning is non-existent. María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 360 Service stands out, above all, in the experiences performed in degree programs unrelated to the Social and Legal Sciences. The average score for this group in the assessment of service is 2.40, whereas for Social Sciences (mainly experiences associated with Education Sciences), the score is 2.15. However, the Student’s t-test showed that there were no significant differences between the two studied groups (t(39)=1.22; p=.228). On the other hand, the learning scores are more even, since the courses in Social and Legal Sciences reach a mean score of 2.08, and in the other areas the score is slightly lower (2.07). Once again, the Student’s t-test showed that there were no significant differences between the two groups (t(39)=-.48; p=.962). Discussion and Conclusions The scientific literature shows the key elements that an SL experience should have, with learning and service as main pillars connected through reflection and planning (Honnet & Poulsen, 1989; Howard, 2001). The first thing that is apparent from our study is precisely the need to integrate service and learning into a well-structured project, avoiding imbalances that may lead to experiences potentially closer to volunteering (where the weight falls on the service) or field education (where learning is the most important aspect) (Furco, 1996). Precisely, this is the contribution that this work seeks to provide to the field of service-learning, since in order to guarantee a sustainable growth of the methodology in universities, one should first recognize the level of assumption that the different implemented experiences make of the educational premises that turn SL into a differentiated methodological strategy (Santos Rego et al., 2015). The analysis of the practices included in this research study shows an encouraging course of action for this methodology in the Spanish university system, highlighting a certain change in the teaching culture, which is imbued with a strong component of social commitment (Lorenzo et al., 2019). However, this change is at an early stage. Indications thereof are that none of the experiences was classified as advanced (value 4), neither in learning nor in service, as most of them are in an incipient state (value 2), or in development (value 3). The implementation process was, in this case, at an intermediate level, a progressive growth being expected (Lambright & Alden, 2012). It can therefore be concluded that Spanish universities are in an initial phase, that of creating critical mass and in which it is urgent to recognize the presence of this type of experience. It is nevertheless true that the institutional progress of this methodology has started heading towards more developed stages, where what matters is no longer only the presence and amount of SL courses, but their quality (Furco, 2002). Only one practice is found to be strictly outside the scope of SL, although pedagogically appropriate. In our opinion, the professors involved should improve their definition of essential elements of learning (7 experiences) or service (4 experiences). Our optimism is not diminished by this situation, considering the data extracted from the study. As Tapia (2010) pointed out, this educational methodology is developed through processes of transition, not always intentional, which, to be defined as service-learning, were often based on another type of activities. Consequently, teacher training in this methodology becomes an essential pillar to guarantee the adequate development of the experiences, according to quality criteria and principles, especially taking into account the shortcomings that have been observed in the Spanish context (Álvarez, Martínez, González, & Buenestado, 2017; Lorenzo et al., 2019). On the other hand, when examined separately, learning comes out better than service, since it sometimes emerges as a means to service. Thus, there is a lack of reciprocity between these elements of the process. In our opinion, this problem is due, to a large extent, to the common assumption of a merely instrumental role by the partners during the development of the experience. However, one of the first scientific articles to address service-learning already warned that it was not only a matter of joining learning and service in a single project, but also of understanding that service situations in the community were configured as learning contexts to which importance should be attached (Sigmon, 1979). Another possibility is that the other two elements of functional conjunction, planning and reflection in this case, had been missing or had not been applied properly; both planning and reflection are necessary ingredients for adequate feedback from the students and the partners participating in the program. For Santos Rego et al. (2015), both elements were clear catalysts for the quality of the project. Thus, the authors proposed that adequate planning guaranteed the pedagogical and academic intentionality of these projects, while reflection allows for the integration of learning and the connection of community service with the curricular framework of the subject. Finally, if teachers are asked to provide a pedagogical basis for their practice, there is a greater reason to ask universities to design a solidly sustainable training and innovation plan, which is safe from political circumstances, needing minimal financial assistance and, obviously, assessable. Its viability could be linked to a three-fold path: a) basic training, including design and mentoring, for teachers, students and partners; b) institutional recognition of the agents involved and their efforts; and c) taking care of the partnership, providing logistical support in search of organizations or entities willing to participate in pedagogical tasks that add value and lead to a desire for corporate social action. In conclusion, it is also important to acknowledge the limitations of the research, as well as the future lines of work arising from them. The key limitation may be that, when assessing how they affect the quality of the experiences, the study did not take into consideration the levels of teacher training in SL. In a direct relationship to what was mentioned in the previous point, it is also urgent to recognize which variables or elements of a project have the greatest impact on students’ learning process. This identified the need to improve and extend the instrument used in order to work on delimiting the main criteria that define a quality SL course, the way in which these variables are determined by the educational level of the teacher and, finally, the impact they have on students, community and university. Figure 1. Overall assessment of learning and service. María del Mar Lorenzo Moledo, Covadonga Ruíz de Miguel, Elena Arbués Radigales, María Jesús Martínez Usarralde, Mariana Buenestado Fernández e Ígor Mella Núñez Aula Abierta, volumen 49, nº4, octubre-diciembre, 2020, págs. 353-362 361 References Álvarez, J. L., Martínez, M. J., González, H., & Buenestado, M. (2017). Service-Learning in teacher training in Spanish universities. Revista Española de Pedagogía, 75(267), 199217. doi: 10.22550/REP75-2-2017-02 Anderman, E. (2010). Reflections on Wittrock´s generative model of learning: a motivation perspective. Educational Psychologist, 45(1), 55-60. doi: 10.1080/00461520903433620 Ash, S., Clayton, P., & Atkinson, M. (2005). Integrating reflection and assessment to capture and improve student learning. Michigan Journal of Community Service Learning, 11(2), 49-60. Boud, D. (2001). Using journal writing to enhance reflective practice. New Directions for Adult and Continuing Education, 90, 9-17. doi: 10.1002/ace.16 Bringle, R. G., & Clayton, P. (2012). Civic education through service-learning: what, how, and why? In L. McIlrath, A. Lyons & R. Munck (Eds.), Higher education and civic engagement. Comparative perspectives (pp. 101-124). New York, NY: Palgrave. Butin, D. W. (2006). The limits of service-learning in higher education. The Review of Higher Education, 29(4), 473-498. doi: 10.1353/rhe.2006.0025 Campo, L. (2015). Una rúbrica para evaluar y mejorar los proyectos de aprendizaje servicio en la universidad [A rubric for assessment and improvement of service-learning projects in the university context]. RIDAS, Revista Iberoamericana de Aprendizaje Servicio, 1, 91-111. doi: 10.1344/RIDAS2015.1.6 Cooks, L., & Scharrer, E. (2006). Assesing learning in community service learning: a social approach. Michigan Journal of Community Service Learning, 13(1), 44-55. Deeley, S. (2015). Critical perspectives on service-learning in higher education. Basingstoke: Palgrave Macmillan. Eyler, J., & Giles, D. (1999). Where´s the learning in service-learning? San Francisco, CA: Jossey-Bass. Furco, A. (1996). Service-learning: a balanced approach to experiential education. In B. Taylor (Ed.), Expanding Boundaries: Serving and Learning (pp. 2-6). Washington, DC: Corporation for National Service. Furco, A. (2002). Self-assessment rubric for the institutionalization of service-learning in higher education (revised 2002). Retrieved from https://bit.ly/2XledLy Gargallo, B., Sahuquillo, P. M., Verde, I., & Almerich, G. (2018). What happens when teachers use learning-centered methods? Effects on learning approaches, on students’ capacities and on their perception of learning environment. Revista de Educación, 382, 157-190. doi: 10.4438/1988-592X-RE-2018-382-396 Honnet, E. P., & Poulsen, S. J. (1989). Principles of good practice for combining service and learning. Retrieved from https://bit. ly/3e4kHFj Hopkins, B., & Hermann, J. (1977). Evaluating inter-observer reliability of interval data. Journal of Applied Behavior Analysis, 10(1), 121-126. doi: 10.1901/jaba.1977.10-121 Howard, J. (2001). Service-learning course design workbook. Ann Arbor, MI: OCSL Press. Jameson, J., Clayton, P., & Ash, S. (2013). Conceptualizing, assessing, and investigating academic learning in service-learning. In P. Clayton, R. G. Bringle & J. A. Hatcher (Eds.), Research on service-learning (vol. 2A) (pp. 85-110). Sterling, VA: Stylus. Krippendorff, K. (2004). Content analysis. An introduction to its methodology. Thousand Oaks, CA: Sage. Lambright, K., & Alden, A. (2012). Voices from the trenches: faculty perspectives on support for sustaining service learning. Journal of Higher Education Outreach and Engagement, 16(2), 9-45. Lim, S., & Bloomquist, C. (2015). Distinguishing service learning from other types of experiential learning. Education for Information, 31(4), 195-207. doi: 10.3233/EFI-150952 Lorenzo, M., Ferraces, M. J., Pérez, C., & Naval, C. (2019). University teaching staff facing service-learning: explanatory variables. Revista de Educación, 386, 37-60. doi: 10.4438/1988592X-RE-2019-386-426 Naval, C., & Arbués, E. (2016). El aprendizaje-servicio y la transición desde la educación superior al mundo del trabajo [Service-learning and transition from higher education to the labor market]. In M. A. Santos Rego (Ed.), Sociedad del conocimiento. Aprendizaje e innovación en la universidad (pp. 219-239). Madrid: Biblioteca Nueva. Puig, J. M., Batlle, R., Bosch, C., & Palos, J. (2007). Aprendizaje servicio. Educar para la ciudadanía [Service learning. Educating for citizenship]. Barcelona: Ministerio de Educación y Ciencia / Octaedro. Ross, E. W., & Gautreaux, M. (2018). Pensando de manera crítica sobre el pensamiento crítico [Thinking critically about critical thinking]. Aula Abierta, 47(4), 383-386. doi: 10.17811/rifie.47.4.2018.383-386 Rubio, L., Puig, J. M., Martín, X., & Palos, J. (2015). Analizar, repensar y mejorar los proyectos: una rúbrica para la autoevaluación de experiencias de aprendizaje servicio [Analyze, rethink and improve projects: a rubric for self-assessment of service-learning experiences]. Profesorado: Revista de curriculum y formación del profesorado, 19(1), 111-126. Santos Rego, M. A., & Lorenzo, M. (2018). A guide for the institutionalization of service-learning at university level. Santiago de Compostela: Universidade de Santiago de Compostela. Santos Rego, M. A., Sotelino, A., & Lorenzo, M. (2015). Aprendizaje-servicio y misión cívica de la universidad: una propuesta de desarrollo [Service-learning and civic mission of the university: a proposal for the development]. Barcelona: Octaedro. Sigmon, R. (1979). Service-Learning: Three principles. Synergist, 9-11. Retrieved from https://bit.ly/3gaSCy8 Simons, L., & Clearly, B. (2006). The influence of service learning on students’ personal and social development. College Teaching, 54(4), 307-319. doi: 10.3200/CTCH.54.4.307-319 Shumer, R. (2003). Self-assessment for service-learning. In S. Billig, & A. Waterman (Eds.), Studying service-learning. Innovations in education research methodology (pp. 149-171). Mahwah, NJ: Lawrence Erlbraum Associates. Sotelino, A., Santos Rego, M. A. & Lorenzo, M. (2016). Aprender y servir en la universidad. Una vía cívica al desarrollo educativo [Learning and serving in the university context. A civic path to educational development]. Teoría de la Educación, 28(2), 225-248. doi: 10.14201/teoredu282225248 Tapia, M. (2010). La propuesta pedagógica del aprendizaje-servicio: una perspectiva latinoamericana [The pedagogical proposal of service-learning: a Latin American perspective]. Tzhoecoen. Revista Científica, 3(5), 23-43. Torrego, L., Martínez, S., & Sonlleva, M. (2018). ¿Es posible la formación de personas éticas y comprometidas en la universidad? La asignatura de Educación para la paz y la igualdad en los Grados de Magisterio [Is it possible to educate ethical and committed people in the university? The subject of Education for peace and equality in the Degrees of Teaching]. Aula Abierta, 47(4), 415-422. doi: 10.17811/rifie.47.4.2018.415-422 Villa, A. (Ed.) (2013). Proyecto Tuning América Latina. Un modelo de evaluación de Innovación Social Universitaria Responsable (ISUR) [The Tuning Latin America Project. An evaluation model for Responsible University Social Innovation (RUSI)]. Bilbao: Universidad de Deusto.