A Benchmarking Exercise for Quality Blended Learning. A Challenge for European Universities in the 21st Century

1 A Benchmarking Exercise for Quality Blended Learning. A Challenge for European Universities in the 21 st Century Authors: Valentina Elvira Comba 1, Ebba Ossiannilsson 2 , Lena Landgren 2, Rasmus Blok 3 , Jose Manuel Martins Ferreira 4, Christopher Kjær 5 and IngerMarie F. Christensen 5 Affiliation: The authors were all part of a benchmarking exercise in 2009 through ESMU, European Centre for Strategic Management of Universities (http://www.esmu.be) in cooperation with EADTU (http://www.eadtu.nl). Authors’ affiliation and e-mail 1 University of Bologna, Italy, [email protected] 2 Lund University, Sweden, ebba.ossiannil[email protected], [email protected] 3 Aarhus University, Denmark, [email protected]k 4 University of Porto – Faculty of Engineering, Portugal, [email protected]t 5 University of Southern Denmark, [email protected], i[email protected] Abstract This paper shares the experiences of 5 universities involved in a Benchmarking Exercise on eLearning in 2009 through ESMU. A total of 9 European universities participated in the exercise with the purpose of evaluating their existing eLearning practices and policies and getting advice on which areas to improve and how. Initially, the paper discusses the benchmarking concept and reasons why institutions should engage in benchmarking exercises. Benchmarking is viewed as a method for quality assurance and enhancement in higher education. Self-assessment is involved at the participating institutions, which leads to a high level of awareness and understanding of existing practices and policies at different levels of the organisation. Benchmarking is thus an efficient self-improvement tool. In chapter two, the planning of the benchmarking exercise is outlined. A combination of an individual, collaborative and expert approach to benchmarking was chosen. Chapter three deals with the creation of the benchmarking questionnaire which was a collaborative effort between all nine participating universities, ESMU and EADTU. Taking their starting point in the online E-xellence benchmarking tool developed by EADTU, participants reformulated, deleted and added benchmarks within the following six categories: Strategic management, curriculum design, course design, course delivery, staff support and student support. An effort was made to translate the E-xellence questions to the blended learning context of the participating universities. The internal data collection and formulation of responses are accounted for in chapter 4 which also contains reflections on the challenges and benefits of the selected approaches. Chapter five presents the overall conclusions of the benchmarking exercise within each of the six benchmark categories. The final chapter discusses and provides examples of how participants can use the benchmarking results to improve existing practices and policies and outlines potential external collaboration opportunities between participants. 2 Keywords: Benchmarking, quality assurance, blended learning, eLearning, collaborative learning, European universities 1. Introduction In 2009, ESMU 1 (European Centre for Strategic Management of Universities) conducted the Benchmarking Exercise on eLearning with 9 European universities. This paper is a joint effort from 5 of these universities to share their experiences and to encourage other institutions to engage in similar benchmarking exercises. The paper explains each of the phases in the benchmarking process and further discusses follow-up actions and their importance to improve the quality of teaching and learning. The paper also discusses the value of benchmarking exercises and gives specific examples of the benefits for universities. Why benchmark on eLearning? Quality development and evaluation make up crucial parts of the activities of educational institutions today and benchmarking has become an increasingly common method for performing quality work (Ubachs, 2009). Benchmarking deals with changes, but also with enhancement and successful implementation of new procedures and efforts (Ossiannilsson, 2010a, b, c). Moriarty (2008) defines the method as: “…an exemplar-driven teleological process operating within an organization with the objectives of intentionally changing an existing state of affairs into a superior state of affairs” (Moriarty, 2008, p. 30). Moriarty & Smallman (2009) further state as follows: “The locus of benchmarking lies between the current and desirable states of affairs and contributes to the transformation process that realizes these improvements” (Moriarty & Smallman, 2009, p. 484). The definition used by ESMU is expressed as: “Benchmarking is an internal organizational process which aims to improve the organization’s performance by learning about possible improvements of its primary and/or support processes by looking at these processes in other, better-performing organizations” (van Vught et al., 2008, p. 16). Benchmarking initiatives are often conducted as self-evaluations, including systematic data and information gathering, from predefined benchmarks, as well as formulating roadmaps. The goal of benchmarking is to formulate, together with others, strengths and challenges and areas for enhancement (Ossiannilsson, 2010a; van Vught et al., 2008). The benefits can be expressed as it is defined in ten (10) statements by ESMU: “self-assess institution, better understand the process, measure and compare, discover new ideas, obtain data to support decision-making, set targets for improvement, strengthen institutional identity, enhance reputation, respond to national performance indicators and benchmarks and set new standards for the sector” (van Vught et al., 2008). When participating in a benchmarking process, not only the state of the art in the investigated area and possible change potentials are obtained. In addition, awareness, both individual and collective, on the organisation itself is a result of participation, which can be considered as a direct and substantial value (Ossiannilsson, 2010a). Often benchmarking is confused with ranking in terms of methodology and outcome in the public sector, as likewise the term “benchmarking” is widely used in the private sector as synonymous with exercises comparing company performance in regard to tenders. However, benchmarking and ranking are very different, as benchmarking is a self-improvement tool for 1 http://www.esmu.be 3 organisations, combining the advantages of quality assurance – mainly based on selfassessment – and of experience sharing. It allows participating institutions to compare themselves with others, identify their comparative strengths and weaknesses, and learn how to improve as described in the introduction. Benchmarking is thus about identifying best practice within certain fields in order to find ways for improving existing practice, and therefore has a strong focus on development and improvement. Rankings do not address development and improvement, but only works to give a snapshot of a certain performance at a certain time, which is then listed and ranked among others. Within the world of universities, Academic Ranking of World Universities 2 compiled by Shanghai Jiao Tong University and the Times Higher Education World University Rankings 3 are the two most prominent rankings. As rankings among universities are often of high priority in a global competitive world for many university managements, and with high direct or indirect impact on funding and student enrolment, it was important to make such differences between benchmarking and ranking very clear for all (especially management) at the outset of the benchmarking exercise. So far, eLearning has not been the subject of benchmarking initiatives to a high extent (Ossiannilsson, 2010a; Ossianilsson & Landgren, 2010a). However the experiences from the Benchmarking Exercise on eLearning (BEEL) clearly show the benefits of such initiatives and emphasize the values of continuous benchmarking exercises. With this paper the participants in BEEL highly recommend institutions to follow and take part in such projects. Participants in the exercise Nine European universities participated in the ESMU project, BEEL, namely Aarhus (DK), Bologna (IT), Copenhagen (DK), Kuopio (FI), Latvia, Lund (SE), Southern Denmark (coordinating institution), Oulu (FI) and Porto (PO). This paper describes and reflects on the individual steps of the benchmarking exercise. The initial planning process will be accounted for, the revision and reformulation of benchmarks are explained and the approach to data collection and formulation of responses is discussed in detail. Following that the conclusions of the benchmarking exercise are presented and both internal and external follow up actions are outlined. 2. Planning a benchmarking exercise The Benchmarking Exercise on eLearning among universities in Europe originated as an idea from the University of Southern Denmark. Here the initial idea was to look beyond national borders in order to compare, collaborate and learn from other universities on the specific subject of eLearning. However, when planning for the benchmarking of eLearning among universities in Europe, some initial considerations had to be made. First and foremost questions about objective, outcome and benefits needed to be addressed in order to determine the framework, process and level of collaboration for the benchmarking exercise. Along with these questions, considerations about number of participants, size and timeframe together with issues on management of the benchmarking exercise and finding and selecting the right participants had to be addressed. From the start, ESMU was chosen as an external partner and manager of the benchmarking exercise. ESMU has for several years been carrying out annual benchmarking exercises within European higher education institutions (of which the initiating university, University of Southern Denmark has been a participant) as well as developing significant work on 2 http://www.arwu.org 3 http://www.timeshighereducation.co.uk/ 4 benchmarking in higher education and producing a handbook as an attempt to provide a clear definition of what benchmarking is and is not. The ESMU benchmarking methodology is precisely based on a collaborative approach, and is about organisations learning to share and identify good practices in order to set targets for improvement as discussed above. The focus is on processes by which results are achieved. Processes often exceed organisational boundaries and involve several departments and levels within the organisation, and the benchmarking exercise should lead organisations to question the impact of such boundaries on the effectiveness of their processes. With ESMU involved as the organizer and manager of the benchmarking, the objective of the exercise was to enable each institution to reference its own policies and practice against the other institutions in the group using a methodology developed and agreed by the whole group. In the absence of national or European codes of practice for benchmarking of eLearning it was necessary to bring in an organisation with expertise in the quality assurance of eLearning. Hence EADTU 4 , European Association of Distance Teaching Universities was chosen as a second managing partner for their subject expertise on eLearning together with their newly developed online assessment tool, E-xcellence, aimed at individual higher educational institutions and their use of ICT for distance teaching. At an initial meeting in Brussels in May 2009 between the initiators from the University of Southern Denmark, ESMU and EADTU, the final benchmarking methodology and collaboration plan was agreed on, as well as two subject experts (Keith Williams 5 & Bob Rotheram 6 ) who should act as reviewers and compilers of the benchmarking questionnaire and responses were named. The Benchmarking Exercise on eLearning should be a collaborative activity in which the participants use mutually agreed performance criteria in order to reference their own performance, relate their performance to the other participants with the help of two subject experts, and finally develop an individual action plan for addressing future improvement within their own organisation. The exercise as such consisted of these five stages: o establishing benchmarking criteria at a workshop (May 2009, Brussels, Belgium) o data collection and preparation of institutional response o compilation of responses and preparation of report o comparison of institutional responses and workshop discussion (November 2009, Odense, Denmark) o development and implementation of improvement action plans The time frame was set to ¾ of a year starting spring 2009 and finishing by the end of that year, leaving roughly 2 months for mutual collaboration on benchmarking criteria; 1 month for setting up the questionnaire; 2 months of individual data collections and institutional response to the questionnaire; 2 months for experts to review and compile an interim report; 1 month for discussing the interim report and developing individual action plans; and finally 1 month for the experts to sum it all up in the final report. The exercise was set to a minimum 4 http://www.eadtu.nl/ 5 Keith Williams is a Senior Lecturer in the Communications and Systems Department of the Open University. In a long career at the Open University he has served as Dean of Technology Faculty and as Director of Academic Development in Open University Worldwide, the university’s international division. He worked as Director of Distance Learning with The British Council during leave of absence from the OU. He has been a member of the E-xcellence project core team throughout its initial phase, E-xcellence Plus and the recently funded E-xcellence Next phase. He is currently engaged in projects concerned with professional skills development with Sector Skills Councils and employers. 6 Bob Rotheram is Reader in Assessment, Learning and Teaching at Leeds Metropolitan University. He is a UK National Teaching Fellow with over 30 years experience of university teaching and faculty development in several countries. He is particularly interested in the uses of technology in higher education. 5 of 8 and a maximum of 15 participants, involving a cost of € 3.400 for each participating institution for the whole exercise (excluding travel and accommodation at the two workshops). With the help of ESMU, the University of Southern Denmark placed a European call for participation – asking universities to join in the benchmarking of eLearning on the above conditions. By spring 2009, luckily 10 universities had asked to participate, of which 9 ended up carrying out the exercise. As the participant group showed a fruitful diversity in terms of geography, use and experience of eLearning, there was no need to engage in a selection process. In order to establish a common ground for developing criteria and benchmarks for the exercise, it was agreed to utilise the E-xcellence criteria developed by the EADTU as the starting point. The criteria were structured on the assumption that higher education institutions operate within a strategic framework and that other policies, resourcing and management practice relate to the institution’s strategic vision and objectives. Hence the criteria developed in the E-xcellence 7 project also cover pedagogic, technical, student and staff related aspects of eLearning and their interrelationships. Amongst the resources available via the E-xcellence website is a “Quick Scan” questionnaire that enables an institution to informally evaluate their eLearning performance against 33 criteria. This Quick Scan questionnaire was used as the starting point for the formulation of the benchmarks used in this exercise. The formulation of benchmarks was a collaborative effort, see details below, however, each institution produced and submitted its response to the questionnaire in isolation from the others. The responses were then collected and combined into a compilation and an associated summary chart that presented an overview of what were judged by reviewers and ESMU staff to be contributions of significant interest. From this material the reviewers produced an interim report, which was circulated to the participants prior to the final workshop held in Odense, Denmark. Additionally participants were requested to draft an action plan for improving performance and practice regarding eLearning at their university. For this an action plan template was provided in order to help the participants identify what needed to be achieved and how. At the final workshop each participating university was invited to make a short presentation of an aspect the reviewers had identified as being of particular interest. The idea was to bring more detail to best practices of different sorts from all participating universities. Finally the workshop addressed the initial action plans, which was discussed and supervised in groups. Universities showing best practice in one area of eLearning would support others addressing development in this particular area and vice versa. By the end of the year, the whole benchmarking exercise was summed up in a final report delivered by the experts Keith Williams & Bob Rotheram (Williams & Rotheram, 2010). The report addressed all the results summarized within the 6 benchmark categories, and additionally went to point to possible conclusions and best practices within each of these categories. The report also included data on background and benchmarking methodology for the whole exercise, as well as a summery chart of participant responses to the questionnaire. 3. Evaluating benchmarks As mentioned above, the first step of the benchmarking exercise has been the completion of the Quick Scan exercise by each university. The Quick Scan exercise has been proposed in the frame of the EADTU’s E-xcellence Project. It is accessible online free of charge 8 and is 7 http://www.eadtu.nl/e-xcellence 8 http://www.eadtu.nl/e-xcellenceqs/ 6 supported by the E-xcellence Manual (Ubachs, 2009). It consists of 33 benchmarks. At the moment, three versions are available (English, French and Italian). This Quick Scan tool was used as a mechanism for initial engagement with the benchmarking criteria, as each participating university was asked to access and finalise the scan before the first workshop in Brussels. The questionnaire was completed online, and each participant received online feedback immediately that made comparisons between the answers to the different questions possible. The feedback also included suggestions for relevant procedures that each participant should consider implementing to drive eLearning forward at the institution. The Quick Scan questionnaire was discussed at our first workshop in Brussels on 26 and 27 May 2009. At the workshop, representatives of the participating universities met with members of ESMU and of the E-xcellence development group in order to discuss the use of the benchmarking criteria and indicators of the E-xellence tool as the basis for this particular benchmarking exercise. Some criteria and indicators were found irrelevant, some were added, some were just revised and yet some were substituted by new criteria more closely related to the blended learning context of the participating universities. The changes concerned student perspectives, library resources and personalisation among other things. But with these changes the group in collaboration decided that the 33 reformulated Excellence criteria could work as benchmarks for the exercise. The agreed reformulated criteria and performance indicators were then transposed into a questionnaire by colleagues from the University of Southern Denmark who also managed the process of collecting and compiling responses. In focus at the first workshop, which was led by Keith Williams (Open University, U.K.), was also and in particular the different approaches to distance learning by the participating universities, all of them defined as “traditional universities”. The mission of traditional universities includes research and teaching, the latter is often offered face-to-face because the direct contact between the researchers and their students play a very large role in university teaching. Therefore, the face-to-face lessons remain the core and eLearning is used to support and enhance the students’ learning. However, the main goal of traditional universities is to provide high level teaching, based on the most advanced research; therefore teaching is sometimes considered a secondary goal. For this reason, these universities typically do not invest high budgets in teaching, nor provide full distance courses, because of the overall costs and the costs of updating materials. The usual trend is to provide “blended learning” courses, where only a part of the course is online. The group therefore had lengthy discussions on terms to be used and the definition of these terms: “eLearning” stands for full distance learning or the online components in a course; “blended learning” for a face-to-face with some eLearning components course. At the workshop in Brussels in May 2009, eLearning was defined as follows: “eLearning covers a wide set of applications and pedagogical processes supported by ICT, such as web-based learning, computer-based learning, virtual classrooms and digital collaboration with an added value of increased accessibility, flexibility and interactivity” (unpublished observations, BEEL workshop, May 2009). The discussion on terminology has been very useful to reach an agreement among the participants on what should be modified in the proposed questionnaire. The main areas which were modified were the curriculum design and course delivery. The benchmarks in these areas very clearly pointed at a distance learning approach to eLearning and had to be reformulated to properly reflect the blended learning approach of the group. 7 The group agreed, with expert Keith Williams and ESMU secretary general Nadine Burquel, that our benchmarking exercise would be devoted mainly to a qualitative assessment and not a quantitative one; only the data about budget and expenditure would provide a quantitative picture of the universities. The final benchmark questionnaire was first discussed in two small groups at the workshop in Brussels, where changes and additions were proposed. The “big work” was done after the meeting because we had the possibility to work together on a virtual platform offered by the University of Southern Denmark, where the final version of the questionnaire was eventually available. The final questionnaire included the following 6 benchmark categories: • Strategic management • Curriculum design • Course design • Course delivery • Staff support • Student support. Everything was now set for the individual data collection at the participating institutions. 4. Internal Data Collection and formulating responses With the questionnaire as a framework, the purpose of the internal data collection was to describe the policies and practices adopted at each university to facilitate a comparison between the participating universities. The guidelines for collecting internal data stressed that one should consult widely with colleagues to ensure that one’s institutional policies were fully represented. In addition each benchmark statement should be described with about 250 words maximum. With the above in mind, all participating universities used different approaches to organising and collecting internal data. In the following, we will briefly describe the different ways in which data collection was organised and approached. Approaches to and the organisation of data collection At the University of Southern Denmark (SDU), the internal data collection was organized and conducted by a task force consisting of two representatives from the Central E-learning Unit. Following an analysis of each benchmark statement and indicator, 18 key persons were identified and interviewed. These persons were categorized with respect to their knowledge regarding eLearning at SDU and the different benchmarks and indicators. From an organisational perspective these key persons were representatives from the following different levels and units at SDU: the E-learning Unit including our LMS system administrator, the E-learning Strategy Committee, the E-learning Coordination Committee and the Quality Organisation. These interviews were supplemented with relevant eLearning documents describing the ELearning and Quality Organisation at SDU. To secure a representative collection of data with respect to the benchmarks Curriculum Design and Course Design, persons from each faculty were interviewed (these are included in the 18 key persons). The questionnaire and its indicators were used as a semi-structured interview guide. Together with a student assistant we conducted and transcribed the interviews simultaneously. This approach forced us to transform each benchmark and indicator to 8 questions that could be asked to our different informants or key persons. Therefore, answering the questions posed in each benchmark became an easy task when using the corresponding indicators. A first draft of the questionnaire with responses was sent back to all informants for corrections and additional feedback. A few corrections were made and the questionnaire was then completed. At the University of Porto, a task force was set up for the purpose of data collection, comprising six members: 1) the institutional coordinator for the benchmarking project, 2) the director of the IT Systems division (supporting the information system and the computer network, including the eLearning servers and infrastructure), 3) the director of the Communication division (providing support for the development of eLearning contents, among other roles not related to eLearning), 4) a member of the academic staff who is also a specialist in eLearning technologies, particularly in Moodle, 5) one degree director (in this case, a member of the academic staff that is responsible for the integrated masters degree in mechanical engineering), and 6) one student (at the time, a 4th year student in the electrical and computer engineering integrated masters degree). A plan of face-to-face meetings and email discussions were agreed upon, enabling data collection and peer-review of information, according to the specific expertise of each member. The plan worked well with the exception of the selected student, who was not able to attend the face-to-face meetings and did not contribute with information or review comments. This was not due to a lack of interest, but rather to a lack of opportunity due to his academic duties (mostly classes and exams). The combined areas of expertise of the remaining team members were, however, considered sufficient to validate the responses to the questionnaire. The approach to collecting data did not involve interviews with end-users, not only for reasons of time, but also because the selected group of people were believed to possess all the knowledge needed to answer the benchmark questionnaire. At an initial meeting, the questionnaire was run through and a first round of information was gathered and used for building the first set of responses. This draft document was then circulated to get written feedback from the group members, which in turn was used to refine the responses. Occasionally there were doubts which led the coordinator to talk with additional people, namely people that were then in the Board of Directors. A second meeting took place to discuss the overall picture, and then the responses were sent. These responses led in turn to some comments from the experts, which we used to rediscuss and further refine some aspects. At Aarhus University it was strategically decided to form a small task force towards the implementation of the benchmarking exercise, which should act as an active and proactive unit with respect to collecting data in terms of the necessary information and documentation. Since the focus was on management processes, a broad representation in the task force seemed unnecessary, but rather a small and efficient task force that could act quickly and flexibly. Thus, the task force consisted of four people including Head of Studies, an IT consultant, the head of The E-learning Unit and a Special Adviser also from The E-learning Unit, with the latter as project manager of the exercise and data collection. Because of a merger between Aarhus University and several other Danish Universities, the collection of data became a challenge. Especially because each of the merging universities had their own eLearning units which used different approaches to eLearning. It was then decided that the point of departure for answering the benchmark questionnaire should be from the perspective of the Aarhus University E-learning unit. Other eLearning activities from the merging universities were used in the questionnaire if considered relevant. 9 At the University of Bologna a task force was set up consisting of the eLearning manager and a young expert in budgeting, data management and presentation, who was paid to support the data collection. Data were collected in August 2009, when most of the central university offices were closed. This limited the collection of data to be from the official reports of the university published on the university website. In addition detailed data about salaries were received from the “Accounting and Control Area” for the relevant periods and staff involved in the eLearning activities. AT Lund University, the internal work on the benchmarking exercise was co-ordinated through the Centre for Educational Development, in collaboration with the Library Head Office and the Director of Strategic Development. The benchmarking process was based on recently submitted written documentation, evidence and material from the EADTU Excellence+ project in 2008 9 as well as from the site visit report and the formulated action plan for Lund University. The extensive body of documents was updated with the help of colleagues at the various infrastructural units and several initiatives, since the action plans mentioned in the E-xcellence+ documentation had now been formally implemented. Experiences from the data collection and formulation of responses Using the questionnaire as an interview guide had significant drawbacks. First and foremost, there were difficulties turning the benchmark statements and indicators into understandable questions which could be posed to the informants. It also played a role that the participants were all non-native English speakers. This forced the group to carefully discuss each benchmark and related indicators which indeed was a time consuming task. After the workshop in Brussels, it was assumed that agreement on the meanings of each benchmark and related indicators had been reached, but trying to explain and turn each benchmark into reasonable questions was for some an unexpected challenge. For future benchmarking exercises it is, therefore, recommendable to use more time on the development of common ground with respect to the identified benchmarks and indicators. Cultural differences between the participating universities, the external partners and the experts also had an impact on the benchmarking exercise and made it a complex business to reach agreement and to translate benchmarks to local contexts. An open and explicit discussion becomes even more important in this light. Nothing should be assumed or taken for granted. Looking back on the experiences from the process of collecting data and formulating responses, it was a challenge to work with the questionnaire. It should be shortened, and an initial glossary should be added to ensure that all institutions work with a common perception of terms and expressions. But from a positive point of view, all agreed that a much deeper and better understanding of eLearning at each university has been achieved. 5. The benchmarking report The final version of our Benchmarking Exercise on eLearning report dates from January 18 th 2010 and comprises the following sections: • Introduction [4 pp.] – offering an overview of the background of this exercise; describing the benchmarking methodology, criteria, and performance indicators that were adopted; and presenting overall observations that facilitate the interpretation of contents and the identification of possible follow-up actions. • The benchmarking exercise and results [16 pp.] – presents a summary of the information gathered within the six main areas addressed by the EADTU E-xcellence 9 Based on the results of this project, Lund University was, as the first university in Europe, awarded the Excellence Associates label for the two benchmarked international Master programs.