Effectiveness Evaluation on Entrepreneurship Education

1 Effectiveness Evaluation on Entrepreneurship Education Gustafsson-Pesonen Anne, Aalto University School of Economics Small Business Center, PhD Student, Training Director Abstract In EU there are hundreds and thousands ESF funded projects and several studies of these projects effectiveness have been done, but how many of these studies are focused on changes in entrepreneurship education specially in teachers thinking and action and changes on the organizational level? Note so many. That’s why I try to discover answers to the question: Is it possible and how it is possible to evaluate the Entrepreneurship Education project or program especially when taking teachers changes in thinking and action and changes on the organizational level. In the AUSESBC (Aalto University School of Economics Small Business Center) there is the effectiveness evaluation project, funded by ESF, whom called “Entrepreneurship Education joint and evaluation “YKOONTI. Project has been carried out 1.9.2010 – 30.04.2012 in Finland. The main idea of the project “YKOONTI” is to collect data and evaluate the Entrepreneurship Education ESF funded project’s which have been taken during 2000 – 2010. In this paper there were described some detail of the frame effectiveness evaluation and described one project which was evaluated during the project “YKOONTI”. The project evaluated in this paper was “FIRMA” which was carried in South Finland in the Vocational College “ESEDU”. Researcher has been decided to use in evaluation the frame of Kirkpatricks (1959, 2006) effectiveness evaluation four level model. Model is well known when evaluated effectiveness in management program but not so known when evaluating entrepreneurship education program or project. When thinking why evaluation is important is has been said, “Evaluation is about revealing the value of a project” (ETUI-REHS Project monitoring and evaluation – guidelines). The EU has been said “The purpose of evaluation is to examine how well a project answers to the need it is done for, that is to evaluate the results and effects of the project. The method of 2 implementation of the evaluation depends, among other things, on the stage it is done in and on who does it. The function of evaluation is also to yield information for the planning of the project, to help in the effective division of resources and to improve the quality of the project.” (European Commission 2004, 9.) Key words: Effectiveness Entrepreneurship Education, Teacher Education, Evaluation of the program, Changes of Thinking and action on entrepreneurship 3 1. INTRODUCTION The attitudes, behavior or opinion of teachers are often seen to be the biggest obstacle for the successful teaching of entrepreneurship and the realization of entrepreneurship education. On the other hand, it has also been noticed that the teaching of entrepreneurship seems to require the using of a concept of learning that is in unison with the phenomenon of entrepreneurship as the learning methodology of the teaching of the subject matter. The conception of learning teaches people ways of thinking, looking and acting needed in entrepreneurship, in other words, the values and attitudes of entrepreneurship, in addition to the information content of the education. The YKOONTI ESF funded project has been carried out 1.9.2010 – 30.4.2012 in Finland. The central actors of the project are the National Board of Education whom financed the project and AUSESBC whom administer the project. The main idea of the YKOONTI is to collect together mostly of the ESF funded projects which main idea was “Entrepreneurship Education” years 2000 – 2010 and evaluate these projects effectiveness. There has been found 154 ESF funded project which main idea was Entrepreneurship Education. One of these projects was the FIRMA -project which was carried out in South Savo and administer was the South Savo Vocational College (ESEDU). Aalto University School of Economics Small Business Center (AUSESBC) was responsible for the realization of the “EDC” (Entrepreneurship Development Coaching) and for doing evaluation research in YKOONTI. Totally 40 persons participated in the entrepreneurship developer coaching of the Firma –project during 2009-2010. The central aim of the project FIRMA has been 1) to increase the entrepreneurial readiness of the students of upper secondary level vocational schools in South Savo, and entrepreneurship and business knowhow and 2) to strengthen the entrepreneurship educational skills of the personnel of upper secondary vocational schools. The target group of the project has been the young and adult students and the teaching personnel of upper secondary vocational schools. The effectiveness evaluation was done by researcher, Gustafsson-Pesonen for the “EDC” program. Main idea in the evaluation was how was the program effect on teacher’s opinion, action and thinking on entrepreneurship education. Target group for the evaluation were the participants on the program, the directors and rector for the ESEDU. 4 In this research were used Kirkpatrick’s (2006, org. 1959) four level effectiveness evaluation method. Model is opened: level 1 Reaction: How participants feel about various aspects of an education program, level 2 Learning: Measure of the knowledge acquired, skills improved or the attitudes change, level 3 Behavior: To which extent changes in learning lead to changes in behaviors, level 4 Results: Measure of final results that occur due to education at multiple level. 1. PREVIOUS STUDIES ENTREPRENEURSHIP EDUCATION It has been studied by several researchers (e.g. Gibb, Kyrö) that Entrepreneurship Education needs to be taught entrepreneurially. This in fact makes the education entrepreneurial. Haahti (1989) has followed the historical text sources and has found the first meaning of entrepreneurship which was Action. Teaching entrepreneurship only by the knowledge-based method does not effect on the character and the nature of entrepreneurship and actually may cause negative consequences. Vice versa to teach whatever subject entrepreneurially effects on values, attitudes, action habits and thinking that will promote entrepreneurship. An extensive view on entrepreneurship and connections to working life and the knowing of the practical activity of entrepreneurs are typical of entrepreneurship education in the Finnish school system (Kyrö & Ripatti 2006, 20). The developing of an entrepreneurship identity on the level of values and attitudes has also been attached to the aims of entrepreneurship education in schools. According to Koiranen and Peltonen (1995), the aims of entrepreneurship education should be seen broadly so that with the help of education and other means of learning, entrepreneurship would be made a part of school and the pupils’ lives. The aim is to promote “the intellectual, skilled and attitudinal development of the pupils into entrepreneurial and self-directed, active agents.” (Koiranen & Peltonen 1995, 10, 32, Koiranen 1993, 8–9, Koiranen & Ruohotie 2001, 110). Ristimäki brings forth (1998a 65-66) the promoting of an entrepreneurial attitude, way of thinking and qualities in an individual in entrepreneurship education. Ristimäki divides entrepreneurship education into two parts. The first part is to promote the entrepreneurial activity of pupils, and the other is to bring entrepreneurship up as an alternative career choice to be reckoned with. (Ristimäki 2004.) According to Ristimäki (2004), entrepreneurship should be seen in schools also elsewhere than in external entrepreneurship, but he also sees that the broad goals of entrepreneurship education will not be achieved if we end up emphasizing only internal entrepreneurship. According 5 to Ristimäki, the division between internal and external entrepreneurship could perhaps be given up in the discussions on entrepreneurship taking place in schools and just talk about entrepreneurship education and the promoting of entrepreneurship. (Ristimäki 2004.) According to Seikkula-Leino (2007a), the task of entrepreneurship education is to educate entrepreneurial citizens, in which case the readiness for the developing of entrepreneurship is also assumed. The prerequisite for the development of entrepreneurial readiness are the individual’s sufficient self-regulation skills (Seikkula-Leino 2007a, 30). Seikkula-Leino summarizes that the aim of entrepreneurship education is to develop entrepreneurial qualities and to give people information on entrepreneurship (2007b). According to Kyrö, Lehtonen and Ristimäki (2007, 22), entrepreneurship education is related to both the study of entrepreneurship and the application of entrepreneurship education into the educational system. Seikkula-Leino (2007a, 26) summarizes, that entrepreneurship education is to be seen as civic education, because skills, behavior and beliefs are affected in education. Ristimäki (2007a) also brings forth that the definition of entrepreneurship and entrepreneurship education is not in conflict with school’s general educational orientations. Ristimäki defines entrepreneurial activity as innovativeness, risk management and a catalytic activity. Simultaneously he believes that the definition guides teachers to understand entrepreneurship education as a methodical question related to any given subject. (see Ristimäki 2007a, 35). Remes (2001; 2003) researched learning paradigms and studied the history of education to find out that traditionally learning and teaching does not support learning to be entrepreneur or entrepreneurial acting . Instead, so-called reform pedagogies do support it. The learning is then child- or student-centered and learning environments activate the learner holistically. Learning is also combined with the social authentic environment and is felt practical and actual. (Remes 2001; 2003.) Teacher’s responsibility is to follow the learning process and help the learner if he/she asks for it. When students learn to think and act entrepreneurially, in future, they have better skills to run a business of your own or act in the environments that demand entrepreneurial attitudes towards action. 6 2. ENTREPRENEURSHIP DEVELOPMENT COACHING “ EDC “ PROGRAM According to entrepreneurial pedagogy (Remes 2001, 2003) the learning process of entrepreneurship is therefore a creative learning event, in which the learners quite independently create new products and services. The teacher’s responsibility is to create outward circumstances for learning, to support and instruct learning and to function as an entrepreneurship mentor. The teacher’s working as an entrepreneurship mentor requires that the teacher is familiar with the learning process of entrepreneurship. As was done in the FIRMA –project’s education, the teacher have to experience the process by her - or himself to know empirically how to instruct the pupils and students (Gustafsson-Pesonen, Remes 2011). This is why entrepreneurial pedagogy should be used as the conception of learning in teacher education where entrepreneurship or entrepreneurship education is studied (Remes 2001, 2003). The structure of the course was (Gustafsson-Pesonen, Remes 2011): 1. Contact days. The main educator and visitors. Purpose: knowledge in entrepreneurship and entrepreneurship education. 2. Workshop days. The main educator. Purpose: To apply the knowledge both from the contact days and from their practical school work. 3. Developmental project. Mentor was the main educator. Purpose: To create new pedagogical solutions in entrepreneurship education. Testing in their work. 4. Mentoring. The main educator was available when there emerged problems or need to talk with someone. As Remes (2001, 2003) said in entrepreneurship the already existing knowledge is applied to the activity. New knowledge is acquired according to the need, and it is also applied to the current process. An entrepreneurial learning process needs knowledge that the actor is seen to already have but that is completed according to the need along the process. The model imitates entrepreneurship that has the doing of a product or a service for someone in the center of the activity. The activity is entrepreneurial and other people are in a significant role in it. The nature of knowledge of entrepreneurship is present. That is why the task of the main instructor is to support learning in real time. When questions or a need for additional information arise in the process, the instructor has to be easily reachable. The main instructor works as a role model for the developing entrepreneurship educator model of those having participated in the training. 7 4 THE FRAME OF EVALUATION RESEARCH When discussion effectiveness evaluation research it could be said it is quite difficult to measure what effects, how and to whom. Professor Fayolle has been studied the effectiveness of the programs and he has been said: “While the impact of the program or session on the trainees, students and other participants may be an acceptable evaluation criterion, the problem of how to measure it still remains. What indicators should be used, and how should they be measured? How can you measure a change in someone’s state of mind or behavior? How can the importance of the time factor be taken into account? And how can factors relating to education, teaching and training be separated from all the other factors that have an impact on the decision to choose a specific career path or profession?” (Fayolle A., 2005). The regulation on the evaluation of education (150/2003) emphasizes the status of evaluation as a means of developing education. According to it the aim of evaluation is: 1. to acquire and analyze information to form the basis of nationwide decision-making of the politics for education and the developing of education; 2. to acquire and analyze information to form the basis of local development of education and decision-making and 3. to support the learning of students, the work of the personnel of education and the developing of schools. According to researchers, evaluation means the quantification of the value of phenomena and activity, based on the publicly expressed criteria, its proportioning to the evaluation base (for example House 1993, 4; House & Howe 1999). As Raivola (2000a, 65-66) states, “the function of evaluation is to yield systematic information, which value- and advantage implications are attached to, so that information can be used to direct some social activity to reach the goals set for it even better and more efficiently than before.” The utilization of evaluation information is a part of the effectiveness of the evaluation of education. In education effectiveness generally means the positive success of the education, pursued by action, and the fulfillment of its objectives and functions (Raivola 2000b, 12). 8 According to a division established in evaluation research, the utilization of evaluation information can be instrumental, conceptual or persuasive (Rossi et. al. 2004, Lampinen 1992, 30-37; Leviton & Hughess 1981, 528-548). According to Chelimsky (1997, 10-18), we can pursue three ends with evaluation: fulfilling accountability, yielding new information or promoting development. Evaluation and its effects can also be examined on different dimensions of time, according to whether it concentrates on evaluation during the activity (ex nunc), before it (exante) or after it (ex post) (Evaluating EU Expenditure Programmes 1997). In the analysis of the article in question and the data of the study related to it, information on the project before, during and after will be partly yielded. In terms of time, the evaluation of different development projects can be carried out in several different stages of the project being evaluated. Traditionally ex post evaluation has been extensively utilized in the evaluation of different development projects, in which case the evaluation has often been concentrated on the effectiveness and implications of the outcome of the development projects. The content themes of evaluation in ex post evaluation are significantly related to the reaching of the set goals, change comparisons, viewpoints on learning and economic and functional things. (Anttila, 2006, 2007) It has been noticed that real-time evaluation of development projects has become common. In realtime evaluation operations taking place during the project are often tried to be clarified and the short-term results of the operations and the effectiveness on the operational processes and the actors are tried to be clarified in real-time. Additionally, ex post evaluation can be more and more utilized in respect to projects, for example so that the preliminary needs and views of the actors under operations are tried to be found out. On the basis of actor feedback, the content of the operations to be done in the project can be planned and guided in real-time, tailored to the actors (Lindström, 1994). Vaherva (1983) considers the effectiveness of education to mean the achievement of the investment-, process- and output objectives set in detail for education in a certain, defined time span (long and/or short span). This assumes that the effectiveness of education can take place on different levels, for example the readiness brought by education can strengthen the mental and functional growth of the individual and more extensively, effectiveness can be manifested on the 9 organizational level as changes in productivity, excellence and networking, and extensively in society as the development of working life and well-being. When the frame of reference of effectiveness evaluation is examined, the four-step model of the effectiveness evaluation of education, developed by Kirkpatrick (2006, org. 1959), is often brought up. On the basis of Kirkpatrick’s original model (2006, org 1959), plenty of similar evaluation models of the effectiveness of education have been created. They, however, mainly include the central elements of Kirkpatrick’s classification. Kirkpatrick’s (2006) effectiveness levels of education can be divided into the following levels: a. The Level of Reactions; In the evaluation of the level of reactions the satisfaction of the trainees in the education in relation to their earlier opinions and experiences is clarified. On the other hand, even positive satisfaction results do not necessarily guarantee the learning of the participants in general or changes in their thinking and actions. Practical relevance: With the received feedback those realizing the education can develop student-inspired education, emphasizing the viewpoints of content and the actor-sensibility of learning in the future. b. The Level of Learning; With the evaluation of the level of learning the lore, skills and attitudes learnt by the trainees are clarified. Learning can be measured in different ways, such as written feedback (evaluations), tests, skills assessments, portfolios and different preliminary and final level measures/enquiries, learning diaries and reflection discussions. Practical relevance: The evaluation of the level of learning is quite reliable in terms of short and intensive courses. On one hand, reliability can suffer when the length of education increases. On the other hand, the evaluation of the level of learning does not often yield information on the using of what has been learnt in the actor’s own task or organizational behavior. Nor does the meaning of the things learnt necessarily unfurl actor-specifically if the actor has not brought one’s own meanings up in terms of things. c. The Level of Behavior: The evaluation of the level of behavior tries to yield information on the application of the things learnt in the education/on the course into one’s own work and /or functional behavior. Functional and behavior-related changes are often connected to the participants’ prior knowledge, knowhow, attitudes and motivation and the prowess to apply the new, learnt things. 16 towards entrepreneurship go to teachers’ entrepreneurship coaching, not these others… Luckily it has been said during the interviews: II. the attitude of the teaching personnel towards entrepreneurship has become more favorable. Respondents also highlighted that very many teacher who still have negative attitude towards entrepreneurship think that it is something very difficult to do: III. resistance towards change is apparent, new things are experienced as a threat because they are not known and people are wondering how much they will increase the amount of work. They do not necessarily see that the new will bring new possibilities. 7. THE OVERALL EVALUATION OF THE “EDC” PROGRAM AND THE FINAL CONCLUSIONS The evaluation of the overall effects of the clarification is based on the evaluation of the change in the thoughts, actions and behavior of the persons having participated in the training, according to Kirkpatrick’s classification. In evaluation study, by using Kirkpatrick’s method, it was possible to use 1, 2 and 3 levels but not the fourth level. The central needs and challenges of the development of the trainings that have possibly affected the satisfaction/dissatisfaction towards the training, brought up by the respondents, have been classified in the following conclusions: the training created an increased need for information, some of the respondents wished for a stronger application of the knowledge gained in the training into practice, time resources for the internalization of the know-how and the development of one’s own thinking are needed from manager of the school, even more functional content into the trainings and more sharing of the good practices and examples are wished. According to the training assessment, it would seem that based on the training feedback it have succeeded in the “EDC” program of the project to create strong prerequisites for learning and 17 positive learning experiences for the learners by finding good instructors, who have used teaching methods supporting learning and been able to create an interactive atmosphere in the training. When evaluating the impressiveness of the training on the level of Learning, three main themes that can be seen to have changed along with the training emerge from the data. The themes were grouped as a change in 1) pedagogy, 2) thinking and 3) action. It can be said that along with the training and partly thanks to it the changes were mainly positive. New or developed pedagogic experiments had been made, thinking and attitudes towards entrepreneurship had changed in a more positive direction and an approach clearly supporting entrepreneurship had emerged in the activity. The evaluation on level 3 behavior it was already got onto a deeper level of interpretation. Also in this stage the data was classified into main themes, with the help of which we try to discern and find the dimensions of the impressiveness of the training. There were five main themes: 1. expectation of organization, 2. team work and co-operation, 3. change of teacher’s thinking and role. 4. the new entrepreneurial pedagogy and 5. teacher’s entrepreneurial attitude. It could be found out that all themes proved the “EDC” program was succeeded well and changes were quite positive but not in theme 5. The teacher’s entrepreneurial attitude should still change and there is quite lot of work to do. It can be stated that the findings yield a good basis for developing entrepreneurship. There is still a lot to be done, that is, change has begun and now we need a communal- and organizational development approach, for which the finding of resources is also essential. The speed of the change of the role of behavior, thinking and the teacher is well under way. The expressions of entrepreneurial pedagogy, a new kind of learning culture and the enterprise-school co-operation has started. Development is still needed in the changing of attitudes. As the next step, according to Kirkpatrick’s evaluation method, have to see how the coaching operates in the community. It will only be able to access the level 4 Behavior on the organization – 18 interpretation when at least three – five years were passed since the coaching ended, preferably more. The perception of organizational level –changes will only become possible when all the levels of the organization have been gone through with the theme of entrepreneurship and the issue has been taken from the individual level into the structures. 19 References A150/2003 koulutuksen arvioinnista. Suomen säädöskokoelma. Anttila, P. (2006). Tutkiva toiminta ja ilmaisu, teos Tekeminen, Akatiimi oy, Hamina. Anttila, P. (2007). Realistinen evaluaatio ja tuloksellinen kehittämistyö, Artefakta 19, Akatiimi oy, Juvanes Print, Tampere. Chelimsky, E. 1997. The coming transformations in evaluation. Teoksessa E. Chelimsky & W. R. Cook, T. 1997. Lessons learned in evaluation over the past 25 years. Teoksessa E. Chelimsky & W. R. Evaluating EU Expenditure Programmes: A Guide. 1997. Ex post and Intermediate Evaluation Directorate- General XIX – Budgets, European Commission. European Commission. First edition. European Commission, Directorate-General for the Budget (2004): Evaluating EU activities: A practical guide for the Commission services. http://ec.europa.eu/budget/library/publications/financial_pub/eval_activities_en.pdf ETUI-REHS (2004): Project monitoring and evaluation – guidelines. http://www.etui.org/education/Resources/Project-monitoring-and-evaluation-guidelines Fayolle, A. 2005: Evaluation of entrepreneurship education: behaviour performing or intention increasing? Int. J. Entrepreneurship and Small Business, Vol. 2, No. 1, 2005 89 – 98 Gibb, A. 1993. The Enterprise Culture and Education. International Small Business Journal, 11 (3), 11-34. Gustafsson-Pesonen, A. and Remes, L 2011. Changing teachers’ thinking and actions -frame. Entrepreneurial teacher Education in Entrepreneurship Education. Paper in EFMD Tallinn 2011. Gustafsson-Pesonen, A. and Remes, L 2011. Changing teachers’ thinking and actions -frame. Entrepreneurial teacher Education in Entrepreneurship Education. Paper in ICSB Stocholm 2011. Haahti A, 1989: Entrepreneurship strategic orientation. Modelling Strategic Behaviour In small Industrial Owner-managed firms. The Helsinki School of Economics and Business Administration, Helsinki. Acta Academiae Oeconomicae Helsingiensis. Series A: 64 House, E. 1993. Professional evaluation. Newbury Park, California: Sage Publications. House, E. & Howe, K. 1999. Values in evaluation and social research. Thousand Oaks, California: Sage Publications. Kirkpatrick, D. and Kirkpatrick, J.D. (2006) Evaluating training programs. The Four Levels. Third Edition. Berret-Koehler Piblishers, Inc. San Francisco. Koiranen, M. 1993. Ole yrittäjä. Sisäinen ja ulkoinen yrittäjyys. Tampere: TTKustannustieto. 20 Koiranen, M. & Peltonen, M. 1995. Yrittäjyyskasvatus. Valkeakoski: Konetuumat. Koiranen, M. & Ruohotie, P. 2001. Yrittäjyyskasvatus: analyyseja, synteesejä ja sovelluksia. Aikuiskasvatus 2 / 2001. 102–111. Kyrö, P. 1997. Yrittäjyyden muodot ja tehtävä ajan murroksessa. Väitöskirja. Jyväskylä Studies in Computer Science, Economics and Statistics. 38. Jyväskylän yliopisto. Kyrö, P. & Ripatti, A. 2006. Yrittäjyyden opetuksen uudet tuulet. Teoksessa Kyrö, P. & Ripatti, A. (toim.) Yrittäjyyskasvatuksen uusia tuulia. Yrittäjyyskasvatuksen julkaisusarja 4/2006, Tampereen yliopiston kauppakorkeakoulu, Hämeenlinna, 10–31. Kyrö, P., Lehtonen, H. & Ristimäki, K. 2007. Yrittäjyyskasvatuksen suuntia etsimässä. Teoksessa Kyrö, P., Lehtonen, H. & Ristimäki, K. (toim.) Yrittäjyyskasvatuksen monia suuntia. Yrittäjyyskasvatuksen julkaisusarja 5/2007, Tampereen yliopiston kauppakorkeakoulu, Hämeenlinna, 12–31. Lampinen, O. 1992. The utilization of social science research on public policy. Suomen Akatemian julkaisuja 4/92. Helsinki: VAPK-kustannus. Leviton, L. & Hughes, E. 1981. Research on the utilization of evaluations. Evaluation Review 4 (5), 525–548. Lindström, K. (1994). Työyhteisön kehittämisen toiminnan arviointi, Lindström (toim.) Terve työyhteisö – kehittämisen malleja ja menetelmiä, Työterveyslaitos, Helsinki. Raivola, R. 2000a. Tehoa vai laatua koulutukseen? Helsinki: WSOY. Raivola, R. 2000b. Vaikuttavuutta koulutukseen. Suomen akatemian koulutuksen vaikuttavuusohjelman tutkimuksia. Helsinki: Edita. Remes, L. 2001. Yrittäjyyskasvatus pedagogisessa toimintatehtävässä. Aikakauskirja Kasvatus 4/2001, 168-181. Remes, L. 2001. Yrittäjyyskasvatus pedagogisessa toimintatehtävässä. Kasvatus journal 4/2001, 168-181. Remes, L. 2003. Yrittäjyyskasvatuksen kolme diskurssia. Väitöskirja. Studies in Education, Psychology and Social Sciences 213. Jyväskylän Yliopisto. http://selene.lib.jyu.fi:8080/vaitos/studies/studeduc/9513914267.pdf Ristimäki, K. 1998 a. Yrittäjyyskasvatus Kokkolan kouluissa. tutkimus opettajien yrittäjyysasenteista ja -arvoista. Jyväskylän yliopisto. Kokkola: Chydenius-Instituutti. Ristimäki, K. 2004. Yrittäjyyskasvatus. Järvenpää: Yrityssanoma Oy. 90 Ristimäki, K. 2007a. Yrittäjyyskasvatuksen osalliset. Teoksessa Kyrö, P., Lehtonen, H. & Ristimäki, K. (toim.) Yrittäjyyskasvatuksen monia suuntia. Yrittäjyyskasvatuksen julkaisusarja 5/2007, Tampereen yliopiston kauppakorkeakoulu, Hämeenlinna, 32–43. 21 Rossi, P., Lipsey, M. & Freeman, H. 2004. Evaluation: A systematic approach. Seventh Edition. Thousand Oaks, California: Sage Publications. Seikkula-Leino, J. 2007a. Opetussuunnitelmauudistus ja yrittäjyyskasvatuksen toteuttaminen. Opetusministeriön julkaisuja 2007:28. Seikkula-Leino, J. 2007b. Yrittäjyyskasvatuksen tulevaisuus – tutkimus, opetussuunnitelma ja käytäntö. Päätössanat. Yrittäjyyskasvatuksen tutkija- ja toimijatapaaminen Vaasassa 10. helmikuuta 2007. lipas.uwasa.fi/yrittajyyskasvatus/paatossanat_Seikkula-Leino.doc Vaherva, T. (1983). Koulutuksen vaikuttavuus, käsiteanalyyttistä tarkastelua ja viitekehyksen hahmottelua, Jyväskylän yliopiston, Kasvatustieteiden laitoksen julkaisuja.