A Framework proposal for monitoring and evaluating training in ERP implementation project

1 A Framework Proposal for Monitoring and Evaluating Training in ERP Implementation Projects Technical Research Report July 2002 José Esteves Universidad Politécnica Catalunya, Barcelona, Spain, jestev[email protected] Joan Pastor Universidad Internacional de Catalunya, Barcelona, Spain, [email protected] Josep Casanovas Universidad Politécnica Catalunya, Barcelona, Spain, [email protected] Abstract. During the last years some researchers have studied the topic of critical success factors in ERP implementations, out of which ‘training’ is cited as one of the most ones. Up to this moment, there is not enough research on the management and operationalization of critical success factors within ERP implementation projects. This technical research report proposes a framework for monitoring and evaluating training in ERP implementation projects. In order to develop a set of metrics for such monitoring and evaluating tasks, we have used the Goals/Questions/Metrics (GQM) approach. The GQM approach is a mechanism for defining and interpreting operational, measurable goals. Because of its intuitive nature the approach has gained widespread appeal. As a result, we propose a GQM preliminary plan with different metrics to monitor, control and evaluate training while implementing an ERP system. We also propose a three dimensional framework to interpret the metrics defined. Keywords: Enterprise Resource Planning, critical success factors, training metrics, GQM, ERP implementations 2 Index: 1. INTRODUCTION...............................................................................................................3 2. RESEARCH METHODOLOGY.........................................................................................4 3. ERP TRAINING OVERVIEW............................................................................................4 3.1 TRAINING METHODS......................................................................................................5 3.2 TRAINING SCHEDULE.....................................................................................................5 3.3 TRAINING CURRICULUM.................................................................................................5 3.4 TRAINING EVALUATION .................................................................................................6 3.4.1. Training Evaluation at Project Level.......................................................................6 3.4.2. Training Evaluation at Organizational Level............................................................7 3.5 KIRKPATRICK EVALUATION TRAINING MODEL................................................................8 3.5.1. Reaction.................................................................................................................8 3.5.2. Learning................................................................................................................8 3.5.3. Behaviour..............................................................................................................9 3.5.4. Results...................................................................................................................9 3.6 AN ERP TRAINING EVALUATION FRAMEWORK PROPOSAL............................................. 10 3.6.1. Training Evaluation Dimensions............................................................................12 3.6.2. ERP User Roles....................................................................................................12 3.6.3. ERP Implementation Phases..................................................................................12 3.7 TRAINING AS A CONTINUOUS PROCESS.......................................................................... 13 4. A GQM PRELIMINARY PLAN FOR TRAINING ON ERP IMPLEMENTATIONS......14 4.1 GQM METHOD OVERVIEW........................................................................................... 14 4.2 GOALS OF THE GQM PRELIMINARY PLAN..................................................................... 14 4.3 QUESTIONS.................................................................................................................. 15 4.3.1. Project level.........................................................................................................15 4.3.2. Project Level: Management Activities....................................................................15 4.3.3. Reaction Level......................................................................................................16 4.3.4. Learning level......................................................................................................17 4.3.5. Behaviour level.....................................................................................................17 4.3.6. Results level.........................................................................................................17 4.3.7. Costs level............................................................................................................17 4.4 DESCRIPTION OF METRICS............................................................................................ 18 5. CONCLUSIONS...............................................................................................................19 6. ACKNOWLEDGEMENTS...............................................................................................19 7. REFERENCES..................................................................................................................19 8. APPENDIX A - METRICS DESCRIPTION .....................................................................22 3 1. Introduction Training is one of the most cited Critical Success Factors (CSFs) in Enterprise Resource Planning (ERP) implementation projects (Bancroft et al. 1998, De Bruin 1997, Gibson and Mann 1998, Sumner 1999, Kale 2000). In order to realize significant benefits from ERP systems a considerable amount of training is required (Wortmann 1998). There must be a training plan and it should take into consideration both technical staff and end-users, with its scope depending on the type of implementation approach selected. Some case studies of ERP implementations have shown the importance of effective training at all levels (e.g. Bancroft et al. 1998, Miller 1999, Kale 2000). However, little has been done in relation to the management and the operationalization of training metrics. Usually, the metrics proposed in the ERP implementation methodologies are related with milestones and costs aspects. Some organizations use an in-house training approach while others prefer to use training consultants from outside (Esteves and Pastor 2000). According to Esteves and Pastor (2002), training activities in a SAP implementation are in the ranking of the most critical activities. Koch (1996) mentioned that “without proper training, about 30 to 40 percent of front-line workers will not be able to handle the demands of the new system”. Hence, ERP systems are complex and demand rigorous training. As Bingi et al. (1999, p. 14) says: ”it is difficult for trainers or consultants to pass on the knowledge to the employees in a short period of time“. From the project management perspective a training evaluation activity may seem to be an add-on, a luxury, another costly element of a project consuming resources. Unlike training monitoring which focuses on technical aspects of delivery of training and control of planned variables. According to Goldstein (1993, p. 9), “most organizations do not collect the information to determine the utility of their own training programs”. We think that the same happens in the case of ERP implementation projects, especially when managers are only worried with the training costs rather than measuring its effectiveness. One of the most important benefits on evaluating training is that it “can serve as a diagnostic technique to permit the revision of programs to meet the large number of goals and objectives” (Mann and Robertson 1996, p. 15). Other benefits gained by evaluating training affect decision making, particularly because evaluations can help decide between alternative training programs and to decide who should participate in future programs (Mann and Robertson 1996). Some of the main arguments for better evaluation of training are: to validate training as a business tool, to justify costs incurred in training, to help improve the design of training, and to help in selecting training methods. This research study attempts to provide a set of metrics to control and monitor training in ERP implementation projects in order to help managers to achieve success in their projects. According to Jurison (1999, p. 28) the purpose of project control is: "to keep the project on course and as close to the plan as possible, to identify problems before they happen and, implement recovery plans before unrecoverable damage is done". Sandoe et al. (2001, p. 164) pointed out that "having both business and project measures to show progress may be the momentum that keeps the project on track at critical times and keeps the project motivated to meet deadlines". As a result of this study, we are interested in a set of metrics to help managers understand the situation of the ERP project. We used the Goals/Question/Metric (GQM) method to develop this set metrics. The result of the application of this method is a GQM plan. The GQM plan is a document that contains the goals, questions, and metrics for a measurement program (Solingen and Berghout 1999), in this case an ERP implementation project. 4 This technical research report is organized as follows. First, we present the research methodology used. Next, we present the background in training. Then, we describe the GQM method and we present the GQM plan proposed. We also propose a framework to interpret the metrics defined. Finally, we present some conclusions and further work. 2. Research Methodology The purpose of this study is to develop a set of metrics to control and monitor training issues in ERP implementation projects. We used the GQM method to develop a measurement program. The steps were: § Literature review on training topic. § Definition of goals related with training in ERP implementation projects. § Definition of questions associated for each goal. § Definition of metrics associated to each question. § Definition of the preliminary GQM plan. § Interpretation of metrics. A literature review on the training topic as related to ERP implementations was made in order to acquire knowledge related with this CSF. The information provided by the literature was the main source of information. We used the concept of preliminary GQM plan due to the fact that the project team that is going to use it must validate the final GQM plan. Here, we only provide a proposal for this plan. 3. ERP Training Overview Training is important in an ERP implementation project not only to adapt users to the new ERP system but also to help in the organizational change process. Thus, training has at least two important advantages: § It helps users to obtain knowleged of the ERP system before it is going live, what reduces uncertainty and allows a better participation in the implementation project. § The fact of a person being called for training allows him/her to create the feeling that organization stills interested on his/her work, reducing the feeling of loosing their jobs or work segregation. ERP implementation projects engender large training requirements. Training is one of the most costly components of an ERP implementation project (Esteves et al. 2001a). According to Kale (2000), a training program may easily involve 10 to 20% of the manpower of an organization. An organization may have to undertake ERP training for three different groups (Kale 2000): § The managerial personnel who are key members of their departments and have been nominated as the members of the functional team. § Key-users who form the core of the super-users to be entrusted with the task of life-scale testing the system and also training other end-users. § All other end-users who would be using the system as part of their routine operational duties. There is another other important group that requires attention, the trainers. They should be carefully selected, since in most projects, trainers demonstrated lack of experience, and courses were inadequately structured, usually because those trainers were junior consultants.Trainers should be natural teachers, must be experts in the business processes they are teaching, and must recognize and understand the relationships between their processes and all the others (Bancroft et al. 1998). Another 5 important task they perform is training evaluation. Therefore, is also important to analyse if they have the skills, time, or resources to do the training evaluation. 3.1 Training Methods Training may include classroom instruction supplied by the ERP vendor or from Web, interactive, and other distance learning courses. It is helpful to ask referral ERP users what training tools proved most useful to them. “the company’s ERP evaluation committee should consider what kinds of training are available from prospective vendors and what percentage of the total cost of the new system should be budgeted for training. Many vendors recommend at least 10 to 15 percent (McAlary 2000, Kale 2000). Some recommend an estimated number of 120 hours per person. It is likely that the training investment will help drive the rollout plan. That is, the more you spend on training, the faster your rollout may be accomplished.” (McAlary 2000). Users are most often trained by a training staff that first learned how to use the ERP system during the project pilot phase. A key benefit in this approach is that after the initial training plan, users and trainers (in this case people from the organization) will work side-by-side (Bancroft et al. 1998). User training is ideally performed on the customer’s premises, using examples from the organization lineof-business data and the new ERP system. However, it is often difficult to get trainees to sit through day-long, onsite training. They are inclined to run in and out of the training sessions, answering telephone calls and responding to everyday problems, within their departments. This is disruptive and reduces the effectiveness of the training task. The figure of the training manager is very important in an ERP project. His major responsibility is “to ensure the availability not only of the instructors, but also to release the various personnel from their normal functions at appropriate times so that they can participate in the scheduled training sessions” (Kale 2000, p. 233). 3.2 Training Schedule Some case studies of ERP implementations have shown the importance of effective training at all levels (e.g. Bancroft et al. 1998, Miller 1999, Kale 2000). Training should be synchronized with the overall implementation project. Formal training of all users is not normally deployed at the beginning of the implementation. Some organizations embark upon training programs several months before the ERP goes live, by which time users have forgotten their training (De Bruin 1997, Welti 1999). As Gupta (2000) states, poor end-user training is a common problem in all ERP implementations. “The ERP system is ready to go live but no one in the organization knows how to use it, and personnel also lack training in the maintenance aspects of the system” (Gupta 2000, p. 116). According to McAlary 2000, training can take place as late as two weeks before the beginning of the implementation cycle that deals with the trainee’s domain. One way to minimize the problems arisen from the time lag between the actual training and the commencement of ERP going into production is doing refresher courses (Kale 2000). 3.3 Training Curriculum Bancroft et al. (1998) focus the importance of begining the training program with an analysis of the needs of users. The purpose of this analysis is to understand what to present and how to deliver the information. From the results of the analysis, the training team can identify the number and types of 6 training courses. It will know the number of users to be trained, their locations, the number of days for each course, the number of courses to be delivered, and the number of trainers needed. The consensus that is emerging in relation to ERP training is that the training that matters is not technological but rather it must develop the ability to figure out the underlying flow of information through the business itself (Wheatley 2000). This is not just training in using the new ERP system, but also in the new processes and in understanding the integration within the ERP system – how the work of one employee influences the work of others (Krammegaard and Moller 2000, Wheatley 2000). The implementation of an ERP system involves a lot of organizational change in organizations. When organizations are asked what they would have done differently, most respond that they would have offered more training on how the system would change processes and to use the system (Kraemmergaard and Moller 2000). Therefore, there is the need to incorporate specific training related with it. Bancroft et al. (1998, p. 138) pointed out “users must be informed as to the business needs for such change as well as which keys to when”, where change refers to the reengineered process associated with the new ERP system and the novel way of operating. We call this type of training organizational training. Most consultants suggest that end-users should be trained by using the new ERP parameterization according to the organization needs as this provides a better and quickly adaptation to what they will find after the go-live phase. Another important aspect is the commitment of users to training. In terms of short term perception of training skills transfer, Axtell and Maitlis (1997) mention that if new skills are to be transferred to the workplace, trainees first need to feel that the training course is relevant to their jobs, and must also be committed to using what they have learned. Therefore, is important at the beginning of the ERP training program to explain to users what are the objectives and benefits of training. After one year, the factors influencing the transfer of training skills are the degree of autonomy in their jobs and their original motivation to use what they have learned. Thus, the predictors of training skills transfer after one year are different from those just after the training course (Axtell and Maitlis 1997). 3.4 Training Evaluation According to Brinkerhoff (1988), a good training evaluation should be able to prove that the program: • Is aimed at important and worthwhile organizational benefits; • Operates smoothly and effectively and is enjoyed by participants; • Achieves important skills, knowledge and attitude objectives; • Uses the best available and most cost-effective designs; • Is used effectively on the job; and • Provides valuable and cost-effective organizational benefits. Training evaluation can be analysed at least at two levels: project level and organizational level. Project level concerns to the training program as a whole while organizational level concerns to the people involved in the training and the effects on the organization. 3.4.1. Training Evaluation at Project Level Mulder (2001) proposes a model for training effectiveness were he distinguishes three variables: project definition, project implementation and project effects. For each variable he defines a set of items (see table 1). The model is oriented to evaluate a training program from a perspective of a 7 training organization and its clients. This is suitable for ERP implementation contexts since most of the training is given by external training organizations during the ERP implementation project. Objective operationalisation The level in which the objectives of the training project are specified. Distribution of responsibility The level in which the training organization carries responsibility for attaining the results. Project Definition Condition registration The delivery conditions about which agreements have been made between the client organization and the training organization. Total satisfaction This is the satisfaction about the total project handling, the preparation of the project. Condition-realisation consistency The delivery reliability of the training organization. Project Implementation Condition-realisation satisfaction The level of satisfaction about the performance of the training organizations with respect to the possible delivery conditions. Expectation realisation The level in which the project results meet the expectations of the client organization. Objective realisation The level in which the intended objectives of the training project are achieved. Project Effectiveness Success attribution The level in which the training organization has been responsible for attaining the intended objectives. Table 1 - The model of training effectiveness proposed by Mulder (2001). The model proposed by Mulder (2001) focus on the effectiveness of training programs perceived by customers in relation to the training organisations. We extended this model by defining another variable which is related with the management and monitoring of the training program itself. We denominate this variable as management activities since it concerns to the management activities associated with the training plan/project. This variable has three important entities: courses (type of courses, goals, number of hours, type of instructional method and number participants per course), type of users and trainers. These three entities should be managed and integrated in a metrics program. 3.4.2. Training Evaluation at Organizational Level Different approaches to training evaluation have been proposed. Following the six general approaches to educational evaluation: goal-based, goal-free, responsive, systems, professional and quasi-legal, Eseryel (2002) categorized and compared the most common training approaches. The most used training evaluation model was proposed by Kirkpatrick (1959) which follows the goal-based evaluation approach and is based on four levels of evaluation. Later, we describe in detail this model. Under the systems approach, the most influential models include: context, input, process product (CIPP) model (Worthen and Sanders 1987); Training Validation System (TVS) approach (Fitz-Enz 1994); and Input, Process, Output, Outcome (IPO) model (Bushnell 1990). According to Philips (1991), goal-based and systems-based approaches are predominantly used in the evaluation of training. Eseryel (2002) mentions that goal-based models may help practitioners think about the purposes of evaluation ranging from purely technical to covertly political purpose. However, he argues that these models do not define the steps necessary to achieve purposes and do not address the ways to utilize results to improve training. Systems-based models seem to be more useful in terms of thinking about the overall context and situation but they may not present the dynamic interaction between the design 8 and the evaluation of training. Again, few of these models provide descriptions of the processes involved in each steps. Furthermore, these models do not address the collaborative process of evaluation, i.e., the different roles and responsibilities that people may play during and evaluation process. 3.5 Kirkpatrick Evaluation Training Model Kirkpatrick created his model in 1959 but it is still the most used and accepted evaluation training model. His model focuses on “what” must be evaluated. Kirkpatrick (1998) has identified four levels of outcomes of training which are hierarchically ordered: reaction, learning, behaviour and results. Since the model was developed in 1959, several training researchers have augmented the model. Kirkpatrick model is especially suited to training contexts where specific outputs are of interest right from the outset (Athanasou 1998). Next we describe these four levels of training taking into account the improvements made by other authors. 3.5.1. Reaction This level assesses the initial reactions of participants to a course. This, in turn, offers insights into participants’ satisfaction with a course, a perception of value. Trainers usually assess this through a survey, often called a ‘smiley sheet’. Warr and Bunce (1995) described three kinds of reactions that are measured: § Enjoyment of training (emotional reaction) – “I found this training program to be enjoyable”. The reactions focus on how trainees view a program as enjoyable. § Usefulness of training (perceived value) – “What level of values does the training content have for your job?”, this kind of reaction attempts to ascertain the perceived utility value, or usefulness, of training for subsequent job performance. § Difficult of training – “I found the issues taught in training difficult to understand”. These reactions cover the cognitive and emotional effort required to perform well in the training. A similar categorization is proposed by Alliger et al. (1997). However, in there case, they proposed only the emotional reaction and perceived value, and the third one is a mix of both. This level is usually evaluated after the training is completed. Warr et al. (1999) propose a fourth state: the extent to which individuals are motivated to apply the material they have learned. According to Noe (1986) there is often scope for variation in the degree to which course material is later apllied in a job, and motivational differences between individuals at the end of a course may be linked to variations in subsequent job behaviour. Therefore, an additional measure is should be taken at the end of training, recording motivation to transfer course into a job setting. 3.5.2. Learning This level measures the amount of learning that results from training and determines how much behaviour can change back on the job. In most cases, focus is upon the acquisition of knowledge, but much training also has as an objective the modification of attitudes and values. Kraiger et al. (1993) identified three types of learning that might result from training: § Cognitive outcomes – An evaluation of cognitive outcomes must focus on trainee knowledge and the processes of knowledge acquisition,organization and application. Usually assessed by multiple choice questions, open-ended responses, listing of facts, or similar methods. Knowledge checks such as these are very similar to tests used in schools to assign grades. 9 Measures of cognitive outcomes can be assessed immediately after training or later to assess knowledge retention over time. § Skill-based outcomes – Kraiger et al. (1993) describe this category of outcomes as the development of technical or motor skills and is frequently measured by observing trainee performance in role plays or actual job behaviors. These outcomes are typically measured by requiring that participants demonstrate their new skills in the training environment. Skillbased outcomes are not the same as behavior-change outcomes (level 3) that occur in the work environment. Learning outcomes that focus on skills only measure participants’ ability to demonstrate the skills. § Attitudinal outcomes – these measures focus on how participants feel or think about the training content. They have implications for participants’ motivation to use the training, their confidence for using the skills, and their ability to reach goals. Kraiger et al.(1993)define affectively based measures of training evaluation as variables measuring attitudes,motivation,and goals that are relevant to the objectives of the training program;they further state,measures of attitudes should indicate the direction (agree or disagree)and strength of the reaction to an object. 3.5.3. Behaviour This level measures the degree of transfer from what was learned to how the trainee behaves on the job, which in turn determines how much organizational impact the training can have. This assessment is based on the objectives of the course and these assessed through tests, observations, surveys, and interviews with co-workers and supervisors. Based on a literature review, Olsen (1998) summarized a set of factors which can influence behaviour: integration of training to the work setting rather than as an isolated occurrence, cues and reinforcement, a connection to the reward system, close and frequent supervisory (coaching and nurturing) feedback, group dynamics, employee attitudes about the work and the organization, the type of training conducted, and consistency between what is being trained and its applicability in the real job setting. 3.5.4. Results This level is a measure of organizational and business impacts of the training. Some assess this measurement by tracking business measurements, others assess it by observations, some by surveys and still others assess by qualitative measures. Examples of might include: productivity, customer satisfaction, efficiency, morale, and profitability. The last level is the most difficult level to assess. First, most of training courses do not have explicitly written their organizational/business goals; second, the methodology for assessing the impact is not yet refined; and third after a long period after training occurs, evaluators have difficulty solely in attributing changes in business results to training. An effective evaluation of a training program must evaluate each level in Kirkpatrick’s hierarchy in order to an organization be able to understand the full effects of the training program (Kirkpatrick 1998). A survey from the American Society of Training and Development Benchmarking Forum suggests that most organizations conduct some evaluations at levels one and two and only a few evaluate levels three and four. The survey also found that participating organizations believed there to be diminishing returns from evaluating at levels 3 and 4. Furthermore, there may even be dubious results from measurement at these levels. The results are shown in Table 2. 16 Q13 What is the proportion of organizational training Vs technical training courses? M13 – ratio technical Vs organizational training courses Q14 What is the proportion of organizational training Vs technical training hours? M14 – ratio technical Vs organizational training hours Q15 What is the proportion of training hours by ERP module? M15 – % of training hours of by ERP module Q16 What is the percentage of training courses made according to the training methods planned? M16 – % of training courses made using the training method planned Q17 What is the percentage of training courses made in the location planned? M17 – % of training courses made in the location planned 4.3.2.2. Users We focused on the following aspects related with users: § Realization of training plan for the users planned. § Assistance of users to the training courses. § Realization of number of hours planned for each user. § Realization of training courses with the number of users planned. Q18 What is the percentage of users trained according to the training plan? M18 – % of users trained Q19 What is the percentage of users per course in relation to what was planned? M19 – % nº. users per course Q20 What is the percentage of assistance of users per course? M20 – % of users assistance per course Q21 what percentage of training hours each user will have? M21 – % of nº. hours per user 4.3.2.3. Trainers We focused on the following aspects related with trainers: § Realization of training courses with the number of trainers planned. § Courses realized by each trainer. § Analysis of the quality of trainers. § Type of trainers used. Q22 What is the percentage of trainers per course in relation to what was planned? M22 – % nº. trainers per course Q23 what sorts of people are selected to provide training and how are they chosen? M23 – quality of trainers Q24 What proportion of training courses made by each trainer in relation to what was planned? M24 – ratio of training courses made by trainers Q25 What is the type of trainers, internal or external per course? M25 – type of trainers per course 4.3.3. Reaction Level Q26 Was the training enjoyable? M26 – enjoyment Q27 To what degree will this training influence trainees’ ability later to perform their job? 17 M27 – usefulness Q28 Was this training course relevant? M27 – usefulness Q29 Was the training of practical value? M27 – usefulness Q30 Was the training difficult? M28 – difficulty of training Q31 Are trainees motivated to apply the material they have learned? M29 – motivation to transfer 4.3.4. Learning level For this level, we adopt the three types of learning suggested by kraigers et al. (1993): cognitive outcomes, skill-based outcomes and attitudinal outcomes. Q32 What Knowledge was learned? M30 – cognitive outcomes Q33 What skills were developed or improved? M31 – skill outcomes Q34 Were attitudes, self-efficacy, or motivation changed?? M32 – attitudinal outcomes 4.3.5. Behaviour level Q35 What skills, knowledge, or attitudes have changed? M33 – Behaviour 4.3.6. Results level Q36 Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? M34 – results 4.3.7. Costs level Q37 What is the overall cost of training? M35 – total cost of training Q38 Are there enough resources for the training program? M36 – training budget 18 4.4 Description of Metrics In this section we show the relationship between the questions defined above and the metrics (see table 4). In appendix A we present a detailed description for each metric using a special form. 1 Objective realization Q1 2 Distribution of responsibility Q2 Project Definition 3 Condition registration Q3 4 Total satisfaction Q4 5 Condition-realization consistency Q5 Project Implementation 6 Condition-realization satisfaction Q6 7 Expectation realization Q7 8 Objective realization Q8 Project Effects 9 Success attribution Q9 10 % training courses made Q10 11 % courses made according to schedule plan. Q11 12 % training hours made Q12 13 ratio technical Vs organizational training courses Q13 14 ratio technical Vs organizational training hours Q14 15 % of training hours of by ERP module Q15 16 % of training courses made using the training method planned Q16 17 % of training courses made in the location planned Q17 18 % of users trained Q18 19 % nº. users per course Q19 20 % users assistance per course Q20 21 % nº. hours per user Q21 22 % nº. trainers per course Q22 23 quality of trainers Q23 24 ratio training courses made by trainers Q24 Project level Management Activities 25 type of trainers per course Q25 26 Enjoyment Q26 27 Usefulness Q27, Q28, Q29 28 Difficulty Q30 Reaction Level 29 Motivation to transfer Q31 30 Cognitive outcomes Q32 31 Skill outcomes Q33 Learning Level 32 Attitudinal outcomes Q34 Behaviour Level 33 Behavioral Q35 Results Level 34 Results Q36 35 Total cost of training Q37 Costs Level 36 Training budget Q38 Table 4 - The definition of metrics and their relationship with questions. 19 5. Conclusions This study attempts to define a first set of metrics for training in ERP implementation projects. Training is cited as one of the most relevant CSFs in ERP implementation projects. We think these metrics have two important proactive characteristics: metrics help to detect deviations from the project plan and to act before damage is made, and second, the feedback from the participants is important for the training manager and the training team for improvement for subsequent training causes. They can analyse the feedback and possibly incorporate improvements in the training plan. The purpose of this study is not to describe an exhaustive list of metrics. Instead, we attempt to present a form to develop these metrics in future ERP implementation projects and we provided the first set of metrics that should be extended and adapted according to the specific needs of ERP implementation projects. Managers should analyzed for each metric if it is relevant in their context and if there are the resources to measure it. Next steps in this research are the validation and interpretation of these metrics. Two possible kinds of validation methods can be applied: case study or control experiments (Calero et al. 2001). We would like to remark that we are conscientious that this GQM preliminary plan will be subject to changes during the next steps of the research due to new information gathered and experience gained in the feedback sessions. Currently, we are developing a software application for the management of the metrics defined here. Additional research will attempt to define metrics to other CSFs defined in the literature of ERP implementation projects. 6. Acknowledgements This work has been partially supported by PRONTIC CICYT program project Tic2001-2165. 7. References Alliger G., Tannenbaum S., Bennett W. 1996. “A comparison and integration of three training evaluation approaches: Effectiveness, utility, and anticipatory evaluation of training”. Interim technical report. Brooks Air Force Base, TX: Armstrong Laboratory, Air Force Materiel Command Alliger G., Tannenbaum S., Bennet W., Traver H., Shotland A. 1997. “A Meta-Analysis of the Relations among Training Criteria”, Personnel Psychology Journal, No.50, pp. 341-358 Athanasou J. 1998. “A Framework for Evaluating the effectiveness of technology-assisted learning”, Industrial and commercial Training, 30(3), 1998, pp. 96-103 Axtell C., Maitlis S. 1997. “Predicting Immediate and Longer-term Transfer of Training”, Personnel Review, 26(3), 1997, pp. 201-213 Bancroft N., Seip H., Sprengel A. 1998. "Implementing SAP R/3", 2nd ed., Manning Publications Basili V., Caldera C., Rombach H. 1994. "Goal Question Metric Paradigm", Encyclopedia of Software Engineering Marciniak J. editor, Vol 1, John Wiley & Sons, pp. 528-532 Bingi P., Sharma M., Godla J. 1999. “Critical Issues Affecting an ERP Implementation”, Information Systems Management, 16(3), 1999, pp. 7-15 Brinkerhoff R. 1988 “Achieving Results from Training”, Joss-Bass, San Francisco Ca. Bushnell D. 1990. “Input, Process, Output: A Model for Evaluating Training”, Training and Development journal, 44(3), march 1990, pp. 41-43 Calero C., Piattini M., Genero M. 2001. "Method for Obtaining Correct Metrics", Third International Conference on Enterprise Information Systems, pp. 779-784 20 Davenport T., Short J. 1990. “The New Industrial Engineering: Information Technology and Business Process Redesign”, Sloan Management Review, summer 1990 De Bruin P. 1997. "Unpublished 1997 Sapphire Conference Notes" in Gibson and Mann 1997 Eseryel D. 2002. “Approaches to Evaluation of Training: Theory & Practice”, Educational Technology & Society Journal, 5(2), 2002, pp. 93-98 Esteves, J., Pastor, J. 2000. "Towards the Unification of Critical Success Factors for ERP Implementations", 10th Annual BIT conference, Manchester, UK Esteves J., Carvalho J., Santos A. 2001. "Towards an ERP Life-cycle Costs Model", Information Resources Management Association IRMA International Conference, Toronto Esteves J., Pastor J. 2001. "Analysis of Critical Success Factors Relevance along SAP Implementation Phases ", Americas Conference on Information Systems Esteves J., Pastor J. 2002. “A Framework to Analyze Most Critical Work Packages in ERP Implementation Projects”, International Conference on Enterprise Information Systems, Spain Fitz-Enz J. 1994. “Yes..you can weigh training values”, Training journal, 31(7), July 1994, pp. 54-58 Gibson J., Mann S. 1997. "A Qualitative Examination of SAP R/3 Implementations in the Western Cape", research report, University of Cape Town Goldstein I. 1993. “Training in Organizations”, Brooks/Cole, Pacific Grove Ca. Gupta A. 2000. “Enterprise Resource Planning: The Emerging Organizational Value Systems”, Industrial Management & Data Systems, 100(3), pp. 114-118 Jurison J. 1999. "Software Project Management: The Manager's View", tutorial, Communications of the Association for Information Systems, 2(17), 1999 Kale V. 2000. “Implementing SAP R/3: The Guide for Business and Technology Managers”, Sams Publishing Kirkpatrick D. 1998. “Evaluating Training Programmes: The Four Levels”, Berrett-Koechler Publishers Koch C. 1996. “Surprise, Surprise”, CIO magazine, June 1996 Kraemmergaard P., Moller C. 2000. "Evaluation of ERP Implementation: A Case-Study of an Implementation", International Conference on Information Systems Analysis and Synthesis, Florida, USA Kraiger K., Ford J., Salas E. 1993. “Application of cognitive, Skill-based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation”, Journal of applied Psychology, 78(2), April 1993, pp. 311-328 Mann S., Robertson I. 1996. “What Should Training Evaluations Evaluate?”, Journal of European Industrial Training, 20(9), 1996, pp. 14-20 McAlary S. 2000. “Three Pitfalls in ERP Implementations”, Strategy &Leadership Magazine, October/November/December, pp. 49-50 Miller G. 1999. “ERP Implementation Lessons Learned”, white paper, Proaction Management Consultants Mulder M. 2001. “Customer Satisfaction with Training Programs”, Journal of European Industrial Training, 25(6), 2001, pp. 321-331. Nah F., Lau J., Kuang J. 2001. "Critical Factors for Successful Implementation of Enterprise Systems", Business Process Management Journal, 7(3), 2001, pp. 285-296 Nickols F. 2000. “Evaluating Training: there is no ‘Cookbook’ Approach”, Distance Consulting Company Noe R. 1986. “Trainees’ atributes and attitudes: neglected influences on training effectiveness”, Academy of management review, 11, pp. 736-749. Olsen J. 1998. “The Evaluation and Enhancement of training transfer”, International journal of training and development, 2(1), 1998, pp. 61-75 Olson M., Ives B. 1980. “Measuring User Involvement in Information Systems Development”, International Conference on Information Systems, 1980. 21 Philips J. 1991. “Handbook of training evaluation and measurement methods”, 2nd edition, Houston, USA. Phillips J. 1994. “Measuring Return on Investment”, American Society for Training and Development, 1994. Sandoe K., Corbitt G., Boykin R. 2001. "Enterprise Integration", John Wiley & Sons Solingen R., Berghout E. 1999. "The Goals/question/Metric Method: A Practical Guide for Quality Improvement of Software Development", McGraw-Hill Sumner M. 1999. "Critical Success Factors in Enterprise Wide Information Management Systems Projects", Americas Conference on Information Systems, Milwaukee, USA Warr P., Bunce D. 1995. “Trainee characteristics and the outcomes of open learning”. Personnel Psychology, 48, pp. 347-375 Warr P. Allan C., Birdi K. 1999. “Predicting Three Levels of Training Outcome”, journal of occupational and organizational psychology, 72, pp. 351-375. Welti N. 1999. "Successful SAP R/3 Implementation: Practical Management of ERP Projects", Addison-Wesley Wheatley M. 2000. “ERP Training Stinks”, CIO Magazine, June 2000 Worthen B., Sanders J. 1987. “Educational Evaluation”, New York, Longman, 1987 Wortmann J. 1998. “Evolution of ERP Systems”, International Conference of the Manufacturing Value-Chain, Kluwer Academic Publishers, pp. 11-23 22 8. Appendix A - Metrics Description In this appendix we present the description of all metrics using a special form we created. For each metric we define the following aspects: what they are measuring, when they must be measured, what possible values they could have, who will measure it, what medium is used for data collection. We created a special form for the metrics description. Most of the metrics proposed are direct measurements except the metrics related with percentages. Training level Direct measurement Indirect measurement Project level Nº. training courses planned Nº. training courses made Nº. training courses in time Nº. training hours made Nº. training hours planned Nº. organizational training hours Nº. technical training hours Nº. training hours by ERP module Nº. training courses made by planned training method Nº. training courses made by planned location Nº. of users trained Nº. users planned to be trained Nº. of users per course Nº. training hours per user Nº. trainers per course Nº. trainers planned per course Nº. courses by trainer Nº. courses planned for each trainer Type of trainers per course M25 – type of trainers per course M1 - Objective realization M2 - Distribution of responsibility M3 - Condition registration M4 - Total satisfaction M5 - Condition-realization consistency M6 - Condition-realization satisfaction M7 - Expectation realization M8 - Objective realization M9 - Success attribution M10 – % of training courses made M11 – % of courses made according to schedule plan. M12 – % of training hours made M13 – ratio technical Vs organizational training courses M14 – ratio technical Vs organizational training hours M15 – % of training hours of by ERP module M16 – % of training courses made using the training method planned M17 – % of training courses made in the location planned M18 – % of users trained M19 – % nº. users per course M20 – % of users assistance per course M21 – % of nº. hours per user M22 – % nº. trainers per course M23 – quality of trainers M24 – % training courses made by trainers Reaction level M26 – enjoyment M27 – usefulness M28 – difficulty M29 – motivation to transfer Learning level M30 – cognitive outcomes M31 – skill outcomes M32 – attitudinal outcomes Behaviour level M33 – behaviour Results level M34 – results Costs level M36 - Training budget M35 – Costs of training 23 Metric name M1 - Objective realization Definition This metric assesses the level in which the objectives of the training project are specified. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M2 - Distribution of responsibility Definition This metric assesses the level in which the training organization carries responsibility for attaining the results. Calculation method Frequency Values Percentage Responsible Training manager Medium Manual Type Indirect Metric name M3 - Condition registration Definition This metric assesses the delivery conditions about which agreements have been made between the client organization and the training organization. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M4 - Total satisfaction Definition This metric assesses the satisfaction about the total project handling, the preparation of the project. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M5 - Condition-realization consistency Definition This metric assesses the delivery reliability of the training organization. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M6 - Condition-realization satisfaction Definition This metric assesses the level of satisfaction about the performance of the training organizations with respect to the possible delivery conditions. Calculation method 24 Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M7 - Expectation realization Definition This metric assesses the level in which the project results meet the expectations of the client organization. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M8 - Objective realization Definition This metric assesses the level in which the intended objectives of the training project are achieved. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M9 - Success attribution Definition This metric assesses the level in which the training organization has been responsible for attaining the intended objectives. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M10 – % of training courses made Definition This metric assesses the percentage of training courses made according to the estimated number of training courses. Calculation method Frequency Values Percentage Responsible Training manager Medium Manual Type Indirect Metric name M11 - % of courses made according to schedule plan. Definition This metric assesses the percentage of training courses that were made during the period that was established in the schedule training plan. Calculation method Frequency Values Percentage Responsible training manager 25 Medium Manual Type Indirect Metric name M12 - % of training hours made Definition This metric assesses the percentage of training hours made according to the estimated number of training hours Calculation method Nº. training hours/nº. estimated training hours*100 Frequency Values Percentage Responsible training manager Medium Manual Type Indirect Metric name M13 - ratio technical Vs organizational training courses Definition This metric assesses the proportion of technical courses versus the number of organizational courses Calculation method Frequency Values Ratio Responsible Training manager Medium Manual Type Indirect Metric name M14 – ratio organizational Vs technical training hours Definition This metric assesses the proportion of technical training hours versus the number of organizational training hours Calculation method Nº. technical hours/nº organizational hours Frequency Values Ratio Responsible Training manager Medium Manual Type Indirect Metric name M15 – % of training hours of by ERP module Definition This metric assesses the percentage of training hours done by ERP module. Calculation method Nº training hours by ERP module/nº. training hours*100 Frequency Values Responsible Training manager Medium Manual Type Indirect Metric name M16 – % of training courses made using the training method planned Definition This metric assesses the percentage of courses made according to the planned training method. Calculation method Frequency Values Responsible Training manager Medium Manual Type Indirect