On the Use of Predictive Tools to Improve the Design of Undergraduate Courses

IEEE EDUCATION SOCIETY SECTION Received November 19, 2021, accepted December 26, 2021, date of publication December 31, 2021, date of current version January 10, 2022. Digital Object Identifier 10.1109/ACCESS.2021.3139803 On the Use of Predictive Tools to Improve the Design of Undergraduate Courses JUAN JOSE ACIEGO 1, ANGEL GONZÁLEZ-PRIETO1, IGNACIO GONZÁLEZ-PRIETO 1, ALICIA CLAROS1, MARIO J. DURAN 1, AND MARIO BERMUDEZ 2 1Department of Electrical Engineering, University of Malaga, 29071 Malaga, Spain 2Department of Electrical Engineering, University of Seville, 41092 Seville, Spain Corresponding author: Ignacio González-Prieto ([email protected]) This research was funded by the University of Malaga under the PIE19-21 with the reference PIE19-047. ABSTRACT Although technical competences are fundamental at engineering degrees, industry is also requesting the promotion of transversal capabilities. Consequently, the map of target competences may vary over time, area and location. In this context, the design of an undergraduate course is not a trivial task if the promotion of several competences is desired. When such design is manually performed by the teacher using his/her previous experience, the perspective of the students and the information of the previous scores are usually disregarded. Furthermore, the determination of the optimal times for the different activities becomes complex to satisfy a multi-objective problem that aims at balancing technical and soft skills. This paper proposes the use of a predictive tool to assist the design of the course. On the one hand, the predictive algorithm automatically determines the duration of the different activities to fit a specific map of competences. Moreover, the predictive tool also offers valuable information about the perspective of the student and the influence of previous scores using objective indices. The assessment of the proposal is done in a course of Electrical Machines at the University of Malaga (Spain), confirming the capability of the proposed predictive tool to provide a valuable insight on the subject and to automatically determine the duration of different methodological tools. INDEX TERMS Predictive control, transversal skills, undergraduate courses. I. INTRODUCTION University courses aim at providing students with some specific competences that will enable them to develop a professional career. The selected competences should match and satisfy the society/market needs and allow bachelors/engineers to successfully accomplish their professional role. Engineering degrees have been traditionally focused on training technical skills, but in the last decades the paramount importance of transversal competences has been highlighted and promoted [1], [2]. The effective acquirement of the desired competences depends on the correct selection of the content, methodology and timing. While the content is mostly pre-established by the subject itself, the teacher has flexibility to choose the most appropriate methodological tools and duration of the activities [3]–[5]. The associate editor coordinating the review of this manuscript and approving it for publication was Chin-Feng Lai . The structure of the course is typically done relying on the previous experience of the teacher and according to what he/she expects that students will learn. Unfortunately, what we (teachers) expect to be learnt, what they (students) think they will learn and the competences that are actually acquired, are typically three different things [3]. In some cases, the students’ opinion and scores are simply disregarded, and the course is consequently designed in open-loop mode. In other words, the information that is taken as an input for the course design is exclusively the previous knowledge of the teacher on the subject. In other cases, the teacher takes into account the opinion of the students and their performance, but this information is processed in an informal and nonstructured manner. In such cases the course is designed in a close-loop mode, but the feedback is considered following a non-systematic procedure [4]. Curiously enough, in the context of engineering studies, teachers typically train the students to perform a correct data analysis and an accurate system control using sophisticated algorithms that automate the process [5], [6]. It is a paradox VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 3105 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses however that these tools are scarcely used in an educational context where the student can be regarded as the system, the competences are the objectives, and the teacher performs the control action. In the field of engineering there is a wide variety of techniques to regulate systems and achieve a good match between the target and measured values of the variables of interest [7]–[9]. Among them, predictive control has emerged as one of the most interesting options with great success in the areas such as chemical or electrical engineering [7], [8]. The core idea of predictive techniques is extremely simple: the future values of some variables are estimated for different inputs, finally selecting the control actions that provide a better tracking of some reference values that are set a priori [9]. This work suggests the use of a predictive approach for the design of the course structure, providing an automated process for the timing of the course and valuable information about the subject. It is worth noting that the use of predictions in an educational context is not new. Previous works have used different algorithmic approaches to perform predictions in education. For example, [10] predicts the likelihood of which university a student may enter using artificial neural networks (ANN) based on the academic merits, background and admission criteria. Data mining techniques and ANN have also been proposed in [11] to predict the instructor performance using questionnaires and scores as inputs. On the other hand, the students’ performance has also been predicted using data mining techniques [12] and providing a detailed analysis in virtual learning environments with online teaching [13]. The analysis in [14] also aims at determining the academic performance by considering self-regulated learning indicators and online events engagement. In [12]–[15], the focus is placed on the student or instructor performance, but the predictions do not provide a clear and specific information on how the course should be designed. Although the information from those analyses can be used by the teacher, the correlation between the educational action and the competences acquirement is absent. Similarly, [14] predicts the students disengaged behaviours in online courses and [16] forecasts the achievement of students in Smart Campus. The identity of graduated students [17] or the identification of students at risk [18]–[22] can also be estimated, but this information is not feedbacked to modify the educational action. Many of these proposals follow an algorithmic approach using ANN [10], multi-objective optimization [21], machine learning techniques [22] or Adaptive Neuro-Fuzzy Inference System (ANFIS) [23], to name a few. Even though all the aforementioned works, in a way or another, perform some predictions based on some input data, the forecasting is not used to modify the course structure in a close-loop manner. Consequently, the information obtained does not directly provide a clear guideline to effectively design the course and select the most appropriate methodological tools and theñ duration of the activities to be implemented in the classroom. The approach followed in this work is closer to the predictive control used in engineering to determine the optimum control actions [7]–[9]. In other words, the present proposal allows determining the optimum times for different methodological tools according to certain scenarios where the different competences have a pre-defined weight. The predictive approach can consequently help the teacher to specifically establish the course structure that better suits his/her objectives (i.e., the competences to be acquired by students). To the best knowledge of the authors, this procedure is tested in this work for the first time, and it is hoped to be a powerful tool for teachers to set optimum timings in the course activities. Apart from determining the hours that the instructor should spend in each methodological tool, it should be highlighted that the predictive tool developed in this work also provides valuable collateral information. Since the calculation of the timing takes into account the students’ opinion through questionnaires, the final scores and the experience of the teacher on the subject, it is possible to compare those three perspectives. From this comparison, it is possible to identify for example the divergence of the students perception and the weighted scores. These differences can be quantified and provide an interesting insight about the progress of the subject. All in all, the predictive tool proposed in this work aims at helping the teacher understand what is going on in his/her subject and to quantify the time that he/her should spend during the course in order to promote a specific patter of competences in his/her students. Fortunately, the process can be automated so that the teacher only needs to perform some questionnaires to the students and export the students’ scores from the Moodle system. In this manner it becomes a simple but powerful tool that can be used in any area. Although the proposal is generally valid for different subject and degrees, for assessment purposes this paper tests the suggested predictive tool in an undergraduate engineering course about Electrical Machines at the University of Malaga (Spain). The results exemplify the capability of the predictive approach to adapt the course design to a specific pattern of competences. The analysis of different scenarios confirms the flexibility of the proposal to promote different technical and soft competences on the engineering students. The paper is structured as follows. Section II provides a brief description of the subject and context where the proposal is tested. Section III describes the predictive approach to determine the times of application of the different methodologies and the indices that can provide a further insight on the subject. Section IV shows and discusses the assessment of the predictive tool at the University of Malaga and the final conclusions are summarized in section V. II. CONTEXT, COMPETENCES AND METHODOLOGIES A. CONTEXT Although the predictive tool that will be detailed in section III is generally valid for any subject, the proposal will be 3106 VOLUME 10, 2022 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses tested, for the sake of illustration, in a course of Electrical Machines (EM) that corresponds to an Engineering Degree at the University of Malaga. The EM course is lectured during the second semester of the year, and it welcomes students from three studies: electrical, electronic & electrical and mechanical & electrical engineering degrees. It is worth noting that the cut-off mark to enter those engineering studies is significantly different, hence the background and nature of the students is diverse. It is an optional subject with 6 ECTS that is lectured 4.5 hours per week from February to June, and it has both theoretical and lab teaching. It has two well defined parts, where the students are taught the fundamentals of the direct current (DC) machines (part 1) and synchronous machines (part 2). This subject is not the first contact with electrical machines since the students have attended a course in the first semester that explains transformers and induction machines. They should also be familiar with electrical engineering since all students have studied theory of circuits, at least, in previous years. The students are in the third (electrical degree) and fourth (electrical & electronic, mechanical & electrical degree) course, hence the students have previous experience during several years at the University. The total number of students attending the course in 2020 and 2021 is 90 and 70, respectively. This is quite a high number of students for a course located in the 3rd and 4th year, this enabling the possibility to perform an educational innovation and obtain some results that can be statistically meaningful. From the point of view of the course design, the teacher needs to select the competences to be acquired by students and the methodological tools that will be used to achieve this goal. Both aspects are detailed in what follows. B. COMPETENCES The number and type of competences that can be used in the proposed predictive tool (see section III) is not limited in any manner. In the specific case of the EM subject under consideration, the main competences that have been identified are: •C1 - Acquisition of theoretical concepts. •C2 - Resolution of practical issues. •C3 - Creativity. •C4 - Teamwork capability. •C5 - Motivation. •C6 - Satisfaction. While C1 and C2 are classical competences in the field of engineering studies, other transversal skills such are C3 and C4 are increasingly welcomed by the electrical engineering industry. This is especially noticeable in the area of electrical machines and drives since the irruption of electrical vehicles and renewable energies are requiring multidisciplinary teams to ultimately improve the existing technology. On the other hand, competences C5 and C6 are included to achieve a lasting knowledge and to promote a life-long learning. Although the number of competences is finite, the number of scenarios is endless if we consider C1 to C6 as continuous variables. In other words, if we rate the importance of each competence from 1 to 10, it is possible to give more importance to some specific group of competences. For the sake of example, high rating for competences C1 and C2 would lead to a classical approach where technical skills are predominant. Conversely, high values to C3 and C4 would also promote transversal skills. Section IV will set some scenarios that analyze the impact of different maps of competences on the course design. C. METHODOLOGICAL TOOLS Once the competences are identified, the next step is to decide which methodological tools will be used to successfully promote those target skills. This work considers the following: •M1 - Master Class. •M2 - Visualization of theoretical videos. •M3 - Visualization of problem-solving videos. •M4 - Students’ problem design. •M5 - Teamwork. •M6 - Analysis of industry brochures. •M7 - Intermediate exams. •M8 - Online questionnaires. •M9 - Couple questionnaires. Tool M1 is the classical lecture in a classroom where the main theoretical concepts are explained, and some practical problems are solved. It is typically done using chalk and blackboard and slides, and it is tremendously useful to transmit knowledge to groups with a high number of students. On the other hand, the interaction in M1 is typically low, even when it is promoted by the teacher. M2 and M3 are alternative tools to M1, in the sense that they share the same aim. In other words, M2 and M3 are the online version of M1, and in this case the interaction is simply null. Nonetheless, videos offer editing possibilities to teachers and the capability to rewind the explanation for students. The covid-19 outbreak in 2020 promoted the use of M2 and M3, and the videos that were created in those months were re-used in 2021 as complementary tools to M1. M4 implies that students should design their own problems involving DC and synchronous machines. They are asked to look for an industrial context and define a sensible problem with reasonable parameters. The problem and its solution must be uploaded to the Moodle platform. M4 aims to place the student in the centre of the teaching-learning process, making him/her protagonist and responsible for the creation of the problem. M5 involves teamwork in any way, including homework to be done by groups of students and classroom tasks that should be solved as a team. M6 requires the analysis and use of industry brochures, hence approaching the students to real-world problems and putting him/her in contact with the most relevant parameters and quality indices that are used in this field of knowledge. VOLUME 10, 2022 3107 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses The methodological tool M7 is related to the use of intermediate exams to evaluate the theoretical and practical capabilities of the students. Although, the realization of exams is not a particularly desired academic choice between the students, they typically want to pass the exam. Consequently, they try to consolidate/increase their theoretical/practical knowledge to obtain a satisfactory score. Analysing previous editions of the considered EM courses, two intermediate exams have been usually carried out, the first one related to the DC-machine whereas the second one is focused on the synchronous machine. In the case of methodological tool M8, the Moodle platform is employed to generate online questionaries. This alternative allows the implementation of a quick test where the student needs to solve some issues related to a specific topic of the course. The realization of these online questionnaires permits a higher detail degree and a lower-stressing situation than in the case of activity M7. Finally, the use of questionnaires in couples is considered as the methodological tool M9 in this work. This methodology can include the realization of some laboratory implementations or the resolution of diverse practical issues, promoting at the same time their teamwork skills. III. DESCRIPTION OF THE PREDICTIVE TOOL A. THE BASES OF THE CONSIDERED PREDICTIVE TOOL As previously exposed, the proposed predictive tool allows the estimation of the corresponding timing for each preselected methodological tool. The objective of this optimization process is to promote the acquisition of certain desired competences. For that purpose, a predictive tool, based on the nature of model predictive control strategies, has been implemented to estimate the optimal timing of the course. The operating principle of this modern regulation technique is the use of a system model to predict the impact of the available control actions on the predicted control variables. Next, a predefined cost-function is employed to select the optimum control action for the considered reference scenario. In this work, the tuple, methodological instrument and its corresponding application time, has been defined as the control action, whereas the selected student competences are the control variables. The goodness of each control action is later evaluated in the cost-function where the control designer establishes the control objectives. Figure 1 illustrates the structure of a generalized MPC scheme. In addition, the use of an MPC strategy typically permits the inclusion in a simple manner of different operating constraints [7]. For instance, in this work, several time restrictions have been added in the design of the control actions. Since an MPC strategy has been implemented, it is necessary to design a predictive model where the behaviour of the analyzed system needs to be satisfactorily reproduced. To this end, some input data can be required, as well as some modelling design decisions. Subsequently, the cost-function and the control constraints are also established according to the control designer criterium. This section describes the FIGURE 1. Simplified scheme of an MPC strategy. design process of the proposed predictive tool. Furthermore, one additional index has been defined to provide meaningful information about the perception of the different agents implied in the teaching-learning process. The first stage to determine the available control actions is the selection of the methodological instruments (see Section II.C). However, it is necessary to know the timing of these learning tools in order to obtain the control actions. Therefore, the control action is formed by the selected methodological tools [M]= [M1,M2,M3,M4,M5,M6,M7,M8,M9] and their corresponding application times [t]=[tM1,tM2,tM3,tM4,tM5, tM6,tM7,tM8,tM9]: Control action ≡[tM1·M1,...,tM9·M9].(1) As the methodological instruments are pre-selected, the achievement of different control actions is obtained with the modification of [t]. This approach has been implemented since the impact on the control variables (the student competences in this work) is related to the application time of each learning tool during the course. Taking advantage of the MPC flexibility to design the available control actions [8], [9], some timing constraints have been imposed: 0.200 ≤tM1≤0.40, 0.025 ≤M2≤0.15, 0.025 ≤tM3≤0.15, 0.100 ≤tM4≤0.25, 0.100 ≤tM5≤0.20, 0.025 ≤tM6≤0.05, 0.050 ≤tM7≤0.20, 0.050 ≤tM8≤0.20, 0.050 ≤tM9≤0.20.(2) These restrictions can be defined by the instructor according to his/her experience or using the specific academic legislation. Moreover, the course duration needs to be added as a constraint: tM1+tM2+ · · · + tM8+tM9=1,(3) 3108 VOLUME 10, 2022 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses in this case a per unit approach has been employed to model the course duration. B. PREDICTIVE MODEL: TEACHER DATA, STUDENT DATA AND SCORES The predictive model needs to satisfactorily reproduce the system behaviour and this task can be typically solved in a simple manner in engineering problems. Although for the considered system the modelling process might seem complex, this can be carried out without an excessive complication if it is assumed that the objective of the model is to determine the effect of each methodological action on the acquisition of each one of the selected competences. For that purpose, the proposed model takes into account the teachers’ experience, the perception of the students, and their achieved scores. In other words, the predictive model has been designed considering all agents involved in the teaching-learning process. Based on the previous information and requirements, the predictive model (PM) to know the relationship between each students’ competence (i) and each methodological instrument (j) is the following: PMij =¯ Qij +¯ Sij.(4) The ¯ Qij components are collected using questionnaires that evaluate the students’ perception about the impact of the methodological tools on each one of the studied competences. This information is arranged in a matrix ¯ Q6×9since the predictive tool considers six different students’ skills and nine methodological instruments. On the other hand, the ¯ Sij term models the relationship between competences and the available methodological tools using the students’ scores. This quantitative information is obtained thanks to the implementation of different tests designed by the instructor, such as online questionnaires, laboratory games or teamwork projects. In this case, 17 testing activities have been defined by the teachers in the considered EM course. These tasks provide a quantitative measurement, but moreover, they can facilitate the learning and the acquisition of the selected competences. Focusing on the teacher labor in this point, he/she must design the testing tasks and their relationship with the competences [SC]6×17 and the considered methodological actions [SM]9×17. Then, the ¯ Sij components can be estimated using the average score ¯ Snof each assessable activity and the corresponding terms from [SC]6×17 and [SM]9×17, as follows: ¯ Sij =Pn i=1[SC]ij ·[SM]ij ·¯ Sn Pn i=1[SC]ij ·[SM]ij ,(5) being nthe number of implemented tests. Unfortunately, (5) cannot be used to obtain an average score when only a single task is related to the competence and the methodological tool. In that scenario the following estimation has been carried out: ¯ Sij =[SC]ij ·[SM]ij ·¯ Sn.(6) C. COST-FUNCTION: MAP OF TARGET COMPETENCES AND TIMING DETERMINATION To obtain the optimal control action, the timing of the methodological tools needs to be solved in the optimization process. For such goal, a cost-function per competence is defined: JC1=t1·PM11 + · · · + t9·PM19, JC2=t1·PM21 + · · · + t9·PM29, JC3=t1·PM31 + · · · + t9·PM39, JC4=t1·PM41 + · · · + t9·PM49, JC5=t1·PM51 + · · · + t9·PM59, JC6=t1·PM61 + · · · + t9·PM69,(7) where the impact of each methodological tool and its corresponding application time are included. Furthermore, a global cost function is created to aggregate the map of target competences in the optimization process: JT=KC1·JC1+KC2·JC2+ · · · + KC6·JC6,(8) the weighting factors (KC1,KC2,KC3,KC4,KC5,KC6) in the proposed cost-function can be tuned up by the instructor to prioritize the development of certain student skills. For example, a map of target competences defined by the weighing factor values (KC1=5, KC2=2, KC3=0, KC4=0, KC5=3, KC6=0) intensively promotes the acquisition of theoretical contents (C1), aims at developing a teaching-learning process with quite motivated students (C5) and desires learners able to solve practical issues (C2). The optimization is carried out with the variation of the timing using an iterative process to determine the maximum value of the global cost-function. As previously exposed, the modification of the application times implies the generation of different control actions. The result of the optimization process provides the teacher with the knowledge of the optimal methodological instruction timing [topt ]=[topt M1,topt M2, topt M3,topt M4,topt M5,topt M6,topt M7,topt M8,topt M9] for the proposed competence map. Figure 2 shows the scheme of the implemented predictive academic tool where the different components have been depicted. D. COLLATERAL INFORMATION Apart from the optimal timing, the proposed algorithm also provides some collateral information to the teacher. This supplementary information is related to the difference between the perception of the students, their scores, and the instructor experience. For that purpose, the index 1ij is defined for each methodological tool and student competence: 1ij(%) = ¯ Qij −¯ Sij 10 ·100.(9) A null value of this index expresses the desired convergence among the perception of the students, their scores, and the instructor experience. Low values of this index can also be considered as a suitable situation. However, some conflicting VOLUME 10, 2022 3109 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses FIGURE 2. Structure of the proposed predictive tool. points in the teaching-learning process can exist when a high absolute value of 1ij is obtained. For instance, this situation can appear due to a high difference between what the students think they have learned and the competences they have actually acquired or due to a significant divergence between the students’ perception and the instructor experience. IV. ASSESSMENT OF THE PREDICTIVE TOOL A. SELECTION OF SCENARIOS The algorithm described in the previous sections has full flexibility and allows determining the times of application of each methodological tool for any scenario. Nevertheless, for assessment purposes, it is necessary to evaluate some specific scenarios. Those considered in this work are listed in Table 1. Scenarios S1 and S2 are deliberately set on the extremes. While scenario S1 aims at exclusively promoting theoretical concepts, scenario S3 fully focuses on the development of the students’ creativity. The idea is to compare very different pedagogical objectives. It is worth noting in any case that even when the scenario only prioritizes one competence, the other competences are not completely disregarded because the method establishes some minimum and maximum time restrictions (see Section III). Those constrains make the results more stable and ensure that some minimum values of each competence will be promoted. On the other hand, scenarios S3 and S4 are more balanced, including weighting factors for several competences. This is likely a more realistic situation because the teacher will look for an adequate trade-off between technical competences, cross-cutting competences, and motivation/satisfaction. Scenario S3 puts more emphasis on practical skills and teamwork capabilities, whereas scenario S4 opts for a somewhat higher promotion of theoretical concepts, creativity and teamwork skills. Finally, scenario S5 sets equal weights for the acquisition of theoretical concepts and satisfaction. The idea is to compare scenario S1 and S5 and evaluate the impact of the TABLE 1. Scenarios of target competences. students’ satisfaction on the course design and its eventual consequences on the life-long learning. Results will show in different figures the times of application that are obtained for each scenario. Additionally, those time will also be shown in tables in percentage values considering the minimum and maximum time restrictions as follows: tratio = topt Mj −tmin Mj tmax Mj −tmin Mj ·100.(10) The dimensionless values from (1) will clarify the percentage value of the applied times for each specific range. B. RESULTS OF THE APPLICATION TIMES The results for scenario S1 are shown in Figure 3and Table 2. An expected result is the promotion of master classes. In fact, the time of application of C1 is indeed close to its maximum value (87.5%). However, it can also be observed that methodological tool M4 is at its maximum level because the design of problems by the students compulsorily requires previous theoretical background. The rest of methodological tools are at (or close to) minimum values, ensuring in this way that other competences are not completely omitted during the teaching-learning process. The results when the creativity is the priority (scenario S2) are however very different (see Figure 4and Table 3). In this case the problem design, teamwork capabilities and use of industrial brochures are clearly promoted. Even though the 3110 VOLUME 10, 2022 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses FIGURE 3. Methodological tool timing for scenario S1. TABLE 2. Application time ratio: Scenario 1. TABLE 3. Application time ratio: Scenario 2. design of problems by the students (M4) requires a theoretical background, it is essentially an open problem where the students need to imagine a context for the problem and make decisions on the issues that will be analyzed. Similarly, when the students are working in teams towards a certain aim it is necessary to discuss and decide the better approach, hence the activities in M5 provide a high flexibility and promote creativity. Finally, the analysis of industrial brochures (M6) is again an open problem where students are provided with the information from industry and are asked to look for a certain solution, but without detailing the specific steps to be followed. This is in contrast with standard problems in classical books on electrical machines where all the input data are provided, and all the expected results are detailed. Apart from M4 to M6, it is remarkable that master classes and theoretical videos (M1 and M2) are not at minimum values since they contribute to have a minimum background. In other words, creativity stems when a certain previous knowledge exists. M9 has a low value for its time of application, but it acknowledges that working in couples also assists somehow creativity since the opinions need to be compared. The results for scenario S3 are then shown in Figure 5 and Table 4. Since the teamwork capabilities are a priority FIGURE 4. Methodological tool timing for scenario S2. FIGURE 5. Methodological tool timing for scenario S3. in this scenario (C4), the algorithm provides a maximum application time to teamwork (M5) and work in couples for the questionnaires (C9). However, the scenario also promotes practical issues (C2), hence the use of industrial brochures is also set to maximum values (M6). Some weight to theoretical concepts is also given in this scenario and consequently the master classes (M1) have an intermediate time of application. Finally, the visualization of theoretical and practical videos (M2 and M3) has low percentage values that collaterally contribute to the competences that are promoted in S3. VOLUME 10, 2022 3111 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses TABLE 4. Application time ratio: Scenario 3. FIGURE 6. Methodological tool timing for scenario S4. TABLE 5. Application time ratio: Scenario 4. Similarly to S3, scenario S4 is again more balanced that S1 and S2 (see Figure 6and Table 5). However, theoretical concepts (C1) are now more important than practical issues and for this reason the time of application of master classes (M1) is now elevated to maximum values. This is done at the expense of reducing the time of application of other methodological tools (e.g., M5). The realization of questionnaires in couples is again in maximum values as in S3, because it promotes both teamwork and motivation. It is clear in any case that any map of application times is searching for a trade-off since the time that is dedicated to a certain methodological tool leaves less time for other activities. Nevertheless, the minimum and maximum time restrictions set by the instructor guarantee that a certain activity is not fully disregarded or captures all the available time, respectively. Scenario S5 (see Figure 7and Table 6) finally brings the opportunity to compare the results with those of S1. It follows from the results that the promotion of the satisfaction clearly elevates the time of application of theoretical videos M2. Although both master classes (M4) and theoretical videos (M2) promote C1, the possibility of M2 to see the explanations multiple times (with pause and rewind option) better satisfies the student. This satisfaction is in turn a key con- FIGURE 7. Methodological tool timing for scenario S5. FIGURE 8. 1ij per methodological tool and student competence. dition to obtain a lasting knowledge and consequently brings benefits to the teaching activity. The price to be paid is a less capability to interact with the teacher, but this is still possible in M1, which is set to maximum values in scenario 5. The motivation has now as a non-null value and it can be observed that time tM1is actually set to maximum values. C. COLLATERAL INFORMATION It is not only the calculation of the times what can be extracted from the analysis. Conversely, the analysis of some terms used in the predictive tool brings information that can assist the teacher in his/her design of the course. Specifically, the coefficient 1ij defined in (9) identifies the difference between the teachers’ perspective, the students’ opinion, and the actual scores. Ideally, these three points of view should converge to avoid frustration in the students. 3112 VOLUME 10, 2022 J. J. Aciego et al.: On Use of Predictive Tools to Improve Design of Undergraduate Courses FIGURE 9. 1ij obtained in the methodological tool M3. FIGURE 10. Number of tasks per methodological tool and student competence. FIGURE 11. 1ij obtained in the methodological tool M6. TABLE 6. Application time ratio: Scenario 5. Figure 8shows the coefficient 1ij for each methodological tool M1 to M9 and for each competence C1 to C6. Most of the values are below 20%, showing a good match between the mark in the questionnaires and the weighted scores. For example, the visualization of problem-solving videos (M3) presents low values for all competences, as it is detailed in Figure 9. This means that what the students expects (first term in 1ij) from this methodological tools approaches what is actually achieved according to the scores and teachers’ experience (second term in 1ij). This good match is favoured by the fact that M3 is included in many different tasks, as it is shown in Figure 10. Since M3 is simultaneously included in more than 10 tasks for each competence, the aggregated value brings more stable results. That is, the visualization of problem-solving videos is not carried in an isolated activity, but in a combination of multiple tasks, narrowing the gap between the students’ opinion and the weighted scores. Conversely, the use of industrial brochures (M6) presents a discrepancy higher than 50% for competence C1 (see Figure 11), indicating that the students thought that M6 would have a high impact on C1 (average score of 8.31 over 10), whereas the average score of the students (7.16/10) and specially the teachers’ perspective (2.5/10) presents a lower value. This low match is conditioned by the low number of tasks related to M1. Only one task is planned for M6 in relation to C1 (see Figure 10), and it is relatively straightforward to obtain a higher difference in the coefficient 1ij. In other words, the higher number of tasks, the lower is the difference between the students’ opinion and the weighted scores. The other way round, isolated tasks provide little time for the students and the coefficient 1ij is more prone to show a higher discrepancy. A direct consequence of this analysis might be that the use of a methodological tool (M) to develop a certain competence (C) is more effective when several tasks are involved. The use of isolated tasks will likely result in a low match between the students’ expectations and the weighted scores. This analysis simply aims at illustrating how the information from the predictive approach can be used to extract conclusions about the subject. The most relevant aspect to be highlighted is that the proposal provides a tool to be used by instructor to improve the students’ training and satisfaction. Both the determination of the times of application (see subsection IV.B) and the data analysis (see subsection IV.C) assists the teacher in the Herculean task of having well-trained motivated students. V. CONCLUSION The design of a university course to promote a specific map of competences in the students is a rather complex task. Furthermore, this mission is typically accomplished with the exclusive use of the teachers’ previous knowledge, disregarding in this manner the students’ opinions and the historical scores. In order to assist the teacher, this work evaluates the use of a predictive approach to determine the application times of several methodological tools. The results confirm that the proposed tool allows the calculation of the application times for very different scenarios with various weights for each competence. While the time restrictions included in the predictive tool guarantee that all competences are promoted to some extent, the algorithm provides higher times of application for those methodological tools that better fit into the target future competences. Consequently, the predictive algorithm proves to find a compromise between the different competences for each scenario and determines the times for this desired trade-off accordingly. Apart from the time calculations, the predictive approach simultaneously considers the students’ opinion, the teachers’ perspective and the historical scores, and this valuable information can also be VOLUME 10, 2022 3113