Construction and validation of a questionnaire to assess student satisfaction with mathematics learning materials

Construction and validation of a questionnaire to assess student satisfaction with mathematics learning materials Alién García-Hernández University of Informatics Sciences, 17100 Cuba University of Seville, 41103 Spain [email protected] Teresa González-Ramírez University of Seville, 41103 Spain [email protected] ABSTRACT Mathematics is an essential branch for the scientific development and its study is mandatory in most university degrees. However, currently the level of academic performance and motivation of students to learn this science is not the desired one. The students can use different learning tools inside and outside the math classroom, enhancing the quality of the learning materials that are designed essentially to facilitate the learning of mathematics. The present research project aims to determine the validity and reliability of a measurement instrument that allows the assessment of the satisfaction of the students with the available learning materials. To fulfill the objectives of this research, the method of survey was used. A study with a quantitative approach was developed, which led to the design and validation of a questionnaire by a group of 7 experts. The validation closed after applying a pilot study with 728 students. It concluded positively, obtaining nine factors that coincide with the revision of the literature: technological quality, quality of content, visual quality, didactic significance, adequacy of content, relationship between theory and practice, involvement, contribution to learning, relevance and interaction between educational actors. The results of this questionnaire provide to the international scientific community with relevant information for the design, selection, and use of study materials in the classrooms, which will contribute to raising the levels of student engagement, and their academic performance in mathematics, secondarily. CCS CONCEPTS • General and reference → Reliability • General and reference → Measurement • General and reference → Evaluation • General and reference → Validation • General and reference → Design • Social and professional topics → Student assessment KEYWORDS Questionnaire, validation, asses, mathematics, learning materials ACM Reference format: Alién García-Hernández and Teresa González-Ramírez. 2018. Construction and validation of a questionnaire to assess student satisfaction with mathematics learning materials. In Proceedings of 6th International Conference Technological Ecosystems for Enhancing Multiculturality, Spain, October 2018 (TEEM’18) (Salamanca, Spain, October 24-26, 2018), F. J. García-Peñalvo Ed. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3284179.3284204 1 INTRODUCTION Mathematics is vital for the training of professionals from various branches. The number of students with lack of academic achievement in mathematics is significant, for this reason it should be propitiated, to engage them even more with their studies [9]. The learning materials have contributed, through history, to the teaching of mathematics, as an essential variable to enhance student learning [35]. For this reason, many efforts have been made by schools to guarantee the learning materials to their students. We can also look at learning materials based on the following three time perspectives [15]: • We can look at the learning material itself, as a text. Here, the learning material is present as potential didactic potential: that is to say, that we can see the potential in the learning material that could help support the teacher’s teaching and promote student learning. • We can look at the learning material as a tool in use. Here, the learning material acts as actualized didactic potential: that is, we look at what actually happens when the teacher and his/her students use the learning material as a tool in the educational context. • We can look at the effects, and how the use of a learning material can make a difference to both, the students’ learning and the teacher’s teaching. Here, what is being registered is the learning aids realized didactic potential as an effect, which becomes apparent over time. In a baseline investigation [5], study materials are defined by dividing them into three types: • Functional learning materials (tools) characterized by their facilitation of learning and teaching: including black and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. TEEM'18, October 24-26, 2018, Salamanca, Spain © 2018 Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-6518-5/18/10…$15.00 http://dx.doi.org/10.1145/3284179.3284204 Construction and validation of a questionnaire to assess student satisfaction with mathematics learning materials TEEM 2018, October 2018, Salamanca, Spain 135 white boards, computer applications, projectors, and mobile phones. • Semantic learning materials (texts) characterized by their meaning as constituted by signs and semantic references: including film, literature, charts, pictures, paintings and other texts and objects with references to specific domains of experience. • “Didacticized” learning materials characterized by combining tools and texts and facilitating learning and teaching: including textbooks, online teaching materials, and educational games. For the purpose of our research, the third type of study material discussed above is assumed. The students use different learning materials inside and outside the classroom, including recommended textbooks, lecturer-developed coursebooks, lecturer class notes and digital material, and other online resources, even multimedia [25]. One of the most used learning materials is the textbook, however, the study materials created by the teacher are preferred by the students. Ensuring the quality of the study materials that are designed is essential to enhance the learning of mathematics. Various variables are analyzed to assess the quality of the study materials, including those associated with the design of these materials. Research results suggest [1, 20] that the graphic design of the material, it colors, images structure, and its content influence the quality of the study materials, which is also related to audiovisual resources. Other studies [13] establish that an essential element of a study material to enable the learning of mathematics, should contain a mathematical language understandable to the student; as result, these materials should be beneficial to meet the student's learning outcome. Another research team establishes that mathematical educational materials should encourage their postulates to be exposed through comprehensible algorithms, according to the cognitive capacity of the student in question [14, 29, 30]. Is important to emphasize that a relationship between theory and practice is intrinsic, hence study materials should be able to express it, guiding the teaching-learning process [19, 22]. Also, a recurring element in the investigations, is the use of the technology in the study materials, related to the current era that students live, being digital natives. There is a strong relationship between the satisfaction of a student with the mathematics study materials and the levels of interactivity and feedback that is achieved [1, 3, 9, 17, 32]. As a final theoretical analysis, the motivational variables associated with the study materials should be highlighted. The research expresses the crucial role of students’ motivation to achieve proper academic performance, using adequate materials to engage students [2, 6, 8, 10, 28]. In a research carried out at the University of Seville, it was found that in order to engage students with their studies, activities must be achieved with levels of relevance, involvement and interaction, where the student appreciates the contribution of the materials used in their learning process [11]. These elements coincide with other studies carried out [2, 7, 16, 26]. For all of the above, it can be expressed that implementing a pilot study in the area of mathematics teaching could contribute to obtaining a consistent instrument with which to evaluate student satisfaction with their math study materials. This article presents the initial phase of construction of the questionnaire. 2 RESEARCH OBJECTIVES The objective of this research, in the current phase, is to determine the validity and reliability of a measurement instrument that allows to assess the satisfaction of the students the mathematics study materials. To meet this goal, we define the following specific objectives: 1. Construct a questionnaire with items related to satisfaction whit mathematics learning materials from the dimensions indicated in the related scientific literature. 2. Conduct expert validation of the questionnaire on teaching mathematics. 3. Conduct a pilot study with students of Engineering in Computer Science applying the questionnaire on transparency. 4. Analyze the reliability and validity of the measure obtained with the application of the questionnaire and determine possible improvements in it. 3 METHODS To fulfill the objectives of this research, a survey tool was used. A study with a quantitative approach was developed, which led to the design and validation of a questionnaire by a group of experts. The validation closed after applying a pilot study. A first version of the instrument was elaborated, based on the theoretical references found in the literature regarding satisfaction with the learning materials. This initial version was presented for consideration by 7 experts in the teaching of mathematics. After two rounds of evaluation by the panel, the questionnaire was applied to 728 students of computer science engineering at the University of information sciences of Havana, the sampling was intentional. Official data held by the institution about the students were used. Of the total, 293 participants were women (40.25%) and 435 were men (59.75%). The twenty-two survey items, distributed in three groups (scales), assessed the students’ satisfaction with their study materials for the Discrete mathematics subject. Such groups are: 1) general quality; 2) didactic adaptation; and 3) ability to motivate. The scales were created from the theoretical study. After experts' reviews, the final version of the questionnaire was set. The pilot study included a reliability study, based on the Cronbach Alpha coefficient, factorial analysis of its principal components (for metric variables) and a categorical analysis of principal components, considering the ordinal and nominal nature of the data [33]. The McDonald's omega coefficient was also used as statistic tool to estimate reliability [27]. Before starting the factorial analysis, the Kaiser-Meyer-Olkin (KMO) test and the Bartlett sphericity test were performed to corroborate that a set of items measures an underlying theoretical factor [12, 31]. In general, a KMO greater than 0.600 is expected, but it is preferable to observe a value greater than 0.800 [18]. For its part, the Bartlett test was desirable if it reached a high chi square and a probability value of less than 5% [4]. All calculations were carried out in the IBM-SPSS v22 statistical package. TEEM 2018, October 2018, Salamanca, Spain A. García-Hernández and T. González-Ramírez 136 4 RESULTS The validation of experts allowed the improvement of the proposed instrument, by adding or rewriting items to be better understood by the students. 4.1 Reliability The overall result of Cronbach's Alpha (.896) shows the internal consistency of the questionnaire, which coincides with the results obtained in each scale (.821, .832, .837). McDonald's omega coefficient confirmed the high reliability of the questionnaire, both globally (.902) and scales (.829, .845, .844). Those results are shown in Table 1. It has been verified that the elimination of any item would not improve the Cronbach's Alpha or McDonald's omega coefficient. Table 1. Cronbach's Alpha and McDonald´s omega (global and by scales) Scale Cronbach's Alpha McDonald's omega Global .896 .902 General quality of the learning materials .821 .829 Didactic adaptation of the learning materials .832 .845 Motivation´s capacity of learning materials .837 .844 With these results we can express that the scales met the objectives for which they were created. 4.2 Validity Below are the results for each of the questionnaire scales, showing the obtained factors. Scale 1. General quality of the learning materials This first scale seeks to gather information about the characteristics of the study materials that provide the necessary quality for student learning. For this, both the visual quality, and the quality of the discrete mathematical content that is exposed are taken into consideration. Inquiring about characteristics associated with the use of technology in study materials was also of interest. Following these criteria, nine items were developed. In the factor analysis, this scale showed a KMO coefficient = 0.821 and a Bartlett test with a chi square = 1470.962; and 5 degrees of freedom; p <0.001. Of the nine factors, three showed auto values higher than 1, justifying the 68.32% of the variance (Table 2). Table 2. List of principal components (factors) of the scale General quality of the learning materials Factor Auto values Auto value % variance % accumulated 1 2,548 29,82 32,82 2 1,987 22,32 52,14 3 1,089 16,18 68.32 Table 3 shows the factors obtained and the list of items associated to each factor, ordered by their factorial weight. The factors were grouped as: 1) Technological quality, 2) Quality of mathematical content, and 3) Visual quality. Table 3. List of items associated with each factor of first scale Factors (Dimensions) factorial weight Technological quality Possess restitution activities through the use of technologies .869 They have audiovisual resources .790 They can be accessed in mobile format .735 Quality of mathematical content They have a level of difficulty adjusted to the level of the race and the characteristics of the subject .845 They have a variety of enough exercises for my study .738 They present an accurate progression of the exercises in terms of complexity .692 Proper extension of readings or topics .657 Visual quality They have an attractive graphic design that encourages my learning .851 They support the ideas or concepts developed in the text through illustrations or graphics .724 As shown, the activities with the use of technology, the adjustment of difficulty level of learning materials according to level of subject and attractiveness of graphic design are the items that best represent those factors. Scale 2. Didactic adaptation of the study materials The second scale, didactic adaptation, was elaborated with the purpose of obtaining information about the characteristics of the learning materials that favor a didactic adaptation for the learning of the mathematics. The didactic relevance based on the levels of help they provide to students, the adequacy of the objectives and contents to the student's major and an adequate relationship between the theory and the practice were explored. In the factor analysis, this scale showed a KMO coefficient = 0.834 and a Bartlett test with a chi square = 1548.024; and 5 degrees of freedom; p <0.001. We obtained 3 factors whose auto value are higher than 1 and that justify 71.29% of the variance (see Table 4). Table 4. List of principal components (factors) of the scale Didactic suitability of the learning materials Factor Auto values Auto value % variance % accumulated 1 3.978 34.25 34.25 2 2.024 25,32 59.57 3 1,289 11.72 71.29 Construction and validation of a questionnaire to assess student satisfaction with mathematics learning materials TEEM 2018, October 2018, Salamanca, Spain 137 In this scale the three factors are: 1) Didactic Significance, 2) Adequacy of mathematical content, and 3) Relation between theory and practice. Table 5 shows the factors obtained from this scale and the list of items associated with each one of them, ordered by their factorial weight. Table 5. List of items associated with each factor of second scale Factors (Dimensions) factorial weight Didactic Significance They describe algorithms step by step for the comprehension of contents of the mathematics. .824 They sufficiently exemplify mathematical definitions, theorems and postulates. .784 They present the concepts developed with clarity and precision. .741 Make a conceptual summary by content blocks .712 Adequacy of mathematical content They present a correspondence between the contents and the information necessary for the realization of mathematical exercises. .755 They integrate the theoretical and practical elements with developer activities. .684 They adapt to my social and cultural reality .678 Relation between theory and practice They adapt to the objectives and contents of the subject. .812 They have a level of difficulty of the theoretical content according to the students. .751 They have the updated contents. .624 The items with the greatest factorial weight refer to the descriptions of step by step algorithms for understanding the contents of the subject and the adaptation to the objectives and contents of the subject. The factor of adequacy of mathematical contents stands out in this analysis, in which all its elements have a factorial weight greater than .700. Scale 3. Ability to motivate of learning materials The last scale, associated to motivational variables, was elaborated to appraise the self-evaluation of motivation, obtaining information about the characteristics of the learning materials that favor an adequate mood and incentive involvement of the students in the learning of the Discrete Mathematics. For this purpose, students’ self-perception about the role and ability of learning materials to contribute to their learning, the relationship between mathematical content with its history, and its real applications to their future profession were explored. It also inquiries about the satisfaction provoked by the study materials, according to the interests of the students, and the possibilities of interaction as a motivational element. Following these criteria, the fourteen proposed items were elaborated. In the factor analysis, this scale showed a KMO coefficient = 0.834 and a Bartlett test with a chi square = 1724.351; and 5 degrees of freedom; p <0.001. Four factors whose auto value are higher than 1 justify 73.92% of the variance (Table 6). Table 7 shows the factors obtained from this scale and the list of items associated with each factor, ordered by their factorial weight. The factors are: 1) Implication, 2) Contribution to learning, 3) Relevance, and 4) Interaction. Table 6. List of principal components (factors) of the scale Motivation´s capacity of learning materials Factor Autovalues Autovalue % variance % accumulated 1 4.187 24.25 24.25 2 1.899 21,32 45.57 3 1,135 16.72 62.29 4 1.015 11.63 73.92 Table 7. List of items associated with each factor of last scale Factors (Dimensions) factorial weight Implication They enhance my satisfaction in the study of mathematics. .824 They are easy to understand and connected to my interests. .784 Contribute to a better learning of the subject. .741 They make me forget how difficult mathematics is. .709 Contribution to learning They link their contents with my Career. .755 They make visible the linking of mathematical content with the real world. .724 They adapt to my learning rhythm .658 Relevance They present in a pleasant way the mathematical content, showing its origin and evolution. .812 They stimulate search and discovery .754 They connect with my interests through activities of didactic motivation such as surprises, riddles, curiosities, etc. .698 They have activities that promote my learning from games (gamification). .672 Interaction They allow asking the author questions and receiving their answers .715 They allow me to interact with my teacher and my classmates. .684 They allow me to evaluate myself in an automated way .632 5 CONCLUSIONS For this research a questionnaire was elaborated, using Likert type scales, to evaluate the satisfaction of the students with their math learning materials. The instrument, initially designed from the TEEM 2018, October 2018, Salamanca, Spain A. García-Hernández and T. González-Ramírez 138 theoretical referents on the subject in question, was validated using experts criteria. The panel of experts was formed with university professors and researchers of recognized prestige in the area of knowledge that concerns us. Instrumental validation was done with its application to 728 students of Computer Science Engineering where a high reliability index was evidenced. A principal component analysis was applied to complete its validation process. The validation process concluded positively, obtaining nine factors that coincide with the revision of the literature: technological quality [3, 9, 17], quality of content [13, 21, 34], visual quality [1, 20], didactic significance [14, 29, 30], adequacy of content [9, 13, 30], relationship between theory and practice [19, 21, 23, 24], involvement [11, 16, 25], contribution to learning [5, 15], relevance [2, 7, 11], and interaction [1, 32]. It is relevant to observe how these dimensions were grouped into three general scales. The objectives of the research were met, and the questionnaire has validity and reliability, allowing its use in the practice, also justifying its application to other samples in different educational contexts. The results of this questionnaire provide relevant information for the design, selection, and use of study materials in the classrooms setting, enhancing engagement, hence academic performance of students in mathematics related subjects. ACKNOWLEDGMENTS We thank the Ibero-American Postgraduate University Association (AUIP), the University of Seville, and the University of Informatics Sciences for their material and financial contributions to the development of this research. REFERENCES [1] Amaya, D. et al. 2017. Perception on the contribution of Interactive and Experimental Learning Objects to the management of learning in Discrete Mathematics. 10th International Conference of Education, Research and Innovation (2017), 8926–8933. [2] Attard, C. 2012. Engagement with mathematics: What does it mean and what does it look like? APMC. 17, 1 (2012), 9–13. [3] Baek, E. and Monaghan, J. 2013. Journey to Textbook Affordability : An Investigation of Students ’ Use of eTextbooks at Multiple Campuses sis ( SNA ) in OnlineCourses. International Review of Research in Open an Distance Learning. 4, 3 (2013), 1–26. [4] Bartlett, M.S. 1950. Test of significance in factor analysis. Br J Psychol. 3, (1950), 77–85. [5] Bundsgaard, J. and Illum, T. 2011. Evaluation of learning materials. Journal of Learning Design. 4, 4 (2011), 31–44. [6] Causer, T. and Terras, M. 2014. Many hands make light work. Many hands together make merry work: transcribe Bentham and crowdsourcing manuscript collections. Crowdsourcing Our Cultural Heritage. 57–88. [7] Contreras, J. et al. 2015. Procesos de e-tutorización y su impacto en el engagement del alumnado universitario. Un diseño microgenético. Universidad de Sevilla. [8] Durksen, T.L. et al. 2017. Motivation and engagement in mathematics : a qualitative framework for teacher-student interactions. Mathematical Educational Research Journal. 29, (2017), 163–181. DOI:https://doi.org/10.1007/s13394-017-0199-1. [9] García-Hernández, A. and González-Ramírez, T. 2017. Design and evaluation of the impact of an e-textbook in the engagement for the learning of Discrete Mathematics. 5th International Conference Technological Ecosystems for Enhancing Multiculturality (2017), 7. [10] Golding, C. 2014. The educational design of textbooks : a text for being interdisciplinary. Higher Education Research & Development. 33, 5 (2014), 921–934. DOI:https://doi.org/10.1080/07294360.2014.890573. [11] González-Ramírez, T. and Reyes, S. 2015. Características de las aulas universitarias que generan engagement desde la perspectiva de los estudiantes. Investigar con y para la sociedad. 691–704. [12] Gorsuch, R.L. 1997. Exploratory factor analysis: its role in item analysis. J Pers Asses. 68, (1997), 532–560. [13] Gustiani, I. et al. 2017. Development and Validation of Science , Technology , Engineering and Mathematics ( STEM ) based Instructional Material. AIP Conference Proceedings 1848, (2017). [14] Hadar, L.L. 2017. Studies in Educational Evaluation Opportunities to learn : Mathematics textbooks and students ’ achievements ☆. Studies in Educational Evaluation. 55, May (2017), 153–166. DOI:https://doi.org/10.1016/j.stueduc.2017.10.002. [15] Hansen, T.I. 2017. Quality of learning materials. IARTEM e-Journal. 9, 1 (2017), 122–141. [16] Hodgson, T.R. et al. 2017. Assessing Behavioral Engagement in Flipped and Non-Flipped Mathematics Classrooms : Teacher Abilities and Other Potential Factors. International Journal of Education in Mathematics, Science and Technology. 5, 4 (2017), 248–261. DOI:https://doi.org/10.18404/ijemst.296538. [17] Huang, R. et al. 2014. The New Development of Technology Enhanced Learning. Springer. [18] Kaiser, H.F. 1974. An index of factorial simplicity. Psychometrica. 34, (1974), 31–36. [19] Kong, Q. et al. 2003. Student Engagement in Mathematics : Development of Instrument and Validation of Construct. Mathematics Education Research Journal. 15, 1 (2003), 4–21. [20] Krauss, F. and Ally, M. 2005. A study of the design and evaluation of a learning object and implications for content development. Interdisciplinary Journal of Knowledge and Learning Objects. (2005). [21] Lazarides, R. and Rubach, C. 2017. Instructional characteristics in mathematics classrooms : relationships to achievement goal orientation and student engagement. Mathematics Education Research Journal. 19, 2 (2017), 201–217. DOI:https://doi.org/10.1007/s13394-017-0196-4. [22] Leis, M. et al. 2015. Using the Partial Credit Model to Evaluate the Student Engagement in Mathematics Scale. Journal of Applied Measurement. 16, 3 (2015), 251–267. [23] Leis, M. et al. 2015. Using the Partial Credit Model to Evaluate the Student Engagement in Mathematics Scale. Journal of Applied Measurement. 16, 3 (2015), 251–267. [24] Leon, J. et al. 2018. Teaching quality: High school students’ autonomy and competence. Psicothema. 30, 2 (2018), 218–223. [25] Maclaren, P. and Maclaren, P. 2017. How is that done ? Student views on resources used outside the engineering classroom engineering classroom. European Journal of Engineering Education. (2017), 1–18. DOI:https://doi.org/10.1080/03043797.2017.1396445. [26] Manzano, G. 2004. Perfil de los estudiantes comprometidos con sus estudios: influencia del burnout y engagement . Anuario de Psicología. 35, 3 (2004), 399–415. [27] McDonald, R.P. 1970. Theoretical foundations of principal factor analysis and alpha factor analysis. British Journal of Mathematical and Statistical Psychology. 23, (1970), 1–21. [28] Norton, S. 2017. Mathematics engagement in an Australian lower secondary school. Journal of Curriculum Studies. 49, 2 (2017), 169–190. DOI:https://doi.org/http://dx.doi.org/10.1080/00220272.2016.1141995. [29] Polikoff, M.S. 2015. How Well Aligned Are Textbooks to the. American Educational Research Journal. XX, X (2015), 1–27. DOI:https://doi.org/10.3102/0002831215584435. [30] Reys, B. et al. 2004. Why mathematics textbooks matter. Educational Leadership. 61, 5 (2004), 61–66. [31] Streiner, D.L. 1994. Figuring out factors: the use and misuse of factor analysis. Can J Psychiatry. 39, (1994), 135–140. [32] Tseng-Yi, C. et al. 2015. Integrating an e-book software with vector graphic technology on cloud platform. Procedia - Social and Behavioral Sciences. 176 (2015), 1012–1019. DOI:https://doi.org/doi: 10.1016/j.sbspro.2015.01.572. [33] Velazco-Martínez, L.-C. and Tójar-Hurtado, J.-C. 2017. Construction of a questionnaire to know the transparency in the evaluation of the learning of Engineering students. 5th International Conference on Technological Ecosystem for Enhancing Multiculturality (2017). [34] Vílchez, E. 2016. Uso de Wolfram Mathematica como apoyo para la enseñanza y el aprendizaje de la Matemática Discreta. 10 Festival Internacional de Matemática (2016), 198–205. [35] Zwart, D.P. et al. 2017. The effects of digital learning material on students ’ mathematics learning in vocational education. Cogent Education. 29, (2017), 1–10. DOI:https://doi.org/10.1080/2331186X.2017.1313581.