The effectiveness of physical and virtual manipulatives on learning and motivation in structural engineering

RESEARCH ARTICLE The effectiveness of physical and virtual manipulatives on learning and motivation in structural engineering Enrique Justo 1 | Antonio Delgado 1 | Carmen Llorente-Cejudo 2 | Rocio Aguilar 3 | Julio Cabero-Almenara 2 1 Departamento de Estructuras de Edificaci on e Ingeniería del Terreno, Escuela de Architectura, Universidad de Sevilla, Sevilla, Spain 2 Departamento de Did actica y Organizaci on Educativa, Facultad de Ciencias de la Educaci on, Universidad de Sevilla, Sevilla, Spain 3 Departamento de Organizaci on de Empresas y Marketing, Universidad Pablo de Olavide, Sevilla, Spain Correspondence Enrique Justo, Escuela de Arquitectura de Sevilla, Avda Reina Mercedes, 2. 41012 Sevilla, Spain. Email: [email protected] Abstract Background: Recent advances in technology and computer simulation have expanded the use of virtual manipulatives in engineering education. However, especially in disciplines in which tactile concepts (e.g., force) are included, it has been claimed that learning could be compromised by substituting physical materials with computer simulations. Purpose: This study examines the comparative effect of physical and virtual materials on students' conceptual understanding and motivation to learn structural concepts with manipulatives. Method: The study followed a between-subjects design involving 209 university students and seven instructors. We used a pretest/posttest quasiexperimental design with a purposely developed conceptual test to measure students' understanding of basic structural engineering concepts. Motivation and its components (attention, relevance, confidence, and satisfaction) were assessed with Keller's Instructional Materials Motivation Survey. Results: Our results revealed that the physical and virtual conditions were equally effective in promoting students' understanding of basic structural engineering concepts in a classroom setting. We also found that the students were more motivated to learn through the use of physical materials than with virtual ones. This difference was especially significant in the attention dimension. Finally, we found that male students showed more confidence than female students when using physical manipulatives but not when using virtual ones. Conclusions: The findings from this study associate the benefits of using physical materials with an increase in motivation rather than with superior learning gains. In particular, engineering educators seeking to enhance students' attention should consider adopting physical manipulatives in their classrooms, although they must be aware of the potential sex differences that could be caused by their use. Received: 9 November 2020 Revised: 2 August 2022 Accepted: 18 August 2022 DOI: 10.1002/jee.20482 This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2022 The Authors. Journal of Engineering Education published by Wiley Periodicals LLC on behalf of American Society for Engineering Education. J Eng Educ. 2022;111:813–851. wileyonlinelibrary.com/journal/jee 813 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KEYWORDS computer-based instruction, conceptual learning, motivation, physical manipulatives, virtual manipulatives 1|INTRODUCTION The use of physical manipulatives in hands-on activities has been a widespread tradition in Science, Technology, Engineering, and Mathematics (STEM) disciplines. Physical hands-on strategies have often been suggested as effective approaches to improve learning (Carbonneau et al., 2013). Some of the educational benefits claimed for this type of instructional activities include facilitating active learning, promoting interaction, and motivating students (Haury & Rillero, 1994; Satterthwait, 2010). Nevertheless, recent advances in technology and computer simulation have challenged STEM educators to consider the use of virtual manipulatives instead of their physical counterparts. The use of computer-based simulations offers some potential advantages, including an improved visualization of abstract concepts, cost-efficiency, portability, extensibility, cleanness, and greater control (Clements & McMillen, 1996; Rutten et al., 2012; Zacharia & Olympiou, 2011). Also, virtual materials can be integrated into online education environments, and this constitutes a crucial advantage in view of the new challenges posed in higher education by the recent global pandemic crisis. In this context, the question arises about whether students' learning can be compromised by the substitution of actual tactile manipulation by computer (point and click) interaction. For this reason, a growing body of research is being devoted to comparing the efficacy of physical and virtual manipulatives for learning (De Jong et al., 2013). Despite the large number of studies investigating the effect of manipulatives on learning, the majority of them are focused on conceptual understanding, while other cognitive and affective goals, such as motivation, have often been neglected (Chen et al., 2014). In this article, we present a comparative study covering the conceptual and motivational outcomes of undergraduate students using physical and virtual manipulatives for learning in a structural engineering course. 1.1 |Physical/virtual manipulative experimentation Physical manipulative experimentation (PME) refers to a learning environment in which students physically manipulate real-world materials to observe and understand the natural world. PME has two distinct characteristics: physicality, which refers to the actual touch of concrete materials, and manipulation, which involves active interaction of the students with the materials (Zacharia & Olympiou, 2011). Virtual manipulative experimentation (VME), on the other hand, is a learning environment in which students interactively manipulate dynamic representations of concrete objects on a computer screen (Moyer et al., 2001). 1.2 |Effects of PME and VME on conceptual understanding 1.2.1 | Theoretical underpinnings A distinctive characteristic of a PME environment is that students receive touch sensory feedback (Zacharia, 2015) during experimentation. Touch sensory feedback refers to a mode of discovery in which objects are explored through active haptic perception, which is a combination of tactile (variations in cutaneous stimulation) and kinesthetic (perception of force and movement in limbs and muscles) sensations. Two theories provide support for the notion that the presence of touch sensory feedback in PME can be beneficial for conceptual understanding: the embodied cognition theory and the additional sensory channel theory (Zacharia, 2015). The embodied cognition theory emphasizes the importance of perception in conceptual learning. According to this theory, understanding abstract concepts requires the construction of mental models of those concepts based on previous perceptual experiences (Pouw et al., 2014). Learners in PME are provided with perceptually rich experiences (including visual, tactile, and kinesthetic sensations) and build multimodal representations of concepts in 814 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License their minds, which scaffold their future understanding (Han & Black, 2011). The additional sensory channel theory explains the benefits that the activation of an additional sensory channel of information (touch) can provide in terms of reducing the cognitive load in the students' working memory (Zacharia, 2015). In a VME environment, touch sensory feedback is not available. Nonetheless, the virtual materials present some potential advantages that may also enhance learning. According to the cognitive theory of multimedia learning (Mayer, 2002), there are two key features of computer simulations that can minimize the risk of students' cognitive overload. The first, called the contiguity effect, is their ability to present related information (e.g., visual and textual) simultaneously, including some type of information not available with physical manipulatives (e.g., nonvisible material properties). The second, called the coherence effect, is their potential to reduce irrelevant details that may distract the learner (Rau, 2020). Finally, the concept of salient theory (Rau, 2020) suggests that what is important for learning is not the mode of experimentation itself but rather the way in which it is used to make the target concept more salient. This may explain (Rau, 2020) the mixed results of past studies in terms of determining the mode of representation that is more effective, as the effectiveness of the manipulatives depends on the concepts that they illustrate. 1.2.2 | Empirical results Given the differing theoretical advantages associated with both modes of experimentation (PME and VME), it is not surprising that the empirical research so far is not conclusive about the one that is better for improving students' conceptual understanding (Rau, 2020;Zacharia,2015). In elementary and secondary education, the results from studies comparing PME and VME in recent decades are mixed: some favor PME (e.g., Lazonder & Ehrenhard, 2014;Zachariaetal.,2012), some VME (e.g., Gecu-Parmaksiz & Delialioglu, 2019; Wang & Tseng, 2018) and most yield no differences between them (e.g., Klahr et al., 2007; Pyatt & Sims, 2012; Renken & Nunez, 2013; Stull et al., 2013;Triona&Klahr,2003;Zacharia& Michael, 2016). At the university level, PME and VME were generally found to be equally conducive to students' learning (e.g., Baki et al., 2011;Han,2013; Pan et al., 2015; Wiesner & Lan, 2004; Zacharia & Constantinou, 2008;Zacharia& Olympiou, 2011). One exception is the study conducted by Finkelstein et al. (2005), in which physics students using a simulated circuit laboratory showed higher conceptual understanding than those using real equipment. In a meta-analysis covering research comparing physical and virtual labs from the past 10 years, Tsichouridis et al. (2019) concluded that, in primary education, physical materials were found in most cases to be more effective for learning than virtual ones, while in secondary and university education, both modes of experimentation were, in general, equally effective. Given the insufficient findings supporting the effectiveness of touch sensory feedback for learning, some authors have hypothesized that it is manipulation and not physicality that is important for learning (Pouw et al., 2014; Zacharia & Olympiou, 2011). Recent research has identified two conditions that should be met for touch sensory feedback to make a difference in students' conceptual understanding. It has been found that physicality is more effective (1) when touch is the only modality that can provide the relevant information about the phenomenon under study (e.g., force-related subject domains) (Lazonder & Ehrenhard, 2014) and (2) when the students have not previously embodied the perceptual experiences that facilitate their understanding of the related concepts (Zacharia et al., 2012). However, these assumptions have not been consistently verified (Zacharia, 2015). Regarding the first condition, some studies (Chini et al., 2012; Klahr et al., 2007) have found significant differences in the effectiveness of physical and virtual manipulatives in cases when the concepts involved were force, torque, or friction, in which tactile information is clearly relevant. Regarding the second condition, the difficulty of measuring individuals' prior embodied knowledge has caused the majority of studies to not account for this effect (Zacharia, 2015). In summary, despite recent research efforts devoted to investigating the relative efficacy of physical and virtual manipulation in STEM education, the mechanisms by which the presence of touch sensory feedback affects the learning process are still unclear. The existence of contradictory findings suggests the need for further research (Zacharia, 2015), especially in subject domains in which tactile information is highly relevant. Some recent studies (e.g., De Jong et al., 2013; Zacharia & Michael, 2016) have investigated the benefits of combining PME and VME, concluding that, in some cases, the use of a blended PME and VME environment can be more conducive to conceptual understanding than the use of PME or VME alone. It has been argued that a PME/VME blended environment should combine the PME and VME affordances that best serve the learning objectives of the course or experiment (Olympiou & Zacharia, 2012). Typical affordances of the physical materials include providing an JUSTO ET AL.815 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License authentic concrete experience or sensing certain characteristics of a concrete object, while affordances often associated with virtual materials include observing phenomena that are not visible in real life or making accurate and quick experiments (Olympiou & Zacharia, 2012). Table 1lists a summary of well-controlled empirical studies using pretest/posttest designs to compare the efficacy of physical and virtual manipulatives for promoting conceptual understanding in STEM domains. We selected studies published in peer-reviewed journals or books and excluded conference papers, unpublished dissertations, and reports. Their findings on motivation were also included. Effect sizes were listed when appropriate. 1.3 |Effects of PME and VME on motivation 1.3.1 | Theoretical underpinnings Motivation is known to be an important factor that drives learning. One of the most important theories that explains motivation is the expectancy-value theory (Wigfield & Eccles, 2000), which assumes that the degree to which students are motivated to achieve a given task is a function of (a) their expectations for success and (b) the perceived task value. Based on the expectancy-value theory of motivation, Keller (1987) developed the ARCS model for the motivational design of learning materials. In Keller's model, the two categories of the expectancy-value theory were expanded to four: Value was subdivided into two categories: interest and relevance, to make a distinction between a set of constructs that are concerned primarily with curiosity and arousal (interest) versus other motives, such as “need for achievement” and “perceived utility”(relevance). According to Keller, both categories have an influence on how the students value the learning task, but “interest refers more to attentional factors in the environment, and relevance refers more to goal directed activity”(Keller, 1987, p. 3). Expectancy refers to one's expectation of being successful. It includes two important characteristics: self-confidence and a feeling of controlling the task. To include the effect on motivation of the reinforcing value of instruction, a fourth category, called outcomes, was added. After that, the four categories were renamed as attention, relevance, confidence, and satisfaction to strengthen the central feature of each and to form a useful acronym (ARCS). Attention has to do with stimulating curiosity and arousal. Attention should be gained and sustained. Common strategies that enhance attention include promoting interaction, introducing cognitive conflict, and pursuing variability in the format and medium of instruction. Relevance refers to the perceived utility of the task. Present worth and the future usefulness of the learning task have a positive influence on the students' engagement. Also, providing personal choices increases relevance. Confidence depends on the expectancy of success at the learning task. Clearly stated goals, self-confidence, a feeling of controlling the task, and an appropriate level of difficulty are key in promoting confidence. Satisfaction is related to reward and reinforcement strategies. It includes intrinsic and extrinsic rewards that should be provided with an appropriate reinforcement schedule in order to stimulate students' continuing desire to learn (Keller, 2010). To measure the four dimensions of motivation in the ARCS model, Keller (2006) developed and validated the Instructional Materials Motivation Survey (IMMS), a 36-item questionnaire that has been extensively used for assessing the levels of students' motivation to learn with a wide range of instructional materials (Cabero et al., 2019; Huang et al., 2006). 1.3.2 | Empirical results Despite the widely accepted view that direct contact with physical materials can increase motivation to learn in STEM disciplines (Haury & Rillero, 1994; Rau, 2020), there is a lack of empirical studies providing quantitative evidence in support of this assumption. Many of the existing studies have a qualitative approach based on observation and interviews. Others are not specifically focused on motivation but rather on students' general attitudes and measure variables, which are closely related to, but not synonymous with, motivation (e.g., engagement or enjoyment). In most cases, the validity of the results is compromised since the authors did not use validated scales of motivation. Besides, the results of the limited number of studies investigating the comparative effect of PME and VME on motivational outcomes are not consistent. Some studies reported higher enjoyment in VME students (Hsiao, 2001; 816 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License TABLE 1 Findings of empirical studies comparing PME and VME, ordered by age level Findings Study NAge level Design Topics Conceptual understanding Motivation Effect size (d) Zacharia et al. (2012) 80 Kindergarten PME versus VME Controlling prior knowledge Mass and beam balance PME > VME if incorrect prior knowledge 5.64 Gecu-Parmaksiz and Delialioglu (2019) 72 Kindergarten PME versus VME Geometric shape recognition PME < VME 1.17 Wang and Tseng (2018) 208 3rd grade PME versus VME versus PME +VME Evaporation Condensation PME +VME =VME > PME 0.50 Triona and Klahr (2003) 92 4th and 5th grade PME versus VME Springs, weights, and ramps PME =VME Lazonder and Ehrenhard (2014) 60 5th grade PME versus VME Mass and size in falling objects PME > VME for mass PME =VME for size 0.82 Zacharia and Michael (2016) 55 6th grade PME versus VME versus PME +VME Electric circuits PME +VME > PME or VME alone Renken and Nunez (2013) 147 7th grade PME versus VME Length, mass in pendulum PME =VME PME =VME Klahr et al. (2007) 56 7th and 8th grade PME versus VME Assemble and test mousetrap cars PME =VME Husnaini and Chen (2019) 68 10th grade PME versus VME Pendulum motion PME =VME for simple PME < VME for difficult concepts PME =VME Chen et al. (2014) 68 11th grade PME versus VME Boyle's law PME =VME PME > VME Zacharia and Constantinou (2008) 68 University PME versus VME Heat Temperature PME =VME Zacharia and Olympiou (2011) 234 University PME versus VME versus PME +VME versus Traditional Heat Temperature PME =VME =PME + VME > Traditional 1.6 Chini et al. (2012) 121 University PME versus VME Force, mechanical advantage, and work in pulleys PME =VME in force and mech. VME > PME in work 0.50 Han (2013) 48 University PME versus VME Gear's direction, ratio, speed, torque, work in car design PME =VME Stull et al. (2013) 29 University PME versus VME Stereochemistry (diagrams) PME =VME (Continues) JUSTO ET AL.817 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License TABLE 1 (Continued) Findings Study NAge level Design Topics Conceptual understanding Motivation Effect size (d) Zacharia and De Jong (2014) 194 University PME versus VME Electric circuits PME =VME simple circuits PME < VME complex circuits 1.64 Pan et al. (2015) 70 University PME versus VME versus Traditional Mechanical reasoning in dynamic systems PME =VME > Traditional Wiesner and Lan (2004) 24 University PME versus VME Heat exchange mass transfer Humidification PME =VME Liu et al. (2019) 26 University PME versus VME Kinematic movement and physical support of mechanisms PME =VME PME =VME Baki et al. (2011) 96 University PME versus VME versus Traditional Spatial visualization PME =VME > Traditional Abbreviations: PME, physical manipulative experimentation; VME, virtual manipulative experimentation. 818 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License Lane, 2010), some in PME students (Chen et al., 2014), while some others found no significant differences in enjoyment between both modes of experimentation (Husnaini & Chen, 2019; Renken & Nunez, 2013). On the other hand, Nolen and Koretsky (2018) measured engagement in a project simulating real-world engineering tasks and reported greater engagement in the students using virtual laboratories. Only a few researchers have used Keller's IMMS survey to measure motivation to learn in PME and VME environments. Liu et al. (2019) compared how virtual and physical learning aids enhanced learners' motivation in an industrial design course and found no significant differences in the four motivation categories of the ARCS model. Nor did Kebritchi (2008) find significant differences in motivation between high school mathematics students using VME and traditional instruction. Interestingly, in most of the studies that favored VME, the authors attributed the increase in motivation to the affordances of the virtual materials in terms of quick feedback and more effective use of time. Lane (2010), for example, found that the PME students got bored waiting for the teacher to check their work, while the VME students received feedback in real time. Similarly, Nolen and Koretsky (2018) argued that the experiments in the virtual laboratory could be performed quickly and were not constrained to a time and place. This enabled a more realistic and complex task design, which resulted in greater student engagement. On the other hand, in the studies in which quick feedback was provided in both experimentation modes, the PME students showed similar or greater levels of motivation (Liu et al., 2019) or enjoyment (Chen et al., 2014) than the VME students. For example, in the experiment by Chen et al., the PME students used a computer-assisted laboratory, which allowed for physical manipulation, but at the same time, provided real-time graphing. Another important factor that can affect motivation is the context and complexity of the project or activity in which the manipulatives are used. In most of the studies reviewed above, the manipulatives were used for learning basic science and mathematics concepts in a classroom environment (e.g., Chen et al., 2014; Hsiao, 2001). Also, in some cases, the tools were used in support of more complex activities situated in more real contexts, such as design tasks (Chesler et al., 2013; Liu et al., 2019). For example, the study by Nolen and Koretsky (2018) examined the engagement of final-year engineering students working on projects that simulated “real world”tasks, meaning students play the role of practicing engineers doing real engineering work. The experiment presented here was designed to test the extent to which students' motivation to learn basic structural engineering concepts through the use of manipulatives in a classroom setting is affected by experimentation with physical or virtual materials. The use of the ARCS model of motivation allowed for determining in close detail the components of motivation that are fostered by each mode of experimentation. 1.4 |The present study This article presents a comparative study on the effects of the mode of experimentation (PME/VME) on motivation and conceptual understanding of basic structural engineering concepts. The participants were 209 undergraduate architecture students, who were administered a pretest/posttest for conceptual understanding and Keller's IMMS to assess their level of motivation to learn the concepts through the use of PME or VME. The study was carried out in a first-year introductory structural engineering course at the Seville School of Architecture (Spain). In contrast to most previous studies, the research presented in this article examines the effect of PME and VME simultaneously on motivation and on conceptual understanding involving a large number of university-level students. We used a well-grounded measurement to assess students' motivation. The classroom activities with manipulatives had an active, self-directed learning approach based on experiential learning theory. The findings reported here can contribute to understanding the contrasting views about the comparative effect of physical and virtual manipulatives in promoting learning and motivation in STEM disciplines. 2|MATERIALS AND METHODS 2.1 |Research design We used a pretest/posttest quasi-experimental design with a purposely developed conceptual test to measure students' understanding of basic structural engineering concepts. The motivation was assessed with Keller's IMMS. JUSTO ET AL.819 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License The experiment followed a between-subjects design. The students were allocated to one of two conditions: PME versus VME. The research questions are: 1. How does the mode of experimentation (physical/virtual) affect students' understanding of basic structural engineering concepts through the use of manipulatives in a classroom context? 2. How does the mode of experimentation (physical/virtual) affect students' motivation to learn basic structural engineering concepts through the use of manipulatives in a classroom context? The teacher, time on task, and method of instruction were controlled to minimize variation between experimental groups. Since structural concepts are force-related concepts (Ji & Bell, 2018), the presence of touch sensory feedback should be expected to make a difference in students' understanding. Thus, the first hypothesis tested was: Hypothesis 1. Physical materials are more effective than virtual materials for improving students' understanding of basic structural engineering concepts in a classroom context In spite of the motivational value traditionally associated with PME, the empirical results of previous research in this area are inconclusive. The second hypothesis was: Hypothesis 2. Physical materials motivate the students to learn basic structural engineering concepts through the use of manipulatives in classroom activities more than virtual materials. Both the PME and VME environments were designed to facilitate the dynamic visualization of the structural response under different loading and support conditions. In addition, the students using PME received touch sensory feedback during experimentation. As they applied forces (or moments) with their own hands, they were able to experience variations in kinesthetic sensations—for example, when the type of constraint at the supports changed. Students using VME were also able to visualize the structure deflection in real time, but they applied forces with the mouse and thus obtained no kinesthetic feedback in terms of stiffness or force. 2.2 |Curriculum materials The Structures 1 course in which the experiment was conducted is a six ECTS credits (European Credit Transfer System) introductory structures course, focused on the key concepts and principles that govern structural behavior. ECTS is a point system used by European universities to express the required workload of a course. One ECTS credit normally corresponds to 30 (10 contact/20 noncontact) hours of student workload. The course has an active learning approach, mainly based on the problem-based learning method (PBL) (Justo et al., 2016). Nine 1-h instructional activities with manipulatives (Table 2) were programmed during the semester to complement the PBL problems. The aim was to facilitate the understanding of important structural concepts that were otherwise difficult for the students to visualize. The activities were focused on two basic types of structures: beams and frames. Their learning objectives were: •LO1. To draw the reaction forces of beams under common loads. •LO2. To draw the internal force diagrams of beams. •LO3. To predict the deflected shape of beams and frames, matching the support deformations to the type of constraint provided. •LO4. To explain the relationship between the bending moment and the deflected shape. •LO5. To distinguish between a structure and a mechanism. •LO6. To understand the behavior of the structure as a system. 820 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 2.2.1 | PME/VME learning environment The experiential learning theory provides a suitable framework for the instructional design of PME and VME environments (Liu et al., 2019) since it emphasizes the importance of concrete experiences for learning. It is based on the principle that “learning is best facilitated in an environment where there is dialectic tension and conflict between immediate, concrete experience and analytic detachment”(Kolb & Fry, 1975, p. 35). Kolb formulated a learning cycle with four stages: Concrete experience, observation and reflection, abstract conceptualization, and active experimentation. We designed the learning activities with manipulatives following Kolb's cycle but incorporating an initial hypothesis stage. This instructional design is aligned with the inquiry-based learning approach (Coller, 2008; Self et al., 2016), which has been widely used in STEM education. The students worked in small groups to facilitate knowledge elaboration and reflection. First, each student formulated a hypothesis. Then, the group discussed a common hypothesis and tested it through experimentation with the physical/virtual manipulatives. After that, the students reflected on the concepts involved and the misconceptions that may have caused differences between their hypotheses and the experiment. Finally, each group wrote a reflective report. The instructor was available for scaffolding and guidance but did not provide answers or explanations of the students' results. Typical scaffolding questions included: Can you explain it further? Can you clarify your idea? What structural concepts can you apply to support your hypothesis? Both learning environments were designed to offer similar manipulation affordances to the students, except for small differences, which are explained in Section 2.2.3. A sample activity is shown in Appendix A. 2.2.2 | Physical manipulative: Structural kit The structural kit (Figure 1) is an assemblable kit with bars and nodes produced by additive manufacturing with a 3D printer. It allows for building simple structures and visualizing their deformation under load. 2.2.3 | Virtual manipulative: Real-time structural simulator The Structural Simulator (Figure 2) uses video game technology to allow visualizing in real-time the response of the structure to variations in loading or support conditions. The interface is simple and straightforward to facilitate interaction and feedback. Common structures can be selected just by pressing a button. The user can apply point loads and TABLE 2 Time schedule of activities and assessment tasks Week Activities Learning objectives Assessment 1 Presentation Pretest 3 Activity 1 LO3 Activity 2 LO1, LO3, LO5 4 Activity 3 LO3, LO4 Activity 4 LO1, LO2, LO3, LO4 Activity 5 LO3, LO4 7 Activity 6 LO3 9 Posttest 1 (beams) 12 Activity 7 LO3, LO6 13 Activity 8 LO3, LO5, LO6 Activity 9 LO3, LO6 14 Posttest 2 (frames) 15 IMMS JUSTO ET AL.821 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 2.5 |Procedures The pretest for conceptual understanding was administered individually on the first day of class. The activities with manipulatives were carried out in nine different sessions during the semester (Table 2). The students worked in groups of four in the classroom. They were grouped heterogeneously based on their grades in previous courses so that lower and higher achieving students were mixed in the groups. The VME students installed the software on their laptop computers and were briefly instructed on its use before the start of the first session. They worked in groups on the activities (e.g., reaching an agreed hypothesis through discussion), but each student could experiment with the structural FIGURE 5 Wright map representing student ability (left) and item difficulty (right) for items in the attention subdimension in a logit scale. Thurstonian thresholds: The ability at which the probability of achieving a particular degree of agreement or higher reaches 50% [Color figure can be viewed at wileyonlinelibrary.com] FIGURE 6 Wright map representing student ability (left) and item difficulty (right) for items in the relevance subdimension in a logit scale. Thurstonian thresholds: The ability at which the probability of achieving a particular degree of agreement or higher reaches 50% [Color figure can be viewed at wileyonlinelibrary.com] 828 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License simulator on their computer. The PME students were given one structural kit per group, which they shared during the exercises. All the students in the group experimented with the kit in turns. Once the activities of each category (beams/frames) were concluded, the corresponding section of the posttest was administered to all the students in the classroom. At the end of the course, all the students completed the IMMS questionnaire during class time. Administering the IMMS survey as a pretest before the intervention would have been pointless since the survey questions presume that the students have actually used the materials. FIGURE 8 Wright map representing student ability (left) and item difficulty (right) for items in the satisfaction subdimension in a logit scale. Thurstonian thresholds: The ability at which the probability of achieving a particular degree of agreement or higher reaches 50% [Color figure can be viewed at wileyonlinelibrary.com] FIGURE 7 Wright map representing student ability (left) and item difficulty (right) for items in the confidence subdimension in a logit scale. Thurstonian thresholds: The ability at which the probability of achieving a particular degree of agreement or higher reaches 50% [Color figure can be viewed at wileyonlinelibrary.com] JUSTO ET AL.829 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License A teacher's guide with a weekly activity schedule and the wording of the activities was distributed to the team of instructors before the start of the course. At the end of the course, the instructors were interviewed to ensure that the intervention was delivered as planned. TABLE 5 Rasch model item fit statistics for the IMMS Outfit Infit Item Difficulty estimate Error MnSq tMnSq tRaw/max Subdimension IMMS2 0.199 0.053 0.90 1.0 0.91 0.9 502/824 Attention IMMS8 0.070 0.053 1.11 1.1 1.15 1.5 525/824 Attention IMMS11 0.070 0.053 0.74 2.8 0.76 2.6 525/824 Attention IMMS12 0.523 0.056 1.21 2.1 1.16 1.6 621/824 Attention IMMS15 0.019 0.053 1.29 2.7 1.36 3.4 534/824 Attention IMMS17 0.253 0.052 0.60 4.7 0.62 4.5 492/824 Attention IMMS20 0.193 0.053 0.67 3.8 0.69 3.5 503/824 Attention IMMS22 0.603 0.052 1.02 0.2 1.03 0.3 426/824 Attention IMMS24 0.171 0.053 0.94 0.6 0.90 1.0 507/824 Attention IMMS28 0.403 0.052 0.70 3.4 0.67 3.9 464/824 Attention IMMS29 0.121 0.054 0.76 2.6 0.78 2.4 558/824 Attention IMMS31 0.311 0.055 0.94 0.5 0.99 0.1 589/824 Attention IMMS6 0.457 0.055 0.97 0.3 0.93 0.7 608/820 Relevance IMMS9 0.338 0.055 0.55 5.4 0.58 5.0 593/824 Relevance IMMS10 0.146 0.054 0.77 2.5 0.79 2.3 562/824 Relevance IMMS16 0.093 0.054 0.78 2.4 0.80 2.2 553/824 Relevance IMMS18 0.428 0.052 0.82 1.9 0.82 1.9 459/824 Relevance IMMS23 0.113 0.053 0.63 4.3 0.63 4.4 517/824 Relevance IMMS26 0.792 0.057 1.16 1.5 1.21 2.0 657/824 Relevance IMMS30 0.222 0.053 1.05 0.5 1.04 0.5 497/824 Relevance IMMS33 0.361 0.055 0.60 4.7 0.61 4.6 596/824 Relevance IMMS1 0.002 0.053 1.30 2.8 1.25 2.4 537/824 Confidence IMMS3 0.240 0.054 1.37 3.4 1.35 3.3 577/824 Confidence IMMS4 0.222 0.053 0.81 2.1 0.77 2.5 497/824 Confidence IMMS7 0.368 0.052 1.26 2.5 1.25 2.4 470/824 Confidence IMMS13 0.016 0.053 0.66 3.9 0.65 4.2 534/824 Confidence IMMS19 0.541 0.056 1.27 2.6 1.26 2.5 623/824 Confidence IMMS25 0.374 0.052 0.86 1.5 0.84 1.8 469/824 Confidence IMMS34 0.964 0.058 0.95 0.4 1.02 0.2 678/824 Confidence IMMS35 0.147 0.053 0.48 6.6 0.49 6.6 511/824 Confidence IMMS5 0.182 0.054 0.78 2.4 0.82 2.0 568/824 Satisfaction IMMS14 0.639 0.052 0.62 4.5 0.63 4.5 419/824 Satisfaction IMMS21 0.181 0.053 0.61 4.5 0.64 4.3 505/824 Satisfaction IMMS27 0.572 0.052 0.67 3.8 0.63 4.4 432/824 Satisfaction IMMS32 0.217 0.054 0.69 3.5 0.69 3.5 574/824 Satisfaction IMMS36 0.025 Note: MnSq: infit and outfit mean square values. t:t-test statistics for the MNSQ calculation. Raw: raw measure of difficulty, calculated as the number of students providing a correct answer for the item. Max: total number of students taking the test. Abbreviation: IMMS, Instructional Materials Motivation Survey; MNSQ, mean square. 830 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 2.6 |Data analysis We used multiple regression analysis to determine whether there were statistically significant differences in conceptual understanding and motivation between the students in the PME and VME conditions while controlling for a number of potentially influential factors. For conceptual understanding, the dependent variable was the posttest score. For motivation, the dependent variables were the IMMS score and the four subscores (attention, relevance, confidence, and satisfaction). The independent variables were the experimental condition (physical/virtual), the pretest score, the teacher, and the students' age and sex. We used six dummy variables to code the teacher variable, with Teacher E serving as the reference group. The experimental condition (physical/virtual) and the students' sex were also coded as dummy variables, with PME and females, respectively, as reference groups. A significance level of 0.05 was applied in all the tests. In addition, we conducted a Spearman correlation analysis to examine the association between motivation and conceptual understanding. 3|RESULTS 3.1 |Conceptual understanding Table 6provides means and standard deviations of pretest and posttest scores. The pretest scores of the PME and VME groups did not differ significantly according to an independent sample t-test (t(196) =1.217, p=.225, CI 95 =[1.68, 7.1]), so we can assume that the students of both groups had similar levels of prior knowledge. The learning gains measured using the average normalized gain ⟨g⟩(Hake, 1998) were almost identical. Figure 9shows a box plot of the posttest results. Table 7shows the results of the multiple regression analysis for conceptual understanding. We found no main effect of media (physical/virtual) on conceptual understanding (B=0.142, SE B =0.333, β=0.037, p=.669). The teacher effect was also not significant. As expected, the pretest score was a positive predictor of conceptual understanding. We also found a main effect of sex on the posttest score (males scored significantly better than females). We calculated an additional regression model, including sex condition as an interaction effect, and found no significant effect for the interaction of sex and experimental condition. The proportion of variance (R 2 ) accounted for by all the predictors was 0.128. The model predicts quite a small amount of the actual variance in conceptual understanding, suggesting that there are still unexplained factors that affect students' learning. The R 2 value was significant with p=.004. Effect sizes were estimated with Cohen's d, using the unstandardized coefficients of the regression model as a measure of mean differences between groups. We obtained a medium effect size (d=0.45) for the effect of sex on the posttest score. 3.2 |Motivation The results of the motivation survey are listed in Table 8. A box plot of the motivation scores is displayed in Figure 10. The results of the multiple regression analysis for motivation (overall score) and its components (ARCS subscores) are presented in Table 7. The analysis shows that PME was more effective than VME in motivating the students (B=0.242, SE B =0.089, β=0.221, p=.007). The difference was highly significant in the attention factor TABLE 6 Means and standard deviations of pretest/posttest scores for PME and VME Pretest Posttest Condition NMSD MSD Learning gain <g> PME 104 3.68 1.57 6.18 1.88 0.40 VME 94 3.41 1.55 6.01 1.98 0.39 Abbreviations: PME, physical manipulative experimentation; VME, virtual manipulative experimentation. JUSTO ET AL.831 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License (B=0.470, SE B =0.104, β=0.344, p< .001), while for the rest of the motivational factors, no significant differences were obtained. The teacher also had a main effect on motivation and its components. The age of the participants was found to be a positive predictor of motivation, attention, relevance, and satisfaction. We also observed a main effect of sex on confidence (women were less confident than men). To check whether this men-women confidence discrepancy was dependent on the instructional medium, we carried out separate analyses in the physical and virtual groups and found that the differences were only encountered in the physical group (B=0.237, SE B =0.099, β=0.223, p=.019). An interaction graph is provided in Figure 11 to depict how the effect of sex on confidence depended on the type of material used (physical/virtual). The proportion of variance (R 2 ) of motivation accounted for by all the predictors was 0.499 (p< .001), which indicates that the model explained the factors affecting motivation fairly well. Medium and large effect sizes were obtained for motivation and attention (d=0.455 and 0.771, respectively). Standard errors were estimated under robust option in order to assess heteroscedasticity issues. To ensure that the presence of outliers did not disturb the models, we tested discrepancy measures in each model. Studentized residuals were within the range (3, 3), which guarantees the absence of problematic extreme values (Belsley et al., 1980). Finally, the results of the Spearman test applied to all the students (ρ=0.043, p=.584) and to each group separately (PME: ρ=0.091, p=.422; VME: ρ=0.128, p=.256) showed no correlation between students' motivation and their conceptual understanding. 4|DISCUSSION The first purpose of the present study was to determine whether the use of physical materials was important in promoting students' understanding of basic structural engineering concepts (Hypothesis 1). Our results indicate that the students using physical manipulatives showed similar learning gains to those using virtual manipulatives. This finding is in line with the conclusions of the majority of previous studies comparing PME and VME for conceptual understanding. The lack of a physical/virtual effect is especially relevant in the case of the present study since the subject domain (forces and deformations) is one in which tactile manipulation might be expected to contribute to the learning process. This seems to contradict the claim made by some authors (Lazonder & Ehrenhard, 2014) that physical manipulation is more effective for learning concepts in which tactile information is relevant. One possible explanation for this lack of effect, according to the embodied cognition theory, is that the students in the virtual condition had, by past experiences, internalized the relevant perceptual knowledge that the structural kit would have given them. We must keep in mind that the majority of studies wherein a physical-virtual effect was found FIGURE 9 Box plot for conceptual understanding results, measured through the posttest [Color figure can be viewed at wileyonlinelibrary.com] 832 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License TABLE 7 Results of multiple regression analysis Dependent variables Conceptual understanding Motivation Attention Relevance Confidence Satisfaction Independent variables B(S.E.) βB(S.E.) βB(S.E.) βB(S.E.) βB(S.E.) βB(S.E.) β Condition PME/VME 0.142 0.037 0.242*** 0.221 0.470*** 0.344 0.135 0.117 0.152 0.145 0.082 0.051 (0.312) (0.089) (0.104) (0.097) (0.100) (0.150) Age 0.089 0.114 0.044*** 0.178 0.049*** 0.159 0.055*** 0.211 0.010 0.042 0.067*** 0.186 (0.061) (0.015) (0.017) (0.017) (0.018) (0.024) Sex 0.859** 0.222 0.100 0.091 0.036 0.026 0.080 0.069 0.148** 0.141 0.183* 0.114 (0.271) (0.062) (0.075) (0.071) (0.075) (0.108) Pretest 0.304*** 0.247 0.011 0.030 0.031 0.070 0.021 0.057 0.045* 0.134 0.038 0.074 (0.086) (0.022) (0.026) (0.023) (0.025) (0.036) Teacher A 0.193 0.040 0.153 0.118 0.186 0.115 0.082 0.060 0.137 0.110 0.215 0.113 (0.527) (0.103) (0.123) (0.131) (0.112) (0.164) Teacher B 0.195 0.041 0.923*** 0.580 0.871*** 0.437 0.92*** 0.546 0.764*** 0.501 1.269*** 0.546 (0.534) (0.107) (0.128) (0.127) (0.135) (0.181) Teacher C 0.319 0.063 0.612*** 0.369 0.699*** 0.337 0.637*** 0.363 0.352** 0.221 0.790*** 0.327 (0.552) (0.147) (0.183) (0.186) (0.159) (0.229) Teacher D 0.756 0.145 0.749*** 0.553 0.767*** 0.453 0.758*** 0.529 0.564*** 0.435 0.975*** 0.493 (0.547) (0.135) (0.144) (0.153) (0.140) (0.238) Teacher G 0.304 0.035 0.438*** 0.174 0.427** 0.136 0.613*** 0.230 0.134 0.056 0.652*** 0.178 (0.725) (0.140) (0.205) (0.167) (0.143) (0.224) Teacher F 0.716 0.115 0.621*** 0.382 0.882*** 0.434 0.400** 0.233 0.313* 0.201 0.891*** 0.376 (0.664) (0.148) (0.181) (0.165) (0.172) (0.243) R-squared 0.128*** 0.499*** 0.524*** 0.422*** 0.309*** 0.355*** Note: Condition: 0 =PME, 1 =VME; Sex: 0 =Female, 1 =Male; Reference teacher =Teacher E; B =unstandardized coefficient; β=standardized coefficient; S.E. =robust standard error. Abbreviations: PME, physical manipulative experimentation; VME, virtual manipulative experimentation. *p< .1; **p< .05; ***p< .01. JUSTO ET AL.833 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License were the studies with young children (Lazonder & Ehrenhard, 2014; Tsichouridis et al., 2019; Zacharia et al., 2012), while in our study, the participants were university-level adults. Adults, unlike children, would have probably embodied the relevant perceptual knowledge from prior physical experiences (De Jong et al., 2013), in which case the presence of touch sensory feedback is no longer a necessary condition for improved understanding. Further research, to validate this assumption, should undertake the design of experiments to assess the availability of perceptual information representing the phenomenon of interest in students' long-term memory. Another possible explanation is that the embodied cognition benefits of the physical materials could have been compensated by the advantages of the virtual materials in terms of reducing the students' cognitive load. The reduction of TABLE 8 Means (M) and standard deviations (SD) of IMMS score and subscores Motivation Attention Relevance Confidence Satisfaction Condition NM SD MSD MSD MSD MSD PME 115 3.73 0.61 3.78 0.67 3.81 0.62 3.71 0.60 3.57 0.93 VME 91 3.41 0.44 3.21 0.55 3.61 0.54 3.55 0.48 3.31 0.66 Abbreviations: IMMS, Instructional Materials Motivation Survey; PME, physical manipulative experimentation; VME, virtual manipulative experimentation. FIGURE 10 Box plot depicting the results of students' motivation to learn structural engineering concepts through the use of manipulatives, measured through the IMMS total score. IMMS, Instructional Materials Motivation Survey [Color figure can be viewed at wileyonlinelibrary.com] 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 elaMelameF Confidence Gender Physical Virtual FIGURE 11 Interaction effect of sex and the type of material used (physical/virtual) on confidence subscore of the IMMS. IMMS, Instructional Materials Motivation Survey [Color figure can be viewed at wileyonlinelibrary.com] 834 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License the cognitive load in the virtual group could be explained in the present case by two main principles of the cognitive theory of multimedia learning (Mayer, 2002): the coherence effect and the contiguity effect. The coherence effect postulates that the cognitive load is reduced when irrelevant details that may distract the learner are excluded. In this case, the structural simulator can deliver a purer and more accurate representation of the structure deflections, while the structural kit is not free from the small inaccuracies that are inherent in real materials. The contiguity effect states that the cognitive load is reduced when related parts of information are presented simultaneously. In our experiment, the structural simulator allowed for a quick and dynamic visualization of the structural response to variations in the support conditions, usually with a simple mouse click. Finally, our findings are consistent with the claim made by some authors (Zacharia & Olympiou, 2011) that manipulation, and not physicality, is what is important for learning, although, in the absence of a control group, this assumption cannot be directly supported. The second purpose of this study was to investigate the effect on students' motivation to learn the concepts of the use of physical materials compared to virtual simulations (Hypothesis 2). The results of the IMMS test revealed that PME was more effective than VME in promoting motivation. Our results apparently contradict those of some previous studies measuring motivation-type variables (e.g., Lane, 2010), which in most cases reported findings favoring virtual materials. Yet, the main reasons claimed for the increase in motivation with the use of virtual materials are related to their superiority in providing quick feedback and facilitating more effective use of time. But in the present study, both modes of experimentation provided real-time feedback, as the students in the physical condition (Structural Kit) were able to apply forces with their hands and observe the structural response in real time. Compared to traditional laboratory exercises in which numerical data should be recorded and analyzed, the qualitative nature of the PME activities proposed here allowed for quicker and more intuitive experimentation of the structural behavior. On the other hand, the results of the current study do not confirm those of Liu et al. (2019), who also used IMMS to assess motivation and found no significant differences between PME and VME, even when the context of both studies was very similar; engineering students experimenting with structures or mechanisms to observe and understand, in real time, their behavior. The main difference is one of scale. The study by Liu et al. included only 13 students per condition, while our study enrolled 111 students in the physical condition and 98 in the virtual condition. Still, to further clarify the controversy about the motivational value of physical and virtual materials, more studies using validated scales of motivation, and preferably a large number of participants, would be needed. A closer inspection of the results of the IMMS test allowed discerning in greater depth the influence of the mode of experimentation on the four categories of motivation. Specifically, our results revealed that physical materials were especially effective in gaining the students' attention. The large effect size for this finding indicates a high practical significance. In the rest of the motivational factors (relevance, confidence, and satisfaction), no significant differences were found. During the development of the ARCS model of motivational design, Keller (1987) found that facilitating the students' active interaction with the materials is an educational strategy that enhances attention. In the case presented here, it could be argued that the students were more active when they interacted directly with the physical materials, applying forces with their hands and receiving tactile and kinesthetic feedback, than when they interacted with the virtual materials through a computer screen. The teacher was also found to affect the students' motivation. This is an unsurprising finding that nevertheless emphasizes the need to control the influence of the teacher variable when assessing the motivational value of physical/ virtual materials. Teacher E (the one achieving higher motivational values) was taken as a reference. We found significant differences in motivation between the reference teacher and the rest of the instructors, with the exception of Teacher A. In some cases (e.g., Teachers B and D), the impact of the instructor was very important. The differences we similar in all dimensions of motivation, except in confidence, in which the instructor's influence seemed to be less important. The students' age was also found to be a positive predictor of motivation, with older students being more motivated. It should be taken into account that, in our study, age variations are small (the standard deviation for students' age is 2.2 years). Thus, the conclusions of previous studies (e.g., Tsichouridis et al., 2019) comparing different age levels (children vs. adolescents vs. adults) are not useful in order to interpret this age effect since the participants in our study are all at the same educational level. An interesting finding of the present study is the lack of a correlation between motivation and conceptual understanding. It seems that the increase in motivation, particularly in the attention variable, did not lead to the students in the physical condition learning more. Attention has often been considered a prerequisite for learning (Keller, 1987). JUSTO ET AL.835 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License Keller's claim and our own findings suggest that attention is a necessary but not a sufficient condition for learning. Thus, gaining students' attention does not guarantee learning. In this particular case, we should also bear in mind that our measure of motivation was based on the students' opinions, while the assessment of learning gains was based on the students' performance. At the same time, the IMMS questions were precisely focused on the manipulatives, while the students' performance in the conceptual test could have been moderated by the rest of the course's learning resources. These issues point to the need for further research to clarify the relationship between motivation and conceptual understanding in PME and VME environments. We found a main effect of sex on conceptual understanding. Women performed significantly worse than men in the posttest, even though their mean scores in the pretest were similar. However, the effect of sex on the posttest seemed to be independent of the mode of experimentation (physical/virtual) that the students were using. As for the effect of sex in the motivational variables, we found that women scored significantly lower than men in confidence. In a similar experiment with secondary students, Klahr et al. (2007) also found that girls had lower confidence in their knowledge than boys, but in their study, the confidence discrepancies did not translate into differences in performance. To verify whether students' confidence affected their posttest performance, we performed an additional multiple regression analysis. The results of the analysis revealed that there is no main effect of confidence on conceptual understanding. This suggests that the sex differences in the posttest must have another explanation. Interestingly, in Klahr's study, the boy-girl confidence difference was not affected by the mode of experimentation, while in our study, the discrepancy was only found in the group using the physical materials. It is known (Klahr et al., 2007) that women are less confident about their performance in “tasks that are stereotypically considered masculine”(Beyer, 2002). This indicates that the women in the physical group might have associated the physical version of the manipulatives (structural kit) with stereotypical masculine tasks. Meanwhile, in the virtual group, it appeared that the “virtualization”of the materials eliminated the sex-related lack of confidence. A more specific, sex-based study, would be needed to investigate these issues in greater detail. 4.1 |Implications The findings of the present study have several implications for engineering educators. First, it was found that PME and VME were equally effective in promoting an understanding of structural engineering concepts. This result seems to confirm the idea that in simple classroom tasks aimed at learning basic concepts, physical manipulatives can be substituted by their virtual equivalents without the danger of compromising learning, even in disciplines such as structural engineering in which tactile concepts (e.g., force) are involved. However, the second finding emerging from the present study is that the actual touch of physical materials motivated the students. Therefore, given the similar learning gains obtained with both types of materials, educators seeking to enhance students' motivation to learn should consider using PME in their classrooms. Furthermore, physically manipulating the materials appeared to have a clear positive effect on gaining and maintaining the students' attention. The closeness of real-world materials provided chances for a more active interaction, including tactile and kinesthetic sensations, which seemed to capture students' attention more than simply interacting with mouse clicks through a computer screen. On the other hand, designers of virtual learning software could help improve motivation in the virtual mode by implementing features to gain and maintain students' attention. Based on our findings, we suggest that engineering educators should as well consider the option of combining physical and virtual manipulatives in a blended learning environment that could benefit from the increase in motivation offered by PME while still profiting from some of the unique affordances offered by VME, such as the possibility of observing phenomena that cannot be observed in real life or of making accurate and quick experiments (Olympiou & Zacharia, 2012). In the case explained in this article, for example, the Structural Simulator could be easily improved to provide important conceptual information that is not visible in real life (e.g., internal forces) while keeping the advantages of the structural kit in terms of increased attention and motivation, and also for “feeling”the forces. The learning activities could be easily reformulated to use a combination of both modes of experimentation, whether sequentially or concurrently. Finally, engineering educators using physical manipulatives should also be aware of their potential negative effects on female students' confidence, especially in cases when their use may be associated with stereotypical male tasks. 836 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 4.2 |Limitations and future research The fact that the study was conducted in a natural setting provides our findings with ecological validity but also implies that they should be taken with caution, as in the context of a normal course, there may be other sources, apart from their experience with manipulatives, that could contribute to the students' learning. An important limitation of the present study is that the students' learning gains were measured through a multiplechoice test. Several authors have historically objected (e.g., Kozma, 1994) that the effect of media on learning cannot be fully understood using responses in a test. As a consequence, further research is needed, using a qualitative approach, in order to achieve a deeper understanding of the cognitive, affective and social processes by which learning with physical/virtual manipulatives occurs. For our part, a follow-up paper is presently being prepared that examines in more depth the learning process that takes place during PME and VME using interviews and open-ended surveys. Although the focus of the present study is on the mode of experimentation (physical/virtual), in a real classroom setting, learning and motivation are influenced by the complete instructional experience, which includes instructor practices, the instructional method, and the mode of experimentation. The instructor, in particular, was found to have a great effect on students' motivation in the present study. Future studies, to take this into account, should investigate in depth the role of the instructor and the instructional method in order to measure all possible interactions among these three factors (instructor, method, medium) on conceptual understanding and motivation. 5|CONCLUSIONS The current study examines the comparative effect of physical and virtual materials on architecture students' conceptual understanding and motivation to learn structural engineering concepts through the use of manipulatives. Our results revealed no significant differences in conceptual understanding between students in the physical and virtual conditions. We also found that the students were more motivated to learn using physical manipulatives than with virtual ones. Among the four motivational factors included in the ARCS model of motivation (attention, relevance, confidence, and satisfaction), physical manipulatives were particularly effective in enhancing the students' attention. These findings suggest that the benefits of using physical materials for learning basic engineering concepts could be associated with increased motivation rather than with superior learning gains. For this reason, engineering educators seeking to enhance students' motivation should consider adopting the use of physical manipulatives in their classrooms. ORCID Enrique Justo https://orcid.org/0000-0003-3860-4419 Antonio Delgado https://orcid.org/0000-0001-7085-1212 Carmen Llorente-Cejudo https://orcid.org/0000-0002-4281-928X Rocio Aguilar https://orcid.org/0000-0002-2599-8159 Julio Cabero-Almenara https://orcid.org/0000-0002-1133-6031 REFERENCES Baki, A., Kosa, T., & Guven, B. (2011). A comparative study of the effects of using dynamic geometry software and physical manipulatives on the spatial visualisation skills of pre-service mathematics teachers. British Journal of Educational Technology,42(2), 291–310. https://doi. org/10.1111/j.1467-8535.2009.01012.x Belsley, D. A., Kuh, E., & Welsch, R. E. (1980). Identifying influential data and sources of collinearity. Regression diagnostics. Wiley. Beyer, S. (2002). The effects of gender, dysphoria, and performance feedback on the accuracy of self-evaluations. Sex Roles,47(9), 453–464. https://doi.org/10.1023/A:1021600510857 Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE—Life Sciences Education,15(4), rm4. https:// doi.org/10.1187/cbe.16-04-0148 Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer Science+Business Media. https://doi.org/10. 1007/978-94-007-6857-4 Cabero, J., Barroso, J., Llorente, C., & Fern andez, M.d M. (2019). Educational uses of augmented reality (AR): Experiences in educational science. Sustainability,11(18), 4990. https://doi.org/10.3390/su11184990 Carbonneau, K., Marley, S., & Selig, J. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology,105(2), 380–400. https://doi.org/10.1037/a0031084 JUSTO ET AL.837 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License APPENDIX C RASCH ANALYSIS WITH SEX DIFFERENCE (DIF) FIGURE C1 Wright map representing student ability and item difficulty for male students (posttest), in a logit scale. Raw: Raw measure of ability, computed as the number of correct answers in the test 844 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License FIGURE C2 Wright map representing student ability and item difficulty for female students (posttest), in a logit scale. Raw: Raw measure of ability, computed as the number of correct answers in the test JUSTO ET AL.845 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License TABLE C1 Rasch model item fit statistics for the posttest: Male students Outfit Infit Item Difficulty estimate Error MnSq tMnSq tRaw/max i1 0.044 0.190 0.76 1.5 0.79 1.5 43/65 i2 1.321 0.186 0.99 0.0 0.99 0.0 25/65 i3 2.073 0.237 0.90 0.5 0.96 0.3 60/65 i4 1.321 0.186 1.06 0.4 1.07 0.5 25/65 i5 1.693 0.190 0.83 1.0 0.84 1.2 20/65 i6 0.271 0.187 0.77 1.4 0.81 1.4 40/65 i7 1.110 0.184 0.92 0.4 0.92 0.6 28/65 i8 1.136 0.216 0.94 0.3 1.01 0.1 55/65 i9 0.764 0.184 1.30 1.6 1.33 2.3 33/65 i10 0.284 0.196 0.78 1.3 0.80 1.4 47/65 i11 2.812 0.249 0.90 0.5 0.92 0.6 62/65 i12 0.561 0.202 0.78 1.3 0.84 1.1 50/65 i13 0.344 Note: MnSq: Infit and outfit mean square values. t:t-test statistics for the MNSQ calculation. Raw: raw measure of difficulty, calculated as the number of students providing a correct answer for the item. Max: total number of students taking the test. Abbreviation: MNSQ, mean square. TABLE C2 Rasch model item fit statistics for the posttest: Female students Outfit Infit Item Difficulty estimate Error MnSq tMnSq tRaw/max i1 0.464 0.172 1.25 1.5 1.27 2.0 52/75 i2 0.716 0.166 1.43 2.4 1.43 3.2 32/75 i3 0.668 0.176 0.99 0.0 1.00 0.0 55/75 i4 0.893 0.168 1.13 0.8 1.13 1.1 29/75 i5 1.075 0.170 1.16 1.0 1.17 1.3 26/75 i6 0.670 0.176 1.11 0.7 1.13 1.0 55/75 i7 0.373 0.165 0.98 0.1 0.98 0.1 38/75 i8 0.819 0.178 0.93 0.4 0.94 0.5 57/75 i9 0.086 0.166 1.06 0.4 1.06 0.6 43/75 i10 0.371 0.165 0.54 3.3 0.55 4.6 38/75 i11 1.346 0.189 0.91 0.5 0.92 0.6 63/75 i12 0.031 0.167 0.78 1.4 0.78 1.9 45/75 i13 0.485 Note: MnSq: infit and outfit mean square values. t:t-test statistics for the MNSQ calculation. Raw: raw measure of difficulty, calculated as the number of students providing a correct answer for the item. Max: total number of students taking the test. Abbreviation: MNSQ, mean square. 846 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License APPENDIX D POSTTEST OF STRUCTURAL CONCEPTS 1. What is the reaction moment at A? Choose the correct option. 2. What are the magnitude and direction of the reaction force at A? 3. Choose the correct deformed shape of the fixed-hinged bar in the figure. JUSTO ET AL.847 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 4. Given the bending moment diagram of the cantilevered beam in the figure, choose its correct deformed shape. 5. Choose the correct sign of the bending moment at the edges (A, B) of the cantilevered beam in the figure. 6. What is the bending moment at A? Choose the correct option. 848 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 7. Choose the correct internal force diagrams of the beam in the figure. 8. Which is the correct stress distribution through the beam's cross section at A? 9. What would change in the beam shown below if we doubled the height h of its cross section? JUSTO ET AL.849 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 10. Choose the correct deformed shape of the structural frame in the figure. 11. Choose the correct deformed shape of the structural frame in the figure. 850 JUSTO ET AL. 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 12. What is the bending moment at sections A, B, and C of the frame in the figure? Choose the correct answer. 13. If the frame in the figure is broken up into individual members, what are the contact forces of each member? Choose the correct answer. JUSTO ET AL.851 21689830, 2022, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/jee.20482 by Universidad De Sevilla, Wiley Online Library on [24/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License