Learners engagement improving and fatigue prevention model based on the analysis of their eye movement traces

The 17th International Scientific Conference eLearning and Software for Education Bucharest, April 22-23, 2021 10.12753/2066-026X-21-090 LEARNERS ENGAGEMENT IMPROVING AND FATIGUE PREVENTION MODEL BASED ON THE ANALYSIS OF THEIR EYE MOVEMENT TRACES Aleksandrs GORBUNOVS, Dr.sc.ing., senior researcher Distance Education Study Centre, Riga Technical University, Kronvalda Blvd. 1, Riga, LV-1010, Latvia [email protected] Abstract: In modern world the technologies permeate our daily lives. They have entered a wide range of our work and leisure areas. Technologies are continuing their expansion with rapid strides. It is quite clear that someone has to know these technologies and be able to use them. The knowledge society needs smart, intelligent, well-educated and trained professionals. Despite the progress made in improving an efficiency of the learning process, including an introduction of new digital teaching methodologies, pedagogical principles and models, the searching attempts for the new tools and approaches, in order to identify students' learning experiences, continues to attract the attention of education professionals. Eye tracking might be considered as the one of such approaches that could help assess the e-learning system users’ experience. Learners’ gaze data can give to researchers an important valuable information, ensure an understanding about information system users’ learning preferences, e.g. the type and form of digital learning object, distraction parameters, and so on. These data may warn us also about learners’ fatigue signs, hence, giving impetus to take appropriate measures to prevent such negative effect during knowledge acquisition process. This paper is dedicated to identify and propose the new technological training model based on the learners’ gaze data in the digital learning environment, i.e. in learning management systems, expressed points and areas of interest there, time spent in them, as well as eye movements sequence. The article provides insights and offers suggestions for increasing users engagement in learning process and reducing or even preventing the effects of possible fatigue. Keywords: area of interest; e-learning; eye-tracking; eye movements; fatigue; gaze; user experience. INTRODUCTION For decades and centuries, teachers and philosophers tried to find the most effective ways to deliver and perceive knowledge, as well as factors that impact knowledge acquisition. Nowadays, in rapidly changing working and social environments, an issue of quality and efficiency for preparing of well-educated and trained professionals has become truly challenging. New digital technologies enrich teaching staff with additional educational capacities and give new possibilities to construct learning process, finding ways to improve the speed and efficiency of knowledge transfer and sharing [34]. In the last few years serious gaming is turned on learners’ engagement and creativity tool. This approach has found its application not only in civilian universities but also in military ones [40]. Artificial intelligence enthusiasts update learning management systems embedding them with more sophisticated tools and plugins, allowing determine each learner performance during e-learning course and provide appropriate learning content for their further education themes or steps by learning object type and difficulty level [20]. Covid-19 pandemic challenges remind us again about the necessity to find effective digital learning solutions [2], [17], [32]. Bio-sensors ought to be considered as another learning efficiency improvement category. In a few previous decades, a lot of significant improvements were made in bio-sensors technologies, eye trackers in particular [28]. They became more affordable for the use not only in limited number of 155 scientific research organizations, but also in the higher education institutions. Moreover, each new software update brought more research opportunities and confidence about data analysis results. Eye tracking tools are used in many application fields to find out information system users preferences and interests [23]. In educational domain researchers are eager to implement each new possibility to get know more about students’ activity and behavioural patterns within learning management system in order to improve learning process efficiency. For the teaching staff, it would be useful to observe learners’ activities within digital learning environment, make their gaze data records, analyse them, and come forward with improvements. Eye tracking technology can help education professionals and instructional designers to recognize weak chains in their digital learning objects and even in a whole Curriculum. Learners’ fatigue signs, attention to details displayed on the computer screen and connectivity with peripheral vision issues – these points play an extremely important role in improving an efficiency of the learning process, particularly, in the better perception of learning objects. This paper offers an insight into the findings of these issues and, based on gaze data, proposes learners’ engagement improving and fatigue prevention model. I. TOOLS AND METHODS 1.1 Background Eye tracking technology allows us to record person’s eye movements. In educational settings student's gaze data can provide a broad information about learning activities and learner’s preferences. Hence, these data could bring the new possibilities in user experience research giving an opportunity to consider more appropriate instructional design approaches to improve the effectiveness of the digital learning process. Eye trackers can give us an overview about learners’ cognitive processes [10]. Despite the big amount of gaze data, obtained during eye tracking tests, researchers are still faced with an interpretation issues of these data. On the one hand, we have a picture considering learners’ eye movements, on the other hand, an understanding and interpretation of the learner’s reasons for each individualised activity seems to be the challenging question [5]. Nevertheless, the new and new eye tracking systems and their extensions are being developed to find out more about learners’ cognition [7]. Usually, in standard knowledge acquisition processes the learners consciously accept provided stimuli. At the same time the question about conscious or unconscious changes in logical task solving and cognition, raised by learners’ subconscious processes, remains open [19]. It was found that short periods of unconscious brain activation processes can improve human’s cognitive abilities and task solving in comparison against their immediate reactions [8]. Conscious and unconscious human emotions are broadly studied also in neuroscience. It is discussed that unconscious thinking process might be considered as a credible event from the cognitive neuroscience perspective [42]. Instructional designers, while developing e-learning course, ought to keep in mind that masked parts of the learning content can influence student’s perception ability [22]. It might be said that teachers have to remember about learners’ para-fovea vision and use this fact to stimulate students’ involvement into e-learning activities. It could be also said that such sometimes hidden objects are able to influence learners by stimuli [4]. From the cognitive psychology point of the view, the control of consciousness decreases after exactly 40 minutes [45]. Technologies enhance education process. Moreover, video-type learning objects and teacher’s presence at video lecture could be considered as an effective education efficiency improvement instrument. Teacher’s presence in videos is emphasized in several findings [47]. Appropriate research is presented also in eye tracking studies [48]. Videos are considered as more illustrative and students engaging lessons [6], [9], [21], [26], [29], [33], [44]. An important factor which enhance e-learning is the time spent at the computer screen; it was found that short videos are more efficient for learning than long ones [18]. The voice over screen can also impact students’ attention [41]. The problem is that in some cases students can lose their interest to the activities in video lessons, and their performance remain weak [27]. In order to keep learners engaged and motivated, the suggested length of video learning objects should not extend 6 min. [15]. 156 Respectfully, further sections will try to find out and put forward possible solutions to keep students involved in learning activities as well make steps to prevent their fatigue. 1.2 Tools and methods Eye tracking technology allows us to record person’s eye movements. In educational settings student's gaze data can provide a broad information about learning activities and learner’s preferences. Hence, these data could bring the new possibilities in user experience research giving an opportunity to consider more appropriate instructional design approaches to improve the effectiveness of the digital learning process. The following stages of this research should be noted: 1) From 01.08.2018 till 31.01.2020 - investigation of the state-of-the-art considerations in the area of eye tracking and gaze analysis, as well as learning management system users’ experience recognition in order to propose a new approach to develop the technological learning and teaching model; 2) From 01.01.2020 till 31.07.2020 – studies on current achievements and models in elearners’ engagement; 3) From 01.08.2020 till 26.02.2021 – initial eye tracking system testing, observations, modelling as well justification and approbation in test-beds. For eye tracking tests and modelling sessions the GazePoint GP3 Bundle equipment [13] was chosen, including Gazepoint GP3 HD 150 Hz eye tracker, Gazepoint Analysis UX Edition software, as well as Laptop Mount. The reason for this choice was made to avoid a limited sampling frequency cases while eye tracking testing [3], as well as to draw the heat maps, record users activity in a form of screen capture, determine gaze fixation paths, export gaze data in usable csv files, add think-aloud method and analyse obtained extra information [13]. Eye tracking test organizer has a possibility to choose the necessary gaze data for the analysis by ticking a check-box from the Analysis UX Edition desktop (figure 1). Keeping in mind that eye movement fixation duration usually stands from 100 till 600 msec, and the most of the information we get from fixations, at initial stages the more useful data could be collected from the user’s fixation maps and areas of interest (AOIs). Moreover, the AOIs with the fixations there and the duration of fixations can give us useful information about learner’s cognitive efforts during knowledge acquisition in digital learning environment [14]. Besides, following previous justification, it could be noted that the chosen eye tracking set provides the optimal price / quality ratio cost-effective solution that is acceptable for the research organisations with the limited funding resources. Figure 1. CSV data selection windows in GP3 HD eye tracking studies 157 II. RESULTS AND DISCUSSION 2.1 Results and observations Based on the state-of-the-art findings of eye tracking implementations in educational domain, initial modelling of prospective eye tracking technological solution and their justification was performed. This section discusses both proposed model and some observations from the user-learner and user-teacher perspective. Taking into account suggested video learning object duration [15], to support developed model in this project, 121 eye tracking experiments were performed: the 45 tests were made for video resources with the length 20 sec-1 min, 56 tests – for the 1 min-2 min long videos, and 20 tests – for the 2 min-6 min long videos. In all the cases the trials showed dominant F-shape viewing model. To be more precise, in general user’s eye movements have followed F-pattern pathway. But this is correct till presenter’s appearance on the screen. And, since the speaker-knowledge deliver appears and starts his story, eye tracker traps the gaze data, mainly corresponding to the presenter’s visage and facial expression (figure 2). Figure 2. Heat maps highlighting the user's attention to the speaker's facial expressions Similarly, the same picture of the user’s attention could be represented in the form saccade directions and fixation durations, giving a comprehension regarding user’s appropriate interests and particular attention paid to the corresponding object (figure 3). Fixations and their duration can speak themselves about learner’s interest, engagement and cognitive processes during the event. At the moment, eye trackers the most gaze data, usable for an interpretation, obtain from fixations. However, in order to develop efficacious learning objects, we ought to think about also those objects which are close and fare behind fixations. 158 Figure 3. Fixation durations Figure 4. Presenter’s activity is immediately heeded It was also observed that static presentation may lead to the loose of learner’s attention. And contrary, presenter’s activity is immediately heeded (figure 4). This observation could be used to develop highly inclusive, engaging video learning objects that would reduce or even prevent the fatigue during learning, involving students in lesson activities, waking them up by unexpected body movements from time to time. That might appear not only within F-pattern vision area, but also throughout all computer screen. Each test, with the same learning content, have been repeated at least twice to check whether information system user’s activity and behaviour patterns have changed. It was noticed that learners’ attention to the speaker was the greatest for the first time they watched the video content. Next time (that might be interpreted also as a repetition case) the learner paid more attention to the things around the speaker, trying to catch and find out more details regarding this video lesson, e.g. bookshelves, textual information and so on. 159 Performed experiments approved initial expectations from the proposed model, discussed in the next subsection. 2.2 Proposed model Pursuant to obtained gaze data from eye tracking testing results, and taking into account provisions regarding e-learning enhancement approaches and appropriate eye tracking findings, the technological educational model could be proposed (figure 5). It envisages the promotion of students’ active involvement in the e-learning process, following the presentation of e-learning content, as well as trying to reduce learners' fatigue, at least - its signs, hereby ensuring efficient knowledge transfer. When starting learning in the e-environment, student's eye movements on the computer screen (gaze) are captured by an eye tracker. They are analysed regarding e-learning content type/length and learner’s activity patterns (figure 5). Figure 5. Proposed technological educational model based on gaze data considerations In the case when learning object lasts longer than 6 minutes, the appropriate stimuli ought to be initiated as follows: - Tutor’s or video content presenter’s appearance on the screen at least once in the 6 minutes; - Embedding of quizzes in each 6-12 min period. Such an approach could help keeping the learner in e-learning process and prevent early drop-outs. Contrary, in the cases when learning object length is less than 6 min, a random tutor’s appearance in the different screen locations and even corners could be suggested for an implementation in instructional design projects in order to force learners to follow learning content. It should be noted that since learners are stimulated not only by foveal (i.e. main focussing), but also parafoveal and peripheral vision, which provide less resolution and could be detected weaker on the computer screen by learner’s eyes [35]-[39], nevertheless, such eyes ability ought to be taken 160 into account in the proposed model. So, tutor’s appearance on the computer screen might be randomised in different screen locations. However, the first introductory lecturer’s speech / appearance ought to be made taking into account F-shape vision approach, escaping screen corners. Blink detection issue has attracted scientists’ attention for decades, emphasizing its importance on cognitive load [1], [49]. Eye blinking parameter has an influential role on students’ ability to follow knowledge flow. It may warn about learners’ fatigue and possible impacts on their health conditions. Humans blink their eyes about 14000 times daily. With each blink our eyelids are closing and the tear film covering eyes surface. If the tear film is insufficient for some reason, e.g. lack of blinks, the vision may blur. As a result, students can lose their interest in current learning activities. In this model, it is supposed to support learners who did not make at least one blink per minute, that could be counted by modern eye tracking systems by option “the number of blinks in the previous 60 second period of time (BKPMIN)”. It might be recommended to generate the pop-ups appearing on the screen after detection of no blinks from the learner’s side. At the same time, we have to keep in mind also eye blink duration [12], [25]. Blinks which last longer than 400 ms [30], [43] might be considered as the learner’s fatigue condition. Flashing lights and animation fragments, as well as sound signals, could be considered as the stimuli, a kind of alarm tool. It is expected that all the gathered gaze data would be processed and analysed, thus providing both teacher and instructional designer the useful information about necessary improvements to be made in particular learning object. This artificial intelligence component of the model could be considered for the development in further studies. 2.3 Discussions It could be useful to take into account learners’ colour preferences which they expect more likely seeing on the screen. It was found that users’ preferences differ by age and gender [16]. Another one useful instrument already embedded in eye tracking analyse software is implementation of the “Think-aloud” method that could provide a powerful feedback and render necessary suggestions, an understanding about the reasons why the learner paid attention to the particular object. Regarding gaze data analysis, the whole learning course or its separate modules might be considered as a chain of the consequent scaffolding and forward-and-back moved learning activities (i.e. a1, a2, …, an). They all occur in particular corresponding time (i.e. t1, t2, …, tn). Thus, we can talk about the connection of these activities from the data obtained in eye tracking studies and learning analytics, assessing the frequency and time of learners' mouse clicks on specific learning objects. Eye tracking technology gives us an extra opportunity to observe and see in details learners’ AOIs, engaging things on the display. The real situation regarding learners’ involvement into e-learning process, their attention, distraction, and fatigue, remains unclear till now. In digital learning environments e-learning objects (LO) differ to each other by size, type and other parameters. As smaller is the LO, the shorter is learner’s interaction time with it. Hence, the detection of real human-computer interaction (HCI) activity and its traits seems problematics. Technically this issue is still unsolved. Moreover, it ought to be noted that students’ distraction, losing attention and fatigue may appear independently of their discipline and diligence [24]. It could be concluded that a smaller LO would result to the smaller HCI, i.e. human - learning object – interaction (H-LO-I) detection errors and vice-versa (figure 6) [46]. Figure 6. H-LO-I time detection error [46] 161 An appropriate involvement of eye tracking technology into existing learning analytic approaches, exploiting user’s log-files, mouse clicks and other statistical data, could be considered as an important consistent component in further UX studies. In order to find out learner’s engagement, motivation to learn and possible fatigue signs, an evaluation of student’s blink parameters may count. They could be collected and processed by options “the duration of preceding blinks in seconds (BKDUR)” and “the number of blinks in the previous 60 second period of time (BKPMIN)”. Moreover, these metrics could play a crucial role in prevention of dryness and redness of the eyes, which can lead to unwanted health problems. As the one of further research directions might be the development of the built-in learning management system interactive plugin that would give the learner signals in the form of bright flashes as a call to blink. However, it might be argued that in such a case the students would need to work/learn on computers equipped with eye trackers. In order to propose more sophisticated instructional design and e-learning objects acquisition ways which would take into account also learners’ stress and fatigue level, it could be useful to consider an implementation of a complex learner usability studies involving not only eye tracking, but also electroencephalography (EEG) and other biosensor systems [11]. Such a combination would improve our understanding about students’ cognitive processes while learning, their emotions, behavior, attentions and fatigue. EEG is a quite effective tool in detection of persons’ cognitive and logical ability, as well as a fatigue level. In Latvia so far an interesting results and useful suggestions in this area are made by Psychoneurophysiology & bioregulation investigation center (PBIC). It has been found that some parameters could signalize about learners’ possible fatigue state starts – they are the deviations in user behavior which appear in a form of different electrical activity in the brain. EEG could record these disturbances as an increase in the overall amount of slow waves and a decrease in the amplitude of all basal rhythms. These fatigue signs could appear more significantly in cognition processes. Figure 7 shows the standard brain electrical activity in age group from 30 till 35 years, and Figure 8 - recorded fatigue signs [31]. Figure 7. Normal brain electrical activity in age group from 30 till 35 years [31] 162 Figure 8. Significant increase in the amount of slow waves - one of the fatigue signs [31] It could be proposed a complex implementation of both eye tracking tools and other bio-signal sensor technology. Such a combination could open the new opportunities to understand learner’s activity and behavior patterns, possible fatigue warnings and provide effective fatigue prevention solution. It might be proposed that the synchronized and justified use of several biosensor systems in the complex research projects could provide additional efficiency to the Eye Movement Modelling Example (EMME) model, strengthening the role of an expert in a specific field for the orientation of students-novices to the relevant subject and skillful guidance through the whole learning process and separate objects. Although research of an implementation of analytical modules enriched with artificial intelligence could be considered in further projects, it could be noticed that in order to improve elearning course efficiency, various learning analytics tools and approaches ought to be involved, including development of several linked educational eco-system modules, in particular, learners, tutors, bio-sensors, analytics, personalization, web gateway and other ones. Learning course assessment ought to be taken into account as well: both from students and instructors. Figure 9 shows possible learners activity data gathering pathway which might be analyzed by neural networks, where: DQ – data obtained from learners’ surveys, DT – data from teachers’ observations, DL – data requested from database about system users log-files,  - data set,  - outcomes data (after analysis). Figure 9. Learning data gathering pathway III. CONCLUSIONS Tutor’s appearance within video learning objects could be considered as an important factor that might attract students’ attention and keep them engaged. The tutor ought to think about dynamic presentations that would be more preferable in comparison to the static ones. In the case of repeated viewing of e-video materials, the student's eyes are concentrated to the places next to the speaker. Respectively, instructional designers ought to take care of these areas of interest on the computer display, and be aware of the dominant F-shape gaze approach. To prevent learners’ fatigue, it would be desirable to change the picture of the speaker's speech on the presentation slides from time to time, repeating the floating or pop-upping of the 163