DESIGN AND IMPLEMENTATION OF EMOAI SMART CLASSROOM: A STUDENT EMOTIONAL AND BEHAVIORAL ENGAGEMENT RECOGNITION SYSTEM

DESIGN AND IMPLEMENTATION OF EMOAI SMART CLASSROOM: A STUDENT EMOTIONAL AND BEHAVIORAL ENGAGEMENT RECOGNITION SYSTEM ABDULSALAM FOLASEWA MARYAM CSC/2016/001 A FINAL YEAR PROJECT SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, FACULTY OF TECHNOLOGY, OBAFEMI AWOLOWO UNIVERSITY, ILE-IFE. IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE AWARD OF BACHELOR OF SCIENCE (B.Sc.) DEGREE IN COMPUTER SCIENCE WITH MATHEMATICS AUGUST, 2023 i CERTIFICATION The undersigned certify that they have read, approved, and hereby recommend to the Faculty of Technology, a research project titled “Design and Implementation of EmoAI Smart Classroom: A Student Emotional and Behavioral Engagement Recognition System” originally written by Folasewa Maryam Abdulsalam in partial fulfilment of the requirements for the award of a Bachelor of Science Degree (B. Sc) in Computer Science with Mathematics. ____________________ ________________ Dr. A.O Afolabi Date (Project Supervisor) ____________________ ________________ Prof. A.O Oluwatope Date (Head of Department) ii DEDICATION This work is dedicated to you, Folasewa, your parents and siblings; and most importantly to your creator. iii ACKNOWLEDGEMENT Grateful to the Almighty for bringing me thus far; throughout my studentship, I had zero visits to the hospital, no cause to miss lectures or exams. My profound gratitude goes to my supervisor Dr. Afolabi. I would not have asked for a better supervisor. The push needed for me to see this to the end came from you, thank you! My gratitude will be incomplete without mentioning my parents and siblings; Mr. and Mrs. Salami, Bolarinwa Salami, and Zainab Salami. You all gave me the reason to see this to the end. To my reading partners, Jane, Precious, Ope, Promise; to my codes debugger, Bolu; to David who gave me his laptop to work; and Oluwaseun for the support; I say thank you very much for contributing to the success of this project. See you all at the top. iv TABLE OF CONTENT Title Page .................................................................................................................................... Certification................................................................................................................................i Dedication .................................................................................................................................ii Acknowledgment.......................................................................................................................iii Table of Content.........................................................................................................................iv List of Tables .............................................................................................................................ix List of Figures………………………………………………………………………………….x Abstract ....................................................................................................................................xii CHAPTER ONE: INTRODUCTION 1.1 Background to the Study..................................................................................................1 1.2 Statement of Problem........................................................................................................6 1.3 Scope and limitation of Project ……..............................................................................6 1.4 Justification....................................................................................................................... 7 1.5 Aim and Objectives............................................................................................................7 1.6 Methodology ……..............................................................................................................8 1.7 Expected Contribution to Knowledge…………………………………………………….8 1.8 Application to Educational Support ..................................................................................8 1.9 Organization of Report.....................................................................................................9 v CHAPTER TWO: LITERATURE REVIEW 2.1 Introduction.........................................................................................................................10 2.2 Conceptual Review..............................................................................................................10 2.2.1 Smart Classroom………......................................................................................................11 2.2.2 Learning Analytics in a Smart Classroom......................................................................12 2.3 Affective Computing………………………...........................................................................14 2.4 Deep Learning …………………………………………………….........................................14 2.4.1 Deep Learning neural networks……………………………………………………15 2.4.2 Deep Learning use cases…………………………………………………………..16. 2.4.3 Deep Learning Challenges…………………………………………………………16 2.5 Computer Vision……………………………………………………………………………..18 2.5.1 Uses of Computer Vision in Education …………………………………………...18 2.5.2 Computer Vision in Education and Privacy………………………………………..19 2.6 Facial Emotion Recognition System………………………………….…...............................20 2.6.1 Face Recognition…………………………………………………………………..20 2.6.2 Head Pose Estimation……………………………………………………………...21 2.6.3 Eye Gaze Measurement……………………………………………………………22 2.6.4 Hand-Over-Face Gestures in Learning and Teaching……………………………..23 vi 2.7 Student Engagement Systems…………………………………………………………..24 2.7.1 Engagement Estimation Methods…………………………………………….25 2.7.1.1 Manual Method…………………………………………………….26. 2.7.1.2 Semi-Automatic Methods…………………………………………..26 2.7.1.3 Automatic Methods………………………………………………….27 2.8 Related Works……………………………………………………………………………28 CHAPTER THREE: METHODOLOGY 3.0 Introduction.......................................................................................................................38 3.1 Software Development Life Cycle (SDLC).......................................................................38 3.1.1 Requirement Gathering and Analysis Phase………………………………….. 40 3.1.2 System Design………………………………………………………………….41 3.1.2.1 Unified Modeling Language (UML)………………………………….42 3.1.2.2 System Framework…………………………………………………….42. 3.1.2.3 Use Case Diagram……………………………………………………...48 3.1.2.4 Activity Diagram……………………………………………………….48 3.1.2.5 Class Diagram………………………………………………………….52 3.1.2.6 Sequence Diagram……………………………………………………..55 vii CHAPTER FOUR: SYSTEM IMPLEMENTATION AND EVALUATION 4.1 Introduction.........................................................................................................................57 4.2 Implementation at Model Level............................................................. ………………….57 4.2.1 The DAiSEE Dataset.............................................................................................58 4.2.2 Data Collection………………...............................................................................58 4.2.3 Frame Extraction………….....................................................................................59 4.2.4 Data Annotation…………………………………………………………………..59 4.2.5 Model Building…………………………………………………………………….60 4.3 Tools Used………………………………………………………………………………….64 4.4 Implementation at Design Level……………………………………………………………66 4.4.1 Functional Requirements……………………………………………………….66 4.4.2 User Interface Prototype……………………………………………………….66 4.5 Workflow of the Integration of the Model and User Interface…………………………….73 4.6 Tools Used…………………………………………………………………………………77 4.7 Evaluation…........................................................................................................................78 4.7.1 Alpha Testing………................................................................................................78. 4.7.2 Objectives of Alpha Testing......................................................................................78 4.7.3 Parameters used in Evaluation…………………………………………………….79 CHAPTER FIVE: SUMMARY, CHALLENGES, AND CONCLUSION 5.1 Introduction.......................................................................................................................81 viii 5.2 Conclusion........................................................................................................................81. 5.3 Challenges Encountered….................................................................................................82 5.4 Recommendations...............................................................................................................82 REFERENCES........................................................................................................................83 APPENDIX A .........................................................................................................................92 3 et al., 2019). The basic-emotion approach, however, continues to operate under the presumption that there exists a basic facial configuration that serves as the prototype and can be used to identify an individual's emotional state in a manner similar to how a fingerprint can be used to uniquely identify a person. This hereby supports the argument that one of the clues for determining a person's emotional state is their facial expression. Numerous research fields, including computer science (Michel and Kaliouby, 2005), neuroscience (Liu et al., 2012), psychology (Ekman and Oster, 1979), and medicine, have extensively studied emotion recognition in both theory and applications. It is a crucial step in the process of a machine understanding human behavior. The capacity to identify and analyze another person's feelings is known as emotion recognition. It is a technique that examines emotions in various media, including images and videos. It is a member of the group of technologies known as "affective computing," a multidisciplinary area of study on how well computers can understand and detect affective states and human emotions. Affective computing frequently builds on Artificial Intelligence technology. Since a few years ago, there has been a sharp increase in interest in creating technology that can identify people's affective states (Fragopanagos & Taylor, 2005), with the idea of deducing affect from body language receiving particular attention. The numerous non-digital applications for security, law enforcement, gaming and entertainment, education, health care, and security show the relevance of body expressions and the advantages of creating apps into which influence detection from body expressions may be integrated. The application field of interest is the education sector in monitoring students’ attention and detecting engagement in a learning environment using facial and body cues. 4 The stress of labor-intensive classroom tasks like attendance tracking, gathering lecture feedback, student involvement, or attention monitoring is decreased with the introduction of intelligent technologies in big offline classroom management, reinforcing the best teaching-learning outcomes (Kim et al., 2018). The quality of the overall classroom learning experience and academic success are improved when students are actively participating in the learning process (De Villiers & Werner, 2016). The issue of student disengagement is currently getting worse every day due to a variety of factors, including poor teaching methods, a limited attention span, and a lack of student-teacher interactions (Bradbury, 2016). The presence of big offline classrooms (student count is greater or equal to 60) exacerbates this issue In order to improve the educational system, student engagement—which occurs when a student participates meaningfully in the learning environment—should be carefully considered (Sharma et al., 2019). According to the definition of student engagement by (Lamborn et al.,2014), it is the psychological commitment a student makes to learning and comprehending the concepts or abilities that their academic work aims to promote. Achievement of students is directly correlated with engagement (Furrer et al., 2014). Students' participation is described using multiple aspects and components in the literature on educational research. It was characterized by (Fredricks et al. 2004) in terms of behavioral, emotional, and cognitive engagements. Behaviors like maintaining good posture and taking notes are examples of behaviors that are described as being engaged in the learning process. Positive and negative emotional responses to learning, such as focus, boredom, and irritation, are referred to as emotional involvement. Learning to improve cognitive abilities, such as knowledge, problemsolving, and creative thinking, comes via cognitive engagement. It is possible to identify a student's 5 level of involvement during a lecture since behavioral and emotional engagements are inversely correlated (Li & Lerner 2013) and behavioral engagement influences cognitive engagement, a crucial result of the learning process. Students' feelings during class participation are crucial in any learning environment (Krithika et al., 2016). Emotions can have an impact on how pupils learn (Turner et al., 2002). It may have an impact on a student's concentration, drive to learn, coping mechanisms, and self-control (Skinner, et al., 2014). It may also provide crucial cues regarding how well they are studying in class. There is some evidence that some emotions, such as boredom (Csikszentmihalyi, 1990), perplexity (Graesser & Olde, 2003; Klein, & Picard, 2002), flow (also known as engagement, Csikszentmihalyi, 1990), and frustration, might affect learning and cognition (Klein, & Picard, 2002). Also thought to go along with learning are curiosity and eureka moments, or "a-ha" moments. The dynamics of a lecture can be better understood and improved upon by knowing and enhancing the students' attention span, which is a fundamental emotion shown in the classroom by the students. In-class attention spans among students range from 46% to 67%, according to studies (Raca et al., 2015). As a result, it can be assumed that up to 50% of the students are not capable of learning effectively. Determining the possible causes of this scenario as well as the circumstances in which some pupils lose focus more frequently than others is vital. With this knowledge in hand, teachers can look for potential issues during lessons and strive to fix them, which could improve the students' learning effectiveness (Canedo & Neves, 2020). Different emotions are evoked in a classroom setting, which might influence whether learning occurs or not. One of the special affective states that cannot be characterized by the fundamental emotions is the academic affective state (Wei et al., 2017). 6 With the tremendous advancement in computer vision techniques, significant academic affective states that are relevant to a learning environment (Tonguc & Ozkara, 2020) will be reviewed and implemented in this study using a non-invasive approach to measure in real time student emotional and behavioral engagement in the classroom. These states include "boredom," "confusion," "focus," "frustration," and "sleepy." 1.2 Statement of Problem The issue of student disengagement has become critical in the current educational system as a result of numerous distractions and lack of interactions between the students and the teachers. The frequent overcrowding in offline classrooms makes it emotionally draining for teachers to monitor students' moods thoroughly, maintain the correct degree of interaction and obtain rapid learning feedback from all students while learning is occurring (Chen, et al.,2019). To better understand and improve the dynamics of lectures delivered, it is necessary to develop an engagement system that takes into account both academic-affective states and the body language displayed by students in relation to engagement or disengagement. 1.3 Scope and limitation of the Project The system developed is context-specific for inferring affect through visible unintended gestures made by students in tutoring situations. It is limited to detecting hand-over-gestures and four academic affective states. Case study is OGD Tutors, a tutorial center situated at OAU Central market, Ile-Ife, Osun State. The video and image data needed for evaluation were collected from the students. 1.4 Justification This study explores the relationship between student emotions and body language and engagement and attention span, as well as if these cues can help lecturers enhance the dynamics of the lecture 7 in the classroom. The real-time student behavioral and emotional engagement recognition system required for effective learning in large classrooms is one that can provide a thorough analysis of the students' emotions while a lecture is going on without bias and without putting an emotional strain on the instructor giving the lecture. The development of this system will become an important tool for educators and schools. Teachers will be able to identify students who may be struggling with learning in the classroom, and adjust their teaching methods accordingly; schools will make informed decisions about the curriculum, teacher training and student support services. This ultimately provides a system with an improved quality of education for students, promote academic success, and support the development of well-rounded individuals. 1.5 Aim and Objectives The aim of this thesis is to develop a student emotional and behavioral engagement recognition system. The Specific objectives are to: i. collect video data ii. identify student emotions and behaviors iii. understand engagement levels of the students and iv. evaluate the system 1.6 Methodology i. Video data (DAiSEE) was collected from Paper with Code ii. Frames were extracted from the data and annotated using LabelAssist on RoboFlow iii. The extracted frames were trained on YOLO V8 pre-trained model iv. The model was evaluated through testing of pre-recorded classroom videos. 8 1.7 Expected Contribution to Knowledge This study will contribute to the existing body of knowledge by i. providing a system that caters for both body gestures and emotions related to learning in the classroom; ii. providing a system in which tutors can evaluate their teaching methodologies through the result of engagement analysis. 1.8 Application to Educational Support It is challenging to observe all students' mental states in time to determine their level of learning for both offline and online instruction. We conducted a thorough investigation of automatic intelligent monitoring technologies in the classroom to help tutors and lecturers better understand the learning level of each student. We recommend that such technologies be put into place in combination with human teachers since they will help them mentally connect with the students' learning status and provide insight into the positive and negative emotions displayed throughout a lecture. This would enable educators to focus more of their attention on pupils who consistently display unfavorable learning emotions, which would improve the quality of the learning materials. The dynamics of the lectures will be much improved, the teacher's workload will be lessened because he has a system serving as his third eye to connect with the students on a similar mental level, and lectures will be presented better taking into account the emotions of the students. 1.9 Organization of Report This project is sectioned into five (5) chapters. Chapter one gives the background of the project, its scope, and objectives. It also states the problems of the study, methodology, and justification. Chapter two presents a review of existing literature on student classroom engagement systems and facial emotion detection systems. 9 Chapter three elaborates the steps, procedures, and methodologies employed in the design, development, and implementation of student classroom engagement systems and facial emotion detection systems. It gives comprehensive information about the tools used to implement the project. Chapter four, discusses the UI design and the implementation of the project which include the logical and physical design; the system is also evaluated for efficiency. The efficiency is determined by how much the system is able to meet the requirements. The flaws noticed are documented so that they can be modified and updated for the improvement of the future version of the system. Chapter five is the concluding chapter which contains the limitations and challenges encountered, a summary of the work done, and recommendations to be used for future project analysis and conclusion. 10 CHAPTER TWO LITERATURE REVIEW 2.1 Introduction Literature review discusses the research and analysis of existing systems and the implementation of possible techniques and methods to use in the project. This is done to determine the best possible techniques and methods so that the feasibility of the project can be determined. In this project, related topics are: smart classroom, emotion recognition, hand-over-face recognition system, real time systems, facial recognition, body pose estimation, student engagement system implementation in the education field. Since these topics are closely related to the project, the researcher is expected to make references in implementation of a real time student emotional and behavioral engagement system. 2.2 Conceptual Review A person’s ability to learn cognitive procedures have been encouraged and accepted by many researchers. Some research also found the need to develop learning systems that can detect the emotional states of learners; hence finding indexes to determine the engagement and concentration levels based on their emotional states has been gaining ground in recent years. Robotic tutors, e-learning, and Intelligent Tutoring Systems (ITSs), which would offer individualized education across a variety of subjects, have recently attracted a lot of attention (Gordon et al., 2016; Andallaza et al., 2012; Woolf, 2010; D'Mello et al., 2005). These systems typically infer affective states merely from facial expressions and can personalize to some level, but they lack the requisite empathic skills, that is, the capacity to fully understand learners' emotions, moods, and temperaments. The following sub-section will give a comprehensive 11 overview of areas related to creating a smart classroom using an automatic student emotional and behavioral system. 2.2.1 Smart Classroom The study and development of novel technologies and sciences for replicating, extending, and enhancing human intelligence is known as artificial intelligence (Zhang and Chen, 2021). Artificial intelligence technology has advanced quickly along with the growth of mobile information, altering every aspect of our lives. It is possible to construct intelligent and efficient classroom instruction and improve students' ideological development by deeply integrating and innovating artificial intelligence technology into classroom education and teaching. The term 'smart classroom' means an intelligent classroom based on artificial intelligence technologies by integrating in an unobtrusive manner the different components in a traditional classroom to improve learning process. In this regard, smart classrooms play a significant role in the transformation of conventional education into modern education (Hicham, and Abdelaziz, 2017), which presents prospects for enhancing educational quality and academic ability, access to quality education, and fairness. The various parts of a smart classroom produce a lot of activity-related data, which provides the opportunity to extract insightful knowledge that might be applied to enhance the learning process and help the component parts' educational decision-making (Aguilar, et al, 2017). However, in a smart classroom, it is extremely difficult to share and manipulate this kind of information in real time. This is the area where learning analytics activities in a smart classroom are applied to improve its behavior. 12 2.2.2 Learning Analytics in a Smart Classroom Understanding and identifying the various learning capacities of the students is crucial in a smart classroom so that teachers may provide the students the direction they need to advance their skills. Sharing and modifying insightful knowledge created in the classroom in real time is the area of application of learning analytics activities in a smart classroom (Aguilar et al., 2017). In a smart classroom, Learning Analytics often aims to achieve two distinct objectives (Valdiviezo et al., 2015) 1. To comprehend the student learning: Learning Analytics will need to produce indicators about it (framework, methodologies, tools, etc.); 2. To comprehend the behavior of the students: in this scenario, the learning analytics tasks must produce metrics indicating the effectiveness of each student. For the proposed EmoAI smart classroom, we propose a set of Learning Analytics tasks such as: 1. Recognition of the students in the smart classroom: this task involves face recognition of the students by matching up each student's face with their name in a database. 2. Monitoring the learning process of students in the smart classroom: this task involves the identification of lecture periods at which the maximum and minimum learning took place through their behavioral engagement in the classroom Some of the technologies to be employed include facial recognition systems that use cameras to capture student responses, algorithms to identify their attention levels, and measuring smiles, frowns and audio to classify student engagement, “each using a combination of psychology and data-mining to detect micro expressions and classify human reactions.” 19 The following is a list of "affective computing" advancements being used in education: 1) Transdermal Optical Imaging: which uses a camera to detect face blood flow information and determine student moods in situations where visual signals are not readily available. 2) Electroencephalogram (EEG) electrical brain activity tests are used to assess pupils' emotional arousal, task performance, and to give individuals with computer mediation. 3) Wearable affective technology: such as a social-emotional intelligence prosthetic, that detects human affects in children in real time by using a small camera and analyzing facial expressions and head movements to infer the child's cognitive-affective state. 4) A glove-like gadget that tracks students' physiological arousal and monitors skin conductivity to determine how stimulated they are. 5) Emotionally intelligent computing systems that can analyze emotion and respond with appropriate expressions, allowing educators to give highly tailored content that inspires youngsters. All of these advancements are part of the ongoing effort to turn "real-time mood monitoring" and "emotional feedback" gadgets into critical technology for understanding, measuring, and managing human emotions in modern society. 2.5.2 Computer Vision in Education and Privacy The use of computer vision in education has been disputed and problematic. The morality of keeping children under continual observation has long been debated. Additionally, it's possible that the technology itself isn't yet prepared for general use. Computer vision facial recognition techniques are currently highly sensitive to errors, with women and people of color reporting 20 higher error rates. In the case of online courses, there is a great deal of debate around the volume of information and duration for which the website can track the user's movements. Face recognition data is obviously at considerable risk of being sold or compromised by a third party, as well as being utilized for other illegal activities. to name just a few. Fig 2.2 – Uses of Computer Vision Analysis in Education (S Peng et al., 2021) 2.6 Facial Emotion Recognition System Due to its focus on the core components of recognizing facial emotion expressions, FER is also known as facial emotion recognition. It involves recognizing expressions that convey fundamental emotions like fear, happiness, contempt, and so forth (Barlett, et al., 2003). This field has grown significantly as a result of the quick advancement of artificial intelligence techniques in fields like human-computer interaction (Dornaika et al., 2007); virtual reality (VR); Hickson et al., 2017; augment reality (AR); Assari et al., 2011; and others; because of its tremendous academic and 21 commercial potential, facial emotion recognition (FER) is an essential topic in the domains of computer vision and artificial intelligence. 2.6.1 Face Recognition Face recognition technology is a significant research endeavor in the fields of pattern identification and computer vision because it has the potential to detect identities and other information based on the visual characteristics of a facial image. (Zhao & Mei, 2017). Face traits have the advantages of being good, straightforward, and practical when compared to other biological features; as a result, face recognition is more agreeable to users. The data capturing module, the feature extraction module, the classification module, and the training classifier database module are the four fundamental parts of a face recognition system. 2.6.2 Head Pose Estimation Head pose estimation is the process of assisting biometric systems that make use of any of the biometric features of the head, including the face, ear, or iris (HPE). (Andrea et al., 2022). It is regarded as a preprocessing step to choose the best frame for face recognition in a video, as well as a behavioral feature to assess the subject's intent and a description to help frontalize the face. (Andrea et al, 2022). By observing the head's rotational angles, it accomplishes this. This specific biometric branch is used in areas (fig 2.3) such as optimal frame selection, facial frontalization, driver concentration detection, and surveillance for identification Finding the head posture orientation is the purpose of HPE (yaw, pitch, roll). The main steps of an HPE algorithm is summarized in the figure below (fig 2.4). 22 Fig 2.3 – Areas of application of HPE (Andrea et al., 2022) Fig 2.4HPE Framework (Andea et al., 2022) 2.6.3 Eye Gaze Measurement Users' gazes and regions of interest from eye trackers have been utilized to understand the moods of learners while engaging in any educational activity in online learning. To ascertain where the users were looking, (Aslan et al.,2014) used statistical facial traits, depth data, and eye tracker data. Nine pilot sessions of the decision trees, random forest, naive bayes, logistic regression, and multilayer perceptron machine learning algorithms were employed for testing. Despite their efficiency, the fundamental challenge with eye-tracking-based approaches is precise ocular calibration. For these techniques to give accurate data, each participant must undergo many 23 calibration rounds. Studies must typically omit participants who wear glasses or have eye disorders due to calibration concerns (Raina et al. 2016). Limiting participant mobility to stay within the eye-tracker range is a substantial challenge as well, although doing so is impractical in a real-world educational context. 2.6.4 Hand-Over-Face Gestures in Learning and Teaching The majority of research on learning and teaching relies on the use of facial expressions as the primary method of emotional communication, despite the fact that (Gelder 2006) claims there are similarities between how the brain reacts to emotional body languages and how expressions are identified. The hand over face gesture was studied by (Whitehill et al., 2014) as one of the faces of interaction. The hand-over-face description, hand motion on the face, and the area occluded by the hand over the face, according to (Mahmoud et al., 2016) are key nonverbal communication channels that significantly reveal a person's cognitive state. Hands over Face gestures (fig 2.5) can amplify the affective cues conveyed by facial expressions, rather than being redundant information. In everyday interactions and conversations, we routinely use our hands to convey nonverbal messages (Rautaray and Agrawal 2015), ranging from simple movements like pointing to objects to more complicated ones like expressing emotions. We tend to put our hands closer to our faces, which are often partially veiled and can be used as a key to determine our emotional states, more often. (Tofighi et al.,2016) also identified disengagement, attention, intention, and actions using hand gestures (DAIA). Different levels of hand speed, raising the hands above the waist, and other hand movements can all be detected using the binary classifiers in DAIA. These classifiers identify the action-taking intent of the user. Finally, by examining the classifiers' judgments, a Finite State 24 Transducer (FST) of engagement detection was employed to flow between various emotional states. Fig 2.5Common Hand-over-face gestures (Pease and Pease (2006)) 2.7 Student Engagement Systems Many different facets of learners' engagements have been examined in the literature (Bosch 2016; Fredrick et al. 2004; Anderson et al. 2004). (Bosch 2016) divides engagement into three categories: emotive, behavioral, and cognitive. In contrast to (Fredrick et al.,2004) who defined engagement as being behavioral, cognitive, and emotional, (Anderson et al.,2004) described it as being academic, behavioral, cognitive, and psychological. Affective engagement refers to the emotional attitude, such as being interested in a topic and enjoying learning about it, whereas academic engagement refers to academic identification (such as getting along with teachers) and participation (such as spending time on tasks, not skipping classes) towards learning (Bosch et al., 2016). The idea of behavioral engagement is built on involvement, which involves engaging in extracurricular and classroom activities, staying focused, completing assignments on time, and paying attention to directions from the instructor (Christenson et al., 2012). Cognitive engagement is the thought and willingness to put up the work needed to understand difficult topics and develop difficult abilities, such as heightened focus, memory, and creative thinking (Anderson et al., 2004). Emotional engagement includes both 25 favorable and unfavorable reactions to teachers, students, and academic success (Fredrick et al., 2004). Examples of psychological engagement include connections with teachers, peers, and a sense of belonging. 2.7.1 Engagement Estimation Methods We categorize the present systems into three broad groups: automatic, semi-automatic, and manual, taking into account the methods' reliance on student engagement. The approaches in each category are then subdivided (fig. 2.6) based on the types of data they employ, such as audio, video, learner log data, etc. Computer vision based methods in the automatic category are specifically looked at because they show promise in an online learning setting, are unobtrusive, and are cost-effective when taking into account the hardware and software needed for gathering and analyzing video data. The manual methods now include self-reporting and observational check-list categories. The methods utilized in engagement tracing are semi-automatic. The automatic approaches in this area are further classified into computer vision-based methods, methods that analyze sensor data, and methods that analyze log files depending on the data that these methods process for engagement detection. The computer vision-based methods are further separated into three sub-categories: eye movement, gestures and postures, and the modalities they use for engagement detection. The aforementioned modalities are used singly in some research projects, while others think that combining two or more of them to boost accuracy has potential. 26 2.7.1.1 Manual Method The manual category covers engagement detection methods that call for active participation from learners. In the manual category, self-reporting is a popular strategy where learners use a series of questions to self-report their degree of focus, distraction, enthusiasm, or boredom. Self-reporting is of significant interest to many academics because it is straightforward to administer and provides some useful information regarding learner involvement. For instance, it is useful to know that between 25% and 60% of students’ report being bored or disinterested (Shernoff et al., 2000). However, a variety of factors outside the researchers' control, such as the students' candor, willingness to disclose their emotions, and the correctness of their evaluation of their emotions, will determine whether the results from the self-reporting are genuine (D'Mello et al., 2014). 2.7.1.2 Semi-Automatic Methods In the semi-automatic category, learners must indirectly participate in the engagement detection step (Dewan et al., 2019). Semi-automatic techniques of gauging engagement also include physiologically based methods and knowledge tracing. Knowledge tracing places a heavy duty on the tutor since they must gauge the level of student engagement by looking at how they respond to the questions that are posed to them. In order to interpret physiological signals like brain signals, heart signals, etc., wearable devices like electro dermal activity sensors are used in the aspect of measuring engagement using physiological approaches (Di Lascio et al., 2018). However, this approach is pricy and intrusive to the wearer. 27 2.7.1.3 Automatic Methods The automatic category of techniques extracts characteristics from a range of traits captured by image sensors (such as gaze, facial gestures, gestures, and postures), biological and neurological sensors (such as heart rate, EEG, blood pressure, or galvanic skin response), or by monitoring students' activities in their learning environments (Dewan et al., 2019). (such as total time spent studying, number of forum posts, average length of time to solve a problem, number of submissions correct, etc.). These methods automatically extract features without interfering with the way that students are motivated to learn. Computer vision-based methods (D'Mello et al. 2009; D'Mello and Graesser 2010; Kapoor and Picard 2007) offer a variety of ways to measure learners' involvement by looking at cues from movements and postures, eye movement, and facial expressions. Computer vision-based systems' primary advantages are their simplicity and absence of interference with the learning process, which is comparable to how a teacher can observe a student's motivation in class without interfering with his or her actions. Because they are valuable in large offline classrooms, this study will analyze computer-based automated methods for evaluating student engagement. 28 Fig 2.6 – Taxonomy of Learner’s engagement detection methods (Dewan et al., 2019) 2.8 Related Works The ability of students to direct their own learning inside and outside of the classroom has long been a heated topic for discussion among educational scholars, legislators, and practitioners. The "one-size-fits-all" attitude of industrial society serves as the foundation for the contemporary educational systems (Watson et al. 2015). Numerous studies have been conducted over the years to improve the effectiveness of the learning environment in the classroom by simulating students' emotions and behaviors. In lieu of exploring the combination of hand-over-gestures with facial expressions, (Gunavathi & Siddappa, 2018) developed a multimodal system for cognitive state recognition. They achieved this by splitting the methodology to three independent tasks which include coding and classifying hand-over-face descriptors, detection of basic facial expressions using facial landmarks and finally 35 hand-raising, hands-onface Ardhendu et al., (2020) They explored the variety of nonverbal behaviors (emotions, head movements, head pose, eye gaze, hand-over-face (HoF) gestures) using IntraFace tool They recognized students’ affective states in real-time using state-of the-art deep models trained on ImageNet dataset High dependence on a tool as the main framework for the facial recognition module. Peng et al., (2021) They trained a set of machine learning algorithms such as support vector machine, random forest and multilayer perceptron using features from facial, heart rate and auditory modalities they monitored students’ mental states (boredom, frustration, concentration, and confusion) during classroom discussions the implementation of this study is expensive because it needs physical devices. Zheng et al., (2020) They trained an improved Faster R-CNN model on the created classroom students’ behavioral dataset consisting of hand They identified students’ engagement by analyzing behaviors detected from the students This study is constrained to only students’ behaviors with no room for academic-based 36 raising, standing and sleeping behaviors emotions for estimating students’ engagement Chen, et al (2019) They combined three methodologies (Viola Jones, Gabor Wavelet, and Multi-Layer Perceptron) to dynamically recognize the students’ basic emotions in real time. They introduced a multicamera-based emotion detection system in a classroom environment to detect and record changes in students’ facial expression and report to the teacher in real time. The study only focused on the basic emotions and not the learningspecific emotions in the classroom. Zeng et al., 2019 A front-end web-based visualization system was implemented to support the emotion analysis using Vue and Flask frameworks. they designed an interactive visualization system called EmotionCues that supports automatic emotion extraction and multi-level visual summarization from recorded classroom videos. Choice of visualization style for emotion trend analysis was tacky. Klein and Celik (2017) Transfer learning was performed on Alexnet using synthesized dataset The study categorized attention levels into interested or not interested. In estimation student engagement, academic affective states were not considered 37 of popular classroom actions and postures. Zaletelj and Kosir (2017) Extracted 2D and 3D features were used to train decision tree and KNN algorithms Attention levels were classified into high, medium and low. The proposed system cannot work in a large offline classroom due to the fact that the Kinect sensor has a short range of imaging system. 38 CHAPTER THREE METHODOLOGY 3.0 Introduction This chapter dwells more on describing the process, methods and tools that will be used to achieve the objectives of the study as earlier stated. A software process model is an abstraction of the software development process. It can be seen as a representation of the order of activities of the process and the sequence in which they are performed. The goal is to provide guidance for controlling and coordinating the tasks to achieve the end product and objectives as effectively as possible. In this chapter, the concept of software process model and Software Development Life Cycle will be discussed; its methods and phases inclusive, existing and proposed frameworks will also be discussed and finally Unified Modeling Language which is a language for describing the model of a system along with the use case and static model will also be discussed. 3.1 Software Development Life Cycle (SDLC) A process that defines the various stages involved in the development of software for delivering high-quality product is referred to as Software Development Life Cycle. It covers the entire lifecycle of a software, that is from inception to retirement of the product. The framework or diagrammatic representation of the SDLC is shown in fig 3.1. SDLC provides a well-structured flow of phases that help an organization to quickly produce highquality software which is well-tested and ready for production use. SDLC being a repetitive methodology, one has to ensure code quality at every cycle to remove typical pitfalls of software 39 development project. It works by lowering the cost of software development while simultaneously improving quality and shortening production time. The process model used for the research work is a prescriptive software model in which activities and tasks occur sequentially with defined guidelines for progress. The model allows a choice of framework activities that is necessary to achieve the complexity of any software project. Fig 3.1-SDLC Framework Requirement Gathering and Analysis Design Implementation and Coding Testing Deployment Maintenance 40 3.1.1 Requirement Gathering and Analysis Phase Gathering requirements for a project is the most important part of SDLC. In this phase, the expectations of the project are stated, including those who will use the system, how the system will be used, and what is expected of the system’s functionalities. When the requirements are stated, the analysis of each requirement starts to check for its feasibility and to ensure the requirements can be included in the software without causing breaks or problems in the system functionality. The requirements for EmoAI Smart Classroom was elicited through extensive review of existing systems. The system requirements are: 1. The system should be able to extract academic affective states and hands-over-face gestures to five overall emotionsengaged, boredom, frustrated, confusion, and distracted 2. From the emotion analysis, the system should be able to classify students’ engagement into High, Medium and Low. 3. Lecturers should be able to sign up and log in to the system. 4. Lecturers should be able to upload pre-recorded classroom video 5. Lecturers should be able to see the engagement analysis of the class as a whole. 6. Students should be able to sign up and login 7. Students should be able to register their faces in the database. 8. The visualization web app should be flexible, usable, operable by an experienced user. 41 3.1.2 System Design After requirement gathering phase, the next phase on the process model is the system design. The list of requirements developed in the definition phase is used to make design choices. One or more designs are created to achieve the project result. Depending on the project, the design phase could include diagrams, flowcharts, sketches, prototypes, screen designs, Unified Modeling Language (UML) Schemas, etc. In the design phase, we move from a requirements specification to a design specification and decisions made at this level will affect the success of software construction. The system design stage consists of four distinct stages which can be carried out in no particular order; these stages are Data design, Interface design, Architecture design and Component design. 1. Data design which involves determining how data will be organized, stored, maintained, updated, accessed and used. 2. Interface design which involves designing an overall user interface; including screens, commands, controls, and features that enable users to interact with an application. 3. Architecture design which determines client/server interaction, network configuration, internet/intranet interface issues and communication mechanism between components. 4. Component Design which determines the processing strategies and how business process can be refined into algorithms that can easily be converted to program codes. In the design of this project, an object oriented modeling approach is taken into consideration. Implementation will be done using an Object oriented programming language. 42 3.1.2.1 Unified Modeling Language (UML) Unified Modeling Language was created to forge a common visual language in the complex world of software development that would also be understandable for business users and anyone who wants to understand a system. UML diagrams describe the boundary, structure, and the behavior of the system and the objects within it. The UML diagrams used to model this project are; Use case diagram, Activity diagram, Class Diagram, Sequence Diagram, state machine diagram and a proposed framework for the EmoAI Smart Classroom. 3.1.2.2 System Framework The system framework is often a layered structure that indicates the type of program to be built and how they would interrelate. System framework serves as a blueprint for how the component of the system is arranged in realizing the functionality of the system. The existing framework as proposed by (Pabba and Kumar, 2021) is shown in fig 3.2. This framework talked about analyzing student engagement levels into medium, low and high using two separate but complementary modules known as the offline and online modules. The offline module is a trained Convolutional Neural Network-based Facial Emotion Recognition Model while the online module uses the model in the offline module to estimate in real-time student engagement levels. The flowchart is shown in fig 3.4. In contrary to the existing framework, a new emotional and behavioral student engagement is proposed as seen in fig 3.3. Adjustment is made to the offline module by introducing a YOLO V8based Hand-over-face recognition model for identification of the learning behaviors of the students in the classroom. Furthermore, in the Pabba and Kumar Model, adjustment is made to the post- 43 processing module by including a visualization module in addition to the engagement level graph of individual students in the classroom. The flowchart is shown in fig 3.5 This new framework specifies an integration of the subsystems in the existing framework and an addition of a new subsystem to allow for an improved recognition accuracy of student engagement in the classroom to improve classroom teaching experience. The EmoAI Smart Classroom proposed framework is in three major stages which is further broken down into sub-stages: which will be briefly explained. [1.] Dataset Construction Public dataset will be used for the data repository. DAiSEE dataset is a public dataset that consists of students’ facial expressions in an E-learning environment. The affective states contained in the dataset are boredom, confusion, engagement, and frustration. [2.] YOLO V8 Object Detection Architecture (You Only Look Once) YOLO is a state-of-the-art algorithm that uses an end-to-end neural network that makes predictions of bounding boxes and class probabilities all at once. It performs all of its predictions with the help of a single fully connected layer. The algorithm takes an image as input and uses simple deep convolutional neural network to detects objects in the image. It introduces a loss function and processes images at higher resolutions making it suitable for detecting smaller objects. [3.] Offline Module i. Video Acquisition In the video acquisition process, the lecturer uploads classroom videos of interest; The video format of the video streams is mp4 or MOV, and finally the video data is stored 44 in a database or a file system for future analysis and improvement of the engagement detection system. The storage format should be chosen based on the requirements of the engagement detection system, such as the storage capacity, data retrieval speed, and data security. ii. Pre-Processing The first step in preprocessing is to detect the faces of the students in the video streams. This was done using a face detection algorithm known as HOG-based face detection. After the faces have been detected, the next step is to align them so that the facial features can be consistently extracted and analyzed. This was done using OpenCV, a deep learning-based method. After the faces have been aligned, they can be cropped and resized to a consistent size for further processing. The next step is important to ensure that the facial features are extracted accurately and consistently. To ensure that the facial features are extracted accurately and consistently, the images were subjected to histogram equalization to adjust the brightness and contrast of the images. The final step in preprocessing is to normalize the data so that the features can be consistently extracted and analyzed. This was done by transforming the images to a standard size, format, and color space iii. Student Affective Classification At this stage, the dataset gotten from a combination of facial expressions and body posture were labeled with their corresponding affective state categories; The model was trained on the extracted features and the corresponding affective state categories using an appropriate optimization algorithm. The performance of the 51 No No No Yes Yes Yes Yes No Fig—3.7 Activity Diagram for EmoAI Smart Classroom System Student Lecturer Function Function Function Login Login successful? Login Login successful? Login Login successful? Upload successful? Upload Picture Load Database Take snapshots Identify student faces Output Name & No Attention Tracking Engagement Assessment Generate Reports View student attention analysis View student engagement analysis Clear classroom videos 52 3.1.2.5 Class Diagram A class diagram describes the structure of a system by showing the system’s classes, their attributes, operations, and relationships among the objects. The model is often used to construct the code used in the implementation of the system. The explanation of drawing the EmoAI is shown below: Class: Student Attributes: name, studentID, engagementScore Methods: getName(), getStudentID(), getEngagementScore(), setEngagementScore(score) Class: Lecturer Attributes: name, lecturerID Methods: getName(), getlecturerID(), viewEngagementAnalysis(), viewAttentionAnalysis() Class: Classroom Attributes: classroomID, studentList Methods: addStudent(student), removeStudent(student), getStudentList(), getClassroomID() Class: VideoProcessor Attributes: videoStream Methods: startCapture(), stopCapture(), processVideo(), getVideoStream() Class: EngagementDetector Attributes: videoProcessor, audioProcessor, engagementScores 53 Methods: processData(), calculateEngagementScores(), getEngagementScores(), setEngagementScores(scores) Class: ReportGenerator Attributes: classroom, engagementScores Methods: generateReport(), getReport(), getClassroom(), getEngagementScores() Class: Database Attributes: data Methods: storeData(data), retrieveData(query), updateData(data) The class diagram in fig 3.8 shows the static structure of the EmoAI Smart Classroom which comprises of different classes such as: Face Recognizer, Engagement Detector, etc., There are also representation of relationship such as inheritance, aggregation and association between the classes. 54 1..* sit 1 Fig 3.8—Class Diagram of EmoAI Smart Classroom 1 captures name lecturerID Lecturer getName() getlecturerID() viewEngagement Analysis() viewAttentionAna lysis() VideoStream VideoProcessor startCapture() stopCapture() getVideoStream() processVideo() VideoProcessor engagementScores Engagement Detector processData() getEngagementsc ore() calculateEngagem entScores() classroom engagementScores Report Generator generateReport() getReport() getClassroom() getEngagementSc ores() data Database storeData() retrieveData() updateData() Name StudentID Dept engagementScore Student getEngagementScor e() setEngagementScor e() getStudentId() getName() handoverface() classroomID studentlist Classroom addStudent() removeStudent() getStudentlist() getClassroomID() teaches 1 1..* 1 1 1 1 has has 1 1 1 1 stores 1 view 55 3.1.2.6 Sequence Diagram A sequence diagram shows object interactions arranged in time sequence. It depicts the objects and classes involved in the scenario and the sequence of messages exchanged between the objects needed to carry out the functionality of the scenario. They are always associated with use case realization. A sequence diagram for a student engagement detection system in a large offline classroom could be as follows:  System Initialization: A teacher starts a lecture and triggers the student engagement detection system by pressing a button on a remote control.  Class Start: The system captures the video input from multiple cameras placed in the classroom and starts processing the data.  Attention Tracking: The video data is analyzed in real-time to identify and track the faces of the students and to extract features such as gaze direction, head position, and facial expressions.  Student Engagement Detection: The extracted features are used to compute the engagement scores of each student. The engagement scores are based on factors such as attention, interest, and participation in the lecture.  Visualization: The engagement scores are displayed in real-time on a monitor for the teacher. The teacher can use the information to make adjustments to their teaching style and content to improve student engagement.  Engagement Feedback: At the end of the lecture, the system generates a report that summarizes the engagement levels of the students during the lecture. The report can be 56 used by the teacher to provide feedback to the students and to track their progress over time.  Storage: The system stores the video and audio data along with the engagement scores in a database for future analysis and improvement. Fig 3.9—Sequence Diagram of EmoAI Smart Classroom. login authenticate Upload face image :student :camera :system :lecturer track gaze Compute engagement score View engagement analysis generate report Video storage clear video login Capture video input authenticate 57 CHAPTER FOUR SYSTEM IMPLEMENTATION AND EVALUATION 4.1 Introduction This chapter provides a comprehensive overview of the system implementation at both the model and interface levels. It encompasses the computer vision aspect, describing the tasks involved in developing the system and outlining the necessary resources for successful implementation. The system implementation is the process that involves acquiring the necessary software resources and the evaluation of the system so as to eliminate any observed bugs. The following subsections will dive more into the resources and processes involved in the development of the engagement system. 4.2 Implementation at Model Level This section talks about the tools and processes involved in building the machine learning model from the data collection to deployment (see fig 4.1). It leverages object detection techniques and utilizes YOLOV8 architecture to identify and classify the affective states of the students. The code for the computer vision computing was written in python programming language and the following are the steps involved: Fig 4.1 Implementation at Model Level 58 4.2.1 The DAiSEE Dataset DAiSEE (Dataset for Affective States in E-Environment) is the first multi-label video classification dataset comprising 9,068 video snippets captured from 112 users for recognizing user affective states of boredom, confusion, engagement, and frustration “in the wild”. The dataset has four levels of labels namely-very low, low, high and very high for each of the affective states. Why DAiSEE? The unique features of DAiSEE which made it desirable for this project are highlighted below:  It is the first publicly available dataset for studying user engagement and related affective states in the wild.  While there has been substantial work in recognition of the seven basic emotions, DAiSEE presents a dataset to understand subtler states such as engagement and boredom which are often not exhibited explicitly on a human.  It contains videos, thus allowing researchers to use the temporal information for effective recognition. 4.2.2 Data Collection DAiSEE was made available by Paper With Code through the link below: https://people.iith.ac.in/vineethnb/resources/daisee/index.html 59 4.2.3 Frame Extraction Frame extraction is the process of capturing individual frames or images from a video sequence. Since videos are essentially a sequence of frames that are displayed rapidly one after another creating the illusion of motion. In order to capture the temporal changes in affect, a tool known as FFMPEG was used to extract frames at 30 frames per second (30fps). The frames extracted from the video (9,068) totaled to 2,723,882. 4.2.4 Data Annotation Data annotation involves labeling data to show the outcome one wants the machine learning model to predict. This is an important step because without data annotation, every image extracted will look the same to the machine. Annotated data reveals features that will train the algorithm to identify the same features in data that has not been annotated. Semi-Manual annotation using Label Assist was done on Roboflow (see fig 4.2). 707 images from the frames were loaded into the Annotation section of RoboFlow, and bounding boxes were drawn on the area of interest to map each affective state detected on the face to a label. One new label was added in addition to the four labels of interested. Hence the faces were mapped against five labels which are: “Engaged, confused, frustrated, distracted, bored.” 60 Fig 4.2 Annotation on RoboFlow (Roboflow.inc) 4.2.5 Model Building  Data Split To prepare the dataset for modeling, the data was split into train, test and validation sets. The splitting ratio used is 70-20-10. The general Kaggle convention could have also been used, however, the RoboFlow convention was used.  Data Pre-Processing To further prepare the dataset for modeling, the following techniques were used to enhance the quality and suitability of the input data (see fig 4.3) for the detection model: 67  Splash Screen Fig 4.7 Splash Screen  Student Sign Up Page This is the signup page for students, where students can register with the school email address, full name, matric no and password Fig 4.8-Student Sign Up Page This is the home page of EmoAI, where students and lecturers can sign up or login and perform the necessary activities on the webapp 68  Student Log-In Page Students log in with their matric no and password Fig 4.9-Student Log-In Page 69  Lecturer Sign Up This is the register page of lecturers where lecturers fill in their details such as the department, registration no, full name and password. Fig 4.10-Lecturer Sign Up 70  Lecturer Log In Lecturers log in with their registration number and password Fig 4.11 Lecturer-Log In Enter your Reg No 71  Lecturer-Classroom Video Upload This is where lecturers upload pre-recorded classroom videos to get the engagement analysis Fig 4.12-Video Upload 72  Lecturer-Engagement Analysis This is where lecturers get to see the engagement analysis of the students. Fig 4.13 –Engagement Analysis 73 4.5 Workflow of the Integration of the Model and User Interface The following are the steps taken to connect the model with the interface. The interface that was built was the lecturer’s part where the lecturer as an admin uploads the video to get an engagement analysis report. Step 1: Developing the ML Model and API 1. First, the ML model is trained to effectively classify engagement levels in classroom videos, distinguishing between low, medium, and high engagement. 2. Next, an API endpoint is established within the backend infrastructure. This endpoint, created using a programming language like Python and frameworks such as Flask or FastAPI, exposes the ML model's classification capabilities. 3. The endpoint's functionality encompasses accepting uploaded classroom videos, performing any necessary preprocessing, and then passing the videos through the ML model. Subsequently, the endpoint returns the determined engagement classification, thus forming a bridge between the backend and the model. Step 2: Setting Up Strapi CMS 1. To manage content, a Strapi CMS is installed and configured on the server. This involves defining requisite content types and corresponding fields. One key content type, labeled "Video," includes fields like title, and a mechanism for uploading video files. 2. Role-based permissions are tailored to allow lecturers to upload videos, while administrators can oversee content management. 74 3. In the Strapi CMS, a webhook or server function is implemented to activate whenever a new video is uploaded. This function serves to connect with the ML model's API endpoint, passing the uploaded video for engagement classification. The obtained classification report and video details are subsequently stored in the Strapi database. Step 3: Developing the Next.js Front End 1. The process begins by configuring a Next.js project for the frontend. 2. A user-friendly interface is crafted, enabling lecturers to seamlessly upload classroom videos. Within this interface, user-friendly form components are integrated, facilitating video detail input and file uploads. 3. Upon uploading a video, the Next.js application initiates an API call to the Strapi server. A loading indicator visually represents the ongoing process. 4. Once the engagement classification report is generated, the interface dynamically presents both the uploaded video and the associated engagement classification, providing a comprehensive overview of the uploaded content. Step 4: Integration Flow and User Interaction 1. Lecturers access the Next.js application and securely log in. 2. Within the application, a designated section facilitates the hassle-free upload of classroom videos. 3. Lecturers select the desired video file 4. By triggering the "Upload" command, the Next.js application seamlessly transmits the video file and its details to the Strapi backend through an API call. 75 5. Strapi, upon receiving the data, promptly activates a webhook or server function, which establishes communication with the ML model's API endpoint. The uploaded video is then sent for engagement classification. 6. After processing, the ML model's classification report is communicated back to Strapi, which dutifully stores both the video details and the engagement report within its database. 7. A subsequent API call from the Next.js application to Strapi retrieves the engagement report, categorizing engagement as low, medium, or high. 8. The resulting engagement report is visually presented on the user interface, providing lecturers with valuable insights into the uploaded classroom video's engagement levels. The pictures of the processes are shown below: Fig 4.14 Interface of the Strapi Dashboard 76 Fig 4.15 A new user (admin) is created Fig 4.16 The Engagement report dashboard 83 REFERENCES 1. Aguilar, J., Sánchez, M., Cordero, J., Valdiviezo-Díaz, P., Barba-Guamán, L., & Chamba-Eras, L. (2018). Learning analytics tasks as services in smart classrooms. Universal Access in the Information Society, 17, 693-709. 2. Aist , G. , B. Kort , R. Reilly , R.W. Picard , and J. Mostow . 2002 . Experimentally augmenting an intelligent tutoring system with human supplied capabilities: Adding human-provided emotional scaffolding to an automated reading tutor that listens . In: Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS2002) , Biarritz, France and San Sebastian, Spain, June 2–7, 2002 3. Alkabbany, A. Ali, A. Farag, I. Bennett, M. Ghanoum and A. Farag, “Measuring Student Engagement Level Using Facial Information”, 2019. IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 3337-3341, doi:10.1109/ICIP.2019.8803590 4. Andallaza, T.C.S., Rodrigo, M.M.T., Lagud, M.C.V., Jimenez, R.J.M., Sugay, J.O. (2012). Modeling the affective states of students using an intelligent tutoring system for algebra. In Proc. International Workshop on Empathic Computing (IWEC). 5. Ashwin T.S., Ram Mohana Reddy Guddeti, Affective database for e-learning and classroom environments using Indian students’ faces, hand gestures and body postures, Future Generation Computer Systems,Volume 108, 2020, Pages 334-348, ISSN 0167739X, https://doi.org/10.1016/j.future.2020.02.075. 6. Barrett LF, Adolphs R, Marsella S, Martinez AM, Pollak SD. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychol Sci Public Interest. 2019 Jul;20(1):1-68. doi: 10.1177/1529100619832930. Erratum in: 84 Psychol Sci Public Interest. 2019 Dec;20(3):165-166. PMID: 31313636; PMCID: PMC6640856. 7. Behera, A., Matthew, P., Keidel, A. et al. Associating Facial Expressions and Upper Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems. Int J Artif Intell Educ 30, 236-270 (2020). https://doi.org/10.1007/s40593-020-00195-2 8. Bradbury, N. A. (2016). Attention span during lectures: 8 seconds, 10 minutes, or more?. Advances in Physiology Education, 40(4), 509–513 9. Calvo RA, D’Mello S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing. 2010;1: 18– 37 10. Canedo, Daniel & Trifan, Alina & Neves, António. (2018). Monitoring Students’ Attention in a Classroom Through Computer Vision. 10.1007/978-3-319-94779-2_32. 11. Chen, S., Dai, J., & Yan, Y. (2019). Classroom teaching feedback system based on emotion detection. In 9th International Conference on Education and Social Science (ICESS 2019) (pp. 940-946). 12. Craig, S.D., Graesser, A. C.,Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning. Journal of Educational Media, 29, 241-250. 13. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: HarperCollins Publishers, Inc. 14. D'Mello , S. K. , S. D. Craig , B. Gholson , S. Franklin , R. Picard , and A. C. Graesser . 2005 . Integrating affect sensors in an intelligent tutoring system. Affective 85 interactions: The computer . In: The Affective Loop Workshop at the 2005 International Conference on Intelligent User Interfaces . New York : AMC Press 15. De Villiers, Bridget & Werner, Amanda. (2018). The relationship between student engagement and academic success. Journal for New Generation Sciences. 14. 36. 16. D’Mello, S.K., Craig, S.D., Withserspoon, A., Mcdaniel, B., & Graesser, A. (2010). Automatic detection of learner’s affect from conversational cues. User Modeling and User-Adapted Interaction, 18 (1-2), 45-80 17. D’Mello, S., Craig, S.D., Gholson, B., Franklin, S., Picard, R., Graesser, A. (2005). Integrating affect sensors in an intelligent tutoring system. In Affective Interactions: The Computer in the Affective Loop Workshop (pp. 7–13). 18. Ekman, P., Friesen, W.V., 1969. The repertoire of nonverbal behavior: Categories, origins, usage, and coding, Semiotica, 1: 49-98 19. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3), 169–200. 20. Fredricks, J.A., Blumenfeld, P.C., Paris, A.H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74 (1), 59–109, http://doi.org/10.3102/00346543074001059. 21. Friedman, I. A. (2006). Classroom management and teacher stress and burnout. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 925– 944). Erlbaum. 22. Furrer, C. J., Skinner, E. A., & Pitzer, J. R. (2014). The influence of teacher and peer relationships on students’ classroom engagement and everyday motivational resilience. Teachers College Record, 116(13), 101-123. 86 23. Gelder, B.D. (2006). Towards the neurobiology of emotional body language. Nature Reviews Neuroscience, 7(3), 242–249. 24. Gordon, G., Spaulding, S., Westlund, J.K., Lee, J.J., Plummer, L., Martinez, M., Das, M., Breazeal, C. (2016). Affective personalization of a social robot tutor for children’s second language skills. In Proc. of the Thirtieth AAAI Conference on Artificial Intelligence (pp. 3951–3957) 25. Graesser, A.C., & Olde, B.A. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524-536. https://doi.org/10.1037/0022-0663.95.3.524 26. Graesser, A. C., & D’Mello, S. (2012). Emotions during the learning of difficult material. In Psychology of learning and motivation (Vol. 57, pp. 183-225). Academic Press. 27. Harley, J.M. (2016). Measuring emotions: A survey of cutting-edge methodologies used in computer-based learning environment research. In S. Tettegah & M. Gartmeier (Eds.), Emotions, technology, design, and learning (pp.89-119). London: Academic Press. 28. Jayasree, R. (2014). Effective communication. Retrieved from http://www.laurieford.com 29. Kaliouby, R., & Robinson, P. (2005). Real-time inference of complex mental states from facial expressions and head gestures. In Proc. Real-Time Vision for HCI (pp. 181– 200). 87 30. Kapoor, A., Burleson, W., Picard, R.W. (2007). Automatic prediction of frustration. International Journal of Human-computer Studies, 65(8), 724–736. 31. Kim, Y., Soyata, T., & Behnagh, R, F. (2018). Towards emotionally aware AI Smart Classroom: Current Issues and Directions for Engineering and Education. IEEE Access, 6, 5308-5331. http://doi.org/10.1109/access.2018.2791861 32. Klein, R., and Celik T. (2017). The Wits Intelligent Teaching System: Detecting student engagement during lectures using Convolutional Neural Network. IEEE International Conference on Image Processing (icip) (pp. 2856-2860). http://doi.org/10.1109/ICIP.2017.8296804 33. Kret, M. E., Stekelenburg, J. J., Roelofs, K., & De Gelder, B. (2013). Perception of face and body expressions using electromyography, pupillometry and gaze measures. Frontiers in psychology, 4, 28. 34. Krithika, L. B., & GG, L. P. (2016). Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Computer Science, 85, 767-776. 35. L. Chen, P. Chen and Z. Lin, "Artificial Intelligence in Education: A Review," in IEEE Access, vol. 8, pp. 75264-75278, 2020, doi: 10.1109/ACCESS.2020.2988510. 36. Li, Y., Li, F., Chen, J., and Li, H. (2012). An ERP study on the understanding of the distinction between real and apparent emotions. Neurosci. Lett. 529, 33–38. doi: 10.1016/j.neulet.2012.08.063 37. Mahmoud, M., & Robinson, P. (2011). Interpreting hand-over-face gestures. In Proc. International Conference on Affective Computing and Intelligent Interaction (pp. 248– 255). 88 38. Mrabet, H. E., & Moussa, A. A. (2017). Smart classroom environment via IoT in basic and secondary education. Transactions on Machine Learning and Artificial Intelligence, 5(4). 39. Oster, H., & Ekman, P. (1978). Facial behavior in child development. In W. A. Collins (Ed.), Minnesota Symposia on Child Psychology (Vol. 11, pp. 231-276). Hillsdale, NJ: Erlbaum. 40. P. Ekman and W. V. Friesen, “Facial Action Coding System: A technique for the measurement of facial movement, Consulting Psychologists Press,” Palo Alto, CA, 1978. 41. Pabba, C., & Kumar, P. (2022). An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition. Expert Systems, 39(1), e12839. 42. Pease, A., & Pease, B. (2006). The definitive book of body language: How to read others’ attitudes by their gestures. Orion, London, England. 43. Picard RW, Klein J. Computers that recognise and respond to user emotion: Theoretical and practical implications. Interacting with computers. 2002;14: 141–169. 44. Prabin Sharma, Shubham Joshi, Subash Gautam, Sneha Maharjan, Vitor Filipe, Manuel Cabral Reis. (2018). Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning. 45. R. Zheng, F. Jiang and R. Shen, "Intelligent Student Behavior Analysis System for Real Classrooms," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020, pp. 9244-9248, doi: 10.1109/ICASSP40776.2020.9053457. 89 46. Raca, M., Dillenbourg, P.: System for assessing classroom attention. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. ACM (2013) 47. Rautaray, S.S., & Agrawal, A. (2015). Vision-based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43(1), 1–54. 48. S. Peng and K. Nagao, "Recognition of Students’ Mental States in Discussion Based on Multimodal Data and its Application to Educational Support," in IEEE Access, vol. 9, pp. 18235-18250, 2021, doi: 10.1109/ACCESS.2021.3054176. 49. Shuck, B., Albornoz, C., Winberg, M. (2007). Emotions and their effect on adult learning: a constructivist perspective. In Nielsen, S.M., & Plakhotnik, M.S. (Eds.) Proc. Sixth Annual College of Education Research Conference: Urban and International Education Section (pp. 108–113). 50. Skinner, E. A., and Pitzer, J. R. (2013). “Developmental dynamics of student engagement, coping, and everyday resilience,” in Handbook of research on student engagement, eds S. L. Christenson, A. L. Reschly, and C. Wylie (New York, NY: Springer), 21–44. doi: 10.1007/978-1-4614-2018-7_2 51. Soloviev (2018). Machine learning approach for student engagement automatic recognition from facial expressions. Scientific Publications of the state university of Novi Pazar Series A: Applied Mathematics informatics and mechanics 52. T. S. Ashwin and R. M. R. Guddeti, "Unobtrusive Behavioral Analysis of Students in Classroom Environment Using Non-Verbal Cues," in IEEE Access, vol. 7, pp. 150693150709, 2019, doi: 10.1109/ACCESS.2019.2947519. 90 53. Taylor, John & Fragopanagos, Nickolaos. (2005). The interaction of attention and emotion. Neural networks*: the official journal of the International Neural Network Society. 18. 353-69. 10.1016/j.neunet.2005.03.005. 54. Tulis, Maria & Fulmer, Sara. (2013). Students' motivational and emotional experiences and their relationship to persistence during academic challenge in mathematics and reading. Learning and Individual Differences. 27. 35-47. 10.1016/j.lindif.2013.06.003. 55. Turner, J.C., Thorpe, P.K., & Meyer, D.K. (2002). Students’ reports of motivation and negative affect: A theoretical and empirical analysis. Journal of Educational Psychology, 90 (4), 758. 56. Valdiviezo, P., Cordero, J., Aguilar, J., Sánchez, M.: Conceptual design of a smart classroom base on multiagent system. In: de International Conference on Artificial Intelligence (ICAI’15), pp. 471–477 (2015) 57. Vanneste P, Oramas J, Verelst T, Tuytelaars T, Raes A, Depaepe F, Van den Noortgate W. Computer Vision and Human Behaviour, Emotion and Cognition Detection: A Use Case on Student Engagement. Mathematics. 2021; 9(3):287. https://doi.org/10.3390/math9030287 58. Vanneste, P., Oramas, J., Verelst, T., Tuytelaars, T., Raes, A., Depaepe, F., & Van den Noortgate, W. (2021). Computer Vision and Human Behavior, Emotion and Cognition Detection: A Use case on Student Engagement. Mathematics, 9(3), 287. http://doi.org/10.3390/math9030287 91 59. Whitehill, J., Serpell, Z., Lin, Y.C., Foster, A., Movellan, J.R. (2014). The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98. 60. Woolf, B. P. (2010). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Morgan Kaufmann. 61. Zaletelj, J., Košir, A. Predicting students’ attention in the classroom from Kinect facial and body features. J Image Video Proc. 2017, 80 (2017). https://doi.org/10.1186/s13640-017-0228-8 62. Zeng Z, Pantic M, Roisman GI, Huang TS. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2009;31: 39–58. 10.1109/TPAMI.2008.52 63. Zhang, X., & Chen, L. (2021). College English smart classroom teaching model based on artificial intelligence technology in mobile information systems. Mobile Information Systems, 2021, 1-12. 64. Zhenzhen Luo, Chen Jingying, Wang Guangshuai & Liao Mengyi (2022) A threedimensional model of student interest during learning using multimodal fusion with natural sensing technology, Interactive Learning Environments, 30:6, 1117 1130, DOI: 10.1080/10494820.2019.1710852. 92 APPENDIX A """ ************************************************************* * * * Project: Student-Engagement Detection * * * * Program name: app.py * * * ************************************************************* """ """ ************************************ * * * EMOTION DETECTION SECTION * * * ************************************ """ # import module # =============== TWO MODEL APPROACH ==================== # using face detection and attention detection import time from threading import Thread from flask_cors import CORS 99 else: break input_video.release() video_out.release() return output_file_name, a_count_all, d_count_all """ ************************************ * * * SERVER CODE SECTION * * * ************************************ """ app = Flask(__name__) CORS(app, resources={r"/*": {"origins": "*"}}) UPLOAD_FOLDER = os.path.join(os.path.dirname(__file__), 'public/videos') ALLOWED_EXTENSIONS = {'mp4'} app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER def allowed_file(filename): return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS @app.route('/', methods=['POST']) def create_video(): try: 100 if 'video' not in request.files: return jsonify({ 'status': "error", 'message': 'No video' }), 400 video = request.files['video'] if video.filename == '': return jsonify({ 'status': "error", 'message': 'No selected video' }), 400 if video and allowed_file(video.filename): timestamp = str(int(time.time())) filename = timestamp + '_' + video.filename filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename) print(filepath, 'filepath') with open(filepath, 'wb') as f: f.write(video.read()) output_filename, attentive_count, drowsy_count = object_detection_module( filepath) print(output_filename, attentive_count, drowsy_count) return jsonify({ 'status': "success", 101 'data': { 'result': output_filename, 'sum_inference': 1000, 'engaged_percent': 1000, 'boredom_percent': 80, 'confusion_percent': 90, 'frustration_percent': 110, 'distracted_percent': 600, 'attentive_count': attentive_count, 'drowsy_count': drowsy_count, 'total_count': attentive_count + drowsy_count }, 'message': 'Video uploaded successfully' }), 201 else: return jsonify({ 'status': "error", 'message': 'You can only upload a video in mp4 format' }), 500 except Exception as e: traceback.print_exc() print(e) return jsonify({ 'status': "error", 'message': 'Unable to upload video' }), 500 @app.route('/stream/<filename>', methods=['GET']) def download_video(filename): 102 try: filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename) print(filepath) if os.path.exists(filepath): def generate(): with open(filepath, 'rb') as f: while True: # adjust the chunk size as needed chunk = f.read(1024) if not chunk: break yield chunk return Response(generate(), mimetype='video/mp4') else: return jsonify({ 'status': 0, 'message': 'File not found' }), 404 except Exception as e: print(e) return jsonify({ 'status': 0, 'message': 'Error while streaming file' }), 500 if __name__ == '__main__': app.run(debug=True, port=8000)