A data value chain to model the processing of multimodal evidence in authentic learning scenarios

A Data Value Chain to Model the Processing of Multimodal Evidence in Authentic Learning Scenarios Shashi Kant Shankar1, Adolfo Ruiz-Calleja2, Sergio Serrano-Iglesias2, Alejandro Ortega-Arranz2, Paraskevi Topali2, and Alejandra Mart´ınez-Mon´es2 1Tallinn University, Tallinn, Estonia [email protected] 2GSIC-EMIC Research Group, Universitdad de Valladolid, Valladolid, Spain {adolfo,sergio,alex,evi.topali}@gsic.uva.es,[email protected] Abstract. Multimodal Learning Analytics (MMLA) uncovers the possibility to get a more holistic picture of a learning situation than traditional Learning Analytics, by triangulating learning evidence collected from multiple modalities. However, current MMLA solutions are complex and typically tailored to specific learning situations. In order to overcome this problem we are working towards an infrastructure that supports MMLA and can be adapted to different learning situations. As a first step in this direction, this paper analyzes four MMLA scenarios, abstracts their data processing activities and extracts a Data Value Chain to model the processing of multimodal evidence of learning. This helps us to reflect on the requirements needed for an infrastructure to support MMLA. Keywords: Multimodal Learning Analytics ·Data Value Chain ·Multimodal Learning Scenarios ·Data Value Chain 1 Introduction In the last decade the Technology-Enhanced Learning (TEL) community has witnessed the emergence of Learning Analytics (LA) [17]. A myriad of research works have been published where learning evidence is collected from a learning situation and processed to analyze the situation or to support evidence-based decision making. However, most of these research works only collect and process software logs like Learning Management System (LMS) log. This is an important restriction as they can only provide a partial view of the learning situation. As an example, we can consider the analysis of collaboration in a blended-learning class where a collaborative text editor is used. If students’ collaboration is assessed only out of the logs from text editor, we will get a partial view of the collaboration process because interactions may also happen face to face or with the support of other software tools. In order to overcome these problems Multimodal Learning Analytics (MMLA) was proposed as an LA sub-field that “triangulates among non-traditional as Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. well as traditional forms of data which represent multimodal learning evidence in order to characterize or model students’ learning in complex learning environments” [20]. To operationalize the definition of modality in a blended learning context, modality represents the different modes of learning progress over different communication channels [22]. These communication channels represent the interaction between stakeholders and (or) learning resources which can be tracked through different techniques like observations, audio, video, observation, and sensors which may end up in one or more than one datasets. Finally, these datasets can be further analyzed based on the low- or high-level features which can help to answer the main objective of MMLA in any specific case. MMLA leverages the possibilities of recent digital advancement to provide a holistic view of a learning situation, but it implies a complex technical ecosystem [3]: multiple modalities of data should be collected, processed and triangulated [19]; different data collection devices are used (e.g., sensors, software tools or questionnaires); and new data visualization tools are required to hide the underlying complexity [18]. For this reason, building MMLA solutions is a complex and time-demanding task. In a recent review to the topic [16], we saw that most of the proposals are tailored for a specific learning situation and it is very difficult to reuse them. Furthermore, there is a lack of guidance about the modelling of different data processing activities required in an MMLA solution. This lack of reusability and guidance can potentially hinder the adoption of MMLA proposals, as a significant effort is required to adapt them to each learning situation. We are working towards a software infrastructure for MMLA that can be adaptable to different learning situations. Our approach draws on the Data Value Chain [9] model to conceptualize the support needed in each phase of the MMLA process by the relevant stakeholders. This paper reports our first step in this direction. We have analyzed four learning scenarios inspired in real learning situations where MMLA has been used to support multimodal evidence-based teaching and learning practices. In order to process multimodal evidence in these four scenarios, we identify the requirements posed by the involved stakeholders and the external information required to guide the data processing activities of DVC. Hence, to model the data processing activities with the external information and the intended stakeholders, we extract and report a DVC in this paper. The rest of this paper is structured as follows: First, we outline the state of the art (Section 2). We present four different learning scenarios that illustrate different cases in which MMLA has been applied (Section 3). Out of them we extract a Data Value Chain [9] (Section 4), so that we can synthesize the support needed by these scenarios in a single conceptual tool. Finally, we reflect on how an infrastructure could provide a common support to these and other learning situations where MMLA is applied (Section 5). Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 72 2 State of the art LA community has seen a rapid growth in the last decade due to the possibilities which support evidence-based decision making to educational stakeholders [17]. However, most of the LA existing projects analyze system logs of the digital platform used in the learning situation [7]. This mono-modal nature of LA solutions paints a partial picture of the learning process, learning context and the environment where learning progresses. To get a wider and holistic picture, learning evidence collected from one modality needs to be complemented with learning evidence collected from other modalities [18]. The availability of low-cost sensors and other Internet-of-Things (IoT) devices provides many opportunities to collect learning evidence from multiple modalities [18]. MMLA is a young research area under the umbrella of LA. MMLA leverages the possibility to collect, process and analyze the multimodal evidence collected from multiple modalities which are available in the digital as well as the physical spaces of a learning situation [21]. As an approach, MMLA potentially provides pedagogically-meaningful information to support the needs of multiple stakeholders. Unlike other domains (e.g., business, finance, entertainment), multimodal data processing in educational context is complex due to the involvement of multiple stakeholders and cognitive practices [10]. Hence, to exploit the benefits which MMLA as an approach can offer in daily teaching and learning practices, recent MMLA studies explore specific learning scenarios. To the best of our knowledge, most of the scenarios are either controlled or semi-controlled [1, 5, 10, 14]. In this exploration, the MMLA community has proposed few adhoc and tailored MMLA solutions to fit the requirements of specific learning scenarios [1, 5]. The heterogeneity and dynamicity of these learning scenarios including the different types of learning activities, learning space, learning context, pedagogy, and involved stakeholders adds complexity in developing MMLA solutions which can be adapted to other learning scenarios than to the scenario for what they were developed for [4]. Moreover, these solutions usually follow different data processing activities to process raw multimodal evidence. MMLA community has started proposing software infrastructure to deal with the reusability issue of most of the existing MMLA solutions. A recent review used the analytical lens of Data Value Chain (DVC) as a model to highlight different data processing activities that are being used by existing MMLA architectures to process multimodal evidence of learning [16]. This study discusses the importance of contextual information of a learning situation which are crucial information to guide the multimodal data processing: the number of students, the number of groups, the attributes of learning situation datasets, etc. However, to the best of our knowledge, the MMLA community has not addressed what type of contextual information is required to guide the processing of multimodal evidence in each data processing activities of DVC. Hence, in order to study the potential of a DVC to model the processing of multimodal evidence, the required extra contextual information of the learning situation, and the stakeholders who can provide such extra information in each data processing activities to make sense out of raw multimodal evidence. Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 73 Table 1. Learning context and multimodal features of the presented scenarios. Learning Context Multimodality Scen. Participants Course Type Spaces Purpose Data Sources 18Teachers 3 Researchers 20 Students Blended- Learning Activity Digital and Physical Spaces To analyze the impact of innovative teaching practices on group engagement Graasp log Human-Observation 21Teacher 1 Researcher 150 Students Online Course Digital Space To study the effect of gamification on student engagement LMS log 3rd-party tools log Gamif. platform log 32Teachers 20 Students Treasure-Hunt Activity Digital and Physical Spaces To adapt the student learning experience in real-time App log Human-Observation Sensors 41Instructor 1000 Students MOOC Digital Space To help the instructor identify the students facing problems in the course Course forums log Self-reported problems 3 Four scenarios In this section, we describe the four MMLA scenarios extracted from our experience in different MMLA related research projects. For each scenario, we present its learning space, the type of learning activities, and the stakeholders’ involvement and goals. These four scenarios illustrate the heterogeneity of existing learning scenarios and the complexity in implementing MMLA (see Table 1). 3.1 Scenario 1: Open-doors Activity Iris and Lea are two researchers working on a digital learning lab. They are currently setting up open school approaches where they co-design open-doors events with school teachers to test innovative teaching practices and their impact on learning. In one of these events, a group of 20 students (aged 13 to 16 years old) and one school teacher, Amy, visited the university and participated in learning activities for approximately four hours. The learning activities combined face-to-face and computer-mediated work, and had an emphasis on collaborative and/or inquiry learning, as well as subject integration. Amy, Iris and Lea plan to study the engagement of each group of students. To this end, they decide to analyze six parameters (totally disengaged, talking to peers, looking to peers, interacting with technology, resources, and other people) from the physical space and four parameters from the digital space (resource access, creation, opening and update). To capture the parameters from the physical space, they use one observation tool through Google Forms 3. Human-observers do the observation based on this form and submit their response every five minutes during the enactment phase. Moreover, to cover the digital space, they plan 3Observational form available at http://tiny.cc/adek5y Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 74 to analyze the four aforementioned parameters from the system log of Graasp4(a digital platform to manage the Inquiry-Based Learning (IBL)). After the data gathering phase, these two heterogeneous datasets have to be fused and analyzed based on contextual information about the learning situation provided by Amy (the teacher). The analytical results are presented in a timeline chart to show the teacher and the researchers the coherent view of the learning situation based on the multimodal evidence collected across-spaces and enriched with the information about context provided by the teacher. In this scenario, human actors are responsible of providing the real-time observations that feed the MMLA tool. The teacher is responsible of providing the contextual information needed to enrich the analysis. Moreover, the observation tool needs to be adapted and a significant amount of work has to be dedicated to plan, process and exploit the multimodal evidence. 3.2 Scenario 2: Multimodality Supporting Gamification Research Maria is the teacher of an online-learning university course about Spanish to English translation, usually launched in Moodle with approximately 150 enrolled students. The course activities are configured to be performed with the Moodle tools (e.g., assessments through quizzes); 3rd party tools (e.g., glossaries through Google Form and Google Spreadsheets), and external social networks (e.g., posting in Twitter), all of them embedded in the LMS. A researcher that works with Maria (Paul) proposed her to include reward-based gamification strategies to study their effect on student behavioral engagement. To this end, the course gamification was co-designed between Maria and Paul. The gamification design consisted on providing 15 badges associated to different activities of the course such as participating in the course glossaries (Google Spreadsheet), watching course videos (H5P), completing course quizzes with a score upper than 90% (Moodle) and posting in Twitter with the course hashtag (Twitter). The gamification is implemented through the GamiTool platform5. Behavioural engagement in online environments is frequently measured through variables such as the number of page-views, submissions, posts or the activity time [6]. However, the use of reward-based gamification provides additional parameters showing the student engagement with the course contents and rewards. One parameter measuring the reward-derived engagement is the time from the moment the student has completed the reward conditions to the moment when the student claims and earns the reward [11]. Students will potentially show a certain interest on earning rewards when such time is low. On the other hand, students completing the reward-conditions and claiming the rewards at the end 4Graasp https://graasp.eu/, last access: May 2019. 5GamiTool (https://gamitool.gsic.uva.es/) is an ecosystem of applications that allow teachers to grain-fine design and deploy reward-based gamifications in multiple LMSs involving activities performed in different 3rd party tools. The system allows the implementation of a reward page into the course where students can claim the rewards, being automatically handled by the system [12]. Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 75 of the course, potentially show a lower engagement level than the previous students. Therefore, research on reward-derived engagement needs both the timestamp when the reward-conditions are fulfilled (from the LMS and the external tools), and the timestamp when the rewards are claimed and issued (from the gamification platform). In this situation, the researcher is responsible of retrieving the needed information from the different data sources: Moodle, Google Spreadsheets, Twitter, GamiTool, etc. (e.g., through API), isolating the timestamp information, homogenizing the timestamps, and combining them to finally get the expected parameter measuring student engagement. This parameter can be also combined with other behavioural indicators to measure student engagement more precisely. 3.3 Scenario 3: Adaptive treasure hunt This scenario has been extracted from the one presented in [15]. Felipe and Eva are the teachers of a Natural Science course in a primary school. In order to help their students understand the features of the local flora, they set a treasure hunt activity in the school playground. During a full session devoted for the activity, 20 students collaborated in four-person groups to spot different trees. Each group of students received a tablet device in which, through a custom application, students received hints to find the next tree. Once the requested tree is reached, they had to scan a QR code that triggered a task related with the current tree. In previous years, Felipe and Eva experienced that some students found the hints and the tasks too difficult so they were not able to finish the activity. In order to overcome this issue, Felipe and Eva decided to support the treasure hunt with MMLA. They decided to detect struggling students in order to adapt the learning experience of the students and decrease the difficulty if necessary (giving more hints or asking for easier tasks). The information required for these analytics comes from both the physical and virtual spaces. On the one hand, a custom application installed in the tablet devices reports the current location of the students every 10 seconds. This information was used to check if the students where in the same place for a long period of time and their distance to the next target tree (registered in Google Maps). On the other hand, when the students accessed each QR code, the server logged the group id that accessed the resource, the initial and final time-stamp and the score achieved within every associated task. Additionally, teachers can communicate with the different groups during the development of the activity by using the custom application (e.g., to solve an unexpected problem). This case study faced multimodality issues, such as the availability of the data sent by the tablets, due to loss of signal in some areas in the playground; and the aggregation and alignment of data, as the data sources are populating the data in different time frames. Moreover, these issues were emphasized as the activity required real time support, so the system had to identify the struggling groups while they were still performing the treasure hunt. Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 76 3.4 Scenario 4: Multimodality to Support MOOC Learners Nacho is a lecturer teaching Machine Learning at a well-known university 6. After a successful experience of designing and delivering a MOOC for a first time, he is planning to deliver a second version of the same MOOC. The course will be launched in the Canvas Network platform, estimating 1000 enrolled participants. The course will be organized in 4 modules (one module per week) involving content pages, forums and compulsory activities -individual and collaborative ones- (e.g., quizzes, peer assignments). One issue that proved challenging for Nacho during his previous MOOC was the in-time support to the enrolled participants; he devoted a lot of time answering forums’ and private messages’ questions that in many cases were repeated among learners, when not a signal that participants had not paid enough attention to the content materials. Since his workload is high, he decided to examine the learners’ effort devoted, previous to the communication of a problem to help him control which learner he will assist. In the current MOOC, Nacho will follow a multimodal approach to tackle the issue of measuring learners’ effort. He will use a system to analyze the learners’ self-reported data from the communication threads (posts in discussion forums and private messages) and a dashboard to create a record of the learners’ activity traces previous to the communication of the problem. The learners’ trace data that the system will consider are the logs available at the Canvas platform (e.g. number of assignments’ attempts, time spent in the content material pages, delay of submissions, general course participation, etc.). Before launching the course, Nacho will have to configure some conditions, so that the gathered information results meaningful. For example, if several learners state that they face a conceptual problem, but according to their logs they have not watched the course’s related video, these learners will receive an alternative way of support and not a direct answer from the instructor (e.g., a notification to recheck the video). This way, Nacho can prioritize the learners and see to whom is more urgent to provide support and avoid a possible dropout. Although this approach implies extra work in advance, Nacho is interested to try and see if in that way he can assist first the learners’ who need help and have put their maximum effort. 4 A Data Value Chain for MMLA As an step towards the proposal of an MMLA infrastructure that could be reused in different scenarios, we analyzed the scenarios described above and we synthesize their data processing activities with the contextual information and the stakeholders which mediate these activities in a DVC. This DVC is represented in Figure 1. The DVC considers seven multimodal data processing activities which are divided in three groups. The first group -data discoverydeals with collecting, 6Reference anonimyzed Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 77 Input Setup the devices and pla  orms to collect mul  modal evidence Annoated dataset(s) and a  ributes Cleaned, synchronised and prepared dataset(s) Organised dataset(s) Fused dataset(s) Analy  cal result dataset(s) Reports Process •Store data streams from mul  ple modali  es to generate dataset(s) •Annotate the collected dataset(s) and their a  ributes •Synchronize all the dataset(s) under one reference  me zone •Clean the dataset •Remove unrequired a  ributes •Structure dataset(s) •Aggregate and transform to unify dataset(s) •Extract feature(s) from mul  ple a  ributes of dataset(s) Perform either lowlevel (data-based) or high-level fusion (feature-based) Performs di  erent analy  cal algorithms from exploratory analysis to machine learning algorithms Di  erent visualiza  on algorithms to represent the analyzed results Iden  fy the crucial points from reports and highlight them Output Cleaned and annotated dataset(s) Prepared dataset(s) with only required a  ributes Organised dataset(s) Fused dataset(s) Analy  cal result dataset(s) Reports like  meline chart, bar graph, etc. Report with highlighted crucial points Extra informa  on required? Yes. List of a  ributes of each data source Yes. Reference  me zone each tool and pla  orm used for data collec  on Yes. List of a  ributes' values, aggrega  on and transforma  on func  ons Yes. List of common features available in certain a  ributes of dataset(s) Yes. List of analy  cal algorithms Yes. List of repor  ng algorithms Yes. List of threshold values to  nd crucial points Stakeholder(s) IT guy, Teacher, Researcher Teacher, Researcher Teacher, Researcher Teacher, Researcher Teacher, Researcher Teacher, Researcher Teacher, Researcher Data Discovery Data Fusion Data Exploita on Collect & Annotate Prepare Organize Integrate Analyze Visualiza on Make Decisions Fig. 1. Derived DVC from the study of four MMLA scenarios Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 78 annotating, cleaning, synchronizing, transforming and structuring heterogeneous datasets(s); it includes three activities: collect & annotate, prepare, and organize. The second group -data fusionintegrates two datasets based on the features that relate datasets and generate a coherent view of multimodal evidence; it includes only one activity: integrate. The third group -data exploitationanalyzes the fused dataset, visualize the analyzed report and highlight the data points to make decisions; it includes three additional activities: analyze, visualization, and make decisions. Note that this DVC is compliant with other DVCs like Big Data [2, 9]. Next we provide details about each step: Collect & Annotate: In this activity, we setup the devices and tools to collect multimodal evidence from multiple modalities which may span over digital as well as physical spaces. Once the dataset(s) are generated, the attributes which need to be included in multimodal processing are annotated. This activity currently needs the help of an IT expert to setup devices and platforms to collect multimodal evidence. The list of attributes is the external information which is planned by either teachers, researchers, or both. These actors provide this information during the planning phase, based on the question(s) they expect MMLA solution to answer with the help of an IT expert. For example, in Scenario 3, a custom application is installed in each tablet to receive the location data of the students. Similarly, in Scenario 4, Nacho decides the attributes (like number of visits and time spent) that he is looking to collect from multiple modalities (self-reported data of students and activity log of Canvas). These two different datasets and their attributes are annotated. Prepare: This activity involves tasks to synchronize the different dataset(s) under a single reference time zone. Once the dataset(s) are unified, they need to be cleaned (like removing missing values) and unwanted attributes need to be removed. The tasks under this activity need external information from researchers like the time zone of every tool or platform that generates a dataset. For example, in Scenario 2, Moodle and Google are hosted in different time zones. Hence, to unify these two datasets, they need to be synchronized under a single reference time zone. Similarly, in Scenario 1, Graasp records a total of ten activities of user interaction but teachers and researchers are interested in only four of them. Hence, data related to six unwanted activities need to be removed from the dataset. Organize: In this activity, dataset(s) need to be structured and aggregated. Moreover, if needed, selective features should be extracted from the dataset(s). To perform these actions, a list of aggregation and transformation functions are required which are decided by either teachers or researchers. Moreover, a list of values of attributes which define the boundary conditions for the aggregation and transformation functions are also needed. For example, in Scenario 1, observational record of students who belong to one group are averaged to generate the group level observation. Also in this scenario, the count of total occurrences of four verbs of Graasp log (which represent the Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. LASI Spain 2019: Learning Analytics in Higher Education 79