Exploring mechanisms for virtual interaction to improve online teaching

Exploring Mechanisms for Virtual Interaction to Improve Online Teaching Computer Science Engineering and Software Engineering FACULTAD DE INFORMÁTICA Authors Alexander De Aguiar, Sara Sacó Baños, Samuel Santos Hernán Directors Pablo Gervás, Gonzalo Méndez January 23, 2024 Summary The following project consists on the specification, design and implementation of a desktop application that allows users both to attend or to give classes in a nonpresential way, yet comfortably and efficiently. This would be achieved by means of a digital model of the faculty (already existing and given to us by our project conductors, but to which we have had to apply a big, general refactorization), and a Graphical User Interface (GUI) that would help the user experience to be pleasant and easy to control. The application also implements a server-client system in order to make the actual interaction between users possible. The application could also give support for virtual reality glasses, but it must work in any case with a desktop environment, in order to make it accessible to all users. KeyWords Education, 3D, VR, online, server, university, C#, Unity, classroom, Virtual Classes 1 Resumen El proyecto expuesto a continuación consiste en la especificación, diseño e implementación de una aplicación de escritorio, que permita a los usuarios tanto asistir como impartir clases de manera no presencial, cómoda y eficiente. Para lograr este fin se utiliza, por un lado, un modelo digital de la facultad ya existente y proporcionado por nuestros directores de proyecto, al cual hemos tenido que aplicar una gran refactorización general; Por otra parte, el sistema también requiere una Interfaz Gráfica de Usuario (GUI) que ayude a que la experiencia del usuario sea agradable y fácil de manejar. La aplicación implementa un sistema cliente-servidor para hacer posible la interacción real entre usuarios. Así mismo, también podría dar soporte para gafas de realidad virtual, pero debe funcionar en cualquier caso con un entorno de escritorio, con el fin de hacerlo accesible para todos los usuarios. Palabras clave Educación, 3D, VR, online, servidor, universidad, C#, unity, aula, enseñanza virtual 2 Contents Summary 1 Resumen 2 1 Introduction 6 2 State of the Art 10 2.1 Traditional platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.1 GoogleMeet............................ 10 2.1.2 MicrosoftTeams ......................... 11 2.2 Immersiveplatforms ........................... 11 2.2.1 Gather.Town ........................... 11 2.2.2 Teamflow ............................. 12 2.2.3 Meetinvr.............................. 12 2.2.4 Project from University of Alicante (see biblio. [5]) . . . . . . 12 3 Considered technologies 14 3.1 3DEngines ................................ 14 3.1.1 Unity ............................... 14 3.1.2 UnrealEngine........................... 15 3.1.3 Cryengine ............................. 16 3.1.4 Chosentool ............................ 16 3.2 Platforms for writing this document . . . . . . . . . . . . . . . . . . . 16 3.2.1 GitHub .............................. 17 3.2.2 Overleaf.............................. 17 3.3 Development platforms . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3.1 VisualStudio ........................... 17 3.4 Programming Languages used . . . . . . . . . . . . . . . . . . . . . . 18 3.4.1 C#................................. 18 3.4.2 PHP................................ 18 3.5 Tools used for the design section . . . . . . . . . . . . . . . . . . . . . 18 3 CONTENTS 4 3.5.1 Uitoolkit ............................. 19 3.5.2 Coherence............................. 19 3.5.3 Vivox ............................... 19 3.6 Work Tool for User Story Map . . . . . . . . . . . . . . . . . . . . . . 20 3.6.1 Choosentool ........................... 21 3.7 Creationofdiagrams ........................... 22 3.7.1 Draw.io .............................. 22 3.7.2 PlantUML............................. 23 3.8 AudioTools................................ 23 3.8.1 Mumble .............................. 23 3.8.2 Vivox ............................... 24 3.8.3 ChosenTool............................ 24 3.9 User Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.9.1 XAMPP Web server and Database . . . . . . . . . . . . . . . 25 3.9.2 Unity Authentication Service . . . . . . . . . . . . . . . . . . 26 3.9.3 ChosenTool............................ 27 4 Project Planning 28 4.1 History Maps Creation Process . . . . . . . . . . . . . . . . . . . . . 28 4.1.1 Users................................ 28 4.2 Definition of User Activities . . . . . . . . . . . . . . . . . . . . . . . 29 4.3 Taskpriorities............................... 30 4.3.1 Musthave............................. 32 4.3.2 Shouldhave............................ 33 4.3.3 Couldhave ............................ 35 4.3.4 Won’thave ............................ 35 4.4 Workplan ................................. 36 4.4.1 Iterations ............................. 37 4.5 Software development method . . . . . . . . . . . . . . . . . . . . . . 44 4.6 RiskManagement............................. 44 4.6.1 Risk Identification . . . . . . . . . . . . . . . . . . . . . . . . 45 4.6.2 RiskAnalysis ........................... 46 4.6.3 RiskPlanning........................... 48 5 Implementation 52 5.1 Thescene ................................. 52 5.2 Object synchronization, with Network Manager . . . . . . . . . . . . 57 5.2.1 NetworkManager Script . . . . . . . . . . . . . . . . . . . . . 57 5.2.2 NetworkBehavior Objects . . . . . . . . . . . . . . . . . . . . 58 5.3 Relay.................................... 58 5.4 Remote Procedure Calls . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.5 UserInterface............................... 60 CONTENTS 5 5.5.1 SuperGUI............................. 60 5.5.2 UIelements ............................ 61 5.6 Translator................................. 62 5.7 Creating and joining rooms . . . . . . . . . . . . . . . . . . . . . . . 64 5.8 Audio communication using Vivox . . . . . . . . . . . . . . . . . . . . 65 5.9 Interacting with doors . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.10 Displaying media on the blackboard . . . . . . . . . . . . . . . . . . . 68 5.11 Displaying user webcam on their avatar . . . . . . . . . . . . . . . . . 70 6 Conclusions 72 6.1 Whatwewantedtodo.......................... 72 6.2 Whatweactuallydid........................... 73 6.3 Whatwedidnotdo ........................... 74 6.3.1 Character Customization . . . . . . . . . . . . . . . . . . . . . 74 6.3.2 Quick emoji reactions . . . . . . . . . . . . . . . . . . . . . . . 75 6.3.3 TextMessaging.......................... 75 6.4 Final version of the project . . . . . . . . . . . . . . . . . . . . . . . . 75 Conclusiones 78 6.4.1 Character Customization . . . . . . . . . . . . . . . . . . . . . 81 7 Work done by each member 84 7.1 Samuel................................... 84 7.2 Alexander ................................. 86 7.3 Sara .................................... 88 8 Bibliography 91 9 Appendix 93 Chapter 1: Introduction Telecommunications have evolved exponentially since the end of the last century, starting from mere, limited and often expensive telephone conversations to nowadays being able to share all types of multimedia content, both on social networks and in video calls. And all of course for free and practically in real time thanks to technologies such as 5G. However, although we have already reahed the end of the first fifth of the 21st century, an online way of communicating as natural as when in person has not yet been achieved. Current tools and platforms are not capable of recreating an environment that is interactive enough so that the event you are having in question (a virtual class, a conference, a work meeting) can take place in a completely normal way. However, in the most recent years there has been a lot of progress in this niche, mostly due to the COVID crisis, and thanks to which a lot of money has been invested for digitization. Yet there is still a long way to go. In order to give more context to this issue, we will have to go back to the beginning of the pandemic, more specifically, around mid-March, when for a few months humanity had to stay at home without being able to go out. The health system collapsed and the authorities did not know exactly how to act in such situation. Regarding the educational system, there was a great period of uncertainty because, with some exceptions, universities, colleges and institutes were places of in-person teaching. Classes were temporarily suspended, 6 CHAPTER 1. INTRODUCTION 7 causing exams to be delayed. Finally, and after observing that this crisis was going to last, classes were resumed completely online. And here is precisely the point of the matter. Adapting to the new virtual model was difficult: many professors did not have essential resources such as coverage and stable internet connection, and others uploaded prerecorded classes without taking them live, which made it impossible to ask questions in real time or ask for a concept to be explained again, had it not been understood clearly. And even in live classes, asking questions became equally difficult due to the problem of latency, which even in the best cases (when the lag would not be higher than a couple of seconds), caused teachers and students to unintentionally interrupt each other when stating these questions, and as a consequence the rhythm of the class would end up being hindered. Added to all of this there were other problems, such as screen freezing when sharing a window, or loading time of the resources used, making the quality of the classes practically zero. Then the situation improved and we moved to a hybrid model, but the problems had not ended there: universities and colleges needed to install cameras and similar infrastructure to transmit what was being written to half of the class that was staying at home that day on the blackboard. But this camera would not be oftenly able to focus correctly on the blackboard, causing it to look blurry or even let part of it go out of plane, ultimately causing important details written by the teacher not to be seen. At the moment of handing in this project, we have practically left the pandemic behind, but the use of all these technologies has not stopped, quite the opposite. What it more, and if we put everything that has been said into evidence, it is clear that virtual communication systems still have a long way to go and need to be improved and evolve, implementing functionalities that allow reducing the separation (today enormous) between in-person and remote interaction experiences. CHAPTER 1. INTRODUCTION 8 Thus, the aim of this project is at developing a desktop program that allows users to connect into a 3D model of the faculty in order to give or receive classes in an online way. More specifically, this goal could be subdivided into the following tasks: • Develop a desktop program that not only allows users to choose either to create or to join a server room, but also: – Allows users to move freely within the campus, providing access to various classrooms. – Allows users to interact with the environment, and have other users see those changes (such as opening doors, knowing the current state of the model, etc.). • Developing a desktop application that allows users to conduct or attend a class remotely. – Implementing a feature that allows users to view avatars of other participants in the same server or room, as well as tracking the attendance and identify participants in class. – Implementing an in-app communication channel for users to engage in conversations with one another, making the task of exchanging feedback and transmitting knowledge easier. – Implementing a feature that allows users to see each other through their webcams, improving the life-likeness of interaction – Implementing a feature that allows users to share content on the blackboard with others and display it while conducting a class • Implementing additional functionalities to promote a more natural interaction among users. – Implementing a feature for swift reactions, allowing users to express agree- 3.1. 3D ENGINES 15 much as possible. • Physical engine that simulates the laws of physics. This allows us to create more realistic movements and interact with the environment. •Animations. These can add more personality and vibrancy to the model. • Sounds. This functionality allows setting and developing communication between users. Some of the disadvantages found were: •None of the team members had previously worked with C#. •Sometimes the terrain creation tool is limited. 3.1.2 Unreal Engine It is an engine that allows to create games, landscapes, interactive environments or virtual reality systems with an stounding realism. Previous knowledge in C++ is required. The advantages of using it that we found were: • It has excellent tools for creating 3D environments, in addition to having great quality and power when it comes to modifying global illumination and creating shaders and materials. • We also already had previous experience with C++, the programming language over which it runs. Finally. •It has easily accessible documentation. The main disadvantage though is that it is not designed to be used by small teams in their development and some concepts, such as Actors, are more complex; as well as, in terms of performance, a computer with very good characteristics is necessary to be able to support the program when developing, since it can collapse and the use of the CPU will skyrocket. 3.2. PLATFORMS FOR WRITING THIS DOCUMENT 16 3.1.3 Cryengine It is a very powerful and versatile engine. Its most outstanding feature is the graphical capacity, being better in this aspect than Unity and similar to that of Unreal Engine. The advantages we found were: • Its interface and development tools are more complex than those of the previous two. • The documentation is less accessible and it is not recommended to use when you are inexperienced in developing interactive environments. 3.1.4 Chosen tool Finally we opted for Unity since the development with Unreal Engine seemed more complex to us as well as the aforementioned hardware limits. Furthermore, Unity can be used for free and the main model of the faculty that was provided to us had been developed in Unity, so we would not need to adapt it (what we had to adapt though was the structure of nodes and components of the faculty model itself, but this was another story). 3.2 Platforms for writing this document The program for writing this document was also important in our case, because we were a group of three people and we wanted to have the possibility of writing all the three of us at the same time in the document. Hence, here are the two possibilities that were considered: 3.3. DEVELOPMENT PLATFORMS 17 3.2.1 GitHub Our first idea was to upload this document into a GitHub repository, in order to have track of the file changes. Nonetheless, it did not allow us to work directly and concurrently on the same text file of the document. Thus, GitHub was finally used solely as a storage location and version control system not for the memory, but for the project code itself. 3.2.2 Overleaf This was another of the options that our tutors proposed to us, since it has a syntax that allows us to structure and maintain the style of the entire document in a simple way. It also allows us to work synchronously on the same document, although its free version only allows two users to work on it at the same time. On the other hand, some of our members, Samuel and Alex, already had prior knowledge on this syntax, resulting this in an additional reason for us to finally deciding to go for this latter option. 3.3 Development platforms In terms of coding, we have considered several IDEs, and the following was the final candidate: 3.3.1 Visual Studio In order to carry out the implementation we use Visual Studio and Visual Studio Code. Being an integrated development environment, we were able to use it to debug, compile, write, edit, and debug the code in order to deploy our application. Furthermore, all team members had experience using this tool. 3.4. PROGRAMMING LANGUAGES USED 18 3.4 Programming Languages used We used C# for scripting within Unity and PHP for server-side scripting. 3.4.1 C# Microsoft programming language, inspired by C and C++. Used especially for developing within the .NET platform but also in products such as Xamarin and Unity. For this reason, we use this language to be able to code the scripts in charge of achieving the necessary functionalities to carry out the project. It is an object-oriented, statically typed and multi-paradigm language. Traditionally it was focused on developing on Windows, but it was opened to other systems, making it possible to use it to develop on MacOS and Linux. Since its syntax is similar to C, C++, Java and others, we could easily adapt to because the aforementioned had been the most used languages during our studies. 3.4.2 PHP Mainly focused on server-side scripting, so it can do anything that another CGI program can do, such as collecting form data, generating pages with dynamic content, or sending and receiving cookies. Although PHP can do much more. It is designed to be embedded in HTML. 3.5 Tools used for the design section In the following section, we will describe what libraries and tools have we used in order to implement the application. This includes not only plugins but also libraries developed for unity. 3.5. TOOLS USED FOR THE DESIGN SECTION 19 3.5.1 Ui toolkit A native Unity tool, it is a collection of resources, features and tools that allow the development of user interfaces and Editor extensions. It allowed us to implement the created prototypes that can be seen in the Design chapter. It was chosen because in addition to being compatible with the engine we decided to use (Unity), it is also user-friendly. 3.5.2 Coherence The platform comes with Unity plugins, which proved to be very useful for developing the project. It allows for the creation of scalable and persistent online games regardless of the team’s size and experience level. It also enables combining modes within a single project, meaning you can have both worlds and rooms. Worlds are always-active servers with optional persistence, while rooms are sessions with optional metadata and player counts. Another advantage is the ability to build and upload the project from Unity to the Coherence cloud for accessibility. The downside is that it doesn’t include tools for developing text or audio channels; it focuses on creating the server-side service. 3.5.3 Vivox Capable of handling server functionality and integrating communication features into the model. Voice and Text Chat (Vivox) allows users to communicate orally with each other, regardless of the platform they are connecting from. Furthermore, regardless of the game engine used to create the project, it allows for seamless integration of communications without limitations on the number of connected users. 3.6. WORK TOOL FOR USER STORY MAP 20 3.6 Work Tool for User Story Map We decided that each one would try several tools in order to have different points of view to finally meet and decide which one would be chosen to create the user story map Here we list the different applications tested and the pros and cons that we have found for each of them: Feature Map The advantages that we found when testing this application were that it is very intuitive and easy to understand and use, the interface is made to avoid distractions from external elements, and it also allows you to work synchronously online with the rest of the members. The drawbacks we found were that in this application, the user who can perform the user story correctly cannot be assigned and after 15 days, only a limited version of the application can be accessed, since the trial period ends. Avion Some of the benefits we found when using this application were that it is quite easy to use, the cards are numbered automatically and it allows you to add arrows to make the flow easier to read. In the drawbacks we saw that it can only be used after the 15-day free trial if the premium service is purchased, which makes it not very accessible. We have contacted the sales department to ask if it is possible to obtain a student license but have not received a response so far. Miro We found this application to be very flexible, it has very useful features such as the stopwatch and it also allows you to vote. It would be very useful in meetings that require a general brainstorming of the project but it is not a specific application to create user maps since it focuses on creating schematics and diagrams. 3.6. WORK TOOL FOR USER STORY MAP 21 Mural One of its best advantages is that it is completely free-to-use and you can create very attractive results, but on the other hand it is very complicated to create user maps in this application. Carboard it! In Cardboard it! you can create a user map quickly and collaboratively but it is not possible to assign user types and it becomes paid after the 15-day trial period has passed. StoriesOnBoard In this application you can add and relocate new cards quickly and easily, but you cannot assign user types and they do not have a free version after the trial days. 3.6.1 Choosen tool In order to be able to decide on the tool in question, each of us had listed the drawbacks and benefits of each of the applications we had tested. We established criteria to choose which would be the tool we would use from that moment on. The factors that had the most influence on our decision were: •How friendly the tool is with the user. •Its availability for usage. The time to learn how to use these tools was limited, so we wanted to use one with the smallest learning curve possible so we could start working with it as soon as possible. In this section we highlight the FeatureMap, Avion and Miro applications. Even if a tool is good, it is not worth starting to work with it if after the trial period we cannot use it again, which would delay the workflow, so we wanted to use an application 3.7. CREATION OF DIAGRAMS 22 that was free or that at least gave us allowed it to retain much of its functionality for free. Some that met this criteria were Mural, which is free, and FeatureMap, which maintains many of its features after the trial period. These were the top-rated apps. Avion, although we found it to be quite complete, was handicapped by the impossibility of accessing its functions for free once the trial period ended. The team’s final decision was to use FeatureMap, since it is very intuitive and easy to use, since you can start using it immediately even if you have no prior knowledge. Additionally, this application has the possibility for the team to work simultaneously on the same map. The biggest disadvantage we found was the fact that the full version is free only for a limited time (2 weeks), but after the trial time passes there is still a free version although with restricted functionalities, so it seemed to us an acceptable inconvenience. 3.7 Creation of diagrams We generated component diagrams to elucidate the structure and organization of our system, offering insights into the interrelationships among its components. Furthermore, we generated flowcharts to grasp the logic of key functionalities, facilitating the visualization and design of information flow within our system. The tools considered for their creation were the wollowing: 3.7.1 Draw.io By having the ability to create a wide variety of diagram types, we could address different representation needs. Additionally, the tool enables the export of diagrams in various formats for inclusion in this report. The accessibility from any location with an internet connection not only facilitates collaboration but also supports remote work 3.8. AUDIO TOOLS 23 3.7.2 PlantUML Facilitating the automatic creation of diagrams by describing code in a textual format, it provides versatility. Its simple and readable syntax eases the updating process of diagrams as the code evolves, ensuring consistency between the documentation outlined in this document and the implemented code. 3.8 Audio Tools Establishing oral communication within our 3D environment involves incorporating a tool or library that enables functionalities such as audio channels. This will allow users to receive and send sound so that they can listen to each other. Two potential options for accomplishing this include Vivox and Mumble, and we will explore each of them more thoroughly in the following analysis. 3.8.1 Mumble It is an open-source project, and when combined with Unity, we can use it to achieve a more immersive experience. It employs an audio codec called Opus, providing exceptional sound quality and low latency. Opus was designed to minimize the delay between voice message transmission and reception. Some advantages of using this tool include operating on a client-server model, allowing us to set up our own Mumble server for full control over the communication environment, including the creation of channels and private rooms. It is compatible with Windows, Linux, and macOS. The MumbleLink protocol facilitates communication between Mumble and other external applications, which we would need for integrating Mumble with Unity. The MumbleLink API allows us to retrieve information, such as the user’s position in 3.8. AUDIO TOOLS 24 3D space or the orientation of their head. In virtual reality environments, we have the capability to synchronize the user’s head movement with the avatar’s orientation in Mumble. This tool further facilitates lip animation in the 3D model to simulate the user speaking during voice chat. It’s worth noting that, in our specific case, this feature may not be necessary since we utilize a webcam image of the user (discussed later). Moreover, the MumbleLink API supports spatial sound effects, enabling adjustments to volume or sound position based on the user’s location in the model. Additionally, visual effects, such as animations or color changes, can be displayed contingent on whether the user is actively speaking or not. 3.8.2 Vivox Vivox is a voice and text communication platform specifically designed for online applications and virtual environments. It enables seamless integration of real-time voice and text chat functionalities, providing a user-friendly application programming interface. Additionally, it offers positional audio, and is easy to integrate with VR headsets. Compatible with multiple platforms, it enables the creation and management of personalized voice channels to cater to our application’s requirements. Designed to handle large numbers of users simultaneously, it ensures the feasibility of supporting the simultaneous connection of all students and professorate in the university. 3.8.3 Chosen Tool We chose Vivox because, in addition to the previously mentioned advantages, we found clear and concise documentation that facilitated its integration with Unity. Since we did not require special effects such as lip animation, we felt more confident using this 4.3. TASK PRIORITIES 31 project overview is needed, unlike MoSCoW, which allowed us to evaluate each card independently. Figure 4.1: UserStoryMap 4.3. TASK PRIORITIES 32 The requirements for task prioritization were finally grouped into four categories, corresponding to the uppercase letters in its name. We will now elaborate on each of them now, since we will explain them in further detail in the following chapters: 4.3.1 Must have These are the functionalities that belong to the walking skeleton. For must-haves, we have selected tasks related to our differentiating element. •Render and display the 3D model of the faculty –Codename: US_SHOW_MODEL –Description: As a user I want to be able to see the model of the faculty. •Move freely around the faculty –Codename: US_MOVE – Description: As a user I want to be able to move and roam freely according with real world-physics laws, in order to access different places from the faculty model. •See other users’ avatars –Codename: US_SEE_AV – Description: As a user I want to be able to see other people’s avatars in order to interact with other users in a more natural way. •Communicate with other users –Codename: US_COMM – Description: As a user I want to be able to communicate with other users by means of either a text or audio chat, in order to let my message reach other users and receive feedback from them. 4.3. TASK PRIORITIES 33 •Share content with other Users –Codename: US_SH_CONTENT – Description: As a user I want to be able to share content with the rest of the class, such as playing a video or showing a series of pictures or screenshots. •Disconnect –Codename: US_DICONNECT –Description: As a user I want to be able to close the application. •To watch shared files in the blackboard –Codename: US_WT_CONTENT – Description: As a user I want to be able to watch the files shared in the blackboard in order to get knowledge or verify that is the correct content. 4.3.2 Should have The tasks we have categorized as should-haves are those that users would expect to have. They are not essential to launch the product, but they would add value to the application, improving the user experience on the site. Here, we include functionalities like. Being these add avatar image, teaching and voice chat. •Create a room –Codename: US_CR_ROOM – Description: As a user I want to create a room in order to give classes in it. •Adjust the room size –Codename: US_LM_ROOM –Description: As a user I want to be able to specify the room size. 4.3. TASK PRIORITIES 34 •Join room –Codename: US_JN_ROOM – Description: As a user I want to be able to join a room with a specific code. •See other users’ webcam –Codename: US_SEE_WCM – Description: As a user I would like to see the rest of participants’ webcam. •See my own changes in environment –Codename: US_SEE_USCH_1 – Description: As a user I would like to see the changes that my own avatar applies on the environment, such as opening or closing doors, for instance. •See other users’ changes in environment –Codename: US_SEE_USCH_2 – Description: As a user I would like to see the changes that the rest of participants apply on the environment. •Change the language –Codename: US_CH_LAN – Description: As a user I would like to be able to change the language at any time. •View Usernames –Codename: US_SEE_NAME – As a user, I want to be able to see the usernames of other users to identify them 4.3. TASK PRIORITIES 35 4.3.3 Could have In the could-have category, we have user stories that we consider interesting ideas but we think that implementing them would not be a high priority to add value to the user, or are not yet sure if we can implement them. Being these modify character, chair interaction quick reactions with emojis and manage account. show assigned classrooms, show schedule, text chat, show resources and submit and get assigments. •Show the time –Codename: US_SH_TIME – Description: As a user, I want to be able to see the current time within the application to effectively manage my schedule. •Interact with the doors of the faculty. –Codename: US_DOOR_INT – Description: As a user, I want to be able to interact with the doors of the faculty to access different areas within it. •To erase blackboard content. –Codename: US_ERASE_BB – Description: As a user I want to be able to remove the content that is in the blackboard in order to get an empty blackboard for next class. 4.3.4 Won’t have Finally, the user stories we have placed in “won’t-have” are those that, we find interesting but due to lack of time or complexity, we have decided to discard them. Perhaps, in the future, they could be reconsidered. •React with emoji 4.4. WORKPLAN 36 –Codename: US_EMOJI – Description: As a user I want to react with emoji in order to let the professor know either whether I agree with them or not, or whether I have any doubts. •Connection with virtual reality headset –Codename: US_CONNECTION_VR – Description: As a user I want to connect whit virtual reality headset into the model for interact in other possible ways with it. Achieve a more immersive experience. •Customizing an Avatar. –Codename: US_CUSTOM_A – Description: As a user, I want to customize my avatar to distinguish myself from other users 4.4 Workplan To successfully achieve the objectives described in the first chapter, a series of iterations took place, in which we held an online meeting every few weeks to monitor the status of the project, see if the members had questions or needed assistance with any of the tasks that were assigned to them, prioritizing the tasks. Nevertheless, during the academic year 2022-23 we established longer milestones since we had a specially busy schedule in terms of subjects and other projects, while shorter yet bigger milestones from June onward. In order to have a comprehensive overview of what needed to be done in each iteration, we employed a Kanban board. This allowed us to organize tasks and monitor their status throughout the development process. The board consisted of four columns representing different stages of the workflow. 4.4. WORKPLAN 37 In the first column, “To Do”, all tasks slated for the iteration were initially placed. These tasks were prioritized to ensure that those at the top were selected first when assigning tasks. Each team member could only be assigned one task at a time, and each task could have more than one assigned team member. The second column, “Doing”, housed tasks that were actively in progress. The third column, “Ready for Review”, served to indicate that tasks in the “Doing” column were ready for evaluation once completed. The entire team conducted the review to assess whether the tasks met the expected standards. Upon successful completion of the review, the task transitioned to the “Done” column. If revisions were deemed necessary during the review, the task returned to the “Doing” column. This type of board proved highly beneficial as it facilitated the assignment of tasks, tracking of tasks in progress, and provided a high-level overview of the ongoing work and the achievement of objectives. 4.4.1 Iterations Iteration 1: Creation of the repository and importing the 3D model of the faculty In this iteration, the tools that we would use to write the report were contrasted and chosen. Additionally, we created a GitHub repository with a unity project where we imported the 3D model of the faculty (see US_SHOW_MODEL). •Start date: 03/10/2022 •End date: 01/01/2023 Iteration 2: Main character creation and first steps with physics In this iteration, the basic model of the character was designed and research was carried out on how to perform movement and camera changes correctly (see User 4.4. WORKPLAN 38 Story US_MOVE). •Start date: 01/01/2023 •End date: 06/03/2023 Iteration 3: Interaction with the environment and start of server development In this interaction we managed to interact with doors, to be able to enter the classrooms and we also began to test and investigate the functionalities provided by Coherence in order to have the server working as soon as possible. This system would be later replaced with theNetwork Manager object, using Relay as the transport layer (see User Stories US_SEE_USCH_1,US_SEE_USCH_2 and US_DOOR_INT). •Start date: 06/03/2023 •End date: 04/06/2023 Refactor: Modularization of the faculty model and issues with repository version We slightly reorganized the structure of the faculty model. We also took the time to add colliders to both walls, ceilings, floors and other big decorative elements, such as lockers or doors. On the other hand, we were having trouble when trying to synchronize our code because just opening the project would add thousands of “changed files” staged for commit and constantly giving merge conflicts, making working asynchronously very difficult. In the end we discovered that the unity version control had been activated, and there was some weird interaction from having both the unity version control and GitHub tracking changes at the same time. When trying to remove the Unity version control, we would not be able to commit the change due to said merge conflict. We 4.4. WORKPLAN 39 ended up deciding to create a new repository and moving our progress to it, but it took some time. •Start date: 10/07/2023 •End date: 07/08/2023 Iteration 4: reaction system attempt In this iteration, an attempt was made to add the reaction functionality, consisting of being able to show in the model a thumbs up if one agreed, a thumbs down if one disagreed, and a question mark if one had doubts when pressing the 1, 2, and 3 keys on the keyboard respectively. Finally it was decided to postpone its implementation (see US_EMOJI). •Start date: 07/08/2023 •End date: 21/08/2023 Iteration 5: Testing and creating a separate repository to Integrate Relay in order to synchronize objects throughout the network Before adding to the project containing the faculty model, a separate repository was used to test and implement the code related to the client/server model (the demo project consisted on a stage with a flat platform and the character were represented as cubes (see User Story US_SEE_AV). •Start date: 21/08/2023 •End date: 28/08/2023 4.4. WORKPLAN 40 Iteration 6: Creation of the database model and choice of the tool for its development. During this iteration we discussed how the login system should be implemented. Several approaches were proposed, such as using XAMPP database services (more on this in the Implementation chapter), but our final decision was to use Unity Native Authentication System, by means of anonymous host and clients, that would join the former one’s room with a code (see User Stories US_CR_ROOM, US_LM_ROOM, US_JN_ROOM). •Start date: 28/08/2023 •End date: 11/09/2023 Iteration 7: Creation of a user interface prototype and choice of the tool for its development In this iteration several prototypes of the user interface were created, there were quite a few changes since it took us a long time to decide what elements there would be. You can see the final results in detail in the Design chapter. Finally we decided that the join code and the group size should be shown on the top left, in addition to a drop-down menu that could allow the user to change the language (see User Story US_SH_TIME). What’s more, we implement the option to write a username and then display it to the rest of the users (see US_SEE_NAME). •Start date: 11/09/2023 •End date: 25/09/2023 Iteration 8: Share image by rendering what a webcam displays. To facilitate non-verbal communication and enable educators to evaluate students’ attentiveness, our emphasis in this iteration was on presenting the image captured 4.6. RISK MANAGEMENT 47 Table 4.1: Risk Analysis Table Risk Probability Impact Lack of experience can lead to delays in completing the work and may result in incorrect execution, worsening performance, and increasing effort. It is a project-related risk. High High Unfamiliarity with development tools also poses a risk to the project as unforeseen issues may arise. It is a project-related risk. High Critical Lack of communication among team members can result in suboptimal outcomes, potentially requiring function modifications, leading to time loss and increased effort. It is a project-related risk. Possible Minor If a team member falls ill, it may necessitate a reorganization of assigned tasks by the rest of the team; otherwise, progress may be hindered. It is a project-related risk. Unlikely Medium Unforeseen events may cause task development to slow down or be postponed, preventing the proper development of the project. It is a project-related risk. High Critical Team members may lack clarity about the goals of a specific functionality, leading to excessive changes in user stories, resulting in poor functionality and dissatisfaction. It is a technical risk. Possible High The loss of a team member, should they decide to leave the project for significant reasons, would make it more challenging to continue with the project. It is a business-related risk. Unlikely Medium The lack of compatibility among different software components used for the project can lead to system failures, resulting in a loss of quality and diversity. It is a technical risk. Possible High 4.6. RISK MANAGEMENT 48 4.6.3 Risk Planning To plan for the various risks, we employ a proactive risk management strategy as we prefer to apply it before the risk occurs. This allows us to avoid any issues before they become too problematic to resolve, or at the very least, to minimize their impact. Each risk has three action plans: avoidance, which includes measures to reduce or eliminate the possibility of the risk occurring; monitoring, where we take measures to keep an eye on the risk and take action before it’s too late; and management, which outlines what will be done if it’s not possible to avoid the risk and measures need to be taken to mitigate it. The following table lists each of the risks along with the specific avoidance, monitoring, and management plans that will be employed for each one. Table 4.2: Risk Planning Table. Risk Action Plan First Avoidance Team members will seek tutorials or literature to acquire the necessary knowledge for project development. Those who gain greater expertise in the language and environments we are using will assist and provide guidance to other team members before the project begins, making the team more efficient. Monitoring Regularly check if any team member is struggling to complete a specific functionality efficiently and provide assistance and training to build experience. Verify later if the assistance was effective. Continued on next page 4.6. RISK MANAGEMENT 49 Table 4.2 – continued from previous page Risk Action Plan Management If the risk materializes, conduct one or more meetings to help affected members and, if necessary, redistribute workload among other team members to prevent delays. Second Avoidance Same as the previous risk. Monitoring Same as the previous risk. Management Same as the previous risk. Third Avoidance Implement Agile planning, which adapts well to changes and includes one to two-week iterations. Monitoring Conduct regular meetings and maintain direct contact among team members. Management In the event of occurrence, increase the frequency of project status monitoring. Fourth Avoidance This risk is uncontrollable due to natural causes and cannot be avoided. Monitoring If any team member experiences potential illness symptoms, they will inform the rest of the team to be prepared for task redistribution. Management If the risk occurs, to reduce its probability, team members will temporarily take over the tasks of the affected member while they recover. Risk Action Plan Continued on next page 4.6. RISK MANAGEMENT 50 Table 4.2 – continued from previous page Risk Action Plan Fifth Avoidance Unforeseen events cannot be avoided. Monitoring Team members facing task completion challenges should inform the rest of the team. Management Depending on the task’s priority, either reassign or postpone it until the risk is resolved or minimized. Sixth Avoidance To reduce the likelihood of this risk, agreements will be reached during the planning meeting to clarify the objectives of each task. Additional meetings will be scheduled if necessary to ensure clarity. Monitoring If unclear indications are observed, plan a team meeting to establish the objectives for that functionality. Management In case the risk materializes, halt the development of that functionality, define its objectives, and begin implementing it again. Seventh Avoidance Maintain close communication with team members to ensure their commitment to the project. Monitoring During meetings, check for members who may have become distant, made less progress than expected, or failed to attend. Address their situation to motivate them if they are considering leaving the project. Continued on next page 4.6. RISK MANAGEMENT 51 Table 4.2 – continued from previous page Risk Action Plan Management If a member ultimately leaves the team, conduct a meeting as soon as possible among all members to reassign roles if necessary and reorganize tasks based on priority. Eighth Avoidance Attempt to investigate compatibility between different development tools that may be used. Monitoring During the sprint development, ensure that all team members are using tested tools. If new features are needed, verify if the current tools can accommodate these needs. Management If the risk becomes real, a team member will search for a tool to bridge the incompatible ones. If this is not possible, the new incompatible tool will be discarded, and an alternative solution will be sought. Finally, three of the identified risks materialized, numbered one, two, and three. However, by following the contingency plan, they did not pose a serious problem and could be rectified, allowing the project to proceed correctly. Chapter 5: Implementation This chapter is focused on describing how each of the functionalities has been implemented in the application. First, we start by describing how the project is structured and what elements constitute it. Afterwards, we go over each of these elements in detail, describing what they are used for, what components are involved for each of them, and how these components interact with each other. Finally, we also show the execution flow for each functionality that has been implemented using these tools. 5.1 The scene All of the objects in our project are encapsulated in one single unity scene called EscenaFacultad.unity (see the component diagram in figure 5.1 and the hierarchy in the editor in figure 5.2). First off, we have the Facultad object, which is parent to all of the 3D objects that are part of the model before any players connect. Under it there are many different assets, such as: windows, tables, chairs and so on, which are always fixed in the same place, and therefore no changes to these objects will ever be applied and hence no need to be synchronized over the network. Due to this, each client has a local copy of these objects in their machine, reducing significantly the traffic in the network when a user connects. Most of these objects also have colliders, which limit the movement of the player. 52 5.1. THE SCENE 53 Figure 5.1: Hierarchy of the main objects in the scene, shown in a component diagram 5.1. THE SCENE 54 Figure 5.2: Hierarchy of the root objects in the scene, shown in the unity editor Inside the Facultad model, there are also blackboards. Blackboards are composed of three main objects, the Prop, which is the 3D asset that actually looks like a physical blackboard; the Plane, which is the surface on which the images and videos are shown in; and the Blackboard object, the invisible rectangle that defines the interactable area for the blackboard, and which is the parent of the previous two. Inside the Blackboard object, there is the Blackboard script, which has references to the other child components of the blackboard, and also defines the logic to be executed when a user interacts with it. Inside the Plane there is the VideoWithAudio script, that takes care of playing video files and showing them on the Plane object. Lastly, in the Facultad, there are also doors. Since doors can change their rotation while the application is running (for instance, a user can open or close a door), their state needs to be synchronized over the network (so that if a user opens the door, it appears open for the rest of the users). Due to this, it contains a NetworkObject script and a NetworkTransform script. We will go over what each does later, but in 5.1. THE SCENE 55 short, they indicate that this object needs to be synchronized throughout the network, is owned by the server, and what properties need to be synchronized. Next up, we have the Player prefab, which is spawned when each user connects to the room. Each player can move around the model, and must therefore be synchronized throughout the network as well. For this sake, it has the NetworkObject script and the ClientNetworkTransform. Similarly to the door, these indicates that the object needs to be synchronized over the network, and which properties need to be synchronized. The difference here is that the player uses a ClientNetworkTranform instead of NetworkTranform to indicate that each client is owner of their own Player instance. The player also has a PlayerNetwork script that is in charge of initializing the object in the local machine when it is spawned (such as setting the name, or creating its controller when their own character is spawned). The PlayerNetwork script also contains the RPC methods available to each player. They are used to request the server to run some logic for which the player does not have direct ownership of, such as opening a door (more on this later), or to broadcast some event, such as updating a player’s webcam The PlayerControllers, when created, are put under the Controllers object, and have a reference to the Player they have control of. To split the responsability, the logic has been split into PlayerMovementController, which is in charge of moving the player object and the MainCamera; and the PlayerInteractionController, which is in charge of checking if the player is looking at an interactable object, and calling the interact method of that object. In the scene, there is also the MainCamera. As indicated before, the MainCamera is moved around the model by the PlayerMovementController, and shows the player point of view on screen. Additionally, it also renders an overlay of the UI defined in 5.1. THE SCENE 56 Figure 5.3: Debug console in the scene UIDocument and Canvas. Regarding the UI, it is split into two parts: the Canvas, which contains the debug console shown in figure 5.3 (accessible by pressing F11) and is useful for understanding better what is happening in the background (specially when having more than one client connected to the room, since the unity editor console only shows logs for one client); and the UIDocument, which defines the rest of the UI elements, such as the starting screen, and the interaction messages. The scripts MainGui,ConnectBoxManager and InfoBoxManager define what each button does, and are in charge of showing, updating and hiding elements of the UI. The NetworkManager object is in charge of instantiating network objects and synchronizing them over the network. It has two scripts: NetworkManager script, which contains a reference to the player prefab (so that it is instantiated each time a user connects), a list of all the network prefabs (to broadcast changes made to these 5.6. TRANSLATOR 63 mentioned list to be updated with the respective new values. Finally, the Translator would initialize this variable with the current translation. Except this was not possible to be done. At all. At least not the way we wanted: in short, since strings are primitive types in C# neither there is an (easy 1 ) pointer treatment in C#, we cannot save references to the text of an object, so as to update their respective value to the new chosen language on runtime. That is to say, unless we worked it out, we could only set a certain language at the beginning of the execution, when all the texts are initialized. In order to solve this issue we considered several options, from wrapping the texts in a custom class and letting the list inside Translator class be of this type (so that we could save a reference to them), to even trying to keep track of the pointers to all those variables. But none of those approaches worked in the end: the former one would not work since in C# the string type is not a class in contrast with languages like Java for instance, and hence we could not wrap it; the latter one was simply impossible to achieve since, once again, there is no easy pointer treatment in C#. The solution, though, relied on a variant of the initial approach, which was not ideal in any case but happened to work perfectly: instead of registering the texts into the Translator listeners array, we would register the entire class object in which those texts are treated (see figure 5.6). This way, whenever a new language is loaded, the Translator would call all the registered elements within its list and tell them to update their texts on their own (see figure 5.9). Inside this updateTexts() procedure, the object would manually call another method from the Translator class to “internationalize” the different strings (and in a display of originality we called it _INTL()). The translator would simply retrieve the current translation for the provided key. But for this to work, we had to create a small interface to make it work. This interface, is the 1 In order to be able to manipulate pointers in C# you have to set a compiler flag to allow unsafe code, and then mark a method, class, or code block as unsafe 5.7. CREATING AND JOINING ROOMS 64 aforementioned ITranslatable interface. Here is the sequence diagram of how translation system works: Figure 5.9: Updating texts to the current language 5.7 Creating and joining rooms Upon starting the program, the user is presented with two options: they can either create a room (specifying the maximum amount of users that can connect to it) and connect as a host; or connect as a client by providing the room code of the room they wish to join. To explain the implementation of room creation, we will go over the flow of the first option. Initially, the user selects the room’s capacity, indicating the maximum number of users that can be connected simultaneously. This value will be displayed in a label, and the user will also be able to modify it by means of a slider, from which its value will be extracted. Subsequently, the connect as host button is pressed. The click event is triggered, calling the connectHost() method of the ConnectBoxManager class. In this function, the roomSize value is retrieved from the label and passed to the createRelay() method of the TestRelay class. From this class, the createAllo- 5.8. AUDIO COMMUNICATION USING VIVOX 65 cation method of the relayService class is executed, passing roomSize as well. RelayService is responsible for requesting the unity servers to create and reserve an allocation. Upon execution of this method, the UnityServers class is utilized to reserve a portion of the Unity server’s memory for the room using the reserve method of the AllocatedRelayServer class. This AllocatedRelayServer method returns the IP address and port to be used to connect to the allocated space for the room. This information is passed from UnityServers to RelayService, eventually reaching TestRelay. From there, the getJoinCodeAsync method of the RelayService class is executed, responsible for retrieving the code to display and share with other users so they can connect as clients to the newly created room. To retrieve this code, the getJoinCodeAsync method is executed, providing the allocationId to recover information about the created room from the RelayService class. From RelayService, the GetJoinCodeAsync method of the UnityServers class is executed, also providing the allocationId, thus obtaining the joinCode and returning it through this flow to the TestRelay class. Subsequently, the SetRelayData method is called from TestRelay, with the IP address and port as parameters from the UnityTransport class. This allows the UnityTransport class to have information about the created room, as it is responsible for determining to whom the information should be sent and how. Finally, to connect to the room, the startHost method of the NetworkManager class is executed from TestRelay, and the code is displayed on the screen through a textbox located in MainGUI.uxml. 5.8 Audio communication using Vivox Vivox is the system that handles the audio channels and definitely all that is related to communication via audio. It all starts respectively in the methods createRelay() 5.9. INTERACTING WITH DOORS 66 and joinRelay() (depending whether the user is creating a room or joining it) from the class TestRelay. In these methods, we assign the given join code as the name of the room. Next, the player enters this room with respectively NetworkManager.Singleton.StartHost() or NetworkManager.Singleton.StartClient(), and the NetworkManager finally spawns the player (which has the Player Network component), initializing all the necessary things when this player connects to the room. After this and in the machine of the recently connected user, the Vivox Service is initialized (VivoxService.Initialize()), the Vivox System signs the player into the audio channel (VivoxService.SignIntoVivox), and the OnUserLoggedIn callback is registered after the VivoxPlayer has been created. VivoxManager will then use all the unity credentials provided in its scene component in order to start the session, and finally the callback OnUserLoggedIn will be triggered, where the user finally joins the audio channel, specifying the name of this channel, that it is a positional channel, and that it is an only-audio channel (it could also be configured it as audio-and-text channel as well). On the other hand, the source point of the player audio is updated in the scene every 0.5 seconds in order to have it up-to-date with the player, using the current position of the player Main Camera (not to be confused with the Webcam). 5.9 Interacting with doors To interact with a door (see diagram in figure 5.10), the user first moves their avatar and camera so that it is facing a nearby door. When that happens, the PlayerInteractionController detects that there is an object in front of the player that implements the IInteraction and calls its interact() method. This method is implemented in DoorInteraction, which calls InfoBoxManager to show how the 5.9. INTERACTING WITH DOORS 67 text on how to interact with the door in the UI (see figure 5.11). After that, if the user presses E, to open or close the door, then it invokes toggleDoorServerRPC on the PlayerNetwork script. Since that method is annotated with ServerRPC, a request is sent through the network to the host machine, where the code inside of it is run there. That method calls toggleDoor on the DoorInteractionScript (on the host machine) and it opens the door locally. The change in rotation is then broadcasted to the rest of the clients by the mechanism described in the network manager section. Figure 5.10: Flow diagram of how to open/close a door 5.10. DISPLAYING MEDIA ON THE BLACKBOARD 68 Figure 5.11: Informational text on how to interact with a door 5.10 Displaying media on the blackboard To display media on the blackboard (see figures 5.12, 5.13, 5.14), the user first moves their avatar and camera so that it is facing the blackboard. When that happens, the PlayerInteractionController detects that there is an object in front of the player that implements the IInteraction and calls its interact() method. This method is implemented in Blackboard, which calls InfoBoxManager to show how the text on how to interact with the blackboard in the UI (see figure 5.15). After that, if the user presses “E”, FileBrowserUpdate is called, to open a pop-up file browser window. Once the user finishes selecting the files to display, the callback DisplayOnBlackboard in the Blackboard script is called, passing the selected paths as a parameter. If the file to be displayed is an image, it will call the UnityWebRequestTexture to download the texture of the image and then the texture of the plane object to that. Id the file to be displayed is a video, it will delegate it to VideoWithAudio, which will in turn set and play the new file in VideoPlayer. For playing or resuming a video, when the user presses P, the Play or Paused method (depending of if the current video is playing or not) will be called, in VideoWithAudio, and later delegated to VideoPlayer. 5.10. DISPLAYING MEDIA ON THE BLACKBOARD 69 Figure 5.12: Flow diagram of how to interact with the blackboard Figure 5.13: When the user want to select a file, a pop up file browser is opened 5.11. DISPLAYING USER WEBCAM ON THEIR AVATAR 70 Figure 5.14: After selecting the file, it’s contents are shown in the blackboard Figure 5.15: Informational text on how to interact with a door 5.11 Displaying user webcam on their avatar Inside the player model there is the FaceCam object. This is a floating rectangle in front of the avatar’s head. The texture of this rectangle is where the webcam image is displayed. The FaceCam object has attched to it the GetCameraImage script that is in 5.11. DISPLAYING USER WEBCAM ON THEIR AVATAR 71 charge of changing the texture of the rectangle with the current image of the webcam. Inside the GetCameraImage, there is a WebCamTexture, which takes the first camera device connected to the machine and record the user, loading it into its texture. Every certain time (low frequency to avoid saturating the network) the webcam texture is read, converting it into a Texture2D and then into a byte array of a JPG to serialize the image. Afterwards, this image is passed to the updateWebcamServerRpc method on PlayerNetwork. Since this is a serverRPC call, this data is sent over the network to the host machine. Then, this data is passed to the updateWebcamClientRpc method, since that method is a clientRPC call, this data is broadcasted over the network, and run on each machine of each user connected. (Note that we needed to call the serverRPC because non-host clients are not able to call clientRPC methods). On each machine, the GetCameraImage component of the corresponding FaceCam is passed the image in the method setWebcamTexture. Inside of it, the image is deserialized, and put as the texture of the webcam. One thing of note here is that, since we are sending each frame as images (no inter-frame compression), we needed to have the image as lightweight as possible or else the network can’t handle it, so the resolution is lowered as much as possible and the framerate is very reduced. For this reason, sharing the webcam over the network is only present on the "experimental" branch, instead of the "main" branch. Chapter 6: Conclusions In this chapter we briefly discuss our initial expectations, contrasting them against what we actually ended up doing, and what we left out of the scope of the project, either due to lack of time, or any other complication. Nevertheless, most of the aspects included in this chapter are explained in further detail in the chapters devoted for Considered Technologies,Implementation and Future Work, respectively, hence we do not overly elaborate in this one. Finally, we include our conclusions and future work at the end of the chapter. 6.1 What we wanted to do The initial idea of this project was to come up with a way of letting students and professors interact with each other virtually, but in a more natural way. The concept included the possibility of sharing files among the class, not only from the teacher to the students but in between them, too. Meanwhile, on the aesthetics side of things, the players themselves would have had the possibility of customizing their own avatar, changing the clothes and their physical aspect. This was important if we did not want all our avatars to look exactly the same: a simple dummy made of a sphere and a tweaked pill. Other minor details, such as the possibility of opening a floating window showing a copy of the blackboard screen (in case the student is far from the front of class), were also initially considered. 72 6.4. FINAL VERSION OF THE PROJECT 79 misma, por ejemplo), también fueron consideradas. Nuestro concepto inicial también tenía en cuenta la posibilidad de dotar a los participantes de la capacidad de habilitar o inhabilitar diferentes características del programa dependiendo del tipo de usuario que fueran. Esto implicaría que los profesores tendrían la oportunidad de cambiar los vídeos e imágenes en la pizarra mientras que los estudiantes no, o por ejemplo que los profesores tuvieran permisos para silenciar a uno o varios participantes si lo necesitaran oportuno. Estos son tan solo unos pocos ejemplos considerados. Por otra parte, la aplicación también permitiría a los usuarios controlar el volumen de otros participantes (es decir, en su propia máquina), aparte de tener un sonido espacial para cada uno de ellos (es decir, que el audio de cada persona se reprodujera más alto o bajo —¡Y también en cuanto a Panning!— en función de lo lejos o cerca que estén del usuario). Además, el programa incluiría un chat para permitir una mayor interacción con el resto de la clase evitando al mismo tiempo interrumpir o molestar al profesor. Finalmente, todos los usuarios tendrían la posibilidad the mostrar su webcam en frente de su avatar con la intención de hacer la interacción más personal y, literalmente hablando, «cara a cara». Adicionalmente y de cara a mejorar la experiencia inmersiva, los «jugadores» tendrían también la posibilidad de interactuar con objetos de su entorno, tales como puertas. Lo que realmente hicimos Cuando hicimos estas primeras sesiones de lluvia de ideas, no nos contuvimos a la hora de proponer todo tipo de ideas creativas; sin embargo no pudimos conseguir 6.4. FINAL VERSION OF THE PROJECT 80 implementar la lista entera de todas esas ideas que teníamos en mente en un principio. Por tanto, en esta sección hacemos un resumen de lo que hemos sido capaces de implementar, desde la tarea más básica y crucial hasta la menos prioritaria de nuestra lista: • Implementar un entorno virtual donde dos o más personas puedieran interactuar entre ellas por medio de avatares. • Añadir canales de voz para permitir a los usuarios hablar entre ellos directamente dentro de la aplicación, usando chat de audio por proximidad que tenga en cuenta la distancia al hablante y la dirección a este. • Añadir la webcam a los avatares de forma que comprendan mejor las reacciones de otros participantes e interactúen con ellos de manera más natural. • Añadir la posibilidad de mostrar imágenes o vídeos en un plano dentro del entrono virtual de manera local y posteriormente permitir que el resto también vea estos cambios (un plano que luego constituiría la pizarra). • Implementar un sistema de traducción de manera que permita a personas de diferentes lugares interactuar con la app de manera sencilla. • Permitir a los usuarios que se autentifiquen como profesores o como estudiantes. • Permitir a los usuarios interactuar con puertas y pizarras (con independencia del tipo de usuario que sean). Lo que no hicimos In this section we enumerate a list of tasks that were initially planned to be done, but due to various reasons we ended up not implementing. This is why we have decided to differentiate these from the future work list. 6.4. FINAL VERSION OF THE PROJECT 81 6.4.1 Character Customization Una posibilidad de arreglar la (falta de) personalización de los avatares habría podido ser a través de un framework llamado Ready Player Me, el cual permite crear, personalizar y vestir tu propio avatar y añadirlo a la aplicación. El framework parece ser 100% compatible con Unity y también viene con la posibilidad de personalizar el avatar fuera del juego, por medio de una web, pero consideramos esto finalmente menos esencial que otras características. Reacciones rápidas con Emoji Intentamos añadir la posibilidad de mostrar un emoji o símbolo encima de los avatares, indicando al profesor expresiones como estar de acuerdo o no, estar intersado o no, estar aburrido, etc. Esto permitiría la expresión de opiniones sin la necesidad de escribir o hablar. Este enfoque facilitaría una comunicación eficiente, especialmente cuando el profesor pregunta si su discurso está siendo lo suficientemente claro, evitando respuestas múltiples solapadas. Sin embargo, a la hora de explorar la forma de mostrar texturas u objetos encima o delante del personaje, consideramos otras funcionalidades más prioritarias. Finalmente concluimos que optimizar nuestro tiempo y esfuerzo era crucial, lo que nos llevó a relegar esta funcionalidad a un segundo plano. Finalmente, esta característica no fue implementada. Mensajes de texto A la hora de investigar todo lo relacionado con la comunicación, propusimos una solución que cubriría tanto los comunicación escrita como oral. Finalmente, consideramos más relevante establecer un canal de voz, ya que esto haría que la experiencia en el modelo de la facultad fuera más parecido a aquel en una clase físical y real. Por tanto diferimos la implementación del canal de texto a futuras iteraciones para finalmente redirigir nuestros esfuerzos a otras tareas. 6.4. FINAL VERSION OF THE PROJECT 82 Versión final de nuestro proyecto La versión final del proyecto consiste en un programa que permite a diferentes personas unirse a una misma clase en un servidor para dar y recibir clases de manera más inmersiva. Los avatares tienen un plano flotando delante de la cabeza que muestra el contenido de la webcam de la persona controlando dicho avatar. Con este sistema, ofrecemos una forma más natural de organizar reuniones virtuales porque no solo podemos ver en tiempo real las caras del resto de la gente en un entorno virtualsino que también podemos interactuar con ellos cara a cara. Sin embargo, y debido a la amplia variedad de aspectos en los cuales este proyecto se puede expandir, habría por tanto una gran cantidad de funcionalidades que podrían considerarse partiendo de este punto, entre las que podemos encontrar: • Implementación de un sistema de inventario: Una de nuestras ideas futuras consiste en hacer posible transferir archivos o documentos entre personas pero como si los avatares se pasaran dichos dicumentos físicamente entre ellos. Con esto, podríamos ver una suerte de inventario en el que podríamos almacenar una cierta cantidad de objetos, o quizás apuntes del profesor, por ejemplo. • Organizador de horarios: Otro aspecto importante pero que aun así nos ha faltado sería que profesores o administradores pudieran crear y organizar sus clases al principio del año, de forma que, cuando un estudiante se matricula en un curso, se les asignara automáticamente a sus clases correspondientes y a la hora correspondiente. • Sistema de Inicio de sesión: Ahora mismo cualquiera puede entrar como profesor en la aplicación (metiéndose como Host). Una de las cosas que deberían estar en una versión futura sería cambiar el sistema de autentificación de manera que se pudiera controlar quién se puede registrar como profesor y quién no. 6.4. FINAL VERSION OF THE PROJECT 83 • Encriptación de contraseña: Por otro lado, no nos podemos olvidar de la seguridad, y por esa razón se debería hacer un tratamiento seguro de las contraseñas y en general de las bases de datos que se hicieran. • Chat de Texto: Otra posibilidad sería añadir un chat de texto, de forma que dos o más personas pudieran enviarse mensjaes sin molestar al resto de la clase, o incluso compartir links entre otras cosas. • Compartición de pantalla: Otra funcionalidad interesante sería dar la oportunidad de compartir el contenido de la pantalla del profesor, o permitir nuevos formatos de archivo en la pizarra tales como pdf, por ejemplo. • Escribir con tiza en la pizarra: Otra opción podría ser laposibilidad de escribir con tiza en la pizarra virtual, permitiendo a todo el mundo unirse y escribir por ellos mismos. • Tener la opción de enmudecer el audio: Otra funcionalidad a considerar sería la posibilidad de enmudecer tu propio audio cuando prefieres no ser escuchado y tener la capacidad de ensordecer a otros usuarios en el caso de que puedan ser molestos. Chapter 7: Work done by each member In this final section, we enumerate the respective tasks each of us has done in the development of this project. 7.1 Samuel • Implemented the authentication system (Frontend side), as well as the ingame GUI and its Backend. For this sake, he: –Created the layout using Unity UI Builder. – Created the C# script to handle the user interaction with the GUI and proceeding to call the corresponding backend systems afterwards. •Contributed to the refactoring of the Faculty model. • Implemented the Player movement and the Following Camera System For that sake, he: –Created the dummy Avatar. –Assigned the main camera to it. –Imported the “Following Camera” script. 84 7.1. SAMUEL 85 –Created the movement controller script and assigned it to the player. – Created and added the username tag over an avatar’s head, as well as implementing a system to let it turn towards the player, so that they can read it regardless the angle. –Finally converted the whole hierarchy into a prefab. • Added a digital clock to the GUI so that both users and professors can know how much time of class is left. •Implemented the translation system. For this sake, he: –Created the ITranslatable interface. –Created the Translator class. – Made the GUI classes implement the ITranslatable class and register them in the Translator class. •Implemented the File Selector system. For this sake, he: –Imported the FileBrowserUpdate script. –Created the Interaction Controllers for the player. –Adjusted the file extensions to be allowed to be chosen. – Made the script to return the list of paths of the selected files in order to let the Blackboard system to display them in the classroom. •Formatted and organized the document template in LaTeX. •Worked on some of the UML diagrams. • Worked on part of the Implementation,Considered Technologies and Project Planning chapters. 7.2. ALEXANDER 86 7.2 Alexander •Creation of the initial project –Creating the github repository with the initial unity project. –Importing the exisiting faculty 3d model to the project. –Configuring the packages and tools we used as we went along. •Contributed to the refactoring of the faculty model – The objects were not organized in any hierarchy when the model was loaded, so gave some identifiable names and more coherent structure (faculty has areas, which have floors, windows, doors ... And those doors would group the different elements that make up the door) so that we could later work with those elements more easily – Added the necessary components to each item. These are mainly colliders (solid objects such as floors, walls, tables, ... need colliders so that the player can not go thorugh them) and DoorInteraction scripts to the doors •Research and draft a solution for coherence as an object synchronization tool – Gain understanding of the coherence tool, reading the documentation, checking existing demos, going through tutorials, to evaluate if the tool was appropiate for our case. – Create a separate mini-project to serve as a working prototype for how the object synchronization would be and encounter any blockers in the way. Also to share this prototype with the rest of the team, and test out the synchronization though different networks. – This investigation served to realize that it didn’t integrate well with the audio communication tools evaluated, so it was discarded in favor of syn- 7.2. ALEXANDER 87 chronization with NetworkManager using Relay as the transport • Implemented and debugged (a lot) the network synchronization with NetworkManager and relay as the transport • Added the in-game debug console to be able to debug some of the synchronization problems when running multiple instances of the application (outside the editor) •Iterated and refactored the player controllers as our project developed – When we allowed to have multiple users in the scene, avoid creating multiples in each machine (causing a player to move all others) while also allowing non-host players to move their own avatar (which led to the creation of things like ClientNetworkTranform, to give the authority of the avatar object to the client that owns it) – Created the ServerRpc and ClientRpc methods, used when synchronizing the doors because the client has no ownership over the doors, and for synchronizing the webcams since there was no direct way to synchronize that data. •Created the component diagram and some of the sequence diagrams – Presented the rest of the team with the plantUml tool, as a way to both more quickly create and edit diagrams, which allowed for more visual conversation on how we could iterate different flows in our code, by more easily comparing the before and after of each proposed change. – Created some of the sequence diagrams and created the component diagram to visualize how all of the relevant objects and scripts in our project interact • Worked on writting this document. Mainly focused on the Implementation and Considered Technologies chapters, but also contributing and reviewing to a lesser extent on the rest of the chapters. 7.3. SARA 88 7.3 Sara • Modifications to the script PlayerInteractionController to support environmental interaction functionalities •Interaction with doors. –Creation of the DoorInteraction. –Modifications to the script PlayerNetwork. – Adjustments to attribute values related to coordinates, necessary for the proper movement of doors. –Performing manual tests to evaluate functionality •Trying out a quick reaction system (emojis on the avatar) –Search for three-dimensional resources for display –Executing tests on displaying or hiding objects within the scene •Interaction with the blackboard. – Research on how to display resources like videos or presentations on a plane. –Modifications to the script InfoBoxManager. – Added a plane to the blackboard to render textures of images or videos intended for display. – Creation of the Blackboard script and addition to the plane located at the blackboard position. –Added a –Performing manual tests to evaluate functionality CHAPTER 9. APPENDIX 95 mejorados y evolucionar, implementando funcionalidades que permitan reducir la separación (hoy enorme) entre las experiencias de interacción presencial y remota.