Deliverable D1.1: Review of UX Methodology and Tool Proposition

Deliverable D1.1: Review of UX Methodology and Tool Proposition Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 1 Document Information Document Identification Status Final version Due Date 31 July 2024 Version 6.0 Submission Date 30 July 2024 Related WP WP1 Document Reference D1.1 Related Task(s) 1.1.2 Document Type R Related Deliverable(s) D1.2 – D1.9 (D2-D9) Dissemination Level PUBLIC Lead Participant JU Lead Author Monika Krakowska Contributors JU; MZ; KUL; UM Reviewers Zoë Vandenhende (KUL) Reviewers Andrew Pace (MZ) Document History Author(s) First Name Last Name Partner Monika Krakowska JU Magdalena Zych JU Marek Deja JU Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 2 Version Date Modified by Modification reason 1.0 14.07.2024 Monika Krakowska (JU) First draft 2.0 19.07.2024 Zoë Vandenhende (KUL) Internal review 23.07.2024 Andrew Pace (MZ) 3.0 22 -23.07.2024 Monika Krakowska (JU) Internal reviewer’s comments implementation 4.0 25.07.2024 Zoë Vandenhende (KUL) Second internal review 5.0 26.07.2024 Monika Krakowska (JU); Magdalena Zych (JU) Evaluation team’s comments implementation 6.0 30.07.2024 Łukasz Pieczonka (JU); Żaneta Kubic (JU) Evaluation by the QA Team and assigned experts. Final version. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 3 Quality Control Role Who (Partner short name) Approval Date Deliverable leader Monika Krakowska (JU) 26 July 2024 Quality manager Łukasz Pieczonka (JU) 30 July 2024 Project Coordinator Żaneta Kubic (JU) 30 July 2024 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 4 Executive summary The Deliverable 1.1 presents an overview of UX methodologies resulting from a scoping review of the available scientific and research literature, as well as from various analysis and exploration activities, a research methodology and a proposal of tools to explore the experiences, behaviours, needs and affective-cognitive factors of different user groups, as envisaged by the IMPULSE project. The report is divided into three main parts, chapters. The first chapter briefly outlines the main objectives of the IMPULSE project, taking into account the subject and aims of Work Package 1, which includes a number of subtasks, including subtask 1.1.2, resulting in the development of a proposal for a methodology for researching users using the final project product. Attention was also drawn to the categories of users, for which adequate research tools were created in this first, key stage of activities, to be implemented at the stage of subsequent subtasks related to the analysis of user experience (UX research). The second chapter presents the process of developing the research procedure, creating the methodology by characterising: (a) the main results of the scoping review and the methods and tools used to study the users of immersive environments across multiple dimensions and disciplines of science and art in the context of the use of digital cultural heritage resources; (b) the activities undertaken in the form of research in action, or team and expert consultations; (c) the initial studies that were carried out as part of participation in the task processes of the CAPHE project (H Communities and Artistic Participation in Hybrid Environments, The HORIZON Europe Program under grant agreement no. 101086391; https://www.caphe.space/) and in collaboration with participants in events created in immersive environments, including artists. In the third part of the report, the final chapter presents the research methodology created, together with a characterisation of the research methods and tools to be used in order to achieve the objectives of the IMPULSE project, in particular concerning the study of user experience and the development of coherent effects of extended storytelling towards vivid users' experience. In addition, possible difficulties and problems that may occur during the implementation of the methodology and the conduct of the research have been taken into account. Finally, there is an Appendix, which adds appendices, including a scoping review report, diagrams of all developed questionnaires for different user groups, and observation and interview patterns to be taken into account when implementing UX research. Key words: experiment, questionnaires, interview, observation, procedure, prototype, report, research methodology, research tools and techniques, scoping review, user experience, user research Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 5 Table of Contents Document Information ....................................................................................... 1 Executive summary ............................................................................................. 4 Table of Contents ................................................................................................. 5 Abbreviations and Acronyms ............................................................................. 7 1 Introduction ................................................................................................... 8 1.1 IMPULSE Project - résumé ......................................................................................................... 8 1.2 Objectives of the Work Package WP 1 ................................................................................... 11 1.3 Objectives of the Task 1.1 ........................................................................................................ 12 1.3.1 User groups as audience and actor categories in UX research in the IMPULSE project ....…………………………………………………………………………………………………………………………….13 1.3.2 Structure of Task 1.1 ........................................................................................................ 18 1.3.3 Objective of the subtask 1.1.2 ........................................................................................ 20 2 The Process of Creating the Methodology and Procedures for User Research ............................................................................................................. 21 2.1 Scoping review main conclusions ........................................................................................... 21 2.1.1 Process scoping review and thematic analysis ............................................................ 21 2.1.2 Scoping review results ..................................................................................................... 24 2.2 Team consultation as a form of research in action in the development of research methodology .......................................................................................................................................... 26 2.3 Preliminary initial studies within CAPHE project .................................................................. 29 2.3.1 CAPHE events used in the preliminary study ............................................................... 30 2.3.2 Planning and implementation of initial studies in CAPHE .......................................... 31 3 Methodology of user research in the IMPULSE project - characteristics 39 3.1 User research methods - proposal ........................................................................................ 40 3.1.1 Surveys (Questionnaires) ................................................................................................ 43 3.1.2 Experiment ........................................................................................................................ 49 3.1.3 Interviews .......................................................................................................................... 52 3.1.4 Focus groups interviews .................................................................................................. 53 3.1.5 Observation ....................................................................................................................... 54 3.1.6 Additional methods: Think-aloud protocol ................................................................... 54 3.2 The IMPULSE project research tools ...................................................................................... 55 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 6 3.2.1 Survey questionnaires ..................................................................................................... 55 3.2.2 Impulse questionnaires scales ....................................................................................... 61 3.2.3 Interviews and focus groups........................................................................................... 62 3.2.4 Observation schemes ...................................................................................................... 65 3.3 Potential problems and barriers to conducting research ................................................... 67 4 References ................................................................................................... 70 5 Appendices ................................................................................................... 73 5.1 Scoping review report .............................................................................................................. 73 5.2 Questionnaires ........................................................................................................................ 123 5.2.1 General Questionnaire for users (Group 1 and Group 2) ......................................... 123 5.2.2 Questionnaire for non-users ........................................................................................ 132 5.2.3 Questionnaire with experiment and prototype usage.............................................. 137 5.3 Scheme of Interviews ............................................................................................................. 148 5.3.1 Scheme of interview for IMPULSE users (experts selected from G1 & G2 as well as G3) .. ………………………………………………………………………………………………………………… 148 5.3.2 Scheme of Interviews with academics, artists and non-users ................................. 149 5.4 Scheme of Observations ........................................................................................................ 154 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 7 Abbreviations and Acronyms Abbreviation / acronym Description App. Appendix CCI Cultural and Creative Sectors and Industries DX.X Deliverable number X belonging to WP X EC European Commission FIG. Figure G X (e.g. G1) Group X (e.g. Group 1) IMCo IMPULSE Community of Practice JU Jagiellonian University MUVE Multi-user Environment p. Page Tab. Table UX User Experience VR Virtual Reality WP Work Package WP X Work Package Number XR Extended Reality Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 8 1 Introduction This section of Deliverable 1.1 presents the subject matter and key objectives of the IMPULSE project, taking into account the aims of WP1 and Subtasks, both 1.1.1 and, most importantly, Subtask 1.1.2, whose priority was to develop a methodology for user research and user experience in immersive environments. Given the tasks required to achieve the main aim of the project, their intended scope was incorporated into Work Package 1 (WP1) and Subtask 1.1, as well as a supporting element, labelled subsequent sub-task 1.1.2. The overarching aim behind subsequent sub-subtask 1.1.2 was developed on the basis of a critical analysis of the literature and a scoping review conducted for the project, as well as a variety of activities, including preliminary studies, undertaken by the partners of each Work Package. The aforementioned studies will serve as a preliminary foundation for the actual preliminary studies that have been proposed in accordance with the IMPULSE framework. This initial stage is for the evaluation and validation of the proposed methodologies and the IMPULSE research procedure, small-scaled studies will be conducted concurrently with the implementation of the prototype under development at a subsequent phase of the project. The research methodology that emerged from these empirical and conceptual explorations will discern the needs, expectations, beliefs, socio-cultural impulses, embodiment and use of technology, together with the dynamics of engagement in domains of virtual heritage facilitated by virtual reality (VR). Thus, the methods detailed in the research procedure and the quantitative-qualitative analysis of the results carried out in the subsequent stages of the project will contribute to conceptualise and update the prototypes while providing recommendations tailored to the needs of the public actors entrusted with the digital preservation of cultural heritage. These directives will be based on insights derived from the conditions, expectations, requirements, behavioural patterns, motivational frameworks and experiences gathered from the three main cohorts of users surveyed. Thus, the methodology used to study the experiences of users using digital resources in MUVE/metaverses will implement the process of their application in the didactic, creative and commercial dimensions. 1.1 IMPULSE Project - résumé The overarching goal of the IMPULSE project is to develop innovative and multifaceted solutions and methodologies for the digitisation and accessibility processes of the collections that make up digital cultural heritage. These efforts aim to facilitate their innovative use but also reuse, to address the challenges of interoperability between platforms and to improve access to pre-existing digitised cultural heritage materials in new contexts, in particular in the metaverse. At the same time, the project aims to develop pioneering standardisation protocols and adapt the legal framework to contemporary changes and creative dynamics in education, the arts and the CCSI Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 15 These principal groups with selected sub-groups are described below and include: • Group 1 (G1): This group comprises students and academic teachers, academics from different university departments and institutes selected in all IMPULSE partners academic environments, such as those from the Jagiellonian University in Krakow. The group of users in question can be described as a general and principal one, which may be further diversified and require further subdivision. This will be based on the characteristics of this group (G1), as well as the identification of such users who already have some experience with VR, immersive environments, as well as very active users who are involved in their use, if only in the teaching process (as this dimension is crucial in this group) and the creation of content. Concurrently, the G1 group may also comprise individuals who do not utilise such technologies, as defined on page 17. These individuals are also crucial for the project, as they must be identified and their needs, expectations and motivations in experiencing immersive environments explored. This overarching group of users (G1) will include additionally within those users who engage creatively with immersive environments (e.g. use VR environments for didactics and create content) there are two sub-groups: experts and non-users. Selected individuals who express a willingness to participate in subsequent stages of the project will be included in both the user and non-user groups. From groups of users from G1, G2 & G3, who agree and express an interest in participating will be invited to participate in research using the prototype (an experimental form based on using resources within the immersive environment of the prototype and performing specific tasks according to the stages of project development and prototype creation, along with completing surveys and possibly using retrospective think-aloud protocols). It is now necessary to provide an overview of the methodology in order to facilitate comprehension of the research procedure as a whole. Given the current stage of the IMPULSE tasks, this preliminary description will focus on the key elements of the methodology, highlighting the aspects that should be taken into account in the selection of respondents for the study. A group of experts may be selected on the basis of an analysis of completed questionnaires or with the use of convenience sample method. The experts in this subgroup of users are defined as highly active and proficient users of virtual environments, capable of not only creating such environments but also their content. They possess extensive experience in utilising immersive spaces and have teaching experience, having either used or are currently using virtual environments for educational purposes. Those who are engaged in the utilisation of virtual environments for the learning process (within the context of higher education), and who are established users and creators within the research community, may be invited to participate in the subsequent phase of the study, wherein they will utilise the prototype to explore the metaverse user experience. Furthermore, they must be willing to participate in the experiment. The primary aspect to be explored within this group is the educational and pedagogical perspective, examining the potential for integrating and applying Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 16 immersive environments and digital cultural heritage into teaching and learning processes. A prototype intended for didactic use within this group, utilising contextual layers, will facilitate the examination of ownership within institutional collections and narratives. • Group 2 (G2): This group includes artists of various specialisations, as well as students and teachers from art schools. As with the preceding cohort, this group constitutes a general, principal grouping of users, which may also be diverse and thus necessitates further subdivision. This will be based on the characteristics of this group (G2), as well as the identification of such users who already have some experience with VR, immersive environments, as well as users demonstrate high levels of activity, those who are involved in VR or metaverse use, if only in the creative, artistic process (because this dimension is crucial in this group) and the creation of content. Concurrently, the G2 group may also comprise individuals who are not users of the technology in question. These individuals, as defined on page 17, are also crucial for the project, as they can provide insights into the needs, expectations and motivations of those who do not currently utilise immersive environments, particularly in the context of artistic activities. G2 will then be divided into two sub-groups of experts and non-users, similarly to G1. After the first phase of research, based on the results of the questionnaire, a group of experts will be selected that will include artists, students and academics who actively use VR and engage with immersive environments, create their work in such spaces and share their immersive experiences in virtual environments. The second sub-group will include non-users selected through a survey. The latter group of non-users, a subset identified through the questionnaire, consists of actors that means users, those who do not use VR due to a lack of (often unacknowledged) knowledge about such environments, a lack of competence, or a lack of willingness or motivation to use the metaverse. Furthermore, as in G1, it is possible to select within G2 a group of experts from within this group on the basis of an analysis of the completed questionnaires and using the convenience sampling method. The term "expert" is defined as a highly active and proficient user of virtual environments, including those who create such environments and their associated content. These individuals possess extensive experience with immersive spaces and are established users and creators within the artistic community. They may be invited to participate in the subsequent phase, utilizing the prototype to examine user experiences within the metaverse, and are willing to engage in the experimental process. Various dimensions of experiencing immersive environments will be investigated within this group, including the creative aspects of perceiving, creating, and utilising digital cultural heritage resources, both in metaverse, as well as with the use of the prototype and selected digital heritage content. The inquiries will focus on the potential of using prototypes in unconventional artistic contexts. Artists will employ the IMPULSE prototype in their creative processes through the implementation of artistic research and speculative methods, intellectually interpreting and reinterpreting information and resources to engage Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 17 performatively with assets in metaverse worlds. A particularly interesting approach to studying this user group will be speculative methods, which aim “to imagine or create futures or conditions that may not yet exist, to provoke new ways of thinking and to focus on specific ideas or issues” (Ross, 2017). • Group 3 (G3): As with previous cohorts, this group represents a general core group of users, which may also be diverse in terms of the specialties and industries that this group represents. These include, for example, computer game developers, screenwriters, game music, and others. The selection of users for the study within this group (G3) will be based on collaboration with the IMPULSE Community of Practice (IMCo), the convenience sampling method, and general familiarity and accessibility to recognized individuals in the specialty and industry. The group is comprised of individuals representing the creative and commercial sectors. This group is of primary importance and its selection is guided by the project's objectives. It has been assumed that this is a highly diverse and specialised user group (G3), consisting of professionals who utilise virtual reality (VR) but do not necessarily employ digital heritage resources in this environment. This assumption is also implicit in the project's objectives, where the commercial dimension as it relates to the group representing cultural heritage institutions (CCSI) and the identification of opportunities for the utilisation (as well as re-use in creative way) of a variety of digitised cultural heritage objects in the creation of immersive environments is made clear. The group is constituted by professionals who are proficient in their respective fields and who will be included in the research. These may include, for example, computer game developers or filmmakers. It is assumed that this group, in line with the subject matter of the project and its stated objectives, constitutes a sector that does not make sufficient use of or indeed make use of digital heritage content at all. Furthermore, it is assumed that this group does not use these resources optimally. However, it is possible that this sector could become one that actively uses and transforms digital heritage content in creative ways. The research will employ interview methods and focus groups with representatives of these industries, who will also be selected using the convenience sample method. This will enable the identification of their motivations, expectations, needs and ways of experiencing the metaverse. This group encompasses CCSI organisations and networks, freelancers, and designers. Research within this user group aims to reframe the mental model, as well as the mental and cognitive representation, of those within CCSI, particularly concerning new understandings and methods of utilising digital cultural heritage. Identifying the needs and expectations of users in this group will be crucial for the proper development of standards for storing, using, and creatively reusing these resources, especially in the context of prototyping new immersive environments. Representatives of CCSI, freelancers, and designers will use IMPULSE to explore methods to facilitate the integration of existing digital resources into metaverse environments, through co-creation based on exemplary sets of open cultural data available, for instance, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 18 on Europeana (https://www.europeana.eu/pl), with various constructive approaches to resource integration. The research will examine all these groups and sub-groups perceptions of virtual environments, identify their potential applications for these users, and, crucially, incorporate knowledge about their needs, motivations, and ways of perceiving and understanding new technologies, including metaverses. It will also assess the likelihood of their engagement with digital cultural heritage and identify potential barriers and issues that may lead to avoidance, abandonment, or a lack of immersion and experience in virtual environments. The project and the user research carried out adopted a definition of non-users, according to which these are people who do not use new information technologies, products or services because they are not familiar with them, do not have knowledge about them or do not perceive the environment and facilities offered as appropriate, adequate or accessible. Therefore, a non-user is an individual who does not use new technologies, including different products or services, despite their knowledge or lack thereof. Non-use of new technologies by an entity defined as a non-user can be attributed to several reasons, including a lack of acceptance of new technologies, a lack of specific needs for their use, inappropriate experiences resulting from the use of new technologies, and a lack of knowledge about new technologies. Additionally, non-use of new technologies may be related to affective dimensions such as resistance, aversion, fear, and other negative emotions. The term 'non-user' also encompasses individuals who are disregarded or inaccessible to the field's interests (Rabello, 2023). The project includes a group of potential and future users, specifically those in heritage contexts where XR apps would be applicable, but who currently lack the knowledge to integrate such approaches into their work. This is a crucial distinction between the current users, comprising the G1, G2, and G3, as well two subgroups: experts and non-users, and the experts from G1 and G2, who were incorporated into the comprehensive research. Additionally, there is a need to consider the potential future users, who would require a compelling reason to utilize XR or other digital heritage resources in XR. This could be for educational, teaching, artistic, creative, or commercial purposes. 1.3.2 Structure of Task 1.1 The research undertaken as part of Task 1.1 will encompass an assessment of the skills and competencies of current and future or potential users and non-users, as well as creators. It will also examine the reactions elicited by the tools used and the proposed personifications of virtual environments generated by institutions (e.g., interactions and responses to avatars and objects, resources). Stratified, random, and self-selecting sample studies will be conducted by UX support groups and quality control teams. These studies will be carried out on a diverse group of participants. The groups identified Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 19 in the project documentation, which have been previously described, include: 1) students, including doctoral candidates, and academic teachers (G1); 2) artists and academic teachers from art schools (G2); 3) the creative industries (including computer games, film, animation, and performance (G3). The entire task has been divided into sub-tasks, which will be undertaken in a compatible, transparent, and iterative manner throughout the project's duration. These sub-tasks are divided into seven parts as follows: • Task 1.1.1 (months 1-3): Critical analysis of literature and scope review. At this stage, exploratory (diagnostic) qualitative and quantitative research will be conducted to identify research methods and analyse users and non-users, along with their interactions with VR using digital cultural heritage resources. • Task 1.1.2 (months 3-6): The objective is to develop research techniques for three overarching, categorised and differentiated user groups in accordance with the project's objectives, designated as G1, G2 and G3, as well as two sub-groups (experts and non-users) that can be scaled within G1 and G2. For further details, please refer to subsection 1.3.1. The proposed methodology and procedures will allow for the examination of active and potential future users of virtual worlds and the identification of their knowledge, competencies, motivations, and ways of experiencing immersive spaces. • Task 1.1.3 (months 7-15): User research: preliminary studies before prototype development. At this stage, the requirements and information needs, behaviours, and practices of users in immersive and virtual environments will be identified, along with their competencies and expertise, such as software knowledge, terminology, and tools. The findings will be integrated into the prototype development process. The UX research will be iterative preliminary, allowing for verification and modification based on preliminary and pilot study results, and will be conducted within selected target groups. • Task 1.1.4 (months 16-18): Correlation with research conducted in various WPs. This stage involves compiling and analysing the results of research conducted concurrently across all WPs to diagnose the significant requirements, expectations, and behaviours of different users. • Task 1.1.5 (months 16-20, months 29-35): Collection, selection, evaluation, and interpretation of research results. This task includes the compilation of results, construction of personas, statistical and qualitative analysis of surveys, and production of UX diagrams (designing). Additionally, data collection will focus on educational and didactic factors—ideas for art schools’ teachers and professional digital art trainers on promoting the assimilation of new users into VR. Internal reports will be created for processing within various WPs. • Task 1.1.6 (months 21-34): UX research (final phase, after prototype development and implementation). This task involves testing the usability and functionality of the technology and metaverse prototypes using digital heritage resources. Additionally, the quality of the prototypes will be evaluated. • Task 1.1.7 (months 20-36): Compilation of all relevant educational and didactic data. At this stage, instructions and guidelines will be developed for activating art Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 20 schools’ teachers and digital art trainers to promote the immersion of new users in VR. Concurrently, principles concerning the information needs and behaviours of users will be established. A crucial element of this task will be knowledge dissemination and transfer, including the co-creation of scientific publications, hackathons, conferences, and workshops. 1.3.3 Objective of the subtask 1.1.2 While this document mainly highlights the results of subsequent sub-task 1.1.2, it is important to point out the important role and combination of both the activities undertaken in sub-task 1.1.1 and the activities carried out in WP 2 (TECH), WP 3 (STAND), WP 4 (LEGAL WP), WP 5 (DISS WP). The creation of a library of resources in Zotero by all the teams of partners participating in the project, the search for interrelationships between WPs, the discussions, the creation of a database of questions, the clues that have been taken into account both in the scoping review process and have contributed significantly to the development and preparation of a set of methods and techniques that will form the methodological basis for any research carried out in analysing the experiences, needs, motivations, cognitive-affective aspects of all users in the different groups to be analysed. Hence, the key results of the scoping review have been included in the document, as a starting point for the development of the user research methodology and the activities that have been carried out during the process of creating this extremely difficult, multifaceted, but also compatible with the stated objectives of the project, user experience exploration scheme. The IMPULSE project represents a clearly defined conceptual framework, whose realisation, construction, and implementation are systematically organised through iterative methodologies, encompassing continuous improvement alongside exploratory analyses and actions. This endeavour is divided into six distinct work packages, each outlining a primary objective while leveraging the actions, outcomes, and insights from all associated tasks within the project to support its execution. Among the early stages in these work packages is Subtask 1.1, focused on user experience (UX) research, and subsequent sub-task 1.1.1, which involves a critical review of literature and scoping review, included within WP1, the EX-STORY activities. IMPULSE initiative seeks to challenge existing narratives by showcasing the diversity and multidimensionality of phenomena related to behaviours and experience dimensions while promoting immersive environments. Furthermore, it aims to include marginalised cultures and communities by integrating hidden stories, thereby creating complex, diverse, and multi-layered narratives intended to engage a broader audience with the presented topics and themes. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 21 2 The Process of Creating the Methodology and Procedures for User Research This chapter will set out the procedure for developing and creating the research methodology. In accordance with the roadmap and task implementation, the methodology should be validated through the preceding proper research phase. It is anticipated that the preliminary research phase will entail the utilisation of methodologies and instruments by all project partners. The subsequent actual research phase will employ the prototype, which will be conducted in two phases: the first phase will entail pilot research utilising the prototype, and the second phase will entail testing and implementation at various stages of the research process, where feasible. Each of the methods, as well as the research phases, can be modified and improved during the subsequent phases, starting from the preliminary studies, as well as a result of diverse and often changing conditions, also taking into account the characteristics of particular user groups, whose experiences, including needs, information practices and behaviour, activities, perception, affective-cognitive predispositions will be diagnosed. 2.1 Scoping review main conclusions The subchapter will describe the main conclusions that were drawn after the scoping review and thematic analysis. The full text of the report of this activity and part of research belonging to subsequent sub-task 1.1.1, which is also a prolegomenon for the development of the research methodology can be found in Appendix 5.1 (p. 73). 2.1.1 Process scoping review and thematic analysis Qualitative and quantitative exploratory (diagnostic) research to identify research and methods used in the analysis of users and their interaction with VR using digital cultural heritage resources. The development of user research methodologies in the IMPULSE project would not have been possible without a thorough evaluation of the available and applied methods, techniques, and research tools for various types of users, not only in the context of experiencing immersive environments but also in utilising digital cultural heritage resources. Previous analyses conducted across different disciplines, using diverse methods, highlighted the necessity of creating a research tool and procedure that can undergo continuous evaluation, verification, and improvement at every stage of the project, particularly in relation to the development and use of a multi-user virtual environment (MUVE) prototype that exclusively utilise digitised cultural heritage content, encompassing a range of formats including e.g. audiovisual material. The perspective Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 22 of an environment applicable in educational and didactic processes, as well as a space for the creative activities of artists and a tool for the creative industry, has also allowed for the use of the scoping review as a means to identify various approaches and possibilities. These insights have become essential to integrate into the research process in the next phase of the IMPULSE project. The scoping review constituted the initial task undertaken within WP1 and involved conducting exploratory qualitative and quantitative research with the aim of identifying the research methods and analyses employed in the investigation of the user experience with virtual reality, particularly in the context of utilising digital cultural heritage resources. The scoping review became a form of knowledge synthesis through a systematic and iterative approach to identifying and synthesising existing or emerging user experience research in the literature. The choice of a scoping review also allowed for mapping the scope and nature of the literature and identifying specific gaps within it. This approach was instrumental in constructing an analytical framework related to the research problems and areas addressed by the IMPULSE project, through exhaustive processes of searching and acquiring information. This supported the synthesis of knowledge, the identification of concepts, terms, procedures, as well as research methods and techniques (Tricco et al., 2018). The analysis was based on the PRISMA 2020 flowchart for systematic reviews and meta-analyses (Page et al., 2020). The research was developed based on a group library created within the IMPULSE project, named HORIZON_IMPULSE in Zotero. The library aimed to collect, classify, and share an extensive resource concerning the issues and research areas related to the specific research groups within the project. Consequently, the library was divided into folders corresponding to the main work packages in the IMPULSE project related to research. The different sub-folders have been named according to the acronyms of the individual WPs: Dissemination, Ex-story, Legal, Standard, Tech. During the literature analysis and scoping review, subfolders were added: Screening (containing all publications selected for critical analysis and review), Screening Additional Searching (containing all publications found during the scoping review), and Trash (where mainly duplicate publications not included in the analysis were moved). The expansion of the library through systematically gathering relevant resources related to the project's subject matter was essential. The multi-layered overarching aim of the project, combined with the literature gathered during the collection, search and review stages, helped to expand the objective in subsequent sub-task 1.1.1 and broaden the scope within which the analysis was conducted. As a result, the focus has shifted to a synthetic, multifaceted and interdisciplinary study of the phenomena, features and contextual factors related to research on user experiences in virtual environments using digital resources, including those relating to cultural heritage, and their implementation across disciplines and domains, meeting the needs of different user groups as well as non-users. Achieving Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 23 the set overarching objective required embedding it in the broader context of the user experience in virtual environments, taking into account diverse but selected aspects and contextual factors, and recognising the interdisciplinary dimension of the overall subject matter. Consequently, additional specific tasks were formulated, including: 1. Determining the most appropriate research methodologies for user experience research in immersive environments. 2. Identifying the areas and domains in which previous research has been undertaken. 3. Recognizing the goals and intentions of research conducted in this area, as well as the dimensions in which such research was conducted. 4. Incorporating diverse technologies associated with virtual environments. 5. Identifying various user groups among which VR experience research was conducted. 6. Recognizing different aspects related to VR experience. 7. Diagnosing the plethora of methods used to study users and their experiences in immersive environments. 8. Capturing diverse terminologies. 9. Attempting to identify ways of studying the needs and behaviours of users in immersive environments. The 201 full texts selected were then subjected to thematic analysis using the MaxQDA software (the thematic analysis tool is a qualitative data analysis tool for which a licence was purchased for JU) by three independent researchers. The software allows for the incorporation of resources selected during the current information and literature acquisition, which will be used during the project and will address methodologies for studying user experience and behaviour in immersive virtual environments. MaxQDA allows for efficient research, various types of analysis and continuous improvement at each stage of the project. The material was divided numerically, and deductiveinductive coding was used. Deductive coding was applied using expert knowledge (domain-specific from project partners involved in user research in technical, information and social sciences) as well as knowledge developed by experts in user experience research. The critical analysis and scoping review started with an analysis and verification of the existing Zotero HORIZON_IMPULSE library. The use of Zotero library resources for IMPULSE included resource verification, during which duplicate bibliographic descriptions and full texts were removed, resource availability was checked, and efforts were made to locate full texts. Meanwhile, during the thematic analysis conducted in MaxQDA, due to the expanded scope of the analysis undertaken and the scoping review broadening the scope and focus of the VR experience research, a fourth researcher conducted a search for publications in Web of Science and Scopus databases. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 24 A full report of the scoping review process and the results of the critical literature analysis and thematic analysis undertaken can be found in Appendix 5.1 (p. 73). 2.1.2 Scoping review results Scoping review has identified a number of research procedures that can be used when studying VR users. Among them, methodologies and methods were distinguished. When categorizing individual procedures into either of these two groups, the declarations of the authors of the analysed publications were followed. The methodologies were assigned 28 sub-codes with different levels of detail. At the general level, the authors of the analysed publications used quantitative, qualitative, mixed methodologies, among others. Empirical research was also used, as well as analytical, iterative methodology, phenomenological approach and triangulations. Of the more specific methodologies noted were grounded-theory, Living Lab, Research by Design methodology, Rovina Paradigm, User Centred Design, user-centred innovation approach, and Whittemore & Knaff methodology, among others. The aforementioned methodologies were merely illustrative examples of those discussed in the context of the D1 study. They were included as a result of the scoping review and were employed in the analytical and other UX studies that fell within the scope of the thematic analysis. It is recommended that these methodologies be considered for use in the subsequent stages of the IMPULSE study. As the reports being prepared are documents that will evolve as the work in IMPULSE progresses, it is appropriate to revisit the question of using additional research methods, as well as methodologies, once the activities undertaken by WP1.2 and WP1.3 and their broader understanding based on the emerging prototype have become established. The continuous improvement of our knowledge and understanding of the requirements will enable us to consider appropriate and optimal methodologies for eventual implementation. In the context of IMPULSE, both the wide range of possible methodologies use, and triangulation of methods are noteworthy. The multitude of methodologies identified in the scoping review process demonstrates the intricacy of research in regard to information requirements, expectations, and the experiences of diverse users of VR (and VR-like technologies). Concurrently, it offers considerable flexibility in designing research within this domain. On the other hand, the triangulation of methods and the possibility of combining different methodologies, especially quantitative and qualitative research, in a single research project allows for a multidimensional understanding of the studied fragment of reality. In addition, due to the research objectives of IMPULSE and the specifics of the user groups under study, this approach enables efficient organization of the research tailored to the specific research objectives and the possibilities and limitations of the research (e.g., the stationary availability of the subjects to organize empirical research in the VR studio, the characteristics of the collected research material and the possibility of its analysis). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 31 of selected categories of IMPULSE users. Additional objectives were to test the pre-developed research procedure and tools and to obtain information on the XR technology used by CAPHE artists (at the request of WP2). The research in Italy focused on one of the user groups considered in IMPULSE, i.e. artists using VR, creators as well as the experts. This was a priority for mobility with CAPHE because of accessibility to this group. 2.3.2 Planning and implementation of initial studies in CAPHE The alignment of objectives related to investigating the needs and user experience in G2 within IMPULSE permitted the conduct of initial studies within CAPHE. This allowed for the development and preliminary verification of the proposed methodological approaches, which will be applied during the main phase of the project. In addition, the refinement of interview and observation protocols, which form part of the methodological toolkit for IMPULSE, will be undertaken. On the basis of the scoping review, three general schemes were developed for the research procedure to be conducted with CAPHE (see Table 1). The choice of the scheme to be implemented depended on the possibilities available during the mobility (e.g. events planned in CAPHE, availability of artists and materials prepared in parallel in IMPULSE). In the end, it was possible to implement the most realistic option, an intermediate plan with 15 observations and 4 interviews. All the interviews carried out were a continuation of the observations, following the technique of contextual inquiry. Tab. 1. General scheme of the research procedure for CAPHE initial studies. Plan minimum Intermediate plan (most realistic) Maximum plan observation some observations some individual interviews some contextual inquiries (individual interview + observation) contextual inquiries (individual interview + observation) + maybe questionnaires to test (must be something for laptop, because printouts are unlikely to be done in the field) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 32 observation scheme + photos (phone camera) observation scheme with a semi-structured interview scenario + photos (phone camera) + recording of the interview (voice recorder) observation scheme with a semi-structured interview scenario + photos (phone camera) + recording of the interview (voice recorder) + survey questionnaires Source: Self-authored, 2024 Conducting the CAPHE initial studies required the preparation of research tools, i.e. observation and interview schemes, optionally also a user survey questionnaire. The starting point for the development of the observation schemes were the dimensions identified in the scoping review and the discussion during the methodological internal meeting of the UJ team on 7.05.2024. Based on these, it was decided to include the following elements in the observations: 1. Type of VR experience (what was it, e.g., opera rehearsal, VR opera, costume sewing workshop, etc.). 2. Environment (e.g. Spatial) 3. VR user interactions with objects and other users a. Communication with other users (does it occur and if so, what is it like) b. Additional information, contextual information, menus, tool selection, etc. c. The type of objects and the ways to interact with them (are they present and, if so, what kind - e.g., you can move a vase, you can pour paint on a sculpture, etc.). 4. Cutting off the user from stimuli from the physical world and reactions to this cutting off 5. Movement - how users are moving in physical space (can they move and is the space limited in some way, do they have to move, do they have a choice) and in VR (e.g., teleportation, walking, choice) and what technical tools are used for this movement (the goggles alone? controllers? pads? with vibration – such as in games, where vibration in the pad can alert you of danger? treadmills?) – this movement is related to (1) motion sickness and (2) freedom in artistic creation The next step in developing the observation scheme was to browse the available frameworks for notetaking during observations. Among others, Spradley's 9 Dimensions (Space, Actors, Activities, Objects, Acts, Events, Time, Goals, Feelings), AEIOU (Activities, Environments, Interactions, Objects, Users) and AX4 (Atmosphere, Actors, Artifacts, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 33 Activities)5 were analysed. Due to the previously identified observation elements, it was decided to use Spradley's 9 Dimensions framework. The adaptation of this framework is shown in Table 2. Tab. 2. Idea of adapting Spradley's 9 Dimensions framework to the initial study conditions during CAPHE mobility and IMPULSE objectives. Original version of dimensions Adjustment of dimensions to CAPHE/IMPULSE space - layout of the physical setting; rooms, outdoor spaces, etc. VR space - (1) general level e.g. spatial, single VR, multi-user VR; (2) details of the virtual setting e.g. classroom, concert hall on the beach, amphitheatre finding the right level of detail: how much do we need? “reality” space – indoors? (where?), outdoors? (where?) actors - the names and relevant details of the people involved actors – users with VR goggles – who? how many? e.g. musicians (1 + 3, over 7), opera singers (2), dancers (2 + 3), conductor (1), audience (0, over 7?) activities - the various activities of the actors activities: 1. communication between VR users - does it occur? if so, how do VR users communicate? 2. use of information and virtual tools by VR users - does it occur? if so, in what ways, with what tools, with what information and when? is the use of tools and information easy/intuitive, does it require focus and time, does it cause users problems to the extent that users repeat the action or abandon the idea of the activity? 3. interactions with objects, e.g. lifting a vase, pouring paint on a sculpture, zooming in or rotating an object in space, touching piano keys, waving a baton, stopping at an object, no interaction with objects other than viewing them according to a script (as in the Van Gogh exhibition) 4. VR user's mode of movement: Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 34 • in VR space - 1. walking, 2, flying, 3. teleportation, 4. ‘rollercoaster’, 5. (others? - < additional modalities to be entered>); • in physical space: 1. moving (walking, bending, jumping, etc.) in a designated space, without the use of additional equipment; 2. moving on a treadmill or with the help of other equipment; 3. no movement other than, for example, moving the head and turning on a chair • other activities – <indicate which (insert)> objects - physical elements: furniture etc. objects in VR – objects in VR - what is available? e.g. sculptures? paintings? musical instruments? paints? brushes? <other objects – insert here>? other elements of the world that can be interacted with, e.g. benches, chairs, stairs, doors? objects in “reality”: VR goggles? controllers? pads? haptic gloves? treadmills? steering wheels? moving seats? regular chairs/seats? acts - specific individual actions acts – is this dimension necessary if we have activities and time? events - particular occasions, e.g. meetings events - type of VR experience e.g. VR opera, rehearsal for VR opera, lecture, conference paper, costume sewing workshop time - the sequence of events time - to link with activities / acts and actors + objects goals - what actors are attempting to accomplish goals - ideally in relation to needs and barriers – to be specified, if possible, during the interview feelings - emotions in particular contexts feeling - to be specified, if possible, during the interview (added dimension for initial studies) cutting off, immersion, embodiment - how does it occur? which of the user's senses are cut off from the outside world and redirected to VR? what are the reactions of the VR user? (this is more for an interview, if possible) Source: Self-authored, 2024 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 35 In addition to interpreting the individual dimensions and proposing a preliminary dictionary of responses (to facilitate the note-taking process), the following modifications to the framework were made: 1. the ACTS dimension was removed, due to the anticipated specificity of the observations (rather short events of up to about two hours); 2. CUTTING OFF, IMMERSION, EMBODIMENT dimension was added, which is linked to the IMPULSE study objectives; 3. the order of the dimensions was changed to better match the observed phenomena and conditions during the study; 4. initial elements identifying the individual observations were introduced (observation code, observation start and end time, date and observation point information). During the observations, notes were kept by hand to ensure smooth operation regardless of the field conditions in which the research was conducted. The handwritten notes were then retyped electronically. The written texts were supplemented by photographs, video recordings or screenshots. Some of the materials described in the observations were also accompanied by publicly available links, e.g. to the VR Opera scene in Spatial. A full scheme of the observation can be found in Appendix 5.4 (p. 154). The second research tool that was essential during the preliminary studies was the interview scheme. The starting point for its development were the results of the scoping review, the IMPULSE objectives and the discussions within WP1 and the collaboration with WP2. Furthermore, it was assumed that the optimal research procedure for the conditions offered by CAPHE would be contextual inquiry, so the interviews were to be preceded by previous observations of the individual artists' work. The first version of the interview scheme covered two basic questions to elicit the basic requirements for WP2 in order to create a prototype: 1. What kind of VR platforms (e.g. Spatial) and tools (e.g. VR goggles) are you using in your work? 2. From your perspective: what are the advantages and disadvantages of those VR platforms and tools that you mentioned in terms of your artistic needs? General questions were followed by specific questions relating to previous observations. Using the example of the interview related to the opera Gianni Schicchi, this was as follows (in brackets were details or examples of the paths of additional questions depending on the respondent's narrative): 1. Are Spatial ways of expressing emotions of the characters enough for you? (If yes - which features are the most useful to express e.g. sadness, being angry etc. If no, do you have any idea of VR tools or features that would allow for better expression?) 2. Do you want to access any additional information or tools directly from the VR environment? (for example, was tracking the libretto using a separate program and splitting the screen during the rehearsal enough for you, or would you prefer to have the text somehow displayed right away in Spatial?) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 36 3. Did you feel immersed in your character during yesterday rehearsal? (What do you think mostly led to this immersion (e.g. other people in the back room, the music from YouTube, the libretto, the Spatial environment, something else)?) 4. Is moving the avatar in Spatial the way it was practiced during yesterday's rehearsal easy for you, and do you see any way to improve the VR user experience in terms of immersion and artistic creation? (e.g. using pads or computer mouse) 5. From your perspective as a musician, how do you feel about the possible sound deformation that can happen when creating an opera in VR? 6. Can you imagine anything that would discourage you from using VR? The second and final interview scheme was augmented with suggestions sent by WP2. It consisted of five main questions on VR platforms and tools, their advantages and disadvantages, ways of interacting in VR with heritage objects, the difficulties with importing 2D or 3D objects into VR and the size of the VR environment. The questions posed to experts and artists during the interview process have been collated and presented in the document entitled 'Interview Protocol'. This document can be found in Appendix 5.3 (p. 148). The results of the initial studies carried out by the CAPHE project include: 1. testing observation schemes. 2. developing and testing interview schemes for artists. 3. collection of empirical material: 15 observations, 4 interviews with artists working in XR environments, and pre-arranging further interviews and making contacts that may be useful when recruiting participants for IMPULSE research. 4. insights into the IMPULSE-applicable research procedure: a. considering conducting research in the native languages of the respondents due to greater freedom of expression. b. including in the interview schemes the possibility to change the order of questions and to modify, add or omit certain questions depending on the respondent. c. considering the possibility of 'written interviews' (some CAPHE artists have expressed a willingness to respond in the form of written statements). d. comments on observation schemes: i. The TIME dimension is difficult to complete during live observation; it will be easier to complete based on recordings. ii. the FEELINGS and GOALS dimensions should be completed/verified during the interview, if such an interview is possible. 5. initial insights into VR user needs and behaviours: a. avatar issues (e.g. limited ways of showing emotions in Spatial, difficulties with costume import and live motion capture, abandoning the VR goggles in favour of communication with the other artists working on the opera, cultural differences). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 37 b. lack of certain functionalities important for artistic and didactic projects, e.g., the inability to follow the libretto text (original and translation) during a VR opera in Spatial. c. technical issues with sound during live VR performances and combining a costumed avatar with scenography and live motion capture of the dancer's movements, as well as sound and live streaming. d. mimicking the physical world in a VR environment (gravity, doors, windows, walls, stairs, etc.) with the goal of grounding the user in something familiar to them. e. not associating the artist using VR with the VR end user (How to empower artists?). • Overview of technologies used by CAPHE artists working in XR environments (additional objective at WP2 request) presented in Table 3. Tab. 3. Technology and software used by artists in CAPHE based on initial studies interviews and observations. Technology or software What it was used for Spatial Opera Gianni Schicchi, opera Orfeo & Lwanda, XR Festival exhibitions – GALERIA KONKRET AR.T VRChat Alternative environment for Orfeo & Lwanda opera due to a problem importing an avatar into Spatial; however, it turned out that the avatar file size was too large for VRChart Metaverse For case VR Meditation https://www.lajetee.it/- /medinitaly2/ Rokoko Studio and haptic costume Live motion capture for the Opera Orfeo & Lwanda MoCap System Live tracking gestures Marvelous Designer Used to create the final costume for the avatar Midjourney Creation of the costume for the opera: collecting sketches and assembling into one design, using Midjourney Blender Preparation of files for VR (e.g. changing the properties of the costume and avatar layers) Unity From ITA9: “Why did they choose Unity? Answer: complex solution, but provides various outputs, e.g., to browsers, to applications, to WebGl; allows control of the development process; enables work on the interface and backend (“the interface links our vision”); ability to export from Unity to various devices, e.g. Mac, Windows, Linux, dedicated server, Android etc.” Unreal Engine For case VR Meditation https://www.lajetee.it/- /medinitaly2/ Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 38 Webgl For case AI in my brain https://www.lajetee.it/-/ai-in-mybrain/ Shadertoy For case AI in my brain https://www.lajetee.it/-/ai-in-mybrain/ Html5 For case AI in my brain https://www.lajetee.it/-/ai-in-mybrain/ Sound Ambisonico For case AI in my brain https://www.lajetee.it/-/ai-in-mybrain/ Reality Capture Photogrammetry VR goggles VR goggles used with controllers Desktop VR Projectors Interactive table Interactive exhibition at the Museo Galileo in Florence Photoshop To create a mask for video mapping Visual studio 2019 Coding, used for extended reality and 3D data visualization systems applied to cultural heritage Adobe Illustrator 2023 Used for extended reality and 3D data visualization systems applied to cultural heritage Adobe Premiere 2023 Used for extended reality and 3D data visualization systems applied to cultural heritage Autodesk Maya 2023 Used for 3D in extended reality and 3D data visualization systems applied to cultural heritage Adobe After Effects To make animation for video mapping Whatchout/ Pandora's Box or Resolume Arena To manage live performance Codecs Video mapping: choosing the right codec is very important in the context of the software we use; it also needs to be working in real time and to sync audio and video Computers Powerful computer with good graphics cards (desktop rather than laptop) Right cables Video mapping: to send the signal between computer(s) and projector(s) Source: Self-authored, 2024 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 39 3 Methodology of user research in the IMPULSE project - characteristics This section of the document outlines the fundamental assumptions, methodology, and approach employed in the examination of all user groups considered within the IMPULSE framework. This encompasses the heterogeneous user groups within the overarching categories of G1, G2, and G3, in addition to the two subgroups selected from G1 and G2: non-users and experts, which are classified within both G1 and G2. The research methodology employed in the IMPULSE project is based on a mixedmethods approach, incorporating both quantitative and qualitative techniques. The adopted perspective and interdisciplinary framework will also facilitate the development of a comprehensive, complementary, and interdisciplinary understanding of immersive environments, as well as the utilisation of digital cultural heritage resources explored across various contexts and by diverse user groups. This will involve identifying the conditions and characteristics associated with experiences in the metaverse and multi-user virtual environments. It is important to note that the proposed user research methodology is a construct derived exclusively from critical analysis of the literature, collective expert discussions, and scholarly considerations, which also incorporate the voices of practitioners. The proposed experimental phase in the research process represents an idea conceived prior to the development of various prototypes, with its credible validation expected to occur in subsequent phases of the IMPULSE project, once specific tools for immersion and the utilisation of digital cultural heritage are made available. The methodological proposal is intended to explore selected dimensions of immersive environment experiences and to identify informational needs and practices within virtual worlds, necessitating an evaluation of the usability of the proposed research methods and tools. In the initial studies conducted as part of the CAPHE project, only two methods—interviews and observations—were partially validated, which nonetheless allowed for the integration of certain findings into the methodology development process for IMPULSE, as well as their inclusion in created tools such as questionnaires. Furthermore, the IMPULSE project employs, as intended, the principles of inclusive design as a methodology, both in its own development processes and in the strategies to be implemented in the workshops. Similarly, the project aims to integrate diverse perspectives, voices, narratives, and access points within digital heritage collections. It is imperative to prioritize addressing disability in technology, as it serves as a gateway to vast knowledge and the development of competencies in a multitude of environments, including traditional and immersive ones. It is acknowledged that there are additional physical, embodied aspects of the user experience that extend beyond considerations of immersion, which must be addressed in conjunction with the technical and social elements in IMPULSE. In adapting Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 40 the research method to encompass these aspects and contexts, it would be preferable to utilise interviews and focus groups in lieu of an extended questionnaire, as they will facilitate a more effective and comprehensive capture of the pertinent issues. Similarly, a retrospective think-aloud protocol method will prove more efficacious than a questionnaire, which will facilitate the uncovering of the cognitive and affective aspects related to the fundamental issue. The optimal methodology for recognising and describing these aspects will be to combine the user and non-user groups in focus group interviews. Thus, the proposed methodology for investigating user experiences in immersive environments serves as a foundational framework for extensive analyses and the adaptation of tools, as well as the incorporation of novel methods for analysing needs, motivations, and behaviours. This approach encompasses iterative prototyping processes and considers the social, affective, and cognitive dimensions that shape VR experiences. In the experimental phase of the study, up to 50 selected participants (10-12 individuals from each four group) representing G1, G2, G3, and experts will engage in exploratory activities. It is anticipated that between 100 and 200 users will respond to the questionnaires distributed to the total number of participants from all users’ groups, primary from G1 and G2, as well subcategories of non-users, and experts. The initial sample size estimates are applicable to all user groups (G1, G2, G3, and selected from G1&G2 category of experts) to be studied in each IMPULSE partner country, where VR user experience research with the prototype will be conducted. This could result in a total number that is multiplied by the number of countries or institutions represented by the partners. Furthermore, it is important to note that the selection of research participants representing G1, G2, G3, non-users, and experts will take into account cultural differences, gender, and disabilities in accordance with the assumptions set forth by IMPULSE. The research results will be subjected to both quantitative and qualitative analysis. In regard to quantitative data analysis, we will utilise Bayesian statistics in conjunction with the Bootstrap method, which entails repeatedly sampling with replacement from the sample in order to estimate results. Similar statistical methods will be employed in the analysis of time series data, which includes the examination of decision-making and interaction times with the prototype. Furthermore, nonlinear estimation with permutations will be employed in the statistical analysis. 3.1 User research methods - proposal The methods identified through the critical analysis of existing literature and the subsequent scoping review were further verified and refined. These included Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 47 Figure 8, in turn, presents the research procedure plan exclusively for Group G3. In the case of this particular group, the survey technique will be excluded from the research procedure plan. It seems likely that a greater quantity of qualitative material will be obtained from interviews and focus groups, as well as observations of the participants' information practices and interactions in an immersive environment. The recruitment of research participants in Group G3 will be conducted through the convenience sampling method and by disseminating information about the research, which will be linked to invitations for interested users from various industries operating within CCSI. The research methods employed for this group are based on interviews and focus groups. It is assumed that this form of research will facilitate a more comprehensive understanding of user experiences in immersive environments, allow for the refinement of questions, elicit detailed responses, and facilitate inquiry into other aspects that may be indicated by interview participants. The G3 group is comprised of users with a high degree of diversity, representing CCSI. A questionnaire designed for this group would be of a very general nature and could prove to be time-consuming, as it would rely heavily on open-ended questions. It is therefore recommended that interviews and focus groups be used as the preferred method. Those users from G3 who are willing and interested in participating in research involving experiments and the use of IMPULSE prototypes will be invited to take part in this research phase. Furthermore, this phase incorporates the survey technique, the observation method, and potentially the retrospective think-aloud protocol. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 48 Fig. 8. Visualisation of the research process for G3 Source: Self-authored, 2024 The research procedure for the group of experts selected from G1 and G2, that is visualised on Figure 9, based on the analysis of questionnaires will include both experiments and a combination of interviews, focus groups, and observations. In the initial phase of the procedure applied to this group, the survey questionnaire will serve two purposes: firstly, to select participants for the expert group, and secondly, to identify participants willing to engage in the subsequent stages of user experience research conducted in the form of experiments. In this case, it will be necessary to provide an email contact in the final question of the questionnaire and to give the appropriate consents. The research procedure will include both experiments, in light of the prototype's construction and subsequent verification of its functionality and applicability in the context of virtual worlds and cultural heritage resources within two proper pilot studies, and interviews, focus groups, and observations. The latter will be conducted with the selected users, who will be contacted via email to invite them to participate in the research. Furthermore, if feasible, the observation method will be employed during artistic and creative practices, interactions, and experiences in immersive environments that are utilised in the course of artistic activities. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 49 Fig. 9. Visualisation of the research process experts Source: Self-authored, 2024 3.1.2 Experiment Following the verification of users through quantitative and qualitative analysis using survey questionnaires, an experiment utilising the prototype developed within the project is planned. In this phase of the research, the participants should represent G1 and G2, as well as G3 and experts, which have been selected in a fair manner, considering cultural differences, gender, and disabilities in accordance with IMPULSE’s assumptions. The involvement of the participants in this particular aspect of the study Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 50 is contingent upon their active engagement with virtual reality (VR), the creation of immersive environments, a demonstrated expertise in the field or a representation of the creative computerized simulation and interactive entertainment (CCSI) industry. They should be artists, students and academics, as determined by the analysis of the initial questionnaire (general) and the acquisition of email addresses to contact the participants, as well as the invitation to take part in the experiment and the corresponding consents to participate in the experiment. It is anticipated that the entire experimental process will be recorded using a range of technologies and methods, including MS Teams and transcription tools, cameras, recorders, voice tracers and immersion recording. An additional tool at this stage of the research will be a specially constructed survey divided into three parts: 1) pre-experimental phase; 2) during the experiment phase; and 3) post-experiment phase. The pre-experimental phase involves preparing participants for the study, checking technical settings of devices and prototypes, and completing the initial section of the questionnaire with general questions to identify basic user needs and experiences. The during-experiment phase signifies the period of prototype testing and research trial execution, where participants will use goggles and undergo immersion into a virtual environment. During this phase, participants will complete sections of the questionnaire and perform designated tasks according to predefined activity scenarios and immersion experiences using the prototype. This phase aims to identify selected dimensions that will be assessed through various types of questionnaires commonly used in studies of users in immersive environments, as identified in the scoping review, along with appropriate scales for quantitatively evaluating hierarchy and compliance levels. Detailed questionnaire categories and scales used in constructing the survey are outlined in subsection 3.2.2 (p. 85). Proper scenarios and tasks will depend on the prototype's capabilities and content. At this juncture, two pilot studies—one to refine the prototype and another to validate the tool for rigorous testing with the final, correct prototype—will be conducted. The findings from these studies will serve as the foundation for IMPULSE. The attention will be given to the timing of task completion by participants, as well as mitigating any factors that could disrupt immersion and the proper experience of the prototype environment. It is planned to remove the goggles after each task for participants to complete the questionnaire. There is also the possibility of developing a tool for completing surveys in the immersive environment. However, challenges may arise with the option of survey completion by the experiment conductor; presenting questions and responses to participants could not only lengthen the experiment but also lead to misunderstandings or concerns arising from answering questions aloud. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 51 The questionnaire completion element during the experiment (clearly, this has been subjected to preliminary investigation, with modifications made according to the outcomes of the pilot studies) is crucial stage for verification during preliminary studies and evaluation by participating partners. Subsequently, after completing the experimental phase and switching off the prototype or when participants exit the immersive environment, the third part of the survey questionnaire is envisaged. This section includes specific scales analysing selected dimensions, feelings, reflections, and post-experiment states, providing retrospective insights into experiences during the experiment and immersion phase. The experiment serves as a tool to diagnose and understand human interaction with immersive environments and digital cultural heritage objects, and it is considered a type of behavioural experiment (Aguilar et al., 2024). It will be understood as a verifiable and modifiable concept aimed at repeatable and scalable experimental actions using VR environments. It will consist of a prototype-based proposal, focusing crucially on exploring the context in which immersion and user experience occur. For the organisation of the entire experiment, which requires a prototype (or in the preliminary phase may rely on available immersive worlds and cultural heritage objects usable for testing the adopted methods and tools), it is necessary to develop individual elements ensuring the quality, correctness, and reliability of the experiment. Essential aspects to be developed within the research (including questionnaire utilisation and experiment execution) include: 1. Management Plan: This encompasses participant acquisition, immersion process, incentivisation of participation (considering possible lack of adequate payment or other incentives), as well as progress monitoring. 2. Documentation: This involves a registration plan, location of procedure execution, experiment protocol, consent forms (e.g., for recording). 3. Infrastructure: Preparation of recording equipment, access to the prototype, prototype content, equipment for both experiment conductors and participants, and ensuring suitable network connectivity (fast internet connection) and other local equipment. 4. Data Collection: This includes experiment content, scenarios, tasks, scoring systems, experiment flow, data collection methods, and integration of various experiment elements. 5. Data Management: Assembly of experiment data, infrastructure data, sensor data, and data pre-processing involving anonymisation procedures. 6. Data Analysis: This encompasses data inspection, validation, editing, visualisation techniques, and statistical and quality assessments. The series of experiments (the aforementioned procedures were conducted during the preliminary studies and the verification of the prototype, as well as during the actual pilot studies) with the prototype will not only be repeatable but will also require adjustments after the prototype's deployment, including checks and verification during stages of refinement. A complete experimental procedure may not be necessary Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 52 at that point; instead, the focus will be on testing selected elements identified during preliminary studies or those not functioning correctly, as well as addressing any technological difficulties (such as usability and functionality issues with the prototype or equipment), and addressing personal reasons (such as interruptions due to psychophysical problems or participant non-attendance due to unforeseen circumstances). Scenarios will be tailored to match what is envisaged in the prototype, its functionalities, and user experiences. Feedback and evaluation of results from studies (related to these aspects) after testing the initial prototype version should be planned. Therefore, it is crucial to collectively develop tasks, scenarios, procedures, and necessary information for applying the prototype in the experiment. It is also essential to integrate efforts aimed at developing data collection procedures with other partners and WP2, and to utilise a universally accepted data management plan. Additionally, developing and refining the experiment protocol after preliminary studies and during proper pilot studies (2 phases of VR platforms evaluation) with prototype creation and deployment will be vital. For preliminary studies, which are also part of this methodology and research process, leveraging available VR technologies and platforms used in other projects, such as CAPHE with accessible content (e.g., Spatial), will be considered. 3.1.3 Interviews In relation to categorising participants in the research process during the analysis of results from the first type of questionnaire and identifying categories such as users, experts and non-users from all studied groups, interviews have been planned for implementation. In G3, these will be prioritised given that it is a highly specific, creative user group that, due to time constraints, the nature of their work, and the need for prompt action, may not complete the survey questionnaire. A more efficacious methodology for elucidating preferences, necessities, expectations, modes of experiencing immersive environments, and identifying the optimal utilisation of digital cultural heritage objects would be to employ interviews and focus groups. The schema for in-depth interviews, along with a potential set of questions for selected participants willing to participate in the study, utilising this method, has been included in Appendices 5.3 (p.148) and 5.4 (p.154). The interview method has also been incorporated into the research methodology schema, visualised in Figures 5-9, as an additional component among the methods planned for application across the user from G1, G2, G3, non-users and experts. Alongside surveys and experiments, it constitutes one of the primary methods for obtaining qualitative data. Interviews will be conducted, similar to the initial studies carried out in the CAPHE project, even before the prototype is developed. This is to gather insights into the capabilities, preferences, and experiences of users, particularly concerning the technologies used and immersion experiences in virtual environments. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 53 Participants will include active users of these spaces, experts, artists, students, or academic teachers with qualifications and experience in XR applications. Interviews may also be used during the prototype testing phase to refine conditions, characteristics, and motivations of user experiences, and to gain a deeper understanding of needs, expectations, and potential for utilising metaverse and digital cultural heritage. 3.1.4 Focus groups interviews For experts and users recruited using convenience sampling, a non-random sampling method (Galloway, 2005), participants are chosen based on the researcher's easy access (temporally and spatially) and the likelihood of their willingness to participate (due to personal acquaintance or knowledge of their qualifications, competencies, and experience, which is crucial in the Creative and Cultural Industries sector and for artists with specific specialisations). Additionally, this recruitment method for focus group studies and interviews can be instrumental in capturing a range of attitudes, opinions, and initial hypotheses that can be rigorously tested in subsequent stages of research. An interesting approach would be to combine users from G1, G2, G3, experts and nonusers in the focus groups interviews sessions. Bringing together these two groups, observing their communication processes, knowledge transfer, perceptions, and understanding of the metaverse and immersive vs. traditional worlds would provide significant dynamics in focus group interviews. Questions would be broad, and expectations would involve comments and references to insights gained during discussions, shaping or deepening the knowledge of users and non-users. For groups of artists and experts, the targeted method would be focussing group interviews, conducted either face-to-face or online (using platforms such as Microsoft Teams or Google Meet). Questions used in these interviews should be based on schemas developed after initial studies in CAPHE or questions designed for general interviews with users and non-users from Group 1. Furthermore, conducting initial analyses and studies with various specialists and users experienced in the metaverse and other virtual environments at different stages f the IMPULSE project will allow for modifications and additional questions. These could expand knowledge about needs, experiences, perceptions of immersive environments, and the utilization of digital cultural heritage resources. The iterative approach to project management, including research processes and continuous refinement of user research methodologies, will enable cyclic verification and the development of a framework for studying UX in the metaverse, adapting to rapidly evolving XR technologies. The scheme for focus groups interview is available in Appendix 5.3 (p. 148). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 54 3.1.5 Observation The observation method is designed to address three key areas: firstly, the reactions of VR users during the experiment; secondly, the ways in which artists react and experience VR when carrying out their creative work using immersive environments; and thirdly, the reactions and behaviours of students and academics when enabling immersion, both when using the prototype and, for example, during teaching activities using VR. The objective of the observation is to identify the factors that contribute to the experience of immersion in metaverse. This will be achieved by monitoring the behaviours, reactions and interactions undertaken by participants during their immersion in a virtual environment using prototypes, as well as during their use of other immersive environments. This will include observations made during the preparation of participants for the experiment, during the experiment itself and after the experimental study has concluded. The term observation can also be used to describe research conducted within a group of artists and CCI representatives, during which the participants' reactions and interactions with digital cultural heritage objects in an immersive environment, behaviour and activities during artistic performances, and other forms of artistic activity are monitored. In the latter case, we intend to integrate observation as extensively as possible into the contextual inquiry procedure, which also encompasses the option of posing questions during the observation or conducting a subsequent interview. Obtaining such direct, oral information from the respondents will facilitate the verification and supplementation of the observations made during the observation. The results of the observations will permit the correlation between the data collected, recordings, transcriptions, questionnaires and the results produced by the observations to be verified. A comprehensive overview of the various stages and elements to be considered during the observation process can be found in Appendix 5.4 (p. 154). 3.1.6 Additional methods: Think-aloud protocol An additional method that is feasible for the project and will be a complementary element of the experiment, as well as the questionnaire to be used during this phase of user research, is the think-aloud protocol. This method will require the use of the prototype under development. The method will be implemented following the completion of the tasks and immersion in the prototype environment. It will serve to complement the analysis of the forms of perception, the perception of this environment, the activities undertaken by the user during the experiment, as well as their emotions and their specific narration of thoughts. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 55 Think-aloud protocol (TAP) comes in three versions: simultaneous, retrospective and co-discovery based on the conversation of the participants. In the IMPULSE project, a retrospective version is being considered, which would involve verbalisation of the user's thoughts, actions and feelings while watching a recording of a previously performed task or set of tasks. In the simultaneous version, the verbalisation is done while the activity is being performed, so the simultaneous version would affect the subjects' immersion. In the retrospective version, there is no such impact on immersion, but the duration of a single test is increased. However, it still is necessary to test the retrospective version of the TAP during the pre-testing of the research procedure, including the estimation of the time of a single study and the determination of the order of the procedures used (TAP versus questionnaire survey completed after the experiment). 3.2 The IMPULSE project research tools The following represents an overview of the fundamental tools, questionnaires, interviews and observations that serve as the basis for the proposed user experience (UX) research methodology. It also serves as a foundation for the ongoing examination of specific tasks, the formulation of assumptions and the delineation of various facets of user experience within the metaverse. 3.2.1 Survey questionnaires The online survey questionnaires, provided by all WP1 partners using appropriate tools in collaboration with WP5, have been meticulously developed to align with the user categorisation framework within the IMPULSE project. This framework differentiates between four categories of users: G1, G2, experts, and non-users. Consequently, different questionnaires have been designated for each of these categories. They function as instruments to investigate predetermined domains concerning needs, experiences, stimuli, behaviours, and information practices, while also encompassing affective and cognitive dimensions. These questionnaires aim to uncover the individual mental models (mental representations) of immersive environments held by users, enabling precise adjustments not only to prototype content but also fostering a comprehensive understanding of the conditions and components influencing virtual space experiences and interactions within immersive frameworks alongside their co-participants and objects. The included questions serve as foundational elements for identifying users and non-users. Furthermore, they serve as the initial phase for identifying factors influencing the inactive engagement with immersive worlds or their rejection and non- Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 56 utilisation, as well as highlighting inhibiting determinants such as lack of motivation, knowledge, or skills. The questionnaires have been tailored for: a) general user cohort representing the project's primary user groups: students, academic teachers, artists, and students and teachers from art schools. b) selected group of participants involved in the experiment, primarily targeting Group 2 comprising artists (including selected students or academic teachers from Group 1 with experience in using immersive environments who have expressed willingness and consent to participate in this segment of the study). c) non-users identified through specific questions in the general questionnaire and directed to a dedicated form tailored for this subset of participants. It was decided that users representing G3 would not be included in the survey questionnaire due to the specific nature of their work and activities, as well as the individuality of the CCSI sector. For this group, alternative research methods, as outlined in the methodological proposal, are to be employed. The structure, sections, and specific questions for all types of proposed questionnaires are detailed in the Appendix 5.2 (p.123). 3.2.1.1 General Questionnaire This questionnaire is designed for users from G1 and G2. These comprising students and academics (G1), encompassing general, exploratory, and educational/didactic dimensions and artists, students and academic teachers from art schools (G2). It is anonymous, with no requirement to provide an email address, allowing for potential invitations to participate in future research experiments involving prototypes. The questionnaire will be disseminated across universities and art schools, as well as within the artistic community. It is applicable not only to Group 1 but also to Group 2, serving as an exploratory survey in addition to those used for targeted groups (utilising a convenience sampling method). Selected elements, types of questionnaires, and scales (described in subsection 3.2.2, p. 85) are incorporated within this survey, which are also reflected in the questionnaires intended for experiment participants and non-users. This questionnaire primarily focuses on educational and didactic aspects, exploring user experiences, perceptions of VR technology, immersion, and various dimensions related to VR tool preferences. It comprises 7 sets of closed and open-ended questions, categorised into sections that address overarching aspects of user experience dimensions in VR. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 63 of the metaverse, and people who are not active participants in immersive spaces and create art or produce artistic creations without using VR (in a traditional way), but also those who use digital cultural heritage objects without using VR (and its implementation into artistic work). The interview scheme for this group (G2 and G3) is therefore applicable to the focus interviews. It may also be appropriate to conduct interviews and focus groups with the non-users' group, selected from G1 and G2, in order to gain further insight. Interview structure for experts and artists, also in focus group, it consisted of five main questions on VR platforms and tools, their advantages and disadvantages, ways of interacting in VR with heritage objects, the difficulties with importing 2D or 3D objects into VR and the size of the VR environment. In addition, a question to non-users about their reasons for not using VR was provided. The interview schema concludes with questions that are specific to the subject, which are based on previous observation or information obtained about the respondent. The full interview scheme can be found in Appendix 5.3 (p. 148), whereas only the main questions from the general pool for all respondents are presented below: • What kind of VR platforms (e.g. Spatial) and tools (e.g. VR goggles) are you using in your work? Additional questions relate to the type of VR most commonly used, the hardware used, participation, immersion and use of metaverse environments, multi-user gaming environments, and the mode of interaction that is preferred by the user. 1. From your perspective: what are the advantages and disadvantages of those VR platforms and tools that you mentioned in terms of your artistic needs? In addition, questions will relate to the usefulness of virtual environments in creative work, speciality, industry, the possibility of implementing metaverse to accomplish tasks, preferences of VR aspects and emotional approach to VR, and possible problems and inconveniences in using VR. 2. Do you use VR mainly to inspect cultural heritage content (or other types of content) or do you also interact with / transform / manipulate the content? If you interact with the content, what type of interaction do you usually engage in? 3. Do you notice any issues related to importing 2D / 3D content onto VR platforms and/or VR development software / tools? 4. What is the average size (area) of the virtual environments you usually enter? (e.g. room-sized, building-sized, more expansive)? 5. (additional) *[IF NOT USING VR] Why have you not been interested in using VR up till now? [not applicable if the audience are VR experts] In addition, the resulting interview scheme is intended for a group of users to identify the potential of the educational dimension of the metaverse environment, the use of digital cultural heritage resources for teaching purposes, it is also intended for a possible group of artists who are attempting to use immersive technologies Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 64 in creative work, to check the potential that such an immersion environment brings, and perceptions about the use of digital heritage. The group will include student users as well as academic staff and artists. The investigation of the perceptions associated with the various dimensions of VR, including those related to creativity, immersion, and educational elements, will facilitate the verification of the ways in which this technology is understood, perceived, and experienced in terms of immersion. The structure is based on four demographic questions (relating to age, gender, education level and field of studies or artistic specialisation) and five main questions on specific aspects, particularly related to didactics and, in the case of artists, to the creative use of VR. In the table (available in Appendix 5.3.2, p. 150), the questions posed to the artists are highlighted in colour. The group of questions included those that relate to the following dimensions and objectives: 1. exposure and awareness, which goal is gauge basic awareness and indirect exposure to VR technologies. 2. perceived relevance, which goal is to understand perceived potential uses of VR in academic contexts without prior direct experience. 3. barriers to adoption that should identify barriers or lack of interest regarding VR use, 4. perception of technology which goal is to explore attitudes towards adopting new technologies and specific thoughts on VR's user-friendliness. In the case of a group of artists, the aim is to indicate the potential, aptitude, competence and perspective of the application of new VR, XR technologies in art, as well as to highlight the potential for experimentation with and in the immersive environment of artists. 5. collaborative potential which goal is to elicit thoughts on the transformative potential of VR in educational settings. Each of these questions is accompanied by a series of more specific, guiding questions. With regard to the dimension pertaining to exposure and awareness, the subsequent questions seek to ascertain the definition of the virtual environment, the extent of familiarity with it, the utilisation of VR technologies in any capacity, and, in the case of artists, the incorporation of VR technologies in artistic and creative work. The dimension related to perceived relevance will be identified through a question that primarily concerns the relevance of VR technologies in education or training (the potential for using VR in teaching methods), enhancing learning or collaboration, as well as the beneficiary aspect of VR in education (e.g. in specific disciplines) or creative activities. The question on barriers will seek to identify the principal reasons for the non-use of VR, rejection of VR, and the lack of acceptance of VR technologies in teaching or artistic work. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 65 A further question will be posed in order to ascertain perceptions of new immersive technologies, the view of technological advancements and possible aspects that could enhance the use of VR in both teaching and artistic work. Artists should indicate the distinctive possibilities for artistic practice and experimentation. In the context of evaluating the potential for collaboration, it is essential to identify the ways in which virtual reality (VR) could influence collaborative workspaces and learning environments, as well as to understand the perceived benefits of the metaverse among users. In the context of artists, it will be important to gain knowledge of the interactions and possible ways in which the immersive environment can be experienced collectively. This will enable us to recognise changes in the way artists and audiences perceive and interact with art, as well as to gain unique group experiences. Do you believe that VR and immersive installations can provide artists with benefits that other art forms cannot? The complete interview schedule for G1, G2, G3, can be found in Appendix 5.3 (p. 148). Interview scheme for non-users (selected from G1 and G2) can be found in Appendix 5.3.2 (p. 149). 3.2.4 Observation schemes The observation schema primarily pertains to experiments and other events, including public ones, utilising VR technology. The observation schema includes details such as the date, observation point, observation code, and the start and end times of the observation. Subsequently, the data collected pertains to the event type, including the following categories: a) the type of VR event, e.g., VR opera, VR performance, VR museum exhibition; b) preparation for the event or activities related to the event; c) lecture; d) conference presentation; e) workshop. The specific elements and types of events may be added as required, depending on the circumstances and the nature of the event in question. Further information pertains to the space in which the event is held, encompassing the type of virtual reality (VR) space and its general level of sophistication. This includes spatial, single-user, multi-user, and other forms of VR. Furthermore, additional details encompass aspects of the virtual environment, including, for example, a classroom, a concert hall situated on a beach, or an amphitheatre. Furthermore, consideration is given to the real-world space in question, including the type of room and location, for example, whether it is outdoors. Another element of observation concerns the users representing specific artistic disciplines, artists. The group designated as G2 may comprise a variety of individuals, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 66 including musicians, vocalists, dancers, conductors, audience members, and other relevant parties, similar in composition to the group of artists who participated in the CAPHE initial studies. The types of participants are determined solely from these initial studies and tool verification during conferences and XR events within CAPHE. Types of actors can be modified and added as necessary based on the observed event. The observation schema also considers the context of observers without VR goggles. This part of the schema includes elements for gathering information about objects in VR, such as sculptures, paintings, musical instruments, paints, brushes, landmarks, and other interactive elements in the world, such as benches, chairs, stairs, and doors. Additional types of objects can be freely added. The second part of the observed elements pertains to objects in reality, such as VR goggles, controllers, pads, haptic gloves, treadmills, steering wheels, moving seats, regular chairs or seats. Individual observed elements can be added as needed. The next dimension concerns actions. The observation schema includes elements that researchers should pay attention to and gather relevant data on. These include the occurrence of communication between users and the manner of communication in VR. It examines whether there is a process of using virtual information and tools by VR users, which tools and information are used, and when. Additionally, the observer needs to ascertain whether the use of tools and information is easy/intuitive, requires concentration and time, or presents such problems that users repeat actions or abandon their intended actions. Other aspects to observe include interactions with objects, such as lifting a vase, pouring paint onto a sculpture, zooming in or rotating an object in space, touching a piano, conducting with a baton, stopping at an object, or not interacting with objects beyond observing them according to the sample (used during preliminary studies) or proper (used during pilot studies) scenarios. The manner in which users move in the virtual environment is crucial, including walking, flying, teleporting, experiencing a rollercoaster ride, and others. Movement in physical space pertains to moving (walking, bending, jumping, etc.) within a designated space without using additional equipment; moving on a treadmill or using other equipment; or no movement other than, for example, moving the head and turning on a chair. The observation schema includes time to link with activities or acts and actors and objects in the context of consecutive or simultaneous events. The recording structure is constituted of a number of elements, which can be broadly classified under the following headings: The aforementioned structure comprises a series of interconnected elements, including actors, activities, objects, and actors. This intricate Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 67 network of relationships can be further delineated as follows: (2) actors – activities – actors – objects. Description scheme is: <actors> - <activities> <actors><objects> - - (2) <actors> - <activities> - <actors><objects>. Furthermore, the observation may encompass dimensions pertaining to disconnection, immersion, embodiment, and the manner in which these phenomena occur, including the identification of which of the user's senses are detached from the external environment and redirected towards the virtual reality (VR) domain. Furthermore, it is possible to observe the reactions of the VR user, which is recommended for further exploration during interviews, if feasible. Furthermore, additional dimensions pertaining to objectives and sentiments were incorporated into the assessment of activities and temporal elements. These should be subjected to scrutiny during an eventual interview. The complete observation scheme is available in Appendix 5.4 (p. 154). 3.3 Potential problems and barriers to conducting research During deliberations on research methods and tools, as well as during initial studies in CAPHE, significant difficulties, obstacles, and issues arose due to various conditions or factors often beyond the control of researchers or participants. Among the most critical issues to consider when using the proposed methods and tools, as well as conducting preliminary studies, are: Organisational problems, including, for example: • Insufficient time intervals between experiments, • Inadequate number of researchers to conduct studies within specified time frames, • Lack of VR laboratories, appropriate equipment, and space for conducting studies, including interviews, experiments, and observations, • Limited access to VR laboratories due to scheduling conflicts with educational activities, • Inadequate number of participants, • Recruitment challenges, • Lack of prepared language versions for specific participant groups from partner countries of the project, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 68 • Withdrawal of consent to participate in the study or incomplete informed consent forms for participation and use of research outcomes, • Overly extensive questionnaire or excessively long duration of interviews, including focus group interviews. Technical issues, including: • Equipment failures, • Inadequate internet connectivity (reliable Wi-Fi and wired connections, especially if participants outside VR goggles use desktop computers), • Insufficient testing of research equipment (from VR goggles to voice recorders and cameras for recording studies), • Data loss (it is essential to establish where and how research data will be stored, backup copies, and data security measures to prevent unauthorized access, including sensitive data), • Technical problems during questionnaire completion. Behavioural and psycho-social issues, such as: • Participant boredom and reluctance, • Psycho-physical problems during experiments (including visual and motor disabilities), • Sudden onset of symptoms like motion sickness in participants, • Issues arising from mismatched equipment and digital resources with user preferences, • Inappropriate way of selection of user groups for specific stages of the research process (for example, this may result from an erroneous analysis of the questionnaire results, the failure to reach a particular user group, an inadequately conducted questionnaire dissemination process, and other factors) Communication barriers, such as: • Inadequate researcher conduct (leading responses, rushing, creating a nervous atmosphere). It will be necessary to develop remedial procedures, consider potential issues, and reconsider the ways in which specific methods and tools are used, along with the development and implementation of prototypes envisaged in the IMPULSE project. Furthermore, preliminary studies serve twofold purposes: firstly, to corroborate the proposed methodology and secondly, to identify any potential issues that may arise during the research process. These preliminary studies also facilitate the identification of challenges pertaining to technology, equipment, participant numbers and organisational issues. Additionally, they assist in the identification of contextual issues and deficiencies in the developed research procedure, experiment and other Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 69 aspects of the study. A process of continuous improvement will be implemented to address potential issues that may arise during the preliminary studies and the subsequent main studies, as well as proper pilot studies. This will ensure the smooth and effective conduct of user experience studies within the IMPULSE framework. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 70 4 References Aguilar, L., Gath-Morad, M., Grübel, J., Ermatinger, J., Zhao, H., Wehrli, S., Sumner, R. W., Zhang, C., Helbing, D. & Hölscher, C. (2024). Experiments as Code and its application to VR studies in human-building interaction. Scientific Reports, 14(1), Article 9883. https://doi.org/10.1038/s41598-024-60791-3 Belkin, N.J., Oddy, R.N., Brooks, H.M. (1982). ASK for information retrieval: Part I. Background and theory. Journal of Documentation, 38(2), s. 61–71 Bower, M., Lee, M. J. W., & Dalgarno, B. (2017). Collaborative learning across physical and virtual worlds: Factors supporting and constraining learners in a blended reality environment. British Journal of Educational Technology, 48(2), 407–430. https://doi.org/10.1111/bjet.12435 CAPHE (2024). International Conference “Enhancing Artistic Experience in Hybrid Environments” 15-17 May, La Spezia, Italy. Available: https://www.caphe.space/international-conference-enhancing-artisticexperience-in-hybrid-environments-15-17-may-la-spezia-italy/ CAPHE (2024a). Architecture in VR. Opera set stage design. Available: https://www.caphe.space/architecture-in-vr-opera-set-stage-design/ CAPHE (2024b). XR Festival Florence 2024. Available: https://www.caphe.space/xrfestival-florence-2024/ Cecotti, H. (2022). Cultural Heritage in Fully Immersive Virtual Reality. Virtual Worlds, 1(1), 82–102. https://doi.org/10.3390/virtualworlds1010006 Dervin, B. (2000). Chaos, order and sense-making: A proposed theory for information design. W: R. Jacobson (red.). Information Design. Cambridge, Massachusetts, London: MIT Press, s. 35–57 Dupont, L., Hubert, J., Guidat, C., & Camargo, M. (2019). Understanding user representations, a new development path for supporting Smart City policy: Evaluation of the electric car use in Lorraine Region. Understanding Smart Cities: Innovation Ecosystems, Technological Advancements, and Societal Challenges, 142, 333–346. https://doi.org/10.1016/j.techfore.2018.10.027 Drummond, J. S., & Themessl-Huber, M. (2007). The cyclical process of action research: The contribution of Gilles Deleuze. Action Research, 5(4), 430-448. Available: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=36abc9a0e24 f7e34a3a3af4300fc7bbed6b1a9a7 Ford, N. (2015). Introduction to Information Behaviour. London: Facet Publishing Galloway, A. (2005). Non-Probability Sampling. In K. Kempf-Leonard (Ed.), Encyclopedia of Social Measurement, Elsevier, 2005, Pages 859-864, https://doi.org/10.1016/B012-369398-5/00382-0 Gonzalez-Franco, M., & Peck, T. C. (2018). Avatar Embodiment. Towards a Standardized Questionnaire. Frontiers in Robotics and AI, 5. https://doi.org/10.3389/frobt.2018.00074 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 71 Greenwood, D.J. (2018). Action Research. In: Ciesielska, M., Jemielniak, D. (Eds.), Qualitative Methodologies in Organization Studies. Palgrave Macmillan, Cham. (p. 75-98). https://doi.org/10.1007/978-3-319-65217-7_5. Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In G. Szwillus & J. Ziegler (Eds.), Mensch & Computer 2003 (Vol. 57, pp. 187–196). Vieweg+Teubner Verlag. https://doi.org/10.1007/978-3-322-80058-9_19 Hudson, M., & Cairns, P. (2014). Chapter 6 Measuring Social Presence in Team-Based Digital Games. In: G. Riva, J. Waterworth, & D. Murray (Eds.), Interacting with Presence: HCI and the Sense of Presence in Computer-mediated Environments (pp. 83– 101). DE GRUYTER OPEN. https://doi.org/10.2478/9783110409697.6 IMPULSE Project proposal (2023). IMPULSE, IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies. PROPOSAL_101132704-IMPULSEHORIZON-CL2-2023-HERITAGE-01-PART_B_Section_1.pdf (Available: https://teams.microsoft.com/v2/) Kuhlthau, C.C. (1993). A principle of uncertainty for information seeking. Journal of Documentation 49(4), s. 339–355 Lim, C. P., Nonis, D., & Hedberg, J. (2006). Gaming in a 3D multiuser virtual environment: Engaging students in science lessons. British Journal of Educational Technology, 37(2), 211–231. https://doi.org/10.1111/j.1467-8535.2006.00531.x McTaggart, R., Nixon, R., & Kemmis, S. (2017). Critical participatory action research. In: L. L. Rowell, C. D. Bruce, J. M. Shosh, M. M. Riel (Eds.). The Palgrave international handbook of action research. Palgrave Macmillan New York, 21-35. Minge, M., Thüring, M., & Wagner, I. (2016). Developing and Validating an English Version of the meCUE Questionnaire for Measuring User Experience. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 2063–2067. https://doi.org/10.1177/1541931213601468 Minge, M., Thüring, M., Wagner, I., & Kuhr, C. V. (2017). The meCUE Questionnaire: A Modular Tool for Measuring User Experience. In M. Soares, C. Falcão, & T. Z. Ahram (Eds.), Advances in Ergonomics Modeling, Usability & Special Populations (Vol. 486, pp. 115–128). Springer International Publishing. https://doi.org/10.1007/978-3-319-41685-4_11 Page, M.J. et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Review, 10, 89. https://doi.org/10.1186/s13643-021-01626-4 Rohrer, Ch. (2022). When to Use Which User-Experience Research Methods. NNgroup.com. Available: https://www.nngroup.com/articles/which-ux-researchmethods Ross, J. (2017). Speculative method in digital education research. Learning, Media and Technology, 42(2), 214-229 Savolainen, R. (2005). Everyday life information seeking. W: K.E. Fisher, S. Erdelez, L. McKechnie (red.). Theories of Information Behavior. Medford, N.J: Information Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 72 Today, Inc. (ASIST Monograph Series) s. 143–148 Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments, 10(3), 266–281. https://doi.org/10.1162/105474601300343603 Souza, L. B. O., Cunha, A. C. S., Lima, M. M. X. de, & Ricca, D. E. P. (2023). Pragmatic and hedonic aspects of user experience in Virtual Reality: Analysis of novice users’ information mediation during their first interaction with a Metaverse platform. InfoDesign - Revista Brasileira de Design Da Informação, 20(3). https://doi.org/10.51358/id.v20i3.1068 Tricco, A. C. et al. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 79 Fig. 1. User experience research methods in 3-dimensional framework. Source: Rohrer, Ch. (2022). When to Use Which User-Experience Research Methods. NNgroup, Nielsen Norman Group. Retrieved from: https://www.nngroup.com/articles/which-ux-research-methods/ (30 April 2024) An additional source for the creation of codes for deductive coding was the initial proposal of social survey methods in the context of the different user and non-user groups that were selected in the project. These groups included: 1) students, postgraduates and young researchers and academics; 2) artists and art schools; 3) representatives of the creative industries (games, film, animation, performance etc.). Figure 2 presents the selected four social survey methods in the context of the user groups and types of users that were also selected (Tomanek, 2024). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 80 Fig. 2. Survey Methods vs Target Groups draft. Source: Tomanek, K. (2024). Available at: https://teams.microsoft.com/v2/ Furthermore, an important aspect became the categorization of immersive technologies and virtual worlds proposed by the National and Kapodistrian University of Athens and the Spatial Media Research Group. This categorization emerged from a presentation discussed during the Impulse Plenary Meeting on 1st March 2024 by Dimitris Charitos (accessible at: https://teams.microsoft.com/v2/). In a later stage of coding and code verification, consultation and review of the correct application of parent and child codes for virtual environments, VR, Virtual worlds, MUVEs, CVEs, and key names of popular applications were required. Coding Based on these expert materials, knowledge, and established research objectives, an initial codebook was developed, containing the primary and secondary codes. Through thorough engagement with the text, reading, and familiarity with full texts, inductive coding was employed, progressively adding new codes in accordance with the individual interpretation of the text by the three researchers conducting thematic analysis in MaxQDA. This process involved selecting excerpts related to specific predetermined primary and secondary codes. This facilitated capturing the complexity and richness of qualitative data, as well as recognizing the multitude of methods, procedures, and forms of user research organization and their user experience (UX) in virtual Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 81 environments. The adopted logic and methodology of content analysis, as well as the application of inductive coding, also aim to prepare the groundwork for intensive, often innovative, and multifaceted explorations of the designated three user groups, their behaviours, needs, and experiences in immersive virtual spaces, which will be planned and conducted in later stages of the project. The code set was continuously updated, verified, and supplemented. Ultimately, the following codes were classified as overarching: aim of research, area, dimension, methodology, methods, organization of research, technology, users, and VR experience. Additionally, it was decided to specify embodiment, citation (content referred to in the state of research or discussion of the results), needs, terminology, tips for using VR/AR/XR, and VR therapy as overarching codes. Each overarching code is further subdivided into deductively-inductively defined sub-codes (secondary codes), which are then subdivided into additional tertiary codes. For example, codes related to the area include secondary codes, which are further divided into tertiary codes: for instance, Area > archaeology > archaeological sites, cultural heritage > digital heritage, or historical accuracy, historical authenticity, or tourism, and likewise design > fashion, furniture design, interior design. Among the secondary codes related to users, examples include codes for artists, children, creative industry, experts, students, teachers, and others, which are also subdivided into lower-level codes (e.g., creative industry > design professionals, film/movie creators, game creators, performance creators, VR or MUVE creators). It is noteworthy that in some cases referring to overarching codes such as methodology, methods, dimensions (in which the research was conducted or the authors of the analysed publications referred to the identified dimensions, perspectives identified in previous research, and which are important planes of VR experience research), or technology constitute very elaborate code trees with numerous sub-codes, including names of specific methods, techniques, scales used in questionnaire research, technologies, applications, names of specific virtual worlds and applications. Additionally, when reading the code tree and the number of codes present at this stage of thematic analysis, it is important to note that not all publications and resources subjected to coding yielded codes, necessitating continuous analyses related to key objectives and user groups, as well as methods for studying VR experience. During the process of inductive-deductive coding by individual researchers, which was subsequently integrated, refined, verified, and supplemented, a total of 482 codes were generated. Additionally, MaxQDA obtained 6798 codes based on keywords and field designations from the bibliography, which were directly imported from the Zotero library. The final, comprehensive list of codes and their categorization is located in the Appendix to Scoping review report (p. 98). It should be noted that the codes utilized and selected at this stage may continue to be verified and updated following any consultations and feedback that arise during the project implementation and will be addressed by the team participating in WP1 and individual tasks. In addition to inductive coding, lexical coding was employed in MaxQDA by using alternatives for specific selected codes (e.g., area > museums, area > cultural heritage, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 82 methods > diary method, methods > contextual inquiry, methods > desirability studies, methods > field studies) according to specified instructions, such as diary "feedback study" "feedback studies" "elicitation study" "elicitation studies", "contextual inquiry" "participant observation" "on-site study", "desirability studies" "desirability study" "desirability survey" "desirability research" "desirability toolkit" "desirability testing". Additionally, automatic coding was applied for selected codes, facilitated by the MaxQDA program. Results (in selection) This section delineates chosen outcomes derived from the thematic analysis and coding conducted through MaxQDA. It is imperative to underscore that the lexicon employed in this project necessitates additional scrutiny and elucidation. Standardising and solidifying the overarching terminology utilised within the project holds paramount significance for advancing user research initiatives. The absence of elucidation and definition identified during the literature review underscores the imperative for this endeavour. Due to the nature of Task 1.1, which primarily entails the determination of research and methods utilised in analysing the interactions of users and non-users with virtual reality (VR) through digital cultural heritage resources, the scope of this objective has been broadened to extend beyond cultural heritage, incorporating various domains and research methodologies concerning user experience (UX). Consequently, an initially adopted definition of user experience, as established by Don Norman and Jacob Nielsen, has been preliminarily embraced. Furthermore, the inductive coding employed also encompassed terminology relating to dimensions of experiences, behaviours, and interactions within virtual environments. The term "user experience" comprehensively encapsulates all aspects of an end-user's engagement with a company, its services, and its products (Norman & Nielsen, 1998). However, further clarification and expansion of this definition are deemed necessary. Given the substantial number of codes and the complexity of the code tree structure, the majority of findings have been visually represented below. As an example, Figure 3 depicts a tag cloud delineating the coded themes, highlighting the codes that occur most frequently, surpassing 10 instances. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 83 Fig. 3. Tags cloud of coded themes (coded above 10 instances). Source: Self-authored, 2024 For comparison, Figure 4 depicts a considerably less legible, overall tag cloud representing the entirety of applied codes, the complete code tree, during the process of coding and thematic analysis. Throughout this process, the most frequently occurring themes were recognized, addressing areas encompassing overarching codes such as aim of research, organization of research, methods, methodology, domain, dimension, users, needs, technologies, VR experience, as well as citation, embodiment, terminology, tips for using VR/AR/XR, and VR therapy. Fig. 4. The overall code map. Source: Self-authored, 2024 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 84 Primary code: Methodology The research methodology encompasses the strategic approach, toolkit, and structured plan of inquiry which, when implemented, aids in addressing the research problem by collating heterogeneous data, employing a spectrum of methods and techniques, interpreting the amassed data, conducting qualitative and quantitative analyses, and deriving conclusions from the research findings. It serves as a theoretical, conceptual, and empirical framework for conducting rigorous research. Consequently, the encoding of the most frequently emerging methodologies in the scrutinized publications was conducted. Among the most commonly used methodologies in the literature reviewed were, for example, quantitative confirmatory approach, triangulations, phenomenological approach, double stimulus methodologies, as well as FADGI (a collaborative effort started in 2007 by federal agencies to articulate common sustainable practices and guidelines for digitized and born digital; standardisation methodology) and photogrammetry methodology & process (a cheap, flexible and accurate solution to obtain 3D point clouds and textured models. The results of the coding were presented on Figure 5. The main advantage of this photogrammetric methodology is that a cloud of points and a precise mesh object with texture can be obtained at the same time (Besoain et al., 2021) Fig. 5. Tag code of commonly used methodologies in VR experience research. Source: Self-authored, 2024 The appearance of FADGI procedures and photogrammetry methodology and process earlier in the tag cloud indicates their relevance for standardisation and digitalisation, making them pivotal for prototyping and accomplishing the objectives of the IMPULSE project. However, they may not necessarily align with investigating user needs. Consequently, they were omitted from the subsequent visualization, enabling the identification of new, more frequently employed methodologies. These encompass, Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 85 among others, user-centred design, collaborative methodologies, research by design methodology, discursive methodology, and others, as illustrated in Figure 6. Fig. 6. The tag cloud pertaining to methodologies excluding procedures used in digitalisation and standardisation. Source: Self-authored, 2024 Primary code: Area/Domain Among the most frequently occurring codes related to area or domain, codes associated with cultural heritage emerged, which may also be attributed to the intention of acquiring specific resources related to user VR experience. Additionally, the studies focused on museums, which could result from a bias towards this type of literature and the application of VR in this area, as well as its placement by project partners in the Zotero library. Subsequent analyses may reveal additional or entirely different domains where VR is applied and research on VR experience is conducted. Furthermore, education, digital heritage, medicine, design, and archaeology were among the most frequent codes assigned. Visualization presented on Figure 7 depicts the most commonly occurring codes related to area/domain. Fig. 7. The code cloud pertaining to the dimensions in which the studies were conducted. Source: Self-authored, 2024 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 86 Primary code: Research objective – generalised Table 1 exclusively presents the selected research objectives identified and coded during the thematic analysis conducted. Studies conducted in the fields of art, including museums, cultural heritage, education, and design, were chosen for visualization. Each of the scrutinised studies featured intricate research objectives and referenced additional studies, case studies, projects, and analyses of diverse instances of VR technologies and VR experiences from varied users, necessitating in-depth analysis. Table 1. Selected research objectives in relation to research areas, users and research methods No. study research objective area users methods 1 (África et al., 2023) “The aim of the study is to ascertain the emotions triggered by a VR stimulus through the expressions on the e-WOM online reviews, the present research focuses more closely on the categorical theory of emotions such as that by Plutchik (1980), and consequently the following research question is formulated: RQ2: What are the emotions triggered by the VR headset stimulus regarding the Van Gogh VR experience?” Art, Museums Audience, General users Observation, Text-mining, Sentiment analysis 2 (Chung et al., 2024) “This study investigated the characteristics of visitor experience in relation to the spatial environment of VR exhibitions, particularly focusing on the representational fidelity of the real-physical world. We experimented with comparing the user experience in reality-based (realistically representing the physical world, high representational fidelity level) and virtuality-based (surreal, low representational fidelity level) VR exhibition settings Art, Museums > VR exhibition s General users Observation, Interviews, VR experiment, Questionnaire s > IPQ, Simulator Sickness Questionnaire, PPM Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 87 to explore design implementation.” 3 (Evans, 2019) “The focus of this paper is to assess what those barriers are, and whether they can be overcome in a manner that will allow for the potential of immersive educational experiences through VR to be realised in the near future. Understanding the barriers to VR as a concept could be investigated in several ways through empirical research. “ Education Experts, Creative industry > performanc e creators, films/movie creators Semi structured interviews, Field study, Clickstream / analytics 4 (Kari & Kosa, 2023) “The present study therefore aimed to explain the factors that drive the use and acceptance of VR games. We extended the hedonicmotivation system acceptance model with utilitarian and inconvenience factors to capture the pertinent features of VR systems more holistically.” Education gamers Questionnaire, covariancebased structural equation modelling, Clickstream / Analytics, Modelling methods, Longitudinal studies 5 (Lam, 2023) “This research examines factors that influence the artists’ decisions to use specific techniques in their processes. The findings of the research will discuss the potential of virtual reality in art creation and the creative process for such artists, with reference to their work.” art Artist > painters Case study 6 (Li et al., 2023) “This study aims to comprehensively understand the existing literature on immersive technology in museum exhibitions, focusing on virtual reality (VR), augmented reality (AR), and the visitor experience.” Art, Cultural heritage > digital heritage, tourism, Museums - Bibliometric analysis, Critical literature review, Clickstream / Analytics Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 88 7 (McveighSchultz et al., 2018) “We aimed to go beyond pain points and surface level desires and instead sought out opportunities to reimagine the broader problem space by exploring alternative interaction rituals for collaboration and creativity in VR.” Design > furniture design Creative industry > design professionals , Scientists, Engineers, Project managers Case study, Field study, Design methods, Experience prototyping, Interviews, Observation 8 (Verhulst et al., 2021) “To summarize, this study aims to help fill three gaps in the existing literature. The first, the lack of comparative work, is addressed by directly comparing the user experience of three different versions (one VR, two AR). The second, the lack of realworld studies, is addressed by using regular visitors to the National Gallery in London, UK, as participants. And the third, the lack of analysis of the drawbacks of immersive technologies, is addressed by including negative side effects (for example, nausea and feeling uncomfortable) in the outcome measures. The research question is: Do VR and AR versions of an immersive cultural experience engender different user perceptions of presence, engagement, enjoyment, and do they have negative side-effects?” Art, museums General users VR experiment, storytelling, Questionnaire (survey) > IPQ, presence questionnaire 9 (Wu & Kim, 2022) “In conclusion, the objective of this study was to explore users’ perceptions of technological features in VR and AR and analyse the advantages and disadvantages of technologies in fashion retailing.” Design > fashion Students, non-users > potential users Focus groups Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 95 Limitations This stage of WP1's work (scoping review) is the first step in developing a procedure for researching VR users and non-users as part of the IMPULSE project - it is an important step, but just an initial one. Using MAXQDA provides us with the flexibility to perform follow-up analyses, although the program has some limitations, so we plan to utilize more tools for data analysis and visualization (e.g., Python). Access to full texts of publications is limited - we have made efforts to obtain full texts that could not be downloaded via Zotero (e.g., we contacted authors of publications via Research Gate), nevertheless not all full texts could be obtained, so we also relied on abstracts. The Zotero library should have started to be created earlier, preferably from the initial work on the project. A review of the literature revealed terminological inconsistencies and instances where the methodology employed was not adequately described. In order to address these shortcomings, we conducted a search for information on selected procedures that were relevant to our research. This process will continue. Moreover, concerns include questionnaire purchase costs, participant recruitment (ethical and legal constraints) and the application of tools for physiological reaction research (issues of study supervision, uniform equipment, and software) - we lack the means to utilize methods such as FMRI, EEG, ECG, despite their application in VR UX research. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 96 References África, R. G., González-Rodríguez, M. R., & Díaz-Fernández, M. C. (2023). Salient features and emotions elicited from a virtual reality experience: the immersive Van Gogh exhibition. Quality & Quantity. https://doi.org/10.1007/s11135-023-01752-2 Besoain F, Jego L, Gallardo I. (2021). Developing a Virtual Museum: Experience from the Design and Creation Process. Information, 12, 6:244. https://doi.org/10.3390/info12060244 Chung, S. J., Kim, S. Y., & Kim, K. H. (2024). Comparison of visitor experiences of virtual reality exhibitions by spatial environment. International Journal of Human-Computer Studies, 181, 103145. https://doi.org/10.1016/j.ijhcs.2023.103145 Evans, L. (2019). Barriers to VR use in HE. In Proceedings of the Virtual and Augmented Reality to Enhance Learning and Teaching in Higher Education Conference 2018 (pp. 3–13). IM Publications Open LLP. https://doi.org/10.1255/vrar2018.ch2 Kari, T., & Kosa, M. (2023). Acceptance and use of virtual reality games: an extension of HMSAM. Virtual Reality, 27(3), 1585–1605. https://doi.org/10.1007/s10055-023-00749-4 Lam, C. K. (2023). Exploring virtual reality painting technology and its potential for artistic purposes [Master’s thesis, Nanyang Technological University]. https://dr.ntu.edu.sg/handle/10356/170485 Li, J., Wider, W., Ochiai, Y., & Fauzi, M. A. (2023). A bibliometric analysis of immersive technology in museum exhibitions: exploring user experience. Frontiers in Virtual Reality, 4. https://doi.org/10.3389/frvir.2023.1240562 Mak, S., Thomas, A. (2022). Steps for Conducting a Scoping Review. Journal of graduate medical education, 14(5), 565–567. https://doi.org/10.4300/JGME-D-22-00621.1 McVeigh-Schultz, J., Kreminski, M., Prasad, K., Hoberman, P., & Fisher, S. S. (2018). Immersive Design Fiction: Using VR to Prototype Speculative Interfaces and Interaction Rituals within a Virtual Storyworld. Proceedings of the 2018 Designing Interactive Systems Conference, 817–829. https://doi.org/10.1145/3196709.3196793 Norman, D., Nielsen, J. (1998). The Definition of User Experience (UX). NNgroup, Nielsen Norman Group. Retrieved from: https://www.nngroup.com/articles/definition-userexperience/ Page, M.J. et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Review 10, 89. https://doi.org/10.1186/s13643-021-016264 Project IMPULSE proposal (2023). IMPULSE, IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies. PROPOSAL_101132704-IMPULSE-HORIZON-CL22023-HERITAGE-01-PART_B_Section_1.pdf (Available: https://teams.microsoft.com/v2/) Rohrer, Ch. (2022). When to Use Which User-Experience Research Methods. NNgroup, Nielsen Norman Group. Retrieved from: https://www.nngroup.com/articles/which-uxresearch-methods/ (30 April, 2024) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 97 Tricco, A. C. et al. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850 Verhulst, I., Woods, A., Whittaker, L., Bennett, J., & Dalton, P. (2021). Do VR and AR versions of an immersive cultural experience engender different user experiences? Computers in Human Behavior, 125, 106951. Wu, Y. F., & Kim, E. Y. (2022). Users’ Perceptions of Technological Features in Augmented Reality (AR) and Virtual Reality (VR) in Fashion Retailing: A Qualitative Content Analysis. Mobile Information Systems, 2022, e3080280. https://doi.org/10.1155/2022/3080280 Young, G. W., O’Dwyer, N., Vargas, M. F., Donnell, R. M., & Smolic, A. (2023). Feel the Music!—Audience Experiences of Audio–Tactile Feedback in a Novel Virtual Reality Volumetric Music Video. Arts, 12(4), 156. https://doi.org/10.3390/arts12040156 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 98 Appendix to Scoping review report: List of Codes Codebook Memo Counts our codes: 6798 (all codes, with imports from Zotero: 11226) aim of research 316 area discipline, subject, object - e.g. medicine 5 archeology 6 archeological sites 6 architecture 4 urban design 2 art 46 immersive theatre 2 cultural heritage 155 digital heritage 46 historical accuracy accuracy is about ‘getting the historical facts correct’ (mochocki2021, P. 3) 3 historical authenticity authenticity is ‘about getting the experience and expectations of the past “right”’ (mochocki2021, P. 3) 22 tourism 10 design 1 fashion 24 furniture design 1 interior design 1 education 52 digital education 10 learning theories 0 constructivist learning 1 enquiry-based learning 1 games-based learning 1 situated learning 1 geography/cartography 5 Industry 4.0 4 media culture 1 medicine 25 museums 124 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 99 VR exhibitions 8 psychology 4 sport 5 citation content referred to in the state of research or discussion of the results 27 co-citation 12 combined citation 1 self-citation 1 dimension cognitive, social, affective, technology etc. - if applicable 15 behavioural 31 simulator/motion sickness VR sickness 273 cognitive 41 affordance 4 Affordance theory 94 cognitive load theory 1 expertise reversal level it refers to the reversal of the effectiveness of instructional techniques on learners with differing levels of prior knowledge. The primary recommendation that stems from the expertise reversal effect is that instructional design methods need to be adjusted as learners acquire more knowledge in a specific domain. Expertise is described as "the ability to perform fluently in a specific class of tasks". 1 mental models 1 mental workload mental ergonomics 3 Yerkes-Dodson law The law dictates that performance increases with physiological or mental arousal, but only up to a point. When levels of arousal become too high, performance decreases. The process is often illustrated graphically as a bell-shaped curve which increases and then decreases with higher levels of arousal. 1 cultural eg. western and eastern cultures; the way that people view the world around them based on their cultural background, experiences, and beliefs 6 educational 23 digital literacy 6 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 100 Experiential learning theory based on several fundamental models of experiential learning, including Lewin, Dewey, and Piaget, which basically refer to learning from experience or learning by doing. Learners immerse in a particular experience and reflect their experiences to develop new skills, attitudes, or ways of thinking (Lewis & Williams, 1994). Experiential learning is defined as “the process whereby knowledge is created through the transformation of experience. Knowledge results from combination of grasping and transforming experi-ence” (Kolb, 1984, p. 41). (fromm2021, P. 2) 2 exploratory 7 multidisciplinary perspective 6 narrative 18 storyworld ‘a storyworld can be fictional or non-fictional or have components of both’ (Schrier, Torner & Hammer 2018, p. 352). Its ‘mental representation’ is based on ‘three conditions . . . being logically consistent, large enough to stimulate the imagination, and experienced as complete’ (Ryan, 2019, p. 82). (mochocki2021, P. 13) 2 worldness the potential to generate a mental image of a believable storyworld 4 philosophical 3 psychological 34 affective 35 affective state 6 Emotion Matching (EM) Feeling as another person feels (kembe2022, P. 7) 5 Empathic Concern (EC) Feeling for another person who is in need (kembe2022, P. 7) 6 emotions 14 empathy 24 ARCS Model of Motivation The ARCS model (Keller 1983) is a motivational design process that includes a synthesis of motivational concepts and theories that are clustered into four categories: attention (A), relevance (R), confidence (C), and satisfaction (S). 1 comfort factor the extent in which VR offers a safe and protected space for learning 1 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 101 Imagine-other Perspective (IOP) Imagining how another person thinks or feels given his or her situation (kembe2022, P. 7) 6 Imagine-self Perspective (ISP) Imagining how one would think and feel in another’s situation or “shoes” (kembe2022, P. 7) 6 SDT self-determination theory 3 Theory of perceived risk 1 social 36 technology adoption/acceptance 0 HMSAM the hedonic-motivation system acceptance model (HMSAM) (Lowry et al. 2013), which we extend by (1) adding utilitarian factors and (2) inconvenience factors pertinent to VR systems 3 Multi-user acceptance model 3 TAM Technology Acceptance Model 40 technology anxiety 1 UTUAT2 Unified theory of adoption and use of technology 2 1 embodiment 36 Human Enhancement Human Enhancement is a broad term covering several disciplines in fields such as electrical, mechanical, and genetic engineering[31]. Moore [24] defines it as "any attempt to temporarily or permanently overcome the current limitations of the human body through natural or artificial means. It is the use of technological means to select or alter human characteristics and capacities, whether or not the alteration results in characteristics and capacities that lie beyond the existing human range" (sadeghian2021, P. 2) 2 methodology qualitative, quantitative 62 analytical framework 5 CFIR Consolidated Framework for Implementation Research (CFIR) framework 3 collaborative methodology 3 discursive methodology A discursive approach enables you to explore the construction of meanings in human interaction. 2 discourse 2 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 102 Double stimulus methodologies 2 empirical 5 FADGI a collaborative effort started in 2007 by federal agencies to articulate common sustainable practices and guidelines for digitized and born digital; standardisation methodology 178 grounded-theory 3 group-think approach 3 hybrid SEM-ANN a hybrid analysis of Structural Equation Modeling (SEM) and Artificial Neural Network (ANN), through SmartPLS and SPSS software 2 in-experience assessment 2 iterative methodology 1 Living Lab 5 mixed-methods 7 phenomenological approach 1 photogrammetry methodology & process a cheap, flexible and accurate solution to obtain 3D point clouds and textured models. The main advantage of this photogrammetric methodology is that a cloud of points and a precise mesh object with texture can be obtained at the same time (Besoain et al. - 2021 - Developing a Virtual Museum Experience from the D.pdf, P. 8) 226 post-experience measurement 1 qualitative 14 quantitative 6 quantitative confirmatory approach 1 Research by Design methodology Research by Design (RbD) is a transdisciplinary and methodologically distinct method of knowledge production, which involves “knowing through making”. (...) “research by design” also differs radically from a commercially motivated “design practice” because it not only deals with the product but more importantly with the process of its creation as a part of the investigation, which is transferrable 2 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 103 and creates useful knowledge and insights. (kocaturk2023, P. 4) Rovina Paradigm a Rovina Paradigm to improve: the state of the art in measuring, documentation and classification (and thus indirectly supporting diagnosis activities) through a novel approach to surveying, data management and fruition based on three main components: • DigiRo, an automated robot for collecting data with high-precision sensors, including laser scanners and cameras; • ARIS, the cloud-based Archaeological Information System, to manage, share and elaborate data in the form of photorealistic and metrically precise 3D models of the explored sites; • Web and VR Visualizers, that allow to virtually navigate the 3D models through a very intuitive interface which also allows for an immersive experience 1 statistical 2 triangulations 1 User Centered Design 7 user-centered innovation approach 2 Wittemore & Knaff metodology five steps: step 1 – identify a framework (i.e., a blueprint for the analysis), step 2 –search the data base for sources, step 3 – filter the sources according to the inclusion cri-teria, step 4 –code the relevant sources according to the framework identified in step 1, and step 5 – aggregate the findings in a detailed QoE model 1 methods transparent, qualitative, quantitative, forms of recording 74 "in the wild" the phrase "in-the-wild" describes in situ HCI research approaches that report user experience phenomena in everyday living (Rogers & Marshall, 2017). Previous studies have demonstrated that these materials impact the observer in suppressing explicit expressions of bias in controlled laboratory settings. (Young et al. - 2022 - Exploring virtual reality for quality immersive em.pdf, P. 6) 2 3D graphics interactions 4 A/B testing 2 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 104 adaptive virtual realitybased training “the training in which the problem, the stimulus, or task is varied as a function of how well the trainee performs.” (Kelley 1969). In order for a training system to be adaptive, it should have three fundamental components: (1) trainee’s performance measurement; (2) adaptive variable; and (3) adaptive logic. Performance measures can be collected prior to training (e.g., user’s pro-fle information, learning style) or using formative (e.g., monitoring trainee’s movement during training and in real time) or summative (e.g., accuracy after each training session) evaluation methods. (Zahabi - 2020, P. 726) 6 archival research 1 behavioral methods 2 bibliometric analysis 8 body mapping post-experience evaluation tool 4 card sorting/tree testing 0 case study 96 clickstream/analytics 56 cognitive methods 2 cognitive walkthroughs 14 common method bias variance infation factors (VIF) 4 comparative study 10 comparison 3 constant comparison method consists of: (1) data reduction, (2) data display, (3) data comparison, and (4) conclusion drawing and verification 1 concept maps 7 concept testing 2 content analysis 11 co-word analysis a content analysis technique, identifies the frequency and co-occurrence of keywords in a literary corpus 1 context mapping 2 contextual inquiry Contextual inquiry is a type of ethnographic field study that involves in-depth observation and interviews of a small sample of users to gain a robust understanding of work practices and behaviours. https://www.nngroup.com/articles/contextualinquiry/ 1 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 111 UEQ User Experience Questionnaire (UEQ) 64 UTAUT Unified Theory of Acceptance and Use of Technology Questionnaire 2 VR UX Virtual Reality User Experience questionnaire = 8 scales: presence, engagement, immersion, flow, emotion, experience consequences, judgment, technology adoption 3 recommendations Recommended methods, mentioned in the publication or mentioned as necessary for application; mentioned in the methodology proposal, but not applied 22 RITE Rapid Iterative Testing and Evaluation - RITE is a method that involves updating the prototype after a usability issue surfaces by participants — rather than waiting until the study is complete. https://medium.com/research-methods2021/rapid-iterative-testing-and-evaluationrite-d6fecdd9e509 1 satisfaction test 1 scenarios 4 ScientoPy scientometric tool 2 sensor-based methods 1 outdoor&indoor 1 sentiment analysis 17 Serial Reaction Time (SRT) task Serial reaction time (SRT) is a commonly used parameter in the measurement of unconscious learning processes. [1] This parameter is operationalised through a SRT task, in which participants are asked to repeatedly respond to a fixed set of stimuli in which each cue signals that a particular response (i.e., button press) needs to be made. https://en.wikipedia.org/wiki/Serial_reaction_ti me 1 simulation 2 snowball method 3 software development 1 statistical analysis 47 hypothesis testing 0 ANOVA Analysis of variance 1 Kruskal-Wallis tests 1 non-parametric tests 1 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 112 Wilcoxon signed rank test 1 pwrSEM is a Shiny web app developed by Y. Andre Wang for power analysis for parameter estimation in structural equation modelling. 1 model testing 0 Bartlett’s test 1 Cronbach alpha 2 MAP Minimum Average Partial 1 power analysis the calculation used to estimate the smallest sample size needed for an experiment, given a required significance level, statistical power, and effect size 1 Principal Component Analysis 2 storyboards 2 storytelling 10 digital storytelling 6 systematic review 11 PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses 4 talk-aloud protocol a related to think-aloud protocol but slightly different data-gathering method. Talk-aloud protocl involves participants only describing their actions but not other thoughts. This method is thought to be more objective in that participants merely report how they go about completing a task rather than interpreting or justifying their actions 13 target location moving the study to the target location (i.e., con-ducting a field study) such as a school, or a living space, researchers can avoid the negative impact of artificial laboratory setting on the external validity of the study, therefore achieving a higher level of experimental realism 3 textCNN convolutional neural networks for sentence classification (textCNN) 4 thematic analysis 20 coding 11 deductive and inductive 1 think-aloud-protocol 17 co-discovery 0 retrospective 0 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 113 simultaneous (concurrent) 0 usability benchmarking 0 user study 6 remote moderating testing 0 unmoderated testing 0 usability testing 10 user testing 19 user/customer feedback 2 visual analysis 3 Wizard of Oz Method the Wizard of Oz method is a moderated research method in which a user interacts with an interface that appears to be autonomous but is (fully or partially) controlled by a human. https://www.nngroup.com/articles/wizard-ofoz/ 5 workload analysis 4 SWAT the Subjective Workload Assessment Technique (SWAT) is a subjective rating technique using three dimensions (time load, mental effort load, and psychological stress load) and three levels (low, medium, and high) for each dimension to assess workload 1 workshop "a workshop is characterized by a group of individuals who have come together with more or less a common goal i.e. to acquire new knowledge, problem solve, brainstorm or innovate. Depending on the size of the group, a workshop is facilitated by a single or multiple facilitators. A workshop removes participants for their everyday contexts and places them in a context that provokes reflection and innovation" https://think.design/userdesign-research/workshop/ 12 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 114 ZMET Zaltman metaphor elicitation technique (ZMET) is a market research tool. ZMET is a technique that elicits both conscious and especially unconscious thoughts by exploring people's non-literal or metaphoric expressions. It was developed by Gerald Zaltman at the Harvard Business School in the early 1990s. As Zaltman described it, "A lot goes on in our minds that we're not aware of. Most of what influences what we say and do occurs below the level of awareness. That's why we need new techniques: to get at hidden knowledge-to get at what people don't know they know." The technique has been used by academic researchers and for marketing purposes to study a variety of topics related to both marketing and the social sciences. Zaltman metaphor elicitation technique - Wikipedia 48 needs 63 organisation of research 91 data collection 8 SMA Social media analytics 1 experience phase 3 limitations 164 online meeting 1 post-experience phase 0 preparation phase 9 digital objects use 3 procedure 61 spiral model introduced by Boehm [1986]. The spiral model is based on cycles where a prototype is constantly improved and redesigned based on the insights provided by reviews and testing. The roots of such iterative—these days often called “agile”—development go back as far as the 1950s [Larman and Basili 2003]. (Reunanen 2015, P. 5) 1 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 115 waterfall model defined by Winston W. Royce [1970] in his article “Managing the Development of Large Software Systems.” The stages originally defined by Royce are system requirements, software requirements, analysis, program design, coding, testing, and deployment. Later on, modified versions have been introduced, with a smaller number of stages or different labelling, but the main concept of sequentially proceeding development has remained the same. As noted already by Royce, the waterfall model is rigid by nature and mistakes made at the beginning will propagate to the later stages, possibly requiring costly and timeconsuming backtracking. (Reunanen 2015, P. 4) 1 recruitment recruitment of users for the study 14 searching database searching, screening, theoretical background preparation 3 keywords keywords used in the procedure of research, scanning databases, literature & bibliometric analysis 5 task description 8 taxonomy design 2 technology VR, XR, MUVE etc. 48 3D models 6 3D scans 3 algorithms 5 eXtreme gradient boosting (XGBoost) XGBoost, which stands for Extreme Gradient Boosting, is a scalable, distributed gradientboosted decision tree (GBDT) machine learning library. 2 Procedural Generation Algorithm Selection 1 stereo parallax estimation algorithm using deep learning to construct a mechanism to simulate human eye vision in a virtual reality (VR) environment 4 AR augmented reality 44 AR applications 31 AR interactions 6 AR systems 1 BIM building information modelling 4 blockchain 27 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 116 chatbots 2 CVEs Collaborative Virtual Environments 3 drones 0 games 0 computer games 44 exergames 4 game design 4 game engines 2 games engines 1 persuasive games a pervasive game is one where the gaming experience is extended out into the real world,[1] or where the fictional world in which the game takes place blends with the physical world. (...) Pervasive games have been associated with ubiquitous games, augmented and mixed reality games, mobile games, alternate reality games, (enhanced) live action role playing, affective gaming, virtual reality games, smart toys, locationbased or location-aware games, cross-media games and augmented tabletop games. https://en.wikipedia.org/wiki/Pervasive_game 5 serious games a serious game or applied game is a game designed for a primary purpose other than pure entertainment.[1] The "serious" adjective is generally prepended to refer to video games used by industries like defence, education, scientific exploration, health care, emergency management, city planning, engineering, politics and art.[2] Serious games are a subgenre of serious storytelling, where storytelling is applied "outside the context of entertainment, where the narration progresses as a sequence of patterns impressive in quality ... and is part of a thoughtful progress".[3] The idea shares aspects with simulation generally, including flight simulation and medical simulation, but explicitly emphasizes the added pedagogical value of fun and competition. https://en.wikipedia.org/wiki/Serious_game 15 location-based AR/MR games 1 VR games 19 haptic technology 5 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 117 heterotopias are unique, multilayered epistemic contexts that connect other systems through the exchange of information; Heterotopias is both a digital zine and website, hosting studies and visual essays that dissect spaces of play, exploration, violence and ideology 7 Immersive technology technologies that simulate visual, auditory, haptic and motion realities along a RealityVirtuality continuum 38 Metaverse “a massively scaled and interoperable network of realtime rendered 3D virtual worlds that can be experienced synchronously and persistently by an effectively unlimited number of users with an individual sense of presence, and with continuity of data, such as identity, history, entitlements, objects, communications, and payments” (Ball, 2020) - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 364 Decentraland 2 Meta Horizon 0 Mozilla Hubs 0 Open Simulator 0 Roblox 4 Sandbox 0 Spatial 0 MMORPGs Massively multi-player online role-playing games 0 MR mixed reality 28 MUVE 1 Active Worlds example from the first generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 1 AltSpace VR example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 1 High Fidelity example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 Rec Room example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 118 Sansar example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 Second Life example from the first generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 56 SurrealVR example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 Teen Second Life example from the first generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 The Sims Online example from the first generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 There example from the first generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 0 VRChat example from the second generation of virtual environments for multiple users - IMPULSE Plenary Meeting (1.03.2024), Charitos D. 83 software 8 software development 4 SVEs Social Virtual Environments 0 TDT Taxonomy of Digital Technology 1 tools 12 drones 1 Google glass 9 Gravity Sketch Gravity Sketch is a 3D design and collaboration tool that enables you to create cars, sneakers, furniture, characters and more. If you are a designer or an artist, this is a great tool to easily express your ideas and solve complex design challenges working directly in 3D at any scale. https://store.steampowered.com/app/551370 /Gravity_Sketch/ 2 MasterpieceVR MasterpieceVR is a 3D sculpting and painting tool that is an extension of traditional artistic workflows and opens up new ways for rapid ideation, creation and collaboration in virtual space. Creative professionals and artists can quickly learn to create high-quality 3D content and collaborate with others 3 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 119 from all over the world. https://www.meta.com/plpl/experiences/pcvr/361221470668584/ Quill Quill by Smoothstep ("Quill") is a VR illustration and animation tool empowering creators to tell immersive stories. https://quill.art/ 1 RiftArt a VR tool for supporting the teaching a nd studying of Art History 1 Tilt Brush painting app from Google VR 30 Tvori create story prototypes such as animatics & previs, prototype XR apps, and make complete animated films https://store.steampowered.com/app/517170 /Tvori/ 1 VR headsets 34 Virtual worlds 1 Horizon Worlds a social VR world 1 volumetric video Volumetric video (VV) is a media format representing 3D content captured and re-constructed from the real world by cameras and other sensors similarly commonly used in computer graphics (Smolic et al. 2022). (Young et al. - 2023 - Feel the Music!— Audience Experiences of Audio–Tact.pdf, P. 5); "Volumetric capture or volumetric video is a technique that captures a threedimensional space, such as a location or performance. [1] This type of volumography acquires data that can be viewed on flat screens as well as using 3D displays and VR goggles." https://en.wikipedia.org/wiki/Volumetric_capt ure 6 VR 169 CVEs Collaborative Virtual Environments 7 Social Virtual Environments 4 Virtual Worlds "VR spaces where thousands of people can interact simultaneously within the same three-dimensional synthetic graphical 9 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 120 context" - IMPULSE Plenary Meeting (1.03.2024), Charitos D. Immersive virtual environments (IVEs) 30 virtual heritage one of the computer-based interactive technologies in virtual reality where it creates a visual representation of monuments, artifacts, buildings and culture to deliver openly to global audiences 18 VR applications 39 VR installation 4 VR systems 6 XR 21 Terminology more precise definitions of terms in the field of VR, AR, XR; comparison of terms and attempts to organize them 9 tips for using VR/AR/XR for example, hints on how to apply VR/AR/XR in the creation of museum exhibitions and what to look out for 66 users 48 artists 16 dancers 0 musicians 4 painters 2 sculptors 1 audience for example, the audience at a music concert 9 children 8 clinicians 1 creative industry 4 design professionals 3 films/movie creators 2 game creators 1 performance creators 3 VR or MUVE creators 0 designers 2 engineers 1 enthusiast 2 experts persons with extensive, in-depth knowledge and experience in a specific field or area 25 gamers 11 general users 20 metafluencers current name of influencers in the metaverse 1 non-users 8 Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 127 3.2.6 Are your experiences in the virtual environment consistent with your experiences in the physical/real world? [ ] strongly disagree [ ] rather disagree [ ] don't know [ ] rather agree [ ] strongly agree 3.3 Immersion (Cooperative Social Presence (CCPIG)) Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 3.3.1 I do not feel present in virtual space. - 3.3.2 I am not aware of my physical/real environment. - 3.3.3 In the computer-generated world, I have a sense of "being there." - 3.3.4 Somehow, I feel that the virtual world surrounded me. - 3.3.5 I feel present in virtual space. - 3.3.6 I still pay attention to the physical/real environment. - 3.3.7 The virtual worlds seem more realistic than the real world. - 3.3.8 I feel like I was just perceiving pictures. - 3.3.9 I am completely captivated by the virtual world. - 3.3.10 I can freely express my emotions via my avatar. - 3.3.11 In VR, I can recognize the emotions and intentions of other users at the same level as in the physical world. - 3.4 Hedonic Quality - Identification (HQ-I): AttrakDiff-2 For each statement, please choose how you usually/often feel about the virtual environment on a scale of 1 to 7 3.4.1 Isolating - Connective The VR feels isolating (1) / connective (7) 3.4.2 Unprofessional - Professional The VR surroundings feel unprofessional (1) / professional (7). 3.4.3 Inventive - Conventional The VR graphics look inventive (1) / conventional (7). 3.4.4 Tacky - Stylish The avatars appear usually tacky (1) / stylish (7). 3.4.5 Dull - Captivating The virtual space seems dull (1) / captivating (7). 3.4.6 Alienating - Integrating The virtual environment feels alienating (1) / integrating (7). 3.4.7 Unimpressive - Impressive The virtual environment is unimpressive (1) / impressive (7). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 128 3.4.8 Conventional - Innovative The virtual environment is conventional (1) / innovative (7). 4. Actors (CCPIG) 4.1 Awareness (CCPIG) The Prism of Awareness represents a relevant and reliable test of the coherence of a measurement scale. 4.1.1 I act in VR with others in mind [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 4.1.2 I react to other users’ actions [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 4.1.3 I know what other users are trying to achieve [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 4.1.4 I am aware that other users might work out my goals [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 4.1.5 The actions of other users affect the way I play [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 4.1.6 I feel I am affecting other users’ actions/ I feel I have an influence on the actions of other users [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5. Social Action (CCPIG) in VR environments 5.1 I feel other VR users are looking out for me [ ] never [ ] rarely [ ] don’t know Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 129 [ ] often [ ] always 5.2 I feel I am contributing to others [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.3 I feel others are helping me [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.4 I feel my actions make a difference to others [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.5 The actions of others affect my thoughts and actions [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.6 Other VR users play a significant role in my experience of the game [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.7 Other VR users communicate well [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 5.8 Other VR users have a mutual understanding [ ] never [ ] rarely [ ] don’t know [ ] often [ ] always 6.Objects (meCUE + AttrakDiff-2) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 130 6.1 User emotions (meCUE) Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 6.1.1 Interacting in VR with objects exhilarates me. - 6.1.2 Interacting in VR with objects makes me tired. - 6.1.3 Interacting in VR with objects annoys me. - 6.1.4 Interacting in VR with objects relaxes me. - 6.1.5 When interacting in VR with objects I feel exhausted. - 6.1.6 The objects in VR make me feel happy. - 6.1.7 The objects in VR frustrate me. - 6.1.8 The objects in VR make me feel euphoric. - 6.1.9 The objects in VR make me feel passive. - 6.1.10 The objects in VR calm me. - 6.1.11 When interacting in VR with objects I feel cheerful. - 6.1.12 VR objects anger me. - 6.1.13 What features are most important to you in a creative VR space? (e.g., ease of use, collaboration tools, realistic rendering) Open 6.1.14 Have you interacted with digitalized cultural heritage objects before? If yes, please describe your experience. Open 7. Demography/Metrics Questionnaire 7.1 Age: [ ] 18-24 [ ] 25-34 [ ] 35-44 [ ] 45-54 [ ] 55-64 [ ] 65 and over [ ] Prefer not to say 7.2 Gender: [ ] Male [ ] Female [ ] Non-binary/Third gender [ ] Prefer not to say Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 131 7.3 Education Level: [ ] High School [ ] Some College [ ] Associate Degree [ ] Bachelor's Degree [ ] Master's Degree [ ] Doctorate [ ] Prefer not to say 7.4 Field of studies/discipline Please indicate:_____________________ 7.5 additional, allows the selection of participants for the research stage with a prototype Enter your email address if you agree to be invited to the rest of the study and to participate in the experiment with the VR environment prototype and data processing. Please indicate: ______________________ Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 132 5.2.2 Questionnaire for non-users NON-USERS' Questionnaire for group 1 - students and academics as well as group 2 – artists, artistic schools' students. Anonymous, without prototype, possibly leaving an e-mail address there is the possibility of an invitation to a research experiment (with prototype). Questionnaire to be filled in if a negative answer is given to the question on the use of VR. It is possible to ask questions without a specific task & scenario. Disseminated in universities - includes group 1 and art schools/art environment for group 2, as an exploratory survey, in addition to the questionnaire used in the target groups (convenient sample = non-random sampling method). Introduction to project aims/invitation ........................................................................[space to enter text] Possible consents - processing of information in the project/ consent to the processing of information/data will be required ................................................................................[space to enter text] Non-Users Questionnaire (General) Number/section Question Cafeteria questions/statements of choice 0.1 Experience with VR 0.1.1 How often do you use VR technology? [ ] Never – if never, questions will be raised like whether they play online games, whether they study together with friends, whether they chat online; what form of online communication do they prefer (apart from social media) 0.1.2 Your role in academic community [ ] Student [ ] Faculty member 0.1.2a (student) Apart from social media, how often do you cooperate with other students through online games? [ ] Never – Non-Virtual Acculturation [ ] Rarely (once a year or less) [ ] Occasionally (a few times a year) [ ] Frequently (a few times a month) [ ] Very frequently (weekly or more) 0.1.2b (scholar) Apart from social media, how often do you teach in online games virtual environment? [ ] Never – Non-Virtual Acculturation [ ] Rarely (once a year or less) [ ] Occasionally (a few times a year) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 133 [ ] Frequently (a few times a month) [ ] Very frequently (weekly or more) 0.1.3a (student) Have you ever attended online classes that were organized in an online game environment? (yes = 1) / (no = -1) 0.1.3b (student) If yes: Please specify: ______________________ 0.1.4 How would you rate your proficiency with online gaming? [ ] Beginner [ ] Intermediate [ ] Advanced [ ] Expert 0.1.5 What online game have you used in collaborative learning/teaching? Open (short) Please specify: ______________________ 0.2 Tools Preferences 0.2.1 What type of communication do you use while collaborating with other people? Item 1 – Individual chat (text) Item 2 – Team chat (text) Item 3 – Voice (individual) Item 4 – Voice (team) Item 5 – external channel (e.g. Discord) [ ] Never [ ] Rarely [ ] Occasionally [ ] Frequently [ ] Very frequently 0.2.2 What are your expectations regarding this game environment for group work? Open 0.2.3 Do you have any concerns about using an online game environment for personal communication? If so, please specify. Open Section 1. Interface (SUS + meCUE) VR environments/MUVEs 1 Task 1. Imagine that you use online game again for learning/taking classes/teaching… Scenario: During online communication, while looking on the screen, having people around you, how can you relate to the quality of such group activity. Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 134 1.2 Consequences of use (meCUE) Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 1.2.1 If I could, I would use the product daily. If I could, I would use this online game environment daily. 1.2.2 I would not swap this product for any other. I would not swap this online game environment for any other. 1.2.3 I can hardly wait to use the product again. I can hardly wait to use this online game environment again. 1.2.4 In comparison to this product, no others come close. In comparison to my favorite online game environment, no others come close. 1.2.5 When using the product, I lose track of time. When using this online game environment, I lose track of time. Section 2. Space and immersion (CCPIG + iPQ + AttrakDiff-2) 2 Task 2. Let's say you need to communicate with the group. Scenario: Using your preferable channel of communication, being gathered in one place, making pauses for game action 2.1 Team Identification (CCPIG [Cooperative Social Presence]) Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements. 2.1.1 I was aware of my team I was aware of my group members near me in the virtual environment. 2.1.2 I acted with my teammates in mind I acted with my group members in mind while being separated around the map. 2.1.3 I considered my teammates’ possible plans/thoughts I considered my group members' possible ideas and thoughts during our interactions. 2.1.4 I felt like I was part of a team I felt like I was part of a group within the virtual environment. 2.1.5 I felt a social connection to my team-mates (camaraderie) I felt a social connection to my group members (camaraderie) in the virtual space. 2.4 Hedonic Quality - Identification (HQ-I): AttrakDiff-2 For each statement, please choose how you feel about the online game environment on a scale of - 1 to 1, where: 1 = Good property/ 0 = Don't know / - 1 = Bad property 2.4.1 Isolating - Connective The virtual room feels isolating (-1) / connective (1). 2.4.2 Unprofessional - Professional The surroundings feel unprofessional (-1) / professional (1). 2.4.3 Cheap - Premium The graphics look cheap (-1) / premium (1). 2.4.4 Tacky - Stylish The avatars appear tacky (-1) / stylish (1). 2.4.5 Dull - Captivating The virtual space seems dull (-1) / captivating (1). 2.4.6 Alienating - Integrating The virtual environment feels alienating (-1) / integrating (1). Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 135 2.4.7 UnimpressiveImpressive The virtual environment is unimpressive (-1) / impressive (1). 2.4.8 Conventional - Innovative The virtual environment is conventional (-1) / innovative (1). Section 3. Actors (CCPIG) 3 Task 3. Considering the online game environment as virtual learning/teaching space Scenario: When seeking help or asking questions in group… 3.1 Social Action (CCPIG) Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 3.1.1 I felt my team-mates were looking out for me I felt my group members were looking out for me. 3.1.2 I felt I contributed to the team I felt I contributed to the group. 3.2.3 I felt the team helped me I felt the group helped me. 3.1.4 I felt my actions made a difference to my teammates I felt my actions made a difference to my group members. 3.1.5 The actions of my teammates affected my thoughts and actions The actions of my group members affected my thoughts and actions. 3.1.6 My team-mates played a significant role in my experience of the game My group members played a significant role in my experience of the virtual environment. 3.1.7 My team communicated well My group communicated well. 3.1.8 The team had a mutual understanding The group had a mutual understanding. Section 4. Avatar embodiment (optional) 4 Task 4. Try to remember your avatar from games or groups meetings Scenario: Steering avatar in group meeting on screen gives you any feeling about connection or sense of presence in a group? 4.1 External appearance and response to external stimuli 4.1.1 4.1.1 - If the virtual body is not collocated with the participants' body: Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 4.1.2 I felt as if my (real) mind were drifting toward the virtual avatar or as if the virtual avatar were drifting toward my (real) mind. - 5. What features are most important to you Open Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 136 in a virtual online game space? (e.g., ease of use, communication tools, realistic rendering) 6. Have you enjoyed taking a break and playing with teammates? If yes, please describe your experience. Open Metrics questions 7 Section 7. Demographic Information 7.1 Age: [ ] 18-24 [ ] 25-34 [ ] 35-44 [ ] 45-54 [ ] 55-64 [ ] 65 and over [ ] Prefer not to say 7.2 Gender: [ ] Male [ ] Female [ ] Non-binary/Third gender [ ] Prefer not to say 7.3 Education Level: [ ] High School [ ] Some College [ ] Associate Degree [ ] Bachelor's Degree [ ] Master's Degree [ ] Doctorate [ ] Prefer not to say 7.4 field of study/discipline Open Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 143 Section 6. Interface (SUS + meCUE) 6.1 Consequences of use (meCUE) Do you agree (yes = 1) / disagree (no = - 1) / don't know (0) with the following statements: 6.1.1 If I could, I would use the product daily. If I could, I would use this VR environment daily. 6.1.2 I would not swap this product for any other. I would not swap this VR environment for any other. 6.1.3 I can hardly wait to use the product again. I can hardly wait to use this VR environment again. 6.1.4 In comparison to this product, no others come close. In comparison to this VR environment, no others come close. 6.1.5 I would get exactly this product for myself (again) at any time. I would get exactly this VR environment for myself again at any time. 6.1.6 When using the product, I lose track of time. When using this VR environment, I lose track of time. Section 7. Space and immersion (CCPIG + iPQ + AttrakDiff-2) 7.2 Presence in VR space (IPQ) Do you agree (yes = 1) / disagree (no = - 1) / don't know (0) with the following statements: 7.2.1 Were there other real people within the virtual environment besides yourself? Were there other artists within the virtual environment besides yourself? 7.2.2 Were there artificial characters (e.g. computer game opponents) within the virtual environment? Were there artificial characters (e.g., extras, virtual assistants or NPCs) within the virtual environment? 7.2.3 How aware were you of the real world surrounding while navigating in the virtual world? (i.e. sounds, room temperature, other people, etc.) How aware were you of the real-world surroundings while navigating in the virtual world? (e.g., sounds, room temperature, other people) 7.2.4 How real did the virtual world seem to you? How real did the virtual art studio seem to you? 7.2.5 I had a sense of acting in virtual space, rather than operating something from outside. I had a sense of creating art within virtual space, rather than operating tools from the outside. 7.2.6 How much did your experience in the virtual environment seem consistent with your real-world experience? How much did your experience in the virtual art studio seem consistent with your real-world experience in an art studio? 7.2.7 How real did the virtual world seem to you? How real did the virtual environment feel to you? Section 8. Actors (CCPIG) Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 144 8.1 Social Action (CCPIG) Do you agree (yes = 1) / disagree (no = - 1) / don't know (0) with the following statements: 8.1.1 I felt my team-mates were looking out for me I felt my group members were looking out for me. 8.1.2 I felt I contributed to the team I felt I contributed to the group. 8.1.3 I felt the team helped me I felt the group helped me. 8.1.4 I felt my actions made a difference to my team-mates I felt my actions made a difference to my group members. 8.1.5 The actions of my team-mates affected my thoughts and actions The actions of my group members affected my thoughts and actions. 8.1.6 My team-mates played a significant role in my experience of the game My group members played a significant role in my experience of the virtual environment. 8.1.7 My team communicated well My group communicated well. 8.1.8 The team had a mutual understanding The group had a mutual understanding. Section 9. Objects (meCUE + AttrakDiff-2) 9.1 4.2 Interaction with the object AttrakDiff-2 [Hedonic Quality - Stimulation (HQ-S)] For each statement, please choose how you feel about the virtual environment on a scale of 1 to 7, where 1 = left impression = 7 = right impression, e.g. 1 = Conservative 7 = Creative 9.1.1 Conservative - Creative The objects feel conservative (1) / creative (7). 9.1.2 Conventional - Inventive The objects feel conventional (1) / inventive (7). 9.1.3 Unimaginative - Creative The objects feel unimaginative (1) / creative (7). 9.1.4 Cautious - Bold The objects feel cautious (1) / bold (7). 9.1.5 Ordinary - Innovative The objects feel ordinary (1) / innovative (7). Section 10. Actions (NASA-TLX + AttrakDiff-2 + FSS) 10.1 Psychophysical demand x [Pragmatic Quality (PQ] For each statement, please choose how you feel about the virtual environment on a scale of 1 to 7, where: 1 = left impression = 7 = right impression, e.g. 1 = Confusing = 7 = Clearly structured Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 145 10.1.1 How mentally demanding was the task? [Confusing - Clearly structured] The task was confusing (1) / clearly structured (7). 10.1.2 How physically demanding was the task? [Complicated - Simple] The task was complicated (1) / simple (7). 10.1.3 How hurried or rushed was the pace of the task? [Unruly - Manageable] The task was unruly (1) / manageable (7). 10.1.4 How successful were you in accomplishing what you were asked to do? [Cumbersome - Efficient] The task was cumbersome (1) / efficient (7). 10.1.5 How hard did you have to work to accomplish your level of performance? [Confusing - Clearly structured] The task was difficult (1) / easy (7). 10.1.6 How insecure, discouraged, irritated, stressed, and annoyed were you? [Unpredictable - Predictable] The task was stressful (1) / relaxing (7). Section 11. Avatar embodiment (optional) 11.1 Body ownership. Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 11.1.1 I felt as if the virtual body I saw when I looked down was my body - 11.1.2 It felt as if the virtual arm I saw was someone else - 11.1.3 It seemed as if I might have more than one body - 11.1.4 I felt as if the avatar I saw when looking in the mirror was my own body. - 11.1.5 I felt as if the virtual me I saw when looking at myself in the mirror was another person - 11.2 Agency and motor control. Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 11.2.1 It felt like I could control the virtual tool as if it was in my own hand - 11.2.2 The movements of the virtual pointer were caused by my movements - 11.2.3 I felt as if the movements of the virtual legs were influencing my own movements - Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 146 11.2.4 I felt as if the avatar was moving by itself - 11.3 External appearance. Do you agree (yes = 1) / disagree (no = -1) / don't know (0) with the following statements: 11.3.1 It felt as if my (real) head were turning into an ‘avatar’ head - 11.3.2 At some point it felt as if my real body was starting to take on the posture or shape of the virtual body that I saw - 11.3.3 At some point it felt that the virtual face resembled my own (real) face in terms of shape, skin tone or other visual features. - 11.3.4 I felt like I was wearing different clothes from when I came to the laboratory - 11.3.5 6.4 Response to external stimuli. - 11.3.6 I felt that my own hearing could be affected by virtual environment' sounds - 11.3.7 I felt a comforting sensation in my body when I heard [sound] - 11.3.8 When opera/immersion started, I felt the instinct to look for sound source - 11.3.9 I felt as if my ____had ____” - 11.3.10 I had the feeling that I might be harmed by the falling objects surrounding me - 11.4.11 If the virtual body is not collocated with the participants' body: - 11.4.12 I felt as if my (real) ____were drifting toward the virtual ____or as if the virtual ____were drifting toward my (real) ____ - 11.5 What features are most important to you in a creative VR space? (e.g., ease of use, collaboration tools, realistic rendering) Open 11.6 Have you interacted with digitalized cultural heritage objects before? If yes, please describe your experience. Open Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 147 Section 12. Demographic Information 12.1 Age: [ ] 18-24 [ ] 25-34 [ ] 35-44 [ ] 45-54 [ ] 55-64 [ ] 65 and over [ ] Prefer not to say 12.2 Gender: [ ] Male [ ] Female [ ] Non-binary/Third gender [ ] Prefer not to say 12.3 Education Level: [ ] High School [ ] Some College [ ] Associate Degree [ ] Bachelor's Degree [ ] Master's Degree [ ] Doctorate [ ] Prefer not to say 12.4 Artistic specialization Please specify: ______________________ Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 148 5.3 Scheme of Interviews 5.3.1 Scheme of interview for IMPULSE users (experts selected from G1 & G2 as well as G3) 1. What kind of VR platforms (e.g. Spatial) and tools (e.g. VR goggles) are you using in your work? 1.1. What type of VR are you mostly using? [e.g. Immersive with HMD, projection-based, desktop?] 1.2. Which (specific) hardware are you using? e.g. specific HMD, or phone (if accessing VR content through mobile phone)? 1.3. Have you used / are you currently using multi-user games (e.g.. MMORPGs) or multiuser social virtual worlds (e.g. Second Life or similar platforms)? 1.4. What mode of interaction (in VR) do you find most convenient or inconvenient (e.g. handheld controllers, hand tracking and gaze-based systems)? 2. From your perspective: what are the advantages and disadvantages of those VR platforms and tools that you mentioned in terms of your artistic needs? 2.1. In what ways is VR useful in your field? For which types of tasks do you find VR welland/or ill-suited? In what ways is it preferable to work with more conventional interfaces? 2.2. Do you experience adverse effects when using VR? How often? How severe? 2.3. What is the aspect of VR that captivates you the most? 3. Do you use VR mainly to inspect cultural heritage content (or other types of content) or do you also interact with / transform / manipulate the content? If you interact with the content, what type of interaction do you usually engage in? 4. Do you notice any issues related to importing 2D / 3D content onto VR platforms and/or VR development software / tools? 5. What is the average size (area) of the virtual environments you usually enter? (e.g. room-sized, building-sized, more expansive)? *[IF NOT USING VR] Why have you not been interested in using VR up till now? [not applicable if the audience are VR experts] Specific questions based on previous observations (will vary depending on who the user is): 1. <space to fill> 2. <space to fill> 3. <space to fill> Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 149 5.3.2 Scheme of Interviews with academics, artists and non-users Interview Guide Thanks for your participation in this interview. Before I start asking questions, I wanted to inform you that the interview will be recorded and then transcribed. The transcriptions will be sent to you for confirmation. You can remove any information that you do not want to be part of the transcriptions. The transcriptions will be anonymized and will be published online in an open data archive. Do you consent to the start of the recording? (YES/NO) Demographic Data Please provide the following information by marking the appropriate category: - Age: 20-30 [ ] 31-40 [ ] 41-50 [ ] 51-60 [ ] 61 and more [ ] I don’t want to say my age [ ] - Gender: Male [ ] Female [ ] Other [ ] - Education Level: [ ] High School [ ] Some College [ ] Associate Degree [ ] Bachelor's Degree [ ] Master's Degree [ ] Doctorate [ ] Prefer not to say - Field of studies/[artists]Artistic specialisation: ______________________________________ Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 150 (General non-users, G1&G2) Interview Questions Question ID Questions Notes for interviewers Q1: Exposure and Awareness How would you define virtual reality? Can you describe your general familiarity with Virtual Reality (VR)? Have you had any exposure to VR technologies, even if you haven't used them personally? [artists] How do you incorporate VR/XR technologies into your artistic practice? Goal: Gauge basic awareness and indirect exposure to VR technologies. Possible follow-up questions: - What have you heard or seen about VR that influences your perception of it? - Where do you typically encounter information about VR? Notes to interviewer: - What sources of information are mentioned? - Are there any misconceptions or accurate understandings evident? Possible follow-up questions: - How do you integrate XR (VR, AR, or immersive experiences) in presenting your artworks? - How does the virtual environment influence your artistic choices compared to a physical studio? Q2: Perceived Relevance In your view, how could VR be relevant or beneficial to your studies or teaching methods? Goal: Understand perceived potential uses of VR in academic contexts without prior direct experience. Possible follow-up questions: - Can you imagine any scenarios where VR might enhance learning or collaboration? Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 151 - Are there particular subjects or activities you think would benefit from VR? Notes to interviewer: - What are the theoretical benefits they can think of? - Are there any specific academic disciplines they mention? Q3: Barriers to Adoption What are the main reasons you have not tried using VR in any form? Goal: Identify barriers or lack of interest regarding VR use. Possible follow-up questions: - Is it a matter of access, cost, lack of interest, or something else? - Have you encountered any negative reviews or opinions that influenced your stance? Notes for interviewer: - What specific barriers are mentioned most frequently? - Are these barriers logistical, financial, perceptual, or cultural? Q4: Perception of Technology How do you generally perceive new technologies (like VR, AR, AI) in terms of accessibility and usability? How do these gaps affect your work? Goal: Explore attitudes towards adopting new technologies and specific thoughts on VR's user-friendliness. Possible follow-up questions: - Do you feel that new technologies are designed with users like you in mind? - What could make new technologies more appealing or easier for you to try? Notes to interviewer: Deliverable D1.1. Review of UX Methodology and Tool Proposition IMPULSE IMmersive digitisation: uPcycling cULtural heritage towards new reviving StratEgies| 152 [artists] How do VR and XR offer unique opportunities for artistic experimentation that other media do not? - How do they view technological advancements? - Are there any specific features or support they believe would encourage usage? Goal: The aim is to indicate the potential, aptitude, competence and perspective of the application of new VR, XR technologies in art, as well as to highlight the potential for experimentation with and in the immersive environment of artists Possible follow-up questions: - Can you share an example of how using VR or XR has changed the outcome of an art project? - How does the use of XR (VR, AR or immersive experiences) impact on the perception of space and dimension in artworks? - Describe an artistic concept you could realise in VR that would be impossible in traditional or other digital formats. Notes to interviewer: How the artist creates immersive space, what the perception of an immersive environment means to him, how he feels this space, how it influences his work, whether it really supports experimentation and interaction with the audience? Q5: Collaborative Potential Considering your current methods of study or teaching, how do you think Goal: Elicit thoughts on the transformative potential of VR in educational settings.