Fostering Engagement through a Latency-Optimized LLM-based Dialogue System for Multimodal ECA Responses - Supplemental Material

Fostering Engagement through a Latency-Optimized LLM-based Dialogue System for Multimodal ECA Responses - Supplemental Material Konstantin W. K¨ uhlem , Jonathan Ehret , Torsten W. Kuhlen , Andrea B¨ onsch Visual Computing Institute, RWTH Aachen University, Aachen, Germany EMail: [konstantin.kuehlem|jonathan.ehret]@rwth-aachen.de and [kuhlen|boensch]@vr.rwth-aachen.de Index Terms—large language model, embodied conversational agents, latency, multi-modal responses, wizard-of-oz, virtual reality We have a supplemental video showing our system in action at: https://youtu.be/cGTPEHuLQ3U. I. IMPLEMENTATION IN UNREAL 5.3 Fig. 1: A high-level class diagram outlining how the main components of the LLM-driven dialogue system were implemented in Unreal Engine 5.3. The components either belong to an ECA actor or the user. On the user actor, the speech input is located. On the agent actor, the input receiver, the decisionmaking, the mind and Output, are located — mimicking the flow of information. This section features a short overview over how the Large Language Model (LLM)-driven dialogue system was implemented in Unreal Engine 5.3. Fig. 1 shows a high-level class diagram of the main components. Only important properties and methods are listed for the main dialogue system. The class diagram does not show any classes that were implemented for controlling the user study. The class diagram also indicates the GameMode class which is used, among other things, for debugging commands which are sent to all Receivers in the scene. The implementation was done mostly in C++ and specific adjustments, for example for different agents, in Blueprints. The classes, except struct and enum types, are so-called ActorComponents within Unreal Engine, meaning they are attached to an owning actor such as the Embodied Conversational Agent (ECA) or the user. Multiple instances of these components might exist in an environment if multiple agents are present. Dashed lines in the class diagram indicate that a component forwards information to another component. For example, in the main pipeline of the dialogue system, the Input component receives a stimulus, either by the user speaking or by an external trigger, then forwards this to the DecisionMaking, which forwards its decision to the MindComponent and the Output. As indicated by the boxes for user and ECA, the abstract class Sender is used by the users SpeechInput and the TTSOutput of the ECA, meaning there is technically no difference in an agent being triggered by the user talking or by another agent talking. Some components feature multiple implementations which are, for example, used for debugging and testing purposes. In general, the system does not need to be powered by an LLM, as the DecisionMaking could also be implemented differently. Concerning the speech streaming feature, this feature cannot be used via the REST Application Programming Interface (API) and no official C/C++ library was available, the Java library was used as part of the backend, which was also used for controlling the study and collecting metrics. As with all communication between this Java backend and the Unreal application, the intermediate audio data was also transferred using REST. II. SERVICE SELECTION This section covers the toughs behind selecting the cloud services we selected for our LLM pipeline. During the implementation, the cloud Speech-to-Text (STT) services by Google Cloud and OpenAI were tested, alongside the local versions of the Whisper1models by OpenAI. The Google Cloud service has support streaming the input audio2 to lower the latency between the user fishing to talk and the text to be ready. Additionally, it has also support for supplying metadata to improve the recognition quality. Because of the additional features, the STT service by Google Cloud was chosen for the implementation. Another advantage of the Google Cloud STT when looking at the pricing is that new customers receive $300 free credits and 60 minutes free transcription per month3. Regarding streaming the input, in the implementation, the audio was captured with a sample rate of 48 kHz and sent to the service in increments of 30000 samples, which is about 0.63 seconds of audio. In the sending process, these samples were down-sampled to 16 kHz for faster processing. During the implementation of this system, the following LLM models were tested: Llama 3.0 (8B), Llama 3.0 (70B) and Llama 3.1 (8B) from the Llama4family by Meta5, GPT3.5 Turbo, GPT-4, GPT-4 Turbo, GPT-4o Mini and GPT-4 from the Generative Pre-Trained Transformer (GPT) family by OpenAI6and Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet and Claude 3.5 Sonnet from the Claude family by Anthropic7. In general, the Llama models had the advantage of being able to run locally and not being tied to a cloud service. However, they were unable to respond in the defined JavaScript Object Notation (JSON) format. The GPT and Claude models both returned good results in quality, giving good responses for the tested situations in the desired JSON format, utilizing all fields and abilities available to them. During testing, slightly higher quality in the responses by the GPT models was observed compared to the Claude models, especially in situations, where information was missing, for example, because the STT service misunderstood a critical word. Within a family, a progression in quality was observed towards the newer models. For the TTS, Amazon Polly8, Google’s Cloud Text-toSpeech service9and the Text-to-Speech (TTS) service by OpenAI10 were tested. The quality between these services is comparable, the services by Amazon and Google offer more voices compared to OpenAI. All services were configured to output as Pulse-Code Modulation (PCM) as this was the fastest option across all services as it has no overhead through file 1https://github.com/openai/whisper 2https://cloud.google.com/STT/docs/transcribe-streaming-audio 3https://cloud.google.com/STT/?hl=en#pricing 4https://www.llama.com 5https://about.meta.com 6https://openai.com/ 7https://www.anthropic.com/ 8https://aws.amazon.com/polly 9https://cloud.google.com/text-to-speech 10https://platform.openai.com/docs/guides/speech-to-text Fig. 2: The three embodiments used in the study. From left to right: A woman, a woman in a driving TV mimicking a video call, and a driving robot. Each of them is wearing a different symbol used to identify the different conditions in the user study. headers, as would be the case with, for example, the WAV or MP3 format. To measure the performance, the services were directly compared in their performance with the sentence ’Hello, welcome to our museum! How can I help you?’, with N= 30 samples, resulting in a time of 0.224 ±0.116 s for Polly, 0.404 ±0.181 s for Google Cloud, and 0.981 ±0.205 s for OpenAI. In the implementation, Amazon Polly was used, as it achieved the fastest generation times during testing and in comparison to related work. Another advantage of using Amazon Polly is that similar to the speech recognition service by Google, the so-called AWS Free Tier grants five million characters for free per month for a total of 12 months. During the development and the user study, a total of about 235000 characters were used over a time of about seven months, which is far away from the limit. The neural engine was used with the Joanna voice, which is a female US-English-speaking voice. III. OTHER EMBODIMENTS This section gives an overview over all embodiments used in our user study, as in the main paper, we mainly report the results for the MetaHuman embodiment. We employed three different embodiment, seen in Fig. 2, as the museums guide. The first one, the female MetaHuman (EMetaHuman), has already been discussed in-depth in the main paper. The second embodiment, ERemote, simulates telepresence by displaying the same MetaHuman during a remote video call on a mobile robot. Finally, ERobot is a mix of the telepresence display and the character Wheatley from Portal 211 and represents a moving robot as an entirely robotic system. All embodiments utilized the US-English voice Joanna from Amazon Polly. While the EM etaHuman utilizes facial expressions, gestures, and attention, ERemote only 11https://store.steampowered.com/app/620/Portal 2 shows facial expressions, and ERobot neither. All embodiments can use the move to field. In the study, six different symbols were used to later identify the conditions in the end questionnaire: Star,Flower,Tower, House,Sun, and Windmill. Three of the symbols and how they were worn by the different embodiments can be seen in Fig. 2. IV. QUESTIONNAIRES In this section the complete questionnaires used after every condition and at the end of the study are displayed. All questionnaires were conducted via SoSciSurvey12. A. Interim Questionnaire This section lists the complete interim questionnaire. 1) StudyID which the participant randomly picked at the start of the study. 2) Social Presence (SP) from the Multimodal Presence Scale for Virtual Reality Environments (MPS) by [1] 5-point Likert Scale (1=completely disagree, 2=disagree, 3=neither disagree nor agree, 4=agree, 5=strongly agree) •SP1 I felt like I was in the presence of a living museum guide in the virtual environment. •SP2 I felt that the museum guide in the virtual environment was aware of my presence. •SP3 The museum guide in the virtual environment appeared to be sentient (conscious and alive) to me. •SP4 During the simulation there were times where the computer interfaces seemed to disappear, and I felt like I was working directly with the museum guide. •SP5 I had a sense that I was interacting with a museum guide in the virtual environment, rather than a computer simulation. 3) Godspeed I: Anthropomorphism (ANT) by [2] 5-point bipolar Likert Scale 1 to 5 •ANT 1 Fake - Natural •ANT 2 Machinelike - Humanlike •ANT 3 Unconscious - Conscious •ANT 4 Artificial - Lifelike •ANT 5 Moving rigidly - Moving elegantly 4) Selection of questions from the Artificial-Social-Agent Questionnaire (ASAQ) by [3] 7-point rating scale [-3, . . . , 3], -3=disagree, 0=neither agree nor disagree, 3=agree •NB2 The museum guide acts naturally. •PF1 The museum guide does its task well. •PF2 The museum guide does not hinder me. •PF3 I am capable of succeeding with the museum guide. •UT1 The museum guide always gives good advice. •UT2 The museum guide acts truthfully. •UT3 I can rely on the museum guide. •AC4 The museum guide appears confused. [Reversed] 12https://www.soscisurvey.de 5) Godspeed IV: Perceived Intelligence (PI) by [2] 5-point bipolar Likert Scale 1 to 5 •PI 1 Incompetent - Competent •PI 2 Ignorant - Knowledgeable •PI 3 Irresponsible - Responsible •PI 4 Unintelligent - Intelligent •PI 5 Foolish - Sensible 6) Custom questions rating the responses by the agent 7-point bipolar Likert Scale 1 to 7 ’The responses by the guide...’ •came too slow - came too fast •were too vague - were too precise •were too short - were too long •were spoken too slow - were spoken too fast 7) Selection of questions from the Chatbot Usability Scale by [4] 5-point Likert Scale (1=completely disagree, 2=disagree, 3=neither disagree nor agree, 4=agree, 5=strongly agree) •8.1.1 I felt that my intentions were understood by the museum guide. •8.1.2 The museum guide was able to guide me to my goal. •8.1.3 I find that the museum guide understands what I want and helps me achieve my goal. •9.1.1 The museum guide gave relevant information during the whole conversation. •9.1.2 The museum guide is good at providing me with a helpful response at any point of the process. •9.1.3 The museum guide provided relevant information as and when I needed it. •12.1.1 I found the museum guide’s responses clear. •12.1.2 The museum guide only states understandable answers. •12.1.3 The museum guide’s responses were easy to understand. 8) Custom questions rating the difficulty for every of the three experienced tasks in this condition. 5-point Likert Scale (1=very easy, 2=easy, 3=normal, 4=hard, 5=very hard) •Task 1 (E.g. How many birds are in the painting ’The Therapist’?) •Task 2 (E.g. Where is the shop located?) •Task 3 (E.g. On which hill does the Mona Lisa stand?) 9) Custom questions rating the perceived helpfulness of the agent for every of the three experienced tasks in this condition. 5-point Likert Scale (1=Not helpful at all, 5=very helpful) •Task 1 (E.g. How many birds are in the painting ’The Therapist’?) •Task 2 (E.g. Where is the shop located?) •Task 3 (E.g. On which hill does the Mona Lisa stand?) 10) For condition 2 (Flower) to 6 (Windmill), custom questions comparing the agent presented to the participant in the last condition to the current condition. 5-point bipolar Likert Scale •I felt a stronger social connection to the last museum guide - I felt a stronger social connection to the current museum guide •I preferred teaming up with the last museum guide for solving my tasks - I preferred teaming up with the current museum guide for solving my tasks •I was able to fulfill the tasks better with the last museum guide - I was able to fulfill the tasks better with the current museum guide •I preferred the conversation dynamics with the last museum guide - I preferred the conversation dynamics with the current museum guide •The conversation with the last guide felt more natural - The conversation with the current guide felt more natural •I felt more trust towards the last museum guide - I felt more trust towards the current museum guide •I think the last museum guide was more trustworthy - I think the current museum guide was more trustworthy •I think the last museum guide was more competent - I think the current museum guide was more competent B. End Questionnaire This section lists all questions from the end questionnaire. All section based questions were single-choice, if not stated otherwise. 1) StudyID which the participant randomly picked at the start of the study. 2) Ranking of the six conditions by dragging cards with the condition name on them together with a textual explanation why they chose the ranking. These rankings were in terms of... •The trust the participant felt towards the agent. •The difficulty of the tasks. •How natural and comfortable the museum guide and the interactions with them felt to the participant. •How competent the museum guide felt to the participant. 3) Single choice of the condition which museum guide the participant would like to see again the most together with a textual explanation. 4) Free text question about what the participant liked about the presented agents. 5) Free text question about what the participant disliked about the presented agents. 6) Free text question about the study and organization. 7) Free text question what participants think how the presented museum guides differed other than in their appearance. 8) It is explained that some of the characters were controlled by an LLM and some by a human (WoZ). The participants were than asked to decide for every condition whether they think they were controlled by an LLM or a human which they could further explain in a free-text question. 1) Gender selectable from Male,Female,Non-binary and Self-described with a free-text field. 2) Age in years. 3) Dominant hand selectable from Right hand,Left hand and Ambidextrous. 4) Vision selectable from the following statements: •I have normal vision. •I have corrected vision (glasses / contact lenses). •Not corrected defective vision: [Free text]. 5) How often the participant used a VR system before, selectable from 0,1,2-10,11-50,50+. 6) Whether they have used ChatGPT by OpenAI or comparable LLMs before, by selecting one of the following statements: •I have never used ChatGPT (or similar AI language models). •I have used ChatGPT (or similar AI language models), but I am not using it on a regular basis. •I am using ChatGPT (or similar AI language models) regularly but not more than a few times per week. •I am using ChatGPT (or similar AI language models) daily. 7) Whether they have experience with ECAs, by selecting one or multiple of the following statements: •I have no experience with ECAs. •I have worked with ECAs in an academic context. •I have designed or programmed ECAs myself. •I have knowledge about LLM-driven ECAs. •I have encountered ECAs in Virtual Reality applications. 8) Whether they have experience with NPCs, by selecting one or multiple of the following statements: •I have no experience with NPCs. •I have encountered NPCs (e.g., in video games). •I have encountered NPCs in Virtual Reality (VR). •I have encountered NPCs in Augmented Reality (AR). •I had conversations with NPCs through multiple choice dialogues. •I had conversations with NPC through natural language (both spoken or written). 9) Whether they know ChatGPT would play a role in the study before attending, by selecting one of the following statements: •I did not know it at all. •I thought so, but did not know it. •I knew it at some time, but was not aware of it in the study. •I knew it before and was aware of it during the study. V. LLM PROMPT EXAMPLE This section holds an example for an LLM prompt, similar to situations in the user study. In this context, the user said ’Hello, can you help me?’ to the ECA (embodied as the MetaHuman). [S] stands for the system role and [U] for the user role. This prompt especially shows, how multimodal controls are embedded into the LLM prompt (see Line 3). The LLM does not answer with empty fields, therefore int this example, there is no move to field. [S]The user is on a scavenger hunt through the museum. You insist on leading the user to paintings they have questions about and do not answer specific questions when you are not at the painting. You only start going to the painting if specifically requested by the user. You and the user are the only persons in the museum. Only the paintings which are available as waypoints do exist,there are also only paintings and no other art pieces in the museum. You can also answer any other questions the user has,if you know the answer. [S]Your name is Vivian. You work in a virtual museum [S]You output a json with the following format,all fields marked with ’?’ are optional:{response: string,face:’neutral’ | ’happiness’ | ’ sadness’ | ’surprise’ | ’fear’ | ’anger’ | ’ disgust’ | ’contempt’ | ’slight_smile’,gesture: ’none’ | ’wave’ | ’talk’,attention:’user’ | ’ user_and_target’,move_to?:string} [S]You use the ’user_and_target’ attention when talking with the user about a waypoint. You can move to the following waypoints using the move_to field. You can move to waypoint if you are already there. This is useful to reposition yourself,if you are blocking the view of the user. [S]Waypoint timeless_hall:Name = Timeless Hall [S]Waypoint modern_nature_gallery:Name = Modern Nature Gallery [S]Waypoint self-portrait_on_the_6 th_wedding_anniversary:Name = Self-Portrait on the 6th Wedding Anniversary,Description = Painted by Paula Modesohn-Becker in 1906. Inside the Chamber of Self-Discovery [S]Waypoint self-portrait_with_model:Name = SelfPortrait with Model,Description = Painted by Ernst Ludwig Kirchner in 1910. Inside the Chamber of Self-Discovery [S]Waypoint self-portrait_in_black:Name = SelfPortrait in Black,Description = Painted by Max Beckmann in 1944.Inside the Chamber of SelfDiscovery [S]Waypoint coastal_landscape:Name = Coastal Landscape,Description = Painted by Edvard Munch in 1918. Inside the Modern Nature Gallery [S]Waypoint winter_landscape_in_moonlight:Name = Winter Landscape in Moonlight,Description = Painted by Ernst Ludwig Kirchner in 1919. Inside the Modern Nature Gallery [S]Waypoint the_temptation_of_saint_antonius:Name = The Temptation of Saint Antonius,Description = Painted by Max Ernst in 1945. Inside the Vision Wing [S]Waypoint the_oxbow:Name = The Oxbow, Description = Painted by Thomas Coloe in 1836. Inside the Modern Nature Gallery [S]Waypoint the_persistence_of_memory:Name = The Persistence of Memory,Description = Painted by Salvador Dali in 1931.Inside the Vision Wing [S]Waypoint the_therapist:Name = The Therapist, Description = Painted by Rene Megritte in 1937. Inside the Vision Wing [S]Waypoint guide_stand:Name = Guide Stand, Description = Your stand [S]Waypoint vision_wing:Name = Vision Wing, Description = Contains surreal paintings [S]Waypoint mosaic_hall:Name = Mosaic Hall, Description = Currently has no paintings [S]Waypoint chamber_of_self-discovery:Name = Chamber of Self-Discovery,Description = Has the three self-portraits in this museum [S]Waypoint mona_lisa:Name = Mona Lisa, Description = Inside the Timeless Hall [S]Waypoint wanderer_above_the_sea_of_fog:Name = Wanderer above the Sea of Fog,Description = Painted by Caspar David Friedrich in 1818. Inside the Timeless Hall. The oldest painting in the museum. [S]Waypoint the_starry_night:Name = The Starry Night,Description = Painted by Vincent van Gogh in 1889. Inside the Timeless Hall [S]Waypoint water_lilies:Name = Water Lilies, Description = Painted by Claude Monet in 1906. Inside the Timeless Hall [S]Waypoint the_weeping_woman:Name = The Weeping Woman,Description = By Pablo Picasso in 1937. Inside the Timeless Hall [S]You can also go to the current position of the user or follow them by using ’user’ as the move_to field [S]You can use the move_to field,if you are asked by the user to come to them,follow them,or a lead them to a painting or hall. You always use the painting waypoint when asked for it instead of the hall. [S]You can follow the user,even if you are currently in the same room as them. Always use ’ user’ if you are asked by the user to follow or come to them. [S]You are currently at:guide_stand. The user is currently satnding at:timeless_hall. [S]The user just said to you: [U]Hello,can you help me? To this, the LLM (GPT-4o) answered: { response:"Of course! I’m here to assist you. Is there a specific painting you’d like to learn about or a particular gallery you want to visit?", face:"happy", gesture:"wave", attention:"user" } REFERENCES [1] G. Makransky, L. Lilleholt, and A. Aaby, “Development and validation of the multimodal presence scale for virtual reality environments: A confirmatory factor analysis and item response theory approach,” Computers in Human Behavior, vol. 72, 2017. [2] C. Bartneck, D. Kuli´ c, E. Croft, and S. Zoghbi, “Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,” International journal of social robotics, vol. 1, 2009. [3] S. Fitrianie, M. Bruijnes, F. Li, A. Abdulrahman, and W.-P. Brinkman, “The artificial-social-agent questionnaire: establishing the long and short questionnaire versions,” in Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents. Association for Computing Machinery, 2022. [4] S. Borsci, A. Malizia, M. Schmettow, F. van der Velde, G. Tariverdiyeva, D. Balaji, and A. Chamberlain, “The chatbot usability scale: the design and pilot of a usability scale for interaction with ai-based conversational agents,” Personal and Ubiquitous Computing, vol. 26, no. 1, Jul. 2021.