Backchanneling Emotions in Human-Robot Interaction

Backchanneling Emotions in Human-Robot Interaction Vishruth DIBRIS - RICE Lab Università degli Studi di Genova Genova, Italy [email protected] Antonio Sgorbissa DIBRIS - RICE Lab Università degli Studi di Genova Genova, Italy [email protected] Carmine Tommaso Recchiuto DIBRIS - RICE Lab Università degli Studi di Genova Genova, Italy [email protected] Abstract—Conversational robots often struggle to produce natural backchanneling cues. We propose here a system that enables a robot to provide timely, empathetic feedback. Its integrated processing architecture combines voice activity detection, speech-to-text transcription, large language model (LLM)-based emotion recognition, and context-appropriate responses. The robot responds with synchronized facial expressions, lip movements, and gestures tailored to the user’s emotional state. By leveraging LLMs, the system can infer nuanced emotional cues and produce emotionally congruent backchannels, thus enhancing the naturalness and empathy of HRI. Keywords—Backchanneling, Social robotics, Emotion-aware interaction, Human-Robot Interaction (HRI) I. INTRODUCTION Backchannels are the brief verbal and non-verbal cues that signal a listener's engagement. They are fundamental to natural, empathetic dialogue. These cues are not merely signals of attention; they are critical for maintaining conversational flow and rapport. In HRI, a robot's ability to provide timely and appropriate feedback significantly impacts how users perceive its engagement and empathy [1, 2]. However, the ideal approach is a balance; both excessive and infrequent feedback can be detrimental, and simple nonverbal cues are often more effective than verbose replies [1]. In recent years, data-driven models have been used to generate more human-like backchannels. This work has explored learning from human conversation timing and even adapting for cultural variations [1, 3, 4]. A key insight comes from Shahverdi et al. (2023), who demonstrated that humans adjust their backchanneling style based on emotional context [5]. Their findings underscored a critical gap in HRI: a need for systems that can handle the emotional nuances of backchanneling [5]. Building on this prior research, our study presents an emotion-driven backchanneling system for robots. Instead of relying on traditional classifiers, we leverage a modern large language model (LLM) to directly infer a user’s emotional state from their speech. The subsequent sections of this paper describe the system architecture, its capacity to generate empathetic backchannels, and our plans for future improvements toward multimodal HRI. II. METHODOLOGY The system operates in a closed-loop processing pipeline (Figure 1). The process begins by using a voice activity detector (VAD) [8] to segment audio, which is then transcribed into text by a speech-to-text (STT) service [9]. Fig. 1. Closed-loop architecture of the emotion-aware backchanneling system. Audio input is segmented by a Voice Activity Detector (VAD) and transcribed by a Speech-to-Text (STT) module. The transcript is analyzed by an LLM-based Emotion Backchanneling Engine, which generates and executes the robot’s multimodal response. This produces a stream of transcribed utterances for further analysis. Next, the transcribed text is fed into the emotion recognition module, which is powered by an LLM. All affect detection in the proposed system is performed via this LLM rather than traditional acoustic emotion classifiers. The LLM processes each incoming text segment and infers the user’s emotional state from linguistic cues and context. We prompt the model to return a structured JSON output indicating the predicted emotion category, aligned with the robot’s available expressive states while conceptually grounded in Ekman’s theory of basic emotions [6]. This approach leverages the language understanding capabilities of recent LLMs to interpret nuanced or context-dependent expressions of feeling. Recent studies have demonstrated that LLMs can serve as reliable zero-shot emotion recognizers in dialogue, enabling real-time control of a robot's affective behaviors [1, 2]. The Emotion Backchanneling Engine is at the core of the system, determining when and how the robot should respond. This engine operates on two levels of emotion analysis derived from the LLM: immediate and aggregate emotions. Immediate emotion detection occurs dynamically, based on short silences identified by the VAD. Rather than following a predefined rule-based mapping, the system directly interprets the user’s inferred affective state as predicted by the LLM 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629860 221 Fig. 2. The robot platform, Buddy, demonstrating a happy facial expression as part of the system's backchanneling output. and selects the most appropriate facial expression available to the robot. For example, when the user’s speech conveys frustration or disappointment, the robot displays a sad facial expression, reflecting the conversational tone and signaling that it is attentively following the dialogue. Aggregate emotion detection occurs at the end of a user’s turn or after a longer silence, where the engine evaluates the overall emotional trajectory of the utterance and generates a corresponding facial expression and verbal response. Since the combined processing time of the STT and LLM modules can introduce latency, the system employs short filler utterances immediately after a long silence to maintain conversational continuity and responsiveness while the final output is being processed. The current implementation maps the inferred emotional state directly to one of the robot’s expressive categories; however, future extensions may incorporate affective reasoning frameworks based on Affect Control Theory (e.g., EmoACT) [7] to achieve contextsensitive and affectively consistent interaction behavior. Finally, the selected backchannel response is executed through the robot’s multimodal expression system. In the current implementation, the software has been adapted to the Buddy robot, a humanoid platform with a screen-based face and actuators for gesture control, enabling multimodal expression (see Figure 2). The system synchronizes speech with corresponding facial expressions, ensuring that text-tospeech (TTS) output aligns temporally with labial animations. Additional gestures, such as nodding and blinking, are used to reinforce the communicative intent of the response. The integration of these modalities results in coherent, perceivable, and socially congruent backchanneling behaviors that enhance the fluidity and naturalness of the interaction. III. FUTURE WORK Two primary avenues will guide our future work. First, we plan to conduct controlled user studies to evaluate how emotion-aware backchanneling influences user perception. The experimental protocol will include three interaction conditions: a control condition without backchanneling, a facial expression only condition, and a combined condition that integrates facial expressions with head nods. Each participant will experience all three conditions, and their responses will be evaluated using standardized questionnaires such as the Robotic Social Attributes Scale (ROSAS) [10] and the Perceived Empathic Trust Scale (PETS) [11]. Second, we aim to extend the system toward multimodal perception by integrating camera-based affect recognition. Combining visual cues such as facial expressions and gaze with linguistic input is expected to enable more subtle and contextually adaptive emotional backchanneling. REFERENCES [1] O. Engwall, R. Cumbal, and A. R. Majlesi, “Socio-cultural perception of robot backchannels,” Frontiers in Robotics and AI, vol. 10, p. 990939, 2023. [2] T. Brown, B. Mann, N. Ryder, et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020. [3] M. F. Jung, J. J. Lee, N. DePalma, S. O. Adalgeirsson, P. J. Hinds, and C. Breazeal, “Engaging robots: Easing complex human-robot teamwork using backchanneling,” in Proc. ACM CSCW, pp. 155– 166, 2013. [4] H. Lian, C. Huang, F. Li, et al., “Deep learning-based multimodal emotion recognition: Speech, text, and face—a survey,” Entropy, vol. 25, no. 10, p. 1440, 2023. [5] P. Shahverdi, K. Rousso, J. Klotz, I. Bakhoda, M. Zribi, and W. Y. G. Louie, “Emotionally specific backchanneling in social human-robot interaction and human-human interaction,” in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 4059–4064, 2023. [6] P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992. [7] F. Corrao, A. Nardelli, J. Renoux, and C. T. Recchiuto, “EmoACT: a Framework to Embed Emotions into Artificial Agents Based on Affect Control Theory,” arXiv preprint arXiv:2504.12125, 2025. [8] Silero Team, “Silero VAD: Pre-trained enterprise-grade voice activity detector,” GitHub, 2024. [Online]. Available: https://github.com/snakers4/silero-vad [9] Microsoft, “Speech to Text – real-time speech recognition,” Azure AI Services Documentation, 2023. [Online]. Available: https://learn.microsoft.com/azure/ai-services/speech-service/speechto-text [10] C. M. Carpinella, A. B. Wyman, M. A. Perez, and S. J. Stroessner, “The Robotic Social Attributes Scale (RoSAS): Development and validation,” Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (HRI), Vienna, Austria, 2017, pp. 254–262. [11] M. Niu, M. Lohse, and V. Evers, “The Perceived Empathic Trust Scale (PETS): Measuring user trust and perceived empathy in social robots,” Frontiers in Robotics and AI, vol. 9, 2022, Art. no. 875425. 222