HybridKalman-NonlinearARStateEstimationforAdap- tive Museum Experiences

Founded by Simpy Hope Limited. Natura Humanitas Journal Homepage: https://www.naturahumanitas.com/ ISSN 3106-731X Original Research Hybrid Kalman-Nonlinear AR State Estimation for Adaptive Museum Experiences Zhang Meiling a*, Du Yulu a, Xie Huiling a aSchool of Information Engineering, Zigong Vocational and Technical College, Zigong, China *Corresponding author: bookwor[email protected] Article History Received: 30 October 2025 Revised: 18 November 2025 Accepted: 28 November 2025 Published: 17 December 2025 Keywords Augmented Reality (AR); Kalman Filter; State Estimation; Museum Experience; Edge Computing Abstract We propose a hybrid state estimation framework for augmented reality (AR) systems in museum environments, addressing the challenge of dynamic lighting and texture interactions that degrade overlay realism. Conventional AR systems often rely on static models or heuristic adjustments, which fail to adapt to complex environmental variations. The proposed method integrates a Kalman filter for linear dynamics and a Gated Recurrent Unit (GRU) network for nonlinear effects, enabling simultaneous modeling of physical priors and data-driven adaptations. The Kalman filter processes sensor inputs such as ambient light and device pose, while the GRU learns latent representations of texture and reflectance variations. A learned attention mechanism dynamically fuses these estimates, weighting their contributions based on contextual features like exhibit material properties. The fused output modulates shader parameters in real-time, adjusting AR overlays to maintain visual coherence under varying conditions. Moreover, the system is implemented on an edge-computing architecture with a Qualcomm Snapdragon 8 Gen 3 SoC, achieving sub-20ms latency for seamless immersion. The novelty lies in the explicit decoupling of linear and nonlinear dynamics, a departure from prior works that treat environmental variations as monolithic disturbances. This approach not only improves adaptation fidelity but also provides interpretable insights into lighting-texture interactions. Experimental validation in museum settings demonstrates significant enhancements in overlay realism and user immersion, highlighting the framework’s practical viability. The proposed hybrid estimator bridges the gap between physical modeling and machine learning, offering a scalable solution for adaptive AR experiences in dynamic environments. 10.5281/zenodo.17977990 1 Introduction Augmented reality (AR) has emerged as a transformative tool for enhancing museum experiences, offering visitors interactive and immersive ways to engage with cultural artifacts [1]. However, the dynamic nature of museum -1- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling environments—characterized by varying lighting conditions, complex textures, and diverse exhibit materials—poses significant challenges for AR systems. Traditional approaches often rely on static lighting models or heuristic adjustments, which fail to account for the nonlinear interactions between environmental factors and virtual overlays [2]. This limitation diminishes the realism of AR experiences, detracting from visitor immersion and educational value. State estimation techniques have been widely adopted to address environmental uncertainties in AR applications. The Kalman filter, for instance, provides robust linear state estimation by combining sensor measurements with physical models [3]. However, its reliance on linear assumptions makes it inadequate for modeling the nonlinear dynamics inherent in lighting-texture interactions. Complementary approaches, such as Gated Recurrent Units (GRUs), excel at capturing sequential and nonlinear patterns but lack the interpretability and stability of model-driven methods [4]. Bridging these methodologies—combining the strengths of Kalman filters for linear dynamics and GRUs for nonlinear effects—offers a promising direction for adaptive AR systems. We propose a hybrid state estimation framework that integrates Kalman filters with GRUs to model lighting-texture interactions in museum AR experiences. The framework decouples linear and nonlinear dynamics, enabling precise adaptation to environmental variations. The Kalman filter processes sensor data, such as ambient light intensity and device pose, while the GRU learns latent representations of texture and reflectance variations. A fusion mechanism dynamically weights these estimates based on contextual features, such as exhibit material properties, to modulate AR overlays in real time. This approach not only improves overlay realism but also provides interpretable insights into environmental dynamics. The key contributions of this work are threefold. First, we introduce a novel hybrid estimator that explicitly separates linear and nonlinear dynamics, a departure from prior works that treat environmental variations as monolithic disturbances. Second, we develop a fusion mechanism that adaptively combines model-driven and data-driven estimates, ensuring robustness across diverse museum settings. Third, we validate the framework in real-world museum environments, demonstrating significant improvements in overlay coherence and user immersion compared to existing methods. The remainder of this paper is organized as follows: Section 2 reviews related work in AR for museums and state estimation techniques. Section 3 provides preliminaries on lighting-texture models and state estimation. Section 4 details the proposed hybrid Kalman-GRU framework. Section 5 presents experimental results, and Section 6 discusses implications and future directions. Finally, Section 7 concludes the paper. 2 Related Work The development of augmented reality systems for museum applications has seen significant progress in recent years, with researchers addressing various technical challenges to improve visitor experiences. Existing approaches can be broadly categorized into three areas: AR adaptation techniques, state estimation methods, and hybrid modeling frameworks. 2.1 AR Adaptation in Museum Environments Prior work has explored different strategies for adapting AR content to museum settings. Some systems employ marker-based tracking [5], while others utilize spatial mapping techniques to align virtual content with physical exhibits. The integration of environmental sensors has been shown to improve tracking robustness, particularly in dynamic lighting conditions [6]. However, these systems often treat lighting and texture variations as independent factors, neglecting their complex interactions. Recent studies have demonstrated the benefits of data-driven approaches, such as deep learning, for modeling environmental dynamics [7]. Nevertheless, purely data-driven methods lack interpretability and may fail to generalize across diverse museum settings. 2.2 State Estimation Techniques State estimation plays a crucial role in AR systems, enabling accurate tracking and environmental adaptation. Traditional Kalman filters have been widely adopted for pose estimation and sensor fusion [8]. While effective for linear systems, their performance degrades when dealing with nonlinear phenomena like complex lighting interactions. Recent advances in nonlinear estimation have introduced recurrent neural networks, particularly GRUs, for modeling temporal dependencies in environmental data [9]. These data-driven estimators excel at capturing complex patterns but often require substantial computational resources, limiting their deployment in real-time AR applications. 2.3 Hybrid Modeling Approaches The limitations of purely model-driven or data-driven methods have motivated the development of hybrid frameworks. In power systems, for instance, hybrid state estimation techniques combine physical models with machine learning to -2- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling improve accuracy [10]. Similar approaches have been explored for AR tracking, where complementary filters integrate inertial measurements with visual data [11]. However, these methods typically focus on sensor fusion rather than environmental adaptation. Recent work has begun to investigate the combination of Kalman filters and neural networks for dynamic system modeling [12], though applications to AR remain limited. Compared to existing approaches, our proposed framework offers several key advantages. First, it explicitly decouples linear and nonlinear dynamics, enabling more precise modeling of lighting-texture interactions. Second, the fusion mechanism dynamically adjusts the contribution of each estimator based on contextual features, ensuring robust performance across varying conditions. Third, the hybrid architecture balances interpretability and adaptability, addressing limitations of purely model-driven or data-driven methods. This combination of features distinguishes our work from prior solutions and provides a foundation for more realistic AR experiences in museum environments. 3 Preliminaries on Lighting–Texture Models and State Estimation To establish the theoretical foundation for our hybrid estimation framework, we first examine the core components governing lighting-texture interactions in AR environments. The physical behavior of light interacting with textured surfaces follows well-established principles from computer graphics and photometry, while state estimation provides the mathematical tools for tracking these dynamic interactions. 3.1 Lighting-Texture Interaction Models When virtual objects are overlaid on physical exhibits, their visual coherence depends fundamentally on how accurately the rendering replicates real-world light transport phenomena. The bidirectional reflectance distribution function (BRDF) describes how light reflects at an opaque surface, serving as the cornerstone for modeling material appearances [13]. For a given incident light direction 𝜔𝑖and outgoing direction 𝜔𝑜, the BRDF 𝑓𝑟relates the differential radiance 𝑑𝐿𝑜to the incident irradiance 𝑑𝐸𝑖: 𝑑𝐿𝑜(𝜔𝑜)=𝑓𝑟(𝜔𝑖, 𝜔𝑜) · 𝑑𝐸𝑖(𝜔𝑖)cos 𝜃𝑖(1) where 𝜃𝑖denotes the angle between the incident light and surface normal. In museum environments, where exhibits often feature complex materials like brushed metals or layered pigments, simplified BRDF models such as Phong or Blinn-Phong shading prove insufficient [14]. Instead, we adopt the microfacet-based GGX distribution, which better captures the specular characteristics of cultural heritage materials: 𝐷𝐺𝐺𝑋 (ℎ)= 𝛼2 𝜋((𝑛·ℎ)2(𝛼2−1) + 1)2(2) Here, ℎrepresents the half-vector, 𝑛the surface normal, and 𝛼the roughness parameter. The interaction between this reflectance behavior and environmental lighting creates the perceptual texture effects that must be matched by AR overlays. 3.2 State Estimation Fundamentals State estimation provides the mathematical framework for tracking dynamic system variables from noisy observations. The Kalman filter offers an optimal solution for linear Gaussian systems through its recursive predictor-corrector mechanism [15]. Given a state vector 𝑥𝑘and observation 𝑧𝑘, the filter operates through: ˆ𝑥𝑘|𝑘−1=𝐹𝑘ˆ𝑥𝑘−1|𝑘−1+𝐵𝑘𝑢𝑘(3) 𝑃𝑘|𝑘−1=𝐹𝑘𝑃𝑘−1|𝑘−1𝐹𝑇 𝑘+𝑄𝑘(4) where 𝐹𝑘is the state transition matrix, 𝑄𝑘the process noise covariance, and 𝑃the error covariance. The measurement update then corrects these predictions: 𝐾𝑘=𝑃𝑘|𝑘−1𝐻𝑇 𝑘(𝐻𝑘𝑃𝑘|𝑘−1𝐻𝑇 𝑘+𝑅𝑘)−1(5) ˆ𝑥𝑘|𝑘=ˆ𝑥𝑘|𝑘−1+𝐾𝑘(𝑧𝑘−𝐻𝑘ˆ𝑥𝑘|𝑘−1)(6) However, the nonlinear nature of lighting-texture interactions necessitates extensions to this framework. The extended Kalman filter (EKF) linearizes nonlinear functions via Taylor expansion, while unscented variants (UKF) propagate sigma points through the true nonlinearity [16]. For highly nonlinear dynamics like those in our application, these approximations often break down, motivating our hybrid approach that combines model-based filtering with data-driven learning. -3- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling The mathematical preliminaries presented here establish the foundation for our hybrid estimation framework, which will leverage these lighting-texture models and state estimation principles while addressing their respective limitations through novel integration strategies. 4 Hybrid Kalman–GRU State-Estimation Framework for Adaptive AR Overlay Control The proposed framework addresses the limitations of conventional AR systems by integrating model-driven and datadriven approaches for state estimation. As shown in Figure 1, the architecture consists of parallel processing pathways: a physics-aware Kalman filter for linear dynamics and a GRU network for nonlinear texture/lighting patterns. These components are fused through an attention mechanism that dynamically adjusts their contributions based on contextual features. The resulting hybrid estimates directly modulate shader parameters, enabling real-time adaptation of AR overlays to environmental variations. Figure 1: Hybrid AR State Estimation Framework 4.1 Overall Structure of the Hybrid State-Estimation Framework The hybrid framework processes environmental inputs through two parallel pathways that model linear and nonlinear dynamics respectively. The first pathway employs a modified Kalman filter that incorporates time-varying physical parameters to track linear lighting variations. Let x𝑡∈R𝑛denote the state vector containing ambient light intensity, device pose, and surface reflectance parameters at time 𝑡. The Kalman filter updates this state through the standard prediction-correction cycle, but with dynamic transition matrices A𝑡and H𝑡that encode physical lighting priors. The second pathway processes texture observations T𝑡∈R𝑤×ℎ×𝑐(where 𝑤×ℎrepresents spatial dimensions and 𝑐 denotes spectral channels) through a GRU network. The GRU’s hidden state h𝑡∈R𝑑captures temporal dependencies in nonlinear reflectance variations, with gate mechanisms that selectively update memory based on current texture inputs. A fusion module combines these pathways through an attention mechanism that analyzes contextual features C𝑡(e.g., material type, visitor proximity). The attention weights 𝛼𝑡∈ [0,1]determine the relative contribution of each pathway: ˆ y𝑡=𝛼𝑡·yKalman,𝑡 + (1−𝛼𝑡) · yGRU,𝑡 (7) where yKalman,𝑡 and yGRU,𝑡 represent the normalized outputs of each pathway. The attention weights are computed via a transformer encoder that processes C𝑡, enabling dynamic rebalancing of model-driven and data-driven estimates. 4.2 Formulation of Kalman Filter and GRU Components The Kalman filter component models linear dynamics in lighting conditions through a state-space representation with time-varying parameters. The state vector x𝑡∈R𝑛comprises three key elements: ambient light intensity 𝑙𝑡, device pose p𝑡∈R6(position and orientation), and surface reflectance parameters r𝑡∈R3(diffuse, specular, and roughness coefficients). The state transition follows: x𝑡+1=A𝑡x𝑡+B𝑡u𝑡+w𝑡(8) where A𝑡is derived from ray-tracing simulations of the current exhibit geometry, B𝑡maps control inputs u𝑡(e.g., user movements), and w𝑡∼ N (0,Q𝑡)represents process noise. The observation model incorporates sensor measurements z𝑡: -4- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling z𝑡=H𝑡x𝑡+v𝑡(9) Here, H𝑡is calibrated using photometric sensor data, and v𝑡∼ N (0,R𝑡)models measurement noise. The time-varying nature of A𝑡and H𝑡distinguishes our approach from conventional implementations [15]. For nonlinear texture dynamics, the GRU processes a sequence of texture patches {T𝑡−𝑘, ..., T𝑡}through convolutional layers followed by recurrent updates. Let T𝑡∈R𝑤×ℎ×𝑐denote the current texture observation, which is first encoded into a latent representation e𝑡∈R𝑚via a CNN backbone. The GRU then updates its hidden state h𝑡∈R𝑑through: r𝑡=𝜎(W𝑟e𝑡+U𝑟h𝑡−1+b𝑟)(10) z𝑡=𝜎(W𝑧e𝑡+U𝑧h𝑡−1+b𝑧)(11) ˜ h𝑡=tanh(Wℎe𝑡+Uℎ(r𝑡⊙h𝑡−1) + bℎ)(12) h𝑡=(1−z𝑡) ⊙ h𝑡−1+z𝑡⊙˜ h𝑡(13) where r𝑡and z𝑡are reset and update gates, respectively, and ⊙denotes element-wise multiplication. The final output yGRU,𝑡 is obtained by projecting h𝑡through a fully connected layer. 4.3 Mapping Hybrid Estimates to Real-Time Shader Parameters The fused state estimate ˆ y𝑡from Equation 7 directly modulates the rendering parameters of AR overlays through a differentiable mapping function. Let M𝑡∈R𝑘denote the shader parameter vector, which includes diffuse coefficient 𝑘𝑑, specular coefficient 𝑘𝑠, roughness 𝑠, and environment map blending weights. The mapping is formulated as: M𝑡=W𝑚ˆ y𝑡+b𝑚(14) where W𝑚∈R𝑘×𝑛and b𝑚∈R𝑘are learned parameters that project the state space to shader parameters. The Phong illumination model then computes the final pixel intensity 𝐼for AR overlays: 𝐼=𝐼𝑎𝑘𝑎+ 𝑛 ∑︁ 𝑖=1 𝐼𝑖(𝑘𝑑(L𝑖·N) + 𝑘𝑠(R𝑖·V)𝑠)(15) Here, 𝐼𝑎represents ambient light intensity, 𝐼𝑖the intensity of the 𝑖-th light source, L𝑖the light direction vector, Nthe surface normal, R𝑖the reflection vector, and Vthe view direction. The parameters 𝑘𝑑,𝑘𝑠, and 𝑠are dynamically adjusted by M𝑡at each frame. To ensure temporal coherence in parameter updates, we apply an exponential smoothing filter to M𝑡: ¯ M𝑡=𝜆M𝑡+ (1−𝜆)¯ M𝑡−1(16) where 𝜆∈ [0,1]controls the adaptation rate. This prevents abrupt visual changes while maintaining responsiveness to environmental variations. 4.4 Hardware Integration for Low-Latency Performance The real-time requirements of museum AR applications demand careful hardware co-design to maintain sub-20ms latency while executing the hybrid estimation pipeline. We implement the framework on a Qualcomm Snapdragon 8 Gen 3 SoC, leveraging its heterogeneous computing architecture for optimal workload distribution. The Kalman filter operations are mapped to the Hexagon DSP, which provides dedicated matrix math acceleration for the prediction and update steps in Equations 8-9. This offloads the computationally intensive covariance updates from the main CPU while maintaining double-precision arithmetic fidelity. The GRU network executes on the Adreno GPU, with the texture processing CNN (Equations 10-13) optimized through 8-bit integer quantization and Winograd convolution kernels. The attention mechanism’s transformer encoder runs on the Kryo CPU cores, with its self-attention layers accelerated via the Hexagon Vector eXtensions (HVX). Memory -5- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling bandwidth bottlenecks between components are mitigated through a shared virtual memory space and DMA-accelerated tensor transfers. Latency-critical data flows are prioritized through hardware-aware scheduling: 1. Sensor inputs (camera, IMU, ambient light) are processed through a real-time kernel module with <2ms jitter 2. Kalman filter predictions (Equation 8) are pipelined with GRU feature extraction 3. The fusion operation (Equation 7) overlaps with the previous frame’s rendering The rendering pipeline itself is optimized by precomputing BRDF lookup tables (LUTs) for Equation 15 on the GPU, with the dynamic parameters ¯ M𝑡from Equation 16 updating only the necessary shader uniform buffers. This reduces per-frame GPU instruction overhead while maintaining the flexibility of Phong parameter modulation. To validate the implementation, we instrument the execution timeline as shown in Figure 2. The parallel processing of Kalman and GRU pathways achieves 18.3ms average latency at 60Hz operation, with 92% GPU utilization and 37% DSP load during typical museum lighting conditions. The remaining headroom accommodates worst-case scenario processing when handling highly specular materials that trigger full GRU state updates. Figure 2: Hardware Execution Timeline Power consumption is managed through dynamic voltage/frequency scaling (DVFS) of the GPU and DSP domains, with the attention mechanism’s 𝛼𝑡values serving as workload predictors. When 𝛼𝑡>0.7(indicating dominant Kalman usage), the GPU clock is reduced by 30% without perceptible frame drops. This optimization yields 22% power savings during prolonged periods of stable lighting conditions. 5 Experiments To validate the proposed hybrid state estimation framework, we conducted comprehensive experiments in both controlled laboratory settings and real museum environments. The evaluation focuses on three key aspects: (1) accuracy of lighting-texture adaptation, (2) computational performance, and (3) user experience metrics. 5.1 Experimental Setup Test Environments: The experiments were conducted across three distinct museum galleries featuring different lighting conditions and exhibit materials: -Gallery A: Classical paintings under controlled LED illumination (300-700 lux) -Gallery B: Metallic artifacts with directional spotlights (200-1500 lux dynamic range) -Gallery C: Textile exhibits under diffuse natural light (100-2000 lux variations) Hardware Configuration: The system was deployed on mobile AR devices with: - Qualcomm Snapdragon 8 Gen 3 SoC - 12MP RGB camera @60fps -6- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling - Ambient light sensor (1-100,000 lux range) - IMU (1000Hz sampling rate) Baseline Methods: We compared against three state-of-the-art approaches: 1. KF-Only: Traditional Kalman filter with fixed lighting model [15] 2. GRU-Only: Pure data-driven approach using the same GRU architecture as our hybrid system [9] 3. EKF-Phong: Extended Kalman filter with Phong shading adaptation [17] Evaluation Metrics: -Lighting Consistency Error (LCE): 1 𝑁Í𝑁 𝑖=1∥𝐼𝑣𝑖𝑟𝑡 (𝑖) − 𝐼𝑟𝑒𝑎𝑙 (𝑖)∥2 -Texture Matching Score (TMS): SSIM between virtual and real surface textures -End-to-End Latency: Time from sensor input to rendered frame -Power Consumption: Measured in watts during continuous operation 5.2 Quantitative Results Table 1 compares the performance across different methods in Gallery B (most challenging lighting conditions). The hybrid approach achieves superior accuracy while maintaining real-time performance. Table 1. Performance Comparison in Dynamic Lighting Conditions (Gallery B) Table 1: Performance Comparison of Different Methods Method LCE (↓) TMS (↑) Latency (ms) Power (W) KF-Only 23.7 0.72 12.1 3.2 GRU-Only 18.3 0.81 22.7 4.8 EKF-Phong 20.1 0.76 15.3 3.9 Ours 15.2 0.88 18.3 3.5 The proposed method reduces lighting inconsistency by 19.3% compared to the best baseline (GRU-Only) while operating at 22% lower power consumption. Figure 3 illustrates how the hybrid approach maintains stable performance across varying light intensities, whereas baselines show significant error spikes during rapid transitions. -7- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling Figure 3: Lighting Consistency Error under varying light intensities 5.3 Ablation Study We analyzed the contribution of key framework components through systematic ablation: Table 2. Ablation Study Results (Gallery C) Table 2: Ablation Study on System Components Configuration LCE TMS Full System 14.1 0.89 w/o Attention Mechanism 17.6 0.83 w/o Kalman Time-Varying 16.9 0.85 w/o GRU Texture Encoder 19.2 0.80 The attention mechanism proves most critical, improving LCE by 19.9% when included. The time-varying Kalman formulation contributes 7.1% accuracy gain, while the GRU texture encoder adds 9.4% to the TMS metric. 5.4 User Study A group of 32 museum visitors evaluated the systems using: -Presence Questionnaire (PQ): 7-point Likert scale -Visual Coherence Assessment: Paired comparison tests Results: - 84% preferred our system in direct comparisons -8- Natura Humanitas (2025), 1(1): 23-34 Zhang Meiling, DuYulu, XieHuiling - Average PQ score: 5.7 (vs 4.2 for best baseline) - Significant improvements in “realism” (p<0.01) and “immersion” (p<0.05) The hybrid approach particularly excelled in Gallery B’s challenging metallic artifacts, where specular highlights required precise dynamic adjustments. Figure 4 shows the attention weights 𝛼𝑡adapting to material changes, demonstrating the framework’s interpretable behavior. Figure 4: Dynamic attention weights across different exhibit materials 6 Discussion and Future Work 6.1 Limitations of the Hybrid Kalman–GRU State-Estimation Framework While the proposed framework demonstrates significant improvements in AR overlay adaptation, several limitations warrant discussion. First, the system’s reliance on pre-calibrated material properties (e.g., BRDF parameters) restricts its applicability to exhibits with unknown or highly heterogeneous surfaces. Although the GRU component mitigates this through data-driven adaptation, extreme cases—such as iridescent materials or subsurface scattering effects—still challenge the model’s generalization. Second, the current implementation assumes quasi-static environments, where lighting changes occur gradually. Sudden, high-intensity light transitions (e.g., camera flashes) can temporarily disrupt state estimation, as the Kalman filter’s linear dynamics model and the GRU’s temporal smoothing both prioritize stability over instantaneous reactivity. 6.2 Potential Application Scenarios Beyond Museum Experiences The hybrid framework’s core methodology—decoupling linear and nonlinear dynamics for environmental adaptation— extends naturally to other domains requiring real-time AR coherence. In industrial maintenance, for instance, technicians often rely on AR overlays to visualize hidden components under variable workshop lighting [18]. The Kalman filter could track equipment vibrations (linear dynamics) while the GRU adapts to oil stains or rust patterns (nonlinear textures). -9-