THEIA-XR - Effective eXtended Reality in Machine Operation

THEIAXR has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101092861. Any information of results reflects only the author’s view and the European Commission is not responsible for any use that may be made of the information it contains. Effective eXtended Reality in Machine Operation THEIAXR MAKING THE NON-PERCEIVABLE PERCEIVABLE Martijn Rooker, PhD Senior Innovation Projects & Funding Manager TTControl GmbH 2 2025. 11. 05. Challenges Off-Highway mobile machinery offer limited visibility to the human operator. ›Environmental conditions ›Mobile machinery design ›Remote operation Following this, the defined challenges are: ›Definition of intuitive HMIs ›Intuitive interaction with XR technologies ›Available data ›Interpreting and presentation of (limited) available data ›Processing on mobile machinery ›Human-centered data (incl. privacy) / Co-Design THEIAXR | Martijn Rooker | Public Source: youtube https://www.cdc.gov/ Source: youtube 3 2025. 11. 05. THEIAXR Goals THEIAXR will deliver the following key innovations: •eXtended Reality technologies applicable to stateof-the-art operating environments (cabin-based, and tele-operational) to make the invisible visible resp. the non-perceivable perceivable to the human operator •XR interaction concept and design principles that facilitate a human friendly and intuitive design of XR interaction in industrial applications •A transdisciplinary co-design methodology to design and develop multimodal XR information presentation and interaction solutions THEIAXR | Martijn Rooker | Public 4 2025. 11. 05. THEIAXR Developments THEIAXR | Martijn Rooker | Public 5 2025. 11. 05. Use Cases Snow Grooming Logistics Construction ©prinoth ©KALMAR Source: liebherr.com THEIAXR | Martijn Rooker | Public 6 2025. 11. 05. Concepts XR user interface key concepts •Comprehensive and compatible XR user interfaces concepts were created •A list of assistive features were realized through different forms of extending information presentation •Interactive functions were also conceptualized and integrated •Basis for elaborating design details and technical realization THEIAXR | Martijn Rooker | Public 7 2025. 11. 05. Integrated Solutions mobile machinery data environment data Sensors / Actuators Camera(s) Camera(s) ECU(s) Gateway(s) Central processor Output devices Simulated environment / machine data THEIAXR | Martijn Rooker | Public 8 2025. 11. 05. Prototype Solutions ›Novel sensor setups: 1. PanoTherm: Panoramic Thermal Imaging for Object Detection and Tracking [1] [1] Kernbauer et al.: PanoTherm: Panoramic Thermal Imaging for Object Detection and Tracking, VISAPP'2024 THEIAXR | Martijn Rooker | Public 9 2025. 11. 05. Panotherm Results THEIAXR | Martijn Rooker | Public 16 2025. 11. 05. Haptic / Force Feedback Interaction Haptic feedback for control of machinery Feedback about: •Motion •Excavating •Handling of cargo •Haptic Track controls for Motion •Haptic handling of actuators THEIAXR | Martijn Rooker | Public 17 2025. 11. 05. Virtual Worlds THEIAXR | Martijn Rooker | Public 18 2025. 11. 05. Virtual Worlds Basis for remote operation •Information indication •Direction •Grasping points •Hidden Information THEIAXR | <Presenter> 19 2025. 11. 05. Virtual Worlds THEIAXR | Martijn Rooker | Public 20 2025. 11. 05. First Feedback from Operators The fact that you can also see the depths where the bucket is in the side view is just cool! World class! That‘s a great idea. I‘ve not tried it yet, but I‘m convinced by the look of it Great concept, if it works like this. Especially the swivelling range and object recognition. We mustn't forget that driving an excavator means you have to be constantly focused. (...) I think that if you try to display too many details at the same time, you will eventually overwhelm the excavator driver. THEIAXR | Martijn Rooker | Public THEIAXR has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101092861. Any information of results reflects only the author’s view and the European Commission is not responsible for any use that may be made of the information it contains. Thank you for your attention Martijn Rooker Senior Innovation Projects & Funding Manager [email protected] +43 1 585 3434 4207 https://www.theia-xr.eu https://www.linkedin.com/company/theia-xr https://zenodo.org/communities/theia-xr/