Performance Evaluation of an XR-based Immersive Room for Climate Change Storytelling Using Multisensory Interaction

Climate change communication increasingly requires innovative approaches to enhance public awareness. However, existing XR-based climate communication systems rarely provide rigorous technical performance evaluation of integrated multimodal environments. This study presents the design and technical performance evaluation of an XR-based immersive system integrating three-wall projection mapping, gesture and voice interaction, and multisensory physical feedback. The system is implemented using a unified pipeline combining Unity, MediaPipe, TouchDesigner, Resolume, and Arduino to enable real-time multimodal interaction. Experimental evaluations were conducted to assess system responsiveness and interaction reliability, including latency analysis using a 240 FPS high-speed camera, gesture recognition accuracy under varying lighting and distance conditions, and voice recognition accuracy across multiple noise levels. Results show a consistent user-perceived latency of 16.68 ms, gesture accuracy of 90–100% in bright conditions and 90% at 160 cm distance in low-light conditions, and voice recognition accuracy ranging from 100% (30 dB) to 30% (80 dB). These findings demonstrate that the system achieves stable and responsive multimodal interaction in an XR immersive environment. The study highlights key design considerations and operational constraints, providing a technical foundation for future development of XR-based interactive systems. This study is positioned as an initial technical performance evaluation of a multimodal XR system.

Climate change storytelling; Extended reality; Gesture recognition; Multisensory interaction; Voice interaction

Bailenson, J. (2018). Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do. W. W. Norton & Company.

Baroroh, D. K., & Agarwal, A. (2022). Immersive Technologies in Indonesia Faces ‘New Normal’ COVID-19. International Journal of Technology, 13(3), 633–642. https://doi.org/10.14716/ijtech.v13i3.5220

Birch, S., Wernert, E. A., Danielson, K., Filippelli, G., & Hines, J. (2023). Visualizing the Causes and Impacts of Climate Change with XR Technologies. PEARC 2023 - Computing for the Common Good: Practice and Experience in Advanced Research Computing. https://doi.org/10.1145/3569951.3603630

Bustos-Lopez, G., Aguirre-Villalobos, E. R., & Meingast, K. (2024). XR for Transformable and Interactive Design. Media and Communication, 12. https://doi.org/10.17645/mac.8603

Climate Division of the Meteorological and Geophysical Agency (BMKG). (2024). Indonesia’s Climate and Air Quality Notes 2024.

Coaction Indonesia. (2024). Dynamics of Indonesia’s Climate Agenda.

Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69. https://doi.org/10.1126/science.1167311

Ecstein, D., Kunzel, V., & Schafer, L. (2021). Global Climate Risk Index 2021.

Fernández Galeote, D., Legaki, N. Z., & Hamari, J. (2023). Climate Connected: An Immersive VR and PC Game for Climate Change Engagement. CHI PLAY 2023 Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play. https://doi.org/10.1145/3573382.3616053

Gregory, J. (2018). Game Engine Architecture. Taylor & Francis. http://taylorandfrancis.com

Hu, H., Tan, T., & Qian, Y. (2018). Generative adversarial networks based data augmentation for noise robust speech recognition. ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings. https://doi.org/10.1109/ICASSP.2018.8462624

Huang, T. C., & Tseng, H. P. (2025). Extended Reality in Applied Sciences Education: A Systematic Review. Applied Sciences. https://doi.org/10.3390/app15074038

Ignacio, J., Dolmans, D., Scherpbier, A., Rethans, J. J., Chan, S., & Liaw, S. Y. (2015). Comparison of standardized patients with high-fidelity simulators for managing stress and improving performance in clinical deterioration: A mixed methods study. Nurse Education Today, 35(12). https://doi.org/10.1016/j.nedt.2015.05.009

Ioannou, A., Bhagat, K. K., & Johnson-Glenberg, M. C. (2021). Guest Editorial: Learning Experience Design: Embodiment, Gesture, and Interactivity in XR. Educational Technology and Society, 24(2).

Jamil, M., Hadiyanto, & Sanjaya, R. (2024). Optimizing 3D Assets and Character Modeling of the Mixed Reality Simulator in a Disaster Mitigation Learning Using Vertex Decimation and Depth-of-Field Algorithm. Lecture Notes in Networks and Systems. https://doi.org/10.1007/978-981-99-6547-2_17

Jeong, J., Lee, C.-K., Hong, K., Yeom, J., & Lee, B. (2014). Projection-type dual-view three-dimensional display system based on integral imaging. Applied Optics, 53(27). https://doi.org/10.1364/AO.53.000G12

Jerald, J. (2015). Human-Centered Interaction. In The VR Book. Morgan & Claypool Publishers. https://doi.org/10.1145/2792790.2792821

Kang, J., Diederich, M., Lindgren, R., & Junokas, M. (2021). Gesture Patterns and Learning in an Embodied XR Science Simulation. Educational Technology and Society, 24(2).

Kim, H., & Shin, J. W. (2021). Target exaggeration for deep learning-based speech enhancement. Digital Signal Processing, 116. https://doi.org/10.1016/j.dsp.2021.103109

Kizhevska, E., Ferreira-Brito, F., Guerreiro, T., & Luštrek, M. (2023). Using Virtual Reality to Elicit Empathy: A Narrative Review. CEUR Workshop Proceedings.

Kopcha, T. J., Ocak, C., & Qian, Y. (2021). Analyzing children’s computational thinking through embodied interaction with technology: A multimodal perspective. Educational Technology Research and Development, 69(4). https://doi.org/10.1007/s11423-020-09832-y

Lee, H. L., & Romero, J. R. (2023). IPCC, 2023: Sections. In: Climate Change 2023: Synthesis Report. Contribution of Working Groups I, II and III to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change [Edited by H. Lee and J. Romero].

Lee-Cultura, S., & Giannakos, M. (2020). Embodied Interaction and Spatial Skills: A Systematic Review of Empirical Studies. Interaction Design and Architecture(s). https://doi.org/10.1093/iwcomp/iwaa023

Liu, S., Xu, X., & Claypool, M. (2022). A Survey and Taxonomy of Latency Compensation Techniques for Network Computer Games. ACM Computing Surveys, 54(11s). https://doi.org/10.1145/3519023

Makransky, G., & Petersen, G. B. (2021). The Cognitive Affective Model of Immersive Learning (CAMIL): A Theoretical Research-Based Model of Learning in Immersive Virtual Reality. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09586-2

Mikawa, Y., Sueishi, T., Watanabe, Y., & Ishikawa, M. (2018). VarioLight: Hybrid Dynamic Projection Mapping Using High-Speed Projector and Optical Axis Controller. SIGGRAPH Asia 2018 Emerging Technologies. https://doi.org/10.1145/3275476.3275481

Miranto, C., Firmanda, A., Rante, H., Sukaridhoto, S., Zainuddin, M. A., & Rahman, H. (2025). Performance analysis of 3D assets in virtual reality simulations for climate change: A case study in sustainable energy systems. Bulletin of Electrical Engineering and Informatics, 14(5), 3659–3670. https://doi.org/10.11591/eei.v14i5.9532

Mufarroha, F. A., & Utaminingrum, F. (2017). Hand gesture recognition using adaptive network based fuzzy inference system and K-nearest neighbor. International Journal of Technology, 8(3), 559–567. https://doi.org/10.14716/ijtech.v8i3.3146

Nossier, S. A., Wall, J., Moniri, M., Glackin, C., & Cannings, N. (2021). An experimental analysis of deep learning architectures for supervised speech enhancement. Electronics, 10(1). https://doi.org/10.3390/electronics10010017

Pham, H. H., Salmane, H., Khoudour, L., Crouzil, A., Velastin, S. A., & Zegers, P. (2020). A unified deep framework for joint 3D pose estimation and action recognition from a single RGB camera. Sensors, 20(7). https://doi.org/10.3390/s20071825

Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers and Education, 147. https://doi.org/10.1016/j.compedu.2019.103778

Rante, H., Achmad, Z., Suaib, N., Prasetya, S., Avianto, T., Hermanu, A., Alfarezi, F., & Wijaya, R. (2024). Preliminary Development of Vircadia Virtual Reality Platform for Monitoring Water Quality Powered by Solar Panels. http://www.joiv.org/index.php/joiv

Sherman, W. R., & Craig, A. B. (2018). Understanding Virtual Reality: Interface, Application, and Design (2nd ed.). Morgan Kaufmann. https://doi.org/10.1016/C2013-0-18583-2

Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 74. https://doi.org/10.3389/frobt.2016.00074

Spence, C., & Gallace, A. (2011). Multisensory design: Reaching out to touch the consumer. Psychology and Marketing, 28(3), 267–308. https://doi.org/10.1002/mar.20392

Treal, T., Jackson, P. L., Jeuvrey, J., Vignais, N., & Meugnot, A. (2021). Natural human postural oscillations enhance the empathic response to a facial pain expression in a virtual character. Scientific Reports, 11(1). https://doi.org/10.1038/s41598-021-91710-5

Viswakumar, A., Rajagopalan, V., Ray, T., Gottipati, P., & Parimi, C. (2022). Development of a Robust, Simple, and Affordable Human Gait Analysis System Using Bottom-Up Pose Estimation With a Smartphone Camera. Frontiers in Physiology, 12. https://doi.org/10.3389/fphys.2021.784865

Vroomen, J., & Keetels, M. (2010). Perception of intersensory synchrony: A tutorial review. Attention, Perception, and Psychophysics, 72(4), 871–884. https://doi.org/10.3758/APP.72.4.871

Wakunami, K., Hsieh, P.-Y., Oi, R., Senoh, T., Sasaki, H., Ichihashi, Y., Okui, M., Huang, Y.-P., & Yamamoto, K. (2016). Projection-type see-through holographic three-dimensional display. Nature Communications, 7. https://doi.org/10.1038/ncomms12954

Wallace, M. T., Woynaroski, T. G., & Stevenson, R. A. (2020). Multisensory integration as a window into orderly and disrupted cognition and communication. Annual Review of Psychology, 71, 193–219. https://doi.org/10.1146/annurev-psych-010419-051112

Zhang, X., Yang, H., Liu, C., Tong, Q., Xiu, A., Kong, L., Dan, M., Gao, C., Gao, M., Che, H., Wang, X., & Wu, G. (2025). XR-based Interactive Visualization Platform for Real-time Exploring Dynamic Earth Science Data [SSRN Preprint]. https://ssrn.com/abstract=4769475