Multimodal User Experience in Extended Reality: Exploring Hand Tracking, Voice, and Passthrough Interactions
Kojic, T., Vergari, M., Warsinke, M., Ali, D., Möller, S. & Voigt-Antons, J.-N.
Paper presented at the international workshop on IMmersive Mixed and Virtual Environment Systems (MMVE 2025), Stellenbosch, South Africa
Abstract
With the increasing adoption of Extended Reality (XR) in various fields, understanding how different interaction techniques influence user experience is of high importance. In this paper, the impact of multimodal interaction techniques---hand tracking, voice commands, and passthrough technology---on User Experience (UX) in XR environments is examined. These modalities are analyzed in relation to user engagement, immersion, and satisfaction in both virtual and passthrough reality settings. A hybrid methodology was employed, combining quantitative and qualitative data collection through user testing, self-assessment tools, and standardized questionnaires such as the Igroup Presence Questionnaire (IPQ), Self-Assessment Manikin (SAM), and the User Experience Questionnaire (UEQ-S). Results indicate that multimodal interactions significantly enhance both hedonic and pragmatic dimensions of UX, with passthrough combined with voice commands emerging as the most preferred configuration. Statistical analysis further reveals that passthrough is generally preferred over non-passthrough, while voice commands improve usability across both conditions. However, passthrough can introduce potential distractions, depending on the task context. These findings contribute to the design of more intuitive and immersive XR applications by providing insights into the opportunities and challenges of multimodal interaction. The results support the development of human-centered extended reality experiences and inform more effective implementations for future XR systems.
Cite as
Kojic, T., Vergari, M., Warsinke, M., Ali, D., Möller, S. & Voigt-Antons, J.-N. (2025, March). Multimodal User Experience in Extended Reality: Exploring Hand Tracking, Voice, and Passthrough Interactions. Paper presented at the international workshop on IMmersive Mixed and Virtual Environment Systems (MMVE 2025), Stellenbosch, South Africa.
