A Preliminary Investigation on a Multimodal Controller and Freehand Based Interaction in Virtual Reality

In the last years, the synergy between VR and HCI has dramatically increased the user’s feeling of immersion within virtual scenes, improving VR applications’ user experience and usability. Two main aspects have emerged with the evolution of these technologies concerning immersion in the 3D scene: locomotion within the 3D scene and interaction with the components of the 3D scene. Locomotion with classical freehand approaches based on hand tracking can be stressful for the user due to the need to keep the hand still in a specific position for a long time to activate the locomotion gesture. Likewise, using a classic Head Mounted Display (HMD) controller for the interaction with the 3D scene components could be unnatural for the user, using the hand to pinch and grab the 3D objects. This paper proposes a multimodal approach, mixing the Leap Motion and 6-DOF controller to navigate and interact with the 3D scene to reduce the locomotion gesture stress based on the hand tracking and increase the immersion and interaction feeling using freehand to interact with the 3D scene objects.