Mesfin, Gebremariam, Hussain, Nadia, Kani-Zabihi, Elahe ORCID: https://orcid.org/0000-0002-5679-8512, Covaci, Alexandra, Saleme, Estêvão B. and Ghinea, Gheorghita (2020) QoE of cross-modally mapped mulsemedia: an assessment using eye gaze and heart rate. Multimedia Tools and Applications, 79 (11-12). pp. 7987-8009. ISSN 1380-7501

Preview	PDF Kani-Zabihi_etal_MMTA_2019_QoE_of_cross-modally_mapped_mulsemedia_an_assessment_using_eye_gaze_and_heart_rate.pdf - Accepted Version Available under License Creative Commons Attribution. Download (696kB) | Preview
Preview	PDF Kani-Zabihi_etal_MMTA_2019_QoE_of_cross-modally_mapped_mulsemedia_an_assessment_using_eye_gaze_and_heart_rate.pdf - Published Version Available under License Creative Commons Attribution. Download (2MB) | Preview

Abstract

A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users’ eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users’ QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.

Item Type:	Article
Identifier:	10.1007/s11042-019-08473-5
Additional Information:	This paper was funded by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503. Estêvão B. Saleme thankfully acknowledges support from IFES and CAPES (process no. 88881.187844/2018-01). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/
Keywords:	Mulsemedia, crossmodal correspondence, quality of experience, gaze tracking, heart rate
Subjects:	Computing > Innovation and user experience > Computing interaction design Computing > Innovation and user experience Computing
Related URLs:	Publisher
Depositing User:	Elahe Kani-Zabihi
Date Deposited:	15 Nov 2019 15:27
Last Modified:	04 Nov 2024 11:51
URI:	https://repository.uwl.ac.uk/id/eprint/6534

Downloads

Actions (login required)

View Item