How do we experience crossmodal correspondent mulsemedia content?

Covaci, Alexandra, B.Saleme, Estevao, Mesfin, Gebremariam, Hussain, Nadia, Kani-Zabihi, Elahe ORCID: and Ghinea, Gheorghita (2019) How do we experience crossmodal correspondent mulsemedia content? IEEE Transactions on Multimedia, 22 (5). pp. 1249-1258. ISSN 1520-9210

[thumbnail of Kani-Zabihi_etal_IEEE_ToM_2019_How_do_we_experience_crossmodal_correspondent_mulsemedia_content.pdf]
Kani-Zabihi_etal_IEEE_ToM_2019_How_do_we_experience_crossmodal_correspondent_mulsemedia_content.pdf - Accepted Version

Download (10MB) | Preview


Sensory studies emerged as a significant influence upon Human Computer Interaction and traditional multimedia. Mulsemedia is an area that extends multimedia addressing issues of multisensorial response through the combination of at least three media, typically a non-traditional media with traditional audio-video content. In this paper, we explore the concepts of Quality of Experience and crossmodal correspondences through a case study of different types of mulsemedia setups. The content is designed following principles of crossmodal correspondence between different sensory dimensions and delivered through olfactory, auditory and vibrotactile displays. The Quality of Experience is evaluated through both subjective (questionnaire) and objective means (eye gaze and heart rate). Results show that the auditory experience has an influence on the olfactory sensorial responses and lessens the perception of lingering odor. Heat maps of the eye gazes suggest that the crossmodality between olfactory and visual content leads to an increased visual attention on the factors of the employed crossmodal correspondence (e.g. colour, brightness and shape).

Item Type: Article
Identifier: 10.1109/TMM.2019.2941274
Additional Information: This paper was partially funded by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503. This study was also part financed by the Coordenacão de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 88881.187844/2018-01. Estêvão B. Saleme also thankfully acknowledges support from the Federal Institute of Espírito Santo. © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This paper appears in IEEE Transactions on Multimedia. First published on-line, 13 September 2019, in IEEEXplore (
Keywords: mulsemedia, QoE, crossmodal correspondence, heart rate, eye gaze
Subjects: Computing > Innovation and user experience > Computing interaction design
Computing > Innovation and user experience > Global software
Computing > Innovation and user experience > Usability
Related URLs:
Depositing User: Elahe Kani-Zabihi
Date Deposited: 10 Sep 2019 08:18
Last Modified: 06 Feb 2024 16:00


Downloads per month over past year

Actions (login required)

View Item View Item