| 1. |
- Al Moubayed, Samer, et al.
(författare)
-
Auditory visual prominence From intelligibility to behavior
- 2009
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer Science and Business Media LLC. - 1783-7677 .- 1783-8738. ; 3:4, s. 299-309
-
Tidskriftsartikel (refereegranskat)abstract
- Auditory prominence is defined as when an acoustic segment is made salient in its context. Prominence is one of the prosodic functions that has been shown to be strongly correlated with facial movements. In this work, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study, a speech intelligibility experiment is conducted, speech quality is acoustically degraded and the fundamental frequency is removed from the signal, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrows raise gestures, which are synchronized with the auditory prominence. The experiment shows that presenting prominence as facial gestures significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a follow-up study examining the perception of the behavior of the talking heads when gestures are added over pitch accents. Using eye-gaze tracking technology and questionnaires on 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch accents opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and the understanding of the talking head.
|
|
| 2. |
- Bresin, Roberto, 1963-, et al.
(författare)
-
Interactive sonification
- 2012
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer Science and Business Media LLC. - 1783-7677 .- 1783-8738. ; 5:3-4, s. 85-86
-
Tidskriftsartikel (refereegranskat)abstract
- In October 2010, Roberto Bresin, Thomas Hermann and Andy Hunt launched a call for papers for a special issue on Interactive Sonification of the Journal on Multimodal User Interfaces (JMUI). The call was published in eight major mailing lists in the field of Sound and Music Computing and on related websites. Twenty manuscripts were submit- ted for review, and eleven of them have been accepted for publication after further improvements. Three of the papers are further developments of works presented at ISon 2010— Interactive Sonification workshop. Most of the papers went through a three-stage review process.The papers give an interesting overview of the field of Interactive Sonification as it is today. Their topics include the sonification of data exploration and of motion, a new sound synthesis model suitable for interactive sonification applications, a study on perception in the everyday periphery of attention, and the proposal of a conceptual framework for interactive sonification.
|
|
| 3. |
- Burger, Birgitta, et al.
(författare)
-
Communication of Musical Expression by Means of Mobile Robot Gestures
- 2010
-
Ingår i: Journal on Multimodal User Interfaces. - Stockholm : Springer Science and Business Media LLC. - 1783-7677 .- 1783-8738. ; 3:1, s. 109-118
-
Tidskriftsartikel (refereegranskat)abstract
- We developed a robotic system that can behave in an emotional way. A 3-wheeled simple robot with limited degrees of freedom was designed. Our goal was to make the robot displaying emotions in music performance by performing expressive movements. These movements have been compiled and programmed based on literature about emotion in music, musicians’ movements in expressive performances, and object shapes that convey different emotional intentions. The emotions happiness, anger, and sadness have been implemented in this way. General results from behavioral experiments show that emotional intentions can be synthesized, displayed and communicated by an artificial creature, also in constrained circumstances.
|
|
| 4. |
- Crook, N., et al.
(författare)
-
Generating context-sensitive ECA responses to user barge-in interruptions
- 2012
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer Science and Business Media LLC. - 1783-7677 .- 1783-8738. ; 6:1-2, s. 13-25
-
Tidskriftsartikel (refereegranskat)abstract
- We present an Embodied Conversational Agent (ECA) that incorporates a context-sensitive mechanism for handling user barge-in. The affective ECA engages the user in social conversation, and is fully implemented. We will use actual examples of system behaviour to illustrate. The ECA is designed to recognise and be empathetic to the emotional state of the user. It is able to detect, react quickly to, and then follow up with considered responses to different kinds of user interruptions. The design of the rules which enable the ECA to respond intelligently to different types of interruptions was informed by manually analysed real data from human-human dialogue. The rules represent recoveries from interruptions as two-part structures: an address followed by a resumption. The system is robust enough to manage long, multi-utterance turns by both user and system, which creates good opportunities for the user to interrupt while the ECA is speaking.
|
|
| 5. |
- d’Alessandro, N., et al.
(författare)
-
RAMCESS 2.x FRAMEWORK - EXPRESSIVE VOICE ANALYSIS FOR REALTIME AND ACCURATE SYNTHESIS OF SINGING
- 2008
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer. - 1783-7677 .- 1783-8738. ; 2, s. 133-144
-
Tidskriftsartikel (refereegranskat)abstract
- In this paper we present the work that has been achieved in the context of the second version of the RAMCESS singing synthesis framework. The main improvement of this study is the integration of new algorithms for expressive voice analysis, especially the separation of the glottal source and the vocal tract. Realtime synthesis modules have also been refined. These elements have been integrated in an existing digital instrument: the HANDSKETCH 1.X, a bimanual controller. Moreover this digital instrument is compared to existing systems.
|
|
| 6. |
- Dubus, Gaël
(författare)
-
Evaluation of four models for the sonification of elite rowing
- 2012
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer. - 1783-7677 .- 1783-8738. ; 5:3-4, s. 143-156
-
Tidskriftsartikel (refereegranskat)abstract
- Many aspects of sonification represent potential benefits for the practice of sports. Taking advantage of the characteristics of auditory perception, interactive sonification offers promising opportunities for enhancing the training of athletes. The efficient learning and memorizing abilities pertaining to the sense of hearing, together with the strong coupling between auditory and sensorimotor systems, make the use of sound a natural field of investigation in quest of efficiency optimization in individual sports at a high level. This study presents an application of sonification to elite rowing, introducing and evaluating four sonification models.The rapid development of mobile technology capable of efficiently handling numerical information offers new possibilities for interactive auditory display. Thus, these models have been developed under the specific constraints of a mobile platform, from data acquisition to the generation of a meaningful sound feedback. In order to evaluate the models, two listening experiments have then been carried out with elite rowers. Results show a good ability of the participants to efficiently extract basic characteristics of the sonified data, even in a non-interactive context. Qualitative assessment of the models highlights the need for a balance between function and aesthetics in interactive sonification design. Consequently, particular attention on usability is required for future displays to become widespread.
|
|
| 7. |
- Elmquist, Elias, 1996-, et al.
(författare)
-
SonAir: the design of a sonification of radar data for air traffic control
- 2023
-
Ingår i: Journal on Multimodal User Interfaces. - : SPRINGER. - 1783-7677 .- 1783-8738. ; 17:3, s. 137-149
-
Tidskriftsartikel (refereegranskat)abstract
- Along with the increase of digitalization and automation, a new kind of working environment is emerging in the field of air traffic control. Instead of situating the control tower at the airport, it is now possible to remotely control the airport at any given location, i.e. in a remote tower center (RTC). However, by controlling the airport remotely, the situational awareness and sense of presence might be compromised. By using directional sound, a higher situational awareness could potentially be achieved while also offloading the visual perception which is heavily used in air traffic control. Suitable use cases for sonification in air traffic control were found through workshops with air traffic controllers. A sonification design named SonAir was developed based on the outcome of the workshops, and was integrated with an RTC simulator for evaluating to what degree SonAir could support air traffic controllers in their work. The results suggest that certain aspects of SonAir could be useful for air traffic controllers. A continuous sonification where the spatial positioning of aircraft were conveyed was experienced to be partially useful, but the intrusiveness of SonAir should be further considered to fit the air traffic controllers’ needs. An earcon that conveyed when an aircraft enters the airspace and from which direction was considered useful to support situational awareness.
|
|
| 8. |
- Fabiani, Marco, et al.
(författare)
-
Interactive sonification of expressive hand gestures on a handheld device
- 2012
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer Science and Business Media LLC. - 1783-7677 .- 1783-8738. ; 6:1-2, s. 49-57
-
Tidskriftsartikel (refereegranskat)abstract
- We present here a mobile phone application called MoodifierLive which aims at using expressive music performances for the sonification of expressive gestures through the mapping of the phone’s accelerometer data to the performance parameters (i.e. tempo, sound level, and articulation). The application, and in particular the sonification principle, is described in detail. An experiment was carried out to evaluate the perceived matching between the gesture and the music performance that it produced, using two distinct mappings between gestures and performance. The results show that the application produces consistent performances, and that the mapping based on data collected from real gestures works better than one defined a priori by the authors.
|
|
| 9. |
|
|
| 10. |
- Frid, Emma, et al.
(författare)
-
Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task
- 2018
-
Ingår i: Journal on Multimodal User Interfaces. - : Springer. - 1783-7677 .- 1783-8738. ; 13:4, s. 279-290
-
Tidskriftsartikel (refereegranskat)abstract
- In this paper we present a study on the effects of auditory- and haptic feedback in a virtual throwing task performed with a point-based haptic device. The main research objective was to investigate if and how task performance and perceived intuitiveness is affected when interactive sonification and/or haptic feedback is used to provide real-time feedback about a movement performed in a 3D virtual environment. Emphasis was put on task solving efficiency and subjective accounts of participants’ experiences of the multimodal interaction in different conditions. The experiment used a within-subjects design in which the participants solved the same task in different conditions: visual-only, visuohaptic, audiovisual and audiovisuohaptic. Two different sound models were implemented and compared. Significantly lower error rates were obtained in the audiovisuohaptic condition involving movement sonification based on a physical model of friction, compared to the visual-only condition. Moreover, a significant increase in perceived intuitiveness was observed for most conditions involving haptic and/or auditory feedback, compared to the visual-only condition. The main finding of this study is that multimodal feedback can not only improve perceived intuitiveness of an interface but that certain combinations of haptic feedback and movement sonification can also contribute with performance-enhancing properties. This highlights the importance of carefully designing feedback combinations for interactive applications.
|
|
