| 1. |
- Förster, Frank, et al.
(författare)
-
Working with troubles and failures in conversation between humans and robots: workshop report
- 2023
-
Ingår i: Frontiers in Robotics and AI. - : Frontiers Media SA. - 2296-9144. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- This paper summarizes the structure and findings from the first Workshop on Troubles and Failures in Conversations between Humans and Robots. The workshop was organized to bring together a small, interdisciplinary group of researchers working on miscommunication from two complementary perspectives. One group of technology-oriented researchers was made up of roboticists, Human-Robot Interaction (HRI) researchers and dialogue system experts. The second group involved experts from conversation analysis, cognitive science, and linguistics. Uniting both groups of researchers is the belief that communication failures between humans and machines need to be taken seriously and that a systematic analysis of such failures may open fruitful avenues in research beyond current practices to improve such systems, including both speech-centric and multimodal interfaces. This workshop represents a starting point for this endeavour. The aim of the workshop was threefold: Firstly, to establish an interdisciplinary network of researchers that share a common interest in investigating communicative failures with a particular view towards robotic speech interfaces; secondly, to gain a partial overview of the “failure landscape” as experienced by roboticists and HRI researchers; and thirdly, to determine the potential for creating a robotic benchmark scenario for testing future speech interfaces with respect to the identified failures. The present article summarizes both the “failure landscape” surveyed during the workshop as well as the outcomes of the attempt to define a benchmark scenario.
|
|
| 2. |
- Jonell, Patrik, et al.
(författare)
-
Crowd-powered design of virtual attentive listeners
- 2017
-
Ingår i: 17th International Conference on Intelligent Virtual Agents, IVA 2017. - Cham : Springer. - 9783319674001 ; , s. 188-191
-
Konferensbidrag (refereegranskat)abstract
- This demo presents a web-based system that generates attentive listening behaviours in a virtual agent acquired from audio-visual recordings of attitudinal feedback behaviour of crowdworkers.
|
|
| 3. |
- Jonell, Patrik, et al.
(författare)
-
Crowdsourced Multimodal Corpora Collection Tool
- 2018
-
Ingår i: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). - Paris. ; , s. 728-734
-
Konferensbidrag (refereegranskat)abstract
- In recent years, more and more multimodal corpora have been created. To our knowledge there is no publicly available tool which allows for acquiring controlled multimodal data of people in a rapid and scalable fashion. We therefore are proposing (1) a novel tool which will enable researchers to rapidly gather large amounts of multimodal data spanning a wide demographic range, and (2) an example of how we used this tool for corpus collection of our "Attentive listener'' multimodal corpus. The code is released under an Apache License 2.0 and available as an open-source repository, which can be found at https://github.com/kth-social-robotics/multimodal-crowdsourcing-tool. This tool will allow researchers to set-up their own multimodal data collection system quickly and create their own multimodal corpora. Finally, this paper provides a discussion about the advantages and disadvantages with a crowd-sourced data collection tool, especially in comparison to a lab recorded corpora.
|
|
| 4. |
- Jonell, Patrik, et al.
(författare)
-
FARMI : A Framework for Recording Multi-Modal Interactions
- 2018
-
Ingår i: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). - Paris : European Language Resources Association. ; , s. 3969-3974
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present (1) a processing architecture used to collect multi-modal sensor data, both for corpora collection and real-time processing, (2) an open-source implementation thereof and (3) a use-case where we deploy the architecture in a multi-party deception game, featuring six human players and one robot. The architecture is agnostic to the choice of hardware (e.g. microphones, cameras, etc.) and programming languages, although our implementation is mostly written in Python. In our use-case, different methods of capturing verbal and non-verbal cues from the participants were used. These were processed in real-time and used to inform the robot about the participants’ deceptive behaviour. The framework is of particular interest for researchers who are interested in the collection of multi-party, richly recorded corpora and the design of conversational systems. Moreover for researchers who are interested in human-robot interaction the available modules offer the possibility to easily create both autonomous and wizard-of-Oz interactions.
|
|
| 5. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction
- 2018
-
Ingår i: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). - Paris. ; , s. 119-127
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present a corpus of multiparty situated interaction where participants collaborated on moving virtual objects on a large touch screen. A moderator facilitated the discussion and directed the interaction. The corpus contains recordings of a variety of multimodal data, in that we captured speech, eye gaze and gesture data using a multisensory setup (wearable eye trackers, motion capture and audio/video). Furthermore, in the description of the multimodal corpus, we investigate four different types of social gaze: referential gaze, joint attention, mutual gaze and gaze aversion by both perspectives of a speaker and a listener. We annotated the groups’ object references during object manipulation tasks and analysed the group’s proportional referential eye-gaze with regards to the referent object. When investigating the distributions of gaze during and before referring expressions we could corroborate the differences in time between speakers’ and listeners’ eye gaze found in earlier studies. This corpus is of particular interest to researchers who are interested in social eye-gaze patterns in turn-taking and referring language in situated multi-party interaction.
|
|
| 6. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
A Systematic Cross-Corpus Analysis of Human Reactions to Robot Conversational Failures
- 2021
-
Ingår i: ICMI 2021 - Proceedings of the 2021 International Conference on Multimodal Interaction. - New York, NY, USA : Association for Computing Machinery (ACM). ; , s. 112-120
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we analyze multimodal behavioral responses to robot failures across different tasks. Two multimodal datasets are examined in which humans interact with guided-task robots in task-oriented dialogues. In both datasets, the robots simulated failures of conversational breakdown and miscommunication typically observed in human-robot interactions. We closely examine human reactions to these failures looking at facial and acoustic features. Our analyses identify the significant behavioral features for automatic detection of such failures in interaction. We also examine human responses to different types of robot failures and if failures occurred early or late in the interaction cause variation in the responses. Our findings indicate that several nonverbal behaviors are consistently present in responses to robots' failures, e.g., gaze and speech prosody, whereas, linguistic features appear to be task-dependent. We discuss how these findings may generalize to other tasks, and how autonomous robots may identify opportunities to detect and recover from failures in interactions with humans.
|
|
| 7. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
Behavioural Responses to Robot Conversational Failures
- 2020
-
Ingår i: HRI '20: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. - New York, NY, USA : ACM Digital Library.
-
Konferensbidrag (refereegranskat)abstract
- Humans and robots will increasingly collaborate in domestic environments which will cause users to encounter more failures in interactions. Robots should be able to infer conversational failures by detecting human users’ behavioural and social signals. In this paper, we study and analyse these behavioural cues in response to robot conversational failures. Using a guided task corpus, where robot embodiment and time pressure are manipulated, we ask human annotators to estimate whether user affective states differ during various types of robot failures. We also train a random forest classifier to detect whether a robot failure has occurred and compare results to human annotator benchmarks. Our findings show that human-like robots augment users’ reactions to failures, as shown in users’ visual attention, in comparison to non-humanlike smart-speaker embodiments. The results further suggest that speech behaviours are utilised more in responses to failures when non-human-like designs are present. This is particularly important to robot failure detection mechanisms that may need to consider the robot’s physical design in its failure detection model.
|
|
| 8. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
Chinese whispers : A multimodal dataset for embodied language grounding
- 2020
-
Ingår i: LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. - : European Language Resources Association (ELRA). ; , s. 743-749
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we introduce a multimodal dataset in which subjects are instructing each other how to assemble IKEA furniture. Using the concept of 'Chinese Whispers', an old children's game, we employ a novel method to avoid implicit experimenter biases. We let subjects instruct each other on the nature of the task: the process of the furniture assembly. Uncertainty, hesitations, repairs and self-corrections are naturally introduced in the incremental process of establishing common ground. The corpus consists of 34 interactions, where each subject first assembles and then instructs. We collected speech, eye-gaze, pointing gestures, and object movements, as well as subjective interpretations of mutual understanding, collaboration and task recall. The corpus is of particular interest to researchers who are interested in multimodal signals in situated dialogue, especially in referential communication and the process of language grounding.
|
|
| 9. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
Embodiment Effects in Interactions with Failing Robots
- 2020
-
Ingår i: CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. - New York, NY, USA : ACM Digital Library.
-
Konferensbidrag (refereegranskat)abstract
- The increasing use of robots in real-world applications will inevitably cause users to encounter more failures in interactions. While there is a longstanding effort in bringing human-likeness to robots, how robot embodiment affects users’ perception of failures remains largely unexplored. In this paper, we extend prior work on robot failures by assessing the impact that embodiment and failure severity have on people’s behaviours and their perception of robots. Our findings show that when using a smart-speaker embodiment, failures negatively affect users’ intention to frequently interact with the device, however not when using a human-like robot embodiment. Additionally, users significantly rate the human-like robot higher in terms of perceived intelligence and social presence. Our results further suggest that in higher severity situations, human-likeness is distracting and detrimental to the interaction. Drawing on quantitative findings, we discuss benefits and drawbacks of embodiment in robot failures that occur in guided tasks.
|
|
| 10. |
- Kontogiorgos, Dimosthenis, 1987-, et al.
(författare)
-
Estimating Uncertainty in Task Oriented Dialogue
- 2019
-
Ingår i: ICMI 2019 - Proceedings of the 2019 International Conference on Multimodal Interaction. - New York, NY, USA : ACM Digital Library. - 9781450368605 ; , s. 414-418
-
Konferensbidrag (refereegranskat)abstract
- Situated multimodal systems that instruct humans need to handle user uncertainties, as expressed in behaviour, and plan their actions accordingly. Speakers’ decision to reformulate or repair previous utterances depends greatly on the listeners’ signals of uncertainty. In this paper, we estimate uncertainty in a situated guided task, as leveraged in non-verbal cues expressed by the listener, and predict that the speaker will reformulate their utterance. We use a corpus where people instruct how to assemble furniture, and extract their multimodal features. While uncertainty is in cases ver- bally expressed, most instances are expressed non-verbally, which indicates the importance of multimodal approaches. In this work, we present a model for uncertainty estimation. Our findings indicate that uncertainty estimation from non- verbal cues works well, and can exceed human annotator performance when verbal features cannot be perceived.
|
|
