| 1. |
|
|
| 2. |
- Axelsson, Agnes, 1992-
(author)
-
Adaptive Robot Presenters : Modelling Grounding in Multimodal Interaction
- 2023
-
Doctoral thesis (other academic/artistic)abstract
- This thesis addresses the topic of grounding in human-robot interaction, that is, the process by which the human and robot can ensure mutual understanding. To explore this topic, the scenario of a robot holding a presentation to a human audience is used, where the robot has to process multimodal feedback from the human in order to adapt the presentation to the human's level of understanding.First, the use of behaviour trees to model real-time interactive processes of the presentation is addressed. A system based on the behaviour tree architecture is used in a semi-automated Wizard-of-oz experiment, showing that audience members prefer an adaptive system to a non-adaptive alternative.Next, the thesis addresses the use of knowledge graphs to represent the content of the presentation given by the robot. By building a small, local knowledge graph containing properties (edges) that represent facts about the presentation, the system can iterate over that graph and consistently find ways to refer to entities by referring to previously grounded content. A system based on this architecture is implemented, and an evaluation using simulated users is presented. The results show that crowdworkers comparing different adaptation strategies are sensitive to the types of adaptation enabled by the knowledge graph approach.In a face-to-face presentation setting, feedback from the audience can potentially be expressed through various modalities, including speech, head movements, gaze, facial gestures and body pose. The thesis explores how such feedback can be automatically classified. A corpus of human-robot interactions is annotated, and models are trained to classify human feedback as positive, negative or neutral. A relatively high accuracy is achieved by training simple classifiers with signals found mainly in the speech and head movements.When knowledge graphs are used as the underlying representation of the system's presentation, some consistent way of generating text, that can be turned into speech, is required. This graph-to-text problem is explored by proposing several methods, both template-based and methods based on zero-shot generation using large language models (LLMs). A novel evaluation method using a combination of factual, counter-factual and fictional graphs is proposed. Finally, the thesis presents and evaluates a fully automated system using all of the components above. The results show that audience members prefer the adaptive system to a non-adaptive system, matching the results from the beginning of the thesis. However, we note that clear learning results are not found, which means that the entertainment aspects of the presentation are perhaps more prominent than the learning aspects.
|
|
| 3. |
- Bell, Linda, et al.
(author)
-
Modality Convergence in a Multimodal Dialogue System
- 2000
-
In: Proceedings of Götalog. ; , s. 29-34
-
Conference paper (peer-reviewed)abstract
- When designing multimodal dialogue systems allowing speech as well as graphical operations, it is important to understand not only how people make use of the different modalities in their utterances, but also how the system might influence a user’s choice of modality by its own behavior. This paper describes an experiment in which subjects interacted with two versions of a simulated multimodal dialogue system. One version used predominantly graphical means when referring to specific objects; the other used predominantly verbal referential expressions. The purpose of the study was to find out what effect, if any, the system’s referential strategy had on the user’s behavior. The results provided limited support for the hypothesis that the system can influence users to adopt another modality for the purpose of referring
|
|
| 4. |
- Bell, Linda, et al.
(author)
-
Real-time Handling of Fragmented Utterances
- 2001
-
In: Proceedings of the NAACL Workshop on Adaption in Dialogue Systems.
-
Conference paper (peer-reviewed)abstract
- this paper, we discuss an adaptive method of handling fragmented user utterances to a speech-based multimodal dialogue system. Inserted silent pauses between fragments present the following problem: Does the current silence indicate that the user has completed her utterance, or is the silence just a pause between two fragments, so that the system should wait for more input? Our system incrementally classifies user utterances as either closing (more input is unlikely to come) or non-closing (more input is likely to come), partly depending on the current dialogue state. Utterances that are categorized as non-closing allow the dialogue system to await additional spoken or graphical input before responding
|
|
| 5. |
- Bell, Linda, et al.
(author)
-
The Swedish NICE Corpus – Spoken dialogues between children and embodied characters in a computer game scenario
- 2005
-
In: Proceedings Interspeech 2005 - Eurospeech. - Lisbon, Portugal : ISCA. ; , s. 2765-2768, s. 2765-2768
-
Conference paper (peer-reviewed)abstract
- This article describes the collection and analysis of a Swedish database of spontaneous and unconstrained children-machine dialogues. The Swedish NICE corpus consists of spoken dialogues between children aged 8 to 15 and embodied fairytale characters in a computer game scenario. Compared to previously collected corpora of children's computer-directed speech, the Swedish NICE corpus contains extended interactions, including three-party conversation, in which the young users used spoken dialogue as the primary means of progression in the game.
|
|
| 6. |
- Boye, Johan, et al.
(author)
-
Contextual reasoning in multimodal dialogue systems : two case studies
- 2004
-
In: Proceedings of The 8th Workshop on the Semantics and Pragmatics of Dialogue Catalogue'04. - Barcelona. ; , s. 19-21
-
Conference paper (peer-reviewed)abstract
- This paper describes an approach to contextual reasoning for interpretation ofspoken multimodal dialogue. The approach is based on combining recencybased search for antecedents with an object-oriented domain representation insuch a way that the search is highly constrained by the type information of theantecedents. By furthermore representingcandidate antecedents from the dialoguehistory and visual context in a uniformway, a single machinery (based on -reduction in lambda calculus) can be usedfor resolving many kinds of underspecified utterances. The approach has beenimplemented in two highly different domains.
|
|
| 7. |
- Boye, Johan
(author)
-
Dependency-based groundness analysis of functional logic programs
- 1993
-
Licentiate thesis (other academic/artistic)abstract
- The object of study in this thesis is a class of functional logic programs, where the functions are implemented in an external functional or imperative language. The contributions are twofold:Firstly, an operational semantics is formally defined. The key idea is that non-ground function calls selected for unification are delayed and retained in form of constraints until their arguments become ground. With this strategy two problems arise: (1) Given a program P and an initial goal, will any delayed unifications remain unresolved after computation? (2) For every function call f(X) in P, find a safe evaluation point for f(X), i.e. a point in P where X always will be bound to a ground term, and thus f(X) can be evaluated.Secondly, we present a static groundness analysis technique which enables us to solve problems (1) and (2) in a uniform way. The analysis method is dependency-based, exploiting analogies between logic programs and attribute grammars.
|
|
| 8. |
- Boye, Johan
(author)
-
Dialogue management for automatic troubleshooting and other problem-solving applications
- 2007
-
In: Proceedings of the 8th SIGDial workshop on discourse and dialogue. ; , s. 247-255
-
Conference paper (peer-reviewed)abstract
- This paper describes a dialogue management method suitable for automatic troubleshooting and other problem-solving applications. The method has a theoremproving flavor, in that it recursively decomposes tasks into sequences of sub-tasks and atomic actions. An explicit objective when designing the method was that it should be usable by other people than the designers themselves, notably IVR application developers. Therefore the method has a transparent execution model, and is configurable using a simple scripting language.
|
|
| 9. |
- Boye, Johan, et al.
(author)
-
How to do dialogue in a fairy-tale world
- 2005
-
In: Proceedings of the 6th SIGDial workshop on discourse and dialogue.
-
Conference paper (peer-reviewed)abstract
- The work presented in this paper is an endeavor tocreate a prototype of a computer game with spokendialogue capabilities. Advanced spoken dialogue hasthe potential to considerably enrich computer games,where it for example would allow players to refer topast events and to objects currently not visible onthe screen. It would also allaow users to interactsocially and to negotiate solutions with the gamecharacters. The game takes place in a fairy-taleworld, and features two different fairy-talecharacters, who can interact with the player and witheach other using spoken dialogue. The fairy-talecharacters are separate entities in the sense that eachcharacter has its own set of goals and its ownperception of the world. This paper gives anoverview of the functionality of the implementeddialogue manager in the NICE fairy-tale gamesystem.
|
|
| 10. |
- Boye, Johan, et al.
(author)
-
Language-Processing Strategies and Mixed-Initiative Dialogues
- 1999
-
Reports (other academic/artistic)abstract
- We describe an implemented spoken-language dialogue system for a travel-planning domain, which accesses a commercially available travel-information web-server and supports a flexible mixed-initiative dialogue strategy. We argue, based on data from initial Wizard-of-Oz experiments, that mixed-initiative strategies are appropriate for many types of user, but require more sophisticated architectures for processing of language and dialogue; we then use these observations to motivate an architecture which combines parallel deep and shallow natural language analysis engines and an agenda-driven dialogue manager. We outline the top-level processing strategy used by the dialogue manager, and also a novel formalism, which we call Flat Utterance Description, that allows us to reduce the output of the deep and shallow language-processing engines to a common representation.
|
|
