Automatic design of multimodal presentations

We describe our attempt to integrate multiple AI components such as planning, knowledge representation, natural language generation, and graphics generation into a functioning prototype called WIP that plans and coordinates multimodal presentations in which all material is generated by the system. WIP allows the generation of alternate presentations of the same content taking into account various contextual factors such as the user\u27s degree of expertise and preferences for a particular output medium or mode. The current prototype of WIP generates multimodal explanations and instructions for assembling, using, maintaining or repairing physical devices. This paper introduces the task, the functionality and the architecture of the WIP system. We show that in WIP the design of a multimodal document is viewed as a non-monotonic process that includes various revisions of preliminary results, massive replanning and plan repairs, and many negotiations between design and realization components in order to achieve an optimal division of work between text and graphics. We describe how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. Finally, we discuss the generation of cross-modal expressions that establish referential relationships between text and graphics elements

Situationally Aware In-Car Information Presentation Using Incremental Speech Generation: Safer, and More Effective

Kousidis S, Kennington C, Baumann T, Buschmeier H, Kopp S, Schlangen D. Situationally Aware In-Car Information Presentation Using Incremental Speech Generation: Safer, and More Effective. In: Proceedings of the EACL 2014 Workshop on Dialogue in Motion. Gothenburg, Sweden; 2014: 68-72.Holding non-co-located conversations while driving is dangerous (Horrey and Wickens, 2006; Strayer et al., 2006), much more so than conversations with physically present, “situated” interlocutors (Drews et al., 2004). In-car dialogue systems typically resemble non-co-located conversations more, and share their negative impact (Strayer et al., 2013). We implemented and tested a simple strategy for making in-car dialogue systems aware of the driving situation, by giving them the capability to interrupt themselves when a dangerous situation is detected, and resume when over. We show that this improves both driving performance and recall of system-presented information, compared to a non-adaptive strategy

Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation

Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D. Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation. In: AutomotiveUI 2014: Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. Seattle, Washington, USA; 2014: 7:1-7:7.It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers infor- mation via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes

Interpretation and generation incremental management in natural interaction systems

Human interaction develops as an exchange of contributions between participants. The construction of a contribution is not an activity unilaterally created by the participant who produces it, but rather it constitutes a combined activity between the producer and the rest of the participants who take part in the interaction, by means of simultaneous feedback. This paper presents an incremental approach (without losing sight of how turns are produced throughout time), in which the interpretation of contributions is done as they take place, and the final generated contributions are the result of constant rectifications, reformulations and cancellations of the initially formulated contributions. The Continuity Manager and the Processes Coordinator components are proposed. The integration of these components in natural interaction systems allow for a joint approach to these problems. Both have been implemented and evaluated in a real framework called LaBDA-Interactor System which has been applied to the "dictation domain". We found that the degree of naturalness of this turn-taking approach is very close to the human one and it significantly improves the interaction cycle. (c) 2012 British Informatics Society Limited.The development of this approach and its construction as part of the Natural Interaction System LaBDA-Interactor has been par-tially supported by MA2VICMR (Regional Government of Madrid, S2009/TIC-1542), SemAnts (Spanish Ministry of Industry, Tourism and Trade, AVANZA I+D TSI-020110-2009-419); THUBAN (Spanish Ministry of Education and Science, TIN2008-02711); and ‘Access Channel to Digital Resources and Contents’ (Spanish Ministry of Education and Science, TSI-020501-2008-54).Publicad

Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation

Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D. Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation. In: AutomotiveUI 2014: Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. Seattle, Washington, USA; 2014: 7:1-7:7.It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers infor- mation via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes

A Multimodal In-Car Dialogue System That Tracks The Driver's Attention

Kousidis S, Kennington C, Baumann T, Buschmeier H, Kopp S, Schlangen D. A Multimodal In-Car Dialogue System That Tracks The Driver's Attention. In: Proceedings of the 16th International Conference on Multimodal Interfaces. Istanbul, Turkey; 2014: 26-33.When a passenger speaks to a driver, he or she is co-located with the driver, is generally aware of the situation, and can stop speaking to allow the driver to focus on the driving task. In-car dialogue systems ignore these important aspects, making them more distracting than even cell-phone conversations. We developed and tested a ``situationally-aware'' dialogue system that can interrupt its speech when a situation which requires more attention from the driver is detected, and can resume when driving conditions return to normal. Furthermore, our system allows driver-controlled resumption of interrupted speech via verbal or visual cues (head nods). Over two experiments, we found that the situationally-aware spoken dialogue system improves driving performance and attention to the speech content, while driver-controlled speech resumption does not hinder performance in either of these two tasks

Plan-based integration of natural language and graphics generation

Multimodal interfaces combining natural language and graphics take advantage of both the individual strength of each communication mode and the fact that several modes can be employed in parallel. The central claim of this paper is that the generation of a multimodal presentation system WIP which allows the generation of alternate presentations of the same content taking into account various contextual factors. We discuss how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. We show that well-known concepts from the area of natural language processing like speech acts, anaphora, and rhetorical relations take on an extended meaning in the context of multimodal communication. Finally, we discuss two detailed examples illustrating and reinforcing our theoretical claims