Description languages for multimodal interaction: a set ofguidelines and its illustration with SMUIML

This article introduces the problem of modeling multimodal interaction, in the form of markup languages. After an analysis of the current state of the art in multimodal interaction description languages, nine guidelines for languages dedicated at multimodal interaction description are introduced, as well as four different roles that such language should target: communication, configuration, teaching and modeling. The article further presents the SMUIML language, our proposed solution to improve the time synchronicity aspect while still fulfilling other guidelines. SMUIML is finally mapped to these guidelines as a way to evaluate their spectrum and to sketch future work

Mobile Distributed User Interfaces

The success of a mobile application is due to the usability that the graphical user interface provides. A feature of mobile devices is the limited space for the interaction and the deployment of the graphical user interface. For this reason, user interfaces can have different interaction modalities. However, to work with information that can be complex to display, the use of modalities may not solve this problem. A possible alternative to provide more workspace to the users is through a distributed user interface (DUI). A mobile DUI allows the mobile applications to use two or more devices to execute the user interface. These devices can be Smart TVs or wearable such as smart watches. In this work the concepts of mobile DUI design are discussed, some use cases are presented and it is shown that its development in mobile devices is feasible

Bridging the Gap between a Behavioural Formal Description Technique and User Interface description language: Enhancing ICO with a Graphical User Interface markup language

International audienceIn the last years, User Interface Description Languages (UIDLs) appeared as a suitable solution for developing interactive systems. In order to implement reliable and efficient applications, we propose to employ a formal description technique called ICO (Interactive Cooperative Object) that has been developed to cope with complex behaviours of interactive systems including event-based and multimodal interactions. So far, ICO is able to describe most of the parts of an interactive system, from functional core concerns to fine grain interaction techniques, but, even if it addresses parts of the rendering, it still not has means to describe the effective rendering of such interactive system. This paper presents a solution to overcome this gap using markup languages. A first technique is based on the Java technology called JavaFX and a second technique is based on the emergent UsiXML language for describing user interface components for multi-target platforms. The proposed approach offers a bridge between markup language based descriptions of the user interface components and a robust technique for describing behaviour using ICO modelling. Furthermore, this paper highlights how it is possible to take advantage from both behavioural and markup language description techniques to propose a new model-based approach for prototyping interactive systems. The proposed approach is fully illustrated by a case study using an interactive application embedded into interactive aircraft cockpits

Formal Verification of Plastic User Interfaces Exploiting Domain Ontologies

This paper presents a formal model to check the interaction plasticity on a user interface (UI). An interaction is seen as an implementation (achievement) of a user task by means of interaction devices and modes of a given platform. The interaction plasticity is the ability of UI to support several interactions to perform the same task. In this work, two task models, containing different sets of interactions, are observed to check if they describe interactions that perform the same task. Each task model is represented by a labelled state-transitions system (lts). Due to the use of different interaction modes and devices, the obtained lts have different set of labels. Weak bi-simulation relationship is revisited to handle these transition systems by defining a relation on labels. This relation is borrowed from an ontology of interaction modes and devices. Model checking techniques are set up to automatically establish such a bi-simulation. A case study is used to illustrate how the approach works

Spoken Language Interaction with Robots: Recommendations for Future Research

With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with

Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey

The emergence of natural language processing has revolutionized the way users interact with tabular data, enabling a shift from traditional query languages and manual plotting to more intuitive, language-based interfaces. The rise of large language models (LLMs) such as ChatGPT and its successors has further advanced this field, opening new avenues for natural language processing techniques. This survey presents a comprehensive overview of natural language interfaces for tabular data querying and visualization, which allow users to interact with data using natural language queries. We introduce the fundamental concepts and techniques underlying these interfaces with a particular emphasis on semantic parsing, the key technology facilitating the translation from natural language to SQL queries or data visualization commands. We then delve into the recent advancements in Text-to-SQL and Text-to-Vis problems from the perspectives of datasets, methodologies, metrics, and system designs. This includes a deep dive into the influence of LLMs, highlighting their strengths, limitations, and potential for future improvements. Through this survey, we aim to provide a roadmap for researchers and practitioners interested in developing and applying natural language interfaces for data interaction in the era of large language models.Comment: 20 pages, 4 figures, 5 tables. Submitted to IEEE TKD

A Human Operator Model for Medical Device Interaction Using Behavior-Based Hybrid Automata

This paper describes the design and implementation of a control-theoretic model that can be used to model both the discrete and continuous behavior of a human operator. The human operator model can be used to compare different device user interfaces in terms of human performance. The implemented human operator model combines an ON–OFF control model and a behavior-based hybrid automaton with three controllers. The controllers, defined as continuous, discrete, and fine-tuning behavior, simulate the user’s conceptual model of the user interface. The device model used is that of a commercial syringe pump with chevron keys, described as a formal specification. Results of the human operator model simulation were generated for 20 different numbers obtained from syringe pump log files. The simulation results werecompared over 33 trials to a lab study employing a device based on the formal specification. The result of the simulation shows a significant similarity to the result of the lab study for all the numbers used