A Framework for Computational Design and Adaptation of Extended Reality User Interfaces

To facilitate high quality interaction during the regular use of computing systems, it is essential that the user interface (UI) deliver content and components in an appropriate manner. Although extended reality (XR) is emerging as a new computing platform, we still have a limited understanding of how best to design and present interactive content to users in such immersive environments. Adaptive UIs offer a promising approach for optimal presentation in XR as the user's environment, tasks, capabilities, and preferences vary under changing context. In this position paper, we present a design framework for adapting various characteristics of content presented in XR. We frame these as five considerations that need to be taken into account for adaptive XR UIs: What?, How Much?, Where?, How?, and When?. With this framework, we review literature on UI design and adaptation to reflect on approaches that have been adopted or developed in the past towards identifying current gaps and challenges, and opportunities for applying such approaches in XR. Using our framework, future work could identify and develop novel computational approaches for achieving successful adaptive user interfaces in such immersive environments.Comment: 5 pages, CHI 2023 Workshop on The Future of Computational Approaches for Understanding and Adapting User Interface

PopNet: a Pop Culture Knowledge Association Network for Supporting Creative Connections

Pop culture is a pervasive and important aspect of communication and self-expression. When people wish to communicate using pop culture references, they need to find connections between their message and the things, people, location and actions of a movie, tv series, or other pop culture domain. However, finding an appropriate match from memory is challenging and search engines are not specific enough to the task. Often domain-specific knowledge graphs provide the structure, specificity and search capabilities that people need. We introduce PopNet - a Pop Culture Knowledge Association Network automatically created from plain text using state-of-the art NLP methods to extract entities and actions from text summaries of movies and tv shows. The interface allows people to browse and search the entries to find connections. We conduct a study showing that this system is accurate and helpful for finding multiple connections between a message and a pop culture domain

Reverseorc:Reverse engineering of resizable user interface layouts with or-constraints

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.Comment: CHI2021 Full Pape

User Interface Plasticity: Model Driven Engineering to the Limit!

Keynote paper.International audienceTen years ago, I introduced the notion of user interface plasticity to denote the capacity of user interfaces to adapt, or to be adapted, to the context of use while preserving usability. The Model Driven Engineering (MDE) approach, which was used for user interface generation since the early eighties in HCI, has recently been revived to address this complex problem. Although MDE has resulted in interesting and convincing results for conventional WIMP user interfaces, it has not fully demonstrated its theoretical promises yet. In this paper, we discuss how to push MDE to the limit in order to reconcile high-level modeling techniques with low-level programming in order to go beyond WIMP user interfaces

SketchWizard: Wizard of Oz Prototyping of Pen-based User Interfaces

SketchWizard allows designers to create Wizard of Oz prototypes of pen-based user interfaces in the early stages of design. In the past, designers have been inhibited from participating in the design of pen-based interfaces because of the inadequacy of paper prototypes and the difficulty of developing functional prototypes. In SketchWizard, designers and end users share a drawing canvas between two computers, allowing the designer to simulate the behavior of recognition or other technologies. Special editing features are provided to help designers respond quickly to end-user input. This paper describes the SketchWizard system and presents two evaluations of our approach. The first is an early feasibility study in which Wizard of Oz was used to prototype a pen-based user interface. The second is a laboratory study in which designers used SketchWizard to simulate existing pen-based interfaces. Both showed that end users gave valuable feedback in spite of delays between end-user actions and wizard updates

Older Generation: Self-Powered IoTs, Home-Life and “Ageing Well”

Internet of Things (IoT) technology is found in many homes. These systems enable tasks to be done more effectively or efficiently – e.g., securing property, monitoring and adjusting resources, trackingbehaviours for well-being, and so on. The system presented here was designed with older adults; the vast majority of home IoT systems marketed to this age group are not growth-oriented but rather decline-focused, monitoring and signalling well-being issues. In contrast to both “mainstream” and “older adult” IoT frameworks, then, we present a toolkit designed only to platform reflections,conversations and insights by occupants and visitors in regards to diverse user-defined meaningful home activities: hobbies, socialisation, fun, relaxation, and so on. Furthermore, mindful of the climatecrisis and the battery recharge or replacement requirements in conventional IoT systems, the toolkit is predominantly self-powered. We detail the design process and home deployments, highlighting the value of alternative data presentations from the simplest to LLM-enabled

Bridging the Gap between a Behavioural Formal Description Technique and User Interface description language: Enhancing ICO with a Graphical User Interface markup language

International audienceIn the last years, User Interface Description Languages (UIDLs) appeared as a suitable solution for developing interactive systems. In order to implement reliable and efficient applications, we propose to employ a formal description technique called ICO (Interactive Cooperative Object) that has been developed to cope with complex behaviours of interactive systems including event-based and multimodal interactions. So far, ICO is able to describe most of the parts of an interactive system, from functional core concerns to fine grain interaction techniques, but, even if it addresses parts of the rendering, it still not has means to describe the effective rendering of such interactive system. This paper presents a solution to overcome this gap using markup languages. A first technique is based on the Java technology called JavaFX and a second technique is based on the emergent UsiXML language for describing user interface components for multi-target platforms. The proposed approach offers a bridge between markup language based descriptions of the user interface components and a robust technique for describing behaviour using ICO modelling. Furthermore, this paper highlights how it is possible to take advantage from both behavioural and markup language description techniques to propose a new model-based approach for prototyping interactive systems. The proposed approach is fully illustrated by a case study using an interactive application embedded into interactive aircraft cockpits