Evaluating Human-Language Model Interaction

Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.Comment: Authored by the Center for Research on Foundation Models (CRFM) at the Stanford Institute for Human-Centered Artificial Intelligence (HAI

Evaluating visitor experiences with interactive art

The Music Room is an interactive installation that allows visitor to compose classical music by moving throughout a space. The distance between them and their average speed maps the emotionality of music: in particular, distance influences the pleasantness of the music, while speed influences its intensity. This paper focuses on the evaluation of visitors' experience with The Music Room by examining log-data, video footages, interviews, and questionnaires, as collected in two public exhibitions of the installation. We examined this data to the identify the factors that fostered the engagement and to understand how players appropriated the original design idea. Reconsidering our design assumptions against behavioural data, we noticed a number of unexpected behaviours, which induced us to make some considerations on design and evaluation of interactive art

Visual melodies : design and evaluation of an interactive art installation for clinical environments

University of Technology, Sydney. Faculty of Arts and Social Sciences.Over the past decades there has been a growing recognition of the value of art activities, such as visual arts, music, drawing, dance, poetry and writing, as therapeutic approaches to enhance healthcare settings. Numerous studies have identified the value of art and design in decreasing depression and anxiety and creating a holistic healing environment for hospital visitors and patients, as well as offering a positive working environment for staff. Therefore, I propose that interactive multimedia art offers an important new therapeutic avenue as a service for engaging visitors, patients and staff in hospitals. Visual Melodies is an interactive art installation that engenders feelings of calm and relaxation in users. In this exegesis, I describe the theoretical background, development and evaluation of Visual Melodies. This creative trajectory draws on practice-based research, with the aims to create an interactive art installation, to evaluate its therapeutic potential, and to identify the semiotic dimensions of multimedia art that are most generally effective for producing therapeutic effects. Based on the literature of the different techniques used in art therapy, colour therapy and music therapy, I propose a bridge between these three therapies through a platform of an interactive multimedia installation – harnessing images, colours and sounds. Eight design principles that form the foundation of the practice were developed along with the discussion of the design elements that have been shown to be effective for enhancing relaxation. In line with the design principles, the design practice was then developed as a series of original landscape artworks and interactive animations accompanied by music specifically composed for the researcher. Audience feedback to the installation in a hospital waiting room was studied as a way of assessing its therapeutic potential. The evaluation feedback has been very positive and welcoming from visitors, patients and staff of all ages. Overall, Visual Melodies provides a relaxing and playful experience for the participants. The feelings most often reported were that of being relaxed, followed by calm, diverted, evoking memories and happy. This project demonstrates that it is beneficial to create a relaxing and supportive therapeutic interactive multimedia artwork for promoting holistic healing environments. The practice-based research and findings in this exegesis extend our understanding of how we can fuse artwork and technology, to transform our healthcare settings from sterile treatment spaces, into healing places where ‘care’ is built into the environment itself

A Guide to Evaluating the Experience of Media and Arts Technology

Evaluation is essential to understanding the value that digital creativity brings to people's experience, for example in terms of their enjoyment, creativity, and engagement. There is a substantial body of research on how to design and evaluate interactive arts and digital creativity applications. There is also extensive Human-Computer Interaction (HCI) literature on how to evaluate user interfaces and user experiences. However, it can be difficult for artists, practitioners, and researchers to navigate such a broad and disparate collection of materials when considering how to evaluate technology they create that is at the intersection of art and interaction. This chapter provides a guide to designing robust user studies of creative applications at the intersection of art, technology and interaction, which we refer to as Media and Arts Technology (MAT). We break MAT studies down into two main kinds: proof-of-concept and comparative studies. As MAT studies are exploratory in nature, their evaluation requires the collection and analysis of both qualitative data such as free text questionnaire responses, interviews, and observations, and also quantitative data such as questionnaires, number of interactions, and length of time spent interacting. This chapter draws on over 15 years of experience of designing and evaluating novel interactive systems to provide a concrete template on how to structure a study to evaluate MATs that is both rigorous and repeatable, and how to report study results that are publishable and accessible to a wide readership in art and science communities alike.Comment: Preprint. Chapter to appear in "Creating Digitally. Shifting Boundaries: Arts and Technologies - Contemporary Applications and Concepts", Anthony L. Brooks (Editor), Springer. https://link.springer.com/book/978303131359

H Space: Interactive Augmented Reality Art

open accessThis artwork exploits recent research into augmented reality systems, such as the HoloLens, for building creative interaction in augmented reality. The work is being conducted in the context of interactive art experiences. The first version of the audience experience of the artwork, “H Space”, was informally tested in the SIGGRAPH 2018 Art Gallery context. Experiences with a later, improved, version was evaluated at Tsinghua University. The latest distributed version will be shown in Sydney. The paper describes the concept, the background in both the art and the technological domain and points to some of the key computer human interaction art research issues that the work highlights

Cognitive Representation Learning of Self-Media Online Article Quality

The automatic quality assessment of self-media online articles is an urgent and new issue, which is of great value to the online recommendation and search. Different from traditional and well-formed articles, self-media online articles are mainly created by users, which have the appearance characteristics of different text levels and multi-modal hybrid editing, along with the potential characteristics of diverse content, different styles, large semantic spans and good interactive experience requirements. To solve these challenges, we establish a joint model CoQAN in combination with the layout organization, writing characteristics and text semantics, designing different representation learning subnetworks, especially for the feature learning process and interactive reading habits on mobile terminals. It is more consistent with the cognitive style of expressing an expert's evaluation of articles. We have also constructed a large scale real-world assessment dataset. Extensive experimental results show that the proposed framework significantly outperforms state-of-the-art methods, and effectively learns and integrates different factors of the online article quality assessment.Comment: Accepted at the Proceedings of the 28th ACM International Conference on Multimedi

Evaluation Strategy for the Re-Development of the Displays and Visitor Facilities at the Museum and Art Gallery, Kelvingrove

No abstract available

Open Challenges of Interactive Video Search and Evaluation

During the last 10 years of Video Browser Showdown (VBS), there were many different approaches tested for known-item search and ad-hoc search tasks. Undoubtedly, teams incorporating state-of-the-art models from the machine learning domain had an advantage over teams focusing just on interactive interfaces. On the other hand, VBS results indicate that effective means of interaction with a search system is still necessary to accomplish challenging search tasks. In this tutorial, we summarize successful deep models tested at the Video Browser Showdown as well as interfaces designed on top of corresponding distance/similarity spaces. Our broad experience with competition organization and evaluation will be presented as well, focusing on promising findings and also challenging problems from the most recent iterations of the Video Browser Showdown

Dance-making on the internet: can on-line choreographic projects foster creativity in the user-participant?

Interactive Internet artworks invite viewers to become involved as user-participants as the creative process unfolds. Through analysis of selected Internet projects, the authors discuss the potential for facilitating an interactive, creative experience for participants in the process of making dance. This study was carried out in 1998 and 1999, but the findings remain relevant, as there have been few subsequent developments in the field