Cognitively Sensitive User Interface for Command and Control Applications

While there are broad guidelines for display or user interface design, creating effective human-computer interfaces for complex, dynamic systems control is challenging. Ad hoc approaches which consider the human as an afterthought are limiting. This research proposed a systematic approach to human / computer interface design that focuses on both the semantic and syntactic aspects of display design in the context of human-in-the-loop supervisory control of intelligent, autonomous multi-agent simulated unmanned aerial vehicles (UAVs). A systematic way to understand what needs to be displayed, how it should be displayed, and how the integrated system needs to be assessed is outlined through a combination of concepts from naturalistic decision making, semiotic analysis, and situational awareness literature. A new sprocket-based design was designed and evaluated in this research. For the practical designer, this research developed a systematic, iterative design process: design using cognitive sensitive principles, test the new interface in a laboratory situation; bring in subject matter experts to examine the interface in isolation; and finally, incorporate the resulting feedback into a full-size simulation. At each one of these steps, the operator, the engineer and the designer reexamined the results

A tangible programming environment model informed by principles of perception and meaning

It is a fundamental Human-Computer Interaction problem to design a tangible programming environment for use by multiple persons that can also be individualised. This problem has its origin in the phenomenon that the meaning an object holds can vary across individuals. The Semiotics Research Domain studies the meaning objects hold. This research investigated a solution based on the user designing aspects of the environment at a time after it has been made operational and when the development team is no longer available to implement the user’s design requirements. Also considered is how objects can be positioned so that the collection of objects is interpreted as a program. I therefore explored how some of the principles of relative positioning of objects, as researched in the domains of Psychology and Art, could be applied to tangible programming environments. This study applied the Gestalt principle of perceptual grouping by proximity to the design of tangible programming environments to determine if a tangible programming environment is possible in which the relative positions of personally meaningful objects define the program. I did this by applying the Design Science Research methodology with five iterations and evaluations involving children. The outcome is a model of a Tangible Programming Environment that includes Gestalt principles and Semiotic theory; Semiotic theory explains that the user can choose a physical representation of the program element that carries personal meaning whereas the Gestalt principle of grouping by proximity predicts that objects can be arranged to appear as if linked to each other.School of ComputingPh. D. (Computer Science

Design da interação na Web pragmática : reduzindo barreiras semióticas na colaboração mediada pela Web

Orientador: Maria Cecília Calani BaranauskasTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Web e suas tecnologias de base facilitam interações entre pessoas que alguns anos atrás não eram imagináveis. A colaboração é um tipo importante de interação que tem um propósito. Pessoas de diferentes contextos sociais e culturais, e com diferentes preferências e habilidades, podem colaborar mediadas pela Web. A colaboração muitas vezes acontece em contextos heterogêneos, que são definidos tanto pelas situações atuais dos parceiros na colaboração, quanto pelas experiências passadas, sejam elas individuais ou coletivas. A Web como um meio/uma mídia tem um impacto na colaboração e facilita certos aspectos da colaboração enquanto dificulta outros. Adotando uma perspectiva informada pela Web Pragmática, nesta tese investigamos questões da colaboração mediada pela Web, relacionadas com o Design da Interação. Nosso objetivo principal é entender barreiras semióticas da colaboração mediada pela Web e propor uma abordagem ao Design da Interação que reduza tais barreiras. Barreiras semióticas são barreiras relacionadas à comunicação, mediação e representação. Estas barreiras surgem na colaboração mediada pela Web pois muitos mecanismos da comunicação interpessoal face-a-face não estão disponíveis. Dependendo do contexto, barreiras semióticas frequentemente exercem um impacto negativo à colaboração; entretanto, em alguns casos o impacto pode ser positivo também. A abordagem ao Design da Interação aqui proposta tem suas bases na Web Pragmática e utiliza a Semiótica Organizacional e a Teoria da Atividade como referenciais teórico-metodológicos. As investigações teóricas contaram com uma contrapartida em termos de um embasamento em práticas reais através da participação em um projeto de pesquisa no domínio da educação inclusiva. Materializamos a abordagem proposta no design de um protótipo e na implementação de uma ferramenta correspondente ao protótipo, que apoia uma prática de profissionais no domínio da educação inclusiva. Além disso, propusemos e conduzimos um método de avaliação guiada pela pragmática dentro do contexto de um estudo de caso longitudinal. O design do protótipo, a implementação da ferramenta e a avaliação conduzida fornecem evidências de que a abordagem proposta ao Design da Interação guiada pela pragmática contribui para a redução de barreiras semióticas e para a promoção da colaboração mediada pela WebAbstract: The Web and its underlying technologies enable interactions among people that were unimaginable a few years ago. An important type of purposeful interaction is collaboration. Mediated by the Web, people from different social and cultural backgrounds, with different needs, preferences and capabilities can collaborate with each other. Collaboration often takes place in heterogeneous contexts that are not only defined by the actual situations of the collaboration partners, but also by individual and collective past experiences. The Web as a medium has an impact on collaboration and facilitates or enables certain aspects of collaboration while making others more difficult. In this PhD thesis we investigate Interaction Design related questions about web-mediated collaboration under a Pragmatic Web perspective. Our prime objective is to understand semiotic barriers to web-mediated collaboration and propose an approach to Interaction Design that reduces these barriers. Semiotic barriers are barriers related to communication, mediation and representation. These barriers emerge during web-based collaboration since many mechanisms of interpersonal face-to-face communication are not available. Depending on the context, semiotic barriers often have a negative impact on collaboration, but in some cases they might also have positive effects. The approach to Interaction Design proposed in this PhD thesis is rooted in the Pragmatic Web and uses Organizational Semiotics and Activity Theory as its theoretical and methodological frames of reference. The theoretic investigations were practically grounded in real world practices by participating in a research project in the domain of inclusive education. We materialized the proposed approach in the design of a prototype and the implementation of the corresponding tool that supports a practice of inclusive education professionals. Furthermore we proposed and applied a pragmatics-driven evaluation method in a longitudinal case study. Prototype design, tool implementation, and the conducted evaluation provided evidence that the proposed approach to pragmatics-driven Interaction Design can reduce semiotic barriers and thus promote web-mediated collaborationDoutoradoCiência da ComputaçãoDoutor em Ciência da Computaçã

A semiotic evaluation of musical meaning in the works of Igor Stravinsky : decoding syntax and markedness and prototypicality theory.

SIGLEAvailable from British Library Document Supply Centre-DSC:DXN026379 / BLDSC - British Library Document Supply CentreGBUnited Kingdo

The effect of Visual Design Quality on Player Experience Components in Tablet Games

Research in the field of Human Computer Interaction Design indicates that there is a need to develop further methods, tools, and frameworks for the design and evaluation of digital game interfaces. This thesis aims to design, develop, and evaluate two different types of tablet games with varying visual design quality interfaces to examine users’ perceptions of hedonic quality, visual design, emotions, and game enjoyment in different channels of experience. The design-oriented approach was adopted to combine both creative practice and scientific inquiry in the game design process and empirical evaluation. Hypotheses were formulated to explore the significance of visual design quality in relation to the components of player experience. The study entailed two phases. In the first phase, participatory design methods were employed to design and develop the tablet games encompassing mind-mapping techniques, focus groups, iterative prototyping with multiple cycles of usability testing of user interfaces. In the second phase, survey instruments were applied to collect and analyze data from 111 participants using tablet games as stimuli in a controlled experimental condition. The main contribution of this research is creation of a player experience model, validated in the domain of tablet gaming, to serve as a new theory. This research will allow for game researchers and practitioners to obtain a deeper understanding of the significance of the player experience framework components to create optimal player experience in tablet games. The finding shows that highly attractive game user interfaces were perceived to have higher utility and ease of use. Participants exhibited higher levels of arousal and valence in the high visual design quality interfaces mediated by hedonic quality. Participants who were highly sensitive to visual design did not necessarily derive the highest level of game enjoyment. Participants derived a heightened level of engagement in the arousal channel of experience and the highest level of enjoyment in the flow state. The use of 2.5D graphics and analogous color schemes created a spatial illusion that captivated users' attention. Practitioners are encouraged to design game artifacts with feature sets and mechanics capable of transporting players into the state of flow, as this is the stage where they experience game control, excitement and relaxation in addition to game immersion in the state of arousal

Paradoxes of interactivity: perspectives for media theory, human-computer interaction, and artistic investigations

Current findings from anthropology, genetics, prehistory, cognitive and neuroscience indicate that human nature is grounded in a co-evolution of tool use, symbolic communication, social interaction and cultural transmission. Digital information technology has recently entered as a new tool in this co-evolution, and will probably have the strongest impact on shaping the human mind in the near future. A common effort from the humanities, the sciences, art and technology is necessary to understand this ongoing co- evolutionary process. Interactivity is a key for understanding the new relationships formed by humans with social robots as well as interactive environments and wearables underlying this process. Of special importance for understanding interactivity are human-computer and human-robot interaction, as well as media theory and New Media Art. "Paradoxes of Interactivity" brings together reflections on "interactivity" from different theoretical perspectives, the interplay of science and art, and recent technological developments for artistic applications, especially in the realm of sound

Paradoxes of Interactivity

Current findings from anthropology, genetics, prehistory, cognitive and neuroscience indicate that human nature is grounded in a co-evolution of tool use, symbolic communication, social interaction and cultural transmission. Digital information technology has recently entered as a new tool in this co-evolution, and will probably have the strongest impact on shaping the human mind in the near future. A common effort from the humanities, the sciences, art and technology is necessary to understand this ongoing co- evolutionary process. Interactivity is a key for understanding the new relationships formed by humans with social robots as well as interactive environments and wearables underlying this process. Of special importance for understanding interactivity are human-computer and human-robot interaction, as well as media theory and New Media Art. »Paradoxes of Interactivity« brings together reflections on »interactivity« from different theoretical perspectives, the interplay of science and art, and recent technological developments for artistic applications, especially in the realm of sound

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

Design revolutions: IASDR 2019 Conference Proceedings. Volume 4: Learning, Technology, Thinking

In September 2019 Manchester School of Art at Manchester Metropolitan University was honoured to host the bi-annual conference of the International Association of Societies of Design Research (IASDR) under the unifying theme of DESIGN REVOLUTIONS. This was the first time the conference had been held in the UK. Through key research themes across nine conference tracks – Change, Learning, Living, Making, People, Technology, Thinking, Value and Voices – the conference opened up compelling, meaningful and radical dialogue of the role of design in addressing societal and organisational challenges. This Volume 4 includes papers from Learning, Technology and Thinking tracks of the conference

Multi-modal usability evaluation.

Research into the usability of multi-modal systems has tended to be device-led, with a resulting lack of theory about multi-modal interaction and how it might differ from more conventional interaction. This is compounded by a confusion over the precise definition of modality between the various disciplines within the HCI community, how modalities can be effectively classified, and their usability properties. There is a consequent lack of appropriate methodologies and notations to model such interactions and assess the usability implications of these interfaces. The role of expertise and craft skill in using HCI techniques is also poorly understood. This thesis proposes a new definition of modality, and goes on to identify issues of importance to multi-modal usability, culminating in the development of a new methodology to support the identification of such usability issues. It additionally explores the role of expertise and craft skill in using usability modelling techniques to assess usability issues. By analysing the problems inherent in current definitions and approaches, as well as issues relevant to cognitive science, a clear understanding of both the requirements for a suitable definition of modality and the salient usability issues are obtained. A novel definition of modality, based on the three elements of sense, information form and temporal nature is proposed. Further, an associated taxonomy is produced, which categorises modalities within the sensory dimension as visual, acoustic and haptic. This taxonomy classifies modalities within the information form dimension as lexical, symbolic or concrete, and classifies the temporal form dimension modalities as discrete, continuous, or dynamic. This results in a twenty-seven cell taxonomy, with each cell representing one taxon, indicating one particular type of modality. This is a faceted classification system, with the modality named after the intersection of the categories, building the category names into a compound modality name. The issues surrounding modality are examined and refined into the concepts of modality types, properties and clashes. Modalities are identified as belonging to either the system or the user, and being expressive or receptive in type. Various properties are described based on issues of granularity and redundancy. The five different types of clashes are described. Problems relating to the modelling of multi-modal interaction are examined by means of a motivating case study based on a portion of an interface for a robotic arm. The effectiveness of five modelling techniques, STN, CW, CPM-GOMS, PUM and Z, in representing multi-modal issues are assessed. From this, and using the collated definition, taxonomy and theory, a new methodology, Evaluating Multi-modal Usability (EMU), is developed. This is applied to a previous case study of the robotic arm to assess its application and coverage. Both the definition and EMU are used by students in a case study to test the definition and methodology's effectiveness, and to examine the leverage such an approach may give. The results shows that modalities can be successfully identified within an interactive context, and that usability issues can be described. Empirical video data of the robotic arm in use is used to confirm the issues identified by the previous analyses, and to identify new issues. A rational re-analysis of the six approaches (STN, CW, CPM-GOMS, PUM, Z and EMU) is conducted in order to distinguish between issues identified through craft skill, based on general HCI expertise and familiarity with the problem, and issues identified due to the core of the method for each approach. This is to gain a realistic understanding of the validity of claims made by each method, and to identify how else issues might be identified, and the consequent implications. Craft skill is found to have a wider role than anticipated, and the importance of expertise in using such approaches emphasised. From the case study and the re-analyses the implications for EMU are examined, and suggestions made for future refinement. The main contributions of this thesis are the new definition, taxonomy and theory, which significantly contribute to the theoretical understanding of multi-modal usability, helping to resolve existing confusion in this area. The new methodology, EMU, is a useful technique for examining interfaces for multi-modal usability issues, although some refinement is required. The importance of craft skill in the identification of usability issues has been explicitly explored, with implications for future work on usability modelling and the training of practitioners in such techniques