On lions, impala, and bigraphs: modelling interactions in physical/virtual spaces

While HCI has a long tradition of formally modelling task-based interactions with graphical user interfaces, there has been less progress in modelling emerging ubiquitous computing systems due in large part to their highly contextual nature and dependence on unreliable sensing systems. We present an exploration of modelling an example ubiquitous system, the Savannah game, using the mathematical formalism of bigraphs, which are based on a universal process algebra that encapsulates both dynamic and spatial behaviour of autonomous agents that interact and move among each other, or within each other. We establish a modelling approach based on four perspectives on ubiquitous systems—Computational, Physical, Human, and Technology—and explore how these interact with one another. We show how our model explains observed inconsistencies in user trials of Savannah, and then, how formal analysis reveals an incompleteness in design and guides extensions of the model and/or possible system re-design to resolve this

Semantic Interaction in Web-based Retrieval Systems : Adopting Semantic Web Technologies and Social Networking Paradigms for Interacting with Semi-structured Web Data

Existing web retrieval models for exploration and interaction with web data do not take into account semantic information, nor do they allow for new forms of interaction by employing meaningful interaction and navigation metaphors in 2D/3D. This thesis researches means for introducing a semantic dimension into the search and exploration process of web content to enable a significantly positive user experience. Therefore, an inherently dynamic view beyond single concepts and models from semantic information processing, information extraction and human-machine interaction is adopted. Essential tasks for semantic interaction such as semantic annotation, semantic mediation and semantic human-computer interaction were identified and elaborated for two general application scenarios in web retrieval: Web-based Question Answering in a knowledge-based dialogue system and semantic exploration of information spaces in 2D/3D

Towards Designing and Generating User Interfaces by Using Expert Knowledge

[ES] La investigación reportada en la presente tesis doctoral se lleva a cabo a través de la metodología de la ciencia del diseño que se centra en la creación y evaluación de artefactos. En esta tesis, el principal artefacto es el novedoso enfoque para diseñar y generar interfaces de usuario utilizando el conocimiento experto. Con el fin de permitir el uso del conocimiento experto, el enfoque propuesto se basa en la reutilización de patrones de diseño que incorporan el conocimiento experto del diseño de la interfaz y proporcionan soluciones reutilizables a diversos problemas de diseño. El objetivo principal de dicho enfoque es abordar el uso de patrones de diseño a fin de garantizar que los conocimientos especializados se integren en el diseño y la generación de interfaces de usuario para aplicaciones móviles y web. Las contribuciones específicas de esta tesis se resumen a continuación: Una primera contribución consiste en el marco AUIDP que se define para apoyar el diseño y la generación de interfaces adaptativas para aplicaciones web y móviles utilizando patrones de diseño HCI. El marco propuesto abarca tanto la etapa de diseño como la de ejecución de dichas interfaces. En el momento del diseño, los modelos de patrones de diseño junto con la interfaz de usuario y el perfil de usuario se definen siguiendo una metodología de desarrollo específica. En tiempo de ejecución, los modelos creados se utilizan para permitir la selección de patrones de diseño de HCI y para permitir la generación de interfaces de usuario a partir de las soluciones de diseño proporcionadas por los patrones de diseño relevantes. La segunda contribución es un método de especificación para establecer un modelo de ontología que convierte la representación tradicional basada en texto en la representación formal del patrón de diseño de HCI. Este método adopta la metodología Neon para lograr la transición de las representaciones informales a las formales. El modelo de ontología creado se llama MIDEP, que es una ontología modular que captura el conocimiento sobre los patrones de diseño, así como la interfaz de usuario y el perfil del usuario. La tercera contribución es el IDEPAR, que es el primer sistema dentro del marco global del AUIDP. Este sistema tiene como objetivo recomendar automáticamente los patrones de diseño más relevantes para un problema de diseño dado. Se basa en un enfoque híbrido que utiliza una combinación mixta de técnicas de recomendación basadas en texto y ontología para producir recomendaciones de patrones de diseño que proporcionan soluciones de diseño apropiadas. La cuarta contribución es un sistema generador de interfaz llamado ICGDEP, que se propone para generar automáticamente el código fuente de la interfaz de usuario para aplicaciones web y móviles. El ICGDEP es el segundo sistema dentro del marco global de AUIDP y se basa en el uso de patrones de diseño de HCI que son recomendados por el sistema IDEPAR. Su objetivo principal es generar automáticamente el código fuente de la interfaz de usuario a partir de las soluciones de diseño proporcionadas por los patrones de diseño. Para lograr esto, el sistema ICGDEP utiliza un método que permite la generación de código fuente de interfaz de usuario para la aplicación de destino. Las contribuciones aportadas en la presente tesis han sido validadas a través de diferentes perspectivas. En primer lugar, la evaluación de la ontología MIDEP desarrollada se realiza utilizando preguntas de competencia, enfoques de evaluación basados en la tecnología y basados en aplicaciones. En segundo lugar, la evaluación del sistema IDEPAR se establece mediante un patrón producido por expertos y un estudio de evaluación centrado en el usuario. Luego, el sistema ICGDEP es evaluado en términos de ser utilizado efectivamente por los desarrolladores, considerando el factor de productividad. Por último, la evaluación del marco mundial de AUIDP se lleva a cabo mediante estudios de casos y estudios de usabilidad.[CA] La investigació reportada en aquesta tesi doctoral es duu a terme a través de la metodologia de la ciència del disseny que se centra en la creació i avaluació d'artefactes. En aquesta tesi, el principal artefacte és el nou enfocament per dissenyar i generar interfícies d'usuari utilitzant el coneixement expert. Per tal de permetre l'ús del coneixement expert, l'enfocament proposat es basa en la reutilització de patrons de disseny que incorporen el coneixement expert del disseny de la interfície i proporcionen solucions reutilitzables a diversos problemes de disseny. L'objectiu principal d'aquest enfocament és abordar l'ús de patrons de disseny per tal de garantir que els coneixements especialitzats s'integrin en el disseny i la generació d'interfícies d'usuari per a aplicacions mòbils i web. Les contribucions específiques d'aquesta tesi es resumeixen a continuació: Una primera contribució consisteix en el marc AUIDP que es defineix per donar suport al disseny i generació d'interfícies adaptatives per a aplicacions web i mòbils utilitzant patrons de disseny HCI. El marc proposat inclou tant l'etapa de disseny com la d'execució de les interfícies esmentades. En el moment del disseny, els models de patrons de disseny juntament amb la interfície d'usuari i el perfil d'usuari es defineixen seguint una metodologia de desenvolupament específica. En temps d'execució, els models creats s'utilitzen per permetre la selecció de patrons de disseny de HCI i per permetre la generació de interfícies d'usuari a partir de les solucions de disseny proporcionades pels patrons de disseny rellevants. La segona contribució és un mètode d'especificació per establir un model d'ontologia que converteix la representació tradicional basada en text en la representació formal del patró de disseny de HCI. Aquest mètode adopta la metodologia Neon per aconseguir la transició de les representacions informals a les formals. El model d'ontologia creat s'anomena MIDEP, una ontologia modular que captura el coneixement sobre els patrons de disseny, així com la interfície d'usuari i el perfil de l'usuari. La tercera contribució és l'IDEPAR, que és el primer sistema dins del marc global de l'AUIDP. Aquest sistema té com a objectiu recomanar automàticament els patrons de disseny més rellevants per a un problema de disseny donat. Es basa en un enfocament híbrid que utilitza una combinació mixta de tècniques de recomanació basades en text i ontologia per produir recomanacions de patrons de disseny que proporcionen solucions de disseny apropiades. La quarta contribució és un sistema generador d'interfície anomenat ICGDEP, que es proposa per generar automàticament el codi font de la interfície d'usuari per a aplicacions web i mòbils. L'ICGDEP és el segon sistema dins del marc global d'AUIDP i es basa en l'ús de patrons de disseny de HCI que són recomanats pel sistema IDEPAR. El seu objectiu principal és generar automàticament el codi font de la interfície d'usuari a partir de les solucions de disseny proporcionades pels patrons de disseny. Per aconseguir-ho, el sistema ICGDEP utilitza un mètode que permet generar codi font d'interfície d'usuari per a l'aplicació de destinació. Les contribucions aportades a la present tesi han estat validades a través de diferents perspectives. En primer lloc, l'avaluació de l'ontologia MIDEP desenvolupada es fa utilitzant preguntes de competència, enfocaments d'avaluació basats en la tecnologia i basats en aplicacions. En segon lloc, l'avaluació del sistema IDEPAR s'estableix mitjançant un patró produït per experts i un estudi d'avaluació centrat en l'usuari. Després, el sistema ICGDEP és avaluat en termes de ser utilitzat efectivament pels desenvolupadors, considerant el factor de productivitat. Finalment, l'avaluació del marc mundial d'AUIDP es fa mitjançant estudis de casos i estudis d'usabilitat.[EN] The research reported in the present PhD dissertation is conducted through the design science methodology that focuses on creating and evaluating artifacts. In the current thesis, the main artifact is the novel approach to design and generate user interfaces using expert knowledge. In order to enable the use of expert knowledge, the present approach is devoted to reuse design patterns that incorporate expert knowledge of interface design and provide reusable solutions to various design problems. The main goal of the proposed approach is to address the use of design patterns in order to ensure that expert knowledge is integrated into the design and generation of user interfaces for mobile and Web applications. The specific contributions of this thesis are summarized below: This first contribution is the AUIDP framework that is defined to support the design and generation of adaptive interfaces for Web and mobile applications using HCI design patterns. The proposed framework spans over design-time and run-time. At design-time, models of design patterns along with user interface and user profile are defined following a specific development methodology. At run-time, the created models are used to allow the selection of HCI design patterns and to enable the generation of user interfaces from the design solutions provided by the relevant design patterns. The second contribution is a specification method to establish an ontology model that turns traditional text-based representation into formal HCI design pattern representation. This method adopts the Neon methodology to achieve the transition from informal to formal representations. The created ontology model is named MIDEP, which is a modular ontology that captures knowledge about design patterns as well as the user interface and user's profile. The third contribution is the IDEPAR, which is the first system within the global AUIDP framework. This system aims to automatically recommend the most relevant design patterns for a given design problem. It is based on a hybrid approach that relies on a mixed combination of text-based and ontology-based recommendation techniques to produce design pattern recommendations that provide appropriate design solutions. The fourth contribution is an interface generator system called ICGDEP, which is proposed to automatically generate the user interface source code for Web and mobile applications. The proposed ICGDEP is the second system within the global AUIDP framework and relies on the use of HCI design patterns that are recommended by the IDEPAR system. It mainly aims at automatically generating the user interface source code from the design solutions provided by design patterns. To achieve this, the ICGDEP system is based on a generation method that allows the generation of user interface source code for the target application. The contributions provided in the present thesis have been validated through different perspectives. First, the evaluation of the developed MIDEP ontology is performed using competency questions, technology-based, and application-based evaluation approaches. Second, the evaluation of the IDEPAR system is established through an expert-based gold standard and a user-centric evaluation study. Then, the ICGDEP system is evaluated in terms of being effectively used by developers, considering the productivity factor. Finally, the evaluation of the global AUIDP framework is conducted through case studies and usability studies.Braham, A. (2022). Towards Designing and Generating User Interfaces by Using Expert Knowledge [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/19092

Understanding Collaborative Sensemaking for System Design — An Investigation of Musicians\u27 Practice

There is surprisingly little written in information science and technology literature about the design of tools used to support the collaboration of creators. Understanding collaborative sensemaking through the use of language has been traditionally applied to non-work domains, but this method is also well-suited for informing hypotheses about the design collaborative systems. The presence of ubiquitous, mobile technology, and development of multi-user virtual spaces invites investigation of design which is based on naturalistic, real world, creative group behaviors, including the collaborative work of musicians. This thesis is considering the co-construction of new (musical) knowledge by small groups. Co-construction of new knowledge is critical to the definition of an information system because it emphasizes coordination and resource sharing among group members (versus individual members independently doing their own tasks and only coming together to collate their contributions as a final product). This work situates the locus of creativity on the process itself, rather than on the output (the musical result) or the individuals (members of the band). This thesis describes a way to apply quantitative observations to inform qualitative assessment of the characteristics of collaborative sensemaking in groups. Conversational data were obtained from nine face-to-face collaborative composing sessions, involving three separate bands producing 18 hours of recorded interactions. Topical characteristics of the discussion, namely objects, plans, properties and performance; as well as emergent patterns of generative, evaluative, revision, and management conversational acts within the group were seen as indicative of knowledge construction. The findings report the use of collaborative pathways: iterative cycles of generation, evaluation and revision of temporary solutions used to move the collaboration forward. In addition, bracketing of temporary solutions served to help collaborators reuse content and offload attentional resources. Ambiguity in language, evaluation criteria, goal formation, and group awareness meant that existing knowledge representations were insufficient in making sense of incoming data and necessitated reformulating those representations. Further, strategic use of affective language was found to be instrumental in bridging knowledge gaps. Based on these findings, features of a collaborative system are proposed to help in facilitating sensemaking routines at various stages of a creative task. This research contributes to the theoretical understanding of collaborative sensemaking during non-work, creative activities in order to inform the design of systems for supporting these activities. By studying an environment which forms a potential microcosm of virtual interaction between groups, it provides a framework for understanding and automating collaborative discussion content in terms of the features of dialogue

Designing Service-Oriented Chatbot Systems Using a Construction Grammar-Driven Natural Language Generation System

Service oriented chatbot systems are used to inform users in a conversational manner about a particular service or product on a website. Our research shows that current systems are time consuming to build and not very accurate or satisfying to users. We find that natural language understanding and natural language generation methods are central to creating an e�fficient and useful system. In this thesis we investigate current and past methods in this research area and place particular emphasis on Construction Grammar and its computational implementation. Our research shows that users have strong emotive reactions to how these systems behave, so we also investigate the human computer interaction component. We present three systems (KIA, John and KIA2), and carry out extensive user tests on all of them, as well as comparative tests. KIA is built using existing methods, John is built with the user in mind and KIA2 is built using the construction grammar method. We found that the construction grammar approach performs well in service oriented chatbots systems, and that users preferred it over other systems

Semiotic machines : software in discourse

Includes abstract.Includes bibliographical references (p. 245-259).This study develops new theoretical and methodological approaches to the study of software as a medium of communication. This study analyses voting software, educational software, search engines, and combat and narrative in digital games. In each case it investigates how proprietary software affords discourse, and suggests a way of characterising users’ experience of this discourse. These affordances constitute the rules of communication, or ‘rules of speaking’, ‘rules of seeing’, and ‘writing-rights’ which proprietary software makes available to users, situating them within specific power-relations in the process

Web Search, Web Tutorials & Software Applications: Characterizing and Supporting the Coordinated Use of Online Resources for Performing Work in Feature-Rich Software

Web search and other online resources serve an integral role in how people learn and use feature-rich software (e.g., Adobe Photoshop) on a daily basis. Users depend on web resources both as a first line of technical support, and as a means for coping with system complexity. For example, people rely on web resources to learn new tasks, to troubleshoot problems, or to remind themselves of key task details. When users rely on web resources to support their work, their interactions are distributed over three user environments: (1) the search engine, (2) retrieved documents, and (3) the application's user interface. As users interact with these environments, their actions generate a rich set of signals that characterize how the population thinks about and uses software systems "in the wild," on a day-to-day basis. This dissertation presents three works that successively connect and associate signals and artifacts across these environments, thereby generating novel insights about users and their tasks, and enabling powerful new end-user tools and services. These three projects are as follows: Characterizing usability through search (CUTS): The CUTS system demonstrates that aggregate logs of web search queries can be leveraged to identify common tasks and potential usability problems faced by the users of any publicly available interactive system. For example, in 2011 I examined query data for the Firefox web browser. Automated analysis uncovered approximately 150 variations of the query "Firefox how to get the menu bar back", with queries issued once every 32 minutes on average. Notably, this analysis did not depend on direct access to query logs. Instead, query suggestions services and online advertising valuations were leveraged to approximate aggregate query data. Nevertheless, these data proved to be timely, to have a high degree of ecological validity, and to be arguably less prone to self-selection bias than data gathered via traditional usability methods. Query-feature graphs (QF-Graphs): Query-feature graphs are structures that map high-level descriptions of a user's goals to the specific features and commands relevant to achieving those goals in software. QF-graphs address an important instance of the more general vocabulary mismatch problem. For example, users of the GIMP photo manipulation software often want to "make a picture black and white", and fail to recognize the relevance of the applicable commands, which include: "desaturate", and "channel mixer". The key insights for building QF-graphs are that: (1) queries concisely express the user's goal in the user's own words, and (2) retrieved tutorials likely include both query terms, as well as terminology from the application's interface (e.g., the names of commands). QF-graphs are generated by mining these co-occurrences across thousands of query-tutorial pairings. InterTwine: InterTwine explores interaction possibilities that arise when software applications, web search, and online support materials are directly integrated into a single productivity system. With InterTwine, actions in the web browser directly impact how information is presented in a software application, and vice versa. For example, when a user opens a web tutorial in their browser, the application's menus and tooltips are updated to highlight the commands mentioned therein. These embellishments are designed to help users orient themselves after switching between the web browser and the application. InterTwine also augments web search results to include details of past application use. Search snippets gain before and after pictures and other metadata detailing how the user's personal work document evolved the last time they visited the page. This feature was motivated by the observation that existing mechanisms (e.g., highlighting visited links) are often insufficient for recalling which resources were previously helpful vs. unhelpful for accomplishing a task. Finally, the dissertation concludes with a discussion of the advantages, limitations and challenges of this research, and presents an outline for future work