On intelligible multimodal visual analysis

Analyzing data becomes an important skill in a more and more digital world. Yet, many users are facing knowledge barriers preventing them to independently conduct their data analysis. To tear down some of these barriers, multimodal interaction for visual analysis has been proposed. Multimodal interaction through speech and touch enables not only experts, but also novice users to effortlessly interact with such kind of technology. However, current approaches do not take the user differences into account. In fact, whether visual analysis is intelligible ultimately depends on the user. In order to close this research gap, this dissertation explores how multimodal visual analysis can be personalized. To do so, it takes a holistic view. First, an intelligible task space of visual analysis tasks is defined by considering personalization potentials. This task space provides an initial basis for understanding how effective personalization in visual analysis can be approached. Second, empirical analyses on speech commands in visual analysis as well as used visualizations from scientific publications further reveal patterns and structures. These behavior-indicated findings help to better understand expectations towards multimodal visual analysis. Third, a technical prototype is designed considering the previous findings. Enriching the visual analysis by a persistent dialogue and a transparency of the underlying computations, conducted user studies show not only advantages, but address the relevance of considering the user’s characteristics. Finally, both communications channels – visualizations and dialogue – are personalized. Leveraging linguistic theory and reinforcement learning, the results highlight a positive effect of adjusting to the user. Especially when the user’s knowledge is exceeded, personalizations helps to improve the user experience. Overall, this dissertations confirms not only the importance of considering the user’s characteristics in multimodal visual analysis, but also provides insights on how an intelligible analysis can be achieved. By understanding the use of input modalities, a system can focus only on the user’s needs. By understanding preferences on the output modalities, the system can better adapt to the user. Combining both directions imporves user experience and contributes towards an intelligible multimodal visual analysis

Enhancing Usability and Explainability of Data Systems

The recent growth of data science expanded its reach to an ever-growing user base of nonexperts, increasing the need for usability, understandability, and explainability in these systems. Enhancing usability makes data systems accessible to people with different skills and backgrounds alike, leading to democratization of data systems. Furthermore, proper understanding of data and data-driven systems is necessary for the users to trust the function of the systems that learn from data. Finally, data systems should be transparent: when a data system behaves unexpectedly or malfunctions, the users deserve proper explanation of what caused the observed incident. Unfortunately, most existing data systems offer limited usability and support for explanations: these systems are usable only by experts with sound technical skills, and even expert users are hindered by the lack of transparency into the systems\u27 inner workings and functions. The aim of my thesis is to bridge the usability gap between nonexpert users and complex data systems, aid all sort of users, including the expert ones, in data and system understanding, and provide explanations that help reason about unexpected outcomes involving data systems. Specifically, my thesis has the following three goals: (1) enhancing usability of data systems for nonexperts, (2) enable data understanding that can assist users in a variety of tasks such as achieving trust in data-driven machine learning, gaining data understanding, and data cleaning, and (3) explaining causes of unexpected outcomes involving data and data systems. For enhancing usability, we focus on example-driven user intent discovery. We develop systems based on example-driven interactions in two different settings: querying relational databases and personalized document summarization. Towards data understanding, we develop a new data-profiling primitive that can characterize tuples for which a machine-learned model is likely to produce untrustworthy predictions. We also develop an explanation framework to explain causes of such untrustworthy predictions. Additionally, this new data-profiling primitive enables interactive data cleaning. Finally, we develop two explanation frameworks, tailored to provide explanations in debugging data system components, including the data itself. The explanation frameworks focus on explaining the root cause of a concurrent application\u27s intermittent failure and exposing issues in the data that cause a data-driven system to malfunction

Visualising multiple overlapping classification hierarchies.

The revision or reorganisation of hierarchical data sets can result in many possible hierarchical classifications composed of the same or overlapping data sets existing in parallel with each other. These data sets are difficult for people to handle and conceptualise, as they tryto reconcile the different perspectives and structures that such data represents. One area where this situation occurs is the study of botanical taxonomy, essentially the classification and naming of plants. Revisions, new discoveries and new dimensions for classifying plants lead to a proliferation of classifications over the same set of plant data. Taxonomists would like a method of exploring these multiple overlapping hierarchies for interesting information, correlations, or anomalies. The application and extension of Information Visualisation (IV) techniques, the graphical display of abstract information, is put forward as a solution to this problem. Displaying the multiple classification hierarchies in a visually appealing manner along with powerful interaction mechanisms for examination and exploration of the data allows taxonomists to unearth previously hidden information. This visualisation gives detail that previous visualisations and statistical overviews cannot offer. This thesis work has extended previous IV work in several respects to achieve this goal. Compact, yet full and unambiguous, hierarchy visualisations have been developed. Linking and brushing techniques have been extended to work on a higher class of structure, namely overlapping trees and hierarchies. Focus and context techniques have been pushed to achieve new effects across the visually distinct representations of these multiple hierarchies. Other data types, such as multidimensional data and large cluster hierarchies have also been displayed using the final version of the visualisation

Nuevas tecnologías aplicadas a la ergonomía ocupacional. Empleo de sensores RGBD y EyeTracking en la mejora ergonómica de puestos de trabajo

Tesis por compendio[ES] A mediados del siglo XX inició en el continente europeo, especialmente en Francia y Bélgica, una nueva disciplina denominada Ergonomía Centrada en la Actividad. Esta disciplina está enfocada en el análisis del trabajo con el fin de optimizar las condiciones laborales. El propósito de la intervención ergonómica es mejorar componentes que interactúan en el sistema o en la actividad del trabajo (las personas, la organización, la tecnología y el ambiente), interrelacionando aspectos de salud, seguridad, productividad y calidad. En investigaciones anteriores a la presente el doctorando identificó la necesidad de mejora de las herramientas empleadas por los profesionales para la evaluación de factores de riesgo ergonómico, motivando así el desarrollo de la presente investigación. Inicialmente se analizaron métodos generales empleados por los diseñadores para establecer aquellos susceptibles de mejora con la introducción de nuevas tecnologías. En una segunda fase se identificaron y analizaron dispositivos tecnológicos orientados al rastreo de la actividad humana aplicables en el ámbito de las metodologías de la ergonomía ocupacional para los sectores productivos. Como resultado se concluyó que los sensores RGB-D y el Eye-Tracking (rastreo ocular) son dispositivos aplicables para ayudar a mejorar las condiciones en los puestos de trabajo, el primero para distribuir las áreas de trabajo y el segundo para la mejora de las interfaces del usuario. En esta TD se desarrollaron técnicas y métodos para el empleo de estos dispositivos logrando el diseño ergonómico de puestos de trabajo con aplicación práctica (artículos 2 y 3). Durante la realización de esta TD, bajo las directrices del Programa de Doctorado en Tecnologías para la Salud y el Bienestar de la Universidad Politécnica de Valencia, se publicaron tres artículos en revistas que durante el año de su publicación estuvieron indexadas en el primer cuartil de su categoría en el Journal Citation Report, las cuáles sustentan los resultados de la investigación. En ellos se evidenció cómo los avances tecnológicos implementados en Ergonomía producen cambios importantes en el diseño de los puestos de trabajo, y minimizan los tiempos y los movimientos que se requieren en las diferentes actividades laborales, garantizando así una ubicación óptima del recurso humano en los sistemas de producción y generando a su vez estrategias que disminuyen los Trastornos Músculo Esqueléticos (TMEs).[CA] A mitjan segle XX es va desenvolupar en el continent europeu, especialment a França i Bèlgica, una nova disciplina denominada Ergonomia Centrada en l'Activitat. Aquesta disciplina està enfocada en l'anàlisi del treball amb la finalitat d'optimitzar les condicions laborals. El propòsit de la intervenció ergonòmica és millorar els components que interactuen en el sistema o en l'activitat del treball (les persones, l'organització, la tecnologia i l'ambient), interrelacionant els aspectes de salut, seguretat, productivitat i qualitat. En investigacions anteriors a la present el doctorand va identificar la necessitat de millora de les eines emprades pels professionals per a l'avaluació de factors de risc ergonòmic, motivant així el desenvolupament de la present investigació. Inicialment es van analitzar els mètodes habituals empleats pels ergónoms per a establir aquells susceptibles de millora amb la introducció de noves tecnologies. En una segona fase es va identificar i analitzar els dispositius tecnològics orientats al monitoratge de l'activitat humana aplicables en l'àmbit de les metodologies de l'ergonomia ocupacional per als sectors productius. Com a resultat es va concloure que els sensors RGB-D i el Eye-Tracking són dispositius aplicables per a ajudar a millorar les condicions en els llocs de treball, el primer per a distribuir les àrees de treball i el segon per a la millora de les interfícies de l'usuari. En aquesta TD es van desenvolupar tècniques i procediments per a l'ús d'aquests dispositius aconseguint el disseny ergonòmic de llocs de treball amb aplicació pràctica (articles 2 i 3). Durant la realització d'aquesta TD, sota les directrius del Programa de Doctorat en Tecnologies per a la Salut i el Benestar de la Universitat Politècnica de València, es van publicar tres articles en revistes, que durant l'any de la seua publicació, van estar indexades en el primer quartil de la seua categoria en el Journal Citation Report, les quals sustenten els resultats de la investigació. En ells es va evidenciar com els avanços tecnològics implementats en Ergonomia produeixen canvis importants en el disseny dels llocs de treball, minimitzen els temps i els moviments que es requereixen en les diferents activitats laborals, garantint així una ubicació òptima del recurs humà en els sistemes de producció i generant al seu torn estratègies que disminueixen els Trastorns Músculesquelètics.[EN] Around the middle of XX century in Europe, especially in France and Belgium, a new discipline named Activity Focused Ergonomics was developed. This concept is based on the task analysis aiming to optimize working conditions. The purpose of the ergonomic intervention is to improve the components interacting within the system or the work activities (people, organization, technology and environment), correlating all health, safety, productivity and quality facts. In previous research, the ergomists identified the improvement need of those tools used by professionals to assess ergonomic risk factors, enhancing therefore the development of this present research. In the beginning, regular methods used by ergonomics specilists were analized in order to define those that could be improved with the introduction of new technologies. In a second phase, technological devices aimed to monitorize human activity, as well as those applicable for the occupational ergonomics methodologies in productive sectors were identified and analyzed. The conclusion that came as a result was that RGB-D sensors and Eye-Tracking are actual workstations conditions' improving devices, the first one is used to better organize working areas and the second one to optimize the user's interfaces. This Doctoral Thesis develops techniques and procedures to correctly use these devices obtaining workstations ergonomic designs with practical applications (articles 2 and 3). Along the development of this Doctoral Thesis, under the Polytechnic University of Valencia Doctoral School for Health and Wellbeing Technologies program directions, three articles were published in magazines which, throughout their publication year were indexed within its cathegory's first quartile at the Journal Citation Report, supporting the research results. The articles evidenced how the technological progress implemented in Ergonomics produce important changes in the workstations design and minimize times and movements required in different working activities, ensuring human resources' optimal location within the production systems and also developing Musculoskeletal Disorders reduction strategies.Garzón Leal, DC. (2020). Nuevas tecnologías aplicadas a la ergonomía ocupacional. Empleo de sensores RGBD y EyeTracking en la mejora ergonómica de puestos de trabajo [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/156052TESISCompendi

Modellierung und Generierung von Benutzeroberflächen für interaktive Softwaresysteme unter der Nutzung von Mustern

Die Dissertation umfasst den Entwurf einer Entwicklungsumgebung für Benutzeroberflächen, der modell- und musterbasierte UI-Entwicklung zu einem integrierten Entwicklungs- und Generierungsansatz kombiniert. Die Grundlagen der modell- und musterbasierten Entwicklung werden dargestellt und existierende Implementierungen analysiert. Die Ergebnisse der Untersuchungen fließen in den Entwurf der kombinierten Entwicklungsumgebung, der detailliert beschrieben wird, ein. Die Arbeit beinhaltet eine Fallstudie, die die praktische Bedeutung der theoretischen Ausführungen verdeutlicht

Perceptible affordances and feedforward for gestural interfaces: Assessing effectiveness of gesture acquisition with unfamiliar interactions

The move towards touch-based interfaces disrupts the established ways in which users manipulate and control graphical user interfaces. The predominant mode of interaction established by the desktop interface is to ‘double-click’ an icon in order to open an application, file or folder. Icons show users where to click and their shape, colour and graphic style suggests how they respond to user action. In sharp contrast, in a touch-based interface, an action may require a user to form a gesture with a certain number of fingers, a particular movement, and in a specific place. Often, none of this is suggested in the interface. This thesis adopts the approach of research through design to address the problem of how to inform the user about which gestures are available in a given touch-based interface, how to perform each gesture, and, finally, the effect of each gesture on the underlying system. Its hypothesis is that presenting automatic and animated visual prompts that depict touch and preview gesture execution will mitigate the problems users encounter when they execute commands within unfamiliar gestural interfaces. Moreover, the thesis claims the need for a new framework to assess the efficiency of gestural UI designs. A significant aspect of this new framework is a rating system that was used to assess distinct phases within the users’ evaluation and execution of a gesture. In order to support the thesis hypothesis, two empirical studies were conducted. The first introduces the visual prompts in support of training participants in unfamiliar gestures and gauges participants’ interpretation of their meaning. The second study consolidates the design features that yielded fewer error rates in the first study and assesses different interaction techniques, such as the moment to display the visual prompt. Both studies demonstrate the benefits in providing visual prompts to improve user awareness of available gestures. In addition, both studies confirm the efficiency of the rating system in identifying the most common problems users have with gestures and identifying possible design features to mitigate such problems. The thesis contributes: 1) a gesture-and-effect model and a corresponding rating system that can be used to assess gestural user interfaces, 2) the identification of common problems users have with unfamiliar gestural interfaces and design recommendations to mitigate these problems, and 3) a novel design technique that will improve user awareness of unfamiliar gestures within novel gestural interfaces

Interactive Fiction in Cinematic Virtual Reality: Epistemology, Creation and Evaluation

This dissertation presents the Interactive Fiction in Cinematic Virtual Reality (IFcVR), an interactive digital narrative (IDN) that brings together the cinematic virtual reality (cVR) and the creation of virtual environments through 360\ub0 video within an interactive fiction (IF) structure. This work is structured in three components: an epistemological approach to this kind of narrative and media hybrid; the creation process of IFcVR, from development to postproduction; and user evaluation of IFcVR. In order to set the foundations for the creation of interactive VR fiction films, I dissect the IFcVR by investigating the aesthetics, narratological and interactive notions that converge and diverge in it, proposing a medium-conscious narratology for this kind of artefact. This analysis led to the production of an IFcVR functional prototype: \u201cZENA\u201d, the first interactive VR film shot in Genoa. ZENA\u2019s creation process is reported proposing some guidelines for interactive and immersive film-makers. In order to evaluate the effectiveness of the IFcVR as an entertaining narrative form and a vehicle for diverse types of messages, this study also proposes a methodology to measure User Experience (UX) on IFcVR. The full evaluation protocol gathers both qualitative and quantitative data through ad hoc instruments. The proposed protocol is illustrated through its pilot application on ZENA. Findings show interactors' positive acceptance of IFcVR as an entertaining experience

Reconnaissance: A Widely Applicable Approach Encouraging Well-Informed Choices in Computer-Based Tasks

The aim of this thesis is to propose and demonstrate a novel approach to human-computer cooperation in exploratory tasks, that encourages the pursuit of thorough explorations and thus increases the likelihood of finding the best available results. The thesis identifies a class of computer-supported tasks - such as artifact design, decision analysis, data presentation, and many kinds of retrieval - that all involve an exploration among alternative result specifications to find one whose outcome best fits the user's current needs. Such explorations cannot be automated, but must be directed by the user. Since there is typically no way of knowing in advance what results are available, the user must continually trade off the effort required for further exploration against the unknown chance of discovering results preferable to those found so far. The more arduous the exploration is perceived to be, the greater the user's temptation to accept early results without due consideration of alternatives. By using a new illumination zone model to analyse existing systems, the root of the problem is characterised as a lack of support for visualising the trade-offs between results in terms of the criteria that the user feels are significant at the time. Reconnaissance - in the familiar sense of using scouts to examine unknown territory - provides a suitable metaphor for the task of instructing a computer to generate a range of results, to summarise them according to the current set of criteria, and to collate these summaries into a single visualisation. A challenge in supporting computer-based reconnaissance is the provision of a convenient interface that integrates the requesting, viewing and comparison of result summaries. An implemented example of reconnaissance support shows that the parallel coordinates plotting technique can be extended into an interactive form that meets this challenge. Tests carried out on this implementation, which provided reconnaissance facilities for an existing document formatting system, confirmed that more thorough explorations were indeed encouraged. The tests also brought to light additional, unforeseen advantages to the use of result ranges and summarisation