On intelligible multimodal visual analysis

Analyzing data becomes an important skill in a more and more digital world. Yet, many users are facing knowledge barriers preventing them to independently conduct their data analysis. To tear down some of these barriers, multimodal interaction for visual analysis has been proposed. Multimodal interaction through speech and touch enables not only experts, but also novice users to effortlessly interact with such kind of technology. However, current approaches do not take the user differences into account. In fact, whether visual analysis is intelligible ultimately depends on the user. In order to close this research gap, this dissertation explores how multimodal visual analysis can be personalized. To do so, it takes a holistic view. First, an intelligible task space of visual analysis tasks is defined by considering personalization potentials. This task space provides an initial basis for understanding how effective personalization in visual analysis can be approached. Second, empirical analyses on speech commands in visual analysis as well as used visualizations from scientific publications further reveal patterns and structures. These behavior-indicated findings help to better understand expectations towards multimodal visual analysis. Third, a technical prototype is designed considering the previous findings. Enriching the visual analysis by a persistent dialogue and a transparency of the underlying computations, conducted user studies show not only advantages, but address the relevance of considering the user’s characteristics. Finally, both communications channels – visualizations and dialogue – are personalized. Leveraging linguistic theory and reinforcement learning, the results highlight a positive effect of adjusting to the user. Especially when the user’s knowledge is exceeded, personalizations helps to improve the user experience. Overall, this dissertations confirms not only the importance of considering the user’s characteristics in multimodal visual analysis, but also provides insights on how an intelligible analysis can be achieved. By understanding the use of input modalities, a system can focus only on the user’s needs. By understanding preferences on the output modalities, the system can better adapt to the user. Combining both directions imporves user experience and contributes towards an intelligible multimodal visual analysis

A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization

Inspired by the great success of machine learning (ML), researchers have applied ML techniques to visualizations to achieve a better design, development, and evaluation of visualizations. This branch of studies, known as ML4VIS, is gaining increasing research attention in recent years. To successfully adapt ML techniques for visualizations, a structured understanding of the integration of ML4VISis needed. In this paper, we systematically survey 88 ML4VIS studies, aiming to answer two motivating questions: "what visualization processes can be assisted by ML?" and "how ML techniques can be used to solve visualization problems?" This survey reveals seven main processes where the employment of ML techniques can benefit visualizations:Data Processing4VIS, Data-VIS Mapping, InsightCommunication, Style Imitation, VIS Interaction, VIS Reading, and User Profiling. The seven processes are related to existing visualization theoretical models in an ML4VIS pipeline, aiming to illuminate the role of ML-assisted visualization in general visualizations.Meanwhile, the seven processes are mapped into main learning tasks in ML to align the capabilities of ML with the needs in visualization. Current practices and future opportunities of ML4VIS are discussed in the context of the ML4VIS pipeline and the ML-VIS mapping. While more studies are still needed in the area of ML4VIS, we hope this paper can provide a stepping-stone for future exploration. A web-based interactive browser of this survey is available at https://ml4vis.github.ioComment: 19 pages, 12 figures, 4 table

Sistema de Sugestões Sensível ao Contexto

Over the last few years, pervasive systems have experienced some interesting development. Nevertheless, human-human interaction can also take advantage of those systems by using their ability to perceive the surrounding environment. In this dissertation, we have developed a pervasive system - named ConversationaL Aware Suggestion SYstem (CLASSY) - which is aware of the conversational context and suggests the users potentially useful documents or that, somehow, save time executing a specific task. We have also proposed two different approaches - the Neighborhood one, that uses semantic similarity, based on proximity data in order to classify the relationship between tokens; and the Reinforcement Learning one, that uses implicit feedback associated with each suggestion as a source of knowledge that can be used to improve the system's performance over time. The conducted tests showed that these two approaches not only enhanced the pervasive behavior of the system, but also increased its global performance. A case study regarding the importance of feedback on context-limited environments was also carried out, whose results showed that it is still a useful source of knowledge regardless the conversational environment's characteristics.Ao longo dos últimos anos, os sistemas pervasivos têm sido fonte de um grande desenvolvimento. Contudo, as interações humano-humano também podem tirar vantagem deste tipo de sistemas recorrendo à sua capacidade para entender o ambiente que o rodeia. Nesta dissertação, foi desenvolvido um sistema pervasivo - chamado Sistema de Sugestões Sensível ao Contexto (CLASSY) - que está consciente dos vários contextos conversacionais e que sugere documentos considerados potencialmente úteis para os utilizadores ou que, de alguma forma, poupam tempo na execução de uma tarefa específica. Foram também propostas duas aproximações diferentes - a de vizinhança, que usa similaridade semântica, baseando-se em proximidades de forma a classificar relações entre palavras; e a de Aprendizagem por Reforço, que usa feedback implícito dos utilizadores associado a cada sugestão, como fonte de conhecimento que pode ser utilizado para melhorar a performance do sistema ao longo do tempo. Os testes realizados mostraram que as aproximações acima referidas melhoraram não só o comportamento pervasivo do sistema, mas também a sua performance global. Foi, ainda, analisado um caso de estudo referente à importância de feedback em ambientes com contexto limitado, onde os resultados mostraram que o mesmo continua a ser uma importante fonte de conhecimento, independentemente das características do ambiente conversacional.Mestrado em Engenharia de Computadores e Telemátic

Online Personalization in Exploratory Search

Modern society produces vast amounts of digital data related to multiple domains of our lives. We produce data in our free time when browsing the net or taking photos with various personal devices, such as phones or ipads. Businesses and governments also gather a lot of information related to our interests, habits or otherwise personal information (legal status, health data, etc.). The amount of data produced is growning too large for us to be handled manually, and so to assist the user, specialized information retrieval systems have been developed to allow efficient perusal of different types of data. Unfortunately, as using such systems often requires expert understanding of the domain in question, many users get lost in their attempt to navigate the search space. This problem will only be exacerbated in the future, as the amount of data keeps growing, giving us less time to learn about the domains involved. Exploratory search is a field of research that studies user behaviour in situations, where users have little familiarity with the search domain, or have not yet decided exactly what their search goal is. Situations such as these arise when the user wishes to explore what is available, or is otherwise synthesizing or investigating the data. To assist the user in exploratory search and in finding relevant information, various methodologies may be employed, such as user modeling techniques or novel interfaces and data visualization techniques. This thesis presents exploratory search techniques for online personalization and feature representations that allow efficient perusal of unknown datasets. These methods are showcased in two different search environments. First, we present a search engine for scientific document retrieval, which takes the user's knowledge level into account in order to provide the user with more or less diverse search results. The second search environment aims at supporting the user when browsing through a dataset of unannotated images. Overall, the research presented here describes a number of techniques based on reinforcement learning and neural networks that, compared to traditional search engines, can provide better support for users who are unsure of the final goal of their search or who cannot easily formulate their search needs.Moderni yhteiskunta tuottaa suuria määriä digitaalista dataa liittyen elämämme eri osa-alueisiin. Tuotamme tätä dataa vapaa-ajallamme kun käymme internetissä, tai tuotamme moninaista multimediaa eri älylaitteillamme. Tämä näkyy vahvasti myös yritysten liiketoiminnassa ja valtioiden hallinnassa, missä erilaisten toimintojen seurantaa kirjataan digitaalisesti, kuten laillisen informaation, terveysdatan ja henkilökohtaisen datan muodossa. Tuotetun datan määrä on kasvamassa liian suureksi yksittäisten ihmisten käsiteltäväksi, ja täten järjestelmiä mitkä automatisoivat hakuprosessia tuotetaan enenevässä määrin. Valitettavasti, useat näistä järjestelmistä vaativat asiantuntemusta annetulta alalta, minkä takia käyttäjät eksyvät helposti hakuavaruuteen. Exploratiivinen haku on tieteenala joka tutkii käyttäjän käyttäytyymistä tilanteissa, missä heillä on vähän tuntemusta alalta, tai he eivät ole vielä päättäneet mikä on heidän hakunsa päämäärä. Tällaiset tilanteet syntyvät kun käyttäjä haluaa kartoittaa saatavilla olevaa aineistoa, tai muuten syntetisoida tai tutkia kyseistä dataa. Oleellisen tiedon löytämiseksi exploratiivinen haku hyödyntää erilaisia menetelmiä, kuten käyttäjän mallintamista, erikoisia käyttöliittymiä tai datan visualisointimenetelmiä. Tämä kirja esittelee exploratiivisen haun menetelmiä ajantasaiseen personalisointiin ja piirrerepresentaatioihin, mitkä mahdollistavat käyttäjälle entuudestaan tuntemattomien tietokantojen tehokkaan käsittelyn. Nämä menetelmät esitellään kahden hakujärjestelmän yhteydessä. Ensiksi, esittelemme hakukoneen tieteellisille artikkeleille, mikä ottaa käyttäjän tietotason huomioon kun hakutuloksia esitellään. Jälkimmäinen hakukone mahdollistaa annotoimattomien kuvien tehokkaan selailun. Kokonaisuudessaan, tämä kirja kertoo useasta tutkimus- ja järjestelmämenetelmästä, mitkä, verrattuna perinteisiin hakukoneisiin, tukevat epävarman käyttäjän hakuprosessia tuntemattomassa ympäristössä paremmin

Revealing Perceptual Proxies in Comparative Data Visualization

Data Visualization has long been shaped by empirical evidence of the efficacies of different encodings, such as length, position, or area, in conveying quantities. Less is known, however, about what may affect comparison of multiple data series, which generally involves extraction of higher-order values, such as means, ranges, and correlations. In this work, we investigate such factors and the underlying visual processes that may account for them. We begin with a case study motivating the research, in which we modify Krona, a Bioinformatics visualization system, to support several types of comparison. Next, we empirically examine the influence of “arrangement”—that is, whether charts are shown side-by-side, stacked vertically, overlaid, etc.—on comparative tasks, in a series of psychophysical experiments. The results suggest a complex interaction of factors, with different comparative arrangements providing benefits for different combinations of tasks and encodings. For example, overlaid charts make detecting differences easier but comparing means or ranges more difficult. While these results offer some guidance to designers, the number of interactions makes it infeasible to provide broad rankings of arrangements, as has been done previously for encodings. Our subsequent efforts thus work toward understanding the visual processes that underlie the extraction of statistical summaries needed for comparison. It has recently been proposed that simpler shortcuts, called Perceptual Proxies, are used by the visual system to estimate these values. We investigate proxies for bar charts in experiments using an “adversarial” framework, in which the ranking of two charts along a task metric (e.g. mean) is opposite their ranking along a proxy metric (e.g. convex hull area). The strongest evidence we find is for use of a “centroid” proxy to estimate means in bar charts. Finally, we attempt to use using human-guided optimization to construct charts de novo, without assuming specific proxies. This work contributes both to perceptual psychology, by offering evidence for underlying visual processes that may be involved in the interpretation of comparative visualizations, and to data visualization, by providing new research methods and straightforward design guidance on how best to lay out charts to support certain tasks

Towards trustworthy machine learning with kernels

Machine Learning has become an indispensable aspect of various safety-critical industries like healthcare, law, and automotive. Hence, it is crucial to ensure that our machine learning models function appropriately and instil trust among their users. This thesis focuses on improving the safety and transparency of Machine Learning by advocating for more principled uncertainty quantification and more effective explainability tools. Specifically, the use of Kernel Mean Embeddings (KME) and Gaussian Processes (GP) is prevalent in this work since they can represent probability distribution with minimal distributional assumptions and capture uncertainty well, respectively. I dedicate Chapter 2 to introduce these two methodologies. Chapter 3 demonstrates an effective use of these methods in conjunction with each other to tackle a statistical downscaling problem, in which a Deconditional Gaussian process is proposed. Chapter 4 considers a causal data fusion problem, where multiple causal graphs are combined for inference. I introduce BayesIMP, an algorithm built using KME and GPs, to draw causal conclusion while accounting for the uncertainty in the data and model. In Chapter 5, I present RKHS-SHAP to model explainability for kernel methods that utilizes Shapley values. Specifically, I propose to estimate the value function in the cooperative game using KMEs, circumventing the need for any parametric density estimations. A Shapley regulariser is also proposed to regulate the amount of contributions certain features can have to the model. Chapter 6 presents a generalised preferential Gaussian processes for modelling preference with non-rankable structure, which sets the scene for Chapter 7, where I built upon my research and propose Pref-SHAP to explain preference models

Exploring Diversity and Fairness in Machine Learning

With algorithms, artificial intelligence, and machine learning becoming ubiquitous in our society, we need to start thinking about the implications and ethical concerns of new machine learning models. In fact, two types of biases that impact machine learning models are social injustice bias (bias created by society) and measurement bias (bias created by unbalanced sampling). Biases against groups of individuals found in machine learning models can be mitigated through the use of diversity and fairness constraints. This dissertation introduces models to help humans make decisions by enforcing diversity and fairness constraints. This work starts with a call to action. Bias is rife in hiring, and since algorithms are being used in multiple companies to filter applicants, we need to pay special attention to this application. Inspired by this hiring application, I introduce new multi-armed bandit frameworks to help assign human resources in the hiring process while enforcing diversity through a submodular utility function. These frameworks increase diversity while using less resources compared to original admission decisions of the Computer Science graduate program at the University of Maryland. Moving outside of hiring I present a contextual multi-armed bandit algorithm that enforces group fairness by learning a societal bias term and correcting for it. This algorithm is tested on two real world datasets and shows marked improvement over other in-use algorithms. Additionally I take a look at fairness in traditional machine learning domain adaptation. I provide the first theoretical analysis of this setting and test the resulting model on two deal world datasets. Finally I explore extensions to my core work, delving into suicidality, comprehension of fairness definitions, and student evaluations

Von Requirements zu Privacy Explanations: Ein nutzerzentrierter Ansatz für Usable Privacy

Im Zeitalter der fortschreitenden Digitalisierung, in dem die Technologie zunehmend in unsere Gesellschaft eindringt, rücken sogenannte human values wie Ethik, Fairness, Privatsphäre und Vertrauen weiter in den Mittelpunkt. Digitale Informationssysteme dringen immer stärker in private und berufliche Bereiche vor und bieten den Nutzern Unterstützung, schnell und einfach mit anderen Menschen in Kontakt zu treten, bei der Informationsbeschaffung und helfen bei der Erledigung täglicher Aufgaben. Im Gegenzug geben die Nutzer bereitwillig große Mengen an persönlichen Daten an diese Systeme weiter. Diese Datenerfassung bedeutet jedoch, dass die Privatsphäre der Nutzer zunehmend gefährdet ist. Daher ist die Aufklärung der Nutzer über die gesammelten Informationen und ihre anschließende Verarbeitung der Schlüssel, die Privatsphäre der Nutzer zu schützen. Der Gesetzgeber hat Datenschutzerklärungen als Mittel zur Kommunikation von Datenpraktiken eingeführt. Leider erweisen sich diese Dokumente für die Endnutzer als praktisch nutzlos, da sie umfangreich, vage formuliert und mit Fachausdrücken gespickt sind, die oft ein tieferes Fachwissen erfordern. Das Ergebnis ist ein Mangel an nutzerorientierten Lösungen zur transparenten und verständlichen Vermittlung von Datenpraktiken. Um diese Lücke zu schließen, wird in dieser Arbeit das Konzept der Erklärbarkeit als entscheidender Qualitätsaspekt zur Verbesserung der Kommunikation zwischen Systemen und Nutzern in Bezug auf Datenpraktiken in einer klaren, verständlichen und nachvollziehbaren Weise untersucht. Zu diesem Zweck wird ein Ansatz vorgeschlagen, der aus drei Theorien besteht, die durch sieben Artefakte gestützt werden, die die Rolle der Erklärbarkeit im Kontext der Privatsphäre skizzieren und Leitlinien für die Kommunikation von Datenschutzinformationen aufstellen. Diese Theorien und Artefakte sollen Software-Experten unterstützen, (a) privatsphärerelevante Aspekte zu identifizieren, (b) diese kontextrelevant und verständlich an den Nutzer zu kommunizieren, um (c) datenschutzfreundliche Systeme zu designen. Um die Wirksamkeit des vorgeschlagenen Ansatzes zu validieren, wurden Evaluierungen durchgeführt, darunter Literaturrecherchen, Workshops und Nutzerstudien. Die Ergebnisse bestätigen die Eignung der entwickelten Theorien und Artefakte und bieten eine vielversprechende Grundlage für die Entwicklung datenschutzfreundlicher, fairer und transparenter Systeme.In the era of ongoing digitalization, where technology increasingly infiltrates our society, fun-damental human values such as ethics, fairness, privacy, and trust have taken center stage. Digital systems have seamlessly penetrated both personal and professional spheres, offering users swift connectivity, information access, and assistance in their daily routines. In exchange, users willingly share copious amounts of personal data with these systems. However, this data collection means that that users’ privacy sphere is increasingly at stake. Therefore, educating users about the information being collected and its subsequent processing is key to protect users’ privacy sphere. Legislation has established privacy policies as a means of communicating data practices. Unfortunately, these documents often prove fruitless for end users due to their extensive, va-gue, and jargon-laden nature, replete with legal terminology that often requires a deeper level of specialized knowledge. The result is a lack of user-centric solutions to communicate privacy information transparently and understandably. To bridge this gap, this thesis explores the concept of explainability as a crucial quality aspect for improving communication between systems and users concerning data practices, in a clear, understandable, and comprehensible manner. To this end, this thesis proposes an approach consisting of three theories supported by seven artifacts that outline the role of explainability in the context of privacy and provide guidelines for communicating privacy information. These theories and artifacts are intended to help software professionals (a) to identify privacy-relevant aspects, (b) to communicate them to users in a contextually relevant and understandable way, and (c) to design privacy-aware systems. To validate the efÏcacy of the proposed approach, evaluations were conducted, including literature reviews, workshops, and user studies. The results endorse the suitability of the de-veloped theories and artifacts, offering a promising foundation for developing privacy-aware, fair, and transparent systems