Learning from Sequential User Data: Models and Sample-efficient Algorithms

Recent advances in deep learning have made learning representation from ever-growing datasets possible in the domain of vision, natural language processing (NLP), and robotics, among others. However, deep networks are notoriously data-hungry; for example, training language models with attention mechanisms sometimes requires trillions of parameters and tokens. In contrast, we can often access a limited number of samples in many tasks. It is crucial to learn models from these `limited\u27 datasets. Learning with limited datasets can take several forms. In this thesis, we study how to select data samples sequentially such that downstream task performance is maximized. Moreover, we study how to introduce prior knowledge in the deep networks to maximize prediction performance. We focus on four sequential tasks: computerized adaptive testing in psychometrics, sketching in recommender systems, knowledge tracing in computer-assisted education, and career path modeling in the labor market. In the first two tasks, we devise novel sample-efficient algorithms to query a minimal number of sequential samples to improve future predictions. We propose a Bilevel Optimization-Based framework for computerized adaptive testing to learn a data-driven question selection algorithm that improves existing data selection policies. We also tackle the sketching problem in the recommender system, with the task of recommending the next item using a stored subset of prior data samples. In this setting, we develop a data-driven sequential selection algorithm that tackles evolving downstream task distribution. In the last two tasks, we devise novel neural models to introduce prior knowledge exploiting limited data samples. For knowledge tracing, we propose a novel neural architecture, inspired by cognitive and psychometric models, to improve the prediction of students\u27 future performance and utilize the labeled data samples efficiently. For career path modeling, we propose a novel and interpretable monotonic nonlinear state-space model to analyze online user professional profiles and provide actionable feedback and recommendations to users on how they can reach their career goals. The data-driven differentiable data selection algorithms for the first two tasks open up future directions to query (a non-differentiable operation) a minimal number of samples optimally to maximize prediction performance. The structures, introduced in the neural architecture for the models in the last two tasks using prior knowledge, open up future directions to learn deep models augmented with prior knowledge using limited data samples

ReLoop2: Building Self-Adaptive Recommendation Models via Responsive Error Compensation Loop

Industrial recommender systems face the challenge of operating in non-stationary environments, where data distribution shifts arise from evolving user behaviors over time. To tackle this challenge, a common approach is to periodically re-train or incrementally update deployed deep models with newly observed data, resulting in a continual training process. However, the conventional learning paradigm of neural networks relies on iterative gradient-based updates with a small learning rate, making it slow for large recommendation models to adapt. In this paper, we introduce ReLoop2, a self-correcting learning loop that facilitates fast model adaptation in online recommender systems through responsive error compensation. Inspired by the slow-fast complementary learning system observed in human brains, we propose an error memory module that directly stores error samples from incoming data streams. These stored samples are subsequently leveraged to compensate for model prediction errors during testing, particularly under distribution shifts. The error memory module is designed with fast access capabilities and undergoes continual refreshing with newly observed data samples during the model serving phase to support fast model adaptation. We evaluate the effectiveness of ReLoop2 on three open benchmark datasets as well as a real-world production dataset. The results demonstrate the potential of ReLoop2 in enhancing the responsiveness and adaptiveness of recommender systems operating in non-stationary environments.Comment: Accepted by KDD 2023. See the project page at https://xpai.github.io/ReLoo

Analyzing library collections with starfield visualizations

This paper presents a qualitative and formative study of the uses of a starfield-based visualization interface for analysis of library collections. The evaluation process has produced feedback that suggests ways to significantly improve starfield interfaces and the interaction process to improve their learnability and usability. The study also gave us clear indication of additional potential uses of starfield visualizations that can be exploited by further functionality and interface development. We report on resulting implications for the design and use of starfield visualizations that will impact their graphical interface features, their use for managing data quality and their potential for various forms of visual data mining. Although the current implementation and analysis focuses on the collection of a physical library, the most important contributions of our work will be in digital libraries, in which volume, complexity and dynamism of collections are increasing dramatically and tools are needed for visualization and analysis

Experiences with starfield visualizations for analysis of library collections

This paper presents a qualitative and formative study of the uses of a starfield-based visualization interface for analysis of library collections. The evaluation process has produced feedback that suggests ways to significantly improve starfield interfaces and the interaction process to improve their learnability and usability. The study also gave us clear indication of additional potential uses of starfield visualizations that can be exploited by further functionality and interface development. We report on resulting implications for the design and use of starfield visualizations that will impact their graphical interface features, their use for managing data quality and their potential for various forms of visual data mining. Although the current implementation and analysis focuses on the collection of a physical library, the most important contributions of our work will be in digital libraries, in which volume, complexity and dynamism of collections are increasing dramatically and tools are needed for visualization and analysis

Identifying Correlated Heavy-Hitters in a Two-Dimensional Data Stream

We consider online mining of correlated heavy-hitters from a data stream. Given a stream of two-dimensional data, a correlated aggregate query first extracts a substream by applying a predicate along a primary dimension, and then computes an aggregate along a secondary dimension. Prior work on identifying heavy-hitters in streams has almost exclusively focused on identifying heavy-hitters on a single dimensional stream, and these yield little insight into the properties of heavy-hitters along other dimensions. In typical applications however, an analyst is interested not only in identifying heavy-hitters, but also in understanding further properties such as: what other items appear frequently along with a heavy-hitter, or what is the frequency distribution of items that appear along with the heavy-hitters. We consider queries of the following form: In a stream S of (x, y) tuples, on the substream H of all x values that are heavy-hitters, maintain those y values that occur frequently with the x values in H. We call this problem as Correlated Heavy-Hitters (CHH). We formulate an approximate formulation of CHH identification, and present an algorithm for tracking CHHs on a data stream. The algorithm is easy to implement and uses workspace which is orders of magnitude smaller than the stream itself. We present provable guarantees on the maximum error, as well as detailed experimental results that demonstrate the space-accuracy trade-off

시각화 초심자에게 시각적 비교를 돕는 정보 시각화 기술의 디자인

학위논문(박사)--서울대학교 대학원 :공과대학 컴퓨터공학부,2020. 2. 서진욱.The visual comparison is one of the fundamental tasks in information visualization (InfoVis) that enables people to organize, evaluate, and combine information fragmented in visualizations. For example, people perform visual comparison tasks to compare data over time, from different sources, or with different analytic models. While the InfoVis community has focused on understanding the effectiveness of different visualization designs for supporting visual comparison tasks, it is still unclear how to design effective comparative visualizations due to several limitations: (1) Empirical findings and practical implications from those studies are fragmented, and (2) we lack user studies that directly investigated the effectiveness of different visualization designs for visual comparison. In this dissertation, we present the results of three studies to build our knowledge on how to support effective visual comparison to InfoVis novices⁠—general people who are not familiar with visual representations and visual data exploration process. Identifying the major stages in the visualization construction process where novices confront challenges with visual comparison tasks, we explored two high-level comparison tasks with actual users: comparing visual mapping (encoding barrier) and comparing information (interpretation barrier) in visualizations. First, we conducted a systematical literature review on research papers (N = 104) that focused on supporting visual comparison tasks to gather and organize the practical insights that researchers gained in the wild. From this study, we offered implications for designing comparative visualizations, such as actionable guidelines, as well as the lucid categorization of comparative designs which can help researchers explore the design space. In the second study, we performed a qualitative user study (N = 24) to investigate how novices compare and understand visual mapping suggested in a visual-encoding recommendation interface. Based on the study, we present novices' main challenges in using visual encoding recommendations and design implications as remedies. In the third study, we conducted a design study in the area on bioinformatics to design and implement a visual analytics tool, XCluSim, that helps users to compare multiple clustering results. Case studies with a bioinformatician showed that our system enables analysts to easily evaluate the quality of a large number of clustering results. Based on the results of three studies in this dissertation, we suggest a future research agenda, such as designing recommendations for visual comparison and distinguishing InfoVis novices from experts.시각적 비교는 정보 시각화를 이용한 핵심적인 데이터 분석 과정 중 하나로써, 분산되어 있는 정보들을 사람들이 서로 정리, 평가, 병합할 수 있도록 돕는다. 예를 들어, 사람들은 시간의 흐름에 따른 데이터의 변화를 보거나, 서로 다른 출처의 데이터를 비교하거나, 같은 데이터를 여러 분석 모델들을 이용해 평가하기 위해 시각적 비교 과업을 흔히 수행하게 된다. 효과적인 시각화 디자인을 위한 여러 연구가 정보 시각화 분야에서 이루어지고 있는 반면, 어떤 디자인을 통해 효과적으로 시각적 비교를 지원할 수 있는지에 대한 이해는 다음의 제약들로 인해 아직까지 불분명하다. (1) 경험적 통찰들과 실용적 설계 지침들이 파편화되어 있으며 (2) 비교 시각화를 지원하는 방법을 이해하기 위한 사용자 실험의 수가 여전히 제한적이다. 본 논문에서는 시각화 초심자들에게 효과적으로 시각적 비교를 지원하기 위한 정보 시각화 디자인 방법을 더 깊이 이해하기 위해서 일련의 세 연구를 진행하고 이에 대한 결과를 제시한다. 특별히, 시각화 초심자들이 시각적 비교를 할 때 어려움을 경험할 수 있는 두 주요 시각화 단계를 확인함으로써, 본 연구에서는 시각적 인코딩 비교 (인코딩 장벽) 및 정보 비교 (해석 장벽) 과업들에 초점을 맞춘다. 첫째, 비교 시각화 디자인을 제시한 문헌들(N = 104)을 체계적으로 조사 및 분석함으로써 시각화 연구자들이 사용자 실험과 시각화 설계 과정을 통해 얻은 실용적 통찰들을 정리하였다. 이 문헌조사를 기반으로 비교 시각화 설계에 대한 지침들을 정립하고, 비교 시각화를 위한 디자인 공간을 더 깊이 이해하고 탐색하는 데 도움을 줄 수 있는 시각화 분류 및 예시들을 제공한다. 둘째, 초심자들이 시각화 추천 인터페이스에서 어떻게 새로운 시각적 인코딩들을 서로 비교하고 사용하는지에 대한 이해를 돕기 위해 사용자 실험(N = 24)을 수행하였다. 이 실험의 결과를 기반으로, 초심자들의 주요 어려움들과 이들을 해결하기 위한 디자인 지침들을 제시한다. 셋째, 생명정보학자가 시각적으로 다수 개의 클러스터링 결과들을 비교 및 분석할 수 있도록 도와주는 시각화 시스템, XCluSim을 디자인하고 구현하는 디자인 스터디를 수행하였다. 사례 연구를 통해 실제로 생명정보학자가 XCluSim을 이용하여 많은 클러스터링 결과들을 쉽게 비교 및 평가할 수 있다는 것을 보였다. 마지막으로, 이 세 연구 결과들을 기반으로 비교 시각화 분야에서 유망한 향후 연구들을 제시한다.CHAPTER 1. Introduction 1 1.1 Background and Motivation 1 1.2 Research Questions and Approaches 4 1.2.1 Revisiting Comparative Layouts: Design Space, Guidelines, and Future Directions 5 1.2.2 Understanding How InfoVis Novices Compare Visual Encoding Recommendation 6 1.2.3 Designing XCluSim: a Visual Analytics System for Comparing Multiple Clustering Results 7 1.3 Dissertation Outline 8 CHAPTER 2. Related Work 9 2.1 Visual Comparison Tasks 9 2.2 Visualization Designs for Comparison 10 2.2.1 Gleicher et al.s Comparative Layout 11 2.3 Understanding InfoVis Novices 12 2.4 Visualization Recommendation Interfaces 13 2.5 Comparative Visualizations for Cluster Analysis 14 CHAPTER 3. Comparative Layouts Revisited: Design Space, Guidelines, and Future Directions 19 3.1 Introduction 19 3.2 Literature Review 21 3.2.1 Method 22 3.3 Comparative Layouts in The Wild 23 3.3.1 Classifying Comparison Tasks in User Studies 25 3.3.2 Same LayoutIs Called Differently 26 3.3.3 Lucid Classification of Comparative Layouts 28 3.3.4 Advantages and Concerns of Using Each Layout 30 3.3.5 Trade-offs between Comparative Layouts 36 3.3.6 Approaches to Overcome the Concerns 38 3.3.7 Comparative Layout Explorer 42 3.4 Discussion 42 3.4.1 Guidelines for Comparative Layouts 44 3.4.2 Promising Directions for Future Research 48 3.5 Summary 49 CHAPTER 4. Understanding How InfoVis Novices Compare Visual Encoding Recommendation 51 4.1 Motivation 51 4.2 Interface 53 4.2.1 Visualization Goals 53 4.2.2 Recommendations 54 4.2.3 Representation Methods for Recommendations 54 4.2.4 Interface 58 4.2.5 Pilot Study 61 4.3 User Study 62 4.3.1 Participants 62 4.3.2 Interface 62 4.3.3 Tasks and Datasets 65 4.3.4 Procedure. 65 4.4 Findings 68 4.4.1 Poor Design Decisions 68 4.4.2 Role of Preview, Animated Transition, and Text 69 4.4.3 Challenges For Understanding Recommendations 70 4.4.4 Learning By Doing 71 4.4.5 Effects of Recommendation Order 71 4.4.6 Personal Criteria for Selecting Recommendations 72 4.5 Discussion 73 4.5.1 Design Implications 73 4.5.2 Limitations and FutureWork 75 4.6 Summary 77 CHAPTER 5. Designing XCluSim: a Visual Analytics System for Comparing Multiple Clustering Results 78 5.1 Motivation 78 5.2 Task Analysis and Design Goals 79 5.3 XCluSim 80 5.3.1 Color Encoding of Clusters Using Tree Colors 82 5.3.2 Overview of All Clustering Results 83 5.3.3 Visualization for Comparing Selected Clustering Results 86 5.3.4 Visualization for Individual Clustering Results 92 5.3.5 Implementation 100 5.4 CaseStudy 100 5.4.1 Elucidating the Role of Ferroxidase in Cryptococcus Neoformans Var. Grubii H99 (CaseStudy 1) 100 5.4.2 Finding a Clustering Result that Clearly Represents Biological Relations (CaseStudy 2) 103 5.5 Discussion 106 5.5.1 Limitations and FutureWork 108 5.6 Summary 108 CHAPTER 6. Future Research Agenda 110 6.0.1 Recommendation for Visual Comparison 110 6.0.2 Understanding the Perception of Subtle Difference 111 6.0.3 Distinguishing InfoVis Novices from Experts 112 CHAPTER 7. Conclusion. 113 Abstract (Korean) 129 Acknowledgments (Korean) 131Docto

Empathy, connectivity, authenticity, and trust: A rhetorical framework for creating and evaluating interaction design

Relationships are synergistic. Relational theories describe how we create and sustain relationships and take into consideration our own experiences, our own social location and include broad cultural signifiers. Part of our development as people is to learn about power; our own power, and others\u27 power. This thesis offers the combinational addition of Relational-Cultural Theory and the Connectivity Model to the spectrum of interaction design. Since interaction design is about designing mediating tools for people and their subsequent behaviors, particular attention is needed into establishing and maintaining relationship between designer and audience. Relational-Cultural Theory pushes against typical patriarchal structures and values in the United States. These typical power over values/structures include men over women, whites over blacks, logic over emotion, provider over nurturer, and so on. Relational-Cultural Theory seeks a flatness of power. It creates a sense of shared power, or power with others. This idea of shared power can lead to collaborative creation in interaction design to produce useful and good designs. Empathy, mutuality, and authenticity are essential in recognizing our own limits and strengths in connection with others. Building trust requires a mix of all three of these tenets, as well as evolution through conflict. Interaction designers can move toward creating an inclusive theory for this discipline by becoming vulnerable and sharing power with the people with whom they design interactions. Therefore, the rhetorical framework of empathy, connectivity, authenticity, and trust (e-CAT) is presented as a means of creating and evaluating interaction design