The recent advances in natural language based interaction methodologies offer promising avenues to enhance the interactive processes within the human-machine dialogue of visual analytics. We envisage \textit{Multimodal Data Analytics} as a novel approach for conducting data analysis that builds on the strengths of visual analytics and natural language as an expressive interaction channel. We investigate the potential enhancements from such a multimodal approach and discusses the preliminary outline for a structured methodology to study the role of natural language in data analytics. Our approach builds on a simple model of human machine dialogue for interactive data analysis which we then propose to instantiate as visual analytics workflows -- representations to study and operationalise interactive data analysis routines empowered by natural language interaction

Henkin, R.

Turkay, C.

City Research Online

              City, University of London Institutional RepositoryCitation: Turkay, C. & Henkin, R. (2018). Towards Natural Language Empowered Interactive Data Analysis. Poster presented at the EuroVis 2018 20th EG/VGTC Conference on Visualization, 04 - 08 June 2018, Brno, Czech Republic. This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  http://openaccess.city.ac.uk/20110/Link to published version: http://dx.doi.org/10.2312/eurp20181128Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineEUROVIS 2018/ A. Puig and R. Raidou PosterTowards Natural Language Empowered Interactive Data AnalysisC. Turkay1 and R. Henkin11giCentre, Department of Computer Science, City, University of London, UKAbstractThe recent advances in natural language based interaction methodologies offer promising avenues to enhance the interactiveprocesses within the human-machine dialogue of visual analytics. We envisage Multimodal Data Analytics as a novel approachfor conducting data analysis that builds on the strengths of visual analytics and natural language as an expressive interactionchannel. We investigate the potential enhancements from such a multimodal approach and discusses the preliminary outlinefor a structured methodology to study the role of natural language in data analytics. Our approach builds on a simple modelof human machine dialogue for interactive data analysis which we then propose to instantiate as visual analytics workflows –representations to study and operationalise interactive data analysis routines empowered by natural language interaction.CCS Concepts•Human-centered computing → Visual analytics;1. IntroductionThe success of visual analytics systems relies on an iterativehuman-computer dialogue where the knowledge and the creativ-ity of the human, and the power of computation operate together todeal with the complexities in the data sources [SSS∗14, ALA∗]. Insuch human-in-the-loop data analysis approaches, interactive visu-alisation methods are core facilitators of this dialogue [TKBH17a]and have proven their merit in several ways. However, despite thewide literature and established products in visualisation [ERT∗17],such iterative visual analytics processes mostly rely on conven-tional, not often intuitive interaction mechanisms that can introduceunnecessary complexities into the process [Lam08].The recent advances in natural language based interactionmethodologies offer promising avenues to rethink the interactionprocesses within visual analytics [BHCDV99] and natural lan-guage is getting increasing interest from the visual analytics com-munity as a new channel to conduct and support visual analy-sis [HSTD18, SS17a, SS17b]. We envisage Multimodal Data An-alytics as a novel potential approach for conducting data analysisthat combines the strengths of visual analytics techniques and nat-ural language – as an expressive channel – for interaction. In thisposter, we discuss preliminary outcomes from an ongoing 18 monthlong research project and present initial considerations towards thisanalysis approach, and present the preliminary attempts to builda structured approach in exploring the open questions within thisemerging approach to data analysis. In the context of this work,we consider settings where a computational method is embeddedwithin an interactive visual analytics framework, with the algorithmhaving a means to responding to user inputs.2. Enhancing Data Analysis with Natural LanguageInteractionThis new paradigm of multimodal analytics offers a number of po-tential enhancements for interactive data analysis processes. In thefollowing, we investigate them under a number of headings andthen provide a few illustrative and speculative scenarios.E1: Enhanced expressiveness for guidance: Active learning isan established technique through which users can provide exam-ples of correct results to algorithms [CGJ96]. Users, however, of-ten have more than only the correct results to provide to the algo-rithms [SRL∗09] and natural language has the potential to facilitateusers in providing guidance to algorithms through complex utter-ances that explain their way of thinking more effectively to revealtheir tacit knowledge of the domain.E2: Enhanced model steering: Natural language inputs can en-able users to steer algorithms by targeting these inputs in the data,in a mixed role of enriching and guiding the system. A key aspectfor this is presenting users with incomplete results and enabling thealgorithm to be steered [TKBH17b].E3: Enhanced comprehension and transparency: Expressive-ness of the natural language opens up new possibilities to improvethe communication of model results and the execution logic of al-gorithms. Greater transparency has been shown to lead to increasedsatisfaction with results [KSBK12] with also increased trust incomputational models. Natural language offers an effective chan-nel in this regard to embellish and enrich visual representations,and draw attention to uncertainties.E4: Enhanced knowledge elicitation: A more natural flow of thec© 2018 The Author(s)Eurographics Proceedings c© 2018 The Eurographics Association.DOI: 10.2312/eurp.20181128C. Turkay & R. Henkin / Towards Natural Language Empowered Interactive Data AnalysisFigure 1: A two-way dialogue model where a number of interac-tion facets are listed as points where useful interactions can occurbetween the user and an algorithm.interaction dialogue can foster more effective knowledge elicita-tion. In particular in cases where the analytics system supports sev-eral workflows, language provides an effective channel to learn andadapt to users’ tasks and expectations within an analysis session.Speculative Examples:Ex-1: In response to a projection of high-dimensional points, anutterance "... the observations on the left do not share the sameincome characteristics, so this looks wrong ..." from the user couldbe translated into updates of the weights of income related variablesin the computation (E1, E2) and further guidance can be gatheredthrough a question "... there seems to be redundancy within incomerelated features as seen above (referring to a new visualisation ofincome related features) , consider excluding some? ... (E1, E4).Ex-2: In the presentation of clustering results of the same pointsand following the guidance from the user above, a visual repre-sentation of the clusters could be supported by an utterance of thesystem – ... here, more importance is given to income, so the twogroups differ significantly in their income characteristics but no-tice now the weaker discrimination overall ... – not only enhancescomprehension but also highlights the issues in the results (E3).3. Investigating the Role for Natural Language InteractionIn order to investigate the role of natural language interaction,we firstly investigate the theoretical models of visual analyt-ics [ERT∗17, ALA∗] and also in other disciplines such as intel-ligent systems [MKF∗15] and robotics [ACKK14]. Building onthese works, a user can: R1 - Adjust/enrich by modifying the in-puts used by algorithm, such as the data items or classes used inclassification, R2 - Assist by mimicking the algorithm through cor-recting the outputs to be fed back to the system, R3 - Guide: bymodifying the data features used by algorithms, R4 - Tune by mod-ifying the parameters of algorithms, such as the number of clusters.In addition to these roles, we also consider a number of ‘inter-action facets’ where the interaction between the user and the com-puter could conceptually take place over (see Figure 1). Here, welist six interaction facets as points where useful interactions canoccur between the user and an algorithm: Data Inputs, Algorith-mic Outputs, Data Features, Algorithmic Parameters, AlgorithmicLogic, Progress State of the algorithm.Figure 2: An analytical workflow illustration (built here using theOrange software https://orange.biolab.si/) that servesas an instantiation of the high-level communication model pre-sented in Figure 1 for a clustering algorithm with facets indicated.3.1. Analytical WorkflowsThe above general communication model provides the high-levelframework to study in which of the interaction facets, natural lan-guage could benefit most in coordination with visualisation meth-ods. As instantiations of this high-level framework, we are develop-ing analytical workflows for a wide range of algorithmic methods,identify the interaction facets within these interactive processes,and use these systematic approaches both for empirically study-ing the benefits of using different modalities and also as analysistemplates for potential users of such approaches. One example ofthis is the case of using the k-means algorithm to cluster a high-dimensional data set. For such an analytical task, we build analyt-ical workflows (Figure 2) where a number of the interaction facetsare explicitly denoted. In Figure 2, for instance, Selecting Columnsfor k-means (R3) to consider is an instance of the interaction facetFeatures that can be captured in a similar fashion as Ex-1, whereas the Selection of k value is an instance of the interaction facet Pa-rameters which is of role type R4. Future work here is to cover awide range of algorithms [ERT∗17] and provide structured work-flows where most of the interaction facets are explicitly denoted.4. ConclusionWith the increasing adoption of complex algorithmic approachesin data analysis, methods to enable analysts to effectively utilisealgorithms and action on their results are of critical importance.In this poster, we present initial discussions and the sketch of asystematic way to approach the study of multimodal analytics. Toadvance the field, further research is needed to: gather empiricalinformation on the language used by analysts both at a generic anddomain specific level, analyse the effectiveness of mediums andtechniques for the different interaction facets, and develop a widerange of algorithm agnostic visual analytics workflows.AcknowledgementsThis work is supported by the UK Engineering and Physical Sci-ences Research Council (EPSRC) [nlvis: Natural Language Inter-action for Visual Data Analysis, grant number: EP/P025501/1]c© 2018 The Author(s)Eurographics Proceedings c© 2018 The Eurographics Association.54C. Turkay & R. Henkin / Towards Natural Language Empowered Interactive Data AnalysisReferences[ACKK14] AMERSHI S., CAKMAK M., KNOX W. B., KULESZA T.:Power to the People: The Role of Humans in Interactive Machine Learn-ing. AI Magazine 35, 4 (dec 2014), 105. 2[ALA∗] ANDRIENKO N., LAMMARSCH T., ANDRIENKO G., FUCHSG., KEIM D., MIKSCH S., RIND A.: Viewing visual analytics as modelbuilding. Computer Graphics Forum 0, 0. 1, 2[BHCDV99] BRADEN-HARDER L., CORSTON S. H., DOLAN W. B.,VANDERWENDE L. H.: Apparatus and methods for an information re-trieval system that employs natural language processing of search resultsto improve overall precision, Aug. 3 1999. US Patent 5,933,822. 1[CGJ96] COHN D. A., GHAHRAMANI Z., JORDAN M. I.: Active learn-ing with statistical models. Journal of artificial intelligence research(1996). 1[ERT∗17] ENDERT A., RIBARSKY W., TURKAY C., WONG B., NAB-NEY I., BLANCO I. D., ROSSI F.: The state of the art in integratingmachine learning into visual analytics. In Computer Graphics Forum(2017), vol. 36, Wiley Online Library, pp. 458–486. 1, 2[HSTD18] HOQUE E., SETLUR V., TORY M., DYKEMAN I.: Apply-ing Pragmatics Principles for Interaction with Visual Analytics. IEEETransactions on Visualization and Computer Graphics 24, 1 (jan 2018),309–318. 1[KSBK12] KULESZA T., STUMPF S., BURNETT M., KWAN I.: Tellme more? The Effects of Mental Model Soundness on Personalizing anIntelligent Agent. Proceedings of the 2012 ACM annual conference onHuman Factors in Computing Systems - CHI ’12 (2012), 1. 1[Lam08] LAM H.: A framework of interaction costs in information visu-alization. IEEE transactions on visualization and computer graphics 14,6 (2008). 1[MKF∗15] MEIGNAN D., KNUST S., FRAYRET J.-M., PESANT G.,GAUD N.: A Review and Taxonomy of Interactive Optimization Meth-ods in Operations Research. ACM Transactions on Interactive IntelligentSystems 5, 3 (2015), 1–43. 2[RD10] ROSENTHAL S. L., DEY A. K.: Towards maximizing the accu-racy of human-labeled sensor data. International Conference on Intelli-gent User Interfaces, Proceedings IUI (2010), 259–268.[SRL∗09] STUMPF S., RAJARAM V., LI L., WONG W. K., BURNETTM., DIETTERICH T., SULLIVAN E., HERLOCKER J.: Interacting mean-ingfully with machine learning systems: Three experiments. Interna-tional Journal of Human Computer Studies 67, 8 (2009), 639–662. 1[SS17a] SRINIVASAN A., STASKO J.: Natural Language Interfaces forData Analysis with Visualization: Considering What Has and Could BeAsked. In EuroVis 2017 - Short Papers (2017), Kozlikova B., SchreckT., Wischgoll T., (Eds.), The Eurographics Association. 1[SS17b] SRINIVASAN A., STASKO J.: Orko : Facilitating MultimodalInteraction for Visual Exploration and Analysis of Networks. InfoVisTVCG 24, SI (2017), 11. 1[SSS∗14] SACHA D., STOFFEL A., STOFFEL F., KWON B. C., EL-LIS G., KEIM D. A.: Knowledge generation model for visual analyt-ics. IEEE transactions on visualization and computer graphics 20, 12(2014), 1604–1613. 1[SWM17] SAMEK W., WIEGAND T., MÜLLER K.-R.: Explainable Ar-tificial Intelligence: Understanding, Visualizing and Interpreting DeepLearning Models.[TKBH17a] TURKAY C., KAYA E., BALCISOY S., HAUSER H.: Design-ing progressive and interactive analytics processes for high-dimensionaldata analysis. IEEE transactions on visualization and computer graphics23, 1 (2017), 131–140. 1[TKBH17b] TURKAY C., KAYA E., BALCISOY S., HAUSER H.: De-signing Progressive and Interactive Analytics Processes for High-Dimensional Data Analysis. IEEE Transactions on Visualization andComputer Graphics 23, 1 (jan 2017), 131–140. 1c© 2018 The Author(s)Eurographics Proceedings c© 2018 The Eurographics Association.55

Towards Natural Language Empowered Interactive Data Analysis

http://openaccess.city.ac.uk/20110/13/Towards%20Natural%20Language%20Empowered%20Interactive%20Data%20Analysis.pdf

Towards Natural Language Empowered Interactive Data Analysis

Abstract

Similar works

Full text

Available Versions

City Research Online