The continuous interaction between users and the system in visual analytics can be considered a dialogue. We propose the use of multiple two-way channels facilitated by a multimodal interface as a central aspect of interactive visualization design, in particular, the use of natural language with interactive visualization. We discuss key related concepts, potential benefits, challenges and opportunities that emerge as a research agenda for multimodal data analysis

Henkin, R.

Turkay, C.

City Research Online

              City, University of London Institutional RepositoryCitation: Turkay, C. ORCID: 0000-0001-6788-251X and Henkin, R. (2018). Towards Multimodal Data Analytics: Integrating Natural Language into Visual Analytics. Paper presented at the Multimodal Interaction for Data Visualization Workshop at AVI 2018, 29 May 2018, Grosseto, Italy. This is the published version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  http://openaccess.city.ac.uk/20111/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineTowards Multimodal Data Analytics:Integrating Natural Language intoVisual AnalyticsRafael HenkingiCentre, City, University ofLondonNorthampton SquareEC1V 0HB, London, UKrafael.henkin@city.ac.ukCagatay TurkaygiCentre, City, University ofLondonNorthampton SquareEC1V 0HB, London, UKcagatay.turkay@city.ac.ukPermission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the owner/author(s).MultimodalVis 18 Workshop at AVI 2018, May 29, 2018, Grosseto, Italy.https://multimodalvis.github.ioc©2018 Copyright is held by the owner/author(s).AbstractThe continuous interaction between users and the systemin visual analytics can be considered a dialogue. Wepropose the use of multiple two-way channels facilitatedby a multimodal interface as a central aspect ofinteractive visualization design, in particular, the use ofnatural language with interactive visualization. We discusskey related concepts, potential benefits, challenges andopportunities that emerge as a research agenda formultimodal data analysis.ACM Classification KeywordsHuman-centered computing [Interaction design]:Interaction design theory, concepts and paradigmsIntroductionThe continuous interaction between users and systems invisual analytics can be considered a dialogue [10]. Fromthe user side, the dialogue often happens through multipleone-way channels – selecting items with directmanipulation or modifying parameters with menus. Thefeedback that the system then returns, however, is mostlyrestricted to a one-way channel in the form ofimplemented visual encodings. This limits the quality andquantity of feedback that can be given to the userregarding the effects of his/her actions. Extendingmultimodal interaction to both sides is the challenge thatwe address in this ongoing work; our focus at this stage ison the use of natural language in interactive visualization.By envisioning the use of multiple two-way channelsfacilitated by a multimodal interface, we argue that thereare benefits to systems and users; maximizing thesebenefits requires design choices that depend on thecontext of the exchange of information, such as the typeof information, user tasks and roles and the stages ofanalysis.Figure 1: Example ofmulti-channel dialogue.We discuss potential benefits and underlying challengesrelated to this context, setting up a research agenda forthe integration of natural language into visual dataanalysis.Concepts for multimodality in analytical pro-cessesRegarding natural language and multimodality, naturallanguage has seen a recent surge of interest in interactivevisualization, with researchers focused on theimplementation of common visualization interactions innatural language interfaces [2, 8]. To move beyondreplication of interactions from the informationvisualization literature, we suggest that it is essential toidentify the role of users at different stages of analysis.On this topic, Meignan et al. [5] described five user rolesfor intelligent systems in operations research, which arebased on various aspects of interacting with algorithms,but primarily, the target of the user interaction. In visualanalytics, equivalent roles can be defined for the targetsthat match the stages and elements of analyticalworkflows, as follows:Adjust/enrich: when the user modifies the inputs usedby algorithm, such as the data items or classes used inclassification problems;Assist: when the user mimics the algorithm by correctingoutputs, which can be fed back to the system forimprovement;Guiding: when the user modifies the features used byalgorithms, such as including or excluding attributes thatare used for a clustering method, or steers algorithms;Tuning: when the user modifies supporting parameters ofalgorithms, such as the number of clusters;Depending on the tasks and expertise, users can take onone or more of these roles at any stage of the analysis.With this in mind, the different channels through whichinformation is sent can be assigned or limited to differentroles: a system can enable the user to tune algorithmsonly using menus and assist it using natural language. Atthe same time, the system might show the outputs onlythrough visual encodings, reporting back its current statewith natural language. Figure 1 shows an example of amulti-channel dialogue based on this design.On the potential of multimodal analyticsThis new paradigm of multimodal analytics has severalpotential benefits for both users and algorithmicapproaches. This section describes these two categories ofbenefits, as well as the underlying challenges that must beaddressed to maximize them.For the algorithmLearning more than outputsActive learning is an established technique through whichusers can provide examples of correct results toalgorithms. Stumpf et al. [9], however, found that users, ifgiven the chance, are keen to provide more than correctresults. Natural language can increase the expressivenessin this type of interaction.Quality of resultsAlthough algorithm-specific solutions to improve resultsby learning from users have been investigated [3], thechallenge in multimodality is maximizing theimprovements by considering not only the appropriatechannels but what feedback is passed on to users.Amershi et al. [1] demonstrated that users can trainbetter models when presented with high-value inputs.Targeting inputs and featuresHigh-value inputs can also help users to steer algorithmsby targeting these inputs in the data, in a mixed role ofenriching and guiding the system. A key aspect for this ispresenting users with incomplete results and enabling thealgorithm to be steered [10].For the userComprehensionUser interaction can lead to better algorithm results whenthey improve their understanding of the methods with theappropriate feedback [6] and new generated knowledge [7].For this, the methods must be adapted for extracting thenew knowledge and converting them into explanations.TransparencySystems can also communicate decisions to users withoutnecessarily bringing new knowledge. Greater transparencywas shown to to lead to increased satisfaction withresults [4], also leading to an increased trust.Challenges and opportunitiesBased on these benefits, we see the following challengesand opportunities as the way forward to achieve them:Empirically-evidenced multimodal analytics workflowsand vocabularies: Designing workflows, such as the onein figure 2, enables systematic design of studies based ondifferent user roles and stages of analysis, leading tostructured vocabularies for improved active learning. Atthe same time, state-of-the-art AI methods facilitateextending the vocabulary with new utterances afterdeployment. These can then be used in explanations forincreased user comprehension and transparency.Figure 2: Example of a clustering workflow using Orange 1Establishing & evaluating user behaviour in differentroles: User interaction and system feedback must matchthe different user roles and aims. If users intend to tweakparameters of algorithms, they must sufficiently informedby how the parameters will change the results. Thisrequires the development of new evaluation approachesthat are targeted to elicit the role of the user and thealgorithm in each step of an analytical workflow.Visualization designs for multimodal analytics:Although auto-completion and suggestions are commonlyused to instruct users about available interactions, speech1https://orange.biolab.sineeds an altogether different approach – visual cues mightbe used as hints for interaction targets. The sameprinciple applies to conveying feedback when thevisualization does not radically change after aninteraction.ConclusionWith the increasing adoption of complex algorithmicapproaches in data analysis, methods to enable analysts toeffectively utilise algorithms and action on their results areof critical importance. With this paper, we argue thatmultimodal analytics offers novel opportunities tospecifically enhance the experience and the value at thisintersection of humans and algorithms. With this paper,we present an initial position to approach multimodalanalytics with a discussion on open challenges andopportunities to motivate further work in this emergingarea.AcknowledgementsThis work was supported by the EPSRC under grantnumber EP/P025501/1.References[1] Amershi, S., Fogarty, J., Kapoor, A., and Tan, D.Overview based example selection in end userinteractive concept learning. In Proceedings of the22nd annual ACM symposium on User interfacesoftware and technology - UIST ’09, ACM Press(New York, New York, USA, 2009), 247.[2] Hoque, E., Setlur, V., Tory, M., and Dykeman, I.Applying Pragmatics Principles for Interaction withVisual Analytics. IEEE Transactions on Visualizationand Computer Graphics 24, 1 (jan 2018), 309–318.[3] Kim, B., Khanna, R., and Koyejo, O. Examples arenot Enough, Learn to Criticize! Criticism forInterpretability. Nips, Nips (2016), 2280–2288.[4] Kulesza, T., Stumpf, S., Burnett, M., and Kwan, I.Tell me more? The Effects of Mental ModelSoundness on Personalizing an Intelligent Agent.Proceedings of the 2012 ACM annual conference onHuman Factors in Computing Systems - CHI ’12(2012), 1.[5] Meignan, D., Knust, S., Frayret, J.-M., Pesant, G.,and Gaud, N. A Review and Taxonomy of InteractiveOptimization Methods in Operations Research. ACMTransactions on Interactive Intelligent Systems 5, 3(2015), 1–43.[6] Rosenthal, S. L., and Dey, A. K. Towards maximizingthe accuracy of human-labeled sensor data.International Conference on Intelligent UserInterfaces, Proceedings IUI (2010), 259–268.[7] Samek, W., Wiegand, T., and Mu¨ller, K.-R.Explainable Artificial Intelligence: Understanding,Visualizing and Interpreting Deep Learning Models.[8] Srinivasan, A., and Stasko, J. Orko : FacilitatingMultimodal Interaction for Visual Exploration andAnalysis of Networks. InfoVis TVCG 24, SI (2017),11.[9] Stumpf, S., Rajaram, V., Li, L., Wong, W. K.,Burnett, M., Dietterich, T., Sullivan, E., andHerlocker, J. Interacting meaningfully with machinelearning systems: Three experiments. InternationalJournal of Human Computer Studies 67, 8 (2009),639–662.[10] Turkay, C., Kaya, E., Balcisoy, S., and Hauser, H.Designing Progressive and Interactive AnalyticsProcesses for High-Dimensional Data Analysis. IEEETransactions on Visualization and ComputerGraphics 23, 1 (jan 2017), 131–140.

Towards Multimodal Data Analytics: Integrating Natural Language into Visual Analytics

http://openaccess.city.ac.uk/20111/1/Towards%20Multimodal%20Data%20Analytics.pdf

Towards Multimodal Data Analytics: Integrating Natural Language into Visual Analytics

Abstract

Similar works

Full text

Available Versions

City Research Online