Models can help scientists study and understand phenomena. Such models need to be informed by theory. We report on the early stages of our ongoing study in which we use interactive visualisation to help improve the construction of theory-driven models, by facilitating the exploration of statistical summaries of input variables, compare the quality of alternative models and keep track of the model-building process. Later work will investigate whether machine-learning techniques can be incorporated without compromising the models' theoretical bases

Butt, S.

Dykes, J.

Lahtinen, K.

Slingsby, A.

Turkay, C.

City Research Online

Turkay, C., Slingsby, A., Lahtinen, K., Butt, S. & Dykes, J. (2016). Enhancing a Social Science Model-building Workflow with Interactive Visualisation. Paper presented at the The European Symposium on Artificial Neural Networks (ESANN 2016), 27-29 Apr 2016, Bruges, Belgium. City Research OnlineOriginal citation: Turkay, C., Slingsby, A., Lahtinen, K., Butt, S. & Dykes, J. (2016). Enhancing a Social Science Model-building Workflow with Interactive Visualisation. Paper presented at the The European Symposium on Artificial Neural Networks (ESANN 2016), 27-29 Apr 2016, Bruges, Belgium. Permanent City Research Online URL: http://openaccess.city.ac.uk/14232/ Copyright & reuseCity University London has developed City Research Online so that its users may access the research outputs of City University London's staff. Copyright © and Moral Rights for this paper are retained by the individual author(s) and/ or other copyright holders.  All material in City Research Online is checked for eligibility for copyright before being made available in the live archive. URLs from City Research Online may be freely distributed and linked to from other web pages. Versions of researchThe version in City Research Online may differ from the final published version. Users are advised to check the Permanent City Research Online URL above for the status of the paper.EnquiriesIf you have any enquiries about any aspect of City Research Online, or if you wish to make contact with the author(s) of this paper, please email the team at publications@city.ac.uk.Enhancing a Social Science Model-buildingWorkflow with Interactive VisualisationCagatay Turkay1, Aidan Slingsby1, Kaisa Lahtinen2,Sarah Butt2, and Jason Dykes11- giCentre, Department of Computer Science, City University London, UK2- Centre for Comparative Social Surveys, City University London, UKAbstract. Models can help scientists study and understand phenom-ena. Such models need to be informed by theory. We report on the earlystages of our ongoing study in which we use interactive visualisation tohelp improve the construction of theory-driven models, by facilitating theexploration of statistical summaries of input variables, compare the qual-ity of alternative models and keep track of the model-building process.Later work will investigate whether machine-learning techniques can beincorporated without compromising the models’ theoretical bases.1 IntroductionTwo main reasons for which scientists build models are to generate data foranalysis or prediction and to understand phenomena better. For our case study,although models with good predictive power are important, the models need tobe sound and strongly informed by theory. Where there are many variable tochoose from, statistical summaries, graphics, statistical techniques are machine-learning techniques often help in the model-building process [1].For social scientists, it is particularly important that models are informed bysocial science theory. This leads to their general mistrust of machine-learningtechniques in favour of more manual variable-selection approaches. This oftenresults in an overwhelmingly manual approach to model-building, suitable forsmall numbers of variables but which does not scale to sitations where there arehundreds. In these cases, social scientists need approaches that produce modelsthey can defend to their peers.The eventual aim of this ongoing study it to use interactive visualisationto incorporate machine-learning techniques to help social scientists deal withlarge numbers of variables, but to do so in a way that does not compromisetheir goal of using models to understand the phenomena and obtaining a modelthey can justify and defend in a social science context. Incorporating interactivevisualisation approaches with computational methods has already shown greatpotential within the field of visual analytics [2]. In this paper, we report on earlystages of this study, in which we have designed and implemented interactive vi-sualisation that implements our collaborator’s existing workflow for constructingmodels. This adds a number of enhancements: (a) it assists in exploring statis-tical summaries of hundreds of variables, (b) it facilitates comparison betweenthe alternative models that are iteratively built and (c) it helps keep track ofthe modelling process and decision made. We describe and discuss our initialideas on designs and functionalities. Subsequent work will investigate how ex-isting machine-learning techniques can be incorporated without compromisingthe theoretic basis for the models.2 Case study: European Social Survey (ESS)We are working with the European Social Survey (ESS), trying to build modelsof the propensity to respond and the responses using a range of different variablesrarely used in this type of modelling [3]. Survey participants are from a geo-graphically structured sample of 4,520 addresses selected to take part in Round6 of the ESS in the UK 2012-13. These data have been linked to auxiliary datafrom administrative sources, commercial consumer profiling, and open-sourcedata through their geographical locations. The resulting dataset consists of401 candidate variables which are considered as proxies for a range of socialphenomena that might be associated with survey non-response. The aim is toinvestigate potential for using auxiliary data (i.e., 401 variables) to understandsurvey non-response bias and how these relate to the survey responses.The usual approach to this [3] is to identify a set of social theories that mightaffect survey nonresponse, concepts that might construct these, and variablesthat can act as proxies to these. In our case, because we already have variables,we assigned them to one or more of these concepts. However, there are multiplepossible interpretations of these elements and building such hierarchies (theories→ concepts → variables) is a process that involves variety and uncertainty.2.1 Existing model building workflowThe current model-building workflow is to construct models by iteratively addingand/or removing variables from a logistic regression model and assessing thechanges through model fitness metrics (e.g., AIC, McFadden). Analyst judg-ment is used to decide on the next steps either to add or remove a variable.The variables chosen depend on the social science theory behind why they werechosen and which variables already exist in the model – a highly manual pro-cess. Statistical summaries and static graphics are used to explore the range anddistributions of variables and assessed along with the social science theory fortheir inclusion. Commands to build models are run in R (statistical package)and the results are summarised on paper as notes and Post-it notes that can bearranged in such as way as to reflect the model-building process.2.2 Key roles for interactive visualisation2.2.1 Incorporating TheoryR1: Provide interactive access to theoretical annotations. Variables’ potentialsuitabilities need to be informed by social science theory (i.e., the relation totheories & concepts as explained above). One way to incorporate such knowledgeis to externalise the relations between the variables, concepts, and the theories,and make these accessible as interaction points throughout the process.2.2.2 Exploring variablesR2: Provide on-demand statistical and graphical summaries of variables, com-parisons and relationships to each other. Variables’ potential suitabilities arealso informed by their statistical and geographical distributions and correlationswith each other. During the process, analysts need to explore, investigate andcompare variables on-demand to maintain a multi-perspective understanding.2.2.3 Interactively building modelsR3: Real-time computation of models on interactively defined domains. Dueto the multidimensional and open-to-interpretation nature of phenomena beingmodelled, methods to easily trigger model computations with interactively de-fined domains are needed to flexibility compare and contrast several alternatives.R4: Immediate feedback on model success. Such interactive model buildingshould also be informed by different metrics of model fitness and success so thatthe analysts can make judgments easily and efficiently.2.2.4 Considering GeographyR5: Provide geographically constrained models and statistics. Geographical as-pects of these response models are important and the drivers of nonresponse biasare likely to vary geographically, thus requires special consideration.2.2.5 Recording the model-building process, i.e., provenanceR6: Maintain history of model building steps. One critical expectation is to beable to document the different models built, compare the variables they containand their performance (often referred to as provenance).R7: Allow models to be annotated. The scientists need to reflect back on theirprocess and explain why a variable is kept or discarded from the model. Thus,they need to annotate models with notes to help them recall their decisions.3 Enhancing the workflowUsing an established approach involving user-centred iterative prototyping/feed-back cycles [4], we designed interactive visualisation to meet these requirements.3.1 VarXplorer prototypeFig. 1 shows the VarXplorer prototype designed to meet requirements relatedto understanding the input variables and how they relate to each other (R1 &R2) and also includes model building and immediate feedback (R3 & R4) withgeographical considerations (R5). Here we employ multiple visualisations whereseveral selections can be combined using Boolean operations. In order to havethe variables as our main visual entities in the graphics, we employ a techniquecalled dual-analysis [5]. To enable this, for each variable we compute severalABCSel.1 : Deprivation Sel.2 : High Corr. DEFFig. 1: VarXplorer to visually explore the variables and interactive model build-ing. Details are given within the text.general and domain-specific measures and projections: i) Statistical momentssuch as µ, σ, skewness, etc. ii) correlations with response rate and other keyvariables such as happiness score, social trust, etc. iii) multidimensional scaling(MDS) on the variables where the distance is pairwise correlations iv) metadatathat externalise the concepts/theories variables relate to (R1).In Fig. 1, a selection of proxies of deprivation is made through an histogram ofvariable metadata (Fig. 1A). The MDS of variables (Fig. 1B) reveals variabilityin deprivation related variables (i.e., not all of them are inter-correlated) and theselected set is refined (with an AND selection) to only those that are correlated.A logistic regression model (response rate as the target) is computed in real-timeon this selection of variables. The visual representation of variables’ importance(Fig. 1C) indicate that living on benefits and electric consumption as havinghigher weights in this model. However, fitness metrics (AIC and McFadden)signals low predictive strength, i.e., 0.034 for McFadden, (Fig. 1C blue bar).Other characteristics of the variables are investigated through a histogram ofvariation (Fig. 1D), a plot of corr. with health and corr. with social trust (Fig.1E), and a plot of corr. with response vs. corr. with happiness score (Fig. 1F).3.2 ModelBuilder prototypeFig. 2 shows the ModelBuilder prototype designed to meet the requirementsrelated to building models including the incorporation of theory and expertknowledge (R1), interactively-building models (R3) with detail of fit (R4) andmodels that are fit to all 11 geographical regions (R5), maintaining a history ofmodels built (R6) and allowing annotations to be attached to the models (R7).For a given outcome variable and starting with an empty model root (Fig.2D), clicking on potential explanatory variables (one of the variables listed inBCEADFFig. 2: ModelBuilder. Prototype that implements the model building workflowfor an outcome variable (A). See text for explanation.Fig. 2B) creates a new model that adds the variable to the set of variables inthe model. As the user adds variables to the model the model tree develops toshow the provenance of the process (Fig. 2D). The variable’s contribution to theselected model (highlighted in yellow in Fig. 2E) is shown for each explanatoryvariable (brown to green in Fig. 2C) and a measure of the model fit (AIC) isshown for each model (red in Fig. 2E). This history is shown as a tree structure(R6), recording the process through which the analyst constructed the model,allowing models to be compared. Free text annotation (R7) allows decisions tobe recorded, and a red cross indicates where the analyst considers the iterativeprocess to have reached a dead-end. A separate model is built for each of the 11geographical regions (R5), laid out geographically in a grid-map [6] accordingto Fig. 2F. This layout depicts both the contribution that the variable makes tothe model (Fig. 2B) and the model fit (Fig. 2E).3.2.1 The Model Building ProcessThe scientists initiate their analysis by investigating the relations within thevariables together with their associations with the known social science theoryconcepts in VarXplorer. This process ends up with a selection of variables whichthey use as a starting point in building a model in ModelBuilder so they don’tstart from an empty model. AndModelBuilder acts as the interactive mechanismto improve the model in a step-wise fashion and keep track of the process.3.2.2 A brief example of the modelling processWe started with a short list of variables that are important to include in themodel to explain non-response (informed by existing literature): % flats in anarea, % people leaving on job-seekers benefit and ethnic fractionlisation, andselect those in VarXplorer. The histogram of concept types revealed that thesevariables only capture deprivation, individual characteristics and heterogeneityas the theoretically matching concepts. We decided that proxies for quality oflife are also needed to achieve a more comprehensive model and investigatedthe correlation relation of variables associated with quality of life to the alreadyexisting three and chose count of cultural facilities as the next variable and savedthis in the ModelBuilder as an alternative. This not only improved the model fitbut also ended up in a more comprehensive coverage of concepts, thus leadingto a model that is easier to defend and explain.4 Discussion, conclusion and further workOur close and multidisciplinary collaboration using an established user-centrediterative prototyping approach is enabling us to design, implement, and evalu-ate appropriate visual analytics tools that are tailored to meet current model-building requirements based on an existing workflow. Our weekly feedback meet-ings as part of our user-centred approach have been instrumental in directingthe development of the design and prototyping. They have also shown strongpotential for visual analytics to more effectively support similar model-buildingprocesses in social sciences and beyond, without taking control away from modelbuilders. We presented two of a number of separate tools with distinct roles usedalongside others, rather then focussing on a single combined tool. We think thisis less disruptive to the modelling-building process and improves adoption. Wehave also implemented a random-forest classifier where variable weights can bevisualised similar to Fig. 1C. The resulting model can be used as a “startingconfiguration” in ModelBuilder and subsequent work will investigate how suchtechniques can be incorporated further without compromising the social scien-tists’ expert knowledge so that greater confidence in the models is achieved.AcknowledgmentsThis work is partly funded by ADDResponse project funded by the UK Economicand Social Research Council (grant ES/L013118/1).References[1] G. James, D. Witten, T. Hastie, and R. Tibshirani. An introduction to statistical learning.Springer, 2013.[2] D. Keim, F. Mansmann, J. Schneidewind, J. Thomas, and H. Ziegler. Visual analytics:Scope and challenges. In Visual Data Mining, pages 76–90. Springer-Verlag, 2008.[3] R.M. Groves and M.P. Couper. Nonresponse in household interview surveys. John Wiley& Sons, 2012.[4] J. Dykes, J. Wood, and A. Slingsby. Rethinking map legends with visualization. IEEETransactions on Visualization and Computer Graphics, 16(6):890 – 899, November 2010.[5] C. Turkay, P. Filzmoser, and H. Hauser. Brushing dimensions – a dual visual analysis modelfor high-dimensional data. Visualization and Computer Graphics, IEEE Transactions on,17(12):2591–2599, Dec. 2011.[6] D. Eppstein, M.J. van Kreveld, B. Speckmann, and B. Staals. Improved grid map layoutby point set matching. Int. J. Comput. Geometry Appl., 25(2):101–122, 2015.

Enhancing a Social Science Model-building Workflow with Interactive Visualisation

Abstract

Similar works

Full text

Available Versions

City Research Online