DeepSI: Interactive Deep Learning for Semantic Interaction

In this paper, we design novel interactive deep learning methods to improve semantic interactions in visual analytics applications. The ability of semantic interaction to infer analysts' precise intents during sensemaking is dependent on the quality of the underlying data representation. We propose the $\text{DeepSI}_{\text{finetune}}$ framework that integrates deep learning into the human-in-the-loop interactive sensemaking pipeline, with two important properties. First, deep learning extracts meaningful representations from raw data, which improves semantic interaction inference. Second, semantic interactions are exploited to fine-tune the deep learning representations, which then further improves semantic interaction inference. This feedback loop between human interaction and deep learning enables efficient learning of user- and task-specific representations. To evaluate the advantage of embedding the deep learning within the semantic interaction loop, we compare $\text{DeepSI}_{\text{finetune}}$ against a state-of-the-art but more basic use of deep learning as only a feature extractor pre-processed outside of the interactive loop. Results of two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment, show that $\text{DeepSI}_{\text{finetune}}$ more accurately captures users' complex mental models with fewer interactions

NeuralSI: Neural Design of Semantic Interaction for Interactive Deep Learning

An increasing number of studies have utilized interactive deep learning as the analytic model of visual analytics systems for complex sensemaking tasks. In these systems, traditional interactive dimensionality reduction (DR) models are commonly utilized to build a bi-directional bridge between high-dimensional deep learning representations and low-dimensional visualizations. While these systems better capture analysts' intents in the context of human-in-the-loop interactive deep learning, traditional DR cannot support several desired properties for visual analytics, including out-of-sample extensions, stability, and real-time inference. To avoid this issue, we propose the neural design framework of semantic interaction for interactive deep learning. In our framework, we replace the traditional DR with a neural projection network and append it to the deep learning model as the task-specific output layer. Therefore, the analytic model (deep learning) and visualization method (interactive DR) form one integrated end-to-end trainable deep neural network. In order to understand the performance of the neural design in comparison to the state-of-the-art, we systematically performed two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment. The results of these studies indicate that the neural design can give semantic interaction systems substantial advantages while still keeping comparable inference ability compared to the state-of-the-art model.Comment: 19 pages, 9 figure

Evaluation methodology for visual analytics software

O desafio do Visual Analytics (VA) é produzir visualizações que ajudem os utilizadores a concentrarem-se no aspecto mais relevante ou mais interessante dos dados apresentados. A sociedade actual enfrenta uma quantidade de dados que aumenta rapidamente. Assim, os utilizadores de informação em todos os domínios acabam por ter mais informação do que aquela com que podem lidar. O software VA deve suportar interacções intuitivas para que os analistas possam concentrar-se na informação que estão a manipular, e não na técnica de manipulação em si. Os ambientes de VA devem procurar minimizar a carga de trabalho cognitivo global dos seus utilizadores, porque se tivermos de pensar menos nas interacções em si, teremos mais tempo para pensar na análise propriamente dita. Tendo em conta os benefícios que as aplicações VA podem trazer e a confusão que ainda existe ao identificar tais aplicações no mercado, propomos neste trabalho uma nova metodologia de avaliação baseada em heurísticas. A nossa metodologia destina-se a avaliar aplicações através de testes de usabilidade considerando as funcionalidades e características desejáveis em sistemas de VA. No entanto, devido à sua natureza quatitativa, pode ser naturalmente utilizada para outros fins, tais como comparação para decisão entre aplicações de VA do mesmo contexto. Além disso, seus critérios poderão servir como fonte de informação para designers e programadores fazerem escolhas apropriadas durante a concepção e desenvolvimento de sistemas de VA