Visualization Evaluation for Cyber Security: Trends and Future Directions

The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. This research effort has created many tools and techniques that could be applied to improve cyber security, yet the community has not yet established unified standards for evaluating these approaches to predict their operational validity. In this paper, we survey and categorize the evaluation metrics, components and techniques that have been utilized in the past decade of VizSec research literature. We also discuss existing methodological gaps in evaluating visualization in cyber security, and suggest potential avenues for future re- search in order to help establish an agenda for advancing the state-of-the-art in evaluating cyber security visualization

Toward Theoretical Techniques for Measuring the Use of Human Effort in Visual Analytic Systems

Visual analytic systems have long relied on user studies and standard datasets to demonstrate advances to the state of the art, as well as to illustrate the efficiency of solutions to domain-specific challenges. This approach has enabled some important comparisons between systems, but unfortunately the narrow scope required to facilitate these comparisons has prevented many of these lessons from being generalized to new areas. At the same time, advanced visual analytic systems have made increasing use of human-machine collaboration to solve problems not tractable by machine computation alone. To continue to make progress in modeling user tasks in these hybrid visual analytic systems, we must strive to gain insight into what makes certain tasks more complex than others. This will require the development of mechanisms for describing the balance to be struck between machine and human strengths with respect to analytical tasks and workload. In this paper, we argue for the necessity of theoretical tools for reasoning about such balance in visual analytic systems and demonstrate the utility of the Human Oracle Model for this purpose in the context of sensemaking in visual analytics. Additionally, we make use of the Human Oracle Model to guide the development of a new system through a case study in the domain of cybersecurity

Software Usability

This volume delivers a collection of high-quality contributions to help broaden developers’ and non-developers’ minds alike when it comes to considering software usability. It presents novel research and experiences and disseminates new ideas accessible to people who might not be software makers but who are undoubtedly software users

A characteristic-based visual analytics approach to detect subtle attacks from NetFlow records

Security is essentially important for any enterprise networks. Denial of service, port scanning, and data exfiltration are among of the most common network intrusions. It\u27s urgent for network administrators to detect such attacks effectively and efficiently from network traffic. Though there are many intrusion detection systems (IDSs) and approaches, Visual Analytics (VA) provides a human-friendly approach to detect network intrusions with situational awareness functionality. Overview visualization is the first and most important step in a VA approach. However, many VA systems cannot effectively identify subtle attacks from massive traffic data because of the incapability of overview visualizations. In this work, we developed two overviews and tried to identify subtle attacks directly from these two overviews. Moreover, zoomed-in visualizations were also provided for further investigation. The primary data source was NetFlow and we evaluated the VA system with datasets from Mini Challenge 3 of VAST challenge 2013. Evaluation results indicated that the VA system can detect all the labeled intrusions (denial of service, port scanning and data exfiltration) with very few false alerts

Comparative Uncertainty Visualization for High-Level Analysis of Scalar- and Vector-Valued Ensembles

With this thesis, I contribute to the research field of uncertainty visualization, considering parameter dependencies in multi valued fields and the uncertainty of automated data analysis. Like uncertainty visualization in general, both of these fields are becoming more and more important due to increasing computational power, growing importance and availability of complex models and collected data, and progress in artificial intelligence. I contribute in the following application areas: Uncertain Topology of Scalar Field Ensembles. The generalization of topology-based visualizations to multi valued data involves many challenges. An example is the comparative visualization of multiple contour trees, complicated by the random nature of prevalent contour tree layout algorithms. I present a novel approach for the comparative visualization of contour trees - the Fuzzy Contour Tree. Uncertain Topological Features in Time-Dependent Scalar Fields. Tracking features in time-dependent scalar fields is an active field of research, where most approaches rely on the comparison of consecutive time steps. I created a more holistic visualization for time-varying scalar field topology by adapting Fuzzy Contour Trees to the time-dependent setting. Uncertain Trajectories in Vector Field Ensembles. Visitation maps are an intuitive and well-known visualization of uncertain trajectories in vector field ensembles. For large ensembles, visitation maps are not applicable, or only with extensive time requirements. I developed Visitation Graphs, a new representation and data reduction method for vector field ensembles that can be calculated in situ and is an optimal basis for the efficient generation of visitation maps. This is accomplished by bringing forward calculation times to the pre-processing. Visually Supported Anomaly Detection in Cyber Security. Numerous cyber attacks and the increasing complexity of networks and their protection necessitate the application of automated data analysis in cyber security. Due to uncertainty in automated anomaly detection, the results need to be communicated to analysts to ensure appropriate reactions. I introduce a visualization system combining device readings and anomaly detection results: the Security in Process System. To further support analysts I developed an application agnostic framework that supports the integration of knowledge assistance and applied it to the Security in Process System. I present this Knowledge Rocks Framework, its application and the results of evaluations for both, the original and the knowledge assisted Security in Process System. For all presented systems, I provide implementation details, illustrations and applications

Cybercopters Swarm: Immersive analytics for alerts classification based on periodic data

This paper assesses the usefulness of an interactive and navigable 3D environment to help decision-making in cybersecurity. Malware programs frequently emit periodic signals in network logs; however, normal periodical network activities, such as software updates and data collection activities, mask them. Thus, if automatic systems use periodicity to successfully detect malware, they also detect ordinary activities as suspicious ones and raise false positives. Hence, there is a need to provide tools to sort the alerts raised by such software. Data visualizations can make it easier to categorize these alerts, as proven by previous research. However, traditional visualization tools can struggle to display a large amount of data that needs to be treated in cybersecurity in a clear way. In response, this paper explores the use of Immersive Analytics to interact with complex dataset representations and collect cues for alert classification. We created a prototype that uses a helical representation to underline periodicity in the distribution of one variable of a dataset. We tested this prototype in an alert triage scenario and compared it with a state-of-the-art 2D visualization with regard to the visualization efficiency, usability, workload, and flow induced

A Situational Awareness Dashboard for a Security Operations Center

As a result of this dissertation, a solution was developed which would provide visibility into an institution’s security posture and its exposure to risk. Achieving this required the development of a Situational Awareness Dashboard in a cybersecurity context. This Dashboard provides a unified point of view where workers ranging from analysts to members of the executive board can consult and interact with a visual interface that aggregates a set of strategically picked metrics. These metrics provide insight regarding two main topics, the performance and risk of the organization’s Security Operations Center (SOC). The development of the dashboard was performed while working with the multinational enterprise entitled EY. During this time frame, two dashboards were developed one for each of two of EY’s clients inserted in the financial sector. Even though the first solution did not enter production, hence not leaving testing, the dashboard that was developed for the second client successfully was delivered fulfilling the set of objectives that were proposed initially. One of those objectives was enabling the solution to be as autonomous and selfsustained as possible, through its system architecture. Despite having different architectural components, both solutions were based on the same three-layered model. Whereas the first component runs all data ingestion, parsing and transformation operations, the second is in charge of the storage of said information into a database. Finally, the last component, possibly the most important one, is the visualization software tasked with displaying the previous information into actionable intelligence through the power of data visualization. All in all, the key points listed above converged into the development of a Situational Awareness Dashboard which ultimately allows organizations to have visibility into the SOC’s activities, as well as a perception of the performance and associated risks it faces.Como resultado desta dissertação, foi desenvolvida uma solução que proporcionaria visibilidade sobre a postura de segurança de uma instituição e sua exposição ao risco. Para tal foi necessário o desenvolvimento de um Situational Awareness Dashboard num contexto de cibersegurança. Este Dashboard pretende fornecer um ponto de vista unificado onde os trabalhadores, desde analistas a membros do conselho executivo, podem consultar e interagir com uma interface visual que agrega um conjunto de métricas escolhidas estrategicamente. Essas métricas fornecem informações sobre dois tópicos principais, o desempenho e o risco do Security Operations Center (SOC) da organização. O desenvolvimento do Dashboard foi realizado em parceria com a empresa multinacional EY. Nesse período, foram desenvolvidos dois dashboards, um para cada um dos dois clientes da EY inseridos no setor financeiro. Apesar de a primeira solução não ter entrado em produção, não saindo de teste, o painel que foi desenvolvido para o segundo cliente foi entregue com sucesso cumprindo o conjunto de objetivos inicialmente proposto. Umdesses objetivos era permitir que a solução fosse o mais autónoma e auto-sustentável possível, através da sua arquitetura de sistema. Apesar de terem diferentes componentes arquiteturais, ambas as soluções foram baseadas no mesmo modelo de três camadas. Enquanto a primeiro componente executa todas as operações de ingestão, análise e transformação de dados, a segundo é responsável pelo armazenamento dessas informações numa base de dados. Finalmente, o último componente, possivelmente o mais importante, é o software de visualização encarregue em exibir as informações anteriores em inteligência acionável através do poder da visualização de dados. Em suma, os pontos-chave listados acima convergiram no desenvolvimento de um Situational Awareness Dashboard que, em última análise, permite que as organizações tenham visibilidade das atividades do SOC, bem como uma percepção do desempenho e dos riscos que esta enfrenta

Data-Driven Anomaly Detection in Industrial Networks

Since the conception of the first Programmable Logic Controllers (PLCs) in the 1960s, Industrial Control Systems (ICSs) have evolved vastly. From the primitive isolated setups, ICSs have become increasingly interconnected, slowly forming the complex networked environments, collectively known as Industrial Networks (INs), that we know today. Since ICSs are responsible for a wide range of physical processes, including those belonging to Critical Infrastructures (CIs), securing INs is vital for the well-being of modern societies. Out of the many research advances on the field, Anomaly Detection Systems (ADSs) play a prominent role. These systems monitor IN and/or ICS behavior to detect abnormal events, known or unknown. However, as the complexity of INs has increased, monitoring them in the search of anomalous trends has effectively become a Big Data problem. In other words, IN data has become too complex to process it by traditional means, due to its large scale, diversity and generation speeds. Nevertheless, ADSs designed for INs have not evolved at the same pace, and recent proposals are not designed to handle this data complexity, as they do not scale well or do not leverage the majority of the data types created in INs. This thesis aims to fill that gap, by presenting two main contributions: (i) a visual flow monitoring system and (ii) a multivariate ADS that is able to tackle data heterogeneity and to scale efficiently. For the flow monitor, we propose a system that, based on current flow data, builds security visualizations depicting network behavior while highlighting anomalies. For the multivariate ADS, we analyze the performance of Multivariate Statistical Process Control (MSPC) for detecting and diagnosing anomalies, and later we present a Big Data, MSPCinspired ADS that monitors field and network data to detect anomalies. The approaches are experimentally validated by building INs in test environments and analyzing the data created by them. Based on this necessity for conducting IN security research in a rigorous and reproducible environment, we also propose the design of a testbed that serves this purpose

Analytic Provenance for Software Reverse Engineers

Reverse engineering is a time-consuming process essential to software-security tasks such as malware analysis and vulnerability discovery. During the process, an engineer will follow multiple leads to determine how the software functions. The combination of time and possible explanations makes it difficult for the engineers to maintain a context of their findings within the overall task. Analytic provenance tools have demonstrated value in similarly complex fields that require open-ended exploration and hypothesis vetting. However, they have not been explored in the reverse engineering domain. This dissertation presents SensorRE, the first analytic provenance tool designed to support software reverse engineers. A semi-structured interview with experts led to the design and implementation of the system. We describe the visual interfaces and their integration within an existing software analysis tool. SensorRE automatically captures user\u27s sense making actions and provides a graph and storyboard view to support further analysis. User study results with both experts and graduate students demonstrate that SensorRE is easy to use and that it improved the participants\u27 exploration process