3 research outputs found
Forming IDEAS Interactive Data Exploration & Analysis System
Modern cyber security operations collect an enormous amount of logging and
alerting data. While analysts have the ability to query and compute simple
statistics and plots from their data, current analytical tools are too simple
to admit deep understanding. To detect advanced and novel attacks, analysts
turn to manual investigations. While commonplace, current investigations are
time-consuming, intuition-based, and proving insufficient. Our hypothesis is
that arming the analyst with easy-to-use data science tools will increase their
work efficiency, provide them with the ability to resolve hypotheses with
scientific inquiry of their data, and support their decisions with evidence
over intuition. To this end, we present our work to build IDEAS (Interactive
Data Exploration and Analysis System). We present three real-world use-cases
that drive the system design from the algorithmic capabilities to the user
interface. Finally, a modular and scalable software architecture is discussed
along with plans for our pilot deployment with a security operation command.Comment: 4 page short paper on IDEAS System, 4 figure
How do information security workers use host data? A summary of interviews with security analysts
Modern security operations centers (SOCs) employ a variety of tools for
intrusion detection, prevention, and widespread log aggregation and analysis.
While research efforts are quickly proposing novel algorithms and technologies
for cyber security, access to actual security personnel, their data, and their
problems are necessarily limited by security concerns and time constraints. To
help bridge the gap between researchers and security centers, this paper
reports results of semi-structured interviews of 13 professionals from five
different SOCs including at least one large academic, research, and government
organization. The interviews focused on the current practices and future
desires of SOC operators about host-based data collection capabilities, what is
learned from the data, what tools are used, and how tools are evaluated.
Questions and the responses are organized and reported by topic. Then broader
themes are discussed. Forest-level takeaways from the interviews center on
problems stemming from size of data, correlation of heterogeneous but related
data sources, signal-to-noise ratio of data, and analysts' time.Comment: interviews with 13 security analysts about host data, tools, desire
An Assessment of the Usability of Machine Learning Based Tools for the Security Operations Center
Gartner, a large research and advisory company, anticipates that by 2024 80%
of security operation centers (SOCs) will use machine learning (ML) based
solutions to enhance their operations. In light of such widespread adoption, it
is vital for the research community to identify and address usability concerns.
This work presents the results of the first in situ usability assessment of
ML-based tools. With the support of the US Navy, we leveraged the national
cyber range, a large, air-gapped cyber testbed equipped with state-of-the-art
network and user emulation capabilities, to study six US Naval SOC analysts'
usage of two tools. Our analysis identified several serious usability issues,
including multiple violations of established usability heuristics form user
interface design. We also discovered that analysts lacked a clear mental model
of how these tools generate scores, resulting in mistrust and/or misuse of the
tools themselves. Surprisingly, we found no correlation between analysts' level
of education or years of experience and their performance with either tool,
suggesting that other factors such as prior background knowledge or personality
play a significant role in ML-based tool usage. Our findings demonstrate that
ML-based security tool vendors must put a renewed focus on working with
analysts, both experienced and inexperienced, to ensure that their systems are
usable and useful in real-world security operations settings