Testing SOAR Tools in Use

Modern security operation centers (SOCs) rely on operators and a tapestry of logging and alerting tools with large scale collection and query abilities. SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information and then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a new technology that promise to collect, filter, and display needed data; automate common tasks that require SOC analysts' time; facilitate SOC collaboration; and, improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and understand them in use. In this paper, we design and administer the first hands-on user study of SOAR tools, involving 24 participants and 6 commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools and a methodology for testing them. We describe configuration of the test environment in a cyber range, including network, user, and threat emulation; a full SOC tool suite; and creation of artifacts allowing multiple representative investigation scenarios to permit testing. We present the first research results on SOAR tools. We found that SOAR configuration is critical, as it involves creative design for data display and automation. We found that SOAR tools increased efficiency and reduced context switching during investigations, although ticket accuracy and completeness (indicating investigation quality) decreased with SOAR use. Our findings indicated that user preferences are slightly negatively correlated with their performance with the tool; overautomation was a concern of senior analysts, and SOAR tools that balanced automation with assisting a user to make decisions were preferred

Data Analytics and Machine Learning to Enhance the Operational Visibility and Situation Awareness of Smart Grid High Penetration Photovoltaic Systems

Electric utilities have limited operational visibility and situation awareness over grid-tied distributed photovoltaic systems (PV). This will pose a risk to grid stability when the PV penetration into a given feeder exceeds 60% of its peak or minimum daytime load. Third-party service providers offer only real-time monitoring but not accurate insights into system performance and prediction of productions. PV systems also increase the attack surface of distribution networks since they are not under the direct supervision and control of the utility security analysts. Six key objectives were successfully achieved to enhance PV operational visibility and situation awareness: (1) conceptual cybersecurity frameworks for PV situation awareness at device, communications, applications, and cognitive levels; (2) a unique combinatorial approach using LASSO-Elastic Net regularizations and multilayer perceptron for PV generation forecasting; (3) applying a fixed-point primal dual log-barrier interior point method to expedite AC optimal power flow convergence; (4) adapting big data standards and capability maturity models to PV systems; (5) using K-nearest neighbors and random forests to impute missing values in PV big data; and (6) a hybrid data-model method that takes PV system deration factors and historical data to estimate generation and evaluate system performance using advanced metrics. These objectives were validated on three real-world case studies comprising grid-tied commercial PV systems. The results and conclusions show that the proposed imputation approach improved the accuracy by 91%, the estimation method performed better by 75% and 10% for two PV systems, and the use of the proposed forecasting model improved the generalization performance and reduced the likelihood of overfitting. The application of primal dual log-barrier interior point method improved the convergence of AC optimal power flow by 0.7 and 0.6 times that of the currently used deterministic models. Through the use of advanced performance metrics, it is shown how PV systems of different nameplate capacities installed at different geographical locations can be directly evaluated and compared over both instantaneous as well as extended periods of time. The results of this dissertation will be of particular use to multiple stakeholders of the PV domain including, but not limited to, the utility network and security operation centers, standards working groups, utility equipment, and service providers, data consultants, system integrator, regulators and public service commissions, government bodies, and end-consumers

Operational Decision Making under Uncertainty: Inferential, Sequential, and Adversarial Approaches

Modern security threats are characterized by a stochastic, dynamic, partially observable, and ambiguous operational environment. This dissertation addresses such complex security threats using operations research techniques for decision making under uncertainty in operations planning, analysis, and assessment. First, this research develops a new method for robust queue inference with partially observable, stochastic arrival and departure times, motivated by cybersecurity and terrorism applications. In the dynamic setting, this work develops a new variant of Markov decision processes and an algorithm for robust information collection in dynamic, partially observable and ambiguous environments, with an application to a cybersecurity detection problem. In the adversarial setting, this work presents a new application of counterfactual regret minimization and robust optimization to a multi-domain cyber and air defense problem in a partially observable environment

The AFIT ENgineer, Volume 4, Issue 4

In this issue: Data Analytics and Data Science Programs Faculty Retirements CSRA 10th Anniversary Distance Learning New Hypersonic Flight Certificate Progra

Compilation of Abstracts, June 2016

NPS Class of June 2016This quarter’s Compilation of Abstracts summarizes cutting-edge, security-related research conducted by NPS students and presented as theses, dissertations, and capstone reports. Each expands knowledge in its field.http://archive.org/details/compilationofabs109454990

and Cost/Benefits Opportunities

Acquisition Research Program Sponsored Report SeriesSponsored Acquisition Research & Technical ReportsThe acquisition of artificial intelligence (AI) systems is a relatively new challenge for the U.S. Department of Defense (DoD). Given the potential for high-risk failures of AI system acquisitions, it is critical for the acquisition community to examine new analytical and decision-making approaches to managing the acquisition of these systems in addition to the existing approaches (i.e., Earned Value Management, or EVM). In addition, many of these systems reside in small start-up or relatively immature system development companies, further clouding the acquisition process due to their unique business processes when compared to the large defense contractors. This can lead to limited access to data, information, and processes that are required in the standard DoD acquisition approach (i.e., the 5000 series). The well-known recurring problems in acquiring information technology automation within the DoD will likely be exacerbated in acquiring complex and risky AI systems. Therefore, more robust, agile, and analytically driven acquisition methodologies will be required to help avoid costly disasters in acquiring these kinds of systems. This research provides a set of analytical tools for acquiring organically developed AI systems through a comparison and contrast of the proposed methodologies that will demonstrate when and how each method can be applied to improve the acquisitions lifecycle for AI systems, as well as provide additional insights and examples of how some of these methods can be applied. This research identifies, reviews, and proposes advanced quantitative, analytically based methods within the integrated risk management (IRM)) and knowledge value added (KVA) methodologies to complement the current EVM approach. This research examines whether the various methodologies—EVM, KVA, and IRM—could be used within the Defense Acquisition System (DAS) to improve the acquisition of AI. While this paper does not recommend one of these methodologies over the other, certain methodologies, specifically IRM, may be more beneficial when used throughout the entire acquisition process instead of within a portion of the system. Due to this complexity of AI system, this research looks at AI as a whole and not specific types of AI.Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited

Toward Theoretical Techniques for Measuring the Use of Human Effort in Visual Analytic Systems

Visual analytic systems have long relied on user studies and standard datasets to demonstrate advances to the state of the art, as well as to illustrate the efficiency of solutions to domain-specific challenges. This approach has enabled some important comparisons between systems, but unfortunately the narrow scope required to facilitate these comparisons has prevented many of these lessons from being generalized to new areas. At the same time, advanced visual analytic systems have made increasing use of human-machine collaboration to solve problems not tractable by machine computation alone. To continue to make progress in modeling user tasks in these hybrid visual analytic systems, we must strive to gain insight into what makes certain tasks more complex than others. This will require the development of mechanisms for describing the balance to be struck between machine and human strengths with respect to analytical tasks and workload. In this paper, we argue for the necessity of theoretical tools for reasoning about such balance in visual analytic systems and demonstrate the utility of the Human Oracle Model for this purpose in the context of sensemaking in visual analytics. Additionally, we make use of the Human Oracle Model to guide the development of a new system through a case study in the domain of cybersecurity

Improving Information Alignment and Distributed Coordination for Secure Information Supply Chains

Industries are constantly striving to incorporate the latest technology systems into their operations so that they can maintain a competitive edge in their respective markets. However, even when they are able to stay up to speed with technological advancement, there continues to be a gap between the workforce skill set and available technologies. Organizations may acquire advanced systems, yet end up spending extended periods of time in the implementation and deployment phases, resulting in lost resources and productivity. The primary focus of this research is on streamlining the implementation and integration of new information technology systems to avoid the dire consequences of the process being prolonged or inefficient. Specifically, the goal of this research is to mitigate business challenges in information sharing and availability for employees and managers interacting with business tools and each other. This was accomplished by first interviewing work professionals in order to identify gap parameters. Based on the interview findings, recommendations were made in order to enhance the usability of existing tools. At this point, the research setting was shifted from network operations to supply chain operations due to the restrictive nature of network operations. The research team succeeded in developing a user-centered methodology to implement and deploy new business systems to mitigate risk during integration of new systems as the transition is made from the classic way of performing tasks. While this methodology was studied in supply chain operations, it enabled the identification of a common trend of challenges in operations work settings, regardless of the business application. Hence the findings of this research can be extrapolated to any business setting, besides the ones actually studied by the team. In addition, this research ensures that operational teams are able to maximize their benefit out of the technology available, thus enabling them to keep up with the rapidly evolving world of technology while minimizing sacrifices in resources or productivity in the process

2014 Annual Report of the Graduate School of Engineering and Management, Air Force Institute of Technology

The Graduate School\u27s Annual Report highlights research focus areas, new academic programs, faculty accomplishments and news, and provides top-level sponsor-funded research data and information

Developing a Framwork for Evaluating Organizational Information Assurance Metrics Programs

The push to secure organizational information has brought about the need to develop better metrics for understanding the state of the organization’s security capability. This thesis utilizes case studies of information security metrics programs within Department of Defense organizations, the United States Air Force (USAF), and the National Aeronautics and Space Administration’s (NASA’s) Jet Propulsion Lab to discover how these organizations make decisions about how the measurement program is designed, how information is collected and disseminated, and how the collected information supports decision making. This research finds that both the DOD and USAF have highly complex information security programs that are primarily focused on determining the return for security investments, meeting budget constraints, and achieving mission objectives while NASA’s Jet Propulsion Lab seeks to improve security processes related to compliance. While the analytical techniques were similar in all of the cases, the DOD and USAF use communication processes still based mostly on manual data calls and communications. In contrast, NASA’s JPL information security metrics program employs a more automated approach for information collection and dissemination