Establishing cyber situational awareness in industrial control systems

The cyber threat to industrial control systems is an acknowledged security issue, but a qualified dataset to quantify the risk remains largely unavailable. Senior executives of facilities that operate these systems face competing requirements for investment budgets, but without an understanding of the nature of the threat cyber security may not be a high priority. Operational managers and cyber incident responders at these facilities face a similarly complex situation. They must plan for the defence of critical systems, often unfamiliar to IT security professionals, from potentially capable, adaptable and covert antagonists who will actively attempt to evade detection. The scope of the challenge requires a coherent, enterprise-level awareness of the threat, such that organisations can assess their operational priorities, plan their defensive posture, and rehearse their responses prior to such an attack. This thesis proposes a novel combination of concepts found in risk assessment, intrusion detection, education, exercising, safety and process models, fused with experiential learning through serious games. It progressively builds a common set of shared mental models across an ICS operation to frame the nature of the adversary and establish enterprise situational awareness that permeates through all levels of teams involved in addressing the threat. This is underpinned by a set of coping strategies that identifies probable targets for advanced threat actors, proactively determining antagonistic courses of actions to derive an appropriate response strategy

Conceptual Model of Visual Analytics for Hands-on Cybersecurity Training

Hands-on training is an effective way to practice theoretical cybersecurity concepts and increase participants’ skills. In this paper, we discuss the application of visual analytics principles to the design, execution, and evaluation of training sessions. We propose a conceptual model employing visual analytics that supports the sensemaking activities of users involved in various phases of the training life cycle. The model emerged from our long-term experience in designing and organizing diverse hands-on cybersecurity training sessions. It provides a classification of visualizations and can be used as a framework for developing novel visualization tools supporting phases of the training life-cycle. We demonstrate the model application on examples covering two types of cybersecurity training programs

Improving and Measuring Learning at Cyber Defence Exercises

Küberõppusi peetakse üheks efektiivseimaks meetodiks erinevate sihtgruppide koolitamisel, see sobib nii (sõjaväelistele) professionaalsetele meeskondadele kui individuaalsetele õpilastele. Samas põhinevad teadmised õppustel saavutatud õpitulemustest peamiselt suulisel infol ja metoodika efektiivsust pole tõestatud. Käesolev töö käsitleb õppimist küberkaitseõppustel ning keskendub õpitulemuste hindamisele. Erinevate õppuste formaatide seast on antud töö aluseks valitud tehnilised küberkaitseõppused, milles on esindatud punaste ja siniste meeskonnad. Töös analüüsitakse kübekaitseõppusi lähtuvalt täiskasvanu õpiteooriatest ja õpitulemuste mõõtmise hetkeolukorda küberkaitseõppuste raamistikus. Õpitulemusi mõõdeti kahel küberkaitseõppusel, Locked Shields ja Crossed Swords. Neist esimene on suurim avalik küberkaitseõppus maailmas peaaegu 900 osalejaga ning peamiseks koolitusgrupiks on siniste meeskonnad. Teine õppus on väiksemahuline punaste meeskonna õppus. Locked Shields ja Crossed Swords on korraldatud NATO küberkaitsekeskuse poolt. Sellised õppused on tehniliselt väga komplekssed ning nii korraldajatele kui osalejatele keerukad. Seetõttu vajavad nii õppuse disain kui õpitulemuste mõõtmine suuremat tähelepanu. Käesolev töö pakub välja uudse ja skaleeritava õpitulemuste mõõtmise metoodika, nn. “5-ajatempli metoodika”. Metoodika hõlmab nii efektiivset tagasisidet (s.h. võrdlusvõimalus) kui õpitulemuste mõõtmist. See võimaldab hinnata meeskondade tegevustulemust, ja väidab, et tulemuste muutus ajas näitab ka õpitulemusi. Ajatempleid saab koguda nii traditsiooniliste meetoditega (nt. intervjuud, vaatlused ja küsimustikud), aga ka potentsiaalselt mitte-intrusiivselt võrgulogidest (nt. pcap’id). Metoodika aitab parandada tagasisidet, tuvastada õppuse disaininõrkusi ja näidata kübekaitseõppuste õpiväärtust. Crossed Swords õppuse hindamisel keskenduti eelkõige osalejatele (punaste meeskond) kohese tagasiside andmisele nende tegevuste kohta. Käesolev töö annab olulise panuse küberkaitseõppuste õpitulemuste hindamise teoreetiliste ja praktiliste aluste kohta ning pakub välja praktilised soovitused õpikogemuse parendamiseks.Cyber security exercises are believed to be the most effective training for all training audiences from top (military) professional teams to individual students. However, evidence of learning outcomes for those exercises are often anecdotal and not validated. This thesis takes a fresh look at learning in Cyber Defence Exercises (CDXs) and focuses on measuring learning outcomes. As such exercises come in a variety of formats, this thesis focuses on technical CDXs with Red and Blue teaming elements. The review of adult learning theories and current state of learning measurement in CDXs context are presented. The learning measurements are performed at two CDXs: Locked Shields and Crossed Swords. First one is the largest unclassified live-fire CDX in the world with nearly 900 participants (with Blue teams as main training audience). Second one is a small scale exercise designed to train Red teams. Both exercises are organised by the NATO Cooperative Cyber Defence Centre of Excellence (CCD COE). Such top-end CDXs are highly complex, which makes it hard for organisers and participants to handle. Therefore, both learning design and measurement need careful consideration. This work proposes a novel and scalable learning measurement methodology, called the “5-timestamp methodology”. This method aims at accommodating for both—effective feedback (including benchmarking opportunity) and learning measurement. The method is capable of assessing team performance, and argues that changes in performance over time equal learning. The timestamps can either be collected using traditional methods, such as interviews, observations and surveys, but also potentially be obtained non-obtrusively from raw network traces (such as pcaps). The method enhances the feedback loop, allows identifying learning design flaws, and provides solid evidence of learning value for CDXs. Crossed Swords measurement focused on providing the training audience (Red team) with instant feedback about their actions to ensure effective learning. This work contributes to theoretical foundations and in practical terms by providing practical recommendations readily applicable for improvement of learning experience in CDXs

Cyber Operator Competencies: The Role of Cognitive Competencies in Cyber Operator Practice and Education

PhD Dissertations in Child and Youth Participation and Competence Development (BUK): 17. Articles 2, 3 and 4 have been removed from the digital thesis due to lack of permission from the publishers. These can be viewed in the relevant journals/books, and in the printed thesis.The theme of this thesis is the role of cognitive competencies in cyber operator practice and education. Cyber operator practice is a new field of research where the importance and attention is growing rapidly. Research has accumulated a solid amount of knowledge about the technical skills required by a cyber operator. However, less is known about the cognitive competencies that support cyber operator proficiency. In order to gain insight into the cognitive demands of cyber operators, the cognitions of young cyber officers(1) attending the Norwegian Defence Cyber Academy have been studied. Findings contributes to the development of theory and evidence-based knowledge needed to develop educational guidelines for the cyber operator workforce. This dissertation proposes and take steps towards validation of a conceptual framework, The Hybrid Space, that describes the cognitive work environment of military cyber operators. The Hybrid Space conceptual framework is introduced in the first article of this thesis and is used in all parts of the study. Methodological contributions include a method and a software to collect quantitative data on cyber operators’ cognitive focus and assess cognitive agility. Cognitive agility is proposed as a competence and a measure of cyber operator performance. Empirical data collected during a cyber defence exercise support our theoretical assumption and helps to further develop The Hybrid Space conceptual framework. Findings indicate that knowledge and understanding of cyberspace as a domain of operations and the cognitive competencies supporting cyber operator proficiency are limited. Cognitive agility is proposed as a cognitive competency and is associated with higher levels of selfregulation. These findings suggest that cognitive competencies can indeed support cyber operator performance. This thesis therefore contributes to cyber operator practice and education by suggesting that education and training would benefit from including the development of cognitive competencies alongside the technical education and training needed to become a cyber operator. In this way, this thesis adds new insight and perspective into the novel area of cyber operator practice. The results provide the first indications that cyber operator performance can be supported by the development of cognitive competencies during education. 1 Cyber officer and cyber operator are used interchangeably throughout the articles and this extended abstract. The reason is that the students undergo the same education, but the position they later get determine their career path and the accompanying title. The use of the terms is maturing in both military and civilian sectors. As of now neither finite guidelines nor agreed upon norms exist that guide the use of the titlesSammendrag Temaet for denne doktoravhandlingen er rollen til kognitive kompetanser i cyber operator praksis og utdanning. Cyber operator praksis er et nytt forskningsfelt som har fatt stor oppmerksomhet de siste arene. Forskning pa omradet har produsert kunnskap om hvilke tekniske kunnskaper og ferdigheter en cyber operator ma ha. Mindre kunnskap finnes om de kognitive kompetansene som en cyber operator trenger for a kunne utove sin praksis effektivt. For a fa bedre innsikt i de kognitive kravene som cyber operatorer stilles ovenfor har jeg studert unge cyber offiserer under utdanning pa Forsvarets Ingeniorhogskole (2) (FIH). Denne avhandlingen bidrar med kunnskap og empirisk grunnlag for a utvikle forskningsbasert utdanning for fremtidens cyber operatorer. Avhandlingen fremholder og starter validering et konseptuelt rammeverk, The Hybrid Space, som beskriver de kognitive kravene militare cyber operatorer ma forholde seg til i utovelsen av sitt virke. Rammeverket blir introdusert i forste artikkel av denne avhandlingen og blir brukt som konseptuelt fundament i resten av avhandlingen. Avhandlingen fremlegger ogsa en metode og et dataverktoy som kan brukes til a samle inn kvantitative data om cyber operatorers kognitive fokus. Dette dataverktoyet kan ogsa benyttes til a undersoke hvordan cyber operatorer utviser kognitiv fleksibilitet over tid nar de gjennomforer en cyber operasjon. Kognitiv fleksibilitet foreslas som et prestasjonsmal for cyber operatorer. Empiriske data innhentet under en cyberforsvars ovelse bekrefter vare teoretiske hypoteser og bidrar til videre utvikling av det konseptuelle rammeverket. Hovedfunnene indikerer at kunnskap om og forstaelse for cyberspace som operasjonsdomene og rollen til kognitive kompetanser i cyber operatorens utforelse av cyber operasjoner er begrenset. Denne avhandlingen argumenter for at evne til fleksibel kognitiv manover i operasjonsmiljoet, definert som ‘cognitive agility’, er en viktig kognitiv kompetanse for cyber operatorer som kan predikeres ved a undersoke evne til selvregulering. Disse funnene indikerer at kognitive kompetanser kan bidra til a understotte cyber operatorers prestasjon. Avhandlingen bidrar til cyber operator praksis og utdanning ved a vise til at utvikling av cyber operator kompetanse bor inkludere utvikling av kognitive kompetanser i tillegg til utvikling av tekniske kunnskaper og ferdigheter. Med disse funnene bidrar denne avhandlingen bidrar til ny innsikt og perspektiv pa cyber operator praksis og utdanning. 2 Forsvarets Ingeniørhøgskole (FIH) endret i 2018 navn til Cyberingeniørskolen (CIS) og ble samtidig underlagt Forsvarets Høgskole (FHS)

AiCEF: An AI-assisted Cyber Exercise Content Generation Framework Using Named Entity Recognition

Content generation that is both relevant and up to date with the current threats of the target audience is a critical element in the success of any Cyber Security Exercise (CSE). Through this work, we explore the results of applying machine learning techniques to unstructured information sources to generate structured CSE content. The corpus of our work is a large dataset of publicly available cyber security articles that have been used to predict future threats and to form the skeleton for new exercise scenarios. Machine learning techniques, like named entity recognition (NER) and topic extraction, have been utilised to structure the information based on a novel ontology we developed, named Cyber Exercise Scenario Ontology (CESO). Moreover, we used clustering with outliers to classify the generated extracted data into objects of our ontology. Graph comparison methodologies were used to match generated scenario fragments to known threat actors' tactics and help enrich the proposed scenario accordingly with the help of synthetic text generators. CESO has also been chosen as the prominent way to express both fragments and the final proposed scenario content by our AI-assisted Cyber Exercise Framework (AiCEF). Our methodology was put to test by providing a set of generated scenarios for evaluation to a group of experts to be used as part of a real-world awareness tabletop exercise

Øving på cybersikkerheit: Ein casestudie av ei cybersikkerheitsøving

Denne artikkelen presenterer ein casestudie av ei cybersikkerheitsøving i militær utdanning, og nyttar denne casestudien til å drøfte nokre utfordringar med cybersikkerheitsøvingar til utdanningsføremål. Casestudien gjer greie for sentrale avgjerder i designet av øvinga, evaluering av øvinga og utfordringar i øvingskonseptet. Gjennom ein litteraturgjennomgang samanliknar vi øvinga med liknande øvingar, og ser på korleis desse øvingane har blitt evaluert. Avslutningsvis nyttar vi casestudien og litteraturgjennomgangen til å gjere betraktningar om vidare undersøkingar av cybersikkerheitsøvingar.Øving på cybersikkerheit: Ein casestudie av ei cybersikkerheitsøvingpublishedVersio

Cyber Ranges and TestBeds for Education, Training, and Research

In recent years, there has been a growing demand for cybersecurity experts, and, according to predictions, this demand will continue to increase. Cyber Ranges can fill this gap by combining hands-on experience with educational courses, and conducting cybersecurity competitions. In this paper, we conduct a systematic survey of ten Cyber Ranges that were developed in the last decade, with a structured interview. The purpose of the interview is to find details about essential components, and especially the tools used to design, create, implement and operate a Cyber Range platform, and to present the findings

A cyber exercise post assessment framework: In Malaysia perspectives

Critical infrastructures are based on complex systems that provide vital services to the nation. The complexities of the interconnected networks, each managed by individual organisations, if not properly secured, could offer vulnerabilities that threaten other organisations’ systems that depend on their services. This thesis argues that the awareness of interdependencies among critical sectors needs to be increased. Managing and securing critical infrastructure is not isolated responsibility of a government or an individual organisation. There is a need for a strong collaboration among critical service providers of public and private organisations in protecting critical information infrastructure. Cyber exercises have been incorporated in national cyber security strategies as part of critical information infrastructure protection. However, organising a cyber exercise involved multi sectors is challenging due to the diversity of participants’ background, working environments and incidents response policies. How well the lessons learned from the cyber exercise and how it can be transferred to the participating organisations is still a looming question. In order to understand the implications of cyber exercises on what participants have learnt and how it benefits participants’ organisation, a Cyber Exercise Post Assessment (CEPA) framework was proposed in this research. The CEPA framework consists of two parts. The first part aims to investigate the lessons learnt by participants from a cyber exercise using the four levels of the Kirkpatrick Training Model to identify their perceptions on reaction, learning, behaviour and results of the exercise. The second part investigates the Organisation Cyber Resilience (OCR) of participating sectors. The framework was used to study the impact of the cyber exercise called X Maya in Malaysia. Data collected through interviews with X Maya 5 participants were coded and categorised based on four levels according to the Kirkpatrick Training Model, while online surveys distributed to ten Critical National Information Infrastructure (CNII) sectors participated in the exercise. The survey used the C-Suite Executive Checklist developed by World Economic Forum in 2012. To ensure the suitability of the tool used to investigate the OCR, a reliability test conducted on the survey items showed high internal consistency results. Finally, individual OCR scores were used to develop the OCR Maturity Model to provide the organisation cyber resilience perspectives of the ten CNII sectors

Towards a capability maturity model for a cyber range

This work describes research undertaken towards the development of a Capability Maturity Model (CMM) for Cyber Ranges (CRs) focused on cyber security. Global cyber security needs are on the rise, and the need for attribution within the cyber domain is of particular concern. This has prompted major efforts to enhance cyber capabilities within organisations to increase their total cyber resilience posture. These efforts include, but are not limited to, the testing of computational devices, networks, and applications, and cyber skills training focused on prevention, detection and cyber attack response. A cyber range allows for the testing of the computational environment. By developing cyber events within a confined virtual or sand-boxed cyber environment, a cyber range can prepare the next generation of cyber security specialists to handle a variety of potential cyber attacks. Cyber ranges have different purposes, each designed to fulfil a different computational testing and cyber training goal; consequently, cyber ranges can vary greatly in the level of variety, capability, maturity and complexity. As cyber ranges proliferate and become more and more valued as tools for cyber security, a method to classify or rate them becomes essential. Yet while a universal criteria for measuring cyber ranges in terms of their capability maturity levels becomes more critical, there are currently very limited resources for researchers aiming to perform this kind of work. For this reason, this work proposes and describes a CMM, designed to give organisations the ability to benchmark the capability maturity of a given cyber range. This research adopted a synthesised approach to the development of a CMM, grounded in prior research and focused on the production of a conceptual model that provides a useful level of abstraction. In order to achieve this goal, the core capability elements of a cyber range are defined with their relative importance, allowing for the development of a proposed classification cyber range levels. An analysis of data gathered during the course of an expert review, together with other research, further supported the development of the conceptual model. In the context of cyber range capability, classification will include the ability of the cyber range to perform its functions optimally with different core capability elements, focusing on the Measurement of Capability (MoC) with its elements, namely effect, performance and threat ability. Cyber range maturity can evolve over time and can be defined through the Measurement of Maturity (MoM) with its elements, namely people, processes, technology. The combination of these measurements utilising the CMM for a CR determines the capability maturity level of a CR. The primary outcome of this research is the proposed level-based CMM framework for a cyber range, developed using adopted and synthesised CMMs, the analysis of an expert review, and the mapping of the results