Improving and Measuring Learning at Cyber Defence Exercises

Küberõppusi peetakse üheks efektiivseimaks meetodiks erinevate sihtgruppide koolitamisel, see sobib nii (sõjaväelistele) professionaalsetele meeskondadele kui individuaalsetele õpilastele. Samas põhinevad teadmised õppustel saavutatud õpitulemustest peamiselt suulisel infol ja metoodika efektiivsust pole tõestatud. Käesolev töö käsitleb õppimist küberkaitseõppustel ning keskendub õpitulemuste hindamisele. Erinevate õppuste formaatide seast on antud töö aluseks valitud tehnilised küberkaitseõppused, milles on esindatud punaste ja siniste meeskonnad. Töös analüüsitakse kübekaitseõppusi lähtuvalt täiskasvanu õpiteooriatest ja õpitulemuste mõõtmise hetkeolukorda küberkaitseõppuste raamistikus. Õpitulemusi mõõdeti kahel küberkaitseõppusel, Locked Shields ja Crossed Swords. Neist esimene on suurim avalik küberkaitseõppus maailmas peaaegu 900 osalejaga ning peamiseks koolitusgrupiks on siniste meeskonnad. Teine õppus on väiksemahuline punaste meeskonna õppus. Locked Shields ja Crossed Swords on korraldatud NATO küberkaitsekeskuse poolt. Sellised õppused on tehniliselt väga komplekssed ning nii korraldajatele kui osalejatele keerukad. Seetõttu vajavad nii õppuse disain kui õpitulemuste mõõtmine suuremat tähelepanu. Käesolev töö pakub välja uudse ja skaleeritava õpitulemuste mõõtmise metoodika, nn. “5-ajatempli metoodika”. Metoodika hõlmab nii efektiivset tagasisidet (s.h. võrdlusvõimalus) kui õpitulemuste mõõtmist. See võimaldab hinnata meeskondade tegevustulemust, ja väidab, et tulemuste muutus ajas näitab ka õpitulemusi. Ajatempleid saab koguda nii traditsiooniliste meetoditega (nt. intervjuud, vaatlused ja küsimustikud), aga ka potentsiaalselt mitte-intrusiivselt võrgulogidest (nt. pcap’id). Metoodika aitab parandada tagasisidet, tuvastada õppuse disaininõrkusi ja näidata kübekaitseõppuste õpiväärtust. Crossed Swords õppuse hindamisel keskenduti eelkõige osalejatele (punaste meeskond) kohese tagasiside andmisele nende tegevuste kohta. Käesolev töö annab olulise panuse küberkaitseõppuste õpitulemuste hindamise teoreetiliste ja praktiliste aluste kohta ning pakub välja praktilised soovitused õpikogemuse parendamiseks.Cyber security exercises are believed to be the most effective training for all training audiences from top (military) professional teams to individual students. However, evidence of learning outcomes for those exercises are often anecdotal and not validated. This thesis takes a fresh look at learning in Cyber Defence Exercises (CDXs) and focuses on measuring learning outcomes. As such exercises come in a variety of formats, this thesis focuses on technical CDXs with Red and Blue teaming elements. The review of adult learning theories and current state of learning measurement in CDXs context are presented. The learning measurements are performed at two CDXs: Locked Shields and Crossed Swords. First one is the largest unclassified live-fire CDX in the world with nearly 900 participants (with Blue teams as main training audience). Second one is a small scale exercise designed to train Red teams. Both exercises are organised by the NATO Cooperative Cyber Defence Centre of Excellence (CCD COE). Such top-end CDXs are highly complex, which makes it hard for organisers and participants to handle. Therefore, both learning design and measurement need careful consideration. This work proposes a novel and scalable learning measurement methodology, called the “5-timestamp methodology”. This method aims at accommodating for both—effective feedback (including benchmarking opportunity) and learning measurement. The method is capable of assessing team performance, and argues that changes in performance over time equal learning. The timestamps can either be collected using traditional methods, such as interviews, observations and surveys, but also potentially be obtained non-obtrusively from raw network traces (such as pcaps). The method enhances the feedback loop, allows identifying learning design flaws, and provides solid evidence of learning value for CDXs. Crossed Swords measurement focused on providing the training audience (Red team) with instant feedback about their actions to ensure effective learning. This work contributes to theoretical foundations and in practical terms by providing practical recommendations readily applicable for improvement of learning experience in CDXs

PERSONALISING INFORMATION SECURITY EDUCATION

Whilst technological solutions go a long way in providing protection for users online, it has been long understood that the individual also plays a pivotal role. Even with the best of protection, an ill-informed person can effectively remove any protection the control might provide. Information security awareness is therefore imperative to ensure a population is well educated with respect to the threats that exist to one’s electronic information, and how to better protect oneself. Current information security awareness strategies are arguably lacking in their ability to provide a robust and personalised approach to educating users, opting for a blanket, one-size-fits-all solution. This research focuses upon achieving a better understanding of the information security awareness domain; appreciating the requirements such a system would need; and importantly, drawing upon established learning paradigms in seeking to design an effective personalised information security education. A survey was undertaken to better understand how people currently learn about information security. It focussed primarily upon employees of organisations, but also examined the relationship between work and home environments and security practice. The survey also focussed upon understanding how people learn and their preferences for styles of learning. The results established that some good work was being undertaken by organisations in terms of security awareness, and that respondents benefited from such training – both in their workplace and also at home – with a positive relationship between learning at the workplace and practise at home. The survey highlighted one key aspect for both the training provided and the respondents’ preference for learning styles. It varies. It is also clear, that it was difficult to establish the effectiveness of such training and the impact upon practice. The research, after establishing experimentally that personalised learning was a viable approach, proceeded to develop a model for information security awareness that utilised the already successful field of pedagogy and individualised learning. The resulting novel framework “Personalising Information Security Education (PISE)” is proposed. The framework is a holistic approach to solving the problem of information security awareness that can be applied both in the workplace environment and as a tool for the general public. It does not focus upon what is taught, but rather, puts into place the processes to enable an individual to develop their own information security personalised learning plan and to measure their progress through the learning experience.Ministry Of Higher Education Malaysi

DVCL:A Distributed Virtual Computer Lab for Security and Network Education

Teaching networking and IT security in higher education requires a safeplayground for students, where they can safely carry out hands-on exercises.This safe playground is known as a computer lab. Universities have todesign and to provide such a lab with respect to certain criteria, e.g.technical opportunities, educational requirements and demands of thelearners. Since there is no one-size-fits-all lab, the labs will be designed tofit into a certain context and thus have own strengths and weaknesses.In this thesis, we investigate and work with two established labs, whichwere designed for hands-on experiences in networking and it security courses.These labs are predominantly different but have an essential overlap ineducational requirements.One lab is developed by the Open University. It is dedicated for distancelearning. It is based on virtualization and every student is able to startthis lab on his own computer. Students can work out exercises wheneverand wherever they want. A shortcoming however is that students have towork alone, (distant) group work is not possible due to the isolated labarchitecture. This lab is the technical base for our research.The other lab is developed by the Cologne University. It is a physicallab, dedicated for on-campus courses and thus it is not portable. Butstudents can meet in the lab, work in groups and are able to get supportfrom a course advisor, who is also able to verify exercises. A shortcominghowever is that students must be present (they have to travel to theuniversity) and they are dependent on the opening hours of the laboratoryand the availability of the course advisers.In two research parts, we show how such two different lab approachescan be combined and what can be achieved.The first research part is about design issues.Initially, we enable group work in our lab for distance education, sincegroup work is an essential part in on-campus classes. Also remote studentsshould be able to work together. Since the lab is designed as an isolatedsystem, the challenge is to connect two of them on the network level butwithout creating a potential bridge between the isolated and the outsideworld. We achieved this by adding a communication interface to the labarchitecture. This communication interface consists of a ghost host toextract and inject network packets, and a remote bridge endpoint, totransport these packets between remote ghost hosts across an intermediateconnection, e.g. the internet. The developed prototype is called DistributedVirtual Computer Lab (DVCL) and enables to connect two or more distantlabs while preserving the isolated character.The DVCL is then extended and improved by a central authority (CA).While the point-to-point connection of the communication interface canconnect two remote networks in a handy way, more connections requirecareful planning by the students. We show that a CA simplifies the usageof our DVCL for the students (and also for academic staff) and in additionto it avoids administrative configuration errors while connecting remotelabs, e.g. a circular flow which leads to an unusable lab.The first part is completed by two applicability enhancements. Thefirst enhancement covers and resolves security issues in order to pushour prototypical implementation of the DVCL and the CA closer to aproductive learning environment. The second enhancement introduces aGraphical User Interface to increase the usability of the DVCL.The second research part is about educational aspects.In the first part, we assume that working independent from a physicalon-campus lab as well as group work is essential for our students. Ourevaluation of more than 200 students participating in an on-campus networkingcourse shows, that nearly half of the students actually say, thatthey would like to work independently from the university at least partiallyand they would welcome the introduction of an e-learning system. Inaddition, a predominant majority think of working in groups as well asreceiving guidance and feedback as crucial to their learning success. Thisresult justifies and confirms our research and also reveals an additional requirement.The challenge is to provide feedback and guidance to a student, who isworking on an exercise and a human course advisor is not available. Thisis e.g. when students use the DVCL at home in the evening hours. Weshow, that captured network traffic of a lab can give some indication ofwhat a student has already configured according to a certain exercise. Weuse this insight to develop an Electronic Exercise Assistant. This softwareprogram is able to recognize the progress of an exercise and can provideappropriate feedback and support, based on preloaded rules and conditions.This significantly improves the learning situation for students workingremotely in lab. Besides this automatic support, the exercise assistant canverify intermediate and complete solutions of an exercise.The second part is completed by an educational enhancement. Ourevaluation and also own observations show, that a lab is more than aroom with computer and network facilities. Rather it is a social placewhere students e.g. meet, form learning groups, talk and discuss. Weuse these insights and enhance the DVCL to support social interactions.Based on our on-campus lab as source, we model a set of communicational,organizational as well as educational activities and implement them in ourDVCL. The result shows, that our DVCL prototype is no longer a technicalplatform but a virtual place, where students can meet, communicate,arrange learning groups, exchange experiences and work on exercises.This thesis shows that aspects of our two different lab environments canbe combined. Our resulting Distributed Virtual Computer Lab incorporatesstrengths of each source lab. It is a gain for distance teaching as well as foron-campus classes. Remote students are now able to utilize the lab being avirtual classroom, where they can learn in groups, assisted by an electronicadvisor and without the need for a face-to-face meeting. On-campus classescan offer students a new learning environment, where they can learn in aclassroom character without the need to travel to the university