An Empirical Assessment of the Effectiveness of Deception for Cyber Defense

The threat of cyber attacks is a growing concern across the world, leading to an increasing need for sophisticated cyber defense techniques. The Tularosa Study, was designed and conducted to understand how defensive deception, both cyber and psychological, affects cyber attackers Ferguson-Walter et al. [2019c]. More specifically, for this empirical study, cyber deception refers to a decoy system and psychological deception refers to false information of the presence of defensive deception techniques on the network. Over 130 red teamers participated in a network penetration test over two days in which we controlled both the presence of and explicit mention of deceptive defensive techniques. To our knowledge, this represents the largest study of its kind ever conducted on a skilled red team population. In addition to the abundant host and network data collected, we conducted a battery of questionnaires, e.g., experience, personality; and cognitive tasks, e.g., fluid intelligence, working memory; as well as physiological measures, e.g., galvanic skin response (GSR), heart rate, to be correlated with the cyber events at a later date. The design and execution of this study and the lessons learned are a major contribution of this thesis. I investigate the effectiveness of decoy systems for cyber defense by comparing performance across all experimental conditions. Results support a new finding that the combination of the presence of deception and the true information that deception is present has the greatest effect on cyber attackers, when compared to a control condition in which no deception was used. Evidence of cognitive biases in the red teamers’ behavior is then detailed and explained, to further support our theory of oppositional human factors (OHF). The final chapter discusses how elements of the experimental design contribute to the validity of assessing the effectiveness of cyber deception and reviews trade-offs and lessons learned

Responsible Integration of Behavioral Science in Computer Science Research and Development

Cross disciplinary research is essential for technological innovation. For decades, computer science (Comp Sci) has leveraged behavior science (Behav Sci) research to create innovative products and improve end user experience. Despite the natural challenges that come with cross disciplinary work, there are no published manuscripts outlining how to responsibly integrate Behav Sci into Comp Sci research and development. This publication fills this critical gap by discussing important differences between Behav Sci and Comp Sci, particularly with regard to how each field fits under the umbrella of science and how each field conceptualizes data. We then discuss the consequences of misusing Behav Sci and provide examples of technology efforts that drew inappropriate or unethical conclusions about their behavioral data. We discuss in detail common errors to avoid at each stage of the research process, which we condensed into a useful checklist to use as a tool for teams integrating Behav Sci in their work. Finally, we include examples of good applications of Behav Sci into Comp Sci research, the design of which can inform and strengthen digital government, e-commerce, defense, and many other areas of information technology

The Moonraker Study: An Experimental Evaluation of Host-Based Deception

Cyber deception has been discussed as providing enhanced cyber defense. This human subjects research, one of the first rigorously controlled studies on this topic, found that host-based deception was effective at preventing completion of a specific exfiltration task against a virtual network. In addition to impeding progress and preventing success, the deception resulted in increased confusion and surprise in the participants. This study provided the necessary rigor to scientifically attest to the effectiveness of cyber deception for cyber defense with computer specialists

Emotional State Classification and Related Behaviors Among Cyber Attackers

Cyber deception is a strategy that defenders can leverage to gain an advantage over cyber attackers. The effects of deception on the attacker however, are not yet well understood. Quantifying the tangible and emotional effects of deception on the attacker’s performance, beliefs, and emotional state are critical to deploying effective, targeted cyber deception. Our work uses data from a human-subjects experiment measuring the impact of cyber and psychological deception on over 100 professional red-teamers. These results demonstrate that an attacker’s cognitive and emotional state can often be inferred from data already observed and collected by cyber defenders world-wide. Future work will leverage this observed data-set to formulate more informed defensive strategies

The Tularosa Study: An Experimental Design and Implementation to Quantify the Effectiveness of Cyber Deception

The Tularosa study was designed to understand how defensive deception--including both cyber and psychological--affects cyber attackers. Over 130 red teamers participated in a network penetration task over two days in which we controlled both the presence of and explicit mention of deceptive defensive techniques. To our knowledge, this represents the largest study of its kind ever conducted on a professional red team population. The design was conducted with a battery of questionnaires (e.g., experience, personality, etc.) and cognitive tasks (e.g., fluid intelligence, working memory, etc.), allowing for the characterization of a ``typical\u27\u27 red teamer, as well as physiological measures (e.g., galvanic skin response, heart rate, etc.) to be correlated with the cyber events. This paper focuses on the design, implementation, data, population characteristics, and begins to examine preliminary results

The James Webb Space Telescope Mission

Twenty-six years ago a small committee report, building on earlier studies, expounded a compelling and poetic vision for the future of astronomy, calling for an infrared-optimized space telescope with an aperture of at least $4m$. With the support of their governments in the US, Europe, and Canada, 20,000 people realized that vision as the $6.5m$ James Webb Space Telescope. A generation of astronomers will celebrate their accomplishments for the life of the mission, potentially as long as 20 years, and beyond. This report and the scientific discoveries that follow are extended thank-you notes to the 20,000 team members. The telescope is working perfectly, with much better image quality than expected. In this and accompanying papers, we give a brief history, describe the observatory, outline its objectives and current observing program, and discuss the inventions and people who made it possible. We cite detailed reports on the design and the measured performance on orbit.Comment: Accepted by PASP for the special issue on The James Webb Space Telescope Overview, 29 pages, 4 figure

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe