Are Intrusion Detection Studies Evaluated Consistently? A Systematic Literature Review

Cyberinfrastructure is increasingly becoming target of a wide spectrum of attacks from Denial of Service to large-scale defacement of the digital presence of an organization. Intrusion Detection System (IDSs) provide administrators a defensive edge over intruders lodging such malicious attacks. However, with the sheer number of different IDSs available, one has to objectively assess the capabilities of different IDSs to select an IDS that meets specific organizational requirements. A prerequisite to enable such an objective assessment is the implicit comparability of IDS literature. In this study, we review IDS literature to understand the implicit comparability of IDS literature from the perspective of metrics used in the empirical evaluation of the IDS. We identified 22 metrics commonly used in the empirical evaluation of IDS and constructed search terms to retrieve papers that mention the metric. We manually reviewed a sample of 495 papers and found 159 of them to be relevant. We then estimated the number of relevant papers in the entire set of papers retrieved from IEEE. We found that, in the evaluation of IDSs, multiple different metrics are used and the trade-off between metrics is rarely considered. In a retrospective analysis of the IDS literature, we found the the evaluation criteria has been improving over time, albeit marginally. The inconsistencies in the use of evaluation metrics may not enable direct comparison of one IDS to another

Control Behavior Integrity for Distributed Cyber-Physical Systems

Cyber-physical control systems, such as industrial control systems (ICS), are increasingly targeted by cyberattacks. Such attacks can potentially cause tremendous damage, affect critical infrastructure or even jeopardize human life when the system does not behave as intended. Cyberattacks, however, are not new and decades of security research have developed plenty of solutions to thwart them. Unfortunately, many of these solutions cannot be easily applied to safety-critical cyber-physical systems. Further, the attack surface of ICS is quite different from what can be commonly assumed in classical IT systems. We present Scadman, a system with the goal to preserve the Control Behavior Integrity (CBI) of distributed cyber-physical systems. By observing the system-wide behavior, the correctness of individual controllers in the system can be verified. This allows Scadman to detect a wide range of attacks against controllers, like programmable logic controller (PLCs), including malware attacks, code-reuse and data-only attacks. We implemented and evaluated Scadman based on a real-world water treatment testbed for research and training on ICS security. Our results show that we can detect a wide range of attacks--including attacks that have previously been undetectable by typical state estimation techniques--while causing no false-positive warning for nominal threshold values.Comment: 15 pages, 8 figure

TEDDI: Tamper Event Detection on Distributed Cyber-Physical Systems

Edge devices, or embedded devices installed along the periphery of a power grid SCADA network, pose a significant threat to the grid, as they give attackers a convenient entry point to access and cause damage to other essential equipment in substations and control centers. Grid defenders would like to protect these edge devices from being accessed and tampered with, but they are hindered by the grid defender\u27s dilemma; more specifically, the range and nature of tamper events faced by the grid (particularly distributed events), the prioritization of grid availability, the high costs of improper responses, and the resource constraints of both grid networks and the defenders that run them makes prior work in the tamper and intrusion protection fields infeasible to apply. In this thesis, we give a detailed description of the grid defender\u27s dilemma, and introduce TEDDI (Tamper Event Detection on Distributed Infrastructure), a distributed, sensor-based tamper protection system built to solve this dilemma. TEDDI\u27s distributed architecture and use of a factor graph fusion algorithm gives grid defenders the power to detect and differentiate between tamper events, and also gives defenders the flexibility to tailor specific responses for each event. We also propose the TEDDI Generation Tool, which allows us to capture the defender\u27s intuition about tamper events, and assists defenders in constructing a custom TEDDI system for their network. To evaluate TEDDI, we collected and constructed twelve different tamper scenarios, and show how TEDDI can detect all of these events and solve the grid defender\u27s dilemma. In our experiments, TEDDI demonstrated an event detection accuracy level of over 99% at both the information and decision point levels, and could process a 99-node factor graph in under 233 microseconds. We also analyzed the time and resources needed to use TEDDI, and show how it requires less up-front configuration effort than current tamper protection solutions