ROMEO: Exploring Juliet through the Lens of Assembly Language

Automatic vulnerability detection on C/C++ source code has benefitted from the introduction of machine learning to the field, with many recent publications considering this combination. In contrast, assembly language or machine code artifacts receive little attention, although there are compelling reasons to study them. They are more representative of what is executed, more easily incorporated in dynamic analysis and in the case of closed-source code, there is no alternative. We propose ROMEO, a publicly available, reproducible and reusable binary vulnerability detection benchmark dataset derived from the Juliet test suite. Alongside, we introduce a simple text-based assembly language representation that includes context for function-spanning vulnerability detection and semantics to detect high-level vulnerabilities. Finally, we show that this representation, combined with an off-the-shelf classifier, compares favorably to state-of-the-art methods, including those operating on the full C/C++ code.Comment: 6 pages, code available at https://gitlab.com/dlr-dw/rome

Cross-Domain Evaluation of a Deep Learning-Based Type Inference System

Optional type annotations allow for enriching dynamic programming languages with static typing features like better Integrated Development Environment (IDE) support, more precise program analysis, and early detection and prevention of type-related runtime errors. Machine learning-based type inference promises interesting results for automating this task. However, the practical usage of such systems depends on their ability to generalize across different domains, as they are often applied outside their training domain. In this work, we investigate Type4Py as a representative of state-of-the-art deep learning-based type inference systems, by conducting extensive cross-domain experiments. Thereby, we address the following problems: class imbalances, out-of-vocabulary words, dataset shifts, and unknown classes. To perform such experiments, we use the datasets ManyTypes4Py and CrossDomainTypes4Py. The latter we introduce in this paper. Our dataset enables the evaluation of type inference systems in different domains of software projects and has over 1,000,000 type annotations mined on the platforms GitHub and Libraries. It consists of data from the two domains web development and scientific calculation. Through our experiments, we detect that the shifts in the dataset and the long-tailed distribution with many rare and unknown data types decrease the performance of the deep learning-based type inference system drastically. In this context, we test unsupervised domain adaptation methods and fine-tuning to overcome these issues. Moreover, we investigate the impact of out-of-vocabulary words.Comment: Preprint for the MSR'23 technical trac

Generalizability of Code Clone Detection on CodeBERT

Transformer networks such as CodeBERT already achieve very good results for code clone detection in benchmark datasets, so one could assume that this task has already been solved. However, code clone detection is not a trivial task. Semantic code clones in particular are difficult to detect. We show that the generalizability of CodeBERT decreases by evaluating two different subsets of Java code clones from BigCloneBench. We observe a significant drop of F1 score when we evaluate different code snippets and different functionality IDs than those used for model building

User-agent as a Cyber Intrusion Artifact: Detection of APT Activity using minimal Anomalies on the User-agent String Traffic

The detection of attacks, especially persistent intrusions, relies on a combination of various artifacts. Despite being manipulable, the user-agent string, a component of HTTP headers, has proven to be a tool for triggering alerts, thereby enhancing detection capabilities. In this paper, we perform a review and analysis of existing malicious user agent strings. We gather relevant data from different sources of threat intelligence and present a dataset of user-agent strings associated with malicious activities gathered from real incident reports. We also propose a categorization of existing user-agent string anomalies with respect to their type (e.g., syntax) and their complexity degre

Machine Learning Applications in Secure Software Engineering

Security is an important concern throughout the software development process. Many of these security "touchpoints" require time-consuming manual procedures, e.g., architectural threat modeling. We want to reduce the barrier to entry and make security more cost-efficient by increasing the degree of automation. Machine learning plays a crucial role in this mission and our poster gives on overview of interesting applications of ML in secure software engineering

ROMEO: A Binary Vulnerability Detection Dataset for Exploring Juliet through the Lens of Assembly Language

Context Automatic vulnerability detection on C/C++ source code has benefitted from the introduction of machine learning to the field, with many recent publications targeting this combination. In contrast, assembly language or machine code artifacts receive less attention, although there are compelling reasons to study them. They are more representative of what is executed, more easily incorporated in dynamic analysis, and in the case of closed-source code, there is no alternative. Objective We evaluate the representative capability of assembly language compared to C/C++ source code for vulnerability detection. Furthermore, we investigate the role of call graph context in detecting function-spanning vulnerabilities. Finally, we verify whether compiling a benchmark dataset compromises an experiment's soundness by inadvertently leaking label information. Method We propose ROMEO, a publicly available, reproducible and reusable binary vulnerability detection benchmark dataset derived from the synthetic Juliet test suite. Alongside, we introduce a simple text-based assembly language representation that includes context for function-spanning vulnerability detection and semantics to detect high-level vulnerabilities. It is constructed by disassembling the .text segment of the respective binaries. Results We evaluate an x86 assembly language representation of the compiled dataset, combined with an off-the-shelf classifier. It compares favorably to state-of-the-art methods, including those operating on the full C/C++ code. Including context information using the call graph improves detection of function-spanning vulnerabilities. There is no label information leaked during the compilation process. Conclusion Performing vulnerability detection on a compiled program instead of the source code is a worthwhile tradeoff. While certain information is lost, e.g., comments and certain identifiers, other valuable information is gained, e.g., about compiler optimizations

Data from: Determinants of between-year burrow re-occupation in a colony of the European Bee-eater Merops apiaster

Re-occupation of existing nesting burrows in the European bee-eater Merops apiaster has only rarely – and if so mostly anecdotically – been documented in the literature record, although such behavior would substantially save time and energy. In this study, we quantify burrow re-occupation in a German colony over a period of eleven years and identify ecological variables determining reuse probability. Of 179 recorded broods, 54% took place in a reused burrow and the overall probability that one of 75 individually recognized burrows would be reused in a given subsequent year was estimated as 26.4%. This indicates that between-year burrow reuse is a common behavior in the study colony which contrasts with findings from studies in other colonies. Furthermore, burrow re-occupation probability declined highly significantly with increasing age of the breeding wall. Statistical separation of within- and between-burrow effects of the age of the breeding wall revealed that a decline in re-occupation probability with individual burrow age was responsible for this and not a selective disappearance of burrows with high re-occupation probability over time. Limited duty cycles of individual burrows may be caused by accumulating detritus or decreasing stability with increasing burrow age. Alternatively, burrow fidelity may presuppose pair fidelity which may also explain the observed restricted burrow reuse duty cycles. A consequent next step would be to extend our within-colony approach to other colonies and compare the ecological circumstances under which bee-eaters reuse breeding burrows