Are mutants really natural? A study on how “naturalness” helps mutant selection

Background: Code is repetitive and predictable in a way that is similar to the natural language. This means that code is ``natural'' and this ``naturalness'' can be captured by natural language modelling techniques. Such models promise to capture the program semantics and identify source code parts that `smell', i.e., they are strange, badly written and are generally error-prone (likely to be defective). Aims: We investigate the use of natural language modelling techniques in mutation testing (a testing technique that uses artificial faults). We thus, seek to identify how well artificial faults simulate real ones and ultimately understand how natural the artificial faults can be. %We investigate this question in a fault revelation perspective. Our intuition is that natural mutants, i.e., mutants that are predictable (follow the implicit coding norms of developers), are semantically useful and generally valuable (to testers). We also expect that mutants located on unnatural code locations (which are generally linked with error-proneness) to be of higher value than those located on natural code locations. Method: Based on this idea, we propose mutant selection strategies that rank mutants according to a) their naturalness (naturalness of the mutated code), b) the naturalness of their locations (naturalness of the original program statements) and c) their impact on the naturalness of the code that they apply to (naturalness differences between original and mutated statements). We empirically evaluate these issues on a benchmark set of 5 open-source projects, involving more than 100k mutants and 230 real faults. Based on the fault set we estimate the utility (i.e. capability to reveal faults) of mutants selected on the basis of their naturalness, and compare it against the utility of randomly selected mutants. Results: Our analysis shows that there is no link between naturalness and the fault revelation utility of mutants. We also demonstrate that the naturalness-based mutant selection performs similar (slightly worse) to the random mutant selection. Conclusions: Our findings are negative but we consider them interesting as they confute a strong intuition, i.e., fault revelation is independent of the mutants' naturalness

Address Space Layout Randomization Next Generation

[EN] Systems that are built using low-power computationally-weak devices, which force developers to favor performance over security; which jointly with its high connectivity, continuous and autonomous operation makes those devices specially appealing to attackers. ASLR (Address Space Layout Randomization) is one of the most effective mitigation techniques against remote code execution attacks, but when it is implemented in a practical system its effectiveness is jeopardized by multiple constraints: the size of the virtual memory space, the potential fragmentation problems, compatibility limitations, etc. As a result, most ASLR implementations (specially in 32-bits) fail to provide the necessary protection. In this paper we propose a taxonomy of all ASLR elements, which categorizes the entropy in three dimensions: (1) how, (2) when and (3) what; and includes novel forms of entropy. Based on this taxonomy we have created, ASLRA, an advanced statistical analysis tool to assess the effectiveness of any ASLR implementation. Our analysis show that all ASLR implementations suffer from several weaknesses, 32-bit systems provide a poor ASLR, and OS X has a broken ASLR in both 32- and 64-bit systems. This is jeopardizing not only servers and end users devices as smartphones but also the whole IoT ecosystem. To overcome all these issues, we present ASLR-NG, a novel ASLR that provides the maximum possible absolute entropy and removes all correlation attacks making ASLR-NG the best solution for both 32- and 64-bit systems. We implemented ASLR-NG in the Linux kernel 4.15. The comparative evaluation shows that ASLR-NG overcomes PaX, Linux and OS X implementations, providing strong protection to prevent attackers from abusing weak ASLRs.Marco-Gisbert, H.; Ripoll-Ripoll, I. (2019). Address Space Layout Randomization Next Generation. Applied Sciences. 9(14):1-25. https://doi.org/10.3390/app9142928S125914Aga, M. T., & Austin, T. (2019). Smokestack: Thwarting DOP Attacks with Runtime Stack Layout Randomization. 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). doi:10.1109/cgo.2019.8661202Object Size Checking to Prevent (Some) Buffer Overflows (GCC FORTIFY) http://gcc.gnu.org/ml/gcc-patches/2004-09/msg02055.htmlShahriar, H., & Zulkernine, M. (2012). Mitigating program security vulnerabilities. ACM Computing Surveys, 44(3), 1-46. doi:10.1145/2187671.2187673Carlier, M., Steenhaut, K., & Braeken, A. (2019). Symmetric-Key-Based Security for Multicast Communication in Wireless Sensor Networks. Computers, 8(1), 27. doi:10.3390/computers8010027Choudhary, J., Balasubramanian, P., Varghese, D., Singh, D., & Maskell, D. (2019). Generalized Majority Voter Design Method for N-Modular Redundant Systems Used in Mission- and Safety-Critical Applications. Computers, 8(1), 10. doi:10.3390/computers8010010Shacham, H., Page, M., Pfaff, B., Goh, E.-J., Modadugu, N., & Boneh, D. (2004). On the effectiveness of address-space randomization. Proceedings of the 11th ACM conference on Computer and communications security - CCS ’04. doi:10.1145/1030083.1030124Marco-Gisbert, H., & Ripoll, I. (2013). Preventing Brute Force Attacks Against Stack Canary Protection on Networking Servers. 2013 IEEE 12th International Symposium on Network Computing and Applications. doi:10.1109/nca.2013.12Friginal, J., de Andres, D., Ruiz, J.-C., & Gil, P. (2010). Attack Injection to Support the Evaluation of Ad Hoc Networks. 2010 29th IEEE Symposium on Reliable Distributed Systems. doi:10.1109/srds.2010.11Jun Xu, Kalbarczyk, Z., & Iyer, R. K. (s. f.). Transparent runtime randomization for security. 22nd International Symposium on Reliable Distributed Systems, 2003. Proceedings. doi:10.1109/reldis.2003.1238076Zhan, X., Zheng, T., & Gao, S. (2014). Defending ROP Attacks Using Basic Block Level Randomization. 2014 IEEE Eighth International Conference on Software Security and Reliability-Companion. doi:10.1109/sere-c.2014.28Iyer, V., Kanitkar, A., Dasgupta, P., & Srinivasan, R. (2010). Preventing Overflow Attacks by Memory Randomization. 2010 IEEE 21st International Symposium on Software Reliability Engineering. doi:10.1109/issre.2010.22Van der Veen, V., dutt-Sharma, N., Cavallaro, L., & Bos, H. (2012). Memory Errors: The Past, the Present, and the Future. Lecture Notes in Computer Science, 86-106. doi:10.1007/978-3-642-33338-5_5PaX Address Space Layout Randomization (ASLR) http://pax.grsecurity.net/docs/aslr.txtKernel Address Space Layout Randomization https://lwn.net/Articles/569635Rahman, M. A., & Asyhari, A. T. (2019). The Emergence of Internet of Things (IoT): Connecting Anything, Anywhere. Computers, 8(2), 40. doi:10.3390/computers8020040Bojinov, H., Boneh, D., Cannings, R., & Malchev, I. (2011). Address space randomization for mobile devices. Proceedings of the fourth ACM conference on Wireless network security - WiSec ’11. doi:10.1145/1998412.1998434Hiser, J., Nguyen-Tuong, A., Co, M., Hall, M., & Davidson, J. W. (2012). ILR: Where’d My Gadgets Go? 2012 IEEE Symposium on Security and Privacy. doi:10.1109/sp.2012.39Xu, H., & Chapin, S. J. (2009). Address-space layout randomization using code islands. Journal of Computer Security, 17(3), 331-362. doi:10.3233/jcs-2009-0322Wartell, R., Mohan, V., Hamlen, K. W., & Lin, Z. (2012). Binary stirring. Proceedings of the 2012 ACM conference on Computer and communications security - CCS ’12. doi:10.1145/2382196.2382216Growable Maps Removal https://lwn.net/Articles/294001/Silent Stack-Heap Collision under GNU/Linux https://gcc.gnu.org/ml/gcc-help/2014-07/msg00076.htmlAMD Bulldozer Linux ASLR Weakness: Reducing Entropy by 87.5% http://hmarco.org/bugs/AMD-Bulldozer-linux-ASLR-weakness-reducing-mmaped-files-by-eight.htmlCVE-2015-1593—Linux ASLR Integer Overflow: Reducing Stack Entropy by Four http://hmarco.org/bugs/linux-ASLR-integer-overflow.htmlLinux ASLR Mmap Weakness: Reducing Entropy by Half http://hmarco.org/bugs/linux-ASLR-reducing-mmap-by-half.htmlLESNE, A. (2014). Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics. Mathematical Structures in Computer Science, 24(3). doi:10.1017/s0960129512000783Scraps of Notes on Remote Stack Overflow Exploitation http://www.phrack.org/issues.html?issue=67&id=13#articleUchenick, G. M., & Vanfleet, W. M. (2005). Multiple independent levels of safety and security: high assurance architecture for MSLS/MLS. MILCOM 2005 - 2005 IEEE Military Communications Conference. doi:10.1109/milcom.2005.1605749Lee, B., Lu, L., Wang, T., Kim, T., & Lee, W. (2014). From Zygote to Morula: Fortifying Weakened ASLR on Android. 2014 IEEE Symposium on Security and Privacy. doi:10.1109/sp.2014.34The Heartbleed Bug http://heartbleed.co

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants