Actor Critic Deep Reinforcement Learning for Neural Malware Control

In addition to using signatures, antimalware products also detect malicious attacks by evaluating unknown files in an emulated environment, i.e. sandbox, prior to execution on a computer's native operating system. During emulation, a file cannot be scanned indefinitely, and antimalware engines often set the number of instructions to be executed based on a set of heuristics. These heuristics only make the decision of when to halt emulation using partial information leading to the execution of the file for either too many or too few instructions. Also this method is vulnerable if the attackers learn this set of heuristics. Recent research uses a deep reinforcement learning (DRL) model employing a Deep Q-Network (DQN) to learn when to halt the emulation of a file. In this paper, we propose a new DRL-based system which instead employs a modified actor critic (AC) framework for the emulation halting task. This AC model dynamically predicts the best time to halt the file's execution based on a sequence of system API calls. Compared to the earlier models, the new model is capable of handling adversarial attacks by simulating their behaviors using the critic model. The new AC model demonstrates much better performance than both the DQN model and antimalware engine's heuristics. In terms of execution speed (evaluated by the halting decision), the new model halts the execution of unknown files by up to 2.5% earlier than the DQN model and 93.6% earlier than the heuristics. For the task of detecting malicious files, the proposed AC model increases the true positive rate by 9.9% from 69.5% to 76.4% at a false positive rate of 1% compared to the DQN model, and by 83.4% from 41.2% to 76.4% at a false positive rate of 1% compared to a recently proposed LSTM model

Design and evaluation of a hybrid multi-task learning model for optimizing deep reinforcement learning agents

Driven by recent technological advancements within the artificial intelligence domain, deep learning has emerged as a promising representation learning technique. This in turn has given rise to the evolution of deep reinforcement learning that combines deep learning with reinforcement learning methods. Subsequently, performance optimization achieved by reinforcement learning intelligent agents designed with model-free based approaches were predominantly limited to systems with reinforcement learning algorithms learning single task. Such a model was found to be quite data inefficient, whenever agents needed to interact with more complex, rich data environments. This thesis introduces a hybrid multi-task learning-oriented approach for optimization of deep reinforcement learning agents operating within different but semantically similar environments with related tasks. Empirical results obtained with OpenAI Gym library-based Atari 2600 video gaming environment demonstrate that the proposed hybrid multi-task learning model is successful in addressing key challenges associated with the performance optimization of deep reinforcement learning agents