Malware can be described as any type of malicious code that has the potential harm to the computer or network. these threats came from various sources like the internet, local networks and portable drives. Virus which replicates itself is growing faster every year and poses a serious global security threat. The purpose of this research is to classify portable executable new malicious files from benign files. In recent years, data mining methods are investigated for detecting unknown malicious executables, and the result show high and acceptable detection rate. Therefore, this project applied machine learning to detect malicious executable files through Support Vector Machine (SVM) and Artificial Neural Network (ANN) algorithms. These algorithms are compared together and selected the best accuracy model. The result of this research indicated that the accuracy of the SVM and ANN rely on the settings of the parameters used; ANN showed higher accuracy of 98.76 than SVM in terms of data set used while SVM performed a speed three times less than ANN and low computational power. The main conclusions drawn from this research were that current detection approaches of the antivirus are deficient because they fail to detect new unseen malicious files and they have higher false negative rates
