Feature Extraction and Feature Selection: Reducing Data Complexity with Apache Spark

Feature extraction and feature selection are the first tasks in pre-processing of input logs in order to detect cyber security threats and attacks while utilizing machine learning. When it comes to the analysis of heterogeneous data derived from different sources, these tasks are found to be time-consuming and difficult to be managed efficiently. In this paper, we present an approach for handling feature extraction and feature selection for security analytics of heterogeneous data derived from different network sensors. The approach is implemented in Apache Spark, using its python API, named pyspark

Deep Neural Network Solution for Detecting Intrusion in Network

In our experiment, we found that deep learning surpassed machine learning when utilizing the DSSTE algorithm to sample imbalanced training set samples. These methods excel in terms of throughput due to their complex structure and ability to autonomously acquire relevant features from a dataset. The current study focuses on employing deep learning techniques such as RNN and Deep-NN, as well as algorithm design, to aid network IDS designers. Since public datasets already preprocess the data features, deep learning is unable to leverage its automatic feature extraction capability, limiting its ability to learn from preprocessed features. To harness the advantages of deep learning in feature extraction, mitigate the impact of imbalanced data, and enhance classification accuracy, our approach involves directly applying the deep learning model for feature extraction and model training on the existing network traffic data. By doing so, we aim to capitalize on deep learning's benefits, improving feature extraction, reducing the influence of imbalanced data, and enhancing classification accuracy

A Critical Evaluation of Business Improvement through Machine Learning: Challenges, Opportunities, and Best Practices

This paper presents a critical evaluation of the impact of machine learning (ML) on business improvement, focusing on the challenges, opportunities, and best practices associated with its implementation. The study examines the hurdles faced by businesses while integrating ML, such as data quality, talent acquisition, algorithm bias, interpretability, and privacy concerns. On the other hand, it highlights the advantages of ML, including data-driven decision-making, enhanced customer experience, process optimization, cost reduction, and the potential for new revenue streams. Furthermore, the paper offers best practices to guide businesses in successfully adopting ML solutions, covering data management, talent development, model evaluation, ethics, and regulatory compliance. Through real-world case studies, the study illustrates successful ML applications in different industries. It also addresses the ethical and social implications of ML adoption and discusses emerging trends for future directions. Ultimately, this evaluation provides valuable insights to enable informed decisions and sustainable growth for businesses leveraging machine learning

IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective

With the wide spread of sensors and smart devices in recent years, the data generation speed of the Internet of Things (IoT) systems has increased dramatically. In IoT systems, massive volumes of data must be processed, transformed, and analyzed on a frequent basis to enable various IoT services and functionalities. Machine Learning (ML) approaches have shown their capacity for IoT data analytics. However, applying ML models to IoT data analytics tasks still faces many difficulties and challenges, specifically, effective model selection, design/tuning, and updating, which have brought massive demand for experienced data scientists. Additionally, the dynamic nature of IoT data may introduce concept drift issues, causing model performance degradation. To reduce human efforts, Automated Machine Learning (AutoML) has become a popular field that aims to automatically select, construct, tune, and update machine learning models to achieve the best performance on specified tasks. In this paper, we conduct a review of existing methods in the model selection, tuning, and updating procedures in the area of AutoML in order to identify and summarize the optimal solutions for every step of applying ML algorithms to IoT data analytics. To justify our findings and help industrial users and researchers better implement AutoML approaches, a case study of applying AutoML to IoT anomaly detection problems is conducted in this work. Lastly, we discuss and classify the challenges and research directions for this domain.Comment: Published in Engineering Applications of Artificial Intelligence (Elsevier, IF:7.8); Code/An AutoML tutorial is available at Github link: https://github.com/Western-OC2-Lab/AutoML-Implementation-for-Static-and-Dynamic-Data-Analytic

A Comprehensive Analysis of the Role of Artificial Intelligence and Machine Learning in Modern Digital Forensics and Incident Response

In the dynamic landscape of digital forensics, the integration of Artificial Intelligence (AI) and Machine Learning (ML) stands as a transformative technology, poised to amplify the efficiency and precision of digital forensics investigations. However, the use of ML and AI in digital forensics is still in its nascent stages. As a result, this paper gives a thorough and in-depth analysis that goes beyond a simple survey and review. The goal is to look closely at how AI and ML techniques are used in digital forensics and incident response. This research explores cutting-edge research initiatives that cross domains such as data collection and recovery, the intricate reconstruction of cybercrime timelines, robust big data analysis, pattern recognition, safeguarding the chain of custody, and orchestrating responsive strategies to hacking incidents. This endeavour digs far beneath the surface to unearth the intricate ways AI-driven methodologies are shaping these crucial facets of digital forensics practice. While the promise of AI in digital forensics is evident, the challenges arising from increasing database sizes and evolving criminal tactics necessitate ongoing collaborative research and refinement within the digital forensics profession. This study examines the contributions, limitations, and gaps in the existing research, shedding light on the potential and limitations of AI and ML techniques. By exploring these different research areas, we highlight the critical need for strategic planning, continual research, and development to unlock AI's full potential in digital forensics and incident response. Ultimately, this paper underscores the significance of AI and ML integration in digital forensics, offering insights into their benefits, drawbacks, and broader implications for tackling modern cyber threats

Automatization of incident resolution

Incident management is a key IT Service Management sub process in every organization as a way to deal with the current volume of tickets created every year. Currently, the resolution process is still extremely human labor intensive. A large number of incidents are not from a new, never seen before problem, they have already been solved in the past and their respective resolution have been previously stored in an Incident Ticket System. Automation of repeatable tasks in IT is an important element of service management and can have a considerable impact in an organization. Using a large real-world database of incident tickets, this dissertation explores a method to automatically propose a suitable resolution for a new ticket using previous tickets’ resolution texts. At its core, the method uses machine learning, natural language parsing, information retrieval and mining. The proposed method explores machine learning models like SVM, Logistic Regression, some neural networks architecture and more, to predict an incident resolution category for a new ticket and a module to automatically retrieve resolution action phrases from tickets using part-of-speech pattern matching. In the experiments performed, 31% to 41% of the tickets from a test set was considered as solved by the proposed method, which considering the yearly volume of tickets represents a significant amount of manpower and resources that could be saved.A Gestão de incidentes é um subprocesso chave da Gestão de Serviços de TI em todas as organizações como uma forma de lidar com o volume atual de tickets criados todos os anos. Atualmente, o processo de resolução ainda exige muito trabalho humano. Um grande número de incidentes não são de um problema novo, nunca visto antes, eles já foram resolvidos no passado e sua respetiva resolução foi previamente armazenada em um Sistema de Ticket de Incidentes. A automação de tarefas repetíveis em TI é um elemento importante do Gestão de Serviços e pode ter um impacto considerável em uma organização. Usando um grande conjunto de dados reais de tickets de incidentes, esta dissertação explora um método para propor automaticamente uma resolução adequada para um novo ticket usando textos de resolução de tickets anteriores. Em sua essência, o método usa aprendizado de máquina, análise de linguagem natural, recuperação de informações e mineração. O método proposto explora modelos de aprendizagem automática como SVM, Regressão Logística, arquitetura de algumas redes neurais e mais, para prever uma categoria de resolução de incidentes para um novo ticket e um módulo para extrair automaticamente ações de resolução de tickets usando padrões de classes gramaticais. Nas experiências realizados, 31% a 41% dos tickets de um conjunto de testes foram considerados como resolvidos pelo método proposto, que considerando o volume anual de tickets representa uma quantidade significativa de mão de obra e recursos que poderiam ser economizados