Association Rule Mining on Big Data Sets

An accurate, complete, and rapid establishment of customer needs and existence of product recommendations are crucial points in terms of increasing customer satisfaction level in various different sectors such as the banking sector. Due to the significant increase in the number of transactions and customers, analyzing costs regarding time and consumption of memory becomes higher. In order to increase the performance of the product recommendation, we discuss an approach, a sample data creation process, to association rule mining. Thus instead of processing whole population, processing on a sample that represents the population is used to decrease time of analysis and consumption of memory. In this regard, sample composing methods, sample size determination techniques, the tests which measure the similarity between sample and population, and association rules (ARs) derived from the sample were examined. The mutual buying behavior of the customers was found using a well-known association rule mining algorithm. Techniques were compared according to the criteria of complete rule derivation and time consumption

Optimality of a Network Monitoring Agent and Validation in a Real Probe

The evolution of commodity hardware makes it possible to use this type of equipment to implement traffic monitoring systems. A preliminary empirical evaluation of a network traffic probe based on Linux indicates that the system performance has significant losses as the network rate increases. To assess this issue, we consider a model with two tandem queues and a moving server. In this system, we formulate a three-dimensional Markov Decision Process in continuous time. The goal of the proposed model is to determine the position of the server in each time slot so as to optimize the system performance which is measured in terms of throughput. We first formulate an equivalent discrete-time Markov Decision Process and we propose a numerical method to characterize the solution of our problem in a general setting. The solution we obtain in this problem has been tested for a wide range of scenarios and, in all the instances, we observe that the optimality is close to a threshold type policy. We also consider a real probe and we validate the good performance of threshold policies in real applications.This research was partially supported by the Department of Education of the Basque Government, Spain through the Consolidated Research Groups NQaS (IT1635-22) and MATHMODE (IT1456-22), by the Marie Sklodowska-Curie, Spain grant agreement No 777778, by the Spanish Ministry of Science and Innovation, Spain with reference PID2019-108111RB-I00 (FEDER/AEI), by grant PID2020-117876RB-I00 funded by MCIN/AEI (10.13039/501100011033) and by Grant KK-2021/00026 funded by the Basque Government

Enhanced IPFIX flow processing mechanism for overlay network monitoring

Cloud computing is an emerging technology. People are adopting cloud at a faster rate, due to this cloud network traffic is increasing at a pace which is challenging to manage. Monitoring tool is an essential aspect of cloud computing and becomes more apparent with the acquired of cloud services. Overlay network provides new path to converge network and run as an independent virtual network on top of physical network. Currently, cloud overlay network technologies in cloud infrastructure have visibility gaps, which mean cloud provider and consumers miss out the major performance issues for troubleshooting of overlay network traffic. Hence, to keep a close watch on network and catch potential problems, a network monitoring tool required, to track and report more in-depth for not only see the hidden traffic but also presents the related information of cloud overlay network technologies specifically suited to the modern cloud-scale data center. Therefore, this study proposes an enhanced IP Flow Information Export (IPFIX) mechanism for cloud overlay network monitoring by adopting flexible flow based technique. Furthermore, the solution provided in this research consist of diverse mechanisms: enhanced packet filtering mechanisms using property match filtering technique and hash-based filtering technique. Virtual Extensible Local Area Network (VXLAN) based flow classification mechanisms using 6-tuple flow pattern and adoptable flow patterns. IPFIX message template mechanisms, which is comprise set of fields for data records within the IPFIX flow processing system. The findings demonstrate that the proposed mechanism can capture multi-tenant overlay network traffic to identify, track, analyze and continuously monitor the performance of cloud overlay network services. The proposed mechanisms are resource efficient where the combination of VFMFM+6tuple+VXLAN Message consume 4.63% less CPU, while the combination of VHFM+AFCM+AFCM Message consume 11.45% less CPU than Standard IPFIX. The contributions of this study would help cloud network operators and end-users to quickly and proactively resolve any overlay network based on performance issues with end-to end visibility and actionable insights

Monitoring Integration Systems and Visualization

The amount of new software and data has grown significantly in our society. In addition, new software solutions often have been built over the old ones to extend their capabilities. Also architectural solutions for software systems include multiple smaller services that have been divided all over the system. This has lead to the situation where more communication happens between the different parts of the system. Monitoring abundant traffic manually, with human resources is really challenging or even an impossible task. That is the reason why monitoring is performed with automated systems. The objective of the thesis is to develop an extension for monitoring system, that will be in charge of monitoring the integration platform. Thesis is done for the SaaS -software company BCB Medical Oy. In the thesis solutions were sought for general monitoring challenges and integration platforms problems in monitoring. Retrieving integration platforms essential information and their visual presentation was one of these problems. In addition to these problems, the differences in monitoring between the integration part of the system and rest of the system, were evaluated. Also determining thresholds that help detecting any anomalies in the system, was considered an issue. The research for in thesis was performed with qualitative methods, by interviewing the employees of BCB Medical Oy. Interviews were constructed with semi structural interview model and they were used to discover solutions for monitoring issues. Implementation part of the thesis was made with the same tools that the company uses for monitoring, Prometheus and Grafana. Results of the work accomplished of determining values that are required from the integration platform. Transferring these values to the monitoring system was performed with using Prometheus exporter. Visualization of these values was done with Grafana to help discover important information and determining certain thresholds for alerts.Uusien ohjelmistojen ja datan määrä on kasvanut yhteiskunnassamme merkittävästi. Lisäksi vanhojen sovellusten toiminnallisuutta on yritetty laajentaa uusien ratkaisujen avulla. Myös sovellusten arkkitehtuuriset ratkaisut ovat usein toteuttettu niin, että järjestelmä on jaettu useisiin osiin. Tämä on myös johtanut lisääntyneeseen kommunikointiin komponenttien välillä. Runsaasta liikenteestä järjestelmässä johtuen sovellusten manuaalinen monitorointi on ihmisresurssein erittäin haastavaa ellei jopa mahdotonta. Tästä johtuen monitorointia suoritetaan automaattisella monitorointi järjestelmällä. Työn tavoitteena on kehittää monitorointi järjestelmän osa, joka vastaa integraatioalustan monitoiroinnista. Lopputyö toteutettiin SaaS -ohjelmistoyritys BCB Medical Oy:lle. Työssä etsittiin ratkaisuja monitoroinnin haasteisiin ja integraationalustan synnyttämiin ongelmiin monitoroinnissa. Erityisesti integraatioalustalta tarpeellisten tietojen hakemiseen ja niiden visuaaliseen esittämiseen etsittiin ratkaisua. Näiden ongelmien lisäksi koitettiin saada vastauksia integraatioalustan monitoroinnin eroavaisuuksiin tavalliseen monitorointiin verrattuna sekä miten määrittää integraatioalustan arvoille raja-arvo, jonka avulla havaitaan alustan ongelmat. Tutkimus suoritettiin kvalitatiivisellä menetelmällä, haastattelemalla BCB Medical:n työntekijöitä. Haastattelut on muodostettu käyttäen puolistrukturoitua rakennetta ja niillä selvitettiin monitorointiin liittyviä ratkaisuja. Työosuus toteutet-tiin yrityksen käyttämillä monitoroitityökaluilla, joita olivat Prometheus ja Grafana. Työn tuloksena saatiin määritettyä integraatioalustan monitoroinnissa välttämättömiä arvoja. Arvojen siirtäminen monitorointi alustalle tapahtui käyttämällä Prometheuksen exportteria. Sen lisäksi näiden arvojen esittäminen Grafana-ilmoitustaululla tärkeiden tietojen havaitsemisen sujuvoittamiseksi ja hälytysjärjestelmän arvojen määrittämiseksi

Data Mining

Data mining is a branch of computer science that is used to automatically extract meaningful, useful knowledge and previously unknown, hidden, interesting patterns from a large amount of data to support the decision-making process. This book presents recent theoretical and practical advances in the field of data mining. It discusses a number of data mining methods, including classification, clustering, and association rule mining. This book brings together many different successful data mining studies in various areas such as health, banking, education, software engineering, animal science, and the environment