FastLMFI: An Efficient Approach for Local Maximal Patterns Propagation and Maximal Patterns Superset Checking

Maximal frequent patterns superset checking plays an important role in the efficient mining of complete Maximal Frequent Itemsets (MFI) and maximal search space pruning. In this paper we present a new indexing approach, FastLMFI for local maximal frequent patterns (itemset) propagation and maximal patterns superset checking. Experimental results on different sparse and dense datasets show that our work is better than the previous well known progressive focusing technique. We have also integrated our superset checking approach with an existing state of the art maximal itemsets algorithm Mafia, and compare our results with current best maximal itemsets algorithms afopt-max and FP (zhu)-max. Our results outperform afopt-max and FP (zhu)-max on dense (chess and mushroom) datasets on almost all support thresholds, which shows the effectiveness of our approach.Comment: 8 Pages, In the proceedings of 4th ACS/IEEE International Conference on Computer Systems and Applications 2006, March 8, 2006, Dubai/Sharjah, UAE, 2006, Page(s) 452-45

Association Rule Mining -- Geometry and Parallel Computing Approach

Mining association rules is a very important aspect in data mining fields. The process to mine association rules not only take much time, but also take huge computing source. How to fast and efficiently find the large itemsets is a crucial point in the association rule algorithms. This paper will focus on two algorithms research and implementation in parallel computing environments. One is Bitmap Combination algorithm, the other is Bitmap FP-Growth algorithm. Compared to Apriori algorithm, both Bitmap Combination and Bitmap FP-Growth algorithms don’t need generate candidate items, avoids costly database scans. Both algorithms need to translate the original database to Bitmap format, analyze bit distribution to reduce database size and apply high-speed bit calculation to improve the algorithms. The divide-and-conquer replace generation-and-test idea as the basic strategy. Bitmap Combination Algorithm shows the quick combination skills between any two, three, four and more rows, then screening the qualified itemsets. Bitmap FP-Growth Algorithm apply special bit calculation to recursively mine association rules. Based on the experimental results in this paper, both algorithms greatly improve the efficiency and performance of mining association rules, especially provide the possibility to mine association rules in highly parallel computing environments

FastLMFI: An Efficient Approach for Local Maximal Patterns Propagation and Maximal Patterns Superset Checking

Maximal frequent patterns superset checking plays an important role in the efficient mining of complete Maximal Frequent Itemsets (MFI) and maximal search space pruning. In this paper we present a new indexing approach, FastLMFI for local maximal frequent patterns (itemset) propagation and maximal patterns superset checking. Experimental results on different sparse and dense datasets show that our work is better than the previous well known progressive focusing technique. We have also integrated our superset checking approach with an existing state of the art maximal itemsets algorithm Mafia, and compare our results with current best maximal itemsets algorithms afopt-max and FP (zhu)-max. Our results outperform afopt-max and FP (zhu)-max on dense (chess and mushroom) datasets on almost all support thresholds, which shows the effectiveness of our approach.Comment: 8 Pages, In the proceedings of 4th ACS/IEEE International Conference on Computer Systems and Applications 2006, March 8, 2006, Dubai/Sharjah, UAE, 2006, Page(s) 452-45

Frequent itemset mining on multiprocessor systems

Frequent itemset mining is an important building block in many data mining applications like market basket analysis, recommendation, web-mining, fraud detection, and gene expression analysis. In many of them, the datasets being mined can easily grow up to hundreds of gigabytes or even terabytes of data. Hence, efficient algorithms are required to process such large amounts of data. In recent years, there have been many frequent-itemset mining algorithms proposed, which however (1) often have high memory requirements and (2) do not exploit the large degrees of parallelism provided by modern multiprocessor systems. The high memory requirements arise mainly from inefficient data structures that have only been shown to be sufficient for small datasets. For large datasets, however, the use of these data structures force the algorithms to go out-of-core, i.e., they have to access secondary memory, which leads to serious performance degradations. Exploiting available parallelism is further required to mine large datasets because the serial performance of processors almost stopped increasing. Algorithms should therefore exploit the large number of available threads and also the other kinds of parallelism (e.g., vector instruction sets) besides thread-level parallelism. In this work, we tackle the high memory requirements of frequent itemset mining twofold: we (1) compress the datasets being mined because they must be kept in main memory during several mining invocations and (2) improve existing mining algorithms with memory-efficient data structures. For compressing the datasets, we employ efficient encodings that show a good compression performance on a wide variety of realistic datasets, i.e., the size of the datasets is reduced by up to 6.4x. The encodings can further be applied directly while loading the dataset from disk or network. Since encoding and decoding is repeatedly required for loading and mining the datasets, we reduce its costs by providing parallel encodings that achieve high throughputs for both tasks. For a memory-efficient representation of the mining algorithms’ intermediate data, we propose compact data structures and even employ explicit compression. Both methods together reduce the intermediate data’s size by up to 25x. The smaller memory requirements avoid or delay expensive out-of-core computation when large datasets are mined. For coping with the high parallelism provided by current multiprocessor systems, we identify the performance hot spots and scalability issues of existing frequent-itemset mining algorithms. The hot spots, which form basic building blocks of these algorithms, cover (1) counting the frequency of fixed-length strings, (2) building prefix trees, (3) compressing integer values, and (4) intersecting lists of sorted integer values or bitmaps. For all of them, we discuss how to exploit available parallelism and provide scalable solutions. Furthermore, almost all components of the mining algorithms must be parallelized to keep the sequential fraction of the algorithms as small as possible. We integrate the parallelized building blocks and components into three well-known mining algorithms and further analyze the impact of certain existing optimizations. Our algorithms are already single-threaded often up an order of magnitude faster than existing highly optimized algorithms and further scale almost linear on a large 32-core multiprocessor system. Although our optimizations are intended for frequent-itemset mining algorithms, they can be applied with only minor changes to algorithms that are used for mining of other types of itemsets

CAMEP - Complex business activity monitoring system with event processing

U današnjem svijetu, složeno globalno poslovno okruženje je u stanju kontinuiranih promjena. Stoga su poslovne organizacije u potrazi za inovativnim načinom optimaliziranja i davanja podrške obavljanju njihovih poslovnih procesa kako bi sačuvale svoj udio na tržištu i održale konkurentnu prednost. Obično se poslovni procesi prate gotovo u realnom vremenu pomoću ključnih pokazatelja uspješnosti (KPI). KPI se pak izračunavaju na temelju podataka iz poslovnih procesa. U slučaju poslovnog outsourcing, dobivanje KPI iz poslovnih podataka mora se provoditi prelazeći granice među organizacijama i obuhvaćajući mnoga područja. Ovaj proces postavlja nekoliko izazova proračunu ključnih pokazatelja uspješnosti. Neki od tih izazova su nesigurnost u metričke vrijednosti, just-in-time računanje KPI i skalabilnost. Za rješavanje navedenih izazova poslovnih organizacija, predlaže se sustav nadzora složenih poslovnih aktivnosti u četiri nivoa s obradom pojedinog postupka (CAMEP). Ocjenjivanje uspješnosti predloženog sustava provodi se kako bi se pokazala učinkovitost predložene CAMEP metode usporedbom s postojećim BAM pristupima. Predložena se metoda CAMEP također analizira pomoću ANOVA i T testa.In today’s world, complex global business environment is in a state of continuous change. Therefore, business organizations are looking for an innovative way to optimize and to support the performance of their business processes to meet out their market share in order to maintain a competitive edge. Usually, business processes are monitored in near real-time using Key Performance Indicators (KPI). KPI is in turn calculated from metrics from business processes. In the case of business outsourcing, the KPI calculation from the metrics of business process is to be carried out cutting across the inter-organizations boundaries and spanning multiple domains. This process poses several challenges in the computation of the KPIs. Some of these challenges include uncertainty in metric values, just-in-time KPI computation and scalability. To address the above challenges of business organizations, a four-tier Complex business Activity Monitoring system with Event Processing (CAMEP) is proposed. The performance evaluation of the proposed system is performed to demonstrate the effectiveness of the proposed CAMEP method by comparing with the existing BAM approaches. The proposed CAMEP is also analysed using the ANOVA and T test

Discovering Interesting Patterns and Associations in Data Streams

A data stream is a sequence of items that arrive in a timely order. Different from data in traditional static databases, data streams are continuous, unbounded, usually come with high speed, and have a data value distribution that often changes with time (Guha, 2001). As more applications such as web transactions, telephone records, and network flows generate a large number of data streams every day, efficient knowledge discovery of data streams is an active and growing research area in data mining with broad applications. Traditional data mining algorithms are developed to work on a complete static dataset and, thus, cannot be applied directly in data stream applications.One area of data mining research is to mine association relationship in a data set. Most of association mining techniques for data streams can be categorized into two types: those developed based on frequent patterns and those developed based on closed patterns. Due to the number of frequent patterns are often huge and redundant, non-informative patterns are contained in frequent patterns. An alternative way is to develop the association mining approaches for data streaming applications based on closed patterns, which generally represent a small subset of all frequent patterns, but provide complete and condensed information. In these researches, the closed pattern mining is the prerequisite condition for non-redundant and informative association mining.In this dissertation, a sliding window technique for dynamic mining of closed patterns in data streams is proposed, and an approach of mining non-redundant and informative associations based on the discovered closed patterns is developed. The closed pattern and relevant association mining techniques are selected research area in this dissertation. First, the closed patterns for a given collection of data are currently the most compact data knowledge that can provide complete support information for all data patterns.Compared with other techniques, the proposed closed pattern mining technique has potential to largely decrease the number of subsequent combinatorial calculations performed on the data patterns. Second, the memory requirement to store the closed patterns and relevant associations is generally lower than the corresponding frequent patterns and associations. In some data streaming applications, memory usage is an important measurement, because in these applications memory usage is the bottleneck for knowledge discovery. Third, the associations generated for data streams are the knowledge used to identify the relations within the data. The discovered relations can find their wide applications in many data streaming environments.Different from the closed pattern mining techniques on traditional databases, which require multiple scans of the entire database, the proposed technique determines the closed patterns with a single scan. It is an incremental mining process; as the sliding window advances, new data transactions enter and old data transactions exit the window. But instead of regenerating closed patterns from the entire window, the proposed technique updates the old set of closed patterns whenever a new transaction arrives and/or an old transaction leaves the sliding window to obtain the current set of closed patterns. This incremental feature allows the user to get the most recent updated closed patterns without rescanning the entire updated database, which saves not only the computation time, but more importantly, the I/O operating time to load and write data from database to memory. Third, the proposed sliding window technique can handle both the insertion and deletion operations independently, which allows the user to adjust the sliding window size in different application environments. Furthermore, the proposed interesting patterns and association mining framework can handle different users' requests at the same time at their specified support and confidence thresholds, and interested input and output patterns.The research includes both theoretical proofs of correctness for the proposed algorithms and simulation experiments to compare the proposed techniques with those existing in the literature using synthetic and real datasets. The utility of the proposed technique is applied to sensor network databases of a traffic management and an environmental monitoring site for missing data estimation purpose

A genetic algorithm coupled with tree-based pruning for mining closed association rules

Due to the voluminous amount of itemsets that are generated, the association rules extracted from these itemsets contain redundancy, and designing an effective approach to address this issue is of paramount importance. Although multiple algorithms were proposed in recent years for mining closed association rules most of them underperform in terms of run time or memory. Another issue that remains challenging is the nature of the dataset. While some of the existing algorithms perform well on dense datasets others perform well on sparse datasets. This paper aims to handle these drawbacks by using a genetic algorithm for mining closed association rules. Recent studies have shown that genetic algorithms perform better than conventional algorithms due to their bitwise operations of crossover and mutation. Bitwise operations are predominantly faster than conventional approaches and bits consume lesser memory thereby improving the overall performance of the algorithm. To address the redundancy in the mined association rules a tree-based pruning algorithm has been designed here. This works on the principle of minimal antecedent and maximal consequent. Experiments have shown that the proposed approach works well on both dense and sparse datasets while surpassing existing techniques with regard to run time and memory