Implications of Z-normalization in the matrix profile

Companies are increasingly measuring their products and services, resulting in a rising amount of available time series data, making techniques to extract usable information needed. One state-of-the-art technique for time series is the Matrix Profile, which has been used for various applications including motif/discord discovery, visualizations and semantic segmentation. Internally, the Matrix Profile utilizes the z-normalized Euclidean distance to compare the shape of subsequences between two series. However, when comparing subsequences that are relatively flat and contain noise, the resulting distance is high despite the visual similarity of these subsequences. This property violates some of the assumptions made by Matrix Profile based techniques, resulting in worse performance when series contain flat and noisy subsequences. By studying the properties of the z-normalized Euclidean distance, we derived a method to eliminate this effect requiring only an estimate of the standard deviation of the noise. In this paper we describe various practical properties of the z-normalized Euclidean distance and show how these can be used to correct the performance of Matrix Profile related techniques. We demonstrate our techniques using anomaly detection using a Yahoo! Webscope anomaly dataset, semantic segmentation on the PAMAP2 activity dataset and for data visualization on a UCI activity dataset, all containing real-world data, and obtain overall better results after applying our technique. Our technique is a straightforward extension of the distance calculation in the Matrix Profile and will benefit any derived technique dealing with time series containing flat and noisy subsequences

A generalized matrix profile framework with support for contextual series analysis

The Matrix Profile is a state-of-the-art time series analysis technique that can be used for motif discovery, anomaly detection, segmentation and others, in various domains such as healthcare, robotics, and audio. Where recent techniques use the Matrix Profile as a preprocessing or modeling step, we believe there is unexplored potential in generalizing the approach. We derived a framework that focuses on the implicit distance matrix calculation. We present this framework as the Series Distance Matrix (SDM). In this framework, distance measures (SDM-generators) and distance processors (SDM-consumers) can be freely combined, allowing for more flexibility and easier experimentation. In SDM, the Matrix Profile is but one specific configuration. We also introduce the Contextual Matrix Profile (CMP) as a new SDM-consumer capable of discovering repeating patterns. The CMP provides intuitive visualizations for data analysis and can find anomalies that are not discords. We demonstrate this using two real world cases. The CMP is the first of a wide variety of new techniques for series analysis that fits within SDM and can complement the Matrix Profile

Calculating the matrix profile from noisy data

The matrix profile (MP) is a data structure computed from a time series which encodes the data required to locate motifs and discords, corresponding to recurring patterns and outliers respectively. When the time series contains noisy data then the conventional approach is to pre-filter it in order to remove noise but this cannot apply in unsupervised settings where patterns and outliers are not annotated. The resilience of the algorithm used to generate the MP when faced with noisy data remains unknown. We measure the similarities between the MP from original time series data with MPs generated from the same data with noisy data added under a range of parameter settings including adding duplicates and adding irrelevant data. We use three real world data sets drawn from diverse domains for these experiments Based on dissimilarities between the MPs, our results suggest that MP generation is resilient to a small amount of noise being introduced into the data but as the amount of noise increases this resilience disappearsComment: 16 page

Music Synchronization, Audio Matching, Pattern Detection, and User Interfaces for a Digital Music Library System

Over the last two decades, growing efforts to digitize our cultural heritage could be observed. Most of these digitization initiatives pursuit either one or both of the following goals: to conserve the documents - especially those threatened by decay - and to provide remote access on a grand scale. For music documents these trends are observable as well, and by now several digital music libraries are in existence. An important characteristic of these music libraries is an inherent multimodality resulting from the large variety of available digital music representations, such as scanned score, symbolic score, audio recordings, and videos. In addition, for each piece of music there exists not only one document of each type, but many. Considering and exploiting this multimodality and multiplicity, the DFG-funded digital library initiative PROBADO MUSIC aimed at developing a novel user-friendly interface for content-based retrieval, document access, navigation, and browsing in large music collections. The implementation of such a front end requires the multimodal linking and indexing of the music documents during preprocessing. As the considered music collections can be very large, the automated or at least semi-automated calculation of these structures would be recommendable. The field of music information retrieval (MIR) is particularly concerned with the development of suitable procedures, and it was the goal of PROBADO MUSIC to include existing and newly developed MIR techniques to realize the envisioned digital music library system. In this context, the present thesis discusses the following three MIR tasks: music synchronization, audio matching, and pattern detection. We are going to identify particular issues in these fields and provide algorithmic solutions as well as prototypical implementations. In Music synchronization, for each position in one representation of a piece of music the corresponding position in another representation is calculated. This thesis focuses on the task of aligning scanned score pages of orchestral music with audio recordings. Here, a previously unconsidered piece of information is the textual specification of transposing instruments provided in the score. Our evaluations show that the neglect of such information can result in a measurable loss of synchronization accuracy. Therefore, we propose an OCR-based approach for detecting and interpreting the transposition information in orchestral scores. For a given audio snippet, audio matching methods automatically calculate all musically similar excerpts within a collection of audio recordings. In this context, subsequence dynamic time warping (SSDTW) is a well-established approach as it allows for local and global tempo variations between the query and the retrieved matches. Moving to real-life digital music libraries with larger audio collections, however, the quadratic runtime of SSDTW results in untenable response times. To improve on the response time, this thesis introduces a novel index-based approach to SSDTW-based audio matching. We combine the idea of inverted file lists introduced by Kurth and Müller (Efficient index-based audio matching, 2008) with the shingling techniques often used in the audio identification scenario. In pattern detection, all repeating patterns within one piece of music are determined. Usually, pattern detection operates on symbolic score documents and is often used in the context of computer-aided motivic analysis. Envisioned as a new feature of the PROBADO MUSIC system, this thesis proposes a string-based approach to pattern detection and a novel interactive front end for result visualization and analysis

IMPLEMENTASI ALGORITMA SAX DAN RANDOM PROJECTION UNTUK TIME SERIES MOTIF DISCOVERY PADA BIG DATA PLATFORM STUDI KASUS: RESONANSI ELEMEN ORBITAL ASTEROID

Fenomena Big Data telah terjadi pada banyak bidang pengetahuan, salah satunya adalah pada bidang astronomi. Data yang memiliki jumlah yang sangat banyak pada bidang astronomi salah satunya adalah data resonansi elemen orbital asteroid. Data tersebut dapat diolah sehingga para ilmuwan dapat mencari mean motion resonance pada suatu partikel asteroid untuk mengetahui pada tahun keberapa asteroid akan beresonansi dengan planet planet tertentu. Untuk itu, penelitian ini membuat sebuah model komputasi untuk mendapatkan mean motion resonance secara cepat dan efektif dengan memodifikasi dan mengimplementasikan algoritma SAX dan algoritma motif discovery random projection pada Big Data Platform yaitu Apache Hadoop dan Apache Spark. Hasil penelitian ini menunjukkan adanya percepatan yang sangat signifikan antara penggunaan stand alone dan penggunaan Big Data platform dengan perancangan 2 skenario. Skenario pertama yaitu penggunaan cluster dengan 4 cores dan beberapa worker node dan skenario kedua yaitu penggunaan cluster dengan 2 worker node dan beberapa jumlah core. Penelitian ini juga membuktikan bahwa model komputasi yang dibangun dibandingkan dengan kelauran software SwiftVis mendapatkan rata-rata akurasi sebesar 83%. The Big Data phenomenon has occurred in many fields of knowledge, one of them is in the field of astronomy. One of data that has a very large amount in the astronomy field is the resonance data of asteroid orbital elements. The data can be processed so that scientists can find the mean motion resonance in an asteroid particle to find out in what year the asteroid will resonate with a particular planet. But now, it cannot be denied that to process the data with a large amount of data will take a lot of time. For this reason, this research makes a computational model to get mean motion resonance quickly and effectively by modifying and implementing the SAX algorithm and motif discovery random projection algorithm on the Big Data platform, using Apache Hadoop and Apache Spark. The results of this study indicate a very significant acceleration between standalone use and the use of Big Data platforms by designing 2 scenarios. The first scenario is the use of clusters with 4 cores and several worker nodes and the second scenario is the use of clusters with 2 worker nodes and a number of cores. This study also proves that the computational model compared to the result from SwiftVis software gets an average accuracy of 83%

Inferring the role of transcription factors in regulatory networks

<p>Abstract</p> <p>Background</p> <p>Expression profiles obtained from multiple perturbation experiments are increasingly used to reconstruct transcriptional regulatory networks, from well studied, simple organisms up to higher eukaryotes. Admittedly, a key ingredient in developing a reconstruction method is its ability to integrate heterogeneous sources of information, as well as to comply with practical observability issues: measurements can be scarce or noisy. In this work, we show how to combine a network of genetic regulations with a set of expression profiles, in order to infer the functional effect of the regulations, as inducer or repressor. Our approach is based on a consistency rule between a network and the signs of variation given by expression arrays.</p> <p>Results</p> <p>We evaluate our approach in several settings of increasing complexity. First, we generate artificial expression data on a transcriptional network of <it>E. coli </it>extracted from the literature (1529 nodes and 3802 edges), and we estimate that 30% of the regulations can be annotated with about 30 profiles. We additionally prove that at most 40.8% of the network can be inferred using our approach. Second, we use this network in order to validate the predictions obtained with a compendium of real expression profiles. We describe a filtering algorithm that generates particularly reliable predictions. Finally, we apply our inference approach to <it>S. cerevisiae </it>transcriptional network (2419 nodes and 4344 interactions), by combining ChIP-chip data and 15 expression profiles. We are able to detect and isolate inconsistencies between the expression profiles and a significant portion of the model (15% of all the interactions). In addition, we report predictions for 14.5% of all interactions.</p> <p>Conclusion</p> <p>Our approach does not require accurate expression levels nor times series. Nevertheless, we show on both data, real and artificial, that a relatively small number of perturbation experiments are enough to determine a significant portion of regulatory effects. This is a key practical asset compared to statistical methods for network reconstruction. We demonstrate that our approach is able to provide accurate predictions, even when the network is incomplete and the data is noisy.</p

Stocks, bonds, money markets and exchange rates: measuring international financial transmission

The paper presents a framework for analyzing the degree of financial transmission between money, bond and equity markets and exchange rates within and between the United States and the euro area. We find that asset prices react strongest to other domestic asset price shocks, and that there are also substantial international spillovers, both within and across asset classes. The results underline the dominance of US markets as the main driver of global financial markets: US financial markets explain, on average, more than 25% of movements in euro area financial markets, whereas euro area markets account only for about 8% of US asset price changes. The international propagation of shocks is strengthened in times of recession, and has most likely changed in recent years: prior to EMU, the paper finds smaller international spillovers. JEL Classification: E44, F3, C5financial market linkages, integration, international financial markets, transmission