30 research outputs found

    RecAD: Towards A Unified Library for Recommender Attack and Defense

    Full text link
    In recent years, recommender systems have become a ubiquitous part of our daily lives, while they suffer from a high risk of being attacked due to the growing commercial and social values. Despite significant research progress in recommender attack and defense, there is a lack of a widely-recognized benchmarking standard in the field, leading to unfair performance comparison and limited credibility of experiments. To address this, we propose RecAD, a unified library aiming at establishing an open benchmark for recommender attack and defense. RecAD takes an initial step to set up a unified benchmarking pipeline for reproducible research by integrating diverse datasets, standard source codes, hyper-parameter settings, running logs, attack knowledge, attack budget, and evaluation results. The benchmark is designed to be comprehensive and sustainable, covering both attack, defense, and evaluation tasks, enabling more researchers to easily follow and contribute to this promising field. RecAD will drive more solid and reproducible research on recommender systems attack and defense, reduce the redundant efforts of researchers, and ultimately increase the credibility and practical value of recommender attack and defense. The project is released at https://github.com/gusye1234/recad

    Parallel Randomized Tucker Decomposition Algorithms

    Full text link
    The Tucker tensor decomposition is a natural extension of the singular value decomposition (SVD) to multiway data. We propose to accelerate Tucker tensor decomposition algorithms by using randomization and parallelization. We present two algorithms that scale to large data and many processors, significantly reduce both computation and communication cost compared to previous deterministic and randomized approaches, and obtain nearly the same approximation errors. The key idea in our algorithms is to perform randomized sketches with Kronecker-structured random matrices, which reduces computation compared to unstructured matrices and can be implemented using a fundamental tensor computational kernel. We provide probabilistic error analysis of our algorithms and implement a new parallel algorithm for the structured randomized sketch. Our experimental results demonstrate that our combination of randomization and parallelization achieves accurate Tucker decompositions much faster than alternative approaches. We observe up to a 16X speedup over the fastest deterministic parallel implementation on 3D simulation data

    Tensor Regression

    Full text link
    Regression analysis is a key area of interest in the field of data analysis and machine learning which is devoted to exploring the dependencies between variables, often using vectors. The emergence of high dimensional data in technologies such as neuroimaging, computer vision, climatology and social networks, has brought challenges to traditional data representation methods. Tensors, as high dimensional extensions of vectors, are considered as natural representations of high dimensional data. In this book, the authors provide a systematic study and analysis of tensor-based regression models and their applications in recent years. It groups and illustrates the existing tensor-based regression methods and covers the basics, core ideas, and theoretical characteristics of most tensor-based regression methods. In addition, readers can learn how to use existing tensor-based regression methods to solve specific regression tasks with multiway data, what datasets can be selected, and what software packages are available to start related work as soon as possible. Tensor Regression is the first thorough overview of the fundamentals, motivations, popular algorithms, strategies for efficient implementation, related applications, available datasets, and software resources for tensor-based regression analysis. It is essential reading for all students, researchers and practitioners of working on high dimensional data.Comment: 187 pages, 32 figures, 10 table

    Harnessing Big Data and Machine Learning for Event Detection and Localization

    Get PDF
    Anomalous events are rare and significantly deviate from expected pattern and other data instances, making them hard to predict. Correctly and timely detecting anomalous severe events can help reduce risks and losses. Many anomalous event detection techniques are studied in the literature. Recently, big data and machine learning based techniques have shown a remarkable success in a wide range of fields. It is important to tailor big data and machine learning based techniques for each application; otherwise it may result in expensive computation, slow prediction, false alarms, and improper prediction granularity.First, we aim to address the above challenges by harnessing big data and machine learning techniques for fast and reliable prediction and localization of severe events. Firstly, to improve storage failure prediction, we develop a new lightweight and high performing tensor decomposition-based method, named SEFEE, for storage error forecasting in large-scale enterprise storage systems. SEFEE employs tensor decomposition technique to capture latent spatio-temporal information embedded in storage event logs. By utilizing the latent spatio-temporal information, we can make accurate storage error forecasting without training requirements of typical machine learning techniques. The training-free method allows for live prediction of storage errors and their locations in the storage system based on previous observations that had been used in tensor decomposition pipeline to extract meaningful latent correlations. Moreover, we propose an extension to include severity of the errors as contextual information to improve the accuracy of tensor decomposition which in turn improves the prediction accuracy. We further provide detailed characterization of NetApp dataset to provide additional insight into the dynamics of typical large-scale enterprise storage systems for the community.Next, we focus on another application -- AI-driven Wildfire prediction. Wildfires cause billions of dollars in property damages and loss of lives, with harmful health threats. We aim to correctly detect and localize wildfire events in the early stage and also classify wildfire smoke based on perceived pixel density of camera images. Due to the lack of publicly available dataset for early wildfire smoke detection, we first collect and process images from the AlertWildfire camera network. The images are annotated with bounding boxes and densities for deep learning methods to use. We then adapt a transformer-based end-to-end object detection model for wildfire detection using our dataset. The dataset and detection model together form as a benchmark named the Nevada smoke detection benchmark, or Nemo for short. Nemo is the first open-source benchmark for wildfire smoke detection with the focus of the early incipient stage. We further provide a weakly supervised Nemo version to enable wider support as a benchmark

    Remote Sensing Data Compression

    Get PDF
    A huge amount of data is acquired nowadays by different remote sensing systems installed on satellites, aircrafts, and UAV. The acquired data then have to be transferred to image processing centres, stored and/or delivered to customers. In restricted scenarios, data compression is strongly desired or necessary. A wide diversity of coding methods can be used, depending on the requirements and their priority. In addition, the types and properties of images differ a lot, thus, practical implementation aspects have to be taken into account. The Special Issue paper collection taken as basis of this book touches on all of the aforementioned items to some degree, giving the reader an opportunity to learn about recent developments and research directions in the field of image compression. In particular, lossless and near-lossless compression of multi- and hyperspectral images still remains current, since such images constitute data arrays that are of extremely large size with rich information that can be retrieved from them for various applications. Another important aspect is the impact of lossless compression on image classification and segmentation, where a reasonable compromise between the characteristics of compression and the final tasks of data processing has to be achieved. The problems of data transition from UAV-based acquisition platforms, as well as the use of FPGA and neural networks, have become very important. Finally, attempts to apply compressive sensing approaches in remote sensing image processing with positive outcomes are observed. We hope that readers will find our book useful and interestin

    Structural pruning for speed in neural machine translation

    Get PDF
    Neural machine translation (NMT) strongly outperforms previous statistical techniques. With the emergence of a transformer architecture, we consistently train and deploy deeper and larger models, often with billions of parameters, as an ongoing effort to achieve even better quality. On the other hand, there is also a constant pursuit for optimisation opportunities to reduce inference runtime. Parameter pruning is one of the staple optimisation techniques. Even though coefficient-wise sparsity is the most popular for compression purposes, it is not easy to make a model run faster. Sparse matrix multiplication routines require custom approaches, usually depending on low-level hardware implementations for the most efficiency. In my thesis, I focus on structural pruning in the field of NMT, which results in smaller but still dense architectures that do not need any further modifications to work efficiently. My research focuses on two main directions. The first one explores Lottery Ticket Hypothesis (LTH), a well-known pruning algorithm, but this time in a structural setup with a custom pruning criterion. It involves partial training and pruning steps performed in a loop. Experiments with LTH produced substantial speed-up when applied to prune heads in the attention mechanism of a transformer. While this method has proven successful, it carries the burden of prolonged training cost that makes an already expensive training routine even more so. From that point, I exclusively concentrate on research incorporating pruning into training via regularisation. I experiment with a standard group lasso, which zeroes-out parameters together in a structural pre-defined way. By targeting feedforward and attention layers in a transformer, group lasso significantly improves inference speed with already optimised state-of-the-art fast models. Improving upon that work, I designed a novel approach called aided regularisation, where every layer penalty is scaled based on statistics gathered as training progresses. Both gradient- and parameter-based approaches aim to decrease the depth of a model, further optimising speed while maintaining the translation quality of an unpruned baseline. The goal of this dissertation is to advance the state-of-the-art efficient NMT with simple but tangible structural sparsity methods. The majority of all experiments in the thesis involve highly-optimised models as baselines to show that this work pushes the Pareto frontier of quality vs speed trade-off forward. For example, it is possible to prune a model to be 50% faster with no change in translation quality

    Scalable and Reliable Sparse Data Computation on Emergent High Performance Computing Systems

    Get PDF
    Heterogeneous systems with both CPUs and GPUs have become important system architectures in emergent High Performance Computing (HPC) systems. Heterogeneous systems must address both performance-scalability and power-scalability in the presence of failures. Aggressive power reduction pushes hardware to its operating limit and increases the failure rate. Resilience allows programs to progress when subjected to faults and is an integral component of large-scale systems, but incurs significant time and energy overhead. The future exascale systems are expected to have higher power consumption with higher fault rates. Sparse data computation is the fundamental kernel in many scientific applications. It is suitable for the studies of scalability and resilience on heterogeneous systems due to its computational characteristics. To deliver the promised performance within the given power budget, heterogeneous computing mandates a deep understanding of the interplay between scalability and resilience. Managing scalability and resilience is challenging in heterogeneous systems, due to the heterogeneous compute capability, power consumption, and varying failure rates between CPUs and GPUs. Scalability and resilience have been traditionally studied in isolation, and optimizing one typically detrimentally impacts the other. While prior works have been proved successful in optimizing scalability and resilience on CPU-based homogeneous systems, simply extending current approaches to heterogeneous systems results in suboptimal performance-scalability and/or power-scalability. To address the above multiple research challenges, we propose novel resilience and energy-efficiency technologies to optimize scalability and resilience for sparse data computation on heterogeneous systems with CPUs and GPUs. First, we present generalized analytical and experimental methods to analyze and quantify the time and energy costs of various recovery schemes, and develop and prototype performance optimization and power management strategies to improve scalability for sparse linear solvers. Our results quantitatively reveal that each resilience scheme has its own advantages depending on the fault rate, system size, and power budget, and the forward recovery can further benefit from our performance and power optimizations for large-scale computing. Second, we design a novel resilience technique that relaxes the requirement of synchronization and identicalness for processes, and allows them to run in heterogeneous resources with power reduction. Our results show a significant reduction in energy for unmodified programs in various fault situations compared to exact replication techniques. Third, we propose a novel distributed sparse tensor decomposition that utilizes an asynchronous RDMA-based approach with OpenSHMEM to improve scalability on large-scale systems and prove that our method works well in heterogeneous systems. Our results show our irregularity-aware workload partition and balanced-asynchronous algorithms are scalable and outperform the state-of-the-art distributed implementations. We demonstrate that understanding different bottlenecks for various types of tensors plays critical roles in improving scalability

    Towards reinforcement learning based N­Clustering

    Get PDF
    Tese de Mestrado, Ciência de Dados, 2022, Universidade de Lisboa, Faculdade de CiênciasBiclustering and triclustering are becoming increasingly popular for unsupervised analysis of two­ and three­dimensional datasets. Among other patterns of interest, using n­clusters in unsupervised data analy sis can identify potential biological modules, illness progression profiles, and communities of individuals with consistent behaviour. Despite this, most algorithms still rely on exhaustive approaches to produce high­quality results. The main limitation of using deep learning to solve this task is that n­clusters are computed assuming that all elements are represented under equal distance. This assumption invalidates the use of locality simplification techniques like neural convolutions. Graphs are flexible structures that could represent a dataset where all elements are at an equal distance through fully connected graphs, thus encouraging the use of graph convolutional networks to learn their structure and generate accurate embeddings of the datasets. Because n­clustering is primarily viewed as an iterative task in which elements are added or re moved from a given cluster, a reinforcement learning framework is a good fit. Deep reinforcement learn ing agents have already been successfully coupled with graph convolutional networks to solve complex combinatorial optimization problems, motivating the adaptation of reinforcement learning architectures to this problem. This dissertation lays the foundations for a novel reinforcement learning approach for n­clustering that could outperform state of the art algorithms while implementing a more efficient algorithm. To this end, three libraries were implemented: a synthetic data generator, a framework that models n­clustering tasks as Markov decision process, and a training library. A proximal policy­based agent was implemented and tunned using population­based training, to evaluate the behaviour of the reinforcement learning en vironments designed. Results show that agents can learn to modify their behaviour while interacting with the environment to maximize their reward signal. However, they are still far from being a solution to n­clustering. This dissertation is the first step towards this solution. Finally, future steps to improve these results are pro posed. This dissertation has presented foundational work that enables modelling n­clustering as an MDP, paving the way for further studies focused on improving task performance.Os seres humanos evoluíram para encontrar padrões. Esta capacidade está presente na nossa vida quotidiana, e não sobreviveríamos sem ela. Na realidade, esta é uma característica que parecemos partilhar com todos os seres inteligentes, a necessidade de compreender padrões e de criar rotinas. Os padrões são lugares seguros onde podemos agir conscientemente, onde as relações causais que ligam as nossas acções às suas consequências são conhecidas por nós. A compreensão de um padrão pode ser a diferença entre vida e morte, o suave som de folhas pode implicar um ataque mortal, a presença de humidade no solo pode indicar um riacho próximo, enquanto um cheiro pode ajudar a distinguir entre amigo ou inimigo. Encontrar padrões e distinguir entre padrões e acontecimentos aleatórios permitiu à nossa sociedade chegar tão longe. Hoje, enfrentamos problemas mais complexos em quase todos os campos de estudo científicos e sociais, por vezes escondidos por detrás de quantidades massivas de eventos aleatórios. É literalmente como encontrar uma agulha num palheiro. Como tal, recorremos mais uma vez a máquinas para nos ajudar neste empreendimento desafiante. Técnicas de aprendizagem sem supervisão começaram a ser propostas por estatísticos e matemáticos muito antes do aparecimento de campos como a prospecção de dados. No entanto, estes campos, juntamente com um significativo interesse restaurado na área pela indústria, na esperança de rentabilizar grandes quantidades de dados guardados ao longo dos anos, deram grandes passos em frente. Nos últimos anos, temos visto muitos avanços notáveis neste campo e uma nova face da inteligência artificial em geral (por exemplo, aprendizagem de máquinas, aprendizagem profunda). Foram propostas abordagens de clusters revigoradas que combinavam técnicas clássicas com aprendizagem profunda para gerar representações precisas e produzir clusters a partir destes vectores de dados. Biclustering e triclustering estão a tornar-­se cada vez mais populares para análises não supervisionadas de conjuntos de dados bidimensionais e tridimensionais. Entre outros padrões de interesse, a utilização de n­clusters na análise não supervisionada de dados pode identificar potenciais módulos biológicos, perfis de progressão de doenças, e comunidades de indivíduos com comportamento consistente. Nos domínios médicos, as aplicações possíveis incluem a análise de sinais fisiológicos multivariados, onde os n­clusters identificados podem capturar respostas fisiológicas coerentes para um grupo de indivíduos; análise de dados de neuroimagem, onde os n­clusters podem capturar funções de resposta hemodinâmica e conectividade entre regiões cerebrais; e análise de registos clínicos, onde os n­clusters podem corresponder a grupos de pacientes com características clínicas correlacionadas ao longo do tempo. Relativamente aos domínios sociais, as aplicações possíveis vão desde a análise de redes sociais até à descoberta de comunidades de indivíduos com actividade e interacção correlacionadas (frequentemente referidas como comunidades em evolução coerente) ou conteúdos de grupo de acordo com o perfil do utilizador; grupos de utilizadores com padrões de navegação coerentes nos dados de utilização da web; análise de dados de comércio electrónico para encontrar padrões de navegação ocultos de conjuntos cor relacionados de utilizadores (web), páginas (web) visitadas, e operações ao longo do tempo; análise de dados de pesquisa de marketing para estudar a utilidade perceptível de vários produtos para diferentes fins, a julgar por diferentes grupos demográficos; dados de filtragem colaborativa para descobrir correlações accionáveis para sistemas de recomendação ou utilizadores de grupo com preferências semelhantes, entre outras aplicações. O clustering tradicional pode ser utilizado para agrupar observações neste contexto, mas a sua utili dade é limitada porque as observações neste domínio de dados são tipicamente apenas significativamente correlacionadas em subespaços do espaço global. Apesar da importância de n­clustering, a maioria dos algoritmos continua a basear­se em abordagens exaustivas para produzir resultados de qualidade. Como o n­clustering é uma tarefa complexa de opti mização combinatória, as abordagens existentes limitam a estrutura permitida, a coerência e a qualidade da solução. A principal limitação da utilização de aprendizagem profunda para resolver esta tarefa é que os n clusters são computados assumindo que todos os elementos são representados sob igual distância. Este pressuposto invalida o uso de técnicas de simplificação da localidade como as convoluções neurais. Os grafos são estruturas flexíveis que podem ser utilizadas para representar um conjunto de dados onde todos os elementos estão a uma distância igual, através de grafos completos, encorajando assim a utilização de redes convolucionais de grafos para aprender a sua estrutura e gerar representações precisas dos conjuntos de dados. Uma vez que o n­clustering é visto principalmente como uma tarefa iterativa em que os elemen tos são adicionados ou removidos de um dado cluster, uma estrutura de aprendizagem de reforço é um bom suporte. Agentes de aprendizagem de reforço profundos já foram acoplados com sucesso a redes convolucionais de grafos para resolver problemas complexos de otimização combinatória, motivando a adaptação de arquitecturas de aprendizagem de reforço a este problema. Esta dissertação lança as bases para uma nova abordagem de aprendizagem por reforço para n clustering que poderia superar os algoritmos de estado da arte, ao mesmo tempo que implementa um algoritmo mais eficiente. Para este fim, foram implementadas três bibliotecas: um gerador de dados sintéticos, uma framework que modela as tarefas de n­clustering como um processo de decisão de Markov, e uma biblioteca de treino. NclustGen foi implementado para melhorar a utilização programática dos geradores de dados sintéti cos de biclustering e triclustering de última geração. O NclustEnv modela n­clustering como um processo de decisão Markov através da implementação de ambientes de biclustering e triclustering. Segue a interface padrão de programação de aplicações proposta pelo Gym para ambientes de aprendizagem por reforço. A implementação de ambientes de qualidade que modelam a interação entre um agente e uma tarefa de n­clustering é da maior importância. Ao implementar esta tarefa utilizando o padrão Gym, o ambi ente pode ser implementado como agente agnóstico. Assim, qualquer agente será capaz de treinar neste ambiente, se correctamente configurado, independentemente da sua implementação. Esta capacidade de construir ambientes que modelam uma dada tarefa de uma forma agnóstica permite a implementação de uma framework geral para n­clustering baseada em aprendizagem por reforço. Os agentes podem depois utilizar esta framework de treino para encontrar uma solução de última geração para esta tarefa. A fim de avaliar o comportamento dos ambientes de aprendizagem por reforço que foram concebidos, foi implementado e calibrado um agente de optimização proximal de políticas utilizando treino baseado em populações. Um agente de optimização proximal de políticas foi escolhido porque pode servir como uma boa base para experiências futuras. Devido à sua versatilidade, os agentes de optimização proximal de políticas são largamente considerados como os agentes de referência para experiências em ambientes não explorados. A solução e as limitações alcançadas por este agente normalmente dão pelo menos uma ideia dos seguintes passos a tomar se o agente não conseguir alcançar uma boa solução. Os resultados mostram que os agentes podem aprender a modificar o seu comportamento enquanto interagem com o ambiente para maximizar o seu sinal de recompensa. No entanto, ainda estão longe de ser uma solução para o n­clustering. Esta dissertação é o primeiro passo para esta solução e apresentou o trabalho fundamental, mas ainda há muito mais trabalho a ser feito para que esta abordagem possa ultrapassar os algoritmos mais avança dos.Por fim, são propostos os próximos passos para melhorar estes resultados, e que para num futuro próximo, esta abordagem possa vir a resolver a tarefa do n­clustering

    Sketch-Based Streaming Anomaly Detection in Dynamic Graphs

    Full text link
    Given a stream of graph edges from a dynamic graph, how can we assign anomaly scores to edges and subgraphs in an online manner, for the purpose of detecting unusual behavior, using constant time and memory? For example, in intrusion detection, existing work seeks to detect either anomalous edges or anomalous subgraphs, but not both. In this paper, we first extend the count-min sketch data structure to a higher-order sketch. This higher-order sketch has the useful property of preserving the dense subgraph structure (dense subgraphs in the input turn into dense submatrices in the data structure). We then propose four online algorithms that utilize this enhanced data structure, which (a) detect both edge and graph anomalies; (b) process each edge and graph in constant memory and constant update time per newly arriving edge, and; (c) outperform state-of-the-art baselines on four real-world datasets. Our method is the first streaming approach that incorporates dense subgraph search to detect graph anomalies in constant memory and time

    Computing Large-Scale Matrix and Tensor Decomposition with Structured Factors: A Unified Nonconvex Optimization Perspective

    Full text link
    The proposed article aims at offering a comprehensive tutorial for the computational aspects of structured matrix and tensor factorization. Unlike existing tutorials that mainly focus on {\it algorithmic procedures} for a small set of problems, e.g., nonnegativity or sparsity-constrained factorization, we take a {\it top-down} approach: we start with general optimization theory (e.g., inexact and accelerated block coordinate descent, stochastic optimization, and Gauss-Newton methods) that covers a wide range of factorization problems with diverse constraints and regularization terms of engineering interest. Then, we go `under the hood' to showcase specific algorithm design under these introduced principles. We pay a particular attention to recent algorithmic developments in structured tensor and matrix factorization (e.g., random sketching and adaptive step size based stochastic optimization and structure-exploiting second-order algorithms), which are the state of the art---yet much less touched upon in the literature compared to {\it block coordinate descent} (BCD)-based methods. We expect that the article to have an educational values in the field of structured factorization and hope to stimulate more research in this important and exciting direction.Comment: Final Version; to appear in IEEE Signal Processing Magazine; title revised to comply with the journal's rul
    corecore