30 research outputs found
RecAD: Towards A Unified Library for Recommender Attack and Defense
In recent years, recommender systems have become a ubiquitous part of our
daily lives, while they suffer from a high risk of being attacked due to the
growing commercial and social values. Despite significant research progress in
recommender attack and defense, there is a lack of a widely-recognized
benchmarking standard in the field, leading to unfair performance comparison
and limited credibility of experiments. To address this, we propose RecAD, a
unified library aiming at establishing an open benchmark for recommender attack
and defense. RecAD takes an initial step to set up a unified benchmarking
pipeline for reproducible research by integrating diverse datasets, standard
source codes, hyper-parameter settings, running logs, attack knowledge, attack
budget, and evaluation results. The benchmark is designed to be comprehensive
and sustainable, covering both attack, defense, and evaluation tasks, enabling
more researchers to easily follow and contribute to this promising field. RecAD
will drive more solid and reproducible research on recommender systems attack
and defense, reduce the redundant efforts of researchers, and ultimately
increase the credibility and practical value of recommender attack and defense.
The project is released at https://github.com/gusye1234/recad
Parallel Randomized Tucker Decomposition Algorithms
The Tucker tensor decomposition is a natural extension of the singular value
decomposition (SVD) to multiway data. We propose to accelerate Tucker tensor
decomposition algorithms by using randomization and parallelization. We present
two algorithms that scale to large data and many processors, significantly
reduce both computation and communication cost compared to previous
deterministic and randomized approaches, and obtain nearly the same
approximation errors. The key idea in our algorithms is to perform randomized
sketches with Kronecker-structured random matrices, which reduces computation
compared to unstructured matrices and can be implemented using a fundamental
tensor computational kernel. We provide probabilistic error analysis of our
algorithms and implement a new parallel algorithm for the structured randomized
sketch. Our experimental results demonstrate that our combination of
randomization and parallelization achieves accurate Tucker decompositions much
faster than alternative approaches. We observe up to a 16X speedup over the
fastest deterministic parallel implementation on 3D simulation data
Tensor Regression
Regression analysis is a key area of interest in the field of data analysis
and machine learning which is devoted to exploring the dependencies between
variables, often using vectors. The emergence of high dimensional data in
technologies such as neuroimaging, computer vision, climatology and social
networks, has brought challenges to traditional data representation methods.
Tensors, as high dimensional extensions of vectors, are considered as natural
representations of high dimensional data. In this book, the authors provide a
systematic study and analysis of tensor-based regression models and their
applications in recent years. It groups and illustrates the existing
tensor-based regression methods and covers the basics, core ideas, and
theoretical characteristics of most tensor-based regression methods. In
addition, readers can learn how to use existing tensor-based regression methods
to solve specific regression tasks with multiway data, what datasets can be
selected, and what software packages are available to start related work as
soon as possible. Tensor Regression is the first thorough overview of the
fundamentals, motivations, popular algorithms, strategies for efficient
implementation, related applications, available datasets, and software
resources for tensor-based regression analysis. It is essential reading for all
students, researchers and practitioners of working on high dimensional data.Comment: 187 pages, 32 figures, 10 table
Harnessing Big Data and Machine Learning for Event Detection and Localization
Anomalous events are rare and significantly deviate from expected pattern and other data instances, making them hard to predict. Correctly and timely detecting anomalous severe events can help reduce risks and losses. Many anomalous event detection techniques are studied in the literature. Recently, big data and machine learning based techniques have shown a remarkable success in a wide range of fields. It is important to tailor big data and machine learning based techniques for each application; otherwise it may result in expensive computation, slow prediction, false alarms, and improper prediction granularity.First, we aim to address the above challenges by harnessing big data and machine learning techniques for fast and reliable prediction and localization of severe events. Firstly, to improve storage failure prediction, we develop a new lightweight and high performing tensor decomposition-based method, named SEFEE, for storage error forecasting in large-scale enterprise storage systems. SEFEE employs tensor decomposition technique to capture latent spatio-temporal information embedded in storage event logs. By utilizing the latent spatio-temporal information, we can make accurate storage error forecasting without training requirements of typical machine learning techniques. The training-free method allows for live prediction of storage errors and their locations in the storage system based on previous observations that had been used in tensor decomposition pipeline to extract meaningful latent correlations. Moreover, we propose an extension to include severity of the errors as contextual information to improve the accuracy of tensor decomposition which in turn improves the prediction accuracy. We further provide detailed characterization of NetApp dataset to provide additional insight into the dynamics of typical large-scale enterprise storage systems for the community.Next, we focus on another application -- AI-driven Wildfire prediction. Wildfires cause billions of dollars in property damages and loss of lives, with harmful health threats. We aim to correctly detect and localize wildfire events in the early stage and also classify wildfire smoke based on perceived pixel density of camera images.
Due to the lack of publicly available dataset for early wildfire smoke detection, we first collect and process images from the AlertWildfire camera network. The images are annotated with bounding boxes and densities for deep learning methods to use. We then adapt a transformer-based end-to-end object detection model for wildfire detection using our dataset. The dataset and detection model together form as a benchmark named the Nevada smoke detection benchmark, or Nemo for short. Nemo is the first open-source benchmark for wildfire smoke detection with the focus of the early incipient stage. We further provide a weakly supervised Nemo version to enable wider support as a benchmark
Remote Sensing Data Compression
A huge amount of data is acquired nowadays by different remote sensing systems installed on satellites, aircrafts, and UAV. The acquired data then have to be transferred to image processing centres, stored and/or delivered to customers. In restricted scenarios, data compression is strongly desired or necessary. A wide diversity of coding methods can be used, depending on the requirements and their priority. In addition, the types and properties of images differ a lot, thus, practical implementation aspects have to be taken into account. The Special Issue paper collection taken as basis of this book touches on all of the aforementioned items to some degree, giving the reader an opportunity to learn about recent developments and research directions in the field of image compression. In particular, lossless and near-lossless compression of multi- and hyperspectral images still remains current, since such images constitute data arrays that are of extremely large size with rich information that can be retrieved from them for various applications. Another important aspect is the impact of lossless compression on image classification and segmentation, where a reasonable compromise between the characteristics of compression and the final tasks of data processing has to be achieved. The problems of data transition from UAV-based acquisition platforms, as well as the use of FPGA and neural networks, have become very important. Finally, attempts to apply compressive sensing approaches in remote sensing image processing with positive outcomes are observed. We hope that readers will find our book useful and interestin
Structural pruning for speed in neural machine translation
Neural machine translation (NMT) strongly outperforms previous statistical techniques. With
the emergence of a transformer architecture, we consistently train and deploy deeper and
larger models, often with billions of parameters, as an ongoing effort to achieve even better
quality. On the other hand, there is also a constant pursuit for optimisation opportunities to
reduce inference runtime.
Parameter pruning is one of the staple optimisation techniques. Even though coefficient-wise
sparsity is the most popular for compression purposes, it is not easy to make a model run
faster. Sparse matrix multiplication routines require custom approaches, usually depending on
low-level hardware implementations for the most efficiency. In my thesis, I focus on structural
pruning in the field of NMT, which results in smaller but still dense architectures that do not
need any further modifications to work efficiently.
My research focuses on two main directions. The first one explores Lottery Ticket Hypothesis
(LTH), a well-known pruning algorithm, but this time in a structural setup with a custom pruning
criterion. It involves partial training and pruning steps performed in a loop. Experiments with
LTH produced substantial speed-up when applied to prune heads in the attention mechanism
of a transformer. While this method has proven successful, it carries the burden of prolonged
training cost that makes an already expensive training routine even more so.
From that point, I exclusively concentrate on research incorporating pruning into training via
regularisation. I experiment with a standard group lasso, which zeroes-out parameters together
in a structural pre-defined way. By targeting feedforward and attention layers in a transformer,
group lasso significantly improves inference speed with already optimised state-of-the-art fast
models. Improving upon that work, I designed a novel approach called aided regularisation,
where every layer penalty is scaled based on statistics gathered as training progresses. Both
gradient- and parameter-based approaches aim to decrease the depth of a model, further
optimising speed while maintaining the translation quality of an unpruned baseline.
The goal of this dissertation is to advance the state-of-the-art efficient NMT with simple but
tangible structural sparsity methods. The majority of all experiments in the thesis involve
highly-optimised models as baselines to show that this work pushes the Pareto frontier of
quality vs speed trade-off forward. For example, it is possible to prune a model to be 50% faster
with no change in translation quality
Scalable and Reliable Sparse Data Computation on Emergent High Performance Computing Systems
Heterogeneous systems with both CPUs and GPUs have become important system architectures in emergent High Performance Computing (HPC) systems. Heterogeneous systems must address both performance-scalability and power-scalability in the presence of failures. Aggressive power reduction pushes hardware to its operating limit and increases the failure rate. Resilience allows programs to progress when subjected to faults and is an integral component of large-scale systems, but incurs significant time and energy overhead. The future exascale systems are expected to have higher power consumption with higher fault rates. Sparse data computation is the fundamental kernel in many scientific applications. It is suitable for the studies of scalability and resilience on heterogeneous systems due to its computational characteristics.
To deliver the promised performance within the given power budget, heterogeneous computing mandates a deep understanding of the interplay between scalability and resilience. Managing scalability and resilience is challenging in heterogeneous systems, due to the heterogeneous compute capability, power consumption, and varying failure rates between CPUs and GPUs. Scalability and resilience have been traditionally studied in isolation, and optimizing one typically detrimentally impacts the other. While prior works have been proved successful in optimizing scalability and resilience on CPU-based homogeneous systems, simply extending current approaches to heterogeneous systems results in suboptimal performance-scalability and/or power-scalability.
To address the above multiple research challenges, we propose novel resilience and energy-efficiency technologies to optimize scalability and resilience for sparse data computation on heterogeneous systems with CPUs and GPUs. First, we present generalized analytical and experimental methods to analyze and quantify the time and energy costs of various recovery schemes, and develop and prototype performance optimization and power management strategies to improve scalability for sparse linear solvers. Our results quantitatively reveal that each resilience scheme has its own advantages depending on the fault rate, system size, and power budget, and the forward recovery can further benefit from our performance and power optimizations for large-scale computing. Second, we design a novel resilience technique that relaxes the requirement of synchronization and identicalness for processes, and allows them to run in heterogeneous resources with power reduction. Our results show a significant reduction in energy for unmodified programs in various fault situations compared to exact replication techniques. Third, we propose a novel distributed sparse tensor decomposition that utilizes an asynchronous RDMA-based approach with OpenSHMEM to improve scalability on large-scale systems and prove that our method works well in heterogeneous systems. Our results show our irregularity-aware workload partition and balanced-asynchronous algorithms are scalable and outperform the state-of-the-art distributed implementations. We demonstrate that understanding different bottlenecks for various types of tensors plays critical roles in improving scalability
Towards reinforcement learning based NClustering
Tese de Mestrado, Ciência de Dados, 2022, Universidade de Lisboa, Faculdade de CiênciasBiclustering and triclustering are becoming increasingly popular for unsupervised analysis of two and
threedimensional datasets. Among other patterns of interest, using nclusters in unsupervised data analy sis can identify potential biological modules, illness progression profiles, and communities of individuals
with consistent behaviour. Despite this, most algorithms still rely on exhaustive approaches to produce
highquality results.
The main limitation of using deep learning to solve this task is that nclusters are computed assuming
that all elements are represented under equal distance. This assumption invalidates the use of locality
simplification techniques like neural convolutions. Graphs are flexible structures that could represent
a dataset where all elements are at an equal distance through fully connected graphs, thus encouraging
the use of graph convolutional networks to learn their structure and generate accurate embeddings of the
datasets. Because nclustering is primarily viewed as an iterative task in which elements are added or re moved from a given cluster, a reinforcement learning framework is a good fit. Deep reinforcement learn ing agents have already been successfully coupled with graph convolutional networks to solve complex
combinatorial optimization problems, motivating the adaptation of reinforcement learning architectures
to this problem.
This dissertation lays the foundations for a novel reinforcement learning approach for nclustering
that could outperform state of the art algorithms while implementing a more efficient algorithm. To this
end, three libraries were implemented: a synthetic data generator, a framework that models nclustering
tasks as Markov decision process, and a training library. A proximal policybased agent was implemented
and tunned using populationbased training, to evaluate the behaviour of the reinforcement learning en vironments designed.
Results show that agents can learn to modify their behaviour while interacting with the environment
to maximize their reward signal. However, they are still far from being a solution to nclustering. This
dissertation is the first step towards this solution. Finally, future steps to improve these results are pro posed.
This dissertation has presented foundational work that enables modelling nclustering as an MDP,
paving the way for further studies focused on improving task performance.Os seres humanos evoluíram para encontrar padrões. Esta capacidade está presente na nossa vida quotidiana, e não sobreviveríamos sem ela. Na realidade, esta é uma característica que parecemos partilhar
com todos os seres inteligentes, a necessidade de compreender padrões e de criar rotinas. Os padrões são
lugares seguros onde podemos agir conscientemente, onde as relações causais que ligam as nossas acções
às suas consequências são conhecidas por nós. A compreensão de um padrão pode ser a diferença entre
vida e morte, o suave som de folhas pode implicar um ataque mortal, a presença de humidade no solo pode
indicar um riacho próximo, enquanto um cheiro pode ajudar a distinguir entre amigo ou inimigo. Encontrar padrões e distinguir entre padrões e acontecimentos aleatórios permitiu à nossa sociedade chegar tão
longe. Hoje, enfrentamos problemas mais complexos em quase todos os campos de estudo científicos e
sociais, por vezes escondidos por detrás de quantidades massivas de eventos aleatórios. É literalmente
como encontrar uma agulha num palheiro. Como tal, recorremos mais uma vez a máquinas para nos
ajudar neste empreendimento desafiante. Técnicas de aprendizagem sem supervisão começaram a ser propostas por estatísticos e matemáticos muito antes do aparecimento de campos como a prospecção de dados. No entanto, estes campos, juntamente com um significativo interesse restaurado na área pela indústria, na esperança de rentabilizar grandes quantidades de dados guardados ao longo dos anos, deram grandes passos em frente. Nos últimos anos, temos visto muitos avanços notáveis neste campo e uma nova face da inteligência artificial em geral (por exemplo, aprendizagem de máquinas, aprendizagem profunda). Foram propostas abordagens de clusters revigoradas que combinavam técnicas clássicas com aprendizagem profunda para gerar representações precisas e produzir clusters a partir destes vectores de dados.
Biclustering e triclustering estão a tornar-se cada vez mais populares para análises não supervisionadas de conjuntos de dados bidimensionais e tridimensionais. Entre outros padrões de interesse, a utilização de nclusters na análise não supervisionada de dados pode identificar potenciais módulos biológicos,
perfis de progressão de doenças, e comunidades de indivíduos com comportamento consistente.
Nos domínios médicos, as aplicações possíveis incluem a análise de sinais fisiológicos multivariados,
onde os nclusters identificados podem capturar respostas fisiológicas coerentes para um grupo de indivíduos; análise de dados de neuroimagem, onde os nclusters podem capturar funções de resposta hemodinâmica e conectividade entre regiões cerebrais; e análise de registos clínicos, onde os nclusters podem corresponder a grupos de pacientes com características clínicas correlacionadas ao longo do tempo. Relativamente aos domínios sociais, as aplicações possíveis vão desde a análise de redes sociais até à descoberta de comunidades de indivíduos com actividade e interacção correlacionadas (frequentemente referidas como comunidades em evolução coerente) ou conteúdos de grupo de acordo com o perfil do utilizador; grupos de utilizadores com padrões de navegação coerentes nos dados de utilização da web; análise de dados de comércio electrónico para encontrar padrões de navegação ocultos de conjuntos cor relacionados de utilizadores (web), páginas (web) visitadas, e operações ao longo do tempo; análise de
dados de pesquisa de marketing para estudar a utilidade perceptível de vários produtos para diferentes fins,
a julgar por diferentes grupos demográficos; dados de filtragem colaborativa para descobrir correlações
accionáveis para sistemas de recomendação ou utilizadores de grupo com preferências semelhantes, entre
outras aplicações.
O clustering tradicional pode ser utilizado para agrupar observações neste contexto, mas a sua utili dade é limitada porque as observações neste domínio de dados são tipicamente apenas significativamente
correlacionadas em subespaços do espaço global.
Apesar da importância de nclustering, a maioria dos algoritmos continua a basearse em abordagens
exaustivas para produzir resultados de qualidade. Como o nclustering é uma tarefa complexa de opti mização combinatória, as abordagens existentes limitam a estrutura permitida, a coerência e a qualidade
da solução.
A principal limitação da utilização de aprendizagem profunda para resolver esta tarefa é que os n clusters são computados assumindo que todos os elementos são representados sob igual distância. Este
pressuposto invalida o uso de técnicas de simplificação da localidade como as convoluções neurais.
Os grafos são estruturas flexíveis que podem ser utilizadas para representar um conjunto de dados
onde todos os elementos estão a uma distância igual, através de grafos completos, encorajando assim a
utilização de redes convolucionais de grafos para aprender a sua estrutura e gerar representações precisas
dos conjuntos de dados.
Uma vez que o nclustering é visto principalmente como uma tarefa iterativa em que os elemen tos são adicionados ou removidos de um dado cluster, uma estrutura de aprendizagem de reforço é um
bom suporte. Agentes de aprendizagem de reforço profundos já foram acoplados com sucesso a redes
convolucionais de grafos para resolver problemas complexos de otimização combinatória, motivando a
adaptação de arquitecturas de aprendizagem de reforço a este problema.
Esta dissertação lança as bases para uma nova abordagem de aprendizagem por reforço para n clustering que poderia superar os algoritmos de estado da arte, ao mesmo tempo que implementa um
algoritmo mais eficiente.
Para este fim, foram implementadas três bibliotecas: um gerador de dados sintéticos, uma framework
que modela as tarefas de nclustering como um processo de decisão de Markov, e uma biblioteca de treino.
NclustGen foi implementado para melhorar a utilização programática dos geradores de dados sintéti cos de biclustering e triclustering de última geração.
O NclustEnv modela nclustering como um processo de decisão Markov através da implementação de
ambientes de biclustering e triclustering. Segue a interface padrão de programação de aplicações proposta
pelo Gym para ambientes de aprendizagem por reforço.
A implementação de ambientes de qualidade que modelam a interação entre um agente e uma tarefa de nclustering é da maior importância. Ao implementar esta tarefa utilizando o padrão Gym, o ambi ente pode ser implementado como agente agnóstico. Assim, qualquer agente será capaz de treinar neste
ambiente, se correctamente configurado, independentemente da sua implementação. Esta capacidade de
construir ambientes que modelam uma dada tarefa de uma forma agnóstica permite a implementação de
uma framework geral para nclustering baseada em aprendizagem por reforço. Os agentes podem depois
utilizar esta framework de treino para encontrar uma solução de última geração para esta tarefa.
A fim de avaliar o comportamento dos ambientes de aprendizagem por reforço que foram concebidos,
foi implementado e calibrado um agente de optimização proximal de políticas utilizando treino baseado
em populações. Um agente de optimização proximal de políticas foi escolhido porque pode servir como
uma boa base para experiências futuras. Devido à sua versatilidade, os agentes de optimização proximal
de políticas são largamente considerados como os agentes de referência para experiências em ambientes
não explorados. A solução e as limitações alcançadas por este agente normalmente dão pelo menos uma
ideia dos seguintes passos a tomar se o agente não conseguir alcançar uma boa solução.
Os resultados mostram que os agentes podem aprender a modificar o seu comportamento enquanto
interagem com o ambiente para maximizar o seu sinal de recompensa. No entanto, ainda estão longe de
ser uma solução para o nclustering.
Esta dissertação é o primeiro passo para esta solução e apresentou o trabalho fundamental, mas ainda
há muito mais trabalho a ser feito para que esta abordagem possa ultrapassar os algoritmos mais avança dos.Por fim, são propostos os próximos passos para melhorar estes resultados, e que para num futuro
próximo, esta abordagem possa vir a resolver a tarefa do nclustering
Sketch-Based Streaming Anomaly Detection in Dynamic Graphs
Given a stream of graph edges from a dynamic graph, how can we assign anomaly
scores to edges and subgraphs in an online manner, for the purpose of detecting
unusual behavior, using constant time and memory? For example, in intrusion
detection, existing work seeks to detect either anomalous edges or anomalous
subgraphs, but not both. In this paper, we first extend the count-min sketch
data structure to a higher-order sketch. This higher-order sketch has the
useful property of preserving the dense subgraph structure (dense subgraphs in
the input turn into dense submatrices in the data structure). We then propose
four online algorithms that utilize this enhanced data structure, which (a)
detect both edge and graph anomalies; (b) process each edge and graph in
constant memory and constant update time per newly arriving edge, and; (c)
outperform state-of-the-art baselines on four real-world datasets. Our method
is the first streaming approach that incorporates dense subgraph search to
detect graph anomalies in constant memory and time
Computing Large-Scale Matrix and Tensor Decomposition with Structured Factors: A Unified Nonconvex Optimization Perspective
The proposed article aims at offering a comprehensive tutorial for the
computational aspects of structured matrix and tensor factorization. Unlike
existing tutorials that mainly focus on {\it algorithmic procedures} for a
small set of problems, e.g., nonnegativity or sparsity-constrained
factorization, we take a {\it top-down} approach: we start with general
optimization theory (e.g., inexact and accelerated block coordinate descent,
stochastic optimization, and Gauss-Newton methods) that covers a wide range of
factorization problems with diverse constraints and regularization terms of
engineering interest. Then, we go `under the hood' to showcase specific
algorithm design under these introduced principles. We pay a particular
attention to recent algorithmic developments in structured tensor and matrix
factorization (e.g., random sketching and adaptive step size based stochastic
optimization and structure-exploiting second-order algorithms), which are the
state of the art---yet much less touched upon in the literature compared to
{\it block coordinate descent} (BCD)-based methods. We expect that the article
to have an educational values in the field of structured factorization and hope
to stimulate more research in this important and exciting direction.Comment: Final Version; to appear in IEEE Signal Processing Magazine; title
revised to comply with the journal's rul