3,028 research outputs found
Autonomous Fault Detection in Self-Healing Systems using Restricted Boltzmann Machines
Autonomously detecting and recovering from faults is one approach for
reducing the operational complexity and costs associated with managing
computing environments. We present a novel methodology for autonomously
generating investigation leads that help identify systems faults, and extends
our previous work in this area by leveraging Restricted Boltzmann Machines
(RBMs) and contrastive divergence learning to analyse changes in historical
feature data. This allows us to heuristically identify the root cause of a
fault, and demonstrate an improvement to the state of the art by showing
feature data can be predicted heuristically beyond a single instance to include
entire sequences of information.Comment: Published and presented in the 11th IEEE International Conference and
Workshops on Engineering of Autonomic and Autonomous Systems (EASe 2014
Show from Tell: Audio-Visual Modelling in Clinical Settings
Auditory and visual signals usually present together and correlate with each
other, not only in natural environments but also in clinical settings. However,
the audio-visual modelling in the latter case can be more challenging, due to
the different sources of audio/video signals and the noise (both signal-level
and semantic-level) in auditory signals -- usually speech. In this paper, we
consider audio-visual modelling in a clinical setting, providing a solution to
learn medical representations that benefit various clinical tasks, without
human expert annotation. A simple yet effective multi-modal self-supervised
learning framework is proposed for this purpose. The proposed approach is able
to localise anatomical regions of interest during ultrasound imaging, with only
speech audio as a reference. Experimental evaluations on a large-scale clinical
multi-modal ultrasound video dataset show that the proposed self-supervised
method learns good transferable anatomical representations that boost the
performance of automated downstream clinical tasks, even outperforming
fully-supervised solutions
Task-Agnostic Graph Neural Network Evaluation via Adversarial Collaboration
It has been increasingly demanding to develop reliable methods to evaluate
the progress of Graph Neural Network (GNN) research for molecular
representation learning. Existing GNN benchmarking methods for molecular
representation learning focus on comparing the GNNs' performances on some
node/graph classification/regression tasks on certain datasets. However, there
lacks a principled, task-agnostic method to directly compare two GNNs.
Additionally, most of the existing self-supervised learning works incorporate
handcrafted augmentations to the data, which has several severe difficulties to
be applied on graphs due to their unique characteristics. To address the
aforementioned issues, we propose GraphAC (Graph Adversarial Collaboration) --
a conceptually novel, principled, task-agnostic, and stable framework for
evaluating GNNs through contrastive self-supervision. We introduce a novel
objective function: the Competitive Barlow Twins, that allow two GNNs to
jointly update themselves from direct competitions against each other. GraphAC
succeeds in distinguishing GNNs of different expressiveness across various
aspects, and has demonstrated to be a principled and reliable GNN evaluation
method, without necessitating any augmentations.Comment: 11th International Conference on Learning Representations (ICLR 2023)
Machine Learning for Drug Discovery (MLDD) Workshop. 17 pages, 6 figures, 4
table
Link Prediction with Non-Contrastive Learning
A recent focal area in the space of graph neural networks (GNNs) is graph
self-supervised learning (SSL), which aims to derive useful node
representations without labeled data. Notably, many state-of-the-art graph SSL
methods are contrastive methods, which use a combination of positive and
negative samples to learn node representations. Owing to challenges in negative
sampling (slowness and model sensitivity), recent literature introduced
non-contrastive methods, which instead only use positive samples. Though such
methods have shown promising performance in node-level tasks, their suitability
for link prediction tasks, which are concerned with predicting link existence
between pairs of nodes (and have broad applicability to recommendation systems
contexts) is yet unexplored. In this work, we extensively evaluate the
performance of existing non-contrastive methods for link prediction in both
transductive and inductive settings. While most existing non-contrastive
methods perform poorly overall, we find that, surprisingly, BGRL generally
performs well in transductive settings. However, it performs poorly in the more
realistic inductive settings where the model has to generalize to links to/from
unseen nodes. We find that non-contrastive models tend to overfit to the
training graph and use this analysis to propose T-BGRL, a novel non-contrastive
framework that incorporates cheap corruptions to improve the generalization
ability of the model. This simple modification strongly improves inductive
performance in 5/6 of our datasets, with up to a 120% improvement in
Hits@50--all with comparable speed to other non-contrastive baselines and up to
14x faster than the best-performing contrastive baseline. Our work imparts
interesting findings about non-contrastive learning for link prediction and
paves the way for future researchers to further expand upon this area.Comment: ICLR 2023. 19 pages, 6 figure
ClaimDistiller: Scientific Claim Extraction with Supervised Contrastive Learning
The growth of scientific papers in the past decades calls for effective claim extraction tools to automatically and accurately locate key claims from unstructured text. Such claims will benefit content-wise aggregated exploration of scientific knowledge beyond the metadata level. One challenge of building such a model is how to effectively use limited labeled training data. In this paper, we compared transfer learning and contrastive learning frameworks in terms of performance, time and training data size. We found contrastive learning has better performance at a lower cost of data across all models. Our contrastive-learning-based model ClaimDistiller has the highest performance, boosting the F1 score of the base models by 3–4%, and achieved an F1=87.45%, improving the state-of-the-art by more than 7% on the same benchmark data previously used for this task. The same phenomenon is observed on another benchmark dataset, and ClaimDistiller consistently has the best performance. Qualitative assessment on a small sample of out-of-domain data indicates that the model generalizes well. Our source codes and datasets can be found here: https://github.com/lamps-lab/sci-claim-distiller
ISD: Self-Supervised Learning by Iterative Similarity Distillation
Recently, contrastive learning has achieved great results in self-supervised
learning, where the main idea is to push two augmentations of an image
(positive pairs) closer compared to other random images (negative pairs). We
argue that not all random images are equal. Hence, we introduce a self
supervised learning algorithm where we use a soft similarity for the negative
images rather than a binary distinction between positive and negative pairs. We
iteratively distill a slowly evolving teacher model to the student model by
capturing the similarity of a query image to some random images and
transferring that knowledge to the student. We argue that our method is less
constrained compared to recent contrastive learning methods, so it can learn
better features. Specifically, our method should handle unbalanced and
unlabeled data better than existing contrastive learning methods, because the
randomly chosen negative set might include many samples that are semantically
similar to the query image. In this case, our method labels them as highly
similar while standard contrastive methods label them as negative pairs. Our
method achieves better results compared to state-of-the-art models like BYOL
and MoCo on transfer learning settings. We also show that our method performs
better in the settings where the unlabeled data is unbalanced. Our code is
available here: https://github.com/UMBCvision/ISD
- …