2,726 research outputs found
Convergence of program transformers in the metric space of trees
AbstractIn recent years increasing consensus has emerged that program transformers, e.g. partial evaluation and unfold/fold transformations, should terminate; a compiler should stop even if it performs fancy optimizations! A number of techniques to ensure termination of program transformers have been invented, but their correctness proofs are sometimes long and involved. We present a framework for proving termination of program transformers, cast in the metric space of trees. We first introduce the notion of an abstract program transformer; a number of well-known program transformers can be viewed as instances of this notion. We then formalize what it means that an abstract program transformer terminates and give a general sufficient condition for an abstract program transformer to terminate. We also consider some specific techniques for satisfying the condition. As applications we show that termination of some well-known program transformers either follows directly from the specific techniques or is easy to establish using the general condition. Our framework facilitates simple termination proofs for program transformers. Also, since our framework is independent of the language being transformed, a single correctness proof can be given in our framework for program transformers that use essentially the same technique in the context of different languages. Moreover, it is easy to extend termination proofs for program transformers to accommodate changes to these transformers. Finally, the framework may prove useful for designing new termination techniques for program transformers
An ADMM Based Framework for AutoML Pipeline Configuration
We study the AutoML problem of automatically configuring machine learning
pipelines by jointly selecting algorithms and their appropriate
hyper-parameters for all steps in supervised learning pipelines. This black-box
(gradient-free) optimization with mixed integer & continuous variables is a
challenging problem. We propose a novel AutoML scheme by leveraging the
alternating direction method of multipliers (ADMM). The proposed framework is
able to (i) decompose the optimization problem into easier sub-problems that
have a reduced number of variables and circumvent the challenge of mixed
variable categories, and (ii) incorporate black-box constraints along-side the
black-box optimization objective. We empirically evaluate the flexibility (in
utilizing existing AutoML techniques), effectiveness (against open source
AutoML toolkits),and unique capability (of executing AutoML with practically
motivated black-box constraints) of our proposed scheme on a collection of
binary classification data sets from UCI ML& OpenML repositories. We observe
that on an average our framework provides significant gains in comparison to
other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical
advantages of this framework
Code Prediction by Feeding Trees to Transformers
We advance the state-of-the-art in the accuracy of code prediction (next
token prediction) used in autocomplete systems. First, we report that using the
recently proposed Transformer architecture even out-of-the-box outperforms
previous neural and non-neural systems for code prediction. We then show that
by making the Transformer architecture aware of the syntactic structure of
code, we further increase the margin by which a Transformer-based system
outperforms previous systems. With this, it outperforms the accuracy of an
RNN-based system (similar to Hellendoorn et al. 2018) by 18.3\%, the Deep3
system (Raychev et al 2016) by 14.1\%, and an adaptation of Code2Seq (Alon et
al., 2018) for code prediction by 14.4\%.
We present in the paper several ways of communicating the code structure to
the Transformer, which is fundamentally built for processing sequence data. We
provide a comprehensive experimental evaluation of our proposal, along with
alternative design choices, on a standard Python dataset, as well as on a
Facebook internal Python corpus. Our code and data preparation pipeline will be
available in open source
Entropy-based feature extraction for electromagnetic discharges classification in high-voltage power generation
This work exploits four entropy measures known as Sample, Permutation, Weighted Permutation, and Dispersion Entropy to extract relevant information from Electromagnetic Interference (EMI) discharge signals that are useful in fault diagnosis of High-Voltage (HV) equipment. Multi-class classification algorithms are used to classify or distinguish between various discharge sources such as Partial Discharges (PD), Exciter, Arcing, micro Sparking and Random Noise. The signals were measured and recorded on different sites followed by EMI expert’s data analysis in order to identify and label the discharge source type contained within the signal. The classification was performed both within each site and across all sites. The system performs well for both cases with extremely high classification accuracy within site. This work demonstrates the ability to extract relevant entropy-based features from EMI discharge sources from time-resolved signals requiring minimal computation making the system ideal for a potential application to online condition monitoring based on EMI
Small Transformers Compute Universal Metric Embeddings
We study representations of data from an arbitrary metric space
in the space of univariate Gaussian mixtures with a transport metric (Delon and
Desolneux 2020). We derive embedding guarantees for feature maps implemented by
small neural networks called \emph{probabilistic transformers}. Our guarantees
are of memorization type: we prove that a probabilistic transformer of depth
about and width about can bi-H\"{o}lder embed any -point
dataset from with low metric distortion, thus avoiding the curse
of dimensionality. We further derive probabilistic bi-Lipschitz guarantees,
which trade off the amount of distortion and the probability that a randomly
chosen pair of points embeds with that distortion. If 's geometry
is sufficiently regular, we obtain stronger, bi-Lipschitz guarantees for all
points in the dataset. As applications, we derive neural embedding guarantees
for datasets from Riemannian manifolds, metric trees, and certain types of
combinatorial graphs. When instead embedding into multivariate Gaussian
mixtures, we show that probabilistic transformers can compute bi-H\"{o}lder
embeddings with arbitrarily small distortion.Comment: 42 pages, 10 Figures, 3 Table
Prediction of Post-Operative Renal and Pulmonary Complication Using Transformers
Postoperative complications pose a significant challenge in the healthcare
industry, resulting in elevated healthcare expenses and prolonged hospital
stays, and in rare instances, patient mortality. To improve patient outcomes
and reduce healthcare costs, healthcare providers rely on various perioperative
risk scores to guide clinical decisions and prioritize care. In recent years,
machine learning techniques have shown promise in predicting postoperative
complications and fatality, with deep learning models achieving remarkable
success in healthcare applications. However, research on the application of
deep learning models to intra-operative anesthesia management data is limited.
In this paper, we evaluate the performance of transformer-based models in
predicting postoperative acute renal failure, postoperative pulmonary
complications, and postoperative in-hospital mortality. We compare our method's
performance with state-of-the-art tabular data prediction models, including
gradient boosting trees and sequential attention models, on a clinical dataset.
Our results demonstrate that transformer-based models can achieve superior
performance in predicting postoperative complications and outperform
traditional machine learning models. This work highlights the potential of deep
learning techniques, specifically transformer-based models, in revolutionizing
the healthcare industry's approach to postoperative care
- …