5,485 research outputs found
Meta-learning framework with applications to zero-shot time-series forecasting
Can meta-learning discover generic ways of processing time series (TS) from a
diverse dataset so as to greatly improve generalization on new TS coming from
different datasets? This work provides positive evidence to this using a broad
meta-learning framework which we show subsumes many existing meta-learning
algorithms. Our theoretical analysis suggests that residual connections act as
a meta-learning adaptation mechanism, generating a subset of task-specific
parameters based on a given TS input, thus gradually expanding the expressive
power of the architecture on-the-fly. The same mechanism is shown via
linearization analysis to have the interpretation of a sequential update of the
final linear layer. Our empirical results on a wide range of data emphasize the
importance of the identified meta-learning mechanisms for successful zero-shot
univariate forecasting, suggesting that it is viable to train a neural network
on a source TS dataset and deploy it on a different target TS dataset without
retraining, resulting in performance that is at least as good as that of
state-of-practice univariate forecasting models
Learning models for semantic classification of insufficient plantar pressure images
Establishing a reliable and stable model to predict a target by using insufficient labeled samples is feasible and
effective, particularly, for a sensor-generated data-set. This paper has been inspired with insufficient data-set
learning algorithms, such as metric-based, prototype networks and meta-learning, and therefore we propose
an insufficient data-set transfer model learning method. Firstly, two basic models for transfer learning are
introduced. A classification system and calculation criteria are then subsequently introduced. Secondly, a dataset
of plantar pressure for comfort shoe design is acquired and preprocessed through foot scan system; and by
using a pre-trained convolution neural network employing AlexNet and convolution neural network (CNN)-
based transfer modeling, the classification accuracy of the plantar pressure images is over 93.5%. Finally,
the proposed method has been compared to the current classifiers VGG, ResNet, AlexNet and pre-trained
CNN. Also, our work is compared with known-scaling and shifting (SS) and unknown-plain slot (PS) partition
methods on the public test databases: SUN, CUB, AWA1, AWA2, and aPY with indices of precision (tr, ts, H)
and time (training and evaluation). The proposed method for the plantar pressure classification task shows high
performance in most indices when comparing with other methods. The transfer learning-based method can be
applied to other insufficient data-sets of sensor imaging fields
An Adaptive Approach for Probabilistic Wind Power Forecasting Based on Meta-Learning
This paper studies an adaptive approach for probabilistic wind power
forecasting (WPF) including offline and online learning procedures. In the
offline learning stage, a base forecast model is trained via inner and outer
loop updates of meta-learning, which endows the base forecast model with
excellent adaptability to different forecast tasks, i.e., probabilistic WPF
with different lead times or locations. In the online learning stage, the base
forecast model is applied to online forecasting combined with incremental
learning techniques. On this basis, the online forecast takes full advantage of
recent information and the adaptability of the base forecast model. Two
applications are developed based on our proposed approach concerning
forecasting with different lead times (temporal adaptation) and forecasting for
newly established wind farms (spatial adaptation), respectively. Numerical
tests were conducted on real-world wind power data sets. Simulation results
validate the advantages in adaptivity of the proposed methods compared with
existing alternatives
Meta-Transformer: A Unified Framework for Multimodal Learning
Multimodal learning aims to build models that can process and relate
information from multiple modalities. Despite years of development in this
field, it still remains challenging to design a unified network for processing
various modalities ( natural language, 2D images, 3D point
clouds, audio, video, time series, tabular data) due to the inherent gaps among
them. In this work, we propose a framework, named Meta-Transformer, that
leverages a encoder to perform multimodal perception without
any paired multimodal training data. In Meta-Transformer, the raw input data
from various modalities are mapped into a shared token space, allowing a
subsequent encoder with frozen parameters to extract high-level semantic
features of the input data. Composed of three main components: a unified data
tokenizer, a modality-shared encoder, and task-specific heads for downstream
tasks, Meta-Transformer is the first framework to perform unified learning
across 12 modalities with unpaired data. Experiments on different benchmarks
reveal that Meta-Transformer can handle a wide range of tasks including
fundamental perception (text, image, point cloud, audio, video), practical
application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
tabular, and time-series). Meta-Transformer indicates a promising future for
developing unified multimodal intelligence with transformers. Code will be
available at https://github.com/invictus717/MetaTransformerComment: Project website: https://kxgong.github.io/meta_transformer
One Fits All:Power General Time Series Analysis by Pretrained LM
Although we have witnessed great success of pre-trained models in natural
language processing (NLP) and computer vision (CV), limited progress has been
made for general time series analysis. Unlike NLP and CV where a unified model
can be used to perform different tasks, specially designed approach still
dominates in each time series analysis task such as classification, anomaly
detection, forecasting, and few-shot learning. The main challenge that blocks
the development of pre-trained model for time series analysis is the lack of a
large amount of data for training. In this work, we address this challenge by
leveraging language or CV models, pre-trained from billions of tokens, for time
series analysis. Specifically, we refrain from altering the self-attention and
feedforward layers of the residual blocks in the pre-trained language or image
model. This model, known as the Frozen Pretrained Transformer (FPT), is
evaluated through fine-tuning on all major types of tasks involving time
series. Our results demonstrate that pre-trained models on natural language or
images can lead to a comparable or state-of-the-art performance in all main
time series analysis tasks, as illustrated in Figure 1. We also found both
theoretically and empirically that the self-attention module behaviors
similarly to principle component analysis (PCA), an observation that helps
explains how transformer bridges the domain gap and a crucial step towards
understanding the universality of a pre-trained transformer.The code is
publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.Comment: Neurips 2023 Spotligh
Entity Aware Modelling: A Survey
Personalized prediction of responses for individual entities caused by
external drivers is vital across many disciplines. Recent machine learning (ML)
advances have led to new state-of-the-art response prediction models. Models
built at a population level often lead to sub-optimal performance in many
personalized prediction settings due to heterogeneity in data across entities
(tasks). In personalized prediction, the goal is to incorporate inherent
characteristics of different entities to improve prediction performance. In
this survey, we focus on the recent developments in the ML community for such
entity-aware modeling approaches. ML algorithms often modulate the network
using these entity characteristics when they are readily available. However,
these entity characteristics are not readily available in many real-world
scenarios, and different ML methods have been proposed to infer these
characteristics from the data. In this survey, we have organized the current
literature on entity-aware modeling based on the availability of these
characteristics as well as the amount of training data. We highlight how recent
innovations in other disciplines, such as uncertainty quantification, fairness,
and knowledge-guided machine learning, can improve entity-aware modeling.Comment: Submitted to IJCAI, Survey Trac
Semantic Attributes for Transfer Learning in Visual Recognition
Angetrieben durch den Erfolg von Deep Learning Verfahren wurden in Bezug auf künstliche Intelligenz erhebliche Fortschritte im Bereich des Maschinenverstehens gemacht. Allerdings sind Tausende von manuell annotierten Trainingsdaten zwingend notwendig, um die Generalisierungsfähigkeit solcher Modelle sicherzustellen. Darüber hinaus muss das Modell jedes Mal komplett neu trainiert werden, sobald es auf eine neue Problemklasse angewandt werden muss. Dies führt wiederum dazu, dass der sehr kostenintensive Prozess des Sammelns und Annotierens von Trainingsdaten wiederholt werden muss, wodurch die Skalierbarkeit solcher Modelle erheblich begrenzt wird. Auf der anderen Seite bearbeiten wir Menschen neue Aufgaben nicht isoliert, sondern haben die bemerkenswerte Fähigkeit, auf bereits erworbenes Wissen bei der Lösung neuer Probleme zurückzugreifen. Diese Fähigkeit wird als Transfer-Learning bezeichnet. Sie ermöglicht es uns, schneller, besser und anhand nur sehr weniger Beispiele Neues zu lernen. Daher besteht ein großes Interesse, diese Fähigkeit durch Algorithmen nachzuahmen, insbesondere in Bereichen, in denen Trainingsdaten sehr knapp oder sogar nicht verfügbar sind.
In dieser Arbeit untersuchen wir Transfer-Learning im Kontext von Computer Vision. Insbesondere untersuchen wir, wie visuelle Erkennung (z.B. Objekt- oder Aktionsklassifizierung) durchgeführt werden kann, wenn nur wenige oder keine Trainingsbeispiele existieren. Eine vielversprechende Lösung in dieser Richtung ist das Framework der semantischen Attribute. Dabei werden visuelle Kategorien in Form von Attributen wie Farbe, Muster und Form beschrieben. Diese Attribute können aus einer disjunkten Menge von Trainingsbeispielen gelernt werden. Da die Attribute eine doppelte, d.h. sowohl visuelle als auch semantische, Interpretation haben, kann Sprache effektiv genutzt werden, um den Übertragungsprozess zu steuern. Dies bedeutet, dass Modelle für eine neue visuelle Kategorie nur anhand der sprachlichen Beschreibung erstellt werden können, indem relevante Attribute selektiert und auf die neue Kategorie übertragen werden. Die Notwendigkeit von Trainingsbildern entfällt durch diesen Prozess jedoch vollständig. In dieser Arbeit stellen wir neue Lösungen vor, semantische Attribute zu modellieren, zu übertragen, automatisch mit visuellen Kategorien zu assoziieren, und aus sprachlichen Beschreibungen zu erkennen. Zu diesem Zweck beleuchten wir die attributbasierte Erkennung aus den folgenden vier Blickpunkten:
1) Anders als das gängige Modell, bei dem Attribute global gelernt werden müssen, stellen wir einen hierarchischen Ansatz vor, der es ermöglicht, die Attribute auf verschiedenen Abstraktionsebenen zu lernen. Wir zeigen zudem, wie die Struktur zwischen den Kategorien effektiv genutzt werden kann, um den Lern- und Transferprozess zu steuern und damit diskriminative Modelle für neue Kategorien zu erstellen. Mit einer gründlichen experimentellen Analyse demonstrieren wir eine deutliche Verbesserung unseres Modells gegenüber dem globalen Ansatz, insbesondere bei der Erkennung detailgenauer Kategorien.
2) In vorherrschend attributbasierten Transferansätzen überwacht der Benutzer die Zuordnung zwischen den Attributen und den Kategorien. Wir schlagen in dieser Arbeit vor, die Verbindung zwischen den beiden automatisch und ohne Benutzereingriff herzustellen. Unser Modell erfasst die semantischen Beziehungen, welche die Attribute mit Objekten koppeln, um ihre Assoziationen vorherzusagen und unüberwacht auszuwählen welche Attribute übertragen werden sollen.
3) Wir umgehen die Notwendigkeit eines vordefinierten Vokabulars von Attributen. Statt dessen schlagen wir vor, Enyzklopädie-Artikel zu verwenden, die Objektkategorien in einem freien Text beschreiben, um automatisch eine Menge von diskriminanten, salienten und vielfältigen Attributen zu entdecken. Diese Beseitigung des Bedarfs eines benutzerdefinierten Vokabulars ermöglicht es uns, das Potenzial attributbasierter Modelle im Kontext sehr großer Datenmengen vollends auszuschöpfen.
4) Wir präsentieren eine neuartige Anwendung semantischer Attribute in der realen Welt. Wir schlagen das erste Verfahren vor, welches automatisch Modestile lernt, und vorhersagt, wie sich ihre Beliebtheit in naher Zukunft entwickeln wird. Wir zeigen, dass semantische Attribute interpretierbare Modestile liefern und zu einer besseren Vorhersage der Beliebtheit von visuellen Stilen im Vergleich zu anderen Darstellungen führen
Networkwide Traffic State Forecasting Using Exogenous Information: A Multi-Dimensional Graph Attention-Based Approach
Traffic state forecasting is crucial for traffic management and control
strategies, as well as user- and system-level decision making in the
transportation network. While traffic forecasting has been approached with a
variety of techniques over the last couple of decades, most approaches simply
rely on endogenous traffic variables for state prediction, despite the evidence
that exogenous factors can significantly impact traffic conditions. This paper
proposes a multi-dimensional spatio-temporal graph attention-based traffic
prediction approach (M-STGAT), which predicts traffic based on past
observations of speed, along with lane closure events, temperature, and
visibility across the transportation network. The approach is based on a graph
attention network architecture, which also learns based on the structure of the
transportation network on which these variables are observed. Numerical
experiments are performed using traffic speed and lane closure data from the
California Department of Transportation (Caltrans) Performance Measurement
System (PeMS). The corresponding weather data were downloaded from the National
Oceanic and Atmospheric Administration (NOOA) Automated Surface Observing
Systems (ASOS). For comparison, the numerical experiments implement three
alternative models which do not allow for the multi-dimensional input. The
M-STGAT is shown to outperform the three alternative models, when performing
tests using our primary data set for prediction with a 30-, 45-, and 60-minute
prediction horizon, in terms of three error measures: Mean Absolute Error
(MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE).
However, the model's transferability can vary for different transfer data sets
and this aspect may require further investigation.Comment: Transportation Research Board Annual Meeting 202
- …