31 research outputs found
Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer
Unsupervised/self-supervised representation learning in time series is
critical since labeled samples are usually scarce in real-world scenarios.
Existing approaches mainly leverage the contrastive learning framework, which
automatically learns to understand the similar and dissimilar data pairs.
Nevertheless, they are restricted to the prior knowledge of constructing pairs,
cumbersome sampling policy, and unstable performances when encountering
sampling bias. Also, few works have focused on effectively modeling across
temporal-spectral relations to extend the capacity of representations. In this
paper, we aim at learning representations for time series from a new
perspective and propose Cross Reconstruction Transformer (CRT) to solve the
aforementioned problems in a unified way. CRT achieves time series
representation learning through a cross-domain dropping-reconstruction task.
Specifically, we transform time series into the frequency domain and randomly
drop certain parts in both time and frequency domains. Dropping can maximally
preserve the global context compared to cropping and masking. Then a
transformer architecture is utilized to adequately capture the cross-domain
correlations between temporal and spectral information through reconstructing
data in both domains, which is called Dropped Temporal-Spectral Modeling. To
discriminate the representations in global latent space, we propose Instance
Discrimination Constraint to reduce the mutual information between different
time series and sharpen the decision boundaries. Additionally, we propose a
specified curriculum learning strategy to optimize the CRT, which progressively
increases the dropping ratio in the training process.Comment: Accepted by IEEE Transactions on Neural Networks and Learning Systems
(TNNLS
TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
This work summarizes two strategies for completing time-series (TS) tasks
using today's language model (LLM): LLM-for-TS, design and train a fundamental
large model for TS data; TS-for-LLM, enable the pre-trained LLM to handle TS
data. Considering the insufficient data accumulation, limited resources, and
semantic context requirements, this work focuses on TS-for-LLM methods, where
we aim to activate LLM's ability for TS data by designing a TS embedding method
suitable for LLM. The proposed method is named TEST. It first tokenizes TS,
builds an encoder to embed them by instance-wise, feature-wise, and
text-prototype-aligned contrast, and then creates prompts to make LLM more open
to embeddings, and finally implements TS tasks. Experiments are carried out on
TS classification and forecasting tasks using 8 LLMs with different structures
and sizes. Although its results cannot significantly outperform the current
SOTA models customized for TS tasks, by treating LLM as the pattern machine, it
can endow LLM's ability to process TS data without compromising the language
ability. This paper is intended to serve as a foundational work that will
inspire further research.Comment: 10 pages, 6 figure
Spatial Autoregressive Coding for Graph Neural Recommendation
Graph embedding methods including traditional shallow models and deep Graph
Neural Networks (GNNs) have led to promising applications in recommendation.
Nevertheless, shallow models especially random-walk-based algorithms fail to
adequately exploit neighbor proximity in sampled subgraphs or sequences due to
their optimization paradigm. GNN-based algorithms suffer from the insufficient
utilization of high-order information and easily cause over-smoothing problems
when stacking too much layers, which may deteriorate the recommendations of
low-degree (long-tail) items, limiting the expressiveness and scalability. In
this paper, we propose a novel framework SAC, namely Spatial Autoregressive
Coding, to solve the above problems in a unified way. To adequately leverage
neighbor proximity and high-order information, we design a novel spatial
autoregressive paradigm. Specifically, we first randomly mask multi-hop
neighbors and embed the target node by integrating all other surrounding
neighbors with an explicit multi-hop attention. Then we reinforce the model to
learn a neighbor-predictive coding for the target node by contrasting the
coding and the masked neighbors' embedding, equipped with a new hard negative
sampling strategy. To learn the minimal sufficient representation for the
target-to-neighbor prediction task and remove the redundancy of neighbors, we
devise Neighbor Information Bottleneck by maximizing the mutual information
between target predictive coding and the masked neighbors' embedding, and
simultaneously constraining those between the coding and surrounding neighbors'
embedding. Experimental results on both public recommendation datasets and a
real scenario web-scale dataset Douyin-Friend-Recommendation demonstrate the
superiority of SAC compared with state-of-the-art methods.Comment: preprin