2,375 research outputs found
Adaptive DCTNet for Audio Signal Classification
In this paper, we investigate DCTNet for audio signal classification. Its
output feature is related to Cohen's class of time-frequency distributions. We
introduce the use of adaptive DCTNet (A-DCTNet) for audio signals feature
extraction. The A-DCTNet applies the idea of constant-Q transform, with its
center frequencies of filterbanks geometrically spaced. The A-DCTNet is
adaptive to different acoustic scales, and it can better capture low frequency
acoustic information that is sensitive to human audio perception than features
such as Mel-frequency spectral coefficients (MFSC). We use features extracted
by the A-DCTNet as input for classifiers. Experimental results show that the
A-DCTNet and Recurrent Neural Networks (RNN) achieve state-of-the-art
performance in bird song classification rate, and improve artist identification
accuracy in music data. They demonstrate A-DCTNet's applicability to signal
processing problems.Comment: International Conference of Acoustic and Speech Signal Processing
(ICASSP). New Orleans, United States, March, 201
Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs
As foundation models continue to exponentially scale in size, efficient
methods of adaptation become increasingly critical. Parameter-efficient
fine-tuning (PEFT), a recent class of techniques that require only modifying a
small percentage of the model parameters, is currently the most popular method
for adapting large language models (LLMs). Several PEFT techniques have
recently been proposed with varying tradeoffs. We provide a comprehensive and
uniform benchmark of various PEFT techniques across a representative LLM, the
FLAN-T5 model, and evaluate model performance across different data scales of
classification and generation datasets. Based on this, we provide a framework
for choosing the optimal fine-tuning techniques given the task type and data
availability. Contrary to popular belief, we also empirically prove that PEFT
techniques converge slower than full tuning in low data scenarios, and posit
the amount of data required for PEFT methods to both perform well and converge
efficiently. Lastly, we further optimize these PEFT techniques by selectively
choosing which parts of the model to train, and find that these techniques can
be applied with significantly fewer parameters while maintaining and even
improving performance.Comment: Short paper, ICLR '23 Workshop on Understanding Foundation Model
From Node Interaction to Hop Interaction: New Effective and Scalable Graph Learning Paradigm
Existing Graph Neural Networks (GNNs) follow the message-passing mechanism
that conducts information interaction among nodes iteratively. While
considerable progress has been made, such node interaction paradigms still have
the following limitation. First, the scalability limitation precludes the wide
application of GNNs in large-scale industrial settings since the node
interaction among rapidly expanding neighbors incurs high computation and
memory costs. Second, the over-smoothing problem restricts the discrimination
ability of nodes, i.e., node representations of different classes will converge
to indistinguishable after repeated node interactions. In this work, we propose
a novel hop interaction paradigm to address these limitations simultaneously.
The core idea of hop interaction is to convert the target of message-passing
from nodes into multi-hop features inside each node. Specifically, it first
pre-computed multi-hop features of nodes to reduce computation costs during
training and inference. Then, it conducts a non-linear interaction among
multi-hop features to enhance the discrimination of nodes. We design a simple
yet effective HopGNN framework that can easily utilize existing GNNs to achieve
hop interaction. Furthermore, we propose a multi-task learning strategy with a
self-supervised learning objective to enhance HopGNN. We conduct extensive
experiments on 12 benchmark datasets in a wide range of domains, scales, and
smoothness of graphs. Experimental results show that our methods achieve
superior performance while maintaining high scalability and efficiency
- …