135 research outputs found
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization
Fine-tuning pre-trained language models (PTLMs), such as BERT and its better
variant RoBERTa, has been a common practice for advancing performance in
natural language understanding (NLU) tasks. Recent advance in representation
learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings
can significantly improve performance on downstream tasks with faster
convergence and better generalization. The isotropy of the pre-trained
embeddings in PTLMs, however, is relatively under-explored. In this paper, we
analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with
straightforward visualization, and point out two major issues: high variance in
their standard deviation, and high correlation between different dimensions. We
also propose a new network regularization method, isotropic batch normalization
(IsoBN) to address the issues, towards learning more isotropic representations
in fine-tuning by dynamically penalizing dominating principal components. This
simple yet effective fine-tuning method yields about 1.0 absolute increment on
the average of seven NLU tasks.Comment: AAAI 202
SNR Enhancement in Brillouin Microspectroscopy using Spectrum Reconstruction
Brillouin imaging suffers from intrinsically low signal-to-noise ratios
(SNR). Such low SNRs can render common data analysis protocols unreliable,
especially for SNRs below . In this work we exploit two denoising
algorithms, namely maximum entropy reconstruction (MER) and wavelet analysis
(WA), to improve the accuracy and precision in determination of Brillouin
shifts and linewidth. Algorithm performance is quantified using Monte-Carlo
simulations and benchmarked against the Cram\'er-Rao lower bound. Superior
estimation results are demonstrated even at low SNRS (). Denoising was
furthermore applied to experimental Brillouin spectra of distilled water at
room temperature, allowing the speed of sound in water to be extracted.
Experimental and theoretical values were found to be consistent to within
at unity SNR
Research on Experiential Marketing Strategy Based on the Sale of Baking Products
To understand the consumer is the prerequisite for the enterprise to enter the consumer market. With the market competition intensified, Traditional marketing strategy is difficult to achieve the expected goal of the enterprise. It needs new marketing theory to guide the market and satisfy the demand. In this paper, the sales of baking products are taken for an example in which 425 questionnaires analyze five models of emotion, culture, service, environment and personality in detail and the paper suppllies some corresponding suggestions & measure to tackle the problems as the lack of large-scale leading enterprises and Enterprise homogeneity
Multivariate analysis of Brillouin imaging data by supervised and unsupervised learning
Brillouin imaging relies on the reliable extraction of subtle spectral
information from hyperspectral datasets. To date, the mainstream practice has
been using line fitting of spectral features to retrieve the average peak shift
and linewidth parameters. Good results, however, depend heavily on sufficient
SNR and may not be applicable in complex samples that consist of spectral
mixtures. In this work, we thus propose the use of various multivariate
algorithms that can be used to perform supervised or unsupervised analysis of
the hyperspectral data, with which we explore advanced image analysis
applications, namely unmixing, classification and segmentation in a phantom and
live cells. The resulting images are shown to provide more contrast and detail,
and obtained on a timescale faster than fitting. The estimated spectral
parameters are consistent with those calculated from pure fitting
Unsupervised Cross-Task Generalization via Retrieval Augmentation
Humans can perform unseen tasks by recalling relevant skills that are
acquired previously and then generalizing them to the target tasks, even if
there is no supervision at all. In this paper, we aim to improve such
cross-task generalization ability of massive multi-task language models such as
T0 (Sanh et al., 2021) in an unsupervised setting. We propose a
retrieval-augmentation method named ReCross that takes a few unlabelled
examples as queries to retrieve a small subset of upstream data and uses them
to update the multi-task model for better generalization. Our empirical results
show that the proposed ReCross consistently outperforms non-retrieval baselines
by a significant margin.Comment: Project website: https://inklab.usc.edu/ReCross
- …