134 research outputs found
Auto-tune: PAC-Bayes Optimization over Prior and Posterior for Neural Networks
It is widely recognized that the generalization ability of neural networks
can be greatly enhanced through carefully designing the training procedure. The
current state-of-the-art training approach involves utilizing stochastic
gradient descent (SGD) or Adam optimization algorithms along with a combination
of additional regularization techniques such as weight decay, dropout, or noise
injection. Optimal generalization can only be achieved by tuning a multitude of
hyperparameters through grid search, which can be time-consuming and
necessitates additional validation datasets. To address this issue, we
introduce a practical PAC-Bayes training framework that is nearly tuning-free
and requires no additional regularization while achieving comparable testing
performance to that of SGD/Adam after a complete grid search and with extra
regularizations. Our proposed algorithm demonstrates the remarkable potential
of PAC training to achieve state-of-the-art performance on deep neural networks
with enhanced robustness and interpretability.Comment: 30 pages, 15 figures, 7 table
Exploring the role of the transactive memory system in virtual team resilience: Evidence from online medical teams
The capacity to resist and recover from challenges and adversities (i.e., resilience capacity) is critical for a virtual team to survive. However, our knowledge of what influence the development of resilient virtual teams have yet to be fully developed. Drawing on the transactive memory system (TMS) theory, we propose that TMS will enhance a virtual team resilience capacity. Applying discontinuous growth modeling, results of an empirical study involving 1974 online medical teams from a popular online healthcare platform in China provide available evidence. We found inconsistent effects of the three dimensions of TMS on online medical team resilience capacity. Specifically, specialization shows no significant impact. Credibility can enhance online medical team resilience capacity for both process and outcome performance. For coordination, voice centralization positively affects online medical team resilience capacity for process performance. These findings advance virtual team resilience literature and inform practitioners about how to build resilient virtual teams
Enhanced prediction accuracy with uncertainty quantification in monitoring CO2 sequestration using convolutional neural networks
Monitoring changes inside a reservoir in real time is crucial for the success
of CO2 injection and long-term storage. Machine learning (ML) is well-suited
for real-time CO2 monitoring because of its computational efficiency. However,
most existing applications of ML yield only one prediction (i.e., the
expectation) for a given input, which may not properly reflect the distribution
of the testing data, if it has a shift with respect to that of the training
data. The Simultaneous Quantile Regression (SQR) method can estimate the entire
conditional distribution of the target variable of a neural network via pinball
loss. Here, we incorporate this technique into seismic inversion for purposes
of CO2 monitoring. The uncertainty map is then calculated pixel by pixel from a
particular prediction interval around the median. We also propose a novel
data-augmentation method by sampling the uncertainty to further improve
prediction accuracy. The developed methodology is tested on synthetic
Kimberlina data, which are created by the Department of Energy and based on a
CO2 capture and sequestration (CCS) project in California. The results prove
that the proposed network can estimate the subsurface velocity rapidly and with
sufficient resolution. Furthermore, the computed uncertainty quantifies the
prediction accuracy. The method remains robust even if the testing data are
distorted due to problems in the field data acquisition. Another test
demonstrates the effectiveness of the developed data-augmentation method in
increasing the spatial resolution of the estimated velocity field and in
reducing the prediction error.Comment: 42 pages (double-space), 14 figures, 1 tabl
PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descent
Fine-tuning pretrained language models (PLMs) for downstream tasks is a
large-scale optimization problem, in which the choice of the training algorithm
critically determines how well the trained model can generalize to unseen test
data, especially in the context of few-shot learning. To achieve good
generalization performance and avoid overfitting, techniques such as data
augmentation and pruning are often applied. However, adding these
regularizations necessitates heavy tuning of the hyperparameters of
optimization algorithms, such as the popular Adam optimizer. In this paper, we
propose a two-stage fine-tuning method, PAC-tuning, to address this
optimization challenge. First, based on PAC-Bayes training, PAC-tuning directly
minimizes the PAC-Bayes generalization bound to learn proper parameter
distribution. Second, PAC-tuning modifies the gradient by injecting noise with
the variance learned in the first stage into the model parameters during
training, resulting in a variant of perturbed gradient descent (PGD). In the
past, the few-shot scenario posed difficulties for PAC-Bayes training because
the PAC-Bayes bound, when applied to large models with limited training data,
might not be stringent. Our experimental results across 5 GLUE benchmark tasks
demonstrate that PAC-tuning successfully handles the challenges of fine-tuning
tasks and outperforms strong baseline methods by a visible margin, further
confirming the potential to apply PAC training for any other settings where the
Adam optimizer is currently used for training.Comment: Accepted to EMNLP23 mai
Can Directed Graph Neural Networks be Adversarially Robust?
The existing research on robust Graph Neural Networks (GNNs) fails to
acknowledge the significance of directed graphs in providing rich information
about networks' inherent structure. This work presents the first investigation
into the robustness of GNNs in the context of directed graphs, aiming to
harness the profound trust implications offered by directed graphs to bolster
the robustness and resilience of GNNs. Our study reveals that existing directed
GNNs are not adversarially robust. In pursuit of our goal, we introduce a new
and realistic directed graph attack setting and propose an innovative,
universal, and efficient message-passing framework as a plug-in layer to
significantly enhance the robustness of GNNs. Combined with existing defense
strategies, this framework achieves outstanding clean accuracy and
state-of-the-art robust performance, offering superior defense against both
transfer and adaptive attacks. The findings in this study reveal a novel and
promising direction for this crucial research area. The code will be made
publicly available upon the acceptance of this work
- …