5,579 research outputs found
Knowledge Distillation and Continual Learning for Optimized Deep Neural Networks
Over the past few years, deep learning (DL) has been achieving state-of-theart performance on various human tasks such as speech generation, language translation, image segmentation, and object detection. While traditional machine learning models require hand-crafted features, deep learning algorithms can automatically extract discriminative features and learn complex knowledge from large datasets. This powerful learning ability makes deep learning models attractive to both academia and big corporations.
Despite their popularity, deep learning methods still have two main limitations: large memory consumption and catastrophic knowledge forgetting. First, DL algorithms use very deep neural networks (DNNs) with many billion parameters, which have a big model size and a slow inference speed. This restricts the application of DNNs in resource-constraint devices such as mobile phones and autonomous vehicles. Second, DNNs are known to suffer from catastrophic forgetting. When incrementally learning new tasks, the model performance on old tasks significantly drops. The ability to accommodate new knowledge while retaining previously learned knowledge is called continual learning. Since the realworld environments in which the model operates are always evolving, a robust neural network needs to have this continual learning ability for adapting to new changes
A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods
Multi-task learning (MTL) has become increasingly popular in natural language
processing (NLP) because it improves the performance of related tasks by
exploiting their commonalities and differences. Nevertheless, it is still not
understood very well how multi-task learning can be implemented based on the
relatedness of training tasks. In this survey, we review recent advances of
multi-task learning methods in NLP, with the aim of summarizing them into two
general multi-task training methods based on their task relatedness: (i) joint
training and (ii) multi-step training. We present examples in various NLP
downstream applications, summarize the task relationships and discuss future
directions of this promising topic.Comment: Accepted to EACL 2023 as regular long pape
International spillovers of US unconventional monetary policy to emerging market economies
This paper analyses whether US unconventional monetary policy (UMP) shocks contribute to the global financial and macroeconomic conditions in EMEs. Using global VAR models, we assessed the possible effects of US UMP on financial and macroeconomic conditions in EMEs and documented the credit channels through which potential spillovers occur, focusing on cross-border portfolio flows. We found that US UMP leads to an increase in bond outflows, in turn, the rise of inflows to EMEs results in a significant response by financial variables, indicating that US UMP generates sizable spillovers by financial terms in EMEs. While these results represent commonalities within a country, there is evidence of cross-country heterogeneity. The magnitude of spillovers depends on the EMEs' trade integration, exchange rate regime, and financial market development. The results of this thesis suggest that EMEs’ policymakers could mitigate their financial vulnerability to US UMP by fostering flexibility of exchange rates as well as domestic financial market development, while such policy might reduce long-run growth
Precise Facial Landmark Detection by Reference Heatmap Transformer
Most facial landmark detection methods predict landmarks by mapping the input
facial appearance features to landmark heatmaps and have achieved promising
results. However, when the face image is suffering from large poses, heavy
occlusions and complicated illuminations, they cannot learn discriminative
feature representations and effective facial shape constraints, nor can they
accurately predict the value of each element in the landmark heatmap, limiting
their detection accuracy. To address this problem, we propose a novel Reference
Heatmap Transformer (RHT) by introducing reference heatmap information for more
precise facial landmark detection. The proposed RHT consists of a Soft
Transformation Module (STM) and a Hard Transformation Module (HTM), which can
cooperate with each other to encourage the accurate transformation of the
reference heatmap information and facial shape constraints. Then, a Multi-Scale
Feature Fusion Module (MSFFM) is proposed to fuse the transformed heatmap
features and the semantic features learned from the original face images to
enhance feature representations for producing more accurate target heatmaps. To
the best of our knowledge, this is the first study to explore how to enhance
facial landmark detection by transforming the reference heatmap information.
The experimental results from challenging benchmark datasets demonstrate that
our proposed method outperforms the state-of-the-art methods in the literature.Comment: Accepted by IEEE Transactions on Image Processing, March 202
Synthetic Aperture Radar (SAR) Meets Deep Learning
This reprint focuses on the application of the combination of synthetic aperture radars and depth learning technology. It aims to further promote the development of SAR image intelligent interpretation technology. A synthetic aperture radar (SAR) is an important active microwave imaging sensor, whose all-day and all-weather working capacity give it an important place in the remote sensing community. Since the United States launched the first SAR satellite, SAR has received much attention in the remote sensing community, e.g., in geological exploration, topographic mapping, disaster forecast, and traffic monitoring. It is valuable and meaningful, therefore, to study SAR-based remote sensing applications. In recent years, deep learning represented by convolution neural networks has promoted significant progress in the computer vision community, e.g., in face recognition, the driverless field and Internet of things (IoT). Deep learning can enable computational models with multiple processing layers to learn data representations with multiple-level abstractions. This can greatly improve the performance of various applications. This reprint provides a platform for researchers to handle the above significant challenges and present their innovative and cutting-edge research results when applying deep learning to SAR in various manuscript types, e.g., articles, letters, reviews and technical reports
Neural inference search for multiloss segmentation models
Semantic segmentation is vital for many emerging surveillance applications, but current models cannot be relied upon to meet the required tolerance, particularly in complex tasks that involve multiple classes and varied environments. To improve performance, we propose a novel algorithm, neural inference search (NIS), for hyperparameter optimization pertaining to established deep learning segmentation models in conjunction with a new multiloss function. It incorporates three novel search behaviors, i.e., Maximized Standard Deviation Velocity Prediction, Local Best Velocity Prediction, and n -dimensional Whirlpool Search. The first two behaviors are exploratory, leveraging long short-term memory (LSTM)-convolutional neural network (CNN)-based velocity predictions, while the third employs n -dimensional matrix rotation for local exploitation. A scheduling mechanism is also introduced in NIS to manage the contributions of these three novel search behaviors in stages. NIS optimizes learning and multiloss parameters simultaneously. Compared with state-of-the-art segmentation methods and those optimized with other well-known search algorithms, NIS-optimized models show significant improvements across multiple performance metrics on five segmentation datasets. NIS also reliably yields better solutions as compared with a variety of search methods for solving numerical benchmark functions
A review of technical factors to consider when designing neural networks for semantic segmentation of Earth Observation imagery
Semantic segmentation (classification) of Earth Observation imagery is a
crucial task in remote sensing. This paper presents a comprehensive review of
technical factors to consider when designing neural networks for this purpose.
The review focuses on Convolutional Neural Networks (CNNs), Recurrent Neural
Networks (RNNs), Generative Adversarial Networks (GANs), and transformer
models, discussing prominent design patterns for these ANN families and their
implications for semantic segmentation. Common pre-processing techniques for
ensuring optimal data preparation are also covered. These include methods for
image normalization and chipping, as well as strategies for addressing data
imbalance in training samples, and techniques for overcoming limited data,
including augmentation techniques, transfer learning, and domain adaptation. By
encompassing both the technical aspects of neural network design and the
data-related considerations, this review provides researchers and practitioners
with a comprehensive and up-to-date understanding of the factors involved in
designing effective neural networks for semantic segmentation of Earth
Observation imagery.Comment: 145 pages with 32 figure
Tradition and Innovation in Construction Project Management
This book is a reprint of the Special Issue 'Tradition and Innovation in Construction Project Management' that was published in the journal Buildings
Inferences on Mixing Probabilities and Ranking in Mixed-Membership Models
Network data is prevalent in numerous big data applications including
economics and health networks where it is of prime importance to understand the
latent structure of network. In this paper, we model the network using the
Degree-Corrected Mixed Membership (DCMM) model. In DCMM model, for each node
, there exists a membership vector ,
where denotes the weight that node puts in
community . We derive novel finite-sample expansion for the
s which allows us to obtain asymptotic distributions and
confidence interval of the membership mixing probabilities and other related
population quantities. This fills an important gap on uncertainty
quantification on the membership profile. We further develop a ranking scheme
of the vertices based on the membership mixing probabilities on certain
communities and perform relevant statistical inferences. A multiplier bootstrap
method is proposed for ranking inference of individual member's profile with
respect to a given community. The validity of our theoretical results is
further demonstrated by via numerical experiments in both real and synthetic
data examples
Complexity Science in Human Change
This reprint encompasses fourteen contributions that offer avenues towards a better understanding of complex systems in human behavior. The phenomena studied here are generally pattern formation processes that originate in social interaction and psychotherapy. Several accounts are also given of the coordination in body movements and in physiological, neuronal and linguistic processes. A common denominator of such pattern formation is that complexity and entropy of the respective systems become reduced spontaneously, which is the hallmark of self-organization. The various methodological approaches of how to model such processes are presented in some detail. Results from the various methods are systematically compared and discussed. Among these approaches are algorithms for the quantification of synchrony by cross-correlational statistics, surrogate control procedures, recurrence mapping and network models.This volume offers an informative and sophisticated resource for scholars of human change, and as well for students at advanced levels, from graduate to post-doctoral. The reprint is multidisciplinary in nature, binding together the fields of medicine, psychology, physics, and neuroscience
- …