22,285 research outputs found
Securing NextG networks with physical-layer key generation: A survey
As the development of next-generation (NextG) communication networks continues, tremendous devices are accessing the network and the amount of information is exploding. However, with the increase of sensitive data that requires confidentiality to be transmitted and stored in the network, wireless network security risks are further amplified. Physical-layer key generation (PKG) has received extensive attention in security research due to its solid information-theoretic security proof, ease of implementation, and low cost. Nevertheless, the applications of PKG in the NextG networks are still in the preliminary exploration stage. Therefore, we survey existing research and discuss (1) the performance advantages of PKG compared to cryptography schemes, (2) the principles and processes of PKG, as well as research progresses in previous network environments, and (3) new application scenarios and development potential for PKG in NextG communication networks, particularly analyzing the effect and prospects of PKG in massive multiple-input multiple-output (MIMO), reconfigurable intelligent surfaces (RISs), artificial intelligence (AI) enabled networks, integrated space-air-ground network, and quantum communication. Moreover, we summarize open issues and provide new insights into the development trends of PKG in NextG networks
Efficient convolution-based pairwise elastic image registration on three multimodal similarity metrics
Producción CientÃficaThis paper proposes a complete convolutional formulation for 2D multimodal pairwise image registration problems based on free-form deformations. We have reformulated in terms of discrete 1D convolutions the evaluation of spatial transformations, the regularization term, and their gradients for three different multimodal registration metrics, namely, normalized cross correlation, mutual information, and normalized mutual information. A sufficient condition on the metric gradient is provided for further extension to other metrics. The proposed approach has been tested, as a proof of concept, on contrast-enhanced first-pass perfusion cardiac magnetic resonance images. Execution times have been compared with the corresponding execution times of the classical tensor product formulation, both on CPU and GPU. The speed-up achieved by using convolutions instead of tensor products depends on the image size and the number of control points considered, the larger those magnitudes, the greater the execution time reduction. Furthermore, the speed-up will be more significant when gradient operations constitute the major bottleneck in the optimization process.Ministerio de EconomÃa, Industria y Competitividad (grants TEC2017-82408-R and PID2020-115339RB-I00)ESAOTE Ltd (grant 18IQBM
Automatic Feature Detection in Lung Ultrasound Images using Wavelet and Radon Transforms
Objective: Lung ultrasonography is a significant advance toward a harmless
lung imagery system. This work has investigated the automatic localization of
diagnostically significant features in lung ultrasound pictures which are
Pleural line, A-lines, and B-lines. Study Design: Wavelet and Radon transforms
have been utilized in order to denoise and highlight the presence of clinically
significant patterns. The proposed framework is developed and validated using
three different lung ultrasound image datasets. Two of them contain synthetic
data and the other one is taken from the publicly available POCUS dataset. The
efficiency of the proposed method is evaluated using 200 real images. Results:
The obtained results prove that the comparison between localized patterns and
the baselines yields a promising F2-score of 62%, 86%, and 100% for B-lines,
A-lines, and Pleural line, respectively. Conclusion: Finally, the high F-scores
attained show that the developed technique is an effective way to automatically
extract lung patterns from ultrasound images.Comment: 13 pages, 12 figures, 4 table
Audio-visual multi-modality driven hybrid feature learning model for crowd analysis and classification
The high pace emergence in advanced software systems, low-cost hardware and decentralized cloud computing technologies have broadened the horizon for vision-based surveillance, monitoring and control. However, complex and inferior feature learning over visual artefacts or video streams, especially under extreme conditions confine majority of the at-hand vision-based crowd analysis and classification systems. Retrieving event-sensitive or crowd-type sensitive spatio-temporal features for the different crowd types under extreme conditions is a highly complex task. Consequently, it results in lower accuracy and hence low reliability that confines existing methods for real-time crowd analysis. Despite numerous efforts in vision-based approaches, the lack of acoustic cues often creates ambiguity in crowd classification. On the other hand, the strategic amalgamation of audio-visual features can enable accurate and reliable crowd analysis and classification. Considering it as motivation, in this research a novel audio-visual multi-modality driven hybrid feature learning model is developed for crowd analysis and classification. In this work, a hybrid feature extraction model was applied to extract deep spatio-temporal features by using Gray-Level Co-occurrence Metrics (GLCM) and AlexNet transferrable learning model. Once extracting the different GLCM features and AlexNet deep features, horizontal concatenation was done to fuse the different feature sets. Similarly, for acoustic feature extraction, the audio samples (from the input video) were processed for static (fixed size) sampling, pre-emphasis, block framing and Hann windowing, followed by acoustic feature extraction like GTCC, GTCC-Delta, GTCC-Delta-Delta, MFCC, Spectral Entropy, Spectral Flux, Spectral Slope and Harmonics to Noise Ratio (HNR). Finally, the extracted audio-visual features were fused to yield a composite multi-modal feature set, which is processed for classification using the random forest ensemble classifier. The multi-class classification yields a crowd-classification accurac12529y of (98.26%), precision (98.89%), sensitivity (94.82%), specificity (95.57%), and F-Measure of 98.84%. The robustness of the proposed multi-modality-based crowd analysis model confirms its suitability towards real-world crowd detection and classification tasks
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging
Neural networks can be significantly compressed by pruning, leading to sparse
models requiring considerably less storage and floating-point operations while
maintaining predictive performance. Model soups (Wortsman et al., 2022) improve
generalization and out-of-distribution performance by averaging the parameters
of multiple models into a single one without increased inference time. However,
identifying models in the same loss basin to leverage both sparsity and
parameter averaging is challenging, as averaging arbitrary sparse models
reduces the overall sparsity due to differing sparse connectivities. In this
work, we address these challenges by demonstrating that exploring a single
retraining phase of Iterative Magnitude Pruning (IMP) with varying
hyperparameter configurations, such as batch ordering or weight decay, produces
models that are suitable for averaging and share the same sparse connectivity
by design. Averaging these models significantly enhances generalization
performance compared to their individual components. Building on this idea, we
introduce Sparse Model Soups (SMS), a novel method for merging sparse models by
initiating each prune-retrain cycle with the averaged model of the previous
phase. SMS maintains sparsity, exploits sparse network benefits being modular
and fully parallelizable, and substantially improves IMP's performance.
Additionally, we demonstrate that SMS can be adapted to enhance the performance
of state-of-the-art pruning during training approaches.Comment: 9 pages, 5 pages references, 7 pages appendi
An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs
In recent years, Transformer-based language models have become the standard
approach for natural language processing tasks. However, stringent throughput
and latency requirements in industrial applications are limiting their
adoption. To mitigate the gap, model compression techniques such as structured
pruning are being used to improve inference efficiency. However, most existing
neural network inference runtimes lack adequate support for structured
sparsity. In this paper, we propose an efficient sparse deep learning inference
software stack for Transformer-based language models where the weights are
pruned with constant block size. Our sparse software accelerator leverages
Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
order of magnitude on a wide range of GEMM shapes under 5 representative
sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
widely used in industry. We apply our sparse accelerator on widely-used
Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
Services under proxy production latency constraints. We also compare our
solution with two framework-based inference solutions, ONNX Runtime and
PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
PyTorch on Xeon under the latency constraints. All the source code is publicly
available on Github: https://github.com/intel/intel-extension-for-transformers
A DeepONet multi-fidelity approach for residual learning in reduced order modeling
In the present work, we introduce a novel approach to enhance the precision
of reduced order models by exploiting a multi-fidelity perspective and
DeepONets. Reduced models provide a real-time numerical approximation by
simplifying the original model. The error introduced by the such operation is
usually neglected and sacrificed in order to reach a fast computation. We
propose to couple the model reduction to a machine learning residual learning,
such that the above-mentioned error can be learned by a neural network and
inferred for new predictions. We emphasize that the framework maximizes the
exploitation of high-fidelity information, using it for building the reduced
order model and for learning the residual. In this work, we explore the
integration of proper orthogonal decomposition (POD), and gappy POD for sensors
data, with the recent DeepONet architecture. Numerical investigations for a
parametric benchmark function and a nonlinear parametric Navier-Stokes problem
are presented
vONTSS: vMF based semi-supervised neural topic modeling with optimal transport
Recently, Neural Topic Models (NTM), inspired by variational autoencoders,
have attracted a lot of research interest; however, these methods have limited
applications in the real world due to the challenge of incorporating human
knowledge. This work presents a semi-supervised neural topic modeling method,
vONTSS, which uses von Mises-Fisher (vMF) based variational autoencoders and
optimal transport. When a few keywords per topic are provided, vONTSS in the
semi-supervised setting generates potential topics and optimizes topic-keyword
quality and topic classification. Experiments show that vONTSS outperforms
existing semi-supervised topic modeling methods in classification accuracy and
diversity. vONTSS also supports unsupervised topic modeling. Quantitative and
qualitative experiments show that vONTSS in the unsupervised setting
outperforms recent NTMs on multiple aspects: vONTSS discovers highly clustered
and coherent topics on benchmark datasets. It is also much faster than the
state-of-the-art weakly supervised text classification method while achieving
similar classification performance. We further prove the equivalence of optimal
transport loss and cross-entropy loss at the global minimum.Comment: 24 pages, 12 figures, ACL findings 202
SignReLU neural network and its approximation ability
Deep neural networks (DNNs) have garnered significant attention in various
fields of science and technology in recent years. Activation functions define
how neurons in DNNs process incoming signals for them. They are essential for
learning non-linear transformations and for performing diverse computations
among successive neuron layers. In the last few years, researchers have
investigated the approximation ability of DNNs to explain their power and
success. In this paper, we explore the approximation ability of DNNs using a
different activation function, called SignReLU. Our theoretical results
demonstrate that SignReLU networks outperform rational and ReLU networks in
terms of approximation performance. Numerical experiments are conducted
comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and
ELU, which illustrate the competitive practical performance of SignReLU
Implicit Loss of Surjectivity and Facial Reduction: Theory and Applications
Facial reduction, pioneered by Borwein and Wolkowicz, is a preprocessing method that is commonly used to obtain strict feasibility in the reformulated, reduced constraint system.
The importance of strict feasibility is often addressed in the context of the convergence results for interior point methods.
Beyond the theoretical properties that the facial reduction conveys, we show that facial reduction, not only limited to interior point methods, leads to strong numerical performances in different classes of algorithms.
In this thesis we study various consequences and the broad applicability of facial reduction.
The thesis is organized in two parts.
In the first part, we show the instabilities accompanied by the absence
of strict feasibility through the lens of facially reduced systems.
In particular, we exploit the implicit redundancies, revealed by each nontrivial facial reduction step, resulting in the implicit loss of surjectivity.
This leads to the two-step facial reduction and two novel related notions of singularity.
For the area of semidefinite programming, we use these singularities to strengthen a known bound on the solution rank, the Barvinok-Pataki bound.
For the area of linear programming, we reveal degeneracies caused by the implicit redundancies.
Furthermore, we propose a preprocessing tool that uses the simplex method.
In the second part of this thesis, we continue with the semidefinite programs that do not have strictly feasible points.
We focus on the doubly-nonnegative relaxation of the binary quadratic program and a semidefinite program with a nonlinear objective function.
We closely work with two classes of algorithms, the splitting method and the Gauss-Newton interior point method.
We elaborate on the advantages in building models from facial reduction. Moreover, we develop algorithms for real-world problems including the quadratic assignment problem, the protein side-chain positioning problem, and the key rate computation for quantum key distribution.
Facial reduction continues to play an important role for
providing robust reformulated models in both the theoretical and the practical aspects, resulting in successful numerical performances
- …