Knowledge Distillation and Continual Learning for Optimized Deep Neural Networks

Over the past few years, deep learning (DL) has been achieving state-of-theart performance on various human tasks such as speech generation, language translation, image segmentation, and object detection. While traditional machine learning models require hand-crafted features, deep learning algorithms can automatically extract discriminative features and learn complex knowledge from large datasets. This powerful learning ability makes deep learning models attractive to both academia and big corporations. Despite their popularity, deep learning methods still have two main limitations: large memory consumption and catastrophic knowledge forgetting. First, DL algorithms use very deep neural networks (DNNs) with many billion parameters, which have a big model size and a slow inference speed. This restricts the application of DNNs in resource-constraint devices such as mobile phones and autonomous vehicles. Second, DNNs are known to suffer from catastrophic forgetting. When incrementally learning new tasks, the model performance on old tasks significantly drops. The ability to accommodate new knowledge while retaining previously learned knowledge is called continual learning. Since the realworld environments in which the model operates are always evolving, a robust neural network needs to have this continual learning ability for adapting to new changes

A Structured SVM Semantic Parser Augmented by Semantic Tagging with Conditional Random Field

PACLIC 19 / Taipei, taiwan / December 1-3, 200

The Mediating Effects of Switching Costs on the Relationship between Service Quality, Customer Satisfaction and Customer Loyalty: A Study in Retail Banking Industry in Vietnam

This paper develops and empirically tests the mediating role of switching costs in service quality - loyalty and satisfaction-loyalty relationships. Especially, different types of switching costs are tested separately providing more insights into their roles. A research model about the interrelationships between service quality, customer satisfaction, switching costs and customer loyalty is developed. Based on this model, a survey is conducted with retail banking customers, with and 261valid respondents. The hypotheses are then proposed and tested using Structural equation modeling technique (SEM). The analysis reveals that: positive switching cost is a significant mediator for both service quality-loyalty and satisfaction-loyalty relationships, while negative switching cost only mediates the service quality-loyalty relationship. These findings suggest that building and managing switching costs are necessary following-up steps after customer satisfaction for achieving long-term customer loyalty

The mediating effects of switching costs on the relationship between service quality, customer satisfaction and customer loyalty: A study in retail banking industry in Vietnam

This paper develops and empirically tests the mediating role of switching costs in service quality - loyalty and satisfaction-loyalty relationships. Specially, different types of switching costs are tested separately providing more insights about their roles. This approach extended the insights on mediating effects of switching costs by differentiate the roles of positive switching costs and negative switching costs in the model. A research model about the interrelationships between service quality, customer satisfaction, switching costs and customer loyalty is developed. Based on this model, a survey is conducted with retail banking customers, with and 261 valid respondents. The hypotheses are then proposed and tested using Structural equation modeling technique (SEM). The analysis reveals that: positive switching cost is a significant mediator for both service quality-loyalty and satisfaction-loyalty relationships, while negative switching cost only mediates the service quality-loyalty relationship. These findings suggest that building and managing switching costs are necessary following-up steps after customer satisfaction for achieving long-term customer loyalty. However, using the right types of switching costs is necessary to significantly boost the loyalty from customers. © ExcelingTech Pub, UK

SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers

This paper investigates the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework and introduces \textbf{SegViTv2}. In this study, we introduce a novel Attention-to-Mask (\atm) module to design a lightweight decoder effective for plain ViT. The proposed ATM converts the global attention map into semantic masks for high-quality segmentation results. Our decoder outperforms the popular decoder UPerNet using various ViT backbones while consuming only about $5\%$ of the computational cost. For the encoder, we address the concern of the relatively high computational cost in the ViT-based encoders and propose a \emph{Shrunk++} structure that incorporates edge-aware query-based down-sampling (EQD) and query-based upsampling (QU) modules. The Shrunk++ structure reduces the computational cost of the encoder by up to $50\%$ while maintaining competitive performance. Furthermore, we propose to adapt SegViT for continual semantic segmentation, demonstrating nearly zero forgetting of previously learned knowledge. Experiments show that our proposed SegViTv2 surpasses recent segmentation methods on three popular benchmarks including ADE20k, COCO-Stuff-10k and PASCAL-Context datasets. The code is available through the following link: \url{https://github.com/zbwxp/SegVit}.Comment: IJCV 2023 accepted, 21 pages, 8 figures, 12 table

BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation

Current knowledge distillation approaches in semantic segmentation tend to adopt a holistic approach that treats all spatial locations equally. However, for dense prediction, students' predictions on edge regions are highly uncertain due to contextual information leakage, requiring higher spatial sensitivity knowledge than the body regions. To address this challenge, this paper proposes a novel approach called boundary-privileged knowledge distillation (BPKD). BPKD distills the knowledge of the teacher model's body and edges separately to the compact student model. Specifically, we employ two distinct loss functions: (i) edge loss, which aims to distinguish between ambiguous classes at the pixel level in edge regions; (ii) body loss, which utilizes shape constraints and selectively attends to the inner-semantic regions. Our experiments demonstrate that the proposed BPKD method provides extensive refinements and aggregation for edge and body regions. Additionally, the method achieves state-of-the-art distillation performance for semantic segmentation on three popular benchmark datasets, highlighting its effectiveness and generalization ability. BPKD shows consistent improvements across a diverse array of lightweight segmentation structures, including both CNNs and transformers, underscoring its architecture-agnostic adaptability. The code is available at \url{https://github.com/AkideLiu/BPKD}.Comment: 17 pages, 9 figures, 9 table

M^2UNet: MetaFormer Multi-scale Upsampling Network for Polyp Segmentation

Polyp segmentation has recently garnered significant attention, and multiple methods have been formulated to achieve commendable outcomes. However, these techniques often confront difficulty when working with the complex polyp foreground and their surrounding regions because of the nature of convolution operation. Besides, most existing methods forget to exploit the potential information from multiple decoder stages. To address this challenge, we suggest combining MetaFormer, introduced as a baseline for integrating CNN and Transformer, with UNet framework and incorporating our Multi-scale Upsampling block (MU). This simple module makes it possible to combine multi-level information by exploring multiple receptive field paths of the shallow decoder stage and then adding with the higher stage to aggregate better feature representation, which is essential in medical image segmentation. Taken all together, we propose MetaFormer Multi-scale Upsampling Network (M$^2$UNet) for the polyp segmentation task. Extensive experiments on five benchmark datasets demonstrate that our method achieved competitive performance compared with several previous methods

D2D Communication Network with the Assistance of Power Beacon under the Impact of Co-channel Interferences and Eavesdropper: Performance Analysis

In this paper, we study and demonstrate the performance analysis of a device-to-device (D2D) com- munication network. Specifically, a source node trans- mits data to the destination node using the power bea- con’s harvested energy in order to overcome the limited energy budget. Besides, an eavesdropper located in the proximal region of a source is trying to overhear secure information. Notably, both eavesdropper and destina- tion are affected by co-channel interferences from other sources when they utilize the same frequency. By con- sidering the above discussions, we derived the closed- form expressions for outage probability (OP), intercept probability (IP), and secrecy outage probability (SOP) in connection with using the system model. The derived analytical expressions are then verified by utilizing both simulation and numerical results. Finally, the inten- sive parameters’ influences on the OP, IP, and SOP are also investigated

Assistive technologies for the older people: Physical activity monitoring and fall detection

The advancements in information and communications technologies (ICT) and micro-nano manufacturing lead to innovative developments of smart sensors and intelligent devices as well as related assistive technologies which have been directly contributing to improving the life quality, from early detection of diseases to assisting daily living activities. Physical activity monitoring and fall detection are two specific examples where assistive technologies with the use of smart sensors and intelligent devices may play a key role in enhancing the life quality, especially improving the musculoskeletal health which is an essential aspect of health and wellbeing; and it is more important for the older people. This paper presents and dis-cusses about how sensors and wearable devices, such as accelerometers and mobile phones, may be employed to promote the musculoskeletal health. Assistive technologies and methods for physical activity monitoring and fall detection are discussed, with the focus on the fall detection using mobile phone technology, and assessments of the loading intensity of physical activity in a non-laboratory environment. The possible research directions, challenges and potential collaborations in the areas of assistive technologies and ICT solutions for the older populations are proposed and addressed