Ske2Grid: Skeleton-to-Grid Representation Learning for Action Recognition

This paper presents Ske2Grid, a new representation learning framework for improved skeleton-based action recognition. In Ske2Grid, we define a regular convolution operation upon a novel grid representation of human skeleton, which is a compact image-like grid patch constructed and learned through three novel designs. Specifically, we propose a graph-node index transform (GIT) to construct a regular grid patch through assigning the nodes in the skeleton graph one by one to the desired grid cells. To ensure that GIT is a bijection and enrich the expressiveness of the grid representation, an up-sampling transform (UPT) is learned to interpolate the skeleton graph nodes for filling the grid patch to the full. To resolve the problem when the one-step UPT is aggressive and further exploit the representation capability of the grid patch with increasing spatial size, a progressive learning strategy (PLS) is proposed which decouples the UPT into multiple steps and aligns them to multiple paired GITs through a compact cascaded design learned progressively. We construct networks upon prevailing graph convolution networks and conduct experiments on six mainstream skeleton-based action recognition datasets. Experiments show that our Ske2Grid significantly outperforms existing GCN-based solutions under different benchmark settings, without bells and whistles. Code and models are available at https://github.com/OSVAI/Ske2GridComment: The paper of Ske2Grid is published at ICML 2023. Code and models are available at https://github.com/OSVAI/Ske2Gri

FwdLLM: Efficient FedLLM using Forward Gradient

Large Language Models (LLMs) are transforming the landscape of mobile intelligence. Federated Learning (FL), a method to preserve user data privacy, is often employed in fine-tuning LLMs to downstream mobile tasks, an approach known as FedLLM. Though recent efforts have addressed the network issue induced by the vast model size, they have not practically mitigated vital challenges concerning integration with mobile devices, such as significant memory consumption and sluggish model convergence. In response to these challenges, this work introduces FwdLLM, an innovative FL protocol designed to enhance the FedLLM efficiency. The key idea of FwdLLM to employ backpropagation (BP)-free training methods, requiring devices only to execute ``perturbed inferences''. Consequently, FwdLLM delivers way better memory efficiency and time efficiency (expedited by mobile NPUs and an expanded array of participant devices). FwdLLM centers around three key designs: (1) it combines BP-free training with parameter-efficient training methods, an essential way to scale the approach to the LLM era; (2) it systematically and adaptively allocates computational loads across devices, striking a careful balance between convergence speed and accuracy; (3) it discriminatively samples perturbed predictions that are more valuable to model convergence. Comprehensive experiments with five LLMs and three NLP tasks illustrate FwdLLM's significant advantages over conventional methods, including up to three orders of magnitude faster convergence and a 14.6x reduction in memory footprint. Uniquely, FwdLLM paves the way for federated learning of billion-parameter LLMs such as LLaMA on COTS mobile devices -- a feat previously unattained.Comment: under revie

Federated NLP in Few-shot Scenarios

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications. For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FFNLP, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0$\times$, 41.2$\times$ and 3000.0$\times$, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled

Towards Practical Few-shot Federated NLP

Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.Comment: EuroSys23 worksho

Accelerating Vertical Federated Learning

Privacy, security and data governance constraints rule out a brute force process in the integration of cross-silo data, which inherits the development of the Internet of Things. Federated learning is proposed to ensure that all parties can collaboratively complete the training task while the data is not out of the local. Vertical federated learning is a specialization of federated learning for distributed features. To preserve privacy, homomorphic encryption is applied to enable encrypted operations without decryption. Nevertheless, together with a robust security guarantee, homomorphic encryption brings extra communication and computation overhead. In this paper, we analyze the current bottlenecks of vertical federated learning under homomorphic encryption comprehensively and numerically. We propose a straggler-resilient and computation-efficient accelerating system that reduces the communication overhead in heterogeneous scenarios by 65.26% at most and reduces the computation overhead caused by homomorphic encryption by 40.66% at most. Our system can improve the robustness and efficiency of the current vertical federated learning framework without loss of security

Mitofusin 2 Participates in Mitophagy and Mitochondrial Fusion Against Angiotensin II-Induced Cardiomyocyte Injury

BackgroundMitochondrial dynamics play a critical role in mitochondrial function. The mitofusin 2 (MFN2) gene encodes a mitochondrial membrane protein that participates in mitochondrial fusion to maintain and operate the mitochondrial network. Moreover, MFN2 is essential for mitophagy. In Ang II-induced cardiac remodeling, the combined effects of MFN2-mediated mitochondrial fusion and mitophagy are unclear. This study was designed to explore a novel strategy for preventing cardiomyocyte injury via modulation of mitochondrial dynamics.MethodsWe studied the function of MFN2 in mitochondrial fusion and mitophagy in Ang II-stimulated cardiomyocyte injury. Cardiomyocyte injury experiments, including reactive oxygen species (ROS) production, mitochondrial membrane potential (MMP), and apoptosis rate of cardiomyocytes were performed. The mitochondrial morphology in cardiomyocytes was examined via transmission electron microscopy (TEM) and confocal microscopy. Autophagic levels in response to Ang II were examined by immunoblotting of autophagy-related proteins. Moreover, PINK1/MFN2/Parkin pathway-related proteins were examined.ResultsWith stimulation by Ang II, MFN2 expression was progressively reduced. MFN2 deficiency impaired mitochondrial quality, resulting in exacerbated mitochondrial damage induced by Ang II. The Ang II-induced increases in ROS production and apoptosis rate were alleviated by MFN2 overexpression. Moreover, MFN2 alleviated the Ang II-induced reduction in MMP. MFN2 promoted mitochondrial fusion, and MFN2 promoted Parkin translocation and phosphorylation, leading to mitochondrial autophagy. The effects of MFN2 overexpression were reversed by autophagy inhibitors.ConclusionMitofusin 2 promotes Parkin translocation and phosphorylation, leading to mitophagy to clear damaged mitochondria. However, the beneficial effects of MFN2 were reversed by autophagy inhibitors. Additionally, MFN2 participates in mitochondrial fusion to maintain mitochondrial quality. Thus, MFN2 participated in mitophagy and mitochondrial fusion against Ang II-induced cardiomyocyte injury

Lightweight Protection for Privacy in Offloaded Speech Understanding

Speech is a common input method for mobile embedded devices, but cloud-based speech recognition systems pose privacy risks. Disentanglement-based encoders, designed to safeguard user privacy by filtering sensitive information from speech signals, unfortunately require substantial memory and computational resources, which limits their use in less powerful devices. To overcome this, we introduce a novel system, XXX, optimized for such devices. XXX is built on the insight that speech understanding primarily relies on understanding the entire utterance's long-term dependencies, while privacy concerns are often linked to short-term details. Therefore, XXX focuses on selectively masking these short-term elements, preserving the quality of long-term speech understanding. The core of XXX is an innovative differential mask generator, grounded in interpretable learning, which fine-tunes the masking process. We tested XXX on the STM32H7 microcontroller, assessing its performance in various potential attack scenarios. The results show that XXX maintains speech understanding accuracy and privacy at levels comparable to existing encoders, but with a significant improvement in efficiency, achieving up to 53.3$\times$ faster processing and a 134.1$\times$ smaller memory footprint.Comment: arXiv admin comment: This version has been removed by arXiv administrators as the submitter did not have the rights to agree to the license at the time of submissio

An Electrochromic Ag-Decorated WO_3−x Film with Adjustable Defect States for Electrochemical Surface-Enhanced Raman Spectroscopy

Electrochemical surface-enhanced Raman scattering (EC-SERS) spectroscopy is an ultrasensitive spectro-electrochemistry technique that provides mechanistic and dynamic information on electrochemical interfaces at the molecular level. However, the plasmon-mediated photocatalysis hinders the intrinsic electrochemical behavior of molecules at electrochemical interfaces. This work aimed to develop a facile method for constructing a reliable EC-SERS substrate that can be used to study the molecular dynamics at electrochemical interfaces. Herein, a novel Ag-WO3−x electrochromic heterostructure was synthesized for EC-SERS. Especially, the use of electrochromic WO3−x film suppresses the influence of hot-electrons-induced catalysis while offering a reliable SERS effect. Based on this finding, the real electrochemical behavior of p-aminothiophenol (PATP) on Ag nanoparticles (NPs) surface was revealed for the first time. We are confident that metal-semiconductor electrochromic heterostructures could be developed into reliable substrates for EC-SERS analysis. Furthermore, the results obtained in this work provide new insights not only into the chemical mechanism of SERS, but also into the hot-electron transfer mechanism in metal-semiconductor heterostructures