Incentive Mechanism Design for Distributed Ensemble Learning

Distributed ensemble learning (DEL) involves training multiple models at distributed learners, and then combining their predictions to improve performance. Existing related studies focus on DEL algorithm design and optimization but ignore the important issue of incentives, without which self-interested learners may be unwilling to participate in DEL. We aim to fill this gap by presenting a first study on the incentive mechanism design for DEL. Our proposed mechanism specifies both the amount of training data and reward for learners with heterogeneous computation and communication costs. One design challenge is to have an accurate understanding regarding how learners' diversity (in terms of training data) affects the ensemble accuracy. To this end, we decompose the ensemble accuracy into a diversity-precision tradeoff to guide the mechanism design. Another challenge is that the mechanism design involves solving a mixed-integer program with a large search space. To this end, we propose an alternating algorithm that iteratively updates each learner's training data size and reward. We prove that under mild conditions, the algorithm converges. Numerical results using MNIST dataset show an interesting result: our proposed mechanism may prefer a lower level of learner diversity to achieve a higher ensemble accuracy.Comment: Accepted to IEEE GLOBECOM 202

FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning

Knowledge distillation (KD) can enable collaborative learning among distributed clients that have different model architectures and do not share their local data and model parameters with others. Each client updates its local model using the average model output/feature of all client models as the target, known as federated KD. However, existing federated KD methods often do not perform well when clients' local models are trained with heterogeneous local datasets. In this paper, we propose Federated knowledge distillation enabled by Adversarial Learning (FedAL) to address the data heterogeneity among clients. First, to alleviate the local model output divergence across clients caused by data heterogeneity, the server acts as a discriminator to guide clients' local model training to achieve consensus model outputs among clients through a min-max game between clients and the discriminator. Moreover, catastrophic forgetting may happen during the clients' local training and global knowledge transfer due to clients' heterogeneous local data. Towards this challenge, we design the less-forgetting regularization for both local training and global knowledge transfer to guarantee clients' ability to transfer/learn knowledge to/from others. Experimental results show that FedAL and its variants achieve higher accuracy than other federated KD baselines

Convergence Analysis of Split Federated Learning on Heterogeneous Data

Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are $O(1/T)$ and $O(1/\sqrt[3]{T})$, respectively, where $T$ denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and where some clients may be unavailable during training. Numerical experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients

Virtual network embedding framework in fiber-wireless access network

This paper focuses on the virtual network embedding problem in fiber-wireless access network, and formulates it as an Integer Linear Programming (ILP). Simulation results verify the effectiveness of proposed framework

Virtual network embedding framework in fiber-wireless access network

This paper focuses on the virtual network embedding problem in fiber-wireless access network, and formulates it as an Integer Linear Programming (ILP). Simulation results verify the effectiveness of proposed framework

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Knowledge Distillation (KD) is a powerful technique for transferring knowledge between neural network models, where a pre-trained teacher model is used to facilitate the training of the target student model. However, the availability of a suitable teacher model is not always guaranteed. To address this challenge, Self-Knowledge Distillation (SKD) attempts to construct a teacher model from itself. Existing SKD methods add Auxiliary Classifiers (AC) to intermediate layers of the model or use the history models and models with different input data within the same class. However, these methods are computationally expensive and only capture time-wise and class-wise features of data. In this paper, we propose a lightweight SKD framework that utilizes multi-source information to construct a more informative teacher. Specifically, we introduce a Distillation with Reverse Guidance (DRG) method that considers different levels of information extracted by the model, including edge, shape, and detail of the input data, to construct a more informative teacher. Additionally, we design a Distillation with Shape-wise Regularization (DSR) method that ensures a consistent shape of ranked model output for all data. We validate the performance of the proposed DRG, DSR, and their combination through comprehensive experiments on various datasets and models. Our results demonstrate the superiority of the proposed methods over baselines (up to 2.87%) and state-of-the-art SKD methods (up to 1.15%), while being computationally efficient and robust. The code is available at https://github.com/xucong-parsifal/LightSKD.Comment: Submitted to IEEE TNNL

Clean Utilization of Limonite Ore by Suspension Magnetization Roasting Technology

As a typical refractory iron ore, the utilization of limonite ore with conventional mineral processing methods has great limitations. In this study, suspension magnetization roasting technology was developed and utilized to recover limonite ore. The influences of roasting temperature, roasting time, and reducing gas concentration on the magnetization roasting process were investigated. The optimal roasting conditions were determined to be a roasting temperature of 480 °C, a roasting time of 12.5 min, and a reducing gas concentration of 20%. Under optimal conditions, an iron concentrate grade of 60.12% and iron recovery of 91.96% was obtained. The phase transformation, magnetism variation, and microstructure evolution behavior were systematically analyzed by X-ray diffraction, vibrating sample magnetometer, and scanning electron microscope. The results indicated that hematite and goethite were eventually transformed into magnetite during the magnetization roasting process. Moreover, the magnetism of roasted products significantly improved due to the formation of ferrimagnetic magnetite in magnetization roasting. This study has implications for the utilization of limonite ore using suspension magnetization roasting technology

Treatment of Pediatric Inflammatory Myofibroblastic Tumor: The Experience from China Children’s Medical Center

Background: Inflammatory myofibroblastic tumor (IMT) is a rare mesenchymal tumor with intermediate malignancy that tends to affect children primarily. To date, no standardized therapies exist for the treatment of IMT. This study aimed to share experience from China Children’s Medical Center for the explorative treatment of IMT. Methods: Patients with newly diagnosed IMT between January 2013 and December 2018 were included. Patients were grouped according to surgical margins and Intergroup Rhabdomyosarcoma Study Group (IRSG) staging. The clinical characteristic, therapeutic schedules, treatment response and clinical outcome were described. Results: Six patients were enrolled in this study, including two boys and four girls, with a median age of 57 months (range 10–148 months). Among them, five patients were anaplastic lymphoma kinase positive. Four patients achieved complete remission and two patients attained partial remission after treatment with this protocol. All patients were alive after a median follow-up of 4 years (range 3–7 years). The most common treatment-related adverse reaction was myelosuppression. Conclusion: In this study, we demonstrated that IMT has a good prognosis and the treatment selected according to risk stratification was effective and feasible