On-device training of machine learning models on microcontrollers with a look at federated learning

Recent progress in machine learning frameworks makes it now possible to run an inference with sophisticated machine learning models on tiny microcontrollers. Model training, however, is typically done separately on powerful computers. There, the training process has abundant CPU and memory resources to process the stored datasets. In this work, we explore a different approach: training the model directly on the microcontroller. We implement this approach for a keyword spotting task. Then, we extend the training process using federated learning among microcontrollers. Our experiments with model training show an overall trend of decreasing loss with the increase of training epochs.This work was partially funded by the Spanish Government under contracts PID2019-106774RB-C21, PCI2019-111850-2 (DiPET CHIST-ERA), PCI2019-111851-2 (LeadingEdge CHIST-ERA), and the Generalitat de Catalunya as Consolidated Research Group 2017- SGR-990.Peer ReviewedPostprint (author's final draft

On-device training of machine learning models on microcontrollers with federated learning

Recent progress in machine learning frameworks has made it possible to now perform inference with models using cheap, tiny microcontrollers. Training of machine learning models for these tiny devices, however, is typically done separately on powerful computers. This way, the training process has abundant CPU and memory resources to process large stored datasets. In this work, we explore a different approach: training the machine learning model directly on the microcontroller and extending the training process with federated learning. We implement this approach for a keyword spotting task. We conduct experiments with real devices to characterize the learning behavior and resource consumption for different hyperparameters and federated learning configurations. We observed that in the case of training locally with fewer data, more frequent federated learning rounds more quickly reduced the training loss but involved a cost of higher bandwidth usage and longer training time. Our results indicate that, depending on the specific application, there is a need to determine the trade-off between the requirements and the resource usage of the system.This work has received funding from the Spanish Government under contracts PID2019-106774RB-C21, PCI2019-111851-2 (LeadingEdge CHIST-ERA), PCI2019-111850-2 (DiPET CHIST-ERA).Peer ReviewedPostprint (published version

Analysis of the common genetic component of large-vessel vasculitides through a meta- Immunochip strategy

Giant cell arteritis (GCA) and Takayasu's arteritis (TAK) are major forms of large-vessel vasculitis (LVV) that share clinical features. To evaluate their genetic similarities, we analysed Immunochip genotyping data from 1,434 LVV patients and 3,814 unaffected controls. Genetic pleiotropy was also estimated. The HLA region harboured the main disease-specific associations. GCA was mostly associated with class II genes (HLA-DRB1/HLA-DQA1) whereas TAK was mostly associated with class I genes (HLA-B/MICA). Both the statistical significance and effect size of the HLA signals were considerably reduced in the cross-disease meta-analysis in comparison with the analysis of GCA and TAK separately. Consequently, no significant genetic correlation between these two diseases was observed when HLA variants were tested. Outside the HLA region, only one polymorphism located nearby the IL12B gene surpassed the study-wide significance threshold in the meta-analysis of the discovery datasets (rs755374, P?=?7.54E-07; ORGCA?=?1.19, ORTAK?=?1.50). This marker was confirmed as novel GCA risk factor using four additional cohorts (PGCA?=?5.52E-04, ORGCA?=?1.16). Taken together, our results provide evidence of strong genetic differences between GCA and TAK in the HLA. Outside this region, common susceptibility factors were suggested, especially within the IL12B locus

A Large-Scale Genetic Analysis Reveals a Strong Contribution of the HLA Class II Region to Giant Cell Arteritis Susceptibility

We conducted a large-scale genetic analysis on giant cell arteritis (GCA), a polygenic immune-mediated vasculitis. A case-control cohort, comprising 1,651 case subjects with GCA and 15,306 unrelated control subjects from six different countries of European ancestry, was genotyped by the Immunochip array. We also imputed HLA data with a previously validated imputation method to perform a more comprehensive analysis of this genomic region. The strongest association signals were observed in the HLA region, with rs477515 representing the highest peak (p = 4.05 × 10−40, OR = 1.73). A multivariate model including class II amino acids of HLA-DRβ1 and HLA-DQα1 and one class I amino acid of HLA-B explained most of the HLA association with GCA, consistent with previously reported associations of classical HLA alleles like HLA-DRB1∗04. An omnibus test on polymorphic amino acid positions highlighted DRβ1 13 (p = 4.08 × 10−43) and HLA-DQα1 47 (p = 4.02 × 10−46), 56, and 76 (both p = 1.84 × 10−45) as relevant positions for disease susceptibility. Outside the HLA region, the most significant loci included PTPN22 (rs2476601, p = 1.73 × 10−6, OR = 1.38), LRRC32 (rs10160518, p = 4.39 × 10−6, OR = 1.20), and REL (rs115674477, p = 1.10 × 10−5, OR = 1.63). Our study provides evidence of a strong contribution of HLA class I and II molecules to susceptibility to GCA. In the non-HLA region, we confirmed a key role for the functional PTPN22 rs2476601 variant and proposed other putative risk loci for GCA involved in Th1, Th17, and Treg cell function

TinyML: From Basic to Advanced Applications

TinyML tiene como objetivo la implementación de aplicaciones de aprendizaje automático en dispositivos de poco tamaño y baja potencia, como los microcontroladores. Normalmente los dispositivos periféricos necesitan estar conectados a centro de datos para poder ejecutar aplicaciones de aprendizaje automático. Sin embargo, este método no es posible en muchos escenarios, como en la falta de conectividad. Este estudio investiga las herramientas y técnicas utilizadas en TinyML, las limitaciones en la utilización de dispositivos de baja potencia y la viabilidad de implementar aplicaciones avanzadas de aprendizaje automático en microcontroladores. Se desarrollaron tres programas para comprobar la viabilidad de implementar aplicaciones de aprendizaje automático en microcontroladores. El primero, una aplicación capaz de reconocer un conjunto de palabras claves. El segundo, un programa capaz de entrenar un modelo de red neuronal en el microcontrolador siguiendo un enfoque de aprendizaje en línea. Y el tercero, un programa de aprendizaje federado capaz de entrenar un único modelo global con la agregación de modelos locales entrenados en múltiples microcontroladores. Los resultados muestran un óptimo rendimiento de las tres aplicaciones una vez desplegadas en los microcontroladores. El desarrollo de aplicaciones básicas de TinyML resulta sencillo una vez entendidos el proceso de aprendizaje automático. Sin embargo, el desarrollo de aplicaciones avanzadas es muy complejo, ya que requiere un profundo conocimiento tanto del aprendizaje automático como de los sistemas embebidos. Estos resultados demuestran la viabilidad de implementar con éxito aplicaciones avanzadas de aprendizaje automático en microcontroladores, y por lo tanto, desvelan un futuro brillante para TinyML.TinyML aims to implement machine learning (ML) applications on small, and low-powered devices like microcontrollers. Typically, edge devices need to be connected to data centers in order to run ML applications. However, this approach is not possible in many scenarios, such as lack of connectivity. This project investigates the tools and techniques used in TinyML, the constraints of using low-powered devices, and the feasibility of implementing advanced machine learning applications on microcontrollers. To test the feasibility of implementing ML applications on microcontrollers, three TinyML programs were developed. The first, a basic keyword spotting application able to recognize a set of words. The second, a program for training a neural network model on a microcontroller following an online learning approach. And the third, a federated learning program able to train a single global model with the aggregation of local models trained on multiple microcontrollers. The results show optimal performance in all three applications once deployed on microcontrollers. The development of basic TinyML applications is straightforward when the machine learning pipeline is understood. However, the development of advanced applications turned out to be very complex, as it requires a deep understanding of both machine learning and embedded systems. These results prove the feasibility of successfully implementing advanced ML applications on microcontrollers, and thus, unveil a bright future for TinyML

Erratum: Corrigendum: Analysis of the common genetic component of large-vessel vasculitides through a meta-Immunochip strategy

Giant cell arteritis (GCA) and Takayasu’s arteritis (TAK) are major forms of large-vessel vasculitis (LVV) that share clinical features. To evaluate their genetic similarities, we analysed Immunochip genotyping data from 1,434 LVV patients and 3,814 unaffected controls. Genetic pleiotropy was also estimated. The HLA region harboured the main disease-specific associations. GCA was mostly associated with class II genes (HLA-DRB1/HLA-DQA1) whereas TAK was mostly associated with class I genes (HLA-B/ MICA). Both the statistical significance and effect size of the HLA signals were considerably reduced in the cross-disease meta-analysis in comparison with the analysis of GCA and TAK separately. Consequently, no significant genetic correlation between these two diseases was observed when HLA variants were tested. Outside the HLA region, only one polymorphism located nearby the IL12B gene surpassed the study-wide significance threshold in the meta-analysis of the discovery datasets (rs755374, P = 7.54E-07; ORGCA = 1.19, ORTAK = 1.50). This marker was confirmed as novel GCA risk factor using four additional cohorts (PGCA = 5.52E-04, ORGCA = 1.16). Taken together, our results provide evidence of strong genetic differences between GCA and TAK in the HLA. Outside this region, common susceptibility factors were suggested, especially within the IL12B locus

Corrigendum: Analysis of the common genetic component of large-vessel vasculitides through a meta-Immunochip strategy (Scientific Reports (2017) 7 (43953) DOI: 10.1038/srep43953)

Giant cell arteritis (GCA) and Takayasu\u2019s arteritis (TAK) are major forms of large-vessel vasculitis (LVV) that share clinical features. To evaluate their genetic similarities, we analysed Immunochip genotyping data from 1,434 LVV patients and 3,814 unaffected controls. Genetic pleiotropy was also estimated. The HLA region harboured the main disease-specific associations. GCA was mostly associated with class II genes (HLA-DRB1/HLA-DQA1) whereas TAK was mostly associated with class I genes (HLA-B/ MICA). Both the statistical significance and effect size of the HLA signals were considerably reduced in the cross-disease meta-analysis in comparison with the analysis of GCA and TAK separately. Consequently, no significant genetic correlation between these two diseases was observed when HLA variants were tested. Outside the HLA region, only one polymorphism located nearby the IL12B gene surpassed the study-wide significance threshold in the meta-analysis of the discovery datasets (rs755374, P = 7.54E-07; ORGCA = 1.19, ORTAK = 1.50). This marker was confirmed as novel GCA risk factor using four additional cohorts (PGCA = 5.52E-04, ORGCA = 1.16). Taken together, our results provide evidence of strong genetic differences between GCA and TAK in the HLA. Outside this region, common susceptibility factors were suggested, especially within the IL12B locus