Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence

In transfer learning, training and testing data sets are drawn from different data distributions. The transfer generalization gap is the difference between the population loss on the target data distribution and the training loss. The training data set generally includes data drawn from both source and target distributions. This work presents novel information-theoretic upper bounds on the average transfer generalization gap that capture $(i)$ the domain shift between the target data distribution $P'_Z$ and the source distribution $P_Z$ through a two-parameter family of generalized $(\alpha_1,\alpha_2)$-Jensen-Shannon (JS) divergences; and $(ii)$ the sensitivity of the transfer learner output $W$ to each individual sample of the data set $Z_i$ via the mutual information $I(W;Z_i)$. For $\alpha_1 \in (0,1)$, the $(\alpha_1,\alpha_2)$-JS divergence can be bounded even when the support of $P_Z$ is not included in that of $P'_Z$. This contrasts the Kullback-Leibler (KL) divergence $D_{KL}(P_Z||P'_Z)$-based bounds of Wu et al. [1], which are vacuous under this assumption. Moreover, the obtained bounds hold for unbounded loss functions with bounded cumulant generating functions, unlike the $\phi$-divergence based bound of Wu et al. [1]. We also obtain new upper bounds on the average transfer excess risk in terms of the $(\alpha_1,\alpha_2)$-JS divergence for empirical weighted risk minimization (EWRM), which minimizes the weighted average training losses over source and target data sets. Finally, we provide a numerical example to illustrate the merits of the introduced bounds.Comment: Submitted for conference publicatio

Damage Detection in Largely Unobserved Structures under Varying Environmental Conditions: An AutoRegressive Spectrum and Multi-Level Machine Learning Methodology

Vibration-based damage detection in civil structures using data-driven methods requires sufficient vibration responses acquired with a sensor network. Due to technical and economic reasons, it is not always possible to deploy a large number of sensors. This limitation may lead to partial information being handled for damage detection purposes, under environmental variability. To address this challenge, this article proposes an innovative multi-level machine learning method by employing the autoregressive spectrum as the main damage-sensitive feature. The proposed method consists of three levels: (i) distance calculation by the log-spectral distance, to increase damage detectability and generate distance-based training and test samples; (ii) feature normalization by an improved factor analysis, to remove environmental variations; and (iii) decision-making for damage localization by means of the Jensen-Shannon divergence. The major contributions of this research are represented by the development of the aforementioned multi-level machine learning method, and by the proposal of the new factor analysis for feature normalization. Limited vibration datasets relevant to a truss structure and consisting of acceleration time histories induced by shaker excitation in a passive system, have been used to validate the proposed method and to compare it with alternate, state-of-the-art strategies

MFLD-net: a lightweight deep learning network for fish morphometry using landmark detection

Monitoring the morphological traits of farmed fish is pivotal in understanding growth, estimating yield, artificial breeding, and population-based investigations. Currently, morphology measurements mostly happen manually and sometimes in conjunction with individual fish imaging, which is a time-consuming and expensive procedure. In addition, extracting useful information such as fish yield and detecting small variations due to growth or deformities, require extra offline processing of the manually collected images and data. Deep learning (DL) and specifically convolutional neural networks (CNNs) have previously demonstrated great promise in estimating fish features such as weight and length from images. However, their use for extracting fish morphological traits through detecting fish keypoints (landmarks) has not been fully explored. In this paper, we developed a novel DL architecture that we call Mobile Fish Landmark Detection network (MFLD-net). We show that MFLD-net can achieve keypoint detection accuracies on par or even better than some of the state-of-the-art CNNs on a fish image dataset. MFLD-net uses convolution operations based on Vision Transformers (i.e. patch embeddings, multi-layer perceptrons). We show that MFLD-net can achieve competitive or better results in low data regimes while being lightweight and therefore suitable for embedded and mobile devices. We also provide quantitative and qualitative results that demonstrate its generalisation capabilities. These features make MFLD-net suitable for future deployment in fish farms and fish harvesting plants

A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources

Electronic Health Records (EHRs) are a valuable asset to facilitate clinical research and point of care applications; however, many challenges such as data privacy concerns impede its optimal utilization. Deep generative models, particularly, Generative Adversarial Networks (GANs) show great promise in generating synthetic EHR data by learning underlying data distributions while achieving excellent performance and addressing these challenges. This work aims to review the major developments in various applications of GANs for EHRs and provides an overview of the proposed methodologies. For this purpose, we combine perspectives from healthcare applications and machine learning techniques in terms of source datasets and the fidelity and privacy evaluation of the generated synthetic datasets. We also compile a list of the metrics and datasets used by the reviewed works, which can be utilized as benchmarks for future research in the field. We conclude by discussing challenges in GANs for EHRs development and proposing recommended practices. We hope that this work motivates novel research development directions in the intersection of healthcare and machine learning

Researching the Potential of Artificial Intelligence to Support the Understanding of Neurological Diseases: The Cases for Frontotemporal Lobar Degeneration Detection and Mice Ultrasonic Communication Analysis

Questa ricerca di dottorato mira a esplorare le intersezioni e le sinergie tra l'apprendimento automatico e la neurologia. Il nucleo dello studio è costituito da tre indagini distinte ma interconnesse che mettono in evidenza il potenziale dell'intelligenza artificiale (IA) nel migliorare la nostra comprensione delle malattie neurodegenerative e della comunicazione animale, nonché nel migliorare i metodi di diagnosi. La prima parte di questa ricerca studia l'applicazione di tecniche di apprendimento automatico nella diagnosi della demenza frontotemporale utilizzando dati di risonanza magnetica. Lo studio utilizza principalmente un approccio di analisi a pattern multivoxel con algoritmi di Support Vector Machine e Random Forest per l'analisi. Questa sezione mira a affrontare le sfide nella rilevazione precoce delle malattie neurodegenerative, fornendo ai professionisti medici uno strumento diagnostico di supporto che potrebbe migliorare potenzialmente gli esiti del trattamento. La seconda parte della ricerca approfondisce l'analisi delle vocalizzazioni ultrasoniche nei topi, concentrandosi in particolare sulle modifiche nei modelli di comunicazione ultrasonica nei topi trattati con olio di Cannabis sativa rispetto ai topi di controllo. L'indagine utilizza attrezzature di registrazione specializzate e software dedicati per analizzare le vocalizzazioni ultrasoniche, facendo chiarezza sulle sfumature della comunicazione animale. Questo segmento discute delle disparità nei modelli di comunicazione ultrasonica tra i due gruppi, correlandoli a comportamenti specifici, presentando un'esaustiva esplorazione statistica. La terza parte dello studio presenta un flusso di elaborazione dei dati per analizzare le registrazioni audio dei topi, impiegando avanzate tecniche di elaborazione del segnale e di apprendimento automatico. Questo segmento introduce un meticoloso sistema di etichettatura dei dati che assegna ciascun segmento audio a una delle otto categorie comportamentali presenti nel dataset. I calcoli degli spettrogrammi vengono utilizzati per visualizzare le caratteristiche acustiche delle vocalizzazioni ad alta frequenza dei topi. Gli esperimenti di deep learning intrapresi in questa parte mirano a scoprire informazioni sui meccanismi neurali alla base delle vocalizzazioni ultrasoniche dei topi, arricchendo così la nostra comprensione della complessa relazione tra vocalizzazione e comportamento.This doctoral research aims to explore the intersections and synergies between machine learning and neurology. The study's core is comprised of three distinct but interconnected investigations that highlight the potential of artificial intelligence (AI) in enhancing our understanding of neurodegenerative diseases and animal communication, as well as improving diagnostic methods. The first part of this research investigates the application of machine learning techniques in the diagnosis of Fronto-Temporal Dementia using Magnetic Resonance Imaging data. The study primarily uses a Multi-Voxel Pattern Analysis approach with Support Vector Machine and Random Forest algorithms for analysis. This section aims to address the challenges in the early detection of neurodegenerative diseases, providing medical professionals with a supportive diagnostic tool that could potentially improve treatment outcomes. The second part of the research delves into the analysis of ultrasonic vocalizations in mice, specifically focusing on the changes in ultrasonic communication patterns in mice treated with Cannabis sativa oil as compared to control mice. The investigation utilizes specialized recording equipment and dedicated software to analyze USVs, shedding light on the nuances of animal communication. This segment discusses the disparities in ultrasonic communication patterns between the two groups, correlating them with specific behaviors, presenting a comprehensive statistical exploration. The third part of the study presents a data processing pipeline to analyze mouse audio data, employing advanced signal processing techniques and machine learning. This segment introduces a meticulous data labeling system that assigns each audio segment to one of eight behavioral categories. Spectrogram computations are used to visualize distinct acoustic characteristics of high-frequency mouse vocalizations. Deep learning experiments undertaken in this part aim to uncover insights into the neural mechanisms underlying mouse ultrasonic vocalizations, thereby enriching our understanding of the intricate relationship between vocalization and behavior

A survey of generative adversarial networks for synthesizing structured electronic health records

Electronic Health Records (EHRs) are a valuable asset to facilitate clinical research and point of care applications; however, many challenges such as data privacy concerns impede its optimal utilization. Deep generative models, particularly, Generative Adversarial Networks (GANs) show great promise in generating synthetic EHR data by learning underlying data distributions while achieving excellent performance and addressing these challenges. This work aims to survey the major developments in various applications of GANs for EHRs and provides an overview of the proposed methodologies. For this purpose, we combine perspectives from healthcare applications and machine learning techniques in terms of source datasets and the fidelity and privacy evaluation of the generated synthetic datasets. We also compile a list of the metrics and datasets used by the reviewed works, which can be utilized as benchmarks for future research in the field. We conclude by discussing challenges in GANs for EHRs development and proposing recommended practices. We hope that this work motivates novel research development directions in the intersection of healthcare and machine learning

Divergence Measures

Data science, information theory, probability theory, statistical learning and other related disciplines greatly benefit from non-negative measures of dissimilarity between pairs of probability measures. These are known as divergence measures, and exploring their mathematical foundations and diverse applications is of significant interest. The present Special Issue, entitled “Divergence Measures: Mathematical Foundations and Applications in Information-Theoretic and Statistical Problems”, includes eight original contributions, and it is focused on the study of the mathematical properties and applications of classical and generalized divergence measures from an information-theoretic perspective. It mainly deals with two key generalizations of the relative entropy: namely, the R_ényi divergence and the important class of f -divergences. It is our hope that the readers will find interest in this Special Issue, which will stimulate further research in the study of the mathematical foundations and applications of divergence measures