Modeling Uncertainty for Reliable Probabilistic Modeling in Deep Learning and Beyond

[ES] Esta tesis se enmarca en la intersección entre las técnicas modernas de Machine Learning, como las Redes Neuronales Profundas, y el modelado probabilístico confiable. En muchas aplicaciones, no solo nos importa la predicción hecha por un modelo (por ejemplo esta imagen de pulmón presenta cáncer) sino también la confianza que tiene el modelo para hacer esta predicción (por ejemplo esta imagen de pulmón presenta cáncer con 67% probabilidad). En tales aplicaciones, el modelo ayuda al tomador de decisiones (en este caso un médico) a tomar la decisión final. Como consecuencia, es necesario que las probabilidades proporcionadas por un modelo reflejen las proporciones reales presentes en el conjunto al que se ha asignado dichas probabilidades; de lo contrario, el modelo es inútil en la práctica. Cuando esto sucede, decimos que un modelo está perfectamente calibrado. En esta tesis se exploran tres vias para proveer modelos más calibrados. Primero se muestra como calibrar modelos de manera implicita, que son descalibrados por técnicas de aumentación de datos. Se introduce una función de coste que resuelve esta descalibración tomando como partida las ideas derivadas de la toma de decisiones con la regla de Bayes. Segundo, se muestra como calibrar modelos utilizando una etapa de post calibración implementada con una red neuronal Bayesiana. Finalmente, y en base a las limitaciones estudiadas en la red neuronal Bayesiana, que hipotetizamos que se basan en un prior mispecificado, se introduce un nuevo proceso estocástico que sirve como distribución a priori en un problema de inferencia Bayesiana.[CA] Aquesta tesi s'emmarca en la intersecció entre les tècniques modernes de Machine Learning, com ara les Xarxes Neuronals Profundes, i el modelatge probabilístic fiable. En moltes aplicacions, no només ens importa la predicció feta per un model (per ejemplem aquesta imatge de pulmó presenta càncer) sinó també la confiança que té el model per fer aquesta predicció (per exemple aquesta imatge de pulmó presenta càncer amb 67% probabilitat). En aquestes aplicacions, el model ajuda el prenedor de decisions (en aquest cas un metge) a prendre la decisió final. Com a conseqüència, cal que les probabilitats proporcionades per un model reflecteixin les proporcions reals presents en el conjunt a què s'han assignat aquestes probabilitats; altrament, el model és inútil a la pràctica. Quan això passa, diem que un model està perfectament calibrat. En aquesta tesi s'exploren tres vies per proveir models més calibrats. Primer es mostra com calibrar models de manera implícita, que són descalibrats per tècniques d'augmentació de dades. S'introdueix una funció de cost que resol aquesta descalibració prenent com a partida les idees derivades de la presa de decisions amb la regla de Bayes. Segon, es mostra com calibrar models utilitzant una etapa de post calibratge implementada amb una xarxa neuronal Bayesiana. Finalment, i segons les limitacions estudiades a la xarxa neuronal Bayesiana, que es basen en un prior mispecificat, s'introdueix un nou procés estocàstic que serveix com a distribució a priori en un problema d'inferència Bayesiana.[EN] This thesis is framed at the intersection between modern Machine Learning techniques, such as Deep Neural Networks, and reliable probabilistic modeling. In many machine learning applications, we do not only care about the prediction made by a model (e.g. this lung image presents cancer) but also in how confident is the model in making this prediction (e.g. this lung image presents cancer with 67% probability). In such applications, the model assists the decision-maker (in this case a doctor) towards making the final decision. As a consequence, one needs that the probabilities provided by a model reflects the true underlying set of outcomes, otherwise the model is useless in practice. When this happens, we say that a model is perfectly calibrated. In this thesis three ways are explored to provide more calibrated models. First, it is shown how to calibrate models implicitly, which are decalibrated by data augmentation techniques. A cost function is introduced that solves this decalibration taking as a starting point the ideas derived from decision making with Bayes' rule. Second, it shows how to calibrate models using a post-calibration stage implemented with a Bayesian neural network. Finally, and based on the limitations studied in the Bayesian neural network, which we hypothesize that came from a mispecified prior, a new stochastic process is introduced that serves as a priori distribution in a Bayesian inference problem.Maroñas Molano, J. (2022). Modeling Uncertainty for Reliable Probabilistic Modeling in Deep Learning and Beyond [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/181582TESI

Adversarial Learning with Ladder Networks

[EN] In this MT we are going to asses the inclusion of adversarial generative models into the Ladder semi-supervised networks. The implementation and experiments are carried out with Theano. Therefore the adversarial generative model has to be included in the derivation graph.[ES] En este TFM se va a analizar el impacto de introducir modlos generativos de muestras adversarias en las redes profundas semi-supervisadas del tipo "ladder". La implementación y experimentos se realizará en Theano con lo que se tendrá que incluir el modelo generativo adversario en el grado de derivación.Maroñas Molano, J. (2016). Adversarial Learning with Ladder Networks. http://hdl.handle.net/10251/77958TFG

Comparación de algoritmos de cálculo de ratios de verosimilitudes para interpretación forense

Este TFG compara varios algoritmos de cálculo de ratios de verosimilitudes (LR), en un entorno de casos forenses reales. El marco de aplicación son los sistemas automáticos de reconocimiento de locutores utilizados en casos forenses reales. Los LR son actualmente la recomendación para emitir conclusiones evaluativas en informes forenses en Europa. En la primera parte del TFG se hace uso de tres métodos populares para calcular LR a partir de puntuaciones de sistemas biométricos (llamadas “scores” en inglés): regresión logística (LogReg), modelado gaussiano de máxima verosimilitud (Gauss- ML) y Kernel Density Function Gaussianas (KDF). La comparación de estos métodos se lleva a cabo en el contexto de la evaluación NIST 2012 de reconocimiento de locutor (NIST SRE 2012), donde se presentan varios escenarios de habla conversacional telefónica y microfónica realista. El análisis pone de manifiesto problemas típicos en este tipo de entornos, como el desajuste de bases de datos, el sobreajuste de los modelos, etc. En la segunda parte del TFG se propone el uso del modelado gaussiano bayesiano (Gauss-Bayes), propuesto recientemente como alternativa a los tres métodos analizados anteriormente. Se concluye que, cuando existe gran cantidad de datos de entrenamiento de los modelos, este método equivale a Gauss-ML. Sin embargo, si los datos de entrenamiento son escasos, Gauss-Bayes supera ampliamente en rendimiento a Gauss-ML. Posteriormente, el TFG presenta una aportación original de aplicación: se propone un escenario real en ciencia forense, en el que las proposiciones definidas en el caso dan lugar a una falta considerable de scores de entrenamiento para los modelos. Este caso es muy común en ciencia forense, tal y como se describe. Se muestra que el método Gauss-Bayes supera con creces al método Gauss-ML en este escenario, dando lugar a cálculos de LR robustos y coherentes. Más aún, se proponen dos esquemas de cálculo de scores de entrenamiento (llamados también esquemas de anclaje, o “anchoring”) que pueden ser adecuados para el uso en casos forenses reales, y se presenta el rendimiento para ambos esquemas, constatando que Gauss-Bayes es la mejor opción en dichos escenarios. Los resultados de esta sección son de relevancia, y previsiblemente se enviarán para publicación tras la finalización del TFG.This TFG compares several algorithm for likelihood ratio (LR) computation, in real forensic enviroments. The implementation framework is automatic speaker recognition systems used for real forensic cases. LR are actually recommended to issue evaluative conclusions on forensic reports in Europe. In the first part, three popular LR computation methods are used for calculating LR based on scores from a biometric system: Logit Regression (LogReg), maximum likelihood Gaussian modeling (Gauss-ML) and Gaussian Kernel Density Funtion (KDF). Comparison of these methods is carried out in the context of the 2012 NIST speaker recognition evaluation (NIST SER 2012), where various call scenarios and realistic conversational speech microphone are present. This analysis shows tipical problems in this scenarios like database mismatch, model overfitting… In the second part it is proposed the use of bayesian gaussian modeling , recently proposed as an alternative for the three methods proposed before. It is concluded that when there is lots of train data this methods is equivalent to Gauss-ML. However, if there is few train data, bayesian method outperforms widely Gauss-ML. Finally, this TFG shows an original framework application: a real forensic scenario is proposed in which de propositions defined by the case imply a considerable lack of train scores. This is a very tipical case in forensic science, as it is described. It is showed that bayesian method is much better tan Gauss-ML in this kind of scenario, computing robust and coherent LR. Furthermore, two score computing schemes (also known as anchoring schemes) can be adecuated for being used in real forensic cases, and performance it is presented for both schemes, noting that bayesian estimation is the best choice in these scenarios. Results from these section are relevant, and likely will be submitted for publication after the end of the TFG

Towards Efficient Modeling and Inference in Multi-Dimensional Gaussian Process State-Space Models

The Gaussian process state-space model (GPSSM) has attracted extensive attention for modeling complex nonlinear dynamical systems. However, the existing GPSSM employs separate Gaussian processes (GPs) for each latent state dimension, leading to escalating computational complexity and parameter proliferation, thus posing challenges for modeling dynamical systems with high-dimensional latent states. To surmount this obstacle, we propose to integrate the efficient transformed Gaussian process (ETGP) into the GPSSM, which involves pushing a shared GP through multiple normalizing flows to efficiently model the transition function in high-dimensional latent state space. Additionally, we develop a corresponding variational inference algorithm that surpasses existing methods in terms of parameter count and computational complexity. Experimental results on diverse synthetic and real-world datasets corroborate the efficiency of the proposed method, while also demonstrating its ability to achieve similar inference performance compared to existing methods. Code is available at \url{https://github.com/zhidilin/gpssmProj}

Gaussianization of LA-ICP-MS features to improve calibration in forensic glass comparison

The forensic comparison of glass aims to compare a glass sample of an unknown source with a control glass sample of a known source. In this work, we use multi-elemental features from Laser Ablation Inductively Coupled Plasma with Mass Spectrometry (LA-ICP-MS) to compute a likelihood ratio. This calculation is a complex procedure that generally requires a probabilistic model including the within-source and betweensource variabilities of the features. Assuming the within-source variability to be normally distributed is a practical premise with the available data. However, the between-source variability is generally assumed to follow a much more complex distribution, typically described with a kernel density function. In this work, instead of modeling distributions with complex densities, we propose the use of simpler models and the introduction of a data pre-processing step consisting on the Gaussianization of the glass features. In this context, to obtain a better fit of the features with the Gaussian model assumptions, we explore the use of different normalization techniques of the LA-ICP-MS glass features, namely marginal Gaussianization based on histogram matching, marginal Gaussianization based on Yeo-Johnson transformation and a more complex joint Gaussianization using normalizing flows. We report an improvement in the performance of the Likelihood Ratios computed with the previously Gaussianized feature vectors, particularly relevant in their calibration, which implies a more reliable forensic glass comparisonThis work has been supported by the Spanish Ministerio de Ciencia e Innovación through grant PID2021-125943OB-I0

Confidence Calibration for Deep Renal Biopsy Immunofluorescence Image Classification

With this work we tackle immunofluorescence classification in renal biopsy, employing state-of-the-art Convolutional Neural Networks. In this setting, the aim of the probabilistic model is to assist an expert practitioner towards identifying the location pattern of antibody deposits within a glomerulus. Since modern neural networks often provide overconfident outputs, we stress the importance of having a reliable prediction, demonstrating that Temperature Scaling (TS), a recently introduced re-calibration technique, can be successfully applied to immunofluorescence classification in renal biopsy. Experimental results demonstrate that the designed model yields good accuracy on the specific task, and that TS is able to provide reliable probabilities, which are highly valuable for such a task given the low inter-rater agreement