Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study

Coping with distributional shifts is an important part of transfer learning methods in order to perform well in real-life tasks. However, most of the existing approaches in this area either focus on an ideal scenario in which the data does not contain noises or employ a complicated training paradigm or model design to deal with distributional shifts. In this paper, we revisit the robustness of the minimum error entropy (MEE) criterion, a widely used objective in statistical signal processing to deal with non-Gaussian noises, and investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common. Specifically, we put forward a new theoretical result showing the robustness of MEE against covariate shift. We also show that by simply replacing the mean squared error (MSE) loss with the MEE on basic transfer learning algorithms such as fine-tuning and linear probing, we can achieve competitive performance with respect to state-of-the-art transfer learning algorithms. We justify our arguments on both synthetic data and 5 real-world time-series data.Comment: Manuscript accepted at ECAI-23. Code available at https://github.com/lpsilvestrin/mee-finetun

MR spectroscopy signal quantification using deep learning

Sinais de espectroscopia por ressonância magnética nuclear são utilizados no diagnóstico de doenças importantes tais como Alzheimer, câncer, entre outras. Isso é possível através da quantificação dos metabólitos presentes nos órgãos sujeitos ao exame de espectroscopia. Porém, a presença de ruído e a sobreposição dos sinais emitidos por alguns metabólitos podem tornar o resultado da quantificação impreciso. Neste trabalho, nós implementamos e testamos diversas arquiteturas de redes neurais convolucionais para quantificar sinais de espectroscopia, e comparamos elas com o QUEST, que é a técnica estado-da-arte nessa área, utilizando o erro relativo como medida. Nossos resultados mostram que o erro das redes neurais é aproximadamente 3 vezes menor que o QUEST para sinais contaminados com ruído gaussiano e sinal de fundo. Esse resultado é promissor e mostra que Deep Learning é uma abordagem para a quantificação de sinais de espectroscopia que merece ser explorada

MR spectroscopy signal quantification using deep learning

Sinais de espectroscopia por ressonância magnética nuclear são utilizados no diagnóstico de doenças importantes tais como Alzheimer, câncer, entre outras. Isso é possível através da quantificação dos metabólitos presentes nos órgãos sujeitos ao exame de espectroscopia. Porém, a presença de ruído e a sobreposição dos sinais emitidos por alguns metabólitos podem tornar o resultado da quantificação impreciso. Neste trabalho, nós implementamos e testamos diversas arquiteturas de redes neurais convolucionais para quantificar sinais de espectroscopia, e comparamos elas com o QUEST, que é a técnica estado-da-arte nessa área, utilizando o erro relativo como medida. Nossos resultados mostram que o erro das redes neurais é aproximadamente 3 vezes menor que o QUEST para sinais contaminados com ruído gaussiano e sinal de fundo. Esse resultado é promissor e mostra que Deep Learning é uma abordagem para a quantificação de sinais de espectroscopia que merece ser explorada

A Comparative Study of State-of-the-Art Machine Learning Algorithms for Predictive Maintenance

Predictive maintenance strives to maximize the availability of engineering systems. Over the last decade, machine learning has started to play a pivotal role in the domain to predict failures in machines and thus contribute to predictive maintenance. Ample approaches have been proposed to exploit machine learning based on sensory data obtained from engineering systems. Traditionally, these were based on feature engineering from the data followed by the application of a traditional machine learning algorithm. Recently, also deep learning approaches that are able to extract the features automatically have been utilized (including LSTMs and Convolutional Neural Networks), showing promising results. However, deep learning approaches need a substantial amount of data to be effective. Also, novel developments in deep learning architectures for time series have not been applied to predictive maintenance so far. In this paper, we compare a variety of different traditional machine learning and deep learning approaches to a representative (and modestly sized) predictive maintenance dataset and study their differences. In the deep learning approaches, we include a recently proposed approach that has not been tested for predictive maintenance yet: the temporal convolutional neural network. We compare the approaches over different sizes of the training dataset. The results show that, when the data is scarce, the temporal convolutional network performs better than the common deep learning approaches applied to predictive maintenance. However, it does not beat the more traditional feature engineering based approaches

Revisiting the Robustness of the Minimum Error Entropy Criterion:A Transfer Learning Case Study

Coping with distributional shifts is an important part of transfer learning methods in order to perform well in real-life tasks. However, most of the existing approaches in this area either focus on an ideal scenario in which the data does not contain noises or employ a complicated training paradigm or model design to deal with distributional shifts. In this paper, we revisit the robustness of the minimum error entropy (MEE) criterion, a widely used objective in statistical signal processing to deal with non-Gaussian noises, and investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common. Specifically, we put forward a new theoretical result showing the robustness of MEE against covariate shift. We also show that by simply replacing the mean squared error (MSE) loss with the MEE on basic transfer learning algorithms such as fine-tuning and linear probing, we can achieve competitive performance with respect to state-of-the-art transfer learning algorithms. We justify our arguments on both synthetic data and 5 real-world time-series data.</p

Transfer learning across datasets with different input dimensions:An algorithm and analysis for the linear regression case

With the development of new sensors and monitoring devices, more sources of data become available to be used as inputs for machine learning models. These can on the one hand help to improve the accuracy of a model. On the other hand, combining these new inputs with historical data remains a challenge that has not yet been studied in enough detail. In this work, we propose a transfer learning algorithm that combines new and historical data with different input dimensions. This approach is easy to implement, efficient, with computational complexity equivalent to the ordinary least-squares method, and requires no hyperparameter tuning, making it straightforward to apply when the new data is limited. Different from other approaches, we provide a rigorous theoretical study of its robustness, showing that it cannot be outperformed by a baseline that utilizes only the new data. Our approach achieves state-of-the-art performance on 9 real-life datasets, outperforming the linear DSFT, another linear transfer learning algorithm, and performing comparably to non-linear DSFT.