Essays on variational Bayes in Econometrics

The first essay (Chapter 1) presents a Variational Bayes (Vb) algorithm for Vector Autoregression (reduced-form VAR). The algorithm is derived based on the evidence lower bound, which is demonstrated to be tight, ensuring efficient convergence. The optimization is carried through the Coordinate descent optimization. To validate the proposed method, its accuracy and computational costs are compared with existing Vb approaches that approximate VAR using a one equation at a time technique (Choleskytransformed VAR), and a more computationally intensive Markov Chain Monte Carlo (MCMC) method using Gibbs sampling. In applications using both US macroeconomic data and artificial data, our Vb for VAR outperforms Vb in Cholesky-transformed VAR in terms of VAR covariance accuracy. Furthermore, compared to the MCMC method, our proposed Vb algorithm for reduced form VAR achieves comparable accuracy while significantly reducing computation time. The second essay (Chapter 2) takes the Variational Bayes (Vb) approach to the next level by extending it to the challenging domain of Mixed Frequency Vector Autoregression (MF-VAR) models. These models tackle the complexities of dealing with multiple frequency data in a single estimation, including the issue of missing lower frequency observations in a higher frequency system. To overcome these challenges, we introduce a robust and innovative Vb method known as the Variational Bayes-Expectation Maximization algorithm (Vb-EM). Our Vb-EM algorithm offers several key contributions to approximate Bayesian inference in the MF-VAR model. We derive an evidence lower bound to the log marginal likelihood, accounting for missing observations, and optimize it with respect to the variational parameters. In doing so, we surpass existing Vb methods in the literature by achieving a tighter evidence lower bound, ensuring optimal convergence. To further validate our approach, we compare it to the more computationally demanding Markov Chain Monte Carlo (MCMC) method using Gibbs sampling. Through extensive empirical evaluations and out-of-sample forecasts of eleven US macroeconomic series, we demonstrate that our Vb EM algorithm performs on par with MCMC in terms of point forecasts. Furthermore, when assessing predictive density, we find no significant empirical evidence to distinguish between the two methods. Notably, our Vb-EM algorithm offers the distinct advantage of significantly lower computational costs, making it an appealing choice for researchers and practitioners alike. The third essay (Chapter 3) begins by emphasizing that the spike of volatilities of macroeconomic variables during the surge of Covid-19 pandemic, which led to poor performance of the workhorse Bayesian VAR with stochastic volatility in terms of forecasting. This has attracted considerable attention from economists towards alternative models, including non-parametric models such as Gaussian process VAR. The approach to estimate VAR one equation at a time, namely Cholesky-transformed VARs, enables the application of more advanced regression models in VAR. In this chapter I explore several advanced Gaussian process VARs, including GP-VAR, GP-DNN-VAR (which incorporates a deep neural network as the mean function in the GP prior), and Heteroscedastic-GP-VAR (HGP-VAR) where the likelihood variance is assumed to be time-varying and parameterized by another latent-GP function. In this chapter the variational inference is utilized to be the approximating method for HGP-VAR. The forecasting results suggest that during non pandemic periods, HGP-VAR and GP-VAR perform similarly to BVAR-SV. However, during the Covid-19 pandemic, the advantage of having time-variant likelihood variance in HGP-VAR becomes more pronounced for predicting macroeconomic variables in a highly turbulent period.The first essay (Chapter 1) presents a Variational Bayes (Vb) algorithm for Vector Autoregression (reduced-form VAR). The algorithm is derived based on the evidence lower bound, which is demonstrated to be tight, ensuring efficient convergence. The optimization is carried through the Coordinate descent optimization. To validate the proposed method, its accuracy and computational costs are compared with existing Vb approaches that approximate VAR using a one equation at a time technique (Choleskytransformed VAR), and a more computationally intensive Markov Chain Monte Carlo (MCMC) method using Gibbs sampling. In applications using both US macroeconomic data and artificial data, our Vb for VAR outperforms Vb in Cholesky-transformed VAR in terms of VAR covariance accuracy. Furthermore, compared to the MCMC method, our proposed Vb algorithm for reduced form VAR achieves comparable accuracy while significantly reducing computation time. The second essay (Chapter 2) takes the Variational Bayes (Vb) approach to the next level by extending it to the challenging domain of Mixed Frequency Vector Autoregression (MF-VAR) models. These models tackle the complexities of dealing with multiple frequency data in a single estimation, including the issue of missing lower frequency observations in a higher frequency system. To overcome these challenges, we introduce a robust and innovative Vb method known as the Variational Bayes-Expectation Maximization algorithm (Vb-EM). Our Vb-EM algorithm offers several key contributions to approximate Bayesian inference in the MF-VAR model. We derive an evidence lower bound to the log marginal likelihood, accounting for missing observations, and optimize it with respect to the variational parameters. In doing so, we surpass existing Vb methods in the literature by achieving a tighter evidence lower bound, ensuring optimal convergence. To further validate our approach, we compare it to the more computationally demanding Markov Chain Monte Carlo (MCMC) method using Gibbs sampling. Through extensive empirical evaluations and out-of-sample forecasts of eleven US macroeconomic series, we demonstrate that our Vb EM algorithm performs on par with MCMC in terms of point forecasts. Furthermore, when assessing predictive density, we find no significant empirical evidence to distinguish between the two methods. Notably, our Vb-EM algorithm offers the distinct advantage of significantly lower computational costs, making it an appealing choice for researchers and practitioners alike. The third essay (Chapter 3) begins by emphasizing that the spike of volatilities of macroeconomic variables during the surge of Covid-19 pandemic, which led to poor performance of the workhorse Bayesian VAR with stochastic volatility in terms of forecasting. This has attracted considerable attention from economists towards alternative models, including non-parametric models such as Gaussian process VAR. The approach to estimate VAR one equation at a time, namely Cholesky-transformed VARs, enables the application of more advanced regression models in VAR. In this chapter I explore several advanced Gaussian process VARs, including GP-VAR, GP-DNN-VAR (which incorporates a deep neural network as the mean function in the GP prior), and Heteroscedastic-GP-VAR (HGP-VAR) where the likelihood variance is assumed to be time-varying and parameterized by another latent-GP function. In this chapter the variational inference is utilized to be the approximating method for HGP-VAR. The forecasting results suggest that during non pandemic periods, HGP-VAR and GP-VAR perform similarly to BVAR-SV. However, during the Covid-19 pandemic, the advantage of having time-variant likelihood variance in HGP-VAR becomes more pronounced for predicting macroeconomic variables in a highly turbulent period

COVID-19 Outbreak Prediction with Machine Learning

Abstract: Several outbreak prediction models for COVID-19 are being used by officials around the world to make informed decisions and enforce relevant control measures. Among the standard models for COVID-19 global pandemic prediction, simple epidemiological and statistical models have received more attention by authorities, and these models are popular in the media. Due to a high Level of uncertainty and lack of essential data, standard models have shown low accuracy for long-term prediction. Although the literature includes several attempts to address this issue, the essential generalization and robustness abilities of existing models need to be improved. This paper presents a comparative analysis ofmachine learning and soft computingmodels to predict the COVID-19 outbreak as an alternative to susceptible–infected–recovered (SIR) and susceptible-exposed-infectious-removed (SEIR) models. Among a wide range of machine learning models investigated, two models showed promising results (i.e., multi-layered perceptron, MLP; and adaptive network-based fuzzy inference system, ANFIS). Based on the results reported here, and due to the highly complex nature of the COVID-19 outbreak and variation in its behavior across nations, this study suggests machine Learning as an effective tool to model the outbreak. This paper provides an initial benchmarking to demonstrate the potential of machine learning for future research. This paper further suggests that a genuine novelty in outbreak prediction can be realized by integrating machine learning and SEIR models.publishedVersio

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

Automatic Image Classification for Planetary Exploration

Autonomous techniques in the context of planetary exploration can maximize scientific return and reduce the need for human involvement. This thesis work studies two main problems in planetary exploration: rock image classification and hyperspectral image classification. Since rock textural images are usually inhomogeneous and manually hand-crafting features is not always reliable, we propose an unsupervised feature learning method to autonomously learn the feature representation for rock images. The proposed feature method is flexible and can outperform manually selected features. In order to take advantage of the unlabelled rock images, we also propose self-taught learning technique to learn the feature representation from unlabelled rock images and then apply the features for the classification of the subclass of rock images. Since combining spatial information with spectral information for classifying hyperspectral images (HSI) can dramatically improve the performance, we first propose an innovative framework to automatically generate spatial-spectral features for HSI. Two unsupervised learning methods, K-means and PCA, are utilized to learn the spatial feature bases in each decorrelated spectral band. Then spatial-spectral features are generated by concatenating the spatial feature representations in all/principal spectral bands. In the second work for HSI classification, we propose to stack the spectral patches to reduce the spectral dimensionality and generate 2-D spectral quilts. Such quilts retain all the spectral information and can result in less convolutional parameters in neural networks. Two light convolutional neural networks are then designed to classify the spectral quilts. As the third work for HSI classification, we propose a combinational fully convolutional network. The network can not only take advantage of the inherent computational efficiency of convolution at prediction time, but also perform as a collection of many paths and has an ensemble-like behavior which guarantees the robust performance

Mining Oncology Data: Knowledge Discovery in Clinical Performance of Cancer Patients

Our goal in this research is twofold: to develop clinical performance databases of cancer patients, and to conduct data mining and machine learning studies on collected patient records. We use these studies to develop models for predicting cancer patient medical outcomes. The clinical database is developed in conjunction with surgeons and oncologists at UMass Memorial Hospital. Aspects of the database design and representation of patient narrative are discussed here. Current predictive model design in medical literature is dominated by linear and logistic regression techniques. We seek to show that novel machine learning methods can perform as well or better than these traditional techniques. Our machine learning focus for this thesis is on pancreatic cancer patients. Classification and regression prediction targets include patient survival, wellbeing scores, and disease characteristics. Information research in oncology is often constrained by type variation, missing attributes, high dimensionality, skewed class distribution, and small data sets. We compensate for these difficulties using preprocessing, meta-learning, and other algorithmic methods during data analysis. The predictive accuracy and regression error of various machine learning models are presented as results, as are t-tests comparing these to the accuracy of traditional regression methods. In most cases, it is shown that the novel machine learning prediction methods offer comparable or superior performance. We conclude with an analysis of results and discussion of future research possibilities

COVID-19 Outbreak Prediction with Machine Learning

Abstract: Several outbreak prediction models for COVID-19 are being used by officials around the world to make informed decisions and enforce relevant control measures. Among the standard models for COVID-19 global pandemic prediction, simple epidemiological and statistical models have received more attention by authorities, and these models are popular in the media. Due to a high Level of uncertainty and lack of essential data, standard models have shown low accuracy for long-term prediction. Although the literature includes several attempts to address this issue, the essential generalization and robustness abilities of existing models need to be improved. This paper presents a comparative analysis ofmachine learning and soft computingmodels to predict the COVID-19 outbreak as an alternative to susceptible–infected–recovered (SIR) and susceptible-exposed-infectious-removed (SEIR) models. Among a wide range of machine learning models investigated, two models showed promising results (i.e., multi-layered perceptron, MLP; and adaptive network-based fuzzy inference system, ANFIS). Based on the results reported here, and due to the highly complex nature of the COVID-19 outbreak and variation in its behavior across nations, this study suggests machine Learning as an effective tool to model the outbreak. This paper provides an initial benchmarking to demonstrate the potential of machine learning for future research. This paper further suggests that a genuine novelty in outbreak prediction can be realized by integrating machine learning and SEIR models.publishedVersio

Max-margin stacking with group sparse regularization for classifier combination

Multiple classifier systems are shown to be effective in terms of accuracy for multiclass classification problems with the expense of increased complexity. Classifier combination studies deal with the methods of combining the outputs of base classifiers of an ensemble. Stacked generalization, or stacking, is shown to be a strong combination scheme among combination algorithms; and in this thesis, we improve stacking's performance further in terms of both accuracy and complexity. We investigate four main issues for this purpose. First, we show that margin maximizing combiners outperform the conventional least-squares estimation of the weights. Second we incorporate the idea of group sparsity into regularization to facilitate classifier selection. Third, we develop non-linear versions of class-conscious linear combination types by transforming datasets into binary classification datasets; then applying the kernel trick. And finally, we derive a new optimization algorithm based on the majorization-minimization framework for a particular linear combination type, which we show is the most preferable one

Machine Learning Small Molecule Properties in Drug Discovery

Machine learning (ML) is a promising approach for predicting small molecule properties in drug discovery. Here, we provide a comprehensive overview of various ML methods introduced for this purpose in recent years. We review a wide range of properties, including binding affinities, solubility, and ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity). We discuss existing popular datasets and molecular descriptors and embeddings, such as chemical fingerprints and graph-based neural networks. We highlight also challenges of predicting and optimizing multiple properties during hit-to-lead and lead optimization stages of drug discovery and explore briefly possible multi-objective optimization techniques that can be used to balance diverse properties while optimizing lead candidates. Finally, techniques to provide an understanding of model predictions, especially for critical decision-making in drug discovery are assessed. Overall, this review provides insights into the landscape of ML models for small molecule property predictions in drug discovery. So far, there are multiple diverse approaches, but their performances are often comparable. Neural networks, while more flexible, do not always outperform simpler models. This shows that the availability of high-quality training data remains crucial for training accurate models and there is a need for standardized benchmarks, additional performance metrics, and best practices to enable richer comparisons between the different techniques and models that can shed a better light on the differences between the many techniques.Comment: 46 pages, 1 figur