Parameter Tuning Using Gaussian Processes

Most machine learning algorithms require us to set up their parameter values before applying these algorithms to solve problems. Appropriate parameter settings will bring good performance while inappropriate parameter settings generally result in poor modelling. Hence, it is necessary to acquire the “best” parameter values for a particular algorithm before building the model. The “best” model not only reflects the “real” function and is well fitted to existing points, but also gives good performance when making predictions for new points with previously unseen values. A number of methods exist that have been proposed to optimize parameter values. The basic idea of all such methods is a trial-and-error process whereas the work presented in this thesis employs Gaussian process (GP) regression to optimize the parameter values of a given machine learning algorithm. In this thesis, we consider the optimization of only two-parameter learning algorithms. All the possible parameter values are specified in a 2-dimensional grid in this work. To avoid brute-force search, Gaussian Process Optimization (GPO) makes use of “expected improvement” to pick useful points rather than validating every point of the grid step by step. The point with the highest expected improvement is evaluated using cross-validation and the resulting data point is added to the training set for the Gaussian process model. This process is repeated until a stopping criterion is met. The final model is built using the learning algorithm based on the best parameter values identified in this process. In order to test the effectiveness of this optimization method on regression and classification problems, we use it to optimize parameters of some well-known machine learning algorithms, such as decision tree learning, support vector machines and boosting with trees. Through the analysis of experimental results obtained on datasets from the UCI repository, we find that the GPO algorithm yields competitive performance compared with a brute-force approach, while exhibiting a distinct advantage in terms of training time and number of cross-validation runs. Overall, the GPO method is a promising method for the optimization of parameter values in machine learning

Machine learning-driven credit risk: a systemic review

Credit risk assessment is at the core of modern economies. Traditionally, it is measured by statistical methods and manual auditing. Recent advances in financial artificial intelligence stemmed from a new wave of machine learning (ML)-driven credit risk models that gained tremendous attention from both industry and academia. In this paper, we systematically review a series of major research contributions (76 papers) over the past eight years using statistical, machine learning and deep learning techniques to address the problems of credit risk. Specifically, we propose a novel classification methodology for ML-driven credit risk algorithms and their performance ranking using public datasets. We further discuss the challenges including data imbalance, dataset inconsistency, model transparency, and inadequate utilization of deep learning models. The results of our review show that: 1) most deep learning models outperform classic machine learning and statistical algorithms in credit risk estimation, and 2) ensemble methods provide higher accuracy compared with single models. Finally, we present summary tables in terms of datasets and proposed models

Forecasting Financial Distress With Machine Learning – A Review

Purpose – Evaluate the various academic researches with multiple views on credit risk and artificial intelligence (AI) and their evolution.Theoretical framework – The study is divided as follows: Section 1 introduces the article. Section 2 deals with credit risk and its relationship with computational models and techniques. Section 3 presents the methodology. Section 4 addresses a discussion of the results and challenges on the topic. Finally, section 5 presents the conclusions.Design/methodology/approach – A systematic review of the literature was carried out without defining the time period and using the Web of Science and Scopus database.Findings – The application of computational technology in the scope of credit risk analysis has drawn attention in a unique way. It was found that the demand for identification and introduction of new variables, classifiers and more assertive methods is constant. The effort to improve the interpretation of data and models is intense.Research, Practical & Social implications – It contributes to the verification of the theory, providing information in relation to the most used methods and techniques, it brings a wide analysis to deepen the knowledge of the factors and variables on the theme. It categorizes the lines of research and provides a summary of the literature, which serves as a reference, in addition to suggesting future research.Originality/value – Research in the area of Artificial Intelligence and Machine Learning is recent and requires attention and investigation, thus, this study contributes to the opening of new views in order to deepen the work on this topic

Supervised classification and mathematical optimization

Data Mining techniques often ask for the resolution of optimization problems. Supervised Classification, and, in particular, Support Vector Machines, can be seen as a paradigmatic instance. In this paper, some links between Mathematical Optimization methods and Supervised Classification are emphasized. It is shown that many different areas of Mathematical Optimization play a central role in off-the-shelf Supervised Classification methods. Moreover, Mathematical Optimization turns out to be extremely useful to address important issues in Classification, such as identifying relevant variables, improving the interpretability of classifiers or dealing with vagueness/noise in the data.Ministerio de Ciencia e InnovaciónJunta de Andalucí

Credit risk evaluation modeling using evolutionary linear SVM classifiers and sliding window approach

AbstractThis paper presents a study on credit risk evaluation modeling using linear Support Vector Machines (SVM) classifiers, combined with evolutionary parameter selection using Genetic Algorithms and Particle Swarm Optimization, and sliding window approach. Discriminant analysis was applied for evaluation of financial instances and dynamic formation of bankruptcy classes. The possibilities of feature selection application were also researched by applying correlation-based feature subset evaluator. The research demonstrates a possibility to develop and apply an intelligent classifier based on original discriminant analysis method evaluation and shows that it might perform bankruptcy identification better than original model

Attribute Reduction for Credit Evaluation using Rough Set Approach

Generation of an Integrated Model is an important technique in the research area. It is a powerful technique to improve the accuracy of classifiers. This approach has been applied to different types of real time data. The unprocessed data leads to give wrong results by using some of the machine learning techniques. For generation of an integrated model attribute reduction and re-sampling technique is necessary. For attribute reduction Rough set is the best approach as it requires less execution time, high Interpretability, high reduction rate and high accurac

Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)

We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for quality. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique. In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.Comment: 10 pages + 2 references. Accepted to International Conference of Software Engineering (ICSE), 201

Developing retail performance measurement and financial distress prediction systems by using credit scoring techniques

The current research develops a theoretical framework based on the ResourceAdvantage Theory of Competition (Hunt, 2000) for the selection of appropriate variables. Using a review of the literature as well as to interviews and a survey, 170 potential retail performance variables were identified as possible for inclusion in the model. To produce a relative simple model with the aim of avoiding over-fitting, a limited number of key variables or principal components were selected to predict default. Five credit-scoring techniques: Naive Bayes, Logistic Regression, Recursive Partitioning, Artificial Neural Network, and Sequential Minimal Optimization (SMO) were employed on a sample of 195 healthy and 51 distressed businesses from the USA market over five time periods: 1994-1998, 1995-1999, 1996-2000, 1997-2001 and 1998-2002.Analyses provide sufficient evidence that the five credit scoring methodologies have sound classification ability in the year before financial distress. Moreover, they still remained sound even five years prior to financial distress. However, it is difficult to conclude which modelling technique has the highest classification ability uniformly, since model performance varied in terms of different time scales. The analysis also showed that external environment influences do impact on default assessment for all five credit-scoring techniques, but these influences are weak. These findings indicate that the developed models are theoretically sound. There is however a need to compare their performance to other approaches.To explore the issue of the model's performance two approaches are taken. First, rankings from the study were compared with those from a standard rating system—in this case the well-established Moody's Credit Rating. It is assumed that the higher the degree of similarity between the two sets of rankings, the greater the credibility of the prediction model. The results indicated that the logistic regression model and the SMO model were most comparable with Moody's. Secondly, the model's performance was assessed by applying it to different geographical areas. The original USA model was therefore applied to a new US data set as well as the European and Japanese markets. Results indicated that all market models displayed similar discriminating ability one year prior to financial distress. However, the USA model performed relatively better than European and Japanese models five years before financial distress. This implied that a financial distress model has potentially better prediction ability when based on a single market.Following this result it was decided to explore the performance of a generic global model, since model construction is time-consuming and costly. A composite model was constructed by combining data from USA, European and Japanese markets. This composite model had sound prediction performance, even up to five years before financial distress, as the accuracy rate was above 85.15% and AUROC value was above 0.7202. Comparing with the original USA model, the composite model has similar prediction performance in terms of the accuracy rate. However, the composite model presented a worse prediction utility based on the AUROC value. A future research direction might be to include more world retailing markets in order to ensure the model's prediction utility and practical applicability