"Task-relevant autoencoding" enhances machine learning for human neuroscience

In human neuroscience, machine learning can help reveal lower-dimensional neural representations relevant to subjects' behavior. However, state-of-the-art models typically require large datasets to train, so are prone to overfitting on human neuroimaging data that often possess few samples but many input dimensions. Here, we capitalized on the fact that the features we seek in human neuroscience are precisely those relevant to subjects' behavior. We thus developed a Task-Relevant Autoencoder via Classifier Enhancement (TRACE), and tested its ability to extract behaviorally-relevant, separable representations compared to a standard autoencoder, a variational autoencoder, and principal component analysis for two severely truncated machine learning datasets. We then evaluated all models on fMRI data from 59 subjects who observed animals and objects. TRACE outperformed all models nearly unilaterally, showing up to 12% increased classification accuracy and up to 56% improvement in discovering "cleaner", task-relevant representations. These results showcase TRACE's potential for a wide variety of data related to human behavior.Comment: 41 pages, 11 figures, 5 tables including supplemental materia

Confidence Inference in Defensive Cyber Operator Decision Making

Cyber defense analysts face the challenge of validating machine generated alerts regarding network-based security threats. Operations tempo and systematic manpower issues have increased the importance of these individual analyst decisions, since they typically are not reviewed or changed. Analysts may not always be confident in their decisions. If confidence can be accurately assessed, then analyst decisions made under low confidence can be independently reviewed and analysts can be offered decision assistance or additional training. This work investigates the utility of using neurophysiological and behavioral correlates of decision confidence to train machine learning models to infer confidence in analyst decisions. Electroencephalography (EEG) and behavioral data was collected from eight participants in a two-task human-subject experiment and used to fit several popular classifiers. Results suggest that for simple decisions, it is possible to classify analyst decision confidence using EEG signals. However, more work is required to evaluate the utility of EEG signals for classification of decision confidence in complex decisions

Psychometrics in Practice at RCEC

A broad range of topics is dealt with in this volume: from combining the psychometric generalizability and item response theories to the ideas for an integrated formative use of data-driven decision making, assessment for learning and diagnostic testing. A number of chapters pay attention to computerized (adaptive) and classification testing. Other chapters treat the quality of testing in a general sense, but for topics like maintaining standards or the testing of writing ability, the quality of testing is dealt with more specifically.\ud All authors are connected to RCEC as researchers. They present one of their current research topics and provide some insight into the focus of RCEC. The selection of the topics and the editing intends that the book should be of special interest to educational researchers, psychometricians and practitioners in educational assessment

Evaluating Classifiers During Dataset Shift

Deployment of a classifier into a machine learning application likely begins with training different types of algorithms on a subset of the available historical data and then evaluating them on datasets that are drawn from identical distributions. The goal of this evaluation process is to select the classifier that is believed to be most robust in maintaining good future performance, and then deploy that classifier to end-users who use it to make predictions on new data. Often times, predictive models are deployed in conditions that differ from those used in training, meaning that dataset shift occurred. In these situations, there are no guarantees that predictions made by the predictive model in deployment will still be as reliable and accurate as they were during the training of the model. This study demonstrated a technique that can be utilized by others when selecting a classifier for deployment, as well as the first comparative study that evaluates machine learning classifier performance on synthetic datasets with different levels of prior-probability, covariate, and concept dataset shifts. The results from this study showed the impact of dataset shift on the performance of different classifiers for two real-world datasets related to teacher retention in Wisconsin and detecting fraud in testing, as well as demonstrated a framework that can be used by others when selecting a classifier for deployment. By using the methods from this study as a proactive approach to evaluate classifiers on synthetic dataset shift, different classifiers would have been considered for deployment of both predictive models, compared to only using evaluation datasets that were drawn from identical distributions. The results from both real-world datasets also showed that there was no classifier that dealt well with prior-probability shift and that classifiers were affected less by covariate and concept shift than was expected. Two supplemental demonstrations of the methodology showed that it can be extended for additional purposes of evaluating classifiers on dataset shift. Results from analyzing the effects of hyperparameter choices on classifier performance under dataset shift, as well as the effects of actual dataset shift on classifier performance, showed that different hyperparameter configurations have an impact on the performance of a classifier in general, but can also have an impact on how robust that classifier might be to dataset shift

Classification rates: non‐parametric verses parametric models using binary data

Estimations of the conditional mean and the marginal effects for particular small changes in the covariates have been of interest in financial, economics and even educational sectors. The standard approach has been to specify a parametric model such as probit or logit and then estimating the coefficients by maximum likelihood method. This is only applicable when the distribution form from which the data has been drawn is known. Non parametric methods have been proposed when the functional form assumptions cannot be ascertained. This research sought to establish if non parametric modeling achieves a higher correct classification ratio than a parametric model. The local likelihood technique was used to model fit the data sets. The same sets of data were modeled using parametric logit and the abilities of the two models to correctly predict the binary outcome compared. The results obtained showed that non‐parametric estimation gives a better prediction rate (classification ratio) for a binary data than parametric estimation. This was achieved both empirically and through simulation. For empirical results two different data sets were used. The first set consisted of loan applications of customers and the second set consisted of approved loans. In both data sets the classification ratio for non‐parametric method was found to be 1 while that for parametric was found to be 0.87 (only 87 out of the 100 observations were correctly classified) and 0.83 respectively. Simulation was done based on sample sizes of 25, 50, 75, 100,150,200,250,300 and 500. The simulated results further showed that the accuracy of both models decrease as sample size increases.Key words: Parametric, non‐parametric, local likelihood, logit, confusion matrix and classification rati

Investigating prediction modelling of academic performance for students in rural schools in Kenya

Academic performance prediction modelling provides an opportunity for learners' probable outcomes to be known early, before they sit for final examinations. This would be particularly useful for education stakeholders to initiate intervention measures to help students who require high intervention to pass final examinations. However, limitations of infrastructure in rural areas of developing countries, such as lack of or unstable electricity and Internet, impede the use of PCs. This study proposed that an academic performance prediction model could include a mobile phone interface specifically designed based on users' needs. The proposed mobile academic performance prediction system (MAPPS) could tackle the problem of underperformance and spur development in the rural areas. A six-step Cross-Industry Standard Process for Data Mining (CRISP-DM) theoretical framework was used to support the design of MAPPS. Experiments were conducted using two datasets collected in Kenya. One dataset had 2426 records of student data having 22 features, collected from 54 rural primary schools. The second dataset had 1105 student records with 19 features, collected from 11 peri-urban primary schools. Evaluation was conducted to investigate: (i) which is the best classifier model among the six common classifiers selected for the type of data used in this study; (ii) what is the optimal subset of features from the total number of features for both rural and peri-urban datasets; and (iii) what is the predictive performance of the Mobile Academic Performance Prediction System in classifying the high intervention class. It was found that the system achieved an F-Measure rate of nearly 80% in determining the students who need high intervention two years before the final examination. It was also found that the system was useful and usable in rural environments; the accuracy of prediction was good enough to motivate stakeholders to initiate strategic intervention measures. This study provides experimental evidence that Educational Data Mining (EDM) techniques can be used in the developing world by exploiting the ubiquitous mobile technology for student academic performance prediction

Risk Analytics in Econometrics

[eng] This thesis addresses the framework of risk analytics as a compendium of four main pillars: (i) big data, (ii) intensive programming, (iii) advanced analytics and machine learning, and (iv) risk analysis. Under the latter mainstay, this PhD dissertation reviews potential hazards known as “extreme events” that could negatively impact the wellbeing of people, profitability of firms, or the economic stability of a country, but which also have been underestimated or incorrectly treated by traditional modelling techniques. The objective of this thesis is to develop econometric and machine learning algorithms that can improve the predictive capacity of those extreme events and improve the comprehension of the phenomena contrary to some modern advanced methods which are black boxes in terms of interpretation. This thesis presents seven chapters that provide a methodological contribution to the existing literature by building techniques that transform the new valuable insights of big data into more accurate predictions that support decisions under risk, and increase robustness for more reliable and real results. This PhD thesis focuses uniquely on extremal events which are trigged into a binary variable, mostly known as class-imbalanced data and rare events in binary response, in other words, whose classes that are not equally distributed. The scope of research tackle real cases studies in the field of risk and insurance, where it is highly important to specify a level of claims of an event in order to foresee its impact and to provide a personalized treatment. After Chapter 1 corresponding to the introduction, Chapter 2 proposes a weighting mechanism to incorporated in the weighted likelihood estimation of a generalized linear model to improve the predictive performance of the highest and lowest deciles of prediction. Chapter 3 proposes two different weighting procedures for a logistic regression model with complex survey data or specific sampling designed data. Its objective is to control the randomness of data and provide more sensitivity to the estimated model. Chapter 4 proposes a rigorous review of trials with modern and classical predictive methods to uncover and discuss the efficiency of certain methods over others, and which and how gaps in machine learning literature can be addressed efficiently. Chapter 5 proposes a novel boosting-based method that overcomes certain existing methods in terms of predictive accuracy and also, recovers some interpretation of the model with imbalanced data. Chapter 6 develops another boosting-based algorithm which is able to improve the predictive capacity of rare events and get approximated as a generalized linear model in terms of interpretation. And finally, Chapter 7 includes the conclusions and final remarks. The present thesis highlights the importance of developing alternative modelling algorithms that reduces uncertainty, especially when there are potential limitations that impede to know all the previous factors that influence on the presence of a rare event or imbalanced-data phenomenon. This thesis merges two important approaches in modelling predictive literature as they are: “econometrics” and “machine learning”. All in all, this thesis contributes to enhance the methodology of how empirical analysis in many experimental and non-experimental sciences have being doing so far