Joint Identification of Location and Dispersion Effects in Unreplicated Two-Level Factorials

Most procedures that have been proposed to identify dispersion effects in unreplicated factorial designs assume that location effects have been identified correctly. Incorrect identi- fication of location effects may impair subsequent identification of dispersion effects. We develop a model for joint identification of location and dispersion effects that can reliably identify active effects of both types. The joint model is estimated using maximum likelihood, and hence effect selection is done using a specially derived information criterion. An exhaustive search through a limited version of the space of possible models is conducted. Both a single-model output and model averaging are considered. The method is shown to be capable of identifying sensible location-dispersion models that are missed by methods that rely on sequential estimation of location and dispersion effects

Video Game Telemetry as a Critical Tool in the Study of Complex Skill Learning

Cognitive science has long shown interest in expertise, in part because prediction and control of expert development would have immense practical value. Most studies in this area investigate expertise by comparing experts with novices. The reliance on contrastive samples in studies of human expertise only yields deep insight into development where differences are important throughout skill acquisition. This reliance may be pernicious where the predictive importance of variables is not constant across levels of expertise. Before the development of sophisticated machine learning tools for data mining larger samples, and indeed, before such samples were available, it was difficult to test the implicit assumption of static variable importance in expertise development. To investigate if this reliance may have imposed critical restrictions on the understanding of complex skill development, we adopted an alternative method, the online acquisition of telemetry data from a common daily activity for many: video gaming. Using measures of cognitive-motor, attentional, and perceptual processing extracted from game data from 3360 Real-Time Strategy players at 7 different levels of expertise, we identified 12 variables relevant to expertise. We show that the static variable importance assumption is false - the predictive importance of these variables shifted as the levels of expertise increased - and, at least in our dataset, that a contrastive approach would have been misleading. The finding that variable importance is not static across levels of expertise suggests that large, diverse datasets of sustained cognitive-motor performance are crucial for an understanding of expertise in real-world contexts. We also identify plausible cognitive markers of expertise

Classic Motor Chunking Theory Fails To Account for Behavioural Diversity and Speed in a Complex Naturalistic Task

In tasks that demand rapid performance, actions must be executed as efficiently as possible. Theories of expert motor performance such as the motor chunking framework suggest that efficiency is supported by automatization, where many serial actions are automatized into smaller chunks, or groups of commonly co-occuring actions. We use the fast-paced, professional eSport StarCraft 2 as a test case of the explanatory power of the motor chunking framework and assess the importance of chunks in explaining expert performance. To do so, we test three predictions motivated by a simple motor chunking framework. (1) StarCraft 2 players should exhibit an increasing number of chunks with expertise. (2) The proportion of actions falling within a chunk should increase with skill. (3) Chunks should be faster than non-chunks containing the same atomic behaviours. Although our findings support the existence of chunks, they also highlight two problems for existing accounts of rapid motor execution and expert performance. First, while better players do use more chunks, the proportion of actions within a chunks is stable across expertise and expert sequences are generally more varied (the diversity problem). Secondly, chunks, which are supposed to enjoy the most extreme automatization, appear to save little or no time overall (the time savings problem). Instead, the most parsimonious description of our latency analysis is that players become faster overall regardless of chunking

Predicting which children with juvenile idiopathic arthritis will not attain early remission with conventional treatment: Results from the Reacch-out cohort

Objective. To estimate the probability of early remission with conventional treatment for each child with juvenile idiopathic arthritis (JIA). Children with a low chance of remission may be candidates for initial treatment with biologics or triple disease-modifying antirheumatic drugs (DMARD). Methods. We used data from 1074 subjects in the Research in Arthritis in Canadian Children emphasizing Outcomes (ReACCh-Out) cohort. The predicted outcome was clinically inactive disease for ≥ 6 months starting within 1 year of JIA diagnosis in patients who did not receive early biologic agents or triple DMARD. Models were developed in 200 random splits of 75% of the cohort and tested on the remaining 25% of subjects, calculating expected and observed frequencies of remission and c-index values. Results. Our best Cox logistic model combining 18 clinical variables a median of 2 days after diagnosis had a c-index of 0.69 (95% CI 0.67-0.71), better than using JIA category alone (0.59, 95% CI 0.56-0.63). Children in the lowest probability decile had a 20% chance of remission and 21% attained remission; children in the highest decile had a 69% chance of remission and 73% attained remission. Compared to 5% of subjects identified by JIA category alone, the model identified 14% of subjects as low chance of remission (probability \u3c 0.25), of whom 77% failed to attain remission. Conclusion. Although the model did not meet our a priori performance threshold (c-index \u3e 0.70), it identified 3 times more subjects with low chance of remission than did JIA category alone, and it may serve as a benchmark for assessing value added by future laboratory/imaging biomarkers

Adjustment uncertainty and variable selection in a Bayesian context

Bayesian Model Averaging (BMA) has previously been proposed as a solution to the variable selection problem when there is uncertainty about the true model in regression. Some recent research discusses the drawbacks; specifically, BMA can (and does) give biased parameter estimates in the presence of confounding. This is because BMA is optimized for prediction rather than parameter estimation. Though some newer research attempts to fix the issue of bias under confounding, none of the current algorithms handle either large data sets or survival outcomes. The Approximate Two-phase Bayesian Adjustment for Confounding (ATBAC) algorithm proposed in this paper does both, and we use it on a large medical cohort study called THIN (The Health Improvement Network) to estimate the effect of statins on risk of stroke. We use simulation and some analytical techniques to discuss two main topics in this paper. Firstly, we demonstrate the ability of ATBAC to perform unbiased parameter estimation on survival data while accounting for model uncertainty. Secondly, we discuss when it is, and isn\u27t, helpful to use variable selection techniques in the first place, and find that in some large data sets variable selection for parameter estimation is unnecessary

Statistical Learning Tools for Heteroskedastic Data

Many regression procedures are affected by heteroskedasticity, or non-constant variance. A classic solution is to transform the response y and model h(y) instead. Common functions h require a direct relationship between the variance and the mean. Unless the transformation is known in advance, it can be found by applying a model for the variance to the squared residuals from a regression fit. Unfortunately, this approach additionally requires the strong assumption that the regression model for the mean is \u27correct\u27, whereas many regression problems involve model uncertainty. Consequently it is undesirable to make the assumption that the mean model can be correctly specified at the outset. An alternative is to model the mean and variance simultaneously, where it is possible to try different mean models and variance models together in different combinations, and to assess the fit of each combination using a single criterion. We demonstrate this approach in three different problems: unreplicated factorials, regression trees, and random forests. For the unreplicated factorial problem, we develop a model for joint identification of mean and variance effects that can reliably identify active effects of both types. The joint model is estimated using maximum likelihood, and effect selection is done using a specially derived information criterion (IC). Our method is capable of identifying sensible location-dispersion models that are not considered by methods that rely on sequential estimation of location and dispersion effects. We take a similar approach to modeling variances in regression trees. We develop an alternative likelihood-based split-selection criterion that has the capacity to account for local variance in the regression in an unstructured manner, and the tree is built using a specially derived IC. Our IC explicitly accounts for the split-selection parameter and our IC also leads to a faster pruning algorithm that does not require crossvalidation. We show that the new approach performs better for mean estimation under heteroskedasticity. Finally we use these likelihood-based trees as base learners in an ensemble much like a random forest, and improve the random forest procedure itself. First, we show that typical random forests are inefficient at fitting flat mean functions. Our first improvement is the novel alpha-pruning algorithm, which adaptively changes the number of observations in the terminal nodes of the regression trees depending on the flatness. Second, we show that random forests are inefficient at estimating means when the data are heteroskedastic, which we address by using our likelihood-based regression trees as a base learner. This allows explicit variance estimation and improved mean estimation under heteroskedasticity. Our unifying and novel contribution to these three problems is the specially derived IC. Our solution is to simulate values of the IC for several models and to store these values in a lookup table. With the lookup table, models can be evaluated and compared without needing either crossvalidation or a holdout set. We call this approach the Corrected Heteroskedastic Information Criterion (CHIC) paradigm and we demonstrate that applying the CHIC paradigm is a principled way to model variance in finite sample sizes

Over the Hill at 24: Persistent Age-Related Cognitive-Motor Decline in Reaction Times in an Ecologically Valid Video Game Task Begins in Early Adulthood

Typically studies of the effects of aging on cognitive-motor performance emphasize changes in elderly populations. Although some research is directly concerned with when age-related decline actually begins, studies are often based on relatively simple reaction time tasks, making it impossible to gauge the impact of experience in compensating for this decline in a real world task. The present study investigates age-related changes in cognitive motor performance through adolescence and adulthood in a complex real world task, the real-time strategy video game StarCraft 2. In this paper we analyze the influence of age on performance using a dataset of 3,305 players, aged 16-44, collected by Thompson, Blair, Chen & Henrey [1]. Using a piecewise regression analysis, we find that age-related slowing of within-game, self-initiated response times begins at 24 years of age. We find no evidence for the common belief expertise should attenuate domain-specific cognitive decline. Domain-specific response time declines appear to persist regardless of skill level. A second analysis of dual-task performance finds no evidence of a corresponding age-related decline. Finally, an exploratory analyses of other age-related differences suggests that older participants may have been compensating for a loss in response speed through the use of game mechanics that reduce cognitive load

Validation of prediction models of severe disease course and non-achievement of remission in juvenile idiopathic arthritis part 2: results of the Nordic model in the Canadian cohort

Background: Validated clinical prediction models to identify children with poor prognosis at the time of juvenile idiopathic arthritis (JIA) diagnosis would be very helpful for tailoring treatments, and avoiding under- or over-treatment. Our objective was to externally validate Nordic clinical prediction models in Canadian patients with JIA. Methods: We used data from 513 subjects at the 3-year follow-up from the Research in Arthritis in Canadian Children emphasizing Outcomes (ReACCh-Out) cohort. The predicted outcomes were non-achievement of remission, severe disease course, and functional disability. The Nordic models were evaluated exactly as published and after fine-tuning the logistic regression coefficients using multiple data splits of the Canadian cohort. Missing data was handled with multiple imputation, and prediction ability was assessed with C-indices. C-index values > 0.7 were deemed to reflect helpful prediction. Results: Overall, 81% of evaluable patients did not achieve remission off medications, 15% experienced a severe disease course, and 38% reported disability (CHAQ score > 0). The Nordic model for predicting non-achievement of remission had a C-index of 0.68 (95% CI 0.62–0.74), and 0.74 (0.67–0.80) after fine-tuning. For prediction of severe disease course, it had a C-index of 0.69 (0.61–0.78), and 0.79 (0.68–0.91) after fine-tuning. The fine-tuned Nordic model identified 85% of the cohort as low risk for a severe disease course ( 60% chance). The Nordic model to predict functional disability had a C-index of 0.57 (0.50–0.63), and 0.51 (0.39–0.63) after fine-tuning. Conclusions: Fine-tuned Nordic models, combining active joint count, physician global assessment of disease activity, morning stiffness, and ankle involvement, predicted well non-achievement of remission and severe disease course in Canadian patients with JIA. The Nordic model for predicting disability could not predict functional disability in Canadian patients.Medicine, Faculty ofOther UBCNon UBCPediatrics, Department ofReviewedFacult

A screenshot from the game StarCraft 2.

<p>A screenshot from the game StarCraft 2.</p