Accuracy assessment in multivariate Bayesian forecasting linear and nonlinear models

Abstract

Reduced alertness and high levels of cognitive fatigue due to sleep loss bring forth substantial risks in today\u27s 24/7 society. Biomathematical models can be used to help mitigate such risks by predicting quantitative levels of fatigue under sleep loss. These models help manage risk by providing information on the timing at which high levels of fatigue will occur; countermeasures can then be taken to reduce accident risk at such critical times. Many quantitative models exist to predict cognitive performance based on homeostatic and circadian processes (Mallis et al., 2004). These models have typically been fitted to group average data. Due to large individual variation, group-average predictions are often inaccurate for a given individual. However, since individual differences are trait-like, between subjects variation can be captured by individualizing model parameters using the technique of Bayesian forecasting. In many cases the amount of data collected, and consequently, the prediction accuracy, will be limited by factors such as cost and availability. However; prediction accuracy may still be improved by including information from alternative, correlated performance measures in a multivariate Bayesian forecasting framework. When collecting data from two performance measures, we consider methods of sampling that obtain a desired average level of prediction accuracy for minimal data collection cost. We assess the prediction accuracy using the Bayesian mean squared error (MSE) and derive this measure for a general Bayesian linear model. To understand how the accuracy depends on the number of measurements from primary and secondary tasks in the simplest case, we apply the equation to specify the accuracy for the bivariate Bayesian linear model of subject means. For this simple model, we further assume that observations from each performance measure have a fixed cost per data point, and use this assumption to determine the number of measurements of each variable needed to minimize the cost while still obtaining no less than the desired level of accuracy. To aid the extension of the findings from the linear case to state of the art nonlinear biomathematical fatigue models, we focus on obtaining our extended measure of accuracy for the nonlinear case. Computing this accuracy analytically is often infeasible without reliance on model approximations. Model simulations can be used to compute this accuracy; however, such simulations can be time consuming, especially for models that lack analytic solutions and require that a system of differential equations be solved to produce model dynamics. Much of this computational burden in assessing estimator accuracy, however, is produced by using the Bayesian MMSE estimator, and could be reduced by taking advantage of the quicker to compute Bayesian MAP estimator. We show how for a nonlinear biomathematical model that the accuracy assessment using repeated simulation with the MAP estimator yields a reasonable estimate of the accuracy obtained using the MMSE estimator. Still, however, for any given case, determination of whether the MMSE accuracy can be approximated with the MAP accuracy requires these time consuming simulations. We begin to analytically identify classes of models where the MMSE accuracy can be approximated by the MAP accuracy. We consider a class of quadratic Bayesian models, and show by analytic approximation that for this class, the MMSE has twice the accuracy of the MAP