Nonparametric predictive inference for diagnostic test thresholds

Measuring the accuracy of diagnostic tests is crucial in many application areas including medicine, machine learning and credit scoring. The receiver operating characteristic (ROC) curve and surface are useful tools to assess the ability of diagnostic tests to discriminate between ordered classes or groups. To define these diagnostic tests, selecting the optimal thresholds that maximize the accuracy of these tests is required. One procedure that is commonly used to find the optimal thresholds is by maximizing what is known as Youden’s index. This article presents nonparametric predictive inference (NPI) for selecting the optimal thresholds of a diagnostic test. NPI is a frequentist statistical method that is explicitly aimed at using few modeling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. Based on multiple future observations, the NPI approach is presented for selecting the optimal thresholds for two-group and three-group scenarios. In addition, a pairwise approach has also been presented for the three-group scenario. The article ends with an example to illustrate the proposed methods and a simulation study of the predictive performance of the proposed methods along with some classical methods such as Youden index. The NPI-based methods show some interesting results that overcome some of the issues concerning the predictive performance of Youden’s index

Nonparametric predictive inference for comparison of two diagnostic tests

An important aim in diagnostic medical research is comparison of the accuracy of two diagnostic tests. In this paper, comparison of two diagnostic tests is presented using nonparametric predictive inference (NPI) for future order statistics. The tests are assumed to be applied on the same individuals from two groups, e.g., healthy and diseased individuals, or from three groups with a known ordering, e.g., adding a group of severely diseased individuals to the two group scenario. Our comparison is explicitly in terms of lower and upper probabilities for proportions of correctly diagnosed future individuals from each group, for a given total number of such individuals. We include in our comparison the possibility that it is more important to get a correct diagnosis for individuals from one group than from another group

Dynamical Probability Distribution Function of the SK Model at High Temperatures

The microscopic probability distribution function of the Sherrington-Kirkpatrick (SK) model of spin glasses is calculated explicitly as a function of time by a high-temperature expansion. The resulting formula to the third order of the inverse temperature shows that an assumption made by Coolen, Laughton and Sherrington in their recent theory of dynamics is violated. Deviations of their theory from exact results are estimated quantitatively. Our formula also yields explicit expressions of the time dependence of various macroscopic physical quantities when the temperature is suddenly changed within the high-temperature region.Comment: LaTeX, 6 pages, Figures upon request (here revised), To be published in J. Phys. Soc. Jpn. 65 (1996) No.

Dynamics of on-line Hebbian learning with structurally unrealizable restricted training sets

We present an exact solution for the dynamics of on-line Hebbian learning in neural networks, with restricted and unrealizable training sets. In contrast to other studies on learning with restricted training sets, unrealizability is here caused by structural mismatch, rather than data noise: the teacher machine is a perceptron with a reversed wedge-type transfer function, while the student machine is a perceptron with a sigmoidal transfer function. We calculate the glassy dynamics of the macroscopic performance measures, training error and generalization error, and the (non-Gaussian) student field distribution. Our results, which find excellent confirmation in numerical simulations, provide a new benchmark test for general formalisms with which to study unrealizable learning processes with restricted training sets.Comment: 7 pages including 3 figures, using IOP latex2e preprint class fil

Generating functional analysis of Minority Games with real market histories

It is shown how the generating functional method of De Dominicis can be used to solve the dynamics of the original version of the minority game (MG), in which agents observe real as opposed to fake market histories. Here one again finds exact closed equations for correlation and response functions, but now these are defined in terms of two connected effective non-Markovian stochastic processes: a single effective agent equation similar to that of the `fake' history models, and a second effective equation for the overall market bid itself (the latter is absent in `fake' history models). The result is an exact theory, from which one can calculate from first principles both the persistent observables in the MG and the distribution of history frequencies.Comment: 39 pages, 5 postscript figures, iop styl

Diagonalization of replicated transfer matrices for disordered Ising spin systems

We present an alternative procedure for solving the eigenvalue problem of replicated transfer matrices describing disordered spin systems with (random) 1D nearest neighbor bonds and/or random fields, possibly in combination with (random) long range bonds. Our method is based on transforming the original eigenvalue problem for a $2^n\times 2^n$ matrix (where $n\to 0$) into an eigenvalue problem for integral operators. We first develop our formalism for the Ising chain with random bonds and fields, where we recover known results. We then apply our methods to models of spins which interact simultaneously via a one-dimensional ring and via more complex long-range connectivity structures, e.g. $1+\infty$ dimensional neural networks and `small world' magnets. Numerical simulations confirm our predictions satisfactorily.Comment: 24 pages, LaTex, IOP macro

Statistical mechanics and stability of a model eco-system

We study a model ecosystem by means of dynamical techniques from disordered systems theory. The model describes a set of species subject to competitive interactions through a background of resources, which they feed upon. Additionally direct competitive or co-operative interaction between species may occur through a random coupling matrix. We compute the order parameters of the system in a fixed point regime, and identify the onset of instability and compute the phase diagram. We focus on the effects of variability of resources, direct interaction between species, co-operation pressure and dilution on the stability and the diversity of the ecosystem. It is shown that resources can be exploited optimally only in absence of co-operation pressure or direct interaction between species.Comment: 23 pages, 13 figures; text of paper modified, discussion extended, references adde

Statistical Mechanics of Dilute Batch Minority Games with Random External Information

We study the dynamics and statics of a dilute batch minority game with random external information. We focus on the case in which the number of connections per agent is infinite in the thermodynamic limit. The dynamical scenario of ergodicity breaking in this model is different from the phase transition in the standard minority game and is characterised by the onset of long-term memory at finite integrated response. We demonstrate that finite memory appears at the AT-line obtained from the corresponding replica calculation, and compare the behaviour of the dilute model with the minority game with market impact correction, which is known to exhibit similar features.Comment: 22 pages, 6 figures, text modified, references updated and added, figure added, typos correcte

Bayes Linear Analysis of Imprecision in Computer Models, with Application to Understanding Galaxy Formation

Imprecision arises naturally in the context of computer models and their relation to reality. An imprecise treatment of general computer models is presented, illustrated with an analysis of a complex galaxy formation simulation known as Galform. The analysis involves several different types of uncertainty, one of which (the Model Discrepancy) comes directly from expert elicitation regarding the deficiencies of the model. The Model Discrepancy is therefore treated within an Imprecise framework to reflect more accurately the beliefs of the expert concerning the discrepancy between the model and reality. Due to the conceptual complexity and computationally intensive nature of such a Bayesian imprecise uncertainty analysis, Bayes Linear Methodology is employed which requires consideration of only expectations and variances of all uncertain quantities. Therefore incorporating an Imprecise treatment within a Bayes Linear analysis is shown to be relatively straightforward. The impact of an imprecise assessment on the input space of the model is determined through the use of an Implausibility measure