Facilitating Variable-Length Computerized Classification Testing Via Automatic Racing Calibration Heuristics

Thesis (Ph.D.) - Indiana University, School of Education, 2015Computer Adaptive Tests (CATs) have been used successfully with standardized tests. However, CATs are rarely practical for assessment in instructional contexts, because large numbers of examinees are required a priori to calibrate items using item response theory (IRT). Computerized Classification Tests (CCTs) provide a practical alternative to IRT-based CATs. CCTs show promise for instructional contexts, since many fewer examinees are required for item parameter estimation. However, there is a paucity of clear guidelines indicating when items are sufficiently calibrated in CCTs. Is there an efficient and accurate CCT algorithm which can estimate item parameters adaptively? Automatic Racing Calibration Heuristics (ARCH) was invented as a new CCT method and was empirically evaluated in two studies. Monte Carlo simulations were run on previous administrations of a computer literacy test, consisting of 85 items answered by 104 examinees. Simulations resulted in determination of thresholds needed by the ARCH method for parameter estimates. These thresholds were subsequently used in 50 sets of computer simulations in order to compare accuracy and efficiency of ARCH with the sequential probability ratio test (SPRT) and with an enhanced method called EXSPRT. In the second study, 5,729 examinees took an online plagiarism test, where ARCH was implemented in parallel with SPRT and EXSPRT for comparison. Results indicated that new statistics were needed by ARCH to establish thresholds and to determine when ARCH could begin. The ARCH method resulted in test lengths significantly shorter than SPRT, and slightly longer than EXSPRT without sacrificing accuracy of classification of examinees as masters and nonmasters. This research was the first of its kind in evaluating the ARCH method. ARCH appears to be a viable CCT method, which could be particularly useful in massively open online courses (MOOCs). Additional studies with different test content and contexts are needed

An Integrated Fuzzy Inference Based Monitoring, Diagnostic, and Prognostic System

To date the majority of the research related to the development and application of monitoring, diagnostic, and prognostic systems has been exclusive in the sense that only one of the three areas is the focus of the work. While previous research progresses each of the respective fields, the end result is a variable grab bag of techniques that address each problem independently. Also, the new field of prognostics is lacking in the sense that few methods have been proposed that produce estimates of the remaining useful life (RUL) of a device or can be realistically applied to real-world systems. This work addresses both problems by developing the nonparametric fuzzy inference system (NFIS) which is adapted for monitoring, diagnosis, and prognosis and then proposing the path classification and estimation (PACE) model that can be used to predict the RUL of a device that does or does not have a well defined failure threshold. To test and evaluate the proposed methods, they were applied to detect, diagnose, and prognose faults and failures in the hydraulic steering system of a deep oil exploration drill. The monitoring system implementing an NFIS predictor and sequential probability ratio test (SPRT) detector produced comparable detection rates to a monitoring system implementing an autoassociative kernel regression (AAKR) predictor and SPRT detector, specifically 80% vs. 85% for the NFIS and AAKR monitor respectively. It was also found that the NFIS monitor produced fewer false alarms. Next, the monitoring system outputs were used to generate symptom patterns for k-nearest neighbor (kNN) and NFIS classifiers that were trained to diagnose different fault classes. The NFIS diagnoser was shown to significantly outperform the kNN diagnoser, with overall accuracies of 96% vs. 89% respectively. Finally, the PACE implementing the NFIS was used to predict the RUL for different failure modes. The errors of the RUL estimates produced by the PACE-NFIS prognosers ranged from 1.2-11.4 hours with 95% confidence intervals (CI) from 0.67-32.02 hours, which are significantly better than the population based prognoser estimates with errors of ~45 hours and 95% CIs of ~162 hours

Product Failure Recognition Via Comparison Of Sequential and Quickest Detection Algorithms

Under similar conditions, products that are designed and used for similar tasks fail similarly. Developers may become aware of various product failure modes during the initial stages of new product generation, where redesign and failure mitigation processes can occur with minimal detriment to consumer safety. Developers strive to mitigate the potential for catastrophic failures. This thesis concentrates on when these failures occur outside of controlled conditions, specifically where the development of processes feature low accuracy sensing techniques that impact the safety and operation of the end user. This thesis develops a set of statistical analysis simulation techniques using two existing methods: Sequential Analysis and Quickest Detection. Through the comparison of method-specific features, this thesis aims to assist future researchers unfamiliar with these methods to understand the individual characteristics of each as they pertain to failure mitigation. Each detection method is subjected to investigation via a pair of sensor models, a strong sensor and a weak sensor. Variable detection settings are used to quantify the operational characteristics of these sensors and their individual means of analysis. This thesis then compares both statistical techniques to recognize their overall usefulness to the topic of product failure analysis and mitigation pertaining to lower accuracy sensing processes that require longer sampling periods for better informed decisions. It is ascertained that the Sequential Analysis technique is best used when the initial system state is not yet known to the observer. The Quickest Detection method should be utilized when the initial state of a system is known and it is imperative to detect, with minimal delay, the occurrence of a random change-point in the operational status of the system

An Adaptive Nonparametric Modeling Technique for Expanded Condition Monitoring of Processes

New reactor designs and the license extensions of the current reactors has created new condition monitoring challenges. A major challenge is the creation of a data-based model for a reactor that has never been built or operated and has no historical data. This is the motivation behind the creation of a hybrid modeling technique based on first principle models that adapts to include operating reactor data as it becomes available. An Adaptive Non-Parametric Model (ANPM) was developed for adaptive monitoring of small to medium size reactors (SMR) but would be applicable to all designs. Ideally, an adaptive model should have the ability to adapt to new operational conditions while maintaining the ability to differentiate faults from nominal conditions. This has been achieved by focusing on two main abilities. The first ability is to adjust the model to adapt from simulated conditions to actual operating conditions, and the second ability is to adapt to expanded operating conditions. In each case the system will not learn new conditions which represent faulted or degraded operations. The ANPM architecture is used to adapt the model\u27s memory matrix from data from a First Principle Model (FPM) to data from actual system operation. This produces a more accurate model with the capability to adjust to system fluctuations. This newly developed adaptive modeling technique was tested with two pilot applications. The first application was a heat exchanger model that was simulated in both a low and high fidelity method in SIMULINK. The ANPM was applied to the heat exchanger and improved the monitoring performance over a first principle model by increasing the model accuracy from an average MSE of 0.1451 to 0.0028 over the range of operation. The second pilot application was a flow loop built at the University of Tennessee and simulated in SIMULINK. An improvement in monitoring system performance was observed with the accuracy of the model improving from an average MSE of 0.302 to an MSE of 0.013 over the adaptation range of operation. This research focused on the theory, development, and testing of the ANPM and the corresponding elements in the surveillance system

Research on new techniques for the analysis of manual control systems Progress report, 15 Jun. 1969 - 15 Jun. 1970

Applying statistical decision theory to manual adaptive control system

Towards Robust Artificial Intelligence Systems

Adoption of deep neural networks (DNNs) into safety-critical and high-assurance systems has been hindered by the inability of DNNs to handle adversarial and out-of-distribution input. State-of-the-art DNNs misclassify adversarial input and give high confidence output for out-of-distribution input. We attempt to solve this problem by employing two approaches, first, by detecting adversarial input and, second, by developing a confidence metric that can indicate when a DNN system has reached its limits and is not performing to the desired specifications. The effectiveness of our method at detecting adversarial input is demonstrated against the popular DeepFool adversarial image generation method. On a benchmark of 50,000 randomly chosen ImageNet adversarial images generated for CaffeNet and GoogLeNet DNNs, our method can recover the correct label with 95.76% and 97.43% accuracy, respectively. The proposed attribution-based confidence (ABC) metric utilizes attributions used to explain DNN output to characterize whether an output corresponding to an input to the DNN can be trusted. The attribution based approach removes the need to store training or test data or to train an ensemble of models to obtain confidence scores. Hence, the ABC metric can be used when only the trained DNN is available during inference. We test the effectiveness of the ABC metric against both adversarial and out-of-distribution input. We experimental demonstrate that the ABC metric is high for ImageNet input and low for adversarial input generated by FGSM, PGD, DeepFool, CW, and adversarial patch methods. For a DNN trained on MNIST images, ABC metric is high for in-distribution MNIST input and low for out-of-distribution Fashion-MNIST and notMNIST input

Curtailment and Stochastic Curtailment to Shorten the CES-D

The Center for Epidemiologic Studies-Depression (CES-D) scale is a well-known self-report instrument that is used to measure depressive symptomatology. Respondents who take the full-length version of the CES-D are administered a total of 20 items. This article investigates the use of curtailment and stochastic curtailment (SC), two sequential analysis methods that have recently been proposed for health questionnaires, to reduce the respondent burden associated with taking the CES-D. A post hoc simulation based on 1,392 adolescents' responses to the CES-D was used to compare these methods with a previously proposed computerized adaptive testing (CAT) approach. Curtailment lowered average test lengths by as much as 22% while always matching the classification decision of the full-length CES-D. SC and CAT achieved further reductions in average test length, with SC's classifications exhibiting more concordance with the full-length CES-D than do CAT's. Advantages and disadvantages of each method are discussed. © The Author(s) 2012

Novel channel sensing and access strategies in opportunistic spectrum access networks

textTraditionally radio spectrum was considered a commodity to be allocated in a fixed and centralized manner, but now the technical community and the regulators approach it as a shared resource that can be flexibly and intelligently shared between competing entities. In this thesis we focus on novel strategies to sense and access the radio spectrum within the framework of Opportunistic Spectrum Access via Cognitive Radio Networks (CRNs). In the first part we develop novel transmit opportunity detection methods that effectively exploit the gray space present in packet based networks. Our methods proactively detect the maximum safe transmit power that does not significantly affect the primary network nodes via an implicit feedback mechanism from the Primary network to the Secondary network. A novel use of packet interarrival duration is developed to robustly perform change detection in the primary network's Quality of Service. The methods are validated on real world IEEE 802.11 WLANs. In the second part we study the inferential use of Goodness-of-Fit tests for spectrum sensing applications. We provide the first comprehensive framework for decision fusion of an ensemble of goodness-of-fit tests through use of p-values. Also, we introduce a generalized Phi-divergence statistic to formulate goodness-of-fit tests that are tunable via a single parameter. We show that under uncertainty in the noise statistics or non-Gaussianity in the noise, the performance of such non-parametric tests is significantly superior to that of conventional spectrum sensing methods. Additionally, we describe a collaborative spatially separated version of the test for robust combining of tests in a distributed spectrum sensing setting. In the third part we develop the sequential energy detection problem for spectrum sensing and formulate a novel Sequential Energy Detector. Through extensive simulations we demonstrate that our doubly hierarchical sequential testing architecture delivers a significant throughput improvement of 2 to 6 times over the fixed sample size test while maintaining equivalent operating characteristics as measured by the Probabilities of Detection and False Alarm. We also demonstrate the throughput gains for a case study of sensing ATSC television signals in IEEE 802.22 systems.Electrical and Computer Engineerin