Making Test Batteries Adaptive By Using Multistage Testing Techniques

The objective of this dissertation research is to investigate the possibility to improve both reliability and validity for test batteries under the framework of multi-stage testing (MST). Two test battery designs that incorporate MST components were proposed and evaluated, one is a multistage test battery (MSTB) design and the other is a hybrid multistage test battery (MSTBH) design. The MSTB design consists of three tests: The first test used the AMI (approximate maximum information) method as the routing strategy; and as for the second and third, the “On-the-Fly” strategy (OMST) was employed. The MSTBH design also consists of three tests; the first two are administered via MST while the third one via CAT. This dissertation presents a new test battery design by combining the strengths from different testing models. To improve estimation precision, each subsequent test in the test battery for an examinee was assembled according to the examinee’s previous ability estimate. A set of simulation studies were conducted to compare MSTB, MSTBH with two baseline models for both measurement accuracy and test security control under various conditions. One of the baseline models is a MST design consisting of three MST procedures without borrowing information from each other’s; the other is a computerized adaptive test battery (CATB) design consisting of 1 to 3 CAT procedures, being the second and the third procedures borrowing information from the previous ones. The results demonstrated that the test battery designs yielded better measurement accuracy when considering previous subtest score as a predictor for the current subtest. All designs yielded acceptable mean exposure rates, but only the CATB design had ideal pool utilization. Finally, the discussion section presents some limitations on current studie

Computationally Efficient and Robust BIC-Based Speaker Segmentation

An algorithm for automatic speaker segmentation based on the Bayesian information criterion (BIC) is presented. BIC tests are not performed for every window shift, as previously, but when a speaker change is most probable to occur. This is done by estimating the next probable change point thanks to a model of utterance durations. It is found that the inverse Gaussian fits best the distribution of utterance durations. As a result, less BIC tests are needed, making the proposed system less computationally demanding in time and memory, and considerably more efficient with respect to missed speaker change points. A feature selection algorithm based on branch and bound search strategy is applied in order to identify the most efficient features for speaker segmentation. Furthermore, a new theoretical formulation of BIC is derived by applying centering and simultaneous diagonalization. This formulation is considerably more computationally efficient than the standard BIC, when the covariance matrices are estimated by other estimators than the usual maximum-likelihood ones. Two commonly used pairs of figures of merit are employed and their relationship is established. Computational efficiency is achieved through the speaker utterance modeling, whereas robustness is achieved by feature selection and application of BIC tests at appropriately selected time instants. Experimental results indicate that the proposed modifications yield a superior performance compared to existing approaches

The effects of routing and scoring within a computer adaptive multi-stage framework

This dissertation examined the overall effects of routing and scoring within a computer adaptive multi-stage framework (ca-MST). Testing in a ca-MST environment has become extremely popular in the testing industry. Testing companies enjoy its efficiency benefits as compared to traditionally linear testing and its quality-control features over computer adaptive testing (CAT). Test takers enjoy being able to go back and change responses in review time before being assigned to the next module. Lord (1980) outlined a few salient characteristics that should be investigated before the implementation of multi-stage testing. Of these characteristics, decisions on routing mechanisms have received the least attention. This dissertation varied both item pool characteristics such as the location of information, and ca-MST configuration characteristics such as the ca-MST configuration design (e.g., 1-3, 1-2-3, 1-2-3-4). The results from this study hope to show that number correct scoring can serve as a capable surrogate for IRT calibrations at each step and that even if three-parameter scoring models are used at the end that the number correct method will not misroute as compared to traditional methods

Information Extraction from Messy Data, Noisy Spectra, Incomplete Data, and Unlabeled Images

Data collected from real-world scenarios are never ideal but often messy because data errors are inevitable and may occur in creative and unexpected ways. And there are always some unexpected tricky troubles between ideal theory and real-world applications. Although with the development of data science, more and more elegant algorithms have been well developed and validated by rigorous proof, data scientists still have to spend 50\% to 80\% of their work time on cleaning and organizing data, leaving little time for actual data analysis. This dissertation research involves three scenarios of statistical modeling with common data issues: quantifying function effect on noisy functional data, multistage decision-making model over incomplete data, and unsupervised image segmentation over imperfect engineering images. And three methodologies are proposed accordingly to solve them efficiently. In Chapter 2, a general two-step procedure is proposed to quantify the effects of a certain treatment on the spectral signals subjecting to multiple uncertainties for an engineering application that involves materials treatment for aircraft maintenance. With this procedure, two types of uncertainties in the spectral signals, offset shift and multiplicative error, are carefully addressed. In the two-step procedure, a novel optimization problem is formulated to estimate the representative template spectrum first, and then another optimization problem is formulated to obtain the pattern of modification $\mathbf{g}$ that reveals how the treatment affects the shape of the spectral signal, as well as a vector $\boldsymbol{\delta}$ that describes the degree of change caused by different treatment magnitudes. The effectiveness of the proposed method is validated in a simulation study. \textcolor{black}{Furtherly, in} a real case study, the proposed method \textcolor{black}{is used} to investigate the effect of plasma exposure on the FTIR spectra. As a result, the proposed method effectively identifies the pattern of modification under uncertainties in the manufacturing environment, which matches the knowledge of the affected chemical components by the plasma treatment. And the recovered magnitude of modification provides guidance in selecting the control parameter of the plasma treatment. In Chapter 3, an active learning-based multistage sequential decision-making model is proposed to assist doctors and patients to make cost-effective treatment recommendations when some clinical data are more expensive or time-consuming to collect than other laboratory data. The main idea is to formulate the incomplete clinical data into a multistage decision-making model where the doctors can make diagnostics decisions sequentially in these stages, and actively collect only the necessary examination data from certain patients rather than all. There are two novelties in estimating parameters in the proposed model. First, unlike the existed ordinal logistic regression model which only models a single stage, a multistage model is built by maximizing the joint likelihood function for all samples in all stages. Second, considering that the data in different stages are nested in a cumulative way, it is assumed that the coefficients for common features in different stages are invariant. Compared with the baseline approach that models each stage individually and independently, the proposed multistage model with common coefficients assumption has significant advantages. It reduces the number of variables to estimate significantly, improves the computational efficiency, and makes the doctors feel intuitive by assuming that newly added features will not affect the weights of existed ones. In a simulation study, the relative efficiency of the proposed method with regards to the baseline approach is 162\% to 1,938\%, proving its efficiency and effectiveness soundly. Then, in a real case study, the proposed method estimates all parameters very efficiently and reasonably. %It estimates all parameters simultaneously to reach the global optimum and fully considers the cumulative characteristics between these stages by making common coefficients assumption. In Chapter 4, a simple yet very effective unsupervised image segmentation method, called RG-filter, is proposed to segment engineering images with no significant contrast between foreground and background for a material testing application. With the challenge of limited data size, imperfect data quality, unreachable binary true label, we developed the RG-filter which thresholding the pixels according to the relative magnitude of the R channel and G channel of the RGB image. %And the other one is called the superpixels clustering algorithm, where we add another layer of clustering over the segmented superpixels to binarize their labels. To test the performance of the existed image segmentation and proposed algorithm on our CFRP image data, we conducted a series of experiments over an example specimen. Comparing all the pixel labeling results, the proposed RG-filter outperforms the others to be the most recommended one. in addition, it is super intuitive and efficient in computation. The proposed RG-filter can help to analyze the failure mode distribution and proportion on the surface of composite material after destructive DCB testing. The result can help engineers better understand the weak link during the bonding of composite materials, which may provide guidance on how to improve the joining of structures during aircraft maintenance. Also, it can be crucial data when modeling together with some downstream data as a whole. And if we can predict it from other variables, the destructive DCB testing can be avoided, a lot of time and money can be saved. In Chapter 5, we concluded the dissertation and summarized the original contributions. In addition, future research topics associated with the dissertation have also been discussed. In summary, the dissertation contributes to the area of \textit{System Informatics and Control} (SIAC) to develop systematic methodologies based on messy real-world data in the field of composite materials and healthcare. The fundamental methodologies developed in this thesis have the potential to be applied to other advanced manufacturing systems.Ph.D

Scene segmentation using similarity, motion and depth based cues

Segmentation of complex scenes to aid surveillance is still considered an open research problem. In this thesis a computational model (CM) has been developed to classify a scene into foreground, moving-shadow and background regions. It has been demonstrated how the CM, with the optional use of a channel ratio test, can be applied to demarcate foreground shadow regions in indoor scenes illuminated by a fixed incandescent source of light. A combined approach, involving the CM working in tandem with a traditional motion cue based segmentation method, has also been constructed. In the combined approach, the CM is applied to segregate the foreground shaded regions in a current frame based on a binary mask generated using a standard background subtraction process (BSP). Various popular outlier detection strategies have been investigated to assess their suitabilities in generating a threshold automatically, required to develop a binary mask from a difference frame, the outcome of the BSP. To evaluate the full scope of the pixel labeling capabilities of the CM and to estimate the associated time constraints, the model is deployed for foreground scene segmentation in recorded real-life video streams. The observations made validate the satisfactory performance of the model in most cases. In the second part of the thesis depth based cues have been exploited to perform the task of foreground scene segmentation. An active structured light based depthestimating arrangement has been modeled in the thesis; the choice of modeling an active system over a passive stereovision one has been made to alleviate some of the difficulties associated with the classical correspondence problem. The model developed not only facilitates use of the set-up but also makes possible a method to increase the working volume of the system without explicitly encoding the projected structured pattern. Finally, it is explained how scene segmentation can be accomplished based solely on the structured pattern disparity information, without generating explicit depthmaps. To de-noise the difference frames, generated using the developed method, two median filtering schemes have been implemented. The working of one of the schemes is advocated for practical use and is described in terms of discrete morphological operators, thus facilitating hardware realisation of the method to speed-up the de-noising process

The Markov-Switching Multifractal Model of asset returns: GMM estimation and linear forecasting of volatility

Multifractal processes have recently been proposed as a new formalism for modelling the time series of returns in insurance. The major attraction of these processes is their ability to generate various degrees of long memory in different powers of returns - a feature that has been found in virtually all financial data. Initial difficulties stemming from non-stationarity and the combinatorial nature of the original model have been overcome by the introduction of an iterative Markov-switching multifractal model in Calvet and Fisher (2001) which allows for estimation of its parameters via maximum likelihood and Bayesian forecasting of volatility. However, applicability of MLE is restricted to cases with a discrete distribution of volatility components. From a practical point of view, ML also becomes computationally unfeasible for large numbers of components even if they are drawn from a discrete distribution. Here we propose an alternative GMM estimator together with linear forecasts which in principle is applicable for any continuous distribution with any number of volatility components. Monte Carlo studies show that GMM performs reasonably well for the popular Binomial and Lognormal models and that the loss incurred with linear compared to optimal forecasts is small. Extending the number of volatility components beyond what is feasible with MLE leads to gains in forecasting accuracy for some time series. --Markov-switching,Multifractal,Forecasting,Volatility,GMM estimation

A g-and-h copula approach to risk measurement in multivariate financial models

We propose and backtest a multivariate Value-at-Risk model for financial returns based on Tukey's g-and-h distribution. This distributional assumption is especially useful if (conditional) asymmetries as well as heavy tails have to be considered and fast random sampling is of importance. To illustrate our methodology, we fit copula GARCH models with g-and-h distributed residuals to three European stock indices and provide results of out-of-sample Value-at-Risk backtests. We find that our g-and-h model outperforms models with less flexible residual distributions and attains similar results as a benchmark model based on Hansen's skewed-t distribution

STANDARD REGRESSION VERSUS MULTILEVEL MODELING OF MULTISTAGE COMPLEX SURVEY DATA

Complex surveys based on multistage design are commonly used to collect large population data. Stratification, clustering and unequal probability of the selection of individuals are the complexities of complex survey design. Statistical techniques such as the multilevel modeling – scaled weights technique and the standard regression – robust variance estimation technique are used to analyze the complex survey data. Both statistical techniques take into account the complexities of complex survey data but the ways are different. This thesis compares the performance of the multilevel modeling – scaled weights and the standard regression – robust variance estimation technique based on analysis of the cross-sectional and the longitudinal complex survey data. Performance of these two techniques was examined by Monte Carlo simulation based on cross-sectional complex survey design. A stratified, multistage probability sample design was used to select samples for the cross-sectional Canadian Heart Health Surveys (CHHS) conducted in ten Canadian provinces and for the longitudinal National Population Health Survey (NPHS). Both statistical techniques (the multilevel modeling – scaled weights and the standard regression – robust variance estimation technique) were utilized to analyze CHHS and NPHS data sets. The outcome of interest was based on the question “Do you have any of the following long-term conditions that have been diagnosed by a health professional? – Diabetes”. For the cross-sectional CHHS, the results obtained from the proposed two statistical techniques were not consistent. However, the results based on analysis of the longitudinal NPHS data indicated that the performance of the standard regression – robust variance estimation technique might be better than the multilevel modeling – scaled weight technique for analyzing longitudinal complex survey data. Finally, in order to arrive at a definitive conclusion, a Monte Carlo simulation was used to compare the performance of the multilevel modeling – scaled weights and the standard regression – robust variance estimation techniques . In the Monte Carlo simulation study, the data were generated randomly based on the Canadian Heart Health Survey data for Saskatchewan province. The total 100 and 1000 number of simulated data sets were generated and the sample size for each simulated data set was 1,731. The results of this Monte Carlo simulation study indicated that the performance of the multilevel modeling – scaled weights technique and the standard regression – robust variance estimation technique were comparable to analyze the cross-sectional complex survey data. To conclude, both statistical techniques yield similar results when used to analyze the cross-sectional complex survey data, however standard regression-robust variance estimation technique might be preferred because it fully accounts for stratification, clustering and unequal probability of selection