Modeling Dependent Structure for Utterances in ASR Evaluation

The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluation. To deal with dependent speech data, the blockwise bootstrap approach is also introduced. By dividing utterances into uncorrelated blocks, this approach resamples these blocks instead of original data. However, it is typically nontrivial to uncover the dependent structure among utterances and identify the blocks, which might lead to subjective conclusions in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate uncorrelated blocks of utterances in a rigorous way, after which blockwise bootstrap is applied on top of the inferred blocks. We show the resulting variance estimator of WER in ASR evaluation is statistically consistent under mild conditions. We also demonstrate the validity of proposed approach on LibriSpeech dataset

Systematic errors in current quantum state tomography tools

Common tools for obtaining physical density matrices in experimental quantum state tomography are shown here to cause systematic errors. For example, using maximum likelihood or least squares optimization for state reconstruction, we observe a systematic underestimation of the fidelity and an overestimation of entanglement. A solution for this problem can be achieved by a linear evaluation of the data yielding reliable and computational simple bounds including error bars.Comment: 8 pages, 8 figure

Computationally Efficient Confidence Intervals for Cross-validated Area Under the ROC Curve Estimates

In binary classification problems, the area under the ROC curve (AUC), is an effective means of measuring the performance of your model. Most often, cross-validation is also used, in order to assess how the results will generalize to an independent data set. In order to evaluate the quality of an estimate for cross-validated AUC, we must obtain an estimate for its variance. For massive data sets, the process of generating a single performance estimate can be computationally expensive. Additionally, when using a complex prediction method, calculating the cross-validated AUC on even a relatively small data set can still require a large amount of computation time. Thus, when the processes of obtaining a single estimate for cross-validated AUC is significant, the bootstrap, as a means of variance estimation, can be computationally intractable. As an alternative to the bootstrap, we demonstrate a computationally efficient influence curve based approach to obtaining a variance estimate for cross-validated AUC

Three Essays on Sampling Techniques: Small Sample Performances of Estimators and Predictors.

The dissertation addresses the issues of small sample properties of estimators and predictors. Economic analysis usually relies on the asymptotic properties of estimators and predictors which may not be the same as their asymptotic counterparts. Furthermore, some biased estimators and predictors used in economic studies have certain asymptotic properties which are not fully understood. Consequently, sampling techniques are used to explore the small sample properties and construct confidence intervals for predictors and estimators. In the dissertation, first, Monte Carlo experiments are used to find an appropriate estimation procedure for a system of simultaneous equations which involves a latent endogenous variable. Second, Monte Carlo experiments are used to explore the small sample property of the \u27equity estimator\u27 and compare it to the small sample properties of the \u27traditional\u27 estimators. Third, bootstrap sampling techniques is utilized to construct confidence intervals for the out-of-sample forecasts obtained via biased predictors which cannot be constructed in the usual way. The findings are (1) an instrumental variables approach is an appropriate alternative estimation technique of the system of simultaneous equation involving a latent endogenous variable; (2) the small sample of the equity estimator is dependent on the vector lengths and the conditioning of the data; and (3) bootstrap method produces reasonable confidence intervals for out-of-sample forecasts

Automated speech tools for helping communities process restricted-access corpora for language revival efforts

Many archival recordings of speech from endangered languages remain unannotated and inaccessible to community members and language learning programs. One bottleneck is the time-intensive nature of annotation. An even narrower bottleneck occurs for recordings with access constraints, such as language that must be vetted or filtered by authorised community members before annotation can begin. We propose a privacy-preserving workflow to widen both bottlenecks for recordings where speech in the endangered language is intermixed with a more widely-used language such as English for meta-linguistic commentary and questions (e.g. What is the word for 'tree'?). We integrate voice activity detection (VAD), spoken language identification (SLI), and automatic speech recognition (ASR) to transcribe the metalinguistic content, which an authorised person can quickly scan to triage recordings that can be annotated by people with lower levels of access. We report work-in-progress processing 136 hours archival audio containing a mix of English and Muruwari. Our collaborative work with the Muruwari custodian of the archival materials show that this workflow reduces metalanguage transcription time by 20% even with minimal amounts of annotated training data: 10 utterances per language for SLI and for ASR at most 39 minutes, and possibly as little as 39 seconds.</p