Multi-Graph Decoding for Code-Switching ASR

In the FAME! Project, a code-switching (CS) automatic speech recognition (ASR) system for Frisian-Dutch speech is developed that can accurately transcribe the local broadcaster's bilingual archives with CS speech. This archive contains recordings with monolingual Frisian and Dutch speech segments as well as Frisian-Dutch CS speech, hence the recognition performance on monolingual segments is also vital for accurate transcriptions. In this work, we propose a multi-graph decoding and rescoring strategy using bilingual and monolingual graphs together with a unified acoustic model for CS ASR. The proposed decoding scheme gives the freedom to design and employ alternative search spaces for each (monolingual or bilingual) recognition task and enables the effective use of monolingual resources of the high-resourced mixed language in low-resourced CS scenarios. In our scenario, Dutch is the high-resourced and Frisian is the low-resourced language. We therefore use additional monolingual Dutch text resources to improve the Dutch language model (LM) and compare the performance of single- and multi-graph CS ASR systems on Dutch segments using larger Dutch LMs. The ASR results show that the proposed approach outperforms baseline single-graph CS ASR systems, providing better performance on the monolingual Dutch segments without any accuracy loss on monolingual Frisian and code-mixed segments.Comment: Accepted for publication at Interspeech 201

Dual sticky hierarchical Dirichlet process hidden Markov model and its application to natural language description of motions

In this paper, a new nonparametric Bayesian model called the dual sticky hierarchical Dirichlet process hidden Markov modle (HDP-HMM) is proposed for mining activities from a collection of time series data such as trajectories. All the time series data are clustered. Each cluster of time series data, corresponding to a motion pattern, is modeled by an HMM. Our model postulates a set of HMMs that share a common set of states (topics in an analogy with topic models for document processing), but have unique transition distributions. The number of HMMs and the number of topics are both automatically determined. The sticky prior avoids redundant states and makes our HDP-HMM more effective to model multimodal observations. For the application to motion trajectory modeling, topics correspond to motion activities. The learnt topics are clustered into atomic activities which are assigned predicates. We propose a Bayesian inference method to decompose a given trajectory into a sequence of atomic activities. The sources and sinks in the scene are learnt by clustering endpoints (origins and destinations of trajectories). The semantic motion regions are learnt using the points in trajectories. On combining the learnt sources and sinks, semantic motion regions, and the learnt sequences of atomic activities. the action represented by the trajectory can be described in natural language in as autometic a way as possible.The effectiveness of our dual sticky HDP-HMM is validated on several trajectory datasets. The effectiveness of the natural language descriptions for motions is demonstrated on the vehicle trajectories extracted from a traffic scene

Dual Mechanisms of Cognitive Control: A Hierarchical Bayesian Approach to Test-Retest Reliability

Cognitive control, also known as attentional control or executive function, is a set of fundamental processes that are utilized in a wide range of cognitive functioning: including working memory, reasoning, problem solving, and decision making. Currently, no existing theory of cognitive control unifies experimental and individual differences approaches. Some even argue that cognitive control as a psychometric construct does not exist at all. These disparities may exist in part because individual differences research in cognitive control utilizes tasks optimized for experimental effects (i.e., Stroop effect). As a result, many cognitive control tasks do not have reliable individual differences despite robust experimental effects (Hedge, Powell, & Sumner, 2018). In the current study, we examine the efficacy of a new task battery based on the Dual Mechanisms of Cognitive Control theory (DMCC; Braver, 2012) to provide reliable estimates of individual differences in cognitive control. With two sets of analyses, the first traditional (e.g., split-half, ICC, and rho), and the second hierarchical Bayesian, we provide evidence that (1) reliable individual differences can be extracted from experimental tasks, and (2) weak correlations between tasks of cognitive control are not solely caused by the attenuation of unreliable estimates. The implications of our findings suggest that it is unlikely that poor measurement practices are the cause of the weak between-task correlations in cognitive control, and that a psychometric construct of cognitive control should be reconsidered