I hear you eat and speak: automatic recognition of eating condition and food type, use-cases, and impact on ASR performance

We propose a new recognition task in the area of computational paralinguistics: automatic recognition of eating conditions in speech, i. e., whether people are eating while speaking, and what they are eating. To this end, we introduce the audio-visual iHEARu-EAT database featuring 1.6 k utterances of 30 subjects (mean age: 26.1 years, standard deviation: 2.66 years, gender balanced, German speakers), six types of food (Apple, Nectarine, Banana, Haribo Smurfs, Biscuit, and Crisps), and read as well as spontaneous speech, which is made publicly available for research purposes. We start with demonstrating that for automatic speech recognition (ASR), it pays off to know whether speakers are eating or not. We also propose automatic classification both by brute-forcing of low-level acoustic features as well as higher-level features related to intelligibility, obtained from an Automatic Speech Recogniser. Prediction of the eating condition was performed with a Support Vector Machine (SVM) classifier employed in a leave-one-speaker-out evaluation framework. Results show that the binary prediction of eating condition (i. e., eating or not eating) can be easily solved independently of the speaking condition; the obtained average recalls are all above 90%. Low-level acoustic features provide the best performance on spontaneous speech, which reaches up to 62.3% average recall for multi-way classification of the eating condition, i. e., discriminating the six types of food, as well as not eating. The early fusion of features related to intelligibility with the brute-forced acoustic feature set improves the performance on read speech, reaching a 66.4% average recall for the multi-way classification task. Analysing features and classifier errors leads to a suitable ordinal scale for eating conditions, on which automatic regression can be performed with up to 56.2% determination coefficient

Computer classification of stop consonants in a speaker independent continuous speech environment

In the English language there are six stop consonants, /b,d,g,p,t,k/. They account for over 17% of all phonemic occurrences. In continuous speech, phonetic recognition of stop consonants requires the ability to explicitly characterize the acoustic signal. Prior work has shown that high classification accuracy of discrete syllables and words can be achieved by characterizing the shape of the spectrally transformed acoustic signal. This thesis extends this concept to include a multispeaker continuous speech database and statistical moments of a distribution to characterize shape. A multivariate maximum likelihood classifier was used to discriminate classes. To reduce the number of features used by the discriminant model a dynamic programming scheme was employed to optimize subset combinations. The top six moments were the mean, variance, and skewness in both frequency and energy. Results showed 85% classification on the full database of 952 utterances. Performance improved to 97% when the discriminant model was trained separately for male and female talkers

Anti-spoofing Methods for Automatic SpeakerVerification System

Growing interest in automatic speaker verification (ASV)systems has lead to significant quality improvement of spoofing attackson them. Many research works confirm that despite the low equal er-ror rate (EER) ASV systems are still vulnerable to spoofing attacks. Inthis work we overview different acoustic feature spaces and classifiersto determine reliable and robust countermeasures against spoofing at-tacks. We compared several spoofing detection systems, presented so far,on the development and evaluation datasets of the Automatic SpeakerVerification Spoofing and Countermeasures (ASVspoof) Challenge 2015.Experimental results presented in this paper demonstrate that the useof magnitude and phase information combination provides a substantialinput into the efficiency of the spoofing detection systems. Also wavelet-based features show impressive results in terms of equal error rate. Inour overview we compare spoofing performance for systems based on dif-ferent classifiers. Comparison results demonstrate that the linear SVMclassifier outperforms the conventional GMM approach. However, manyresearchers inspired by the great success of deep neural networks (DNN)approaches in the automatic speech recognition, applied DNN in thespoofing detection task and obtained quite low EER for known and un-known type of spoofing attacks.Comment: 12 pages, 0 figures, published in Springer Communications in Computer and Information Science (CCIS) vol. 66

In-situ crack and keyhole pore detection in laser directed energy deposition through acoustic signal and deep learning

Cracks and keyhole pores are detrimental defects in alloys produced by laser directed energy deposition (LDED). Laser-material interaction sound may hold information about underlying complex physical events such as crack propagation and pores formation. However, due to the noisy environment and intricate signal content, acoustic-based monitoring in LDED has received little attention. This paper proposes a novel acoustic-based in-situ defect detection strategy in LDED. The key contribution of this study is to develop an in-situ acoustic signal denoising, feature extraction, and sound classification pipeline that incorporates convolutional neural networks (CNN) for online defect prediction. Microscope images are used to identify locations of the cracks and keyhole pores within a part. The defect locations are spatiotemporally registered with acoustic signal. Various acoustic features corresponding to defect-free regions, cracks, and keyhole pores are extracted and analysed in time-domain, frequency-domain, and time-frequency representations. The CNN model is trained to predict defect occurrences using the Mel-Frequency Cepstral Coefficients (MFCCs) of the lasermaterial interaction sound. The CNN model is compared to various classic machine learning models trained on the denoised acoustic dataset and raw acoustic dataset. The validation results shows that the CNN model trained on the denoised dataset outperforms others with the highest overall accuracy (89%), keyhole pore prediction accuracy (93%), and AUC-ROC score (98%). Furthermore, the trained CNN model can be deployed into an in-house developed software platform for online quality monitoring. The proposed strategy is the first study to use acoustic signals with deep learning for insitu defect detection in LDED process.Comment: 36 Pages, 16 Figures, accepted at journal Additive Manufacturin

Spectral feature classification of oceanographic processes using an autonomous underwater vehicle

Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution June 2000The thesis develops and demonstrates methods of classifying ocean processes using an underwater moving platform such as an Autonomous Underwater Vehicle (AUV). The "mingled spectrum principle" is established which concisely relates observations from a moving platform to the frequency-wavenumber spectrum of the ocean process. It clearly reveals the role of the AUV speed in mingling temporal and spatial information. For classifying different processes, an AUV is not only able to jointly utilize the time-space information, but also at a tunable proportion by adjusting its cruise speed. In this respect, AUVs are advantageous compared with traditional oceanographic platforms. Based on the mingled spectrum principle, a parametric tool for designing an AUVbased spectral classifier is developed. An AUV's controllable speed tunes the separability between the mingled spectra of different processes. This property is the key to optimizing the classifier's performance. As a case study, AUV-based classification is applied to distinguish ocean convection from internal waves. The mingled spectrum templates are derived from the MIT Ocean Convection Model and the Garrett-Munk internal wave spectrum model. To allow for mismatch between modeled templates and real measurements, the AUVbased classifier is designed to be robust to parameter uncertainties. By simulation tests on the classifier, it is demonstrated that at a higher AUV speed, convection's distinct spatial feature is highlighted to the advantage of classification. Experimental data are used to test the AUV-based classifier. An AUV-borne flow measurement system is designed and built, using an Acoustic Doppler Velocimeter (ADV). The system is calibrated in a high-precision tow tank. In February 1998, the AUV acquired field data of flow velocity in the Labrador Sea Convection Experiment. The Earth-referenced vertical flow velocity is extracted from the raw measurements. The classification test result detects convection's occurrence, a finding supported by more traditional oceanographic analyses and observations. The thesis work provides an important foundation for future work in autonomous detection and sampling of oceanographic processes.This thesis research has been funded by the Office of Naval Research (ONR) under Grants NOOOl4-95-1-1316, NOO0l4-97-1-0470, and by the MIT Sea Grant College Program under Grant NA46RG0434

Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors

The work provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation

Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors

The work provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation

A Comparative Study of Spectral Peaks Versus Global Spectral Shape as Invariant Acoustic Cues for Vowels

The primary objective of this study was to compare two sets of vowel spectral features, formants and global spectral shape parameters, as invariant acoustic cues to vowel identity. Both automatic vowel recognition experiments and perceptual experiments were performed to evaluate these two feature sets. First, these features were compared using the static spectrum sampled in the middle of each steady-state vowel versus features based on dynamic spectra. Second, the role of dynamic and contextual information was investigated in terms of improvements in automatic vowel classification rates. Third, several speaker normalizing methods were examined for each of the feature sets. Finally, perceptual experiments were performed to determine whether vowel perception is more correlated with formants or global spectral shape. Results of the automatic vowel classification experiments indicate that global spectral shape features contain more information than do formants. For both feature sets, dynamic features are superior to static features. Spectral features spanning a time interval beginning with the start of the on-glide region of the acoustic vowel segment and ending at the end of the off-glide region of the acoustic vowel segment are required for maximum vowel recognition accuracy. Speaker normalization of both static and dynamic features can also be used to improve the automatic vowel recognition accuracy. Results of the perceptual experiments with synthesized vowel segments indicate that if formants are kept fixed, global spectral shape can, at least for some conditions, be modified such that the synthetic speech token will be perceived according to spectral shape cues rather than formant cues. This result implies that overall spectral shape may be more important perceptually than the spectral prominences represented by the formants. The results of this research contribute to a fundamental understanding of the information-encoding process in speech. The signal processing techniques used and the acoustic features found in this study can also be used to improve the preprocessing of acoustic signals in the front-end of automatic speech recognition systems