I hear you eat and speak: automatic recognition of eating condition and food type, use-cases, and impact on ASR performance

We propose a new recognition task in the area of computational paralinguistics: automatic recognition of eating conditions in speech, i. e., whether people are eating while speaking, and what they are eating. To this end, we introduce the audio-visual iHEARu-EAT database featuring 1.6 k utterances of 30 subjects (mean age: 26.1 years, standard deviation: 2.66 years, gender balanced, German speakers), six types of food (Apple, Nectarine, Banana, Haribo Smurfs, Biscuit, and Crisps), and read as well as spontaneous speech, which is made publicly available for research purposes. We start with demonstrating that for automatic speech recognition (ASR), it pays off to know whether speakers are eating or not. We also propose automatic classification both by brute-forcing of low-level acoustic features as well as higher-level features related to intelligibility, obtained from an Automatic Speech Recogniser. Prediction of the eating condition was performed with a Support Vector Machine (SVM) classifier employed in a leave-one-speaker-out evaluation framework. Results show that the binary prediction of eating condition (i. e., eating or not eating) can be easily solved independently of the speaking condition; the obtained average recalls are all above 90%. Low-level acoustic features provide the best performance on spontaneous speech, which reaches up to 62.3% average recall for multi-way classification of the eating condition, i. e., discriminating the six types of food, as well as not eating. The early fusion of features related to intelligibility with the brute-forced acoustic feature set improves the performance on read speech, reaching a 66.4% average recall for the multi-way classification task. Analysing features and classifier errors leads to a suitable ordinal scale for eating conditions, on which automatic regression can be performed with up to 56.2% determination coefficient

Development Of A Sensory Descriptive Lexicon For Calamondins (Citrus Mitis Blanco) And Consumer Acceptance Of Calamondin Yogurt

The objectives of this research were to establish a semi-descriptive sensory language that describes the attributes of calamondin fruit and develop yogurt products flavored with calamondin and test consumers’ acceptance of the products. A total of 89 sensory terms were established by trained panelists to describe calamondins. Yogurt products were developed with 0%, 5%, 10%, and 15% calamondin pulp. Results demonstrated that no significant difference (P\u3e0.05) occurred for consumers’ acceptance of yogurt appearance and texture, and significant differences (P\u3c0.05) occurred for consumers’ acceptance of yogurt flavor and overall liking. Generally, the yogurt with 10% honey and the yogurt with 10% honey and 5% calamondin pulp were well accepted by consumers. The consumers were grouped into 6 clusters based on their prefences. The majority (60%) of the consumers liked moderately yogurts with 10% honey, and 0% and 5% calamondin pulp respectively

Acoustical characteristics of mastication sounds : application of speech analysis techniques

Food scientists have used acoustical methods to study characteristics of mastication sounds in relation to food texture. However, a model for analysis of the sounds has not been identified, and reliability of the methods has not been reported. Therefore, speech analysis techniques were applied to mastication sounds, and variation in measures of the sounds was examined. To meet these objectives, two experiments were conducted. In the first experiment, a digital sound spectrograph generated waveforms and wideband spectrograms of sounds by 3 adult subjects (1 male, 2 females) for initial chews of food samples differing in hardness and fracturability. Acoustical characteristics were described and compared. For all sounds, formants appeared in the spectrograms, and energy occurred across a 0 to 8000-Hz range of frequencies. Bursts characterized waveforms for peanut, almond, raw carrot, ginger snap, and hard candy. Duration and amplitude of the sounds varied with the subjects. In the second experiment, the spectrograph was used to measure the duration, amplitude, and formants of sounds for the initial 2 chews of cylindrical food samples (raw carrot, teething toast) differing in diameter (1.27, 1.90, 2.54 cm). Six adult subjects (3 males, 3 females) having normal occlusions and temporomandibular Joints chewed the samples between the molar teeth and with the mouth open. Ten repetitions per subject were examined for each food sample. Analysis of estimates of variation indicated an inconsistent intrasubject variation in the acoustical measures. Food type and sample diameter also affected the estimates, indicating the variable nature of mastication. Generally, intrasubject variation was greater than intersubject variation. Analysis of ranks of the data indicated that the effect of sample diameter on the acoustical measures was inconsistent and depended on the subject and type of food. If inferences are to be made concerning food texture from acoustical measures of mastication sounds, intrasubject variability must be stabilized. Research should be directed toward examining experimental techniques and designs to stabilize the variability. Possibilities include identifying factors that may covary with acoustical measures, developing procedures for screening and selecting subjects, establishing procedures for chewing food samples, and defining dimensions for food samples

LeviSense: a platform for the multisensory integration in levitating food and insights into its effect on flavour perception

Eating is one of the most multisensory experiences in everyday life. All of our five senses (i.e. taste, smell, vision, hearing and touch) are involved, even if we are not aware of it. However, while multisensory integration has been well studied in psychology, there is not a single platform for testing systematically the effects of different stimuli. This lack of platform results in unresolved design challenges for the design of taste-based immersive experiences. Here, we present LeviSense: the first system designed for multisensory integration in gustatory experiences based on levitated food. Our system enables the systematic exploration of different sensory effects on eating experiences. It also opens up new opportunities for other professionals (e.g., molecular gastronomy chefs) looking for innovative taste-delivery platforms. We describe the design process behind LeviSense and conduct two experiments to test a subset of the crossmodal combinations (i.e., taste and vision, taste and smell). Our results show how different lighting and smell conditions affect the perceived taste intensity, pleasantness, and satisfaction. We discuss how LeviSense creates a new technical, creative, and expressive possibilities in a series of emerging design spaces within Human-Food Interaction

FoodFab: Creating Food Perception Illusions using Food 3D Printing

Personalization of eating such that everyone consumes only what they need allows improving our management of food waste. In this paper, we explore the use of food 3D printing to create perceptual illusions for controlling the level of perceived satiety given a defined amount of calories. We present FoodFab, a system that allows users to control their food intake through modifying a food's internal structure via two 3D printing parameters: infill pattern and infill density. In two experiments with a total of 30 participants, we studied the effect of these parameters on users' chewing time that is known to affect people's feeling of satiety. Our results show that we can indeed modify the chewing time by varying infill pattern and density, and thus control perceived satiety. Based on the results, we propose two computational models and integrate them into a user interface that simplifies the creation of personalized food structures

Development of an Intervention for Improving Food Acceptance of People with Hearing Loss

Hearing loss, defined as the partial or total inability to hear sound in one or both ears, is the most common sensory deficit in adults to date. Approximately 15% of American adults aged 18 and over report some trouble hearing. The impact of hearing loss may be profound, with consequences for the social, functional, and psychological well-being of the person. Surprisingly, very little attention has been paid on whether auditory loss can significantly impact consumers’ sensory perception and overall enjoyment of food. There were four objectives of this dissertation study. Chapter 1 aimed to determine the impacts of hearing loss on the sensory perception and acceptance of solid, and liquid food matrices with various intensities of textural attributes. Chapter 2 was designed to understand the relationships between hearing loss and aroma, flavor and taste perception and acceptance. Chapter 3 aimed to determine the impacts of environmental cues on consumers’ with hearing loss perception of their eating environments and food liking and perception in a social dining context. Finally, Chapter 4 aimed to develop an appropriate intervention that improves consumers’ with hearing loss overall food acceptance. Results showed that auditory loss impacted the overall acceptance and loudness perception of solid food samples. Pitch intensity was found as a significant negative contributor to the overall liking of solid food samples in individuals with hearing loss. In addition, subjects with hearing loss were not able to discriminate solid food samples with smaller differences in crispness. Loudness perception of liquid foods was also impacted by hearing loss. The group with hearing loss rated liquid samples as less loud compared to the group with normal hearing. No impact of hearing loss was observed on the overall enjoyment of liquid samples. Hearing loss decreased the aroma, flavor perception, and flavor acceptance of applesauce, and orange juice, but little effects were observed on taste perception. Loud external auditory cues negatively impacted the texture liking and flavor perception of food, as well as the general comfort and engagement of subjects with hearing loss during social dining. Finally, a flavor-enhanced food product proved to be an appropriate intervention plan to improve individuals with hearing loss overall food acceptance. The outcomes of this dissertation study may offer new strategies for the improvement of the enjoyment of food for consumers with auditory loss. Additionally, this research may motivate the food industry to develop new products for the growing consumer segment that are people with hearing loss

FoodFab: creating food perception illusions using food 3D printing

Food 3D printing enables the creation of customized food structures based on a person’s individual needs. In this paper, we explore the use of food 3D printing to create perceptual illusions for controlling the level of perceived satiety given a defined amount of calories. We present FoodFab, a system that allows users to control their food intake through modifying a food’s internal structure via two 3D printing parameters: infill pattern and infill density. In two experiments with a total of 30 participants, we studied the effect of these parameters on users’ chewing time that is known to affect people’s feeling of satiety. Our results show that we can indeed modify the chewing time by varying infill pattern and density, and thus control perceived satiety. Based on the results, we propose two computational models and integrate them into a user interface that simplifies the creation of personalized food structures

Development of an Intervention for Improving Food Acceptance of People with Hearing Loss

Hearing loss, defined as the partial or total inability to hear sound in one or both ears, is the most common sensory deficit in adults to date. Approximately 15% of American adults aged 18 and over report some trouble hearing. The impact of hearing loss may be profound, with consequences for the social, functional, and psychological well-being of the person. Surprisingly, very little attention has been paid on whether auditory loss can significantly impact consumers’ sensory perception and overall enjoyment of food. There were four objectives of this dissertation study. Chapter 1 aimed to determine the impacts of hearing loss on the sensory perception and acceptance of solid, and liquid food matrices with various intensities of textural attributes. Chapter 2 was designed to understand the relationships between hearing loss and aroma, flavor and taste perception and acceptance. Chapter 3 aimed to determine the impacts of environmental cues on consumers’ with hearing loss perception of their eating environments and food liking and perception in a social dining context. Finally, Chapter 4 aimed to develop an appropriate intervention that improves consumers’ with hearing loss overall food acceptance. Results showed that auditory loss impacted the overall acceptance and loudness perception of solid food samples. Pitch intensity was found as a significant negative contributor to the overall liking of solid food samples in individuals with hearing loss. In addition, subjects with hearing loss were not able to discriminate solid food samples with smaller differences in crispness. Loudness perception of liquid foods was also impacted by hearing loss. The group with hearing loss rated liquid samples as less loud compared to the group with normal hearing. No impact of hearing loss was observed on the overall enjoyment of liquid samples. Hearing loss decreased the aroma, flavor perception, and flavor acceptance of applesauce, and orange juice, but little effects were observed on taste perception. Loud external auditory cues negatively impacted the texture liking and flavor perception of food, as well as the general comfort and engagement of subjects with hearing loss during social dining. Finally, a flavor-enhanced food product proved to be an appropriate intervention plan to improve individuals with hearing loss overall food acceptance. The outcomes of this dissertation study may offer new strategies for the improvement of the enjoyment of food for consumers with auditory loss. Additionally, this research may motivate the food industry to develop new products for the growing consumer segment that are people with hearing loss