Somatic ABC's: A Theoretical Framework for Designing, Developing and Evaluating the Building Blocks of Touch-Based Information Delivery

abstract: Situations of sensory overload are steadily becoming more frequent as the ubiquity of technology approaches reality--particularly with the advent of socio-communicative smartphone applications, and pervasive, high speed wireless networks. Although the ease of accessing information has improved our communication effectiveness and efficiency, our visual and auditory modalities--those modalities that today's computerized devices and displays largely engage--have become overloaded, creating possibilities for distractions, delays and high cognitive load; which in turn can lead to a loss of situational awareness, increasing chances for life threatening situations such as texting while driving. Surprisingly, alternative modalities for information delivery have seen little exploration. Touch, in particular, is a promising candidate given that it is our largest sensory organ with impressive spatial and temporal acuity. Although some approaches have been proposed for touch-based information delivery, they are not without limitations including high learning curves, limited applicability and/or limited expression. This is largely due to the lack of a versatile, comprehensive design theory--specifically, a theory that addresses the design of touch-based building blocks for expandable, efficient, rich and robust touch languages that are easy to learn and use. Moreover, beyond design, there is a lack of implementation and evaluation theories for such languages. To overcome these limitations, a unified, theoretical framework, inspired by natural, spoken language, is proposed called Somatic ABC's for Articulating (designing), Building (developing) and Confirming (evaluating) touch-based languages. To evaluate the usefulness of Somatic ABC's, its design, implementation and evaluation theories were applied to create communication languages for two very unique application areas: audio described movies and motor learning. These applications were chosen as they presented opportunities for complementing communication by offloading information, typically conveyed visually and/or aurally, to the skin. For both studies, it was found that Somatic ABC's aided the design, development and evaluation of rich somatic languages with distinct and natural communication units.Dissertation/ThesisPh.D. Computer Science 201

Robust Modeling of Epistemic Mental States and Their Applications in Assistive Technology

This dissertation presents the design and implementation of EmoAssist: Emotion-Enabled Assistive Tool to Enhance Dyadic Conversation for the Blind . The key functionalities of the system are to recognize behavioral expressions and to predict 3-D affective dimensions from visual cues and to provide audio feedback to the visually impaired in a natural environment. Prior to describing the EmoAssist, this dissertation identifies and advances research challenges in the analysis of the facial features and their temporal dynamics with Epistemic Mental States in dyadic conversation. A number of statistical analyses and simulations were performed to get the answer of important research questions about the complex interplay between facial features and mental states. It was found that the non-linear relations are mostly prevalent rather than the linear ones. Further, the portable prototype of assistive technology that can aid blind individual to understand his/her interlocutor\u27s mental states has been designed based on the analysis. A number of challenges related to the system, communication protocols, error-free tracking of face and robust modeling of behavioral expressions /affective dimensions were addressed to make the EmoAssist effective in a real world scenario. In addition, orientation-sensor information from the phone was used to correct image alignment to improve the robustness in real life deployment. It was observed that the EmoAssist can predict affective dimensions with acceptable accuracy (Maximum Correlation-Coefficient for valence: 0.76, arousal: 0.78, and dominance: 0.76) in natural conversation. The overall minimum and maximum response-times are (64.61 milliseconds) and (128.22 milliseconds), respectively. The integration of sensor information for correcting the orientation has helped in significant improvement (16% in average) of accuracy in recognizing behavioral expressions. A user study with ten blind people shows that the EmoAssist is highly acceptable to them (Average acceptability rating using Likert: 6.0 where 1 and 7 are the lowest and highest possible ratings, respectively) in social interaction

Experimental elicitation of gaze aversion in the context of neonate-caregiver interaction /

Infant behaviors during the newborn period were observed in response to the manipulation of maternal behaviors. Data analyses revealed that this sample of newborns was responsive to increases in maternal behaviors, particularly maternal vocal behaviors. Infants looked more when mothers increased their vocal stimulation of the newborns, but only if the infants' baseline levels of looking at mother were low. When baseline levels of looking at mother were high and mothers increased their vocal stimulation, and maintained the increase for an extended period of time, infants looked less at mothers; that is, the newborns gaze avoided more. In addition, maternal tactile stimulation and infant tactile behaviors were differentiated in accordance with the factor of infant gender.Research with older infants indicates that maternal stimulating behaviors may have the effect of either increasing or decreasing infant visual attentiveness, dependent primarily upon the type of maternal behavior examined. Those maternal behaviors which serve to decrease attentiveness have been termed overstimulating and seem to produce a state of overload for the infant. This effect assumes particular importance since visual communication between mothers and infants may serve as an important context for the elaboration of other types of communicative behaviors. In addition, gaze avoidance in infants and young children has been related to a variety of pathological anomalies. There have been no studies, however, which have examined the impact of levels of maternal stimulation on the visual behaviors of neonates in the context of nonfeeding social interaction.Mothers in the present study were asked to increase either their visual behaviors, vocal behaviors, or both, in an attempt to ascertain whether alterations in these aspects of their behavior would have an impact on neonatal visual behaviors. Of particular interest was whether it is possible to elicit neonatal gaze avoidance through maternal overstimulation. In addition, this design allowed examination of the specific types of maternal behavior changes which might produce an effect

Development of an Intervention for Improving Food Acceptance of People with Hearing Loss

Hearing loss, defined as the partial or total inability to hear sound in one or both ears, is the most common sensory deficit in adults to date. Approximately 15% of American adults aged 18 and over report some trouble hearing. The impact of hearing loss may be profound, with consequences for the social, functional, and psychological well-being of the person. Surprisingly, very little attention has been paid on whether auditory loss can significantly impact consumers’ sensory perception and overall enjoyment of food. There were four objectives of this dissertation study. Chapter 1 aimed to determine the impacts of hearing loss on the sensory perception and acceptance of solid, and liquid food matrices with various intensities of textural attributes. Chapter 2 was designed to understand the relationships between hearing loss and aroma, flavor and taste perception and acceptance. Chapter 3 aimed to determine the impacts of environmental cues on consumers’ with hearing loss perception of their eating environments and food liking and perception in a social dining context. Finally, Chapter 4 aimed to develop an appropriate intervention that improves consumers’ with hearing loss overall food acceptance. Results showed that auditory loss impacted the overall acceptance and loudness perception of solid food samples. Pitch intensity was found as a significant negative contributor to the overall liking of solid food samples in individuals with hearing loss. In addition, subjects with hearing loss were not able to discriminate solid food samples with smaller differences in crispness. Loudness perception of liquid foods was also impacted by hearing loss. The group with hearing loss rated liquid samples as less loud compared to the group with normal hearing. No impact of hearing loss was observed on the overall enjoyment of liquid samples. Hearing loss decreased the aroma, flavor perception, and flavor acceptance of applesauce, and orange juice, but little effects were observed on taste perception. Loud external auditory cues negatively impacted the texture liking and flavor perception of food, as well as the general comfort and engagement of subjects with hearing loss during social dining. Finally, a flavor-enhanced food product proved to be an appropriate intervention plan to improve individuals with hearing loss overall food acceptance. The outcomes of this dissertation study may offer new strategies for the improvement of the enjoyment of food for consumers with auditory loss. Additionally, this research may motivate the food industry to develop new products for the growing consumer segment that are people with hearing loss

Development of an Intervention for Improving Food Acceptance of People with Hearing Loss

Hearing loss, defined as the partial or total inability to hear sound in one or both ears, is the most common sensory deficit in adults to date. Approximately 15% of American adults aged 18 and over report some trouble hearing. The impact of hearing loss may be profound, with consequences for the social, functional, and psychological well-being of the person. Surprisingly, very little attention has been paid on whether auditory loss can significantly impact consumers’ sensory perception and overall enjoyment of food. There were four objectives of this dissertation study. Chapter 1 aimed to determine the impacts of hearing loss on the sensory perception and acceptance of solid, and liquid food matrices with various intensities of textural attributes. Chapter 2 was designed to understand the relationships between hearing loss and aroma, flavor and taste perception and acceptance. Chapter 3 aimed to determine the impacts of environmental cues on consumers’ with hearing loss perception of their eating environments and food liking and perception in a social dining context. Finally, Chapter 4 aimed to develop an appropriate intervention that improves consumers’ with hearing loss overall food acceptance. Results showed that auditory loss impacted the overall acceptance and loudness perception of solid food samples. Pitch intensity was found as a significant negative contributor to the overall liking of solid food samples in individuals with hearing loss. In addition, subjects with hearing loss were not able to discriminate solid food samples with smaller differences in crispness. Loudness perception of liquid foods was also impacted by hearing loss. The group with hearing loss rated liquid samples as less loud compared to the group with normal hearing. No impact of hearing loss was observed on the overall enjoyment of liquid samples. Hearing loss decreased the aroma, flavor perception, and flavor acceptance of applesauce, and orange juice, but little effects were observed on taste perception. Loud external auditory cues negatively impacted the texture liking and flavor perception of food, as well as the general comfort and engagement of subjects with hearing loss during social dining. Finally, a flavor-enhanced food product proved to be an appropriate intervention plan to improve individuals with hearing loss overall food acceptance. The outcomes of this dissertation study may offer new strategies for the improvement of the enjoyment of food for consumers with auditory loss. Additionally, this research may motivate the food industry to develop new products for the growing consumer segment that are people with hearing loss

Design and semantics of form and movement (DeSForM 2006)

Design and Semantics of Form and Movement (DeSForM) grew from applied research exploring emerging design methods and practices to support new generation product and interface design. The products and interfaces are concerned with: the context of ubiquitous computing and ambient technologies and the need for greater empathy in the pre-programmed behaviour of the ‘machines’ that populate our lives. Such explorative research in the CfDR has been led by Young, supported by Kyffin, Visiting Professor from Philips Design and sponsored by Philips Design over a period of four years (research funding £87k). DeSForM1 was the first of a series of three conferences that enable the presentation and debate of international work within this field: • 1st European conference on Design and Semantics of Form and Movement (DeSForM1), Baltic, Gateshead, 2005, Feijs L., Kyffin S. & Young R.A. eds. • 2nd European conference on Design and Semantics of Form and Movement (DeSForM2), Evoluon, Eindhoven, 2006, Feijs L., Kyffin S. & Young R.A. eds. • 3rd European conference on Design and Semantics of Form and Movement (DeSForM3), New Design School Building, Newcastle, 2007, Feijs L., Kyffin S. & Young R.A. eds. Philips sponsorship of practice-based enquiry led to research by three teams of research students over three years and on-going sponsorship of research through the Northumbria University Design and Innovation Laboratory (nuDIL). Young has been invited on the steering panel of the UK Thinking Digital Conference concerning the latest developments in digital and media technologies. Informed by this research is the work of PhD student Yukie Nakano who examines new technologies in relation to eco-design textiles