LipLearner: Customizable Silent Speech Interactions on Mobile Devices

Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.Comment: Conditionally accepted to the ACM CHI Conference on Human Factors in Computing Systems 2023 (CHI '23

Android Flash Based Game for Hard Hearing Kids to Learn Malay Language through Cued Speech and Sign Language (MYKIU)

The purpose of this project is to build android application that use as compliment to conventional education system. It use to assist hard hearing kids learning environment to be more interactive and portable. Therefore Android Flash Based Game for Hard Hearing Kids to Learn Malay Language through Cued Speech and Sign Language (MYKIU) developed to assists in hard hearing learning process in reading. This application is using Cued Speech and Malay Sign Language as learning approach. Advantage of MYKIU is act as compliment for traditional system where hard hearing kids would be able to learn through game based approach even though they are not at school. In the android market, android applications that are developed using Malay Sign Language and Cued Speech are not exist yet; most of the application is in American Sign Language (ASL) and Cued Speech that using English vocabulary. Therefore, MYKIU is developed to break the barrier. MYKIU is developed using Cued Speech and Malay Sign Language (MSL) in Malay vocabulary; this application is specifically design to assist hard hearing kids in Malaysia. The scope of the study for this project is focusing for hard hearing kids from 6 to 9 years old. MYKIU developed using phase development life cycle. MYKIU is using Action Script 3 as the programming language. It is developed using Adobe Flash CS5.5 and Adobe Photoshop Portable CS5. MYKIU prototype is tested in Pusat Pertuturan Kiu, Kampung Pandan. The author is able to gather 10 students age from 6 to 9 years old to test the prototype. From the testing, MYKIU get good response when it use by hard hearing kids

Development of an auditory rehabilitation training programme for hearing impaired adults in Saudi Arabia

Hearing impairment is one of the most common sensory disabilities, affecting millions of people worldwide. The current management of acquired hearing impairment in adults is usually in the form of hearing aids. An auditory rehabilitation programme may or may not be provided although literature suggests that such programmes may enhance hearing aid benefits. This study was conducted in an Arabic-speaking country, specifically Saudi Arabia, and is the first to develop an auditory rehabilitation intervention programme in the Arabic language that focuses on working age adults who are new hearing aid users. Method: Randomised intervention study design. Thirty-five participants with mild to severe acquired hearing loss (intervention group n=18, control group n=17). Results: Significant changes were found in the performance of the intervention group in the Minimal Audible Capabilities (MAC) test, a lip reading test and the Glasgow Hearing Aid Benefit Profile (GHABP) questionnaire when compared with the control group. No significant difference was found in speech test scores between the two groups. Conclusions: A short individualised auditory rehabilitation intervention programme can yield a significant improvement in the performance of hearing impaired adults by enhancing the benefits of hearing aid use

Caloric vestibular stimulation in aphasic syndrome

Caloric vestibular stimulation (CVS) is commonly used to diagnose brainstem disorder but its therapeutic application is much less established. Based on the finding that CVS increases blood flow to brain structures associated with language and communication, we assessed whether the procedure has potential to relieve symptoms of post-stroke aphasia. Three participants, each presenting with chronic, unilateral lesions to the left hemisphere, were administered daily CVS for 4 consecutive weeks. Relative to their pre-treatment baseline scores, two of the three participants showed significant improvement on both picture and responsive naming at immediate and one-week follow-up. One of these participants also showed improved sentence repetition, and another showed improved auditory word discrimination. No adverse reactions were reported. These data provide the first, albeit tentative, evidence that CVS may relieve expressive and receptive symptoms of aphasia. A larger, sham-controlled study is now needed to further assess efficacy

Personalization in object-based audio for accessibility : a review of advancements for hearing impaired listeners

Hearing loss is widespread and significantly impacts an individual’s ability to engage with broadcast media. Access can be improved through new object-based audio personalization methods. Utilizing the literature on hearing loss and intelligibility this paper develops three dimensions which are evidenced to improve intelligibility: spatial separation, speech to noise ratio and redundancy. These can be personalized, individually or concurrently, using object based audio. A systematic review of all work in object-based audio personalization is then undertaken. These dimensions are utilized to evaluate each project’s approach to personalisation, identifying successful approaches, commercial challenges and the next steps required to ensure continuing improvements to broadcast audio for hard of hearing individuals