Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages.  This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems

Continuous Density Hidden Markov Model for Hindi Speech Recognition

State of the art automatic speech recognitionsystem uses Mel frequency cepstral coefficients as featureextractor along with Gaussian mixture model for acousticmodeling but there is no standard value to assign number ofmixture component in speech recognition process.Currentchoice of mixture component is arbitrary with littlejustification. Also the standard set for European languagescan not be used in Hindi speech recognition due to mismatchin database size of the languages.Parameter estimation withtoo many or few component may inappropriately estimatethe mixture model. Therefore, number of mixture isimportant for initial estimation of expectation maximizationprocess. In this research work, the authors estimate numberof Gaussian mixture component for Hindi database basedupon the size of vocabulary.Mel frequency cepstral featureand perceptual linear predictive feature along with itsextended variations with delta-delta-delta feature have beenused to evaluate this number based on optimal recognitionscore of the system . Comparitive analysis of recognitionperformance for both the feature extraction methods onmedium size Hindi database is also presented in thispaper.HLDA has been used as feature reduction techniqueand also its impact on the recognition score has beenhighlighted

On Developing an Automatic Speech Recognition System for Commonly used English Words in Indian English

Speech is one of the easiest and the fastest way to communicate. Recognition of speech by computer for various languages is a challenging task. The accuracy of Automatic speech recognition system (ASR) remains one of the key challenges, even after years of research. Accuracy varies due to speaker and language variability, vocabulary size and noise. Also, due to the design of speech recognition that is based on issues like- speech database, feature extraction techniques and performance evaluation. This paper aims to describe the development of a speaker-independent isolated automatic speech recognition system for Indian English language. The acoustic model is build using Carnegie Mellon University (CMU) Sphinx tools. The corpus used is based on Most Commonly used English words in everyday life. Speech database includes the recordings of 76 Punjabi Speakers (north-west Indian English accent). After testing, the system obtained an accuracy of 85.20 %, when trained using 128 GMMs (Gaussian Mixture Models)

A Comprehensive Review on Speech Recognition and Its Techniques

Abstract: This paper provides a review on speech recognition system and its techniques. And provide the advancement in the field of speech recognition system. As speech is a way for the communication between the sender and receiver. A speech recognition system takes speech signal as the input and gives the output in the form of text. This paper describes the basic Automatic Speech Recognition (ASR) System. Provide various Speech recognition techniques such as speech analysis, feature extraction techniques, and matching techniques. This paper gives brief description of feature extraction techniques such as Linear Prediction coding (LPC), Mel frequency Cepstral coefficient (MFCC) and Perceptual Linear Predictive (PLP) technique

Empowering Communication: Speech Technology for Indian and Western Accents through AI-powered Speech Synthesis

Neural Text-to-speech (TTS) synthesis is a powerful technology that can generate speech using neural networks. One of the most remarkable features of TTS synthesis is its capability to produce speech in the voice of different speakers. This paper introduces voice cloning and speech synthesis https://pypi.org/project/voice-cloning/ an open-source python package for helping speech disorders to communicate more effectively as well as for professionals seeking to integrate voice cloning or speech synthesis capabilities into their projects. This package aims to generate synthetic speech that sounds like the natural voice of an individual, but it does not replace the natural human voice. The architecture of the system comprises a speaker verification system, a synthesizer, a vocoder, and noise reduction. Speaker verification system trained on a varied set of speakers to achieve optimal generalization performance without relying on transcriptions. Synthesizer is trained using both audio and transcriptions that generate Mel spectrogram from a text and vocoder which converts the generated Mel Spectrogram into corresponding audio signal. Then the audio signal is processed by a noise reduction algorithm to eliminate unwanted noise and enhance speech clarity. The performance of synthesized speech from seen and unseen speakers are then evaluated using subjective and objective evaluation such as Mean Opinion Score (MOS), Gross Pitch Error (GPE), and Spectral distortion (SD). The model can create speech in distinct voices by including speaker characteristics that are chosen randomly

A Review on Human-Computer Interaction and Intelligent Robots

In the field of artificial intelligence, human–computer interaction (HCI) technology and its related intelligent robot technologies are essential and interesting contents of research. From the perspective of software algorithm and hardware system, these above-mentioned technologies study and try to build a natural HCI environment. The purpose of this research is to provide an overview of HCI and intelligent robots. This research highlights the existing technologies of listening, speaking, reading, writing, and other senses, which are widely used in human interaction. Based on these same technologies, this research introduces some intelligent robot systems and platforms. This paper also forecasts some vital challenges of researching HCI and intelligent robots. The authors hope that this work will help researchers in the field to acquire the necessary information and technologies to further conduct more advanced research

Phonetic Dictionary for Natural Language Processing: Kannada

India has 22 officially recognized languages: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Tamil, Telugu, and Urdu. Clearly, India owns the language diversity problem. In the age of Internet, the multiplicity of languages makes it even more necessary to have sophisticated Systems for Natural Language Process. In this paper we are developing the phonetic dictionary for natural language processing particularly for Kannada. Phonetics is the scientific study of speech sounds. Acoustic phonetics studies the physical properties of sounds and provides a language to distinguish one sound from another in quality and quantity. Kannada language is one of the major Dravidian languages of India. The language uses forty nine phonemic letters, divided into three groups: Swaragalu (thirteen letters); Yogavaahakagalu (two letters); and Vyanjanagalu (thirty-four letters), similar to the vowels and consonants of English, respectively

ISOLARED HANDWRITTEN ARABIC NUMERALS RECOGNITION USING THE K- NEAREST NEIGHBOR AND THE HIDDEN MARKOV MODEL CLASSIFIERS

This work deals with a recognition system of handwritten Arabic numeralsextracted to the MNIST standard database (Arabic numerals), this system is composedby three main phases: the preprocessing of numerals followed by the extraction of primitiveswith the zoning method in order to convert each image into a vector number whichis nothing other than an information extracted from this numeral just to differentiatethe others. Finally, our recognition system will end with a classification phase by thetwo methods: the K-nearest neighbours (K-NN) and Hidden Markov Model (HMM).This work has achieved a recognition rate of approximately 82 of success