AUTOMATIC LIP-READING OF HEARING IMPAIRED PEOPLE

Inability to use speech interfaces greatly limits the deaf and hearing impaired people in the possibility of human-machine interaction. To solve this problem and to increase the accuracy and reliability of the automatic Russian sign language recognition system it is proposed to use lip-reading in addition to hand gestures recognition. Deaf and hearing impaired people use sign language as the main way of communication in everyday life. Sign language is a structured form of hand gestures and lips movements involving visual motions and signs, which is used as a communication system. Since sign language includes not only hand gestures, but also lip movements that mimic vocalized pronunciation, it is of interest to investigate how accurately such a visual speech can be recognized by a lip-reading system, especially considering the fact that the visual speech of hearing impaired people is often characterized with hyper-articulation, which should potentially facilitate its recognition. For this purpose, thesaurus of Russian sign language (TheRusLan) collected in SPIIRAS in 2018–19 was used. The database consists of color optical FullHD video recordings of 13 native Russian sign language signers (11 females and 2 males) from “Pavlovsk boarding school for the hearing impaired”. Each of the signers demonstrated 164 phrases for 5 times. This work covers the initial stages of this research, including data collection, data labeling, region-of-interest detection and methods for informative features extraction. The results of this study can later be used to create assistive technologies for deaf or hearing impaired people

AUTOMATIC DETECTION AND RECOGNITION OF 3D MANUAL GESTURES FOR HUMAN-MACHINE INTERACTION

In this paper, we propose an approach to detect and recognize 3D one-handed gestures for human-machine interaction. The logical structure of the modules of the system for recording a gestural database is described. The logical structure of the database of 3D gestures is presented. Examples of frames showing gestures in the format of Full High Definition, in the map depth mode and in the infrared illustrated. Models of a deep convolutional network for detecting faces and hand shapes are described. The results of automatic detection of the area with the face and the shape of the hand are given. Identified the distinctive features of the gesture at a certain point in time. The process of recognizing 3D one-handed gestures is described. Due to its versatility, this method can be used in tasks of biometrics, computer vision, machine learning, automatic systems of face recognition, sign languages

Неврологические проявления реактивации вируса опоясывающего герпеса

Herpes zoster remains an actual problem for elderly patients and those with immunosuppression. Usually the clinic of Herpes zoster is considered as a combination of neuralgia and typical vesicular rash (sometimes with intoxication). However, the rash does not appear in all patients, and the spectrum of neurological manifestations of the virus reactivation is extremely wide. Most of them reduce the quality of life, and some may result in disability or even death. Due to polymorphic symptoms and difficulties in laboratory diagnosis, many cases of the disease may remain unrecognized, the patient will not receive the necessary etiotropic therapy, so the course of the disease will become more protracted, and the prognosis will worsen. This review is based on 62 domestic and foreign publications without limiting the depth of the search. Here we describe relatively rare forms of the disease with injury of the sensory and motor peripheral nerves, brain and cranial vessels. We offer You a detailed description of pain: herpetic and postherpetic neuralgia. Also in the article we describe examples of damage of motor neurons and give some basic information about damage to the central nervous system. A separate section is devoted to diagnostic methods. This section describes the advantages and disadvantages of diagnostic methods used in real clinical practice, indicates the materials for the study, and the markers of subclinical virus reactivation. The “treatment” section is mainly devoted to the peculiarities of etiotropic therapy for various forms of the disease (with an indication of the recommended drugs, their doses and duration of the course). At the end of the article a list of clinical indications for examination for reactivation of the herpes zoster virus is defined.Опоясывающий герпес остается актуальной проблемой для пожилых пациентов и лиц с иммуносупрессией различного генеза. Традиционно клинику этого заболевания рассматривают как сочетание невралгий и типичных везикулезных высыпаний (иногда с синдромом общей интоксикации). Однако высыпания появляются не у всех больных, а спектр неврологических проявлений реактивации вируса опоясывающего герпеса чрезвычайно широк, причем большинство из них снижают качество жизни, а некоторые могут закончиться инвалидизацией или даже летальным исходом. Из-за полиморфной клиники и трудностей лабораторной диагностики многие случаи болезни могут остаться нераспознанными, пациент не получит этиотропную терапию, вследствие чего течение болезни станет более затяжным, а прогноз ухудшится. Данный обзор выполнен по материалам 62 отечественных и зарубежных публикаций без ограничения глубины поиска. В нём описаны варианты течения болезни с поражением чувствительных и двигательных периферических нервов, центральной нервной системы и сосудов. Дана подробная характеристика болей: герпетических и постгерпетических невралгий. Также в статье приведены примеры поражения мотонейронов и даны основные сведения о поражении центральной нервной системы. Отдельный раздел посвящен актуальным методам диагностики различных форм опоясывающего герпеса. В этом разделе описаны достоинства и недостатки используемых в реальной клинической практике методов диагностики, указаны материалы для исследования, а также маркеры субклинической реактивации вируса. Раздел «Лечение», в основном, посвящен особенностям этиотропной терапии при разных формах болезни (с указанием рекомендуемых препаратов, их доз и длительности курса). Наконец, определен перечень клинических показаний для обследования на предмет реактивации вируса опоясывающего герпеса

IMPROVED AUTOMATIC LIP-READING BASED ON THE EVALUATION OF INTENSITY LEVEL OF SPEAKER’S EMOTION

Automatic audio-visual speech recognition systems (AVSRs) have recently achieved tremendous success, especially in limited vocabulary tasks by far surpassing human abilities to recognize speech, especially in acoustically noisy conditions. Automatic speech recognition systems based on processing of audio and video information are being actively researched and developed all over the world. However, scientific studies aimed at analyzing the influence of the speaker's emotional state (anger, disgust, fear, happy, neutral, and sad), and, most importantly, intensity level of emotion (low - LO, medium - MD, high - HI) on automatic lip-reading have not been conducted. In this regard, the relevance of this research topic cannot be overestimated and requires detailed study. In this paper, we present a novel approach for emotional speech lip-reading, that includes evaluation of a speaker’s emotion and its intensity level. The proposed approach uses visual speech data to detect a person’s emotion type and its intensity level and based on this information assigns it to one of the trained emotional lip-reading models. This essentially resolves the multi-emotional lip-reading issue associated with most real-life scenarios. The proposed approach improves the state-of-the-art results due to the consideration of the intensity of the pronounced audio-visual speech up to 8.2% in terms of the accuracy. Current research is the first step in the creation of emotion-robust speech recognition systems and leaves open a wide field for further research

Норманская теория и становление российской государственности

The authors analyze the question of the origin of Russian statehood in ancient times. The main goal of the research is to study economic and social state of Russia at the beginning of the VIth century. The authors conduct a detailed analysis of the Norman and anti-Norman theories. They conclude assessment of the degree of influence of each of the theories on the formation of Russia as a full-fledged state.В статье рассматривается вопрос зарождения Российской государственности. Изучается экономическое и социальное состояние Руси в начале VI в. Авторами проводится подробный анализ норманской и антинорманской теорий. Дается оценка степени влияния каждой из теорий на дальнейшую концептуализацию формирования Руси как полноценного государства