George, J.

Gnanayutham, Paul

Joumun, F.

English

Name not available

Multimedia Interfaces for BSL Using Lip Readers 
Faramaz Eyasim Joumun1, Paul Gnanayutham2 and Jennifer George3 
1Researcher 
2Department of Computing, University of Portsmouth, Buckingham Building, Lion Terrace,  
Portsmouth PO1 3HE, United Kingdom 
 3SAE Institute, United House, North Road, London, N7 9DP, United Kingdom 
{fara_142@hotmail.com, jennifer.george@sae.edu, paul.gnanayutham@port.ac.uk} 
 
Abstract. This paper deals with mainly the profoundly deaf people who are 
beginning to learn the British Sign Language (BSL) as their first language for 
communication.  These beginners could be children, teenagers or even adults 
who have hearing problems.  There are already quite a few BSL learning 
websites and Lip Reading software available on the market.  However, there is 
no such software that associates the two associated problems although there 
might be BSL users who can benefit from Lip Reading.  This could help them 
not only for communication but also enable them to interact more with people 
with a normal hearing range, who as a matter of fact, form a major part of the 
society we live in.  The artifact aims to be run on most computers making it 
easier for people to access and use it without the need for any additional costly 
features. 
1 Rationale 
The main reason for developing a software program to assist BSL users to learn Lip 
Reading was initiated by sibling of the first author who was born deaf and currently a 
proficient BSL user.  She is currently 15 and attends a specialised educational 
institution in London.  Although she has qualified teachers for teaching the main 
subjects, a few subjects like Science and Art have to be taught in different mainstream 
schools.  This is due to the lack of resources such as science laboratories.  The 
subjects she learns at these schools are taught by teachers without BSL qualification 
and therefore require an interpreter for the deaf pupils. This procedure is exhausting, 
since these children have to follow two teachers simultaneously to understand the 
lessons.  This school offers Lip Reading classes for the students once every week. 
These classes are very limited and the learners are unable to practice any of the 
lessons on their own time.  They neither have any books nor resources to do so.  
Hence, they are not trained enough to Lip Read in real life.  Another major problem 
faced by the deaf in general is their feeling left out within a group of normal-hearing 
people because of the communication and language barrier.  It’s not realistic to get 
every single normal-hearing person to learn BSL and communicate using the sign 
language on a regular basis. 
Multimedia Interfaces for BSL Using Lip Readers      2 
If deaf people could lip read efficiently, problems like the ones faced by students 
such as the researcher’s sister, who has to follow two teachers at the same time for 
one lesson and also being unable to integrate with the majority of other students, 
could be reduced considerably.  Overall, the deaf people as well as the hearing people 
will benefit from this developed software, since this software can be used as a 
teaching tool for both deaf learners and teachers in the main stream schools who do 
not know the BSL language.  The originality of the project lies in the fact that the 
application developed links Lip Reading to BSL.  Since deaf people use BSL as their 
first language for communication, Lip Reading is most helpful for them. It is useful to 
associate these two areas to make the learning process more effective and appropriate 
for this group of disabled personnel.   
2 Introduction   
There are Lip Reading courses available at various educational institutions in the UK, 
especially through organisations for the deaf people community.  The Lip Reading 
students are required to attend classes and interact with their teachers, either on a one-
to-one basis or as a group with the other students.  Despite having the right tutoring in 
their subject, the students may find it quite hard to review their lessons on their own 
outside the classes.  This is because the only resources available for them to do this 
may be books and handouts that have been provided by teachers. As a result of this, 
the Lip Reading students depend on their teachers or other Lip Reading students to 
practice what they have learnt. BSL users with the ability to Lip Read are a very 
limited group in our society.  This means that it is very difficult for the Lip reading 
students to get help from family or friends outside the deaf community.  Technology 
is an integral part of our life and the profoundly deaf people can make use of it as 
efficiently as, if not better, than anybody else.  Therefore, it will indeed be beneficial 
to use technology to assist the BSL users to learn Lip Reading.   
The main aim of this research is to encourage Lip Reading within the deaf 
community, more specifically for BSL users, by providing them with a simple and 
easy-to-use aid that can be utilized by beginners.  Literature search was carried out in 
deafness as a disability and the use of BSL as a language to communicate within the 
deaf people community and Lip Reading, by interacting with the students, teachers 
and any other individuals who could provide useful information relevant to the this 
project. The information to determine the specific requirements has been analysed.  
Similar products or other products related to deafness, BSL or Lip reading have been 
explored.  After identifying exactly what this developed application should do, it was 
designed, implemented and evaluated with two deaf participants.  
Deafness refers to a hearing impairment or loss that affects people’s ability for the 
perception of sound in general.  As a result, the deaf personnel struggle or cannot 
communicate verbally.  There is no age limit associated to deafness.  Babies can be 
born with a hearing impairment just as adults can become deaf after an accident or 
illness. Deafness can be caused by various factors resulting in any kind of damage or 
disruption to any part of the hearing system.  These conditions can be temporary ones 
like earwax, infections or foreign body obstructions that may block the ear canal.  
Multimedia Interfaces for BSL Using Lip Readers      3 
 
Table 1. – Causes of deafness 
(http://www.rnid.org.uk/information_resources/aboutdeafness/causes/other_ear_problems/) 
Table 2. – Levels of deafness 
(http://www.rnid.org.uk/information_resources/aboutdeafness/meaning_of_deafness/) 
 
Other permanent circumstances may be the outcome of inherited conditions like 
for instance having deaf parents, infection that affects women during their pregnancy 
putting their babies at risk to being born deaf or head injury.  According to the RNID 
[1] the main causes for deafness can be categorised as described in Table 1.  There are 
Cause Definition  
 
Age  People are likely to have a deterioration or loss of hearing as they 
grow older.    
Noise  Prolonged exposure to excessive noise can gradually cause hearing 
loss in the long run.   
Obstructions like ear wax that block the ear canal.   
Inflammation caused by viral or bacterial infection, like Otitis 
media (also known as ‘glue ear’).   
The abnormal growth of bone in the middle ear, which prevents the 
ear from functioning properly.  This condition is known as 
‘Otosclerosis’. 
Damaged ‘Ossicles’, which are tiny boned found in the middle ear. 
Other Ear 
Problems 
Perforated Eardrums. 
Genetics   Through heredity and inheritance.   
Deafness Level Difficulty Caused Quietest Sound  
(In decibels) 
Mild Speech, especially in a noisy 
environment. 
25 to 39 
 
Moderate Speech, without a Hearing Aid. 40 to 69 
Severe Unable to communicate verbally. 
Rely on Lip Reading Or British 
Sign Language (BSL) 
70 to 94 
Profound  Unable to communicate verbally. 
Rely on Lip Reading Or British 
Sign Language (BSL) 
95 or higher 
Multimedia Interfaces for BSL Using Lip Readers      4 
various levels of deafness namely, ‘Mild’, ‘Moderate’, ‘Severe’ and ‘Profound’ based 
on the quietest sound they can hear.  These levels have been summarised and 
compared in Table 2. 
The British Sign Language (BSL) is widely used by the deaf people in the United 
Kingdom.  In fact, BSL is their first language for communicating both within and 
outside the deaf community.  Wang and his team [2] describe sign language as the 
primary model of communication among the hearing impaired.  It is a visual language 
that utilises the hand and arm together with facial expressions and lip movements.  
The recognition of sign language aims to provide a competent and precise method of 
interpreting sign language into text or speech for others to understand.   
Waters and his team [3] describe fingerspelling as a manual encoding of the 
written language that can be used by people using sign language to fill lexical gaps.  
BSL users employ a two-hand fingerspelling that includes 26 hand arrangements 
based on the 26 letters of the Roman alphabet.  Hearing people are able to spell 
unfamiliar words by uttering the alphabets that make up the words.  Similarly, deaf 
people utilise the manual alphabets to “spell-out” unfamiliar words.  Thus, 
fingerspelling encodes the writing structure of a language.   
Lip Reading, also referred to as Speechreading, is a method of identifying words 
and/or alphabets by visually translating lip, face and tongue movements to decipher 
communication from other people.  Lay and his team [4] state that Lip Reading is 
used on a regular basis by the normal-hearing, especially in noisy environments, and 
also the hearing impaired even though it is not entirely reliable and accurate.  Apart 
from using hearing aids, the hearing impaired and deaf people will find learning Lip 
Reading essentially useful to improve their communication with other people.  Using 
hearing aids or lip recognition to assist them in communication, the hearing impaired 
recognise 65% of the words.  On the other hand, if they use hearing aids and Lip 
Reading simultaneously, the recognition rate of these words raise to 90%.   
Bauman [5] said that during normal speech about 13 to 15 speech movements are 
made per second but the human eyes can pick up only 8 or 9 of these movements.  
Many English words bear the same movements on the speaker’s lips despite having a 
different sound.  These words are known as homophenes. This means that one is 
unable to distinguish the homophenes unless they can hear the sounds.  An example is 
‘Queen’ and ‘White’.  10% of the population have lip movements that are totally 
impossible to Lip Read while others are simply unable to Speechread.  Statistics show 
that only 23% of the hearing impaired are able to Speechread effectively.   It has also 
been proven that women are better Speechreaders compared to men.  Another 
drawback of Lip Reading is the fact that the process can be very tiring, making it very 
hard to have a smooth and easy conversation.  All these facts make Lip Reading or 
Speechreading impossible to be a perfect method of deciphering words.  
However, these limitations can be handled by employing measures like requesting 
the speaker to talk slowly or allow adequate pauses between words.  It also helps to 
make the lip reader aware of the context and subject of discussion beforehand.  
Multimedia Interfaces for BSL Using Lip Readers      5 
3 Methodology 
The spiral methodology was used for the development for the artifact developed for 
this research. The Spiral methodology is quite similar to the Waterfall methodology 
but also allows additional concepts like prototyping and iteration to take place.  It also 
requires documentation to be presented at each stage during the project life cycle.  
The choice of the Spiral methodology was justified by the fact that the project dealt 
with a group, termed ‘Deaf community using BSL as their first language’. Due to user 
frustration, prototypes were tested with normal hearing participants prior to being 
tested with the deaf people. This meant that the stages need to be re-iterated a number 
of times.  The Spiral methodology met these requirements appropriately.   
4 Design and Development 
Both technical and usability needs were considered and the artefact was tested 
continuously using able and disabled users. The interface consisted of a welcome 
screen, the personalisation screen, the options screen, the lip movement screens and 
the help screen (Figures 1 and 2). Starting from the welcome screen, which has an 
extremely simple layout, the users have the option to click on a ‘Personalise 
Application Settings’ button that will allow them to select a specific:  
  Text Font 
  Text Size 
  Text Style 
  Text Colour  
  Background Colour 
Once they have changed these settings they should be able to save these new 
settings and carry on using the application with the new changes being effective. 
The Options screen is the main screen that gives the users an opportunity to select 
one among the three options available for them to see the lip movements. The user 
will be able to use the application by selecting OPTION 1 – Lip Movements through 
Alphabets.  Similarly, deaf people who are more comfortable with BSL may be prefer 
OPTION 2 – Lip Movements through BSL.  OPTION 3 – Lip Movements through 
Alphabets and BSL allows the users to combine both OPTION 1 and OPTION 2 to 
the Lip Movements.  
 
Multimedia Interfaces for BSL Using Lip Readers      6 
 
Fig. 1. – Options page 
Similarly to the ‘Personalise Application Settings’ button, the ‘Help’ button is 
included at a consistent spot on all the screens of the software.  When clicked, the 
‘Help’ button brings up a very simple and concise screen that will allow the users to 
get some kind of assistance on how they can make use of the application more 
effectively. 
5 Testing 
After two students with hearing impairment used the artefact, qualitative and 
quantitative feedback was obtained. They were questioned on whether they liked the 
artefact, whether they wanted to use it again, what score they would rate it in a scale 
of 1-10 and if there was anything they didn’t like about it.  Their reply was rather 
positive as they said they did like the website, they would like to use it again and 
rated it as 8 and 6 respectively.   
The meetings with the students were not planned and were informal.  Also, the 
participants were rather shy and were a bit hesitant to answer questions directly.  The 
fact that there was no interpreter present, again because the meeting was not planned, 
the communication was rather difficult because they mainly use BSL to communicate 
and the researchers were not proficient at BSL. Overall they were satisfied with the 
artifact and admitted that it was the first time they had encountered such a product.   
 
Multimedia Interfaces for BSL Using Lip Readers      7 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Fig. 2. –Personalise Settings Page 
The interface was tested with these two participants, the results of the testing is 
given below both participants were able to use the developed interface is 100% 
success as the results show in Table 3 
 
Participant Number Learn Lip Reading 
Through Alphabets 
Learn Lip Reading Through 
Alphabets and BSL  
1 Yes Yes 
2 Yes Yes 
Table 3. – Evaluation with deaf participants 
6 Conclusion  
Overall, the developed artefact ‘Lip Reading Assistive Technology for BSL Users’ 
was a success.  This is because it not only met the specified requirements but also was 
an innovation for BSL users to learn Lip Reading. Being an innovation may also 
entail certain limitations as the users, especially the target audience, may be rather 
reluctant to use it initially.  The fact that the BSL users who assisted in testing this 
artifact had to ask questions on how to use the artifact or pause to figure out how to 
use this artifact showed that there was further work to be done in this area of this 
research.  However, their appreciation of the lip movements associated with BSL was 
promising.  They had never encountered such a product before and yet found it rather 
interesting to use.   
Multimedia Interfaces for BSL Using Lip Readers      8 
References  
1. RNID - http://www.rnid.org.uk/information_resources/aboutdeafness/causes/, 
retrieved on 10th September 2007  
2. Qi Wang, Xilin Chen, Liang-Guo Zhang, Chunli Wang and Wen Gao - Viewpoint 
invariant sign language recognition. Computer Vision and Image 
Understanding, Volume 108, Issues 1-2, October-November 2007, Pages 87-97, also 
available at OCG http://www.ogc.gov.uk/methods_prince_2.asp, accessed on 14th 
September 2007. Accessed on 15th September 2007  
3. Dafydd Waters, Ruth Campbell, Cheryl M. Capek, Bencie Woll, Anthony S. David, 
Philip K. McGuire, Michael J. Brammer and Mairéad MacSweeney - Fingerspelling, 
signed language, text and picture processing in deaf native signers: The role of the 
mid-fusiform gyrus NeuroImage, Volume 35, Issue 3, 15 April 2007, Pages 1287-
1302, retrieved on 15th September 2007 
4. Yun-Long Lay, Chung-Ho Tsai, Hui-Jen Yang, Chern-Sheng Lin and Chuan-Zhao 
Lai - The application of extension neuro-network on computer-assisted lip-reading 
recognition for hearing impaired. Expert Systems with Applications, In Press,  
5. Bauman, 2000, http://www.hearinglosshelp.com/articles/speechreading.htm, accessed 
on 12th September 2007 
 


Multimedia Interfaces for BSL Using Lip Readers

Portsmouth University Research Portal (Pure)

Brammer and Mairéad MacSweeney - Fingerspelling, signed language, text and picture processing in deaf native signers:

Chung-Ho Tsai, Hui-Jen Yang, Chern-Sheng Lin and Chuan-Zhao Lai - The application of extension neuro-network on computer-assisted lip-reading recognition for hearing impaired. Expert Systems with Applications,

http://www.hearinglosshelp.com/articles/speechreading.htm, accessed on 12th

Liang-Guo Zhang, Chunli Wang and Wen Gao - Viewpoint invariant sign language recognition. Computer Vision and Image Understanding, Volume 108, Issues 1-2, October-November

https://researchportal.port.ac.uk/portal/services/downloadRegister/134534/BSL.pdf

Multimedia Interfaces for BSL Using Lip Readers

Abstract

Similar works

Full text

Available Versions

Name not available

Portsmouth University Research Portal (Pure)