This project is designed to help the visually
impaired become a part of modern forms of communication. In
this paper, we will present a hands-free device that is meant to be
wirelessly connected to a smartphone through an accompanying
app which will provide access to all forms of messengers and
enable the user to send, listen, respond to a text message, etc.
These many options have been made possible by combining the
movement of different fingers, without the need of any kind of
buttons or keyboards. The system is based on accelerometers and
Morse code. It's easy to use, discreet and it also provides privacy,
opposed to the voice input. So far, we have made a non-wireless
device and programmed and tested it in LabView and Matlab

Aranđelović, Nevena

Popović-Maneski, L.

machinery

   Abstract—This project is designed to help the visually impaired become a part of modern forms of communication. In this paper, we will present a hands-free device that is meant to be wirelessly connected to a smartphone through an accompanying app which will provide access to all forms of messengers and enable the user to send, listen, respond to a text message, etc.  These many options have been made possible by combining the movement of different fingers, without the need of any kind of buttons or keyboards. The system is based on accelerometers and Morse code. It's easy to use, discreet and it also provides privacy, opposed to the voice input. So far, we have made a non-wireless device and programmed and tested it in LabView and Matlab.  Index Terms—visually impaired, text messaging, Morse code, accelerometers I. INTRODUCTION Access to information for the visually-impaired is a growing issue in modern society. Proliferation of touch screens increased the gap between able-bodied and visually-disabled in participation in every-day social activities, especially among the youth. Plenty of people who are blind or with low vision use special apps to type messages into their phones (mainly developed and supported for use with iPhone). The standard app that comes with iPhone is  VoiceOver [1], a gesture-based screen reader which echoes each character on the keyboard as user touches it, and again when user enters it. Many users have found this app cumbersome and slow for text messaging. It is far more easier to use one of the available speech recognizers (e.g. for Android platform [2]); however, the state of the art performance for speech recognizers is limited to a certain number of words and phrases. Moreover, these algorithms are mainly developed for the English language, and perform poorly in cases of noise or distorted speech (e.g. in persons with speech difficulties). On top of all these drawbacks, speech recognizers don't allow privacy as anyone can hear what the user wants to type. One may say that the easiest solution is to simply calling instead of messaging. However, we live in an era of text messaging. Most of the interpersonal information flow occurs on social networks (e.g. Facebook, Twitter, etc.) by posting or group messaging, making it difficult for the visually impaired to stay up to date and participate in conversations. Therefore, the need for a method for easy, fast and reliable sight-free text input to a phone, PC or other devices is becoming highly  Nevena Aranđelović is student with the School of Mechanical Engineering, University of Belgrade, 16 Kraljice Marije, 11020 Belgrade, Serbia (e-mail: n.arandjelovic@gmail.com).  Lana Popović-Maneski is with the Institute of Technical Sciences of the Serbian Academy of Sciences and Art, Knez Mihailova 35/IV, 11000 Belgrade, Serbia (e-mail: lanapm13@gmail.com). important. There are number of options allowing direct or wireless connection of peripheries such as classical or Braille keyboard [3]. iPhone support recently released new app called BrailleTouch [4] which allows sight-free typing of 30 words or more per minute with Braille code. This app requires a user to hold the phone with both hands having three fingers of each hand on the screen. One of the most talked about apps is PocketSMS, developed for Android platform, by Anmol Anand at Bapsi, primarily for the deafblind [5]. When a message is received, the reading section can be activated. It translates text to Morse code and uses vibrations to communicate it to the user. Then, a further action can be chosen by typing reserved combinations of characters or pressing the screen with different durations. The app enables the user to send a new message or reply to a received SMS, also in Morse code. But, besides other obvious deficiencies, only the last message can be accessed. Sight-free phone texting is not only required for the visually impaired but also could be useful for normal sighted in occasions  when they don't want to be seen typing, or e.g. driving. Both of these situations require use of a single hand for texting. All of these requirements led us to the development of a new solution that allows simple, reliable and fast texting with three accelerometers placed on one hand's fingertips, based on Morse code.  II. METHOD Three triple axis accelerometers (MMA7260, sensitivity 600 mv/g) were used to register finger movements. They were connected to a computer through a National Instruments Data Acquisition device (NI USB-6008) with sampling frequency set to 1kHz. The program was written in LabView.  Sensors were placed to the dorsal sides of the fingers in the small elastic strap (Fig. 1). Only z-axes were used to detect finger movements.   Fig. 1.  Three accelerometers placed in elastic straps Text messaging for the visually impaired Nevena Aranđelović and Lana Popović-Maneski     Proceedings of 2nd International Conference on Electrical, Electronic and Computing Engineering IcETRAN 2015, Silver Lake, Serbia, June 8 – 11, 2015, ISBN 978-86-80509-71-6     pp. MEI1.3.1-4 Based on the dexterity level of different fingers and the methods used for manipulating the signals, we recommend for the accelerometers to be placed on the index, middle and little finger. The ring finger follows the movement of the middle finger, while its own movement causes twitching of surrounding fingers. We avoided placement of the accelerometer on the thumb,  as thumb can be conveniently used as twitch signal amplifier during impact with opposed fingers which have sensors on them. However, different configurations can be used based on user's preferences. Based on measurements in 10 healthy volunteers, we determined that the minimal time between two movements of the same type is greater than 120 ms. We used this time window as an indicator of involuntary movements (when the time between movements of the same finger is less than 120 ms). Also, we noticed that the average time difference between the simultaneous movement of index and middle finger is 20 ms. Consequently, if the movements of the middle and index finger are detected within a time window of 30 ms, the movement is recognized as simultaneous. If at least two simultaneous movements occur during 300 ms, the movement is recognized as double simultaneous movement. These values were not changed during the experiments, but in general, the user is allowed to change them based on his/her own preferences.    The index and middle finger are used for typing dashes and dots. Simultaneous movement of the index and middle finger results in accepting the previously typed combination of dashes and dots, and placing them in the growing word. Each time the letter is placed in the word the indicator "letter" shows the letter represented by that combination of dashes and dots. In the version of the program for the visually impaired, the indicator is replaced by the voice feedback pronouncing the corresponding letter. Double simultaneous movement of index and middle finger results in entering a space to separate words in the growing sentence. The little finger's movement is used as a backspace.  There is also a sound feedback (250 ms long 1000 Hz sine, creates quite an irritating sound) if there is an attempt to exceed the maximum number of signs (6), and for non-existent combinations of characters. Different sound (250 ms long 400 Hz sine) is used as a notification when a letter is being deleted.  Thus far, the device has been tested by normal sighted individuals with a hand opposed to the table. User interface for the experiments is shown in Fig.3. The key goal was to test the responsiveness of the software, the average percentage of errors made by the test subjects and the general ability of the subjects to adapt and get used to typing in Morse code. The experiment consisted of three phases: Exercise phase: The subject was provided with the table of The International Morse Code and allowed to test the program by typing and deleting the words of choice for 1 minute. Then, the subject was asked to type all the letters from the English alphabet and 10 numbers. This phase was repeated one more time. Test phase I: The subject was provided with a sentence (“Anything worth doing is worth doing right.”) written on the piece of paper in both, Morse code and the alphabet, one over another, to prevent confusion. The aim was to mimic the typing speed of someone who uses the system for the first time, but has good knowledge of Morse code.  Fig. 2.  The International Morse Code as shown to the test subjects.    Fig. 3.  The user interface designed for normal sighted test subjects and the sentence task in Test phase II.   The task was to retype the text as fast as possible by using the accelerometer system on the fingers. Test phase II: Typing a sentence shown in Fig.4 (different from the sentence in phase one), written only in alphabet, with assistance of a table of The International Morse Code. The text was  designed to have many repeating words and letters, to test the ability of the subject to remember the combinations of characters and to measure the increase of the typing speed. III. RESULTS The experiment was performed with 10 random individuals with normal vision, aged 24±7. An example of recorded signal, high pass filtered by 7rd order Butterworth filter at 2 Hz, is shown in Fig.4. During the exercise phase, test subjects have shown great interest in the system, but in the first few minutes they had some problems with adapting to it. They mixed the functions of different fingers and made mistakes. Three of them were making pushing movements, as if pressing buttons. Such slow and soft movements didn't provide signals with peaks which exceeded the threshold we used. It was enough to touch the table for an instant, like typing on standard keyboards.   Fig. 4. One segment of the signal recorded from three accelerometers (high-pass filtered at 2Hz) and detected commands. Thin blue line- acceleration of index finger, thin red line - acceleration of middle finger, thin black line - acceleration of little finger, thick blue line - detected dash, thick red line - detected dot, thick black line -  detected space, thick green line - detected backspace, thick yellow line - detected send command.   After a couple of minutes, they completely adapted to the system, but some of them complained afterwards of not being able to make double simultaneous movement.  Even though they had elastic straps around their fingertips and wires on the back of their hand (as shown in Fig.1) they were able to perform every other action (such as grabbing, scratching etc.), without any problems. During the first test phase, we were able to calculate the average time needed by a beginner to type a complex sentence.  The average time was 1.83 minutes for 42 character (>22 char/min). The best time was 1.17 minutes (>35 char/min) and the worst 3.48 minutes. The average number of mistakes (mistyped characters) was 7.5. These mistakes usually occurred when subjects accidentally skipped a letter or rushed and typed more characters than they intended to. Also, they were slowed down by constantly looking from the paper to the interface. TABLE I ELAPSED TYPING TIME [MIN]  Test subject No. Test phase I Test phase II 1 2.23 3.21 2 2.46 3.96 3 3.48 4.21 4 1.25 1.62 5 1.41 2.08 6 1.67 1.94 7 2.09 4.38 8 1.32 1.55 9 1.17 1.52 10 1.22 1.47 Average 1.83 2.59  During the second test phase, we were able to see that they were typing very fast while typing the repeating words, but, in general, they had trouble with finding the letters in The International Morse Code Table and that slowed them down. The average time was 2.59 minutes for 71 characters (>27 char/min). One test subject remembered most of the character combinations during the previous phases and had the best score: 1.47 minutes (>48 char/min). The average number of mistakes was 13. The easiest way to explain the mistake that occurred most often during this phase is by giving an example: if the combination was – . – (letter K) the subjects typed . – . (letter R). In general, the relation between typing speed and number of mistyped characters was proportional, but the overall typing speed increased in test phase II.  TABLE II NUMBER OF MISTYPED CHARACTERS  Test subject No. Test phase I Test phase II 1 15 20 2 9 13 3 17 16 4 2 5 5 3 9 6 5 18 7 7 14 8 7 11 9 6 15 10 4 5 Average 7.5 13   Some of the test subjects tried the method of using the thumb as a signal amplifier. We noticed that it was a lot easier with the third accelerometer placed on the ring finger. It was obvious that this kind of movement comes more naturally, but the typing speed was drastically decreased compared to "typing" on the table.  The program was designed in a way that prevents involuntary typing, and that kind of mistake hasn’t been noticed. But, sometimes it happened that a person rushed to type three dots or dashes in a row and moved finger more than once in 150 ms, so the movement wasn't recognized. They usually didn't understand the problem and started typing aggressively with a result of exceeding the maximum number of characters, etc.   Also, some of them had a problem with double simultaneous movement. They typed either too slow, or too fast and barely lifted the fingers. To the naked eye, it seemed as if they couldn’t catch the required rhythm. So far, the involuntary simultaneous movement of the index and middle finger in less than 30 ms has never happened. IV. DISCUSSION In test phase II, even though the subjects had to look for the adequate character representation in the Morse code table, the typing speed increased from 22 to 27 characters/minute. This led us to the conclusion that the system performance partly depends on the abilities of the user but mostly depends on the exercise time. In the future we plan on turning completely to sound feedback, such as different sounds for dots and dashes, or pronouncing entered letters and words. We plan on making a smart phone app which will be able to connect to Facebook Messenger, WhatsApp, etc. Functions of accelerometers will change, adapting to the current environment. For example, while choosing the messenger or whether to reply or forward the message.  The system will connect to smart phone by Bluetooth. We will build the apps for all current platforms. V. CONCLUSION The presented system is simple and easy to use in various occasions, even concurrently with other everyday activities (e.g. walking with a white cane) as it requires use of a single hand regardless of the environment (e.g. typing in the air or typing on the steering wheel while driving for normal sighted). It is privacy friendly as it doesn't require voice input. The algorithm is robust and prevents detections of false positives. Is has also shown its faults (e.g. waiting time between two entries), however it is left to user personal preferences to manipulate the two time windows for optimal performance. Although the system requires the user to learn Morse code, as most of the test subjects pointed out as its main fault, this program is primarily intended to be used by the visually impaired. We count on their motivation to ease their access to electronic devices and become more independent.  ACKNOWLEDGMENT This research was partly supported by Ministry of Education, Science and Technological Development of Serbia (Contract No: III 44008). REFERENCES [1] http://www.apple.com/accessibility/ios/voiceover (accessed 11/05/2015) [2] https://play.google.com/store/search?q=speech%20recognizer  (accessed 11/05/2015) [3] http://braillebug.afb.org/braille_technology.asp (accessed 11/05/2015) [4] http://brailletouchapp.com/ (accessed 11/05/2015) [5] http://m.tech.firstpost.com/news-analysis/pocketsms-app-for-the-deafblind-35178.html (accessed 11/05/2015) 

Text messaging for visually impaired

Machinery - Repository of the Faculty of Mechanical Engineering, University of Belgrade

http://machinery.mas.bg.ac.rs/bitstream/id/11177/bitstream_11177.pdf

Text messaging for visually impaired

Abstract

Similar works

Full text

Available Versions

machinery

Machinery - Repository of the Faculty of Mechanical Engineering, University of Belgrade