Neural Networks for Computer-Human Interfaces: Glove-TalkII

Abstract

Glove-TalkII is system which has an adaptive interface built with neural networks. Glove-TalkII maps hand gestures continuously to 10 control parameters of a parallel formant speech synthesizer. The mapping allows the hand to act as an artificial vocal tract that produces speech in real time giving an unlimited vocabulary and unlimited control of fundamental frequency and volume. The best version of Glove-TalkII uses several input devices (including a CyberGlove, a ContactGlove, a 3-space tracker, and a foot-pedal), a parallel formant speech synthesizer and 3 neural networks. The gesture-to-speech task is divided into vowel and consonant production by using a gating network to weight the outputs of a vowel and a consonant neural network. The gating network and the consonant network are trained with examples from the user. The vowel network implements a fixed, user-defined relationship between hand-position and vowel sound and does not require any training examples from the user. Volume, fundamental frequency and stop consonants are produced with a fixed mapping from the input devices. One subject has trained to speak intelligibly with Glove-TalkII. He speaks slowly with speech quality similar to a text-tospeech synthesizer but with far more natural-sounding pitch variations. Characteristics of the neural networks both enhance and detract from control intimacy