Towards a new approach for integrating multimodal user input based on evolutionary computation

Multimodal interfaces provide flexible, intuitive, and errorrobust interaction with complex information systems. In this work we describe an innovative statistical approach for combining multimodal user input that is based on principles adopted from evolution theory. A population of individuals, each representing a solution to the integration problem, compete for an optimal interpretation of the user interactions. Specially designed genetic operators recombine various characteristics of these solutions. The fitness of a single individual, measuring the certainty and the confidence of an integration result, is calculated according to a weighted scheme including the various information resources and the current system context. Our integration algorithm works extremely robust. Moreover, it can easily be scaled up to additional input devices and various application domains. 1