Engineering multimodal interactions through rapid integration of heterogeneous components

Since pioneering studies in the area of Human-Computer Interactions (HCI), there have been leaps in technologies such as graphical user interfaces, input devices, computer vision, speech processing, artificial intelligence, etc. Today we are witnessing a growing interest and a shift from traditional computer interfaces (WIMP) to advanced, entertaining, pervasive and more natural systems (post-WIMP). Due to its cross-disciplinary nature, the design and development of advanced human-computer interactions is not only conceptually but also practically a very challenging task. Seamless reuse and adaptation of the plethora of existing technologies is a key to successful exploration of non-conventional techniques. However, to this date, researchers and practitioners willing to explore new paths, design and implement innovative systems usually have to build ad hoc tools in order to replicate existing results or implement new interaction techniques. Incremental development is hard to achieve mainly because these ad hoc tools have not been thought in a reusable way. This thesis studies how to support HCI researchers and practitioners in their exploration and experimentation activities towards non-conventional human-computer interactions. With this work we strive for supporting innovation by providing methods, software architectures and tools which aim to unify mature techniques from communities such as HCI and Signal Processing (eye-gaze tracking, brain interface, input devices, computer vision, etc.). We propose a multi-fidelity design method based on a unifying component model and supported by an advanced tool suite, the OpenInterface Platform Workbench. This solution allows leveraging of existing resources and fosters creation of non-conventional interaction techniques. More specifically we provide a coarse-grained component-based model and a runtime platform (OIKernel) tailored for seamless reuse and composition of heterogeneous interaction components. Additionally an Eclipse-based design platform (SKEMMI) that supports collaboration of application designers and programmers and centralizes the complete multi-fidelity experimentation process (from design to implementation) is also delivered. The workbench allows exploring novel interaction techniques through seamless integration and adaptation of heterogeneous components, high-fidelity rapid prototyping and runtime evaluation of designed systems.(FSA 3) -- UCL, 201