Design, implementation and evaluation of a dynamic layout of a vision based virtual keyboard

A hands-free text entry system is needed when the typical text entry with hands is not feasible due to the user's physical disability or other limitations. Use of head/face tracking is one of the options to interact with virtual keyboards for hands-free text entry. Performance and usability impacts of the layout of the virtual keyboard used with such hands-free text entry systems have not been studied enough. This thesis introduced a novice layout design of virtual keyboard to be used with face/head input. The aim of this thesis was to check if the performance of the new and traditional layouts will be any different. The new layout was inspired by Fitts law. In the new layout, the size of each key was calculated dynamically in proportion to its distance from the last pressed key. The performance of the new layout was tested against the traditional static QWERTY layout in a user experiment with 16 able bodied participants where each user entered 8 text phrases of approximately 30 characters with each layout. Face tracking was used to control the cursor movement and a key from the physical keyboard was used to enter the selected character. Text entry speed was 5.03 and 5.14 words per minute, error rate was 0.83% and 1.28% for dynamic and static layouts, correspondently. Keystrokes per character was 1.05 with both layouts. Statistical analysis did not show significant differences in the performance of these two layouts. The subjective rating revealed that the participants liked both layouts equally but felt that the dynamic layout requires more mental effort and is less accurate than the static layout. Directions for further improvement of the dynamic layout are suggested as a future work

Interactive form creation: exploring the creation and manipulation of free form through the use of interactive multiple input interface

Most current CAD systems support only the two most common input devices: a mouse and a keyboard that impose a limit to the degree of interaction that a user can have with the system. However, it is not uncommon for users to work together on the same computer during a collaborative task. Beside that, people tend to use both hands to manipulate 3D objects; one hand is used to orient the object while the other hand is used to perform some operation on the object. The same things could be applied to computer modelling in the conceptual phase of the design process. A designer can rotate and position an object with one hand, and manipulate the shape [deform it] with the other hand. Accordingly, the 3D object can be easily and intuitively changed through interactive manipulation of both hands.The research investigates the manipulation and creation of free form geometries through the use of interactive interfaces with multiple input devices. First the creation of the 3D model will be discussed; several different types of models will be illustrated. Furthermore, different tools that allow the user to control the 3D model interactively will be presented. Three experiments were conducted using different interactive interfaces; two bi-manual techniques were compared with the conventional one-handed approach. Finally it will be demonstrated that the use of new and multiple input devices can offer many opportunities for form creation. The problem is that few, if any, systems make it easy for the user or the programmer to use new input devices

Collaborative Deep Reinforcement Learning for Joint Object Search

We examine the problem of joint top-down active search of multiple objects under interaction, e.g., person riding a bicycle, cups held by the table, etc.. Such objects under interaction often can provide contextual cues to each other to facilitate more efficient search. By treating each detector as an agent, we present the first collaborative multi-agent deep reinforcement learning algorithm to learn the optimal policy for joint active object localization, which effectively exploits such beneficial contextual information. We learn inter-agent communication through cross connections with gates between the Q-networks, which is facilitated by a novel multi-agent deep Q-learning algorithm with joint exploitation sampling. We verify our proposed method on multiple object detection benchmarks. Not only does our model help to improve the performance of state-of-the-art active localization models, it also reveals interesting co-detection patterns that are intuitively interpretable

Sensing and mapping for interactive performance

This paper describes a trans-domain mapping (TDM) framework for translating meaningful activities from one creative domain onto another. The multi-disciplinary framework is designed to facilitate an intuitive and non-intrusive interactive multimedia performance interface that offers the users or performers real-time control of multimedia events using their physical movements. It is intended to be a highly dynamic real-time performance tool, sensing and tracking activities and changes, in order to provide interactive multimedia performances. From a straightforward definition of the TDM framework, this paper reports several implementations and multi-disciplinary collaborative projects using the proposed framework, including a motion and colour-sensitive system, a sensor-based system for triggering musical events, and a distributed multimedia server for audio mapping of a real-time face tracker, and discusses different aspects of mapping strategies in their context. Plausible future directions, developments and exploration with the proposed framework, including stage augmenta tion, virtual and augmented reality, which involve sensing and mapping of physical and non-physical changes onto multimedia control events, are discussed

Web-based haptic applications for blind people to create virtual graphs

Haptic technology has great potentials in many applications. This paper introduces our work on delivery haptic information via the Web. A multimodal tool has been developed to allow blind people to create virtual graphs independently. Multimodal interactions in the process of graph creation and exploration are provided by using a low-cost haptic device, the Logitech WingMan Force Feedback Mouse, and Web audio. The Web-based tool also provides blind people with the convenience of receiving information at home. In this paper, we present the development of the tool and evaluation results. Discussions on the issues related to the design of similar Web-based haptic applications are also given

Virtual reality and its role in removing the barriers that turn cognitive impairments into intellectual disability

Early expectations of the contribution that virtual reality (VR) could make to education far exceeded actual applications. This was largely due to the initial immaturity of the technology and a lack of evidence base on which to base design and utilisation. While the early developments in computer based learning largely concentrated on mainstream education, leaving those with special needs behind, the potential of VR as an educational tool was exploited for those with intellectual disabilities right from the start. This paper describes the empirical evidence that has contributed to the development of educational virtual reality for those with intellectual disabilities: studies on transfer of learning from the virtual to the real world; how teachers might support those using VR; the design of virtual environments and what input/control devices best facilitate use of desktop VR. Future developments and ethical issues are also considered