Vision-Based Finger Detection, Tracking, and Event Identification Techniques for Multi-Touch Sensing and Display Systems

This study presents efficient vision-based finger detection, tracking, and event identification techniques and a low-cost hardware framework for multi-touch sensing and display applications. The proposed approach uses a fast bright-blob segmentation process based on automatic multilevel histogram thresholding to extract the pixels of touch blobs obtained from scattered infrared lights captured by a video camera. The advantage of this automatic multilevel thresholding approach is its robustness and adaptability when dealing with various ambient lighting conditions and spurious infrared noises. To extract the connected components of these touch blobs, a connected-component analysis procedure is applied to the bright pixels acquired by the previous stage. After extracting the touch blobs from each of the captured image frames, a blob tracking and event recognition process analyzes the spatial and temporal information of these touch blobs from consecutive frames to determine the possible touch events and actions performed by users. This process also refines the detection results and corrects for errors and occlusions caused by noise and errors during the blob extraction process. The proposed blob tracking and touch event recognition process includes two phases. First, the phase of blob tracking associates the motion correspondence of blobs in succeeding frames by analyzing their spatial and temporal features. The touch event recognition process can identify meaningful touch events based on the motion information of touch blobs, such as finger moving, rotating, pressing, hovering, and clicking actions. Experimental results demonstrate that the proposed vision-based finger detection, tracking, and event identification system is feasible and effective for multi-touch sensing applications in various operational environments and conditions

An Abstraction Framework for Tangible Interactive Surfaces

This cumulative dissertation discusses - by the example of four subsequent publications - the various layers of a tangible interaction framework, which has been developed in conjunction with an electronic musical instrument with a tabletop tangible user interface. Based on the experiences that have been collected during the design and implementation of that particular musical application, this research mainly concentrates on the definition of a general-purpose abstraction model for the encapsulation of physical interface components that are commonly employed in the context of an interactive surface environment. Along with a detailed description of the underlying abstraction model, this dissertation also describes an actual implementation in the form of a detailed protocol syntax, which constitutes the common element of a distributed architecture for the construction of surface-based tangible user interfaces. The initial implementation of the presented abstraction model within an actual application toolkit is comprised of the TUIO protocol and the related computer-vision based object and multi-touch tracking software reacTIVision, along with its principal application within the Reactable synthesizer. The dissertation concludes with an evaluation and extension of the initial TUIO model, by presenting TUIO2 - a next generation abstraction model designed for a more comprehensive range of tangible interaction platforms and related application scenarios

A Coarse imaging sensor for detecting embedded signals in infrared light

Machine vision technology has become prevalent in touch technology, however, it is still limited by background noise. To reduce the background noise present in the images of interest it is important to consider the imaging device and the signal source. The architecture, size, sampling scheme, programming, and technology of the imaging device must be considered as well as the response characteristics of the signal source. Several pixel architectures are explained and implemented with discrete components. Their performance was measured through their ability to track a modulated signal source. Potentially, an imaging sensor comprised of a system designed to modulate the light to be imaged could drastically reduce background noise. Further, with a less noisy image, the processing steps required for touch event detection may be simplified

Phalanger: controlling music software with hand movement using a computer vision and machine learning approach

Phalanger is a system which facilitates the control of music software with hand and finger motion, with the aim of creating a fluid style of interaction that promotes musicality. The system is purely video based, requires no wearables or accessories and uses affordable and accessible technology. It employs a neural network for background segmentation, a combination of imaging techniques for frame analysis, and a support vector machine (SVM) for recognition of hand positions. System evaluation showed the SVM to reliably differentiate between eight different classes. An initial formative user evaluation with ten musicians was carried out to help build a picture of how users responded to the system; this highlighted areas that need improvement and lent some insight into useful features for the next version

A homography-based multiple-camera person-tracking algorithm

It is easy to install multiple inexpensive video surveillance cameras around an area. However, multiple-camera tracking is still a developing field. Surveillance products that can be produced with multiple video cameras include camera cueing, wide-area traffic analysis, tracking in the presence of occlusions, and tracking with in-scene entrances. All of these products require solving the consistent labelling problem. This means giving the same meta-target tracking label to all projections of a realworld target in the various cameras. This thesis covers the implementation and testing of a multiple-camera peopletracking algorithm. First, a shape-matching single-camera tracking algorithm was partially re-implemented so that it worked on test videos. The outputs of the single-camera trackers are the inputs of the multiple-camera tracker. The algorithm finds the feet feature of each target: a pixel corresponding to a point on a ground plane directly below the target. Field of view lines are found and used to create initial meta-target associations. Meta-targets then drop a series of markers as they move, and from these a homography is calculated. The homographybased tracker then refines the list of meta-targets and creates new meta-targets as required. Testing shows that the algorithm solves the consistent labelling problem and requires few edge events as part of the learning process. The homography-based matcher was shown to completely overcome partial and full target occlusions in one of a pair of cameras

Development platform for elderly-oriented tabletop games

Tese de mestrado integrado. Engenharia Informática e Computação. Universidade do Porto. Faculdade de Engenharia. 201