Vision-Based Finger Detection, Tracking, and Event Identification Techniques for Multi-Touch Sensing and Display Systems

This study presents efficient vision-based finger detection, tracking, and event identification techniques and a low-cost hardware framework for multi-touch sensing and display applications. The proposed approach uses a fast bright-blob segmentation process based on automatic multilevel histogram thresholding to extract the pixels of touch blobs obtained from scattered infrared lights captured by a video camera. The advantage of this automatic multilevel thresholding approach is its robustness and adaptability when dealing with various ambient lighting conditions and spurious infrared noises. To extract the connected components of these touch blobs, a connected-component analysis procedure is applied to the bright pixels acquired by the previous stage. After extracting the touch blobs from each of the captured image frames, a blob tracking and event recognition process analyzes the spatial and temporal information of these touch blobs from consecutive frames to determine the possible touch events and actions performed by users. This process also refines the detection results and corrects for errors and occlusions caused by noise and errors during the blob extraction process. The proposed blob tracking and touch event recognition process includes two phases. First, the phase of blob tracking associates the motion correspondence of blobs in succeeding frames by analyzing their spatial and temporal features. The touch event recognition process can identify meaningful touch events based on the motion information of touch blobs, such as finger moving, rotating, pressing, hovering, and clicking actions. Experimental results demonstrate that the proposed vision-based finger detection, tracking, and event identification system is feasible and effective for multi-touch sensing applications in various operational environments and conditions

InDepth: Real-time Depth Inpainting for Mobile Augmented Reality

Peer reviewe

Building smart cameras on mobile tablets for hand gesture recognition

Mobile tablets have become very popular due to their portability and the wide diversity of the applications available. The touch screens and built-in accelerometers facilitate different forms of input instead of keyboards and mice. Nowadays, high-resolution cameras become a standard feature in a mobile device. Nevertheless, this camera is seldom considered to serve as a form of user inputs to the application, although similar technology is realized in some home entertainment systems. This paper describes our experience on making a smart camera on an iPad that can recognize pre-defined hand gestures. We study the time performance of performing image processing on an iPad. We find that due to the limited computational power of the mobile device, recognition results may not be available fast enough for a real-time application. We explore applying cloud computing to solve the problem. To the best of our knowledge, this is the first study on recognizing hand gestures on an iPad. Our results facilitate the development of a brand new type of applications that require smart cameras. © 2012 ACM.published_or_final_versio

An annotated bibligraphy of multisensor integration

technical reportIn this paper we give an annotated bibliography of the multisensor integration literature

Image Understanding and Robotics Research at Columbia University

The research investigations of the Vision/Robotics Laboratory at Columbia University reflect the diversity of interests of its four faculty members, two staff programmers and 15 Ph.D. students. Several of the projects involve either a visiting computer science post-doc, other faculty members in the department or the university, or researchers at AT&T Bell Laboratories or Philips laboratories. We list below a summary of our interest and results, together with the principal researchers associated with them. Since it is difficult to separate those aspects of robotic research that are purely visual from those that are vision-like (for example, tactile sensing) or vision-related (for example, integrated vision-robotic systems), we have listed all robotic research that is not purely manipulative

Flexible Electronics Sensors for Tactile Multi-Touching

Flexible electronics sensors for tactile applications in multi-touch sensing and large scale manufacturing were designed and fabricated. The sensors are based on polyimide substrates, with thixotropy materials used to print organic resistances and a bump on the top polyimide layer. The gap between the bottom electrode layer and the resistance layer provides a buffer distance to reduce erroneous contact during large bending. Experimental results show that the top membrane with a bump protrusion and a resistance layer had a large deflection and a quick sensitive response. The bump and resistance layer provided a concentrated von Mises stress force and inertial force on the top membrane center. When the top membrane had no bump, it had a transient response delay time and took longer to reach steady-state. For printing thick structures of flexible electronics sensors, diffusion effects and dimensional shrinkages can be improved by using a paste material with a high viscosity. Linear algorithm matrixes with Gaussian elimination and control system scanning were used for multi-touch detection. Flexible electronics sensors were printed with a resistance thickness of about 32 μm and a bump thickness of about 0.2 mm. Feasibility studies show that printing technology is appropriate for large scale manufacturing, producing sensors at a low cost

Light on horizontal interactive surfaces: Input space for tabletop computing

In the last 25 years we have witnessed the rise and growth of interactive tabletop research, both in academic and in industrial settings. The rising demand for the digital support of human activities motivated the need to bring computational power to table surfaces. In this article, we review the state of the art of tabletop computing, highlighting core aspects that frame the input space of interactive tabletops: (a) developments in hardware technologies that have caused the proliferation of interactive horizontal surfaces and (b) issues related to new classes of interaction modalities (multitouch, tangible, and touchless). A classification is presented that aims to give a detailed view of the current development of this research area and define opportunities and challenges for novel touch- and gesture-based interactions between the human and the surrounding computational environment. © 2014 ACM.This work has been funded by Integra (Amper Sistemas and CDTI, Spanish Ministry of Science and Innovation) and TIPEx (TIN2010-19859-C03-01) projects and Programa de Becas y Ayudas para la Realización de Estudios Oficiales de Máster y Doctorado en la Universidad Carlos III de Madrid, 2010