RGB-D datasets using microsoft kinect or similar sensors: a survey

RGB-D data has turned out to be a very useful representation of an indoor scene for solving fundamental computer vision problems. It takes the advantages of the color image that provides appearance information of an object and also the depth image that is immune to the variations in color, illumination, rotation angle and scale. With the invention of the low-cost Microsoft Kinect sensor, which was initially used for gaming and later became a popular device for computer vision, high quality RGB-D data can be acquired easily. In recent years, more and more RGB-D image/video datasets dedicated to various applications have become available, which are of great importance to benchmark the state-of-the-art. In this paper, we systematically survey popular RGB-D datasets for different applications including object recognition, scene classification, hand gesture recognition, 3D-simultaneous localization and mapping, and pose estimation. We provide the insights into the characteristics of each important dataset, and compare the popularity and the difficulty of those datasets. Overall, the main goal of this survey is to give a comprehensive description about the available RGB-D datasets and thus to guide researchers in the selection of suitable datasets for evaluating their algorithms

A Software Development Kit for Camera-Based Gesture Interaction

Human-Computer Interaction is a rapidly expanding field, in which new implementations of ideas are consistently being released. In recent years, much of the concentration in this field has been on gesture-based control, either touch-based or camera-based. Even though camera-based gesture recognition was previously seen more in science fiction than in reality, this method of interaction is rising in popularity. There are a number of devices readily available to the average consumer that are designed to support this type of input, including the popular Microsoft Kinect and Leap Motion devices. Despite this rise in availability and popularity, development for these devices is currently an arduous task, unless only the most simple of gestures is required. The goal of this thesis is to develop a Software Development Kit (SDK) with which developers can more easily develop interfaces that utilize gesture-based control. If successful, this SDK could significantly reduce the amount of work (both in effort and in lines of code) necessary for a programmer to implement gesture control in an application. This, in turn, could help reduce the intellectual barrier which many face when attempting to implement a new interface. The developed SDK has three main goals. The SDK will place an emphasis on simplicity of code for developers using it; will allow for a variety of gestures, including gestures made by single or multiple trackable objects (e.g., hands and fingers), gestures performed in stages, and continuously-updating gestures; and will be device-agnostic, in that it will not be written exclusively for a single device. The thesis presents the results of a system validation study that suggests all of these goals have been met

Web GIS in practice X: a Microsoft Kinect natural user interface for Google Earth navigation

This paper covers the use of depth sensors such as Microsoft Kinect and ASUS Xtion to provide a natural user interface (NUI) for controlling 3-D (three-dimensional) virtual globes such as Google Earth (including its Street View mode), Bing Maps 3D, and NASA World Wind. The paper introduces the Microsoft Kinect device, briefly describing how it works (the underlying technology by PrimeSense), as well as its market uptake and application potential beyond its original intended purpose as a home entertainment and video game controller. The different software drivers available for connecting the Kinect device to a PC (Personal Computer) are also covered, and their comparative pros and cons briefly discussed. We survey a number of approaches and application examples for controlling 3-D virtual globes using the Kinect sensor, then describe Kinoogle, a Kinect interface for natural interaction with Google Earth, developed by students at Texas A&M University. Readers interested in trying out the application on their own hardware can download a Zip archive (included with the manuscript as additional files 1, 2, &3) that contains a 'Kinnogle installation package for Windows PCs'. Finally, we discuss some usability aspects of Kinoogle and similar NUIs for controlling 3-D virtual globes (including possible future improvements), and propose a number of unique, practical 'use scenarios' where such NUIs could prove useful in navigating a 3-D virtual globe, compared to conventional mouse/3-D mouse and keyboard-based interfaces

A SDK improvement towards gesture support

Human-Computer Interaction have been one of the main focus of the technological community, specially the Natural User Interfaces (NUI) field of research as, since the launch of the Kinect Sensor, the goal to achieve fully natural interfaces just got a lot closer to reality. Taking advantage of this conditions the following research work proposes to compute the hand skeleton in order to recognize Sign Language Shapes. The proposed solution uses the Kinect Sensor to achieve a good segmentation and image analysis algorithms to extend the skeleton from the extraction of high-level features. In order to recognize complex hand shapes the current research work proposes the redefinition of the hand contour making it immutable to translation, rotation and scaling operations, and a set of tools to achieve a good recognition. The validation of the proposed solution extended the Kinects Software Development Kit to allow the developer to access the new set of inferred points and created a template-matching based platform that uses the contour to define the hand shape, this prototype was tested in a set of predefined conditions and showed to have a good success ration and has proven to be eligible for real-time scenarios

Hand Gesture-based Process Modeling for Updatable Processes

The increasing popularity of process models leads to the need of alternative interaction methods to view and manipulate process models. One big research field are the gesturebased manipulation methods. Although there are already works in this research area, they utilize only two dimensions for gesture recognition. The objective of this work is to introduce a system that manipulates process models using a three dimensional hand gesture input interface utilizing the RGB-D camera of the Microsoft Kinect. With this, an input interface can be created that is more natural and, thus, is easier to learn and use than its two dimensional counterpart. This work therefore discusses how gestures are recognized as well as technical implementation aspects (e.g., how process models are painted, accessed and manipulated). Furthermore, it explains the problems arising from the use of the Kinect as a hand tracking system and shows which steps have been taken to solve these problems

Gesture Based PC Interface with Kinect Sensor

Käesoleva töö eesmargiks on puutevaba kasutajaliidese loomine Kinect sensori alusel. Töö põhimõtte seisneb selles et eemaldada kasutaja ja arvuti vahelisest suhtlusest klaviatuuri, hiirt ja muid mehaanilisi lüliteid võimaldades juhtida arvutit vaid käeliigutustega. Töö tulemuseks on väljatöötatud kolm tarkvara rakendust. Esimise programmi ettemääratus seisneb selles et juhtida Windows Operatsioon Süsteemi Töölaua kasutades vaid käeliigutusi (ehk zeste). Süsteemil on olemas (ettenähtud) erireziim presentatsiooni juhtimiseks. Teine programm on loodud puhtalt teaduslike eesmärkidega et võimaldada inimkeha luukere liigeste koordinaatide salvestamist reaalajas. Kolmas programm võimaldab juhtida "Pioneer"-roboti ja manipulaatori käeliigutuste abil. Töö kuulub inimese ja masina vahelisse koostoime alale ja võib leida oma rakendust sellistel aladel, kus mehaaniliste lülitite peal baseeruva liidese kasutamine on võimatu või raskendatud (näiteks, kirurgiline meditsiin, kus arsti kindad peavad olema steriilsed).Present research targets development and implementation of the clickless interface on the basis of Kinect Sensor. The idea behind this work is to allow user to explore full functionality of Windows PC equipped with Kinect sensor, using just gestures and eliminating necessity to use any mechanical devices such as mouse or keyboard. Main result of the work are three standalone applications. First implements clickless interface to control Windows Desktop. Second application has purely scientific role and allows to acquire key coordinates of the human skeleton joints in real-time. Third application implements gesture based control of the Pioneer robot with robotic arm. Results of present research belong to the area of human-machine interaction and may found their applications in such areas where usage of mechanical interface elements may be impossible or complicated, for example surgical medicine, where surgeon gloves should be sterile

Recognition of gestures through artificial intelligence techniques

El reconocimiento de gestos consiste en la interpretación de secuencias de acciones humanas captadas por cualquier tipo de sensor, ya sea táctil o no requiera de contacto alguno con el dispositivo, como una cámara. En las últimas décadas ha experimentado un gran avance debido al auge de la Inteligencia Artificial y al desarrollo de sensores cada vez más complejos y precisos. Un ejemplo concreto ha sido la publicación y el mantenimiento de un SDK oficinal de Microsoft Kinect, con el que los desarrolladores han podido acceder a las capacidades de esta cámara para crear interfaces de usuario más naturales e intuitivas. También se ha incentivado el uso de aplicaciones que van más allá de la industria del entretenimiento, como aquellas que asisten en los cuidados médicos o que permiten la automatización de tareas rutinarias. Es por ello que en este proyecto hemos desarrollado un conjunto de herramientas para la generación de modelos de aprendizaje capaces de reconocer gestos personalizados para la Kinect v2. El conjunto de herramientas que se ha diseñado e implementado está orientado a facilitar la tarea completa de reconocimiento para cualquier gesto, comenzando con la captura de los ejemplos de entrenamiento, continuando con el pre-procesado y el tratamiento de los datos, y finalizando con la generación de modelos de aprendizaje mediante técnicas de aprendizaje automático. Finalmente, para evaluar el funcionamiento de la plataforma se ha propuesto y ejecutado una experimentación con un gesto sencillo. Los resultados positivos motivan el empleo de las herramientas desarrolladas para incorporar reconocedores de gestos en cualquier aplicación que utilice el sensor Kinect v2.The gesture recognition consists of the interpretation of sequences of human actions captured by any type of sensor either touchable or non-touchable like a camera. It has experimented a high progress in the last decades, due to the rise of the Artificial Intelligence and the development of more complex and precise sensors. One example of this advances was the publish and maintenance of an official SDK of Microsoft Kinect, which were used by developers to access to the capabilities of this camera, so they could create more natural and intuitive applications. This has motivated the use of applications that go beyond the entertainment industry, like those which assists in healthcare or automate routine tasks. For that reason, this project develops a set of tools for the generation of learning models that are able to recognize personalized gestures for Kinect v2. The set of designed and implemented tools is oriented to ease the task of the recognition of any gesture, starting in the capturing of training examples, continuing with the pre-processing and the treatment of data, and ending with the generation of the recognition models trough machine learning techniques. Finally, in order to test the functionality of the complete system, an experimentation with a simple gesture has been proposed and executed. The positive results motivate to use the set of developed tools to incorporate gesture recognizers in any application that uses the Kinect v2 sensor.Ingeniería Informática (Plan 2011

TiFEE : an input event-handling framework with touchless device support

Tese de mestrado integrado. Engenharia Informática e Computação. Universidade do Porto. Faculdade de Engenharia. 201