Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns

A more natural, intuitive, user-friendly, and less intrusive Human–Computer interface for controlling an application by executing hand gestures is presented. For this purpose, a robust vision-based hand-gesture recognition system has been developed, and a new database has been created to test it. The system is divided into three stages: detection, tracking, and recognition. The detection stage searches in every frame of a video sequence potential hand poses using a binary Support Vector Machine classifier and Local Binary Patterns as feature vectors. These detections are employed as input of a tracker to generate a spatio-temporal trajectory of hand poses. Finally, the recognition stage segments a spatio-temporal volume of data using the obtained trajectories, and compute a video descriptor called Volumetric Spatiograms of Local Binary Patterns (VS-LBP), which is delivered to a bank of SVM classifiers to perform the gesture recognition. The VS-LBP is a novel video descriptor that constitutes one of the most important contributions of the paper, which is able to provide much richer spatio-temporal information than other existing approaches in the state of the art with a manageable computational cost. Excellent results have been obtained outperforming other approaches of the state of the art

Temporal pyramid Matching of local binary sub-patterns for hand-gesture recognition

Human–computer Interaction systems based on hand-gesture recognition are nowadays of great interest to establish a natural communication between humans and machines. However, the visual recognition of gestures and other human poses remains a challenging problem. In this paper, the original volumetric spatiograms of local binary patterns descriptor has been extended to efficiently and robustly encode the spatial and temporal information of hand gestures. This enhancement mitigates the dimensionality problems of the previous approach, and considers more temporal information to achieve a higher recognition rate. Excellent results have been obtained, outperforming other existing approaches of the state of the art

Tiny hand gesture recognition without localization via a deep convolutional network

Visual hand-gesture recognition is being increasingly desired for human-computer interaction interfaces. In many applications, hands only occupy about 10% of the image, whereas the most of it contains background, human face, and human body. Spatial localization of the hands in such scenarios could be a challenging task and ground truth bounding boxes need to be provided for training, which is usually not accessible. However, the location of the hand is not a requirement when the criteria is just the recognition of a gesture to command a consumer electronics device, such as mobiles phones and TVs. In this paper, a deep convolutional neural network is proposed to directly classify hand gestures in images without any segmentation or detection stage that could discard the irrelevant not-hand areas. The designed hand-gesture recognition network can classify seven sorts of hand gestures in a user-independent manner and on real time, achieving an accuracy of 97.1% in the dataset with simple backgrounds and 85.3% in the dataset with complex backgrounds

Pelacakan Gerak Tangan dengan Metode Metode Pelacakan Objek Berbasis Korelasi

Pelacakan tangan (hand tracking) saat ini tengah mendapat perhatian dari para peneliti di bidang visi komputer (computer vision). Tujuan utama dari pelacakan tangan adalah untuk mengetahui lokasi tangan di setiap deretan frame video atau deretan citra. Ada tiga buah fitur yang dapat digunakan untuk melacak objek yakni bentuk, warna, dan gerak. Fitur bentuk susah untuk digunakan melacak tangan, sedangkan fitur warna dan gerak lebih mudah dan lebih reliabel untuk melacak gerak tangan. Fitur warna dan gerak memiliki kelemahan-kelemahan masing-masing, sehingga pada penelitian sebelumnya diusulkanlah penggabungan kedua buah fitur tersebut untuk digunakan dalam melacak tangan secara bersama-sama agar dapat saling menutupi kelemahan masing-masing fitur. Namun melihat dari proses yang dilakukan pada metode yang diusulkan pada penelitian sebelumnya, peran yang lebih memiliki andil adalah peran dari fitur gerak. Pergerakan tangan yang cepat juga belum dapat ditangani dengan baik padahal dalam bahasa isyarat, gestur, dan aktivitas keseharian manusia, tidak jarang tangan bergerak dengan cepat. Oleh karena itu, pada penelitian ini diusulkan sebuah metode pelacakan tangan dengan menggunakan fitur gerak dan warna menggunakan metode pelacakan berbasis korelasi. Secara sederhana, metode berbasis korelasi ini akan melacak tangan di frame selanjutnya secara melingkar di sekitar lokasi sebelumnya dan setelah ditemukan dilakukan pelacakan menjauh dari lokasi awal ke arah yang ditemukan di pelacakan pertama. Fitur warna dan gerak akan digunakan secara seimbang dalam melacak arah pergerakan tangan. Diharapkan dengan menggunakan metode pelacakan yang diusulkan ini, baik gerak tangan yang cepat atau lambat dapat dilacak dengan baik

Deep Siamese Networks toward Robust Visual Tracking

Recently, Siamese neural networks have been widely used in visual object tracking to leverage the template matching mechanism. Siamese network architecture contains two parallel streams to estimate the similarity between two inputs and has the ability to learn their discriminative features. Various deep Siamese-based tracking frameworks have been proposed to estimate the similarity between the target and the search region. In this chapter, we categorize deep Siamese networks into three categories by the position of the merging layers as late merge, intermediate merge and early merge architectures. In the late merge architecture, inputs are processed as two separate streams and merged at the end of the network, while in the intermediate merge architecture, inputs are initially processed separately and merged intermediate well before the final layer. Whereas in the early merge architecture, inputs are combined at the start of the network and a unified data stream is processed by a single convolutional neural network. We evaluate the performance of deep Siamese trackers based on the merge architectures and their output such as similarity score, response map, and bounding box in various tracking challenges. This chapter will give an overview of the recent development in deep Siamese trackers and provide insights for the new developments in the tracking field

Myo-Elektriksel Sinyaller İle İnsansız Kara Aracının Uzaktan Kontrolü

Bu çalışma kapsamında insansız bir kara aracının kişinin el ve parmak hareketleri ile uzaktan kontrolü gerçekleştirilmiştir. Beyinden kol kaslarına iletilen ve kişinin el hareketlerini gerçekleştirmesini sağlayan Elektromiyografi (EMG) sinyalleri, kişinin koluna giydiği sekiz EMG sensör içeren bileklik vasıtası ile gerçek zamanlı olarak alınmıştır. Raspberry pi 3 gömülü sistem kartı üzerinde geliştirilen sinyal işleme, öznitelik çıkarımı ve sınıflandırma algoritmaları kullanılarak anlamlandırılmıştır. Başka bir deyişle el hareketin örüntüsü (el kapama, parmak açma, serçe parmak temas, bilek dışa bükme, vs.) ile EMG sinyal grubu arasındaki ilişkiler tanımlanmıştır. Anlamlandırılan her bir el hareketi araç için bir hareketi kontrol komutu (el kapama: araç ileri, parmak açma: araç dur, serçe parmağa temas: sola dönüş, bilek dışa bükme: sağa dönüş, vs.) olarak kullanılmıştır. Böylece insan – mobil araç etkileşim ağı kurulmuştur. Kurulan insan- mobil araç etkileşim ağı sayesinde el hareketleri ile mobil aracın gerçek zamanlı hareket kontrolü ortalama % 92 başarı ile gerçekleştirilmiştir

Development of an EMG-controlled mobile robot

This paper presents the development of a Robot Operating System (ROS)-based mobile robot control using electromyography (EMG) signals. The proposed robot’s structure is specifically designed to provide modularity and is controlled by a Raspberry Pi 3 running on top of an ROS application and a Teensy microcontroller. The EMG muscle commands are sent to the robot with hand gestures that are captured using a Thalmic Myo Armband and recognized using a k-Nearest Neighbour (k-NN) classifier. The robot’s performance is evaluated by navigating it through specific paths while solely controlling it through the EMG signals and using the collision avoidance approach. Thus, this paper aims to expand the research on the topic, introducing a more accurate classification system with a wider set of gestures, hoping to come closer to a usable real-life applicatio

Human behavior understanding for worker-centered intelligent manufacturing

“In a worker-centered intelligent manufacturing system, sensing and understanding of the worker’s behavior are the primary tasks, which are essential for automatic performance evaluation & optimization, intelligent training & assistance, and human-robot collaboration. In this study, a worker-centered training & assistant system is proposed for intelligent manufacturing, which is featured with self-awareness and active-guidance. To understand the hand behavior, a method is proposed for complex hand gesture recognition using Convolutional Neural Networks (CNN) with multiview augmentation and inference fusion, from depth images captured by Microsoft Kinect. To sense and understand the worker in a more comprehensive way, a multi-modal approach is proposed for worker activity recognition using Inertial Measurement Unit (IMU) signals obtained from a Myo armband and videos from a visual camera. To automatically learn the importance of different sensors, a novel attention-based approach is proposed to human activity recognition using multiple IMU sensors worn at different body locations. To deploy the developed algorithms to the factory floor, a real-time assembly operation recognition system is proposed with fog computing and transfer learning. The proposed worker-centered training & assistant system has been validated and demonstrated the feasibility and great potential for applying to the manufacturing industry for frontline workers. Our developed approaches have been evaluated: 1) the multi-view approach outperforms the state-of-the-arts on two public benchmark datasets, 2) the multi-modal approach achieves an accuracy of 97% on a worker activity dataset including 6 activities and achieves the best performance on a public dataset, 3) the attention-based method outperforms the state-of-the-art methods on five publicly available datasets, and 4) the developed transfer learning model achieves a real-time recognition accuracy of 95% on a dataset including 10 worker operations”--Abstract, page iv

Development of an EMG-controlled mobile robot

This paper presents the development of a Robot Operating System (ROS)-based mobile robot control using electromyography (EMG) signals. The proposed robot’s structure is specifically designed to provide modularity and is controlled by a Raspberry Pi 3 running on top of an ROS application and a Teensy microcontroller. The EMG muscle commands are sent to the robot with hand gestures that are captured using a Thalmic Myo Armband and recognized using a k-Nearest Neighbour (k-NN) classifier. The robot’s performance is evaluated by navigating it through specific paths while solely controlling it through the EMG signals and using the collision avoidance approach. Thus, this paper aims to expand the research on the topic, introducing a more accurate classification system with a wider set of gestures, hoping to come closer to a usable real-life applicatio