An Immersive Telepresence System using RGB-D Sensors and Head Mounted Display

We present a tele-immersive system that enables people to interact with each other in a virtual world using body gestures in addition to verbal communication. Beyond the obvious applications, including general online conversations and gaming, we hypothesize that our proposed system would be particularly beneficial to education by offering rich visual contents and interactivity. One distinct feature is the integration of egocentric pose recognition that allows participants to use their gestures to demonstrate and manipulate virtual objects simultaneously. This functionality enables the instructor to ef- fectively and efficiently explain and illustrate complex concepts or sophisticated problems in an intuitive manner. The highly interactive and flexible environment can capture and sustain more student attention than the traditional classroom setting and, thus, delivers a compelling experience to the students. Our main focus here is to investigate possible solutions for the system design and implementation and devise strategies for fast, efficient computation suitable for visual data processing and network transmission. We describe the technique and experiments in details and provide quantitative performance results, demonstrating our system can be run comfortably and reliably for different application scenarios. Our preliminary results are promising and demonstrate the potential for more compelling directions in cyberlearning.Comment: IEEE International Symposium on Multimedia 201

Securing Cyber-Physical Social Interactions on Wrist-worn Devices

Since ancient Greece, handshaking has been commonly practiced between two people as a friendly gesture to express trust and respect, or form a mutual agreement. In this article, we show that such physical contact can be used to bootstrap secure cyber contact between the smart devices worn by users. The key observation is that during handshaking, although belonged to two different users, the two hands involved in the shaking events are often rigidly connected, and therefore exhibit very similar motion patterns. We propose a novel key generation system, which harvests motion data during user handshaking from the wrist-worn smart devices such as smartwatches or fitness bands, and exploits the matching motion patterns to generate symmetric keys on both parties. The generated keys can be then used to establish a secure communication channel for exchanging data between devices. This provides a much more natural and user-friendly alternative for many applications, e.g., exchanging/sharing contact details, friending on social networks, or even making payments, since it doesn’t involve extra bespoke hardware, nor require the users to perform pre-defined gestures. We implement the proposed key generation system on off-the-shelf smartwatches, and extensive evaluation shows that it can reliably generate 128-bit symmetric keys just after around 1s of handshaking (with success rate >99%), and is resilient to different types of attacks including impersonate mimicking attacks, impersonate passive attacks, or eavesdropping attacks. Specifically, for real-time impersonate mimicking attacks, in our experiments, the Equal Error Rate (EER) is only 1.6% on average. We also show that the proposed key generation system can be extremely lightweight and is able to run in-situ on the resource-constrained smartwatches without incurring excessive resource consumption

Understanding face and eye visibility in front-facing cameras of smartphones used in the wild

Commodity mobile devices are now equipped with high-resolution front-facing cameras, allowing applications in biometrics (e.g., FaceID in the iPhone X), facial expression analysis, or gaze interaction. However, it is unknown how often users hold devices in a way that allows capturing their face or eyes, and how this impacts detection accuracy. We collected 25,726 in-the-wild photos, taken from the front-facing camera of smartphones as well as associated application usage logs. We found that the full face is visible about 29% of the time, and that in most cases the face is only partially visible. Furthermore, we identified an influence of users' current activity; for example, when watching videos, the eyes but not the entire face are visible 75% of the time in our dataset. We found that a state-of-the-art face detection algorithm performs poorly against photos taken from front-facing cameras. We discuss how these findings impact mobile applications that leverage face and eye detection, and derive practical implications to address state-of-the art's limitations

Industry 4.0: Hand Recognition on Assembly Supervision Process

In the assembly industry, the process of assembling components is very important in order to produce a quality product. Assembly of components should be carried out sequentially based on the standards set by the company. For companies that still operate the assembly process manually by employee, sometimes errors occur in the assembly process, which can affect the quality of production. In order to be carried out the assembly process according to the procedure, a system is needed that can detect employee hands when carrying out the assembly process automatically. This study proposes an artificial intelligence-based real-time employee hand detection system. This system will be the basis for the development of an automatic industrial product assembly process to welcome the Industry 4.0. To verify system performance, several experiments were carried out, such as; detecting the right and left hands of employees and detecting hands when using accessories or not. From the experimental results it can be concluded that the system is able to detect the right and left hands of employees well with the resulting FPS average of 15.4.Pada industri perakitan, proses merakit komponen merupakan hal yang sangat penting guna menghasilkan produk yang berkualitas. Perakitan komponen hendaklah dilakukan secara urut berdasarkan standar yang telah ditentukan oleh perusahaan. Bagi perusahaan yang masih menggunakan proses perakitan secara manual yakni dengan menggunakan tenaga manusia, terkadang terjadi kesalahan dalam proses perakitan, sehingga dapat mempengaruhi kualitas produksi. Agar proses perakitan dapat dilakukan sesuai prosedur, maka diperlukan sebuah sistem yang dapat mendeteksi tangan karyawan ketika melakukan proses perakitan secara otomatis. Penelitian ini mengusulkan sistem pendeteksian tangan karyawan secara real-time berbasis kecerdasan buatan. Sistem ini akan menjadi dasar untuk pengembangan proses perakitan produk industri secara otomatis untuk menyambut industri 4.0. Untuk memverifikasi kinerja sistem, beberapa percobaan dilakukan yaitu mendeteksi tangan kanan dan kiri karyawan serta mendeteksi tangan ketika menggunakan aksesoris atau tidak. Dari hasil percobaan dapat disimpulkan bahwa sistem mampu mendeteksi tangan kanan dan kiri karyawan dengan baik dengan rata-rata FPS yang dihasilkan adalah 15.4

FocalSpace: Multimodal Activity Tracking, Synthetic Blur and Adapative Presentation for Video Conferencing

We introduce FocalSpace, a video conferencing system that dynamically recognizes relevant activities and objects through depth sensing and hybrid tracking of multimodal cues, such as voice, gesture, and proximity to surfaces. FocalSpace uses this information to enhance users' focus by diminishing the background through synthetic blur effects. We present scenarios that support the suppression of visual distraction, provide contextual augmentation, and enable privacy in dynamic mobile environments. Our user evaluation indicates increased memory accuracy and user preference for FocalSpace techniques compared to traditional video conferencing

Augmented Reality based monitoring of the remote-lab

© 2014 IEEE. Augmented Reality technology approach has been being adopted within the education sector. The advanced technology tools in many classes have the potential changed of users' attitudes toward the pedagogical and psychological objectives and goals. Moreover, augmented reality has not elicited so much attention within the corridors of education sector as it is now. In order to improve the interactive effectiveness in the smart classroom environment, there is a demand to tailor the innovation technology and align it with every changing requirements and capabilities of various users. Consequently, the educators are increasingly finding augmented reality suitable for deployment in education. In this paper, a project shows how Augmented Reality utilised with overlay Smart-Grid can support the learning process in attractive methods for monitoring events of captured scenes in remote-lab such as video stream, Web-link from smart devices' camera

Two Hand Gesture Based 3D Navigation in Virtual Environments

Natural interaction is gaining popularity due to its simple, attractive, and realistic nature, which realizes direct Human Computer Interaction (HCI). In this paper, we presented a novel two hand gesture based interaction technique for 3 dimensional (3D) navigation in Virtual Environments (VEs). The system used computer vision techniques for the detection of hand gestures (colored thumbs) from real scene and performed different navigation (forward, backward, up, down, left, and right) tasks in the VE. The proposed technique also allow users to efficiently control speed during navigation. The proposed technique is implemented via a VE for experimental purposes. Forty (40) participants performed the experimental study. Experiments revealed that the proposed technique is feasible, easy to learn and use, having less cognitive load on users. Finally gesture recognition engines were used to assess the accuracy and performance of the proposed gestures. kNN achieved high accuracy rates (95.7%) as compared to SVM (95.3%). kNN also has high performance rates in terms of training time (3.16 secs) and prediction speed (6600 obs/sec) as compared to SVM with 6.40 secs and 2900 obs/sec