Skeleton-based Action Recognition of People Handling Objects

In visual surveillance systems, it is necessary to recognize the behavior of people handling objects such as a phone, a cup, or a plastic bag. In this paper, to address this problem, we propose a new framework for recognizing object-related human actions by graph convolutional networks using human and object poses. In this framework, we construct skeletal graphs of reliable human poses by selectively sampling the informative frames in a video, which include human joints with high confidence scores obtained in pose estimation. The skeletal graphs generated from the sampled frames represent human poses related to the object position in both the spatial and temporal domains, and these graphs are used as inputs to the graph convolutional networks. Through experiments over an open benchmark and our own data sets, we verify the validity of our framework in that our method outperforms the state-of-the-art method for skeleton-based action recognition.Comment: Accepted in WACV 201

Learning to Discriminate Information for Online Action Detection

From a streaming video, online action detection aims to identify actions in the present. For this task, previous methods use recurrent networks to model the temporal sequence of current action frames. However, these methods overlook the fact that an input image sequence includes background and irrelevant actions as well as the action of interest. For online action detection, in this paper, we propose a novel recurrent unit to explicitly discriminate the information relevant to an ongoing action from others. Our unit, named Information Discrimination Unit (IDU), decides whether to accumulate input information based on its relevance to the current action. This enables our recurrent network with IDU to learn a more discriminative representation for identifying ongoing actions. In experiments on two benchmark datasets, TVSeries and THUMOS-14, the proposed method outperforms state-of-the-art methods by a significant margin. Moreover, we demonstrate the effectiveness of our recurrent unit by conducting comprehensive ablation studies.Comment: To appear in CVPR 202

Measuring Patient Similarity on Multiple Diseases by Joint Learning via a Convolutional Neural Network

Patient similarity research is one of the most fundamental tasks in healthcare, helping to make decisions without incurring additional time and costs in clinical practices. Patient similarity can also apply to various medical fields, such as cohort analysis and personalized treatment recommendations. Because of this importance, patient similarity measurement studies are actively being conducted. However, medical data have complex, irregular, and sequential characteristics, making it challenging to measure similarity. Therefore, measuring accurate similarity is a significant problem. Existing similarity measurement studies use supervised learning to calculate the similarity between patients, with similarity measurement studies conducted only on one specific disease. However, it is not realistic to consider only one kind of disease, because other conditions usually accompany it; a study to measure similarity with multiple diseases is needed. This research proposes a convolution neural network-based model that jointly combines feature learning and similarity learning to define similarity in patients with multiple diseases. We used the cohort data from the National Health Insurance Sharing Service of Korea for the experiment. Experimental results verify that the proposed model has outstanding performance when compared to other existing models for measuring multiple-disease patient similarity

Improving counseling effectiveness with virtual counselors through nonverbal compassion involving eye contact, facial mimicry, and head-nodding

An effective way to reduce emotional distress is by sharing negative emotions with others. This is why counseling with a virtual counselor is an emerging methodology, where the sharer can consult freely anytime and anywhere without having to fear being judged. To improve counseling effectiveness, most studies so far have focused on designing verbal compassion for virtual counselors. However, recent studies showed that virtual counselors' nonverbal compassion through eye contact, facial mimicry, and head-nodding also have significant impact on the overall counseling experience. To verify this, we designed the virtual counselor's nonverbal compassion and examined its effects on counseling effectiveness (i.e., reduce the intensity of anger and improve general affect). A total of 40 participants were recruited from the university community. Participants were then randomly assigned to one of two virtual counselor conditions: a neutral virtual counselor condition without nonverbal compassion and a compassionate virtual counselor condition with nonverbal compassion (i.e., eye contact, facial mimicry, and head-nodding). Participants shared their anger-inducing episodes with the virtual counselor for an average of 16.30 min. Note that the virtual counselor was operated by the Wizard-of-Oz method without actually being technically implemented. Results showed that counseling with a compassionate virtual counselor reduced the intensity of anger significantly more than counseling with a neutral virtual counselor (F(1, 37) = 30.822, p < 0.001, ηp2 = 0.454). In addition, participants who counseled with a compassionate virtual counselor responded that they experienced higher empathy than those who counseled with a neutral virtual counselor (p < 0.001). These findings suggest that nonverbal compassion through eye contact, facial mimicry, and head-nodding of the virtual counselor makes the participants feel more empathy, which contributes to improving the counseling effectiveness by reducing the intensity of anger