Making the Invisible Visible: Action Recognition Through Walls and Occlusions

Understanding people's actions and interactions typically depends on seeing them. Automating the process of action recognition from visual data has been the topic of much research in the computer vision community. But what if it is too dark, or if the person is occluded or behind a wall? In this paper, we introduce a neural network model that can detect human actions through walls and occlusions, and in poor lighting conditions. Our model takes radio frequency (RF) signals as input, generates 3D human skeletons as an intermediate representation, and recognizes actions and interactions of multiple people over time. By translating the input to an intermediate skeleton-based representation, our model can learn from both vision-based and RF-based datasets, and allow the two tasks to help each other. We show that our model achieves comparable accuracy to vision-based action recognition systems in visible scenarios, yet continues to work accurately when people are not visible, hence addressing scenarios that are beyond the limit of today's vision-based action recognition.Comment: ICCV 2019. The first two authors contributed equally to this pape

Align, Perturb and Decouple: Toward Better Leverage of Difference Information for RSI Change Detection

Change detection is a widely adopted technique in remote sense imagery (RSI) analysis in the discovery of long-term geomorphic evolution. To highlight the areas of semantic changes, previous effort mostly pays attention to learning representative feature descriptors of a single image, while the difference information is either modeled with simple difference operations or implicitly embedded via feature interactions. Nevertheless, such difference modeling can be noisy since it suffers from non-semantic changes and lacks explicit guidance from image content or context. In this paper, we revisit the importance of feature difference for change detection in RSI, and propose a series of operations to fully exploit the difference information: Alignment, Perturbation and Decoupling (APD). Firstly, alignment leverages contextual similarity to compensate for the non-semantic difference in feature space. Next, a difference module trained with semantic-wise perturbation is adopted to learn more generalized change estimators, which reversely bootstraps feature extraction and prediction. Finally, a decoupled dual-decoder structure is designed to predict semantic changes in both content-aware and content-agnostic manners. Extensive experiments are conducted on benchmarks of LEVIR-CD, WHU-CD and DSIFN-CD, demonstrating our proposed operations bring significant improvement and achieve competitive results under similar comparative conditions. Code is available at https://github.com/wangsp1999/CD-Research/tree/main/openAPDComment: To appear in IJCAI 202

Visibility Video Detection with Dark Channel Prior on Highway

Dark channel prior (DCP) has advantages in image enhancement and image haze removal and is explored to detect highway visibility according to the physical relationship between transmittance and extinction coefficient. However, there are three major error sources in calculating transmittance. The first is that sky regions do not satisfy the assumptions of DCP algorithm. So the optimization algorithms combined with region growing and coefficient correction method are proposed. When extracting atmospheric brightness, different values lead to the second error. Therefore, according to different visibility conditions, a multimode classification method is designed. Image blocky effect causes the third error. Then guided image filtering is introduced to obtain accurate transmittance of each pixel of image. Next, according to the definition meteorological optical visual range and the relationship between transmittance and extinction coefficient of Lambert-Beer’s Law, accurate visibility value can be calculated. A comparative experimental system including visibility detector and video camera was set up to verify the accuracy of these optimization algorithms. Finally, a large number of highway section videos were selected to test the validity of DCP method in different models. The results indicate that these detection visibility methods are feasible and reliable for the smooth operation of highways

Dashboards for Real-time Monitoring of Winter Operations Activities and After-action Assessment

The Indiana Department of Transportation (INDOT) operates a fleet of nearly 1100 snowplows and spends up to $60M annually on snow removal and de-icing as part of their winter operation maintenance activities. Systematically allocating resources and optimizing material application rates can potentially save revenue that can be reallocated for other roadway maintenance operations. Modern snowplows are beginning to be equipped with a variety of Mobile Road Weather Information Sensors (MARWIS) which can provide a host of analytical data characterizing on-the-ground conditions during periods of wintry precipitation. Traffic speeds fused with road conditions and precipitation data from weather stations provide a uniquely detailed look at the progression of a winter event and the performance of the fleet. This research uses a combination of traffic speeds, MARWIS and North American Land Data Assimilation System (NLDAS) data to develop real-time dashboards characterizing the impact of precipitation and pavement surface temperature on mobility. Twenty heavy snow events were identified for the state of Indiana from November 2018 through April 2019. Two particular instances, that impacted 182 miles and 231 miles of interstate at their peaks occurred in January and March, respectively, and were used as a case study for this paper. The dashboards proposed in this paper may prove to be particularly useful for agencies in tracking fleet activity through a winter storm, helping in resource allocation and scheduling and forecasting resource needs

BAAP: A Behavioral Animation Authoring Platform for Emotion Driven 3D Virtual Characters

Abstract. Emotion, as an important aspect of human intelligence, has been playing a significant role in virtual characters. We propose an improved threelevel structure of affective model as "personality-emotion-mood" for intelligent and emotional virtual characters. We also present the emotion state space, as well as the emotion updating functions, to generate authentic and expressive emotions. In order to achieve the complexity and variety of behaviors, we bring forward a behavior organizing structure as the behavior tree, which defines four kinds of behavior organizations as well as the behavior tag and behavior message, to manage virtual characters' behaviors. At the end, we achieve an experimental platform BAAP, which prove our emotion model and behavior organizing structure to be effective and practical in generating intelligent and emotional behavioral animations. Keywords: virtual character, affective computing, behavior tree, behavioral animation, authoring platform Introduction In previous work [1], we proposed a computational emotion model which can be incorporated into physiological and social components of emotions. We improve it in this paper and present an experimental platform BAAP. We mainly focus on two subjects: emotion modeling and behavior organizing. Based on the OCEAN mode

Gray matter asymmetry atypical patterns in subgrouping minors with autism based on core symptoms

Abnormal gray matter (GM) asymmetry has been verified in autism spectrum disorder (ASD), which is characterized by high heterogeneity. ASD is distinguished by three core symptom domains. Previous neuroimaging studies have offered support for divergent neural substrates of different core symptom domains in ASD. However, no previous study has explored GM asymmetry alterations underlying different core symptom domains. This study sought to clarify atypical GM asymmetry patterns underlying three core symptom domains in ASD with a large sample of 230 minors with ASD (ages 7–18 years) and 274 matched TD controls from the Autism Brain Imaging Data Exchange I (ABIDE I) repository. To this end, the scores of the revised autism diagnostic interview (ADI-R) subscales were normalized for grouping ASD into three core-symptom-defined subgroups: social interaction (SI), verbal communication (VA), and restricted repetitive behaviors (RRB). We investigated core-symptom-related GM asymmetry alterations in ASD resulting from advanced voxel-based morphometry (VBM) by general linear models. We also examined the relationship between GM asymmetry and age and between GM asymmetry and symptom severity assessed by the Autism Diagnostic Observation Schedule (ADOS). We found unique GM asymmetry alterations underlying three core-symptom-defined subgroups in ASD: more rightward asymmetry in the thalamus for SI, less rightward asymmetry in the superior temporal gyrus, anterior cingulate and caudate for VA, and less rightward asymmetry in the middle and inferior frontal gyrus for RRB. Furthermore, the asymmetry indexes in the thalamus were negatively associated with ADOS_SOCIAL scores in the general ASD group. We also showed significant correlations between GM asymmetry and age in ASD and TD individuals. Our results support the theory that each core symptom domain of ASD may have independent etiological and neurobiological underpinnings, which is essential for the interpretation of heterogeneity and the future diagnosis and treatment of ASD