Deep Video Color Propagation

Traditional approaches for color propagation in videos rely on some form of matching between consecutive video frames. Using appearance descriptors, colors are then propagated both spatially and temporally. These methods, however, are computationally expensive and do not take advantage of semantic information of the scene. In this work we propose a deep learning framework for color propagation that combines a local strategy, to propagate colors frame-by-frame ensuring temporal stability, and a global strategy, using semantics for color propagation within a longer range. Our evaluation shows the superiority of our strategy over existing video and image color propagation methods as well as neural photo-realistic style transfer approaches.Comment: BMVC 201

Relevant for us? We-prioritization in cognitive processing

Humans are social by nature. We ask whether this social nature operates as a lens through which individuals process the world even in the absence of immediate interactions or explicit goals to collaborate. Is information that is potentially relevant to a group one belongs to (“We”) processed with priority over information potentially relevant to a group one does not belong to (“They”)? We conducted three experiments using a modified version of Sui, He, and Humphreys’ (2012) shape–label matching task. Participants were assigned to groups either via a common preference between assigned team members (Experiment 1) or arbitrarily (Experiment 2). In a third experiment, only personal pronouns were used. Overall, a processing benefit for we-related information (we-prioritization) occurred regardless of the type of group induction. A final experiment demonstrated that we-prioritization did not extend to other individual members of a short-term transitory group. We suggest that the results reflect an intrinsic predisposition to process information “relevant for us” with priority, which might feed into optimizing collaborative processes

Learning Task Priorities from Demonstrations

Bimanual operations in humanoids offer the possibility to carry out more than one manipulation task at the same time, which in turn introduces the problem of task prioritization. We address this problem from a learning from demonstration perspective, by extending the Task-Parameterized Gaussian Mixture Model (TP-GMM) to Jacobian and null space structures. The proposed approach is tested on bimanual skills but can be applied in any scenario where the prioritization between potentially conflicting tasks needs to be learned. We evaluate the proposed framework in: two different tasks with humanoids requiring the learning of priorities and a loco-manipulation scenario, showing that the approach can be exploited to learn the prioritization of multiple tasks in parallel.Comment: Accepted for publication at the IEEE Transactions on Robotic

Eye contact facilitates awareness of faces during interocular suppression

Eye contact captures attention and receives prioritized visual processing. Here we asked whether eye contact might be processed outside conscious awareness. Faces with direct and averted gaze were rendered invisible using interocular suppression. In two experiments we found that faces with direct gaze overcame such suppression more rapidly than faces with averted gaze. Control experiments ruled out the influence of low-level stimulus differences and differential response criteria. These results indicate an enhanced unconscious representation of direct gaze, enabling the automatic and rapid detection of other individuals making eye contact with the observer

Applied deep learning in intelligent transportation systems and embedding exploration

Deep learning techniques have achieved tremendous success in many real applications in recent years and show their great potential in many areas including transportation. Even though transportation becomes increasingly indispensable in people’s daily life, its related problems, such as traffic congestion and energy waste, have not been completely solved, yet some problems have become even more critical. This dissertation focuses on solving the following fundamental problems: (1) passenger demand prediction, (2) transportation mode detection, (3) traffic light control, in the transportation field using deep learning. The dissertation also extends the application of deep learning to an embedding system for visualization and data retrieval. The first part of this dissertation is about a Spatio-TEmporal Fuzzy neural Network (STEF-Net) which accurately predicts passenger demand by incorporating the complex interaction of all known important factors, such as temporal, spatial and external information. Specifically, a convolutional long short-term memory network is employed to simultaneously capture spatio-temporal feature interaction, and a fuzzy neural network to model external factors. A novel feature fusion method with convolution and an attention layer is proposed to keep the temporal relation and discriminative spatio-temporal feature interaction. Experiments on a large-scale real-world dataset show the proposed model outperforms the state-of-the-art approaches. The second part is a light-weight and energy-efficient system which detects transportation modes using only accelerometer sensors in smartphones. Understanding people’s transportation modes is beneficial to many civilian applications, such as urban transportation planning. The system collects accelerometer data in an efficient way and leverages a convolutional neural network to determine transportation modes. Different architectures and classification methods are tested with the proposed convolutional neural network to optimize the system design. Performance evaluation shows that the proposed approach achieves better accuracy than existing work in detecting people’s transportation modes. The third component of this dissertation is a deep reinforcement learning model, based on Q learning, to control the traffic light. Existing inefficient traffic light control causes numerous problems, such as long delay and waste of energy. In the proposed model, the complex traffic scenario is quantified as states by collecting data and dividing the whole intersection into grids. The timing changes of a traffic light are the actions, which are modeled as a high-dimension Markov decision process. The reward is the cumulative waiting time difference between two cycles. To solve the model, a convolutional neural network is employed to map states to rewards, which is further optimized by several components, such as dueling network, target network, double Q-learning network, and prioritized experience replay. The simulation results in Simulation of Urban MObility (SUMO) show the efficiency of the proposed model in controlling traffic lights. The last part of this dissertation studies the hierarchical structure in an embedding system. Traditional embedding approaches associate a real-valued embedding vector with each symbol or data point, which generates storage-inefficient representation and fails to effectively encode the internal semantic structure of data. A regularized autoencoder framework is proposed to learn compact Hierarchical K-way D-dimensional (HKD) discrete embedding of data points, aiming at capturing semantic structures of data. Experimental results on synthetic and real-world datasets show that the proposed HKD embedding can effectively reveal the semantic structure of data via visualization and greatly reduce the search space of nearest neighbor retrieval while preserving high accuracy

Flexible Congestion Management for Error Reduction in Wireless Sensor Networks

The dissertation is concerned with the efficient resolution of data congestion on wireless sensor networks (WSNs). WSNs are of increasing relevance due to their applications in automation, industrial processes, natural-disaster detection, weather prediction, and climate monitoring. In large WSNs where measurements are periodically made at each node in the network and sent in a multi-hop fashion via the network tree to a single base-station node, the volume of data at a node may exceed the transmission capabilities of the node. This type of congestion can negatively impact data accuracy when packets are lost in transmission. We propose flexible congestion management for sensor networks (FCM) as a data-collection scheme to reduce network traffic and minimize the error resulting from data-volume reduction. FCM alleviates all congestion by lossy data fusion, encourages opportunistic fusion with an application-specific distortion tolerance, and balances network traffic. We consider several data-fusion methods including the k-means algorithm and two forms of adaptive summarization. Additional fusion is allowed when like data may be fused with low error up to some limit set by the user of the data-collection application on the network. Increasing the error limit tends to reduce the overall traffic on the network at the cost of data accuracy. When a node fuses more data than is required to alleviate congestion, its siblings are notified that they may increase the sizes of their transmissions accordingly. FCM is further improved to re-balance the network traffic of subtrees such that subtrees whose measurements have lower variance may decrease their output rates while subtrees whose measurements have higher variance may increase their output rates, while still addressing all congestion in the network. We verify the effectiveness of FCM with extensive simulations