Action Tubelet Detector for Spatio-Temporal Action Localization

Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level that are then linked or tracked across time. In this paper, we leverage the temporal continuity of videos instead of operating at the frame level. We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, i.e., sequences of bounding boxes with associated scores. The same way state-of-the-art object detectors rely on anchor boxes, our ACT-detector is based on anchor cuboids. We build upon the SSD framework. Convolutional features are extracted for each frame, while scores and regressions are based on the temporal stacking of these features, thus exploiting information from a sequence. Our experimental results show that leveraging sequences of frames significantly improves detection performance over using individual frames. The gain of our tubelet detector can be explained by both more accurate scores and more precise localization. Our ACT-detector outperforms the state-of-the-art methods for frame-mAP and video-mAP on the J-HMDB and UCF-101 datasets, in particular at high overlap thresholds.Comment: 9 page

Structured Mesh Generation : Open-source automatic nonuniform mesh generation for FDTD simulation

This article describes a cuboid structured mesh generator suitable for 3D numerical modelling using techniques such as finite-difference time-domain (FDTD) and transmission-line matrix (TLM). The mesh generator takes as its input an unstructured triangular surface mesh such as is available from many CAD systems, determines a suitable variable mesh discretisation and generates solid and surface meshes in a format suitable for import by the numerical solver. The mesher is implemented in the MATLAB language and is available as open source software

Impulse-based discrete element modelling of rock impact and fragmentation, with applications to block cave mining

Impulse-based methods efficiently and accurately model high-frequency collisions of complex shapes based on the enforcement of non-penetrating constraints. It does not rely on penalty parameters nor requires the computation of penetration between bodies. This work presents a novel necessary condition for energy conservation in impulse-based methods. In previous versions of the impulse methods, such as sequential and simultaneous impulse methods, the relative velocity at the contact points after collision is directly derived from the relative velocity before collision, in a purely simultaneous or sequential manner. This work presents a novel energy tracking method (ETM), in which the relative velocities are iteratively but gradually adjusted, simultaneously modelling their interaction at each iteration. ETM ensures the energy conservation while capturing the propagation of forces during collision. The ETM is applied to model the dynamics of fragment collision in the context of fragmentation. Two approaches of fragmentation are proposed: a finite-discrete element approach, and a low cost, fragmentation pattern-based approach. The first approach models the growth of fractures using the finite element method (FEM) and advanced re-meshing technology. This finite-discrete element approach suffers from the drawback of massive computational cost. The low-cost, fragmentation pattern-based approach separate colliding bodies directly. The fragmentation pattern is generated using Weibull distribution equations, the patterns and size distributions computed using full finite/discrete element simulations and experimental results. This work investigates the influence of fragmentation on the frequency of hang-up events and on the gravity flow of rock fragments within a block caving system. Numerical results indicate that models that do not consider fragmentation tend to overestimate the frequency of hang-up accidents.Open Acces

3D printed neuromorphic sensing systems

Thanks to the high energy efficiency, neuromorphic devices are spotlighted recently by mimicking the calculation principle of the human brain through the parallel computation and the memory function. Various bio-inspired \u27in-memory computing\u27 (IMC) devices were developed during the past decades, such as synaptic transistors for artificial synapses. By integrating with specific sensors, neuromorphic sensing systems are achievable with the bio-inspired signal perception function. A signal perception process is possible by a combination of stimuli sensing, signal conversion/transmission, and signal processing. However, most neuromorphic sensing systems were demonstrated without signal conversion/transmission functions. Therefore, those cannot fully mimic the function provides by the sensory neuron in the biological system. This thesis aims to design a neuromorphic sensing system with a complete function as biological sensory neurons. To reach such a target, 3D printed sensors, electrical oscillators, and synaptic transistors were developed as functions of artificial receptors, artificial neurons, and artificial synapses, respectively. Moreover, since the 3D printing technology has demonstrated a facile process due to fast prototyping, the proposed 3D neuromorphic sensing system was designed as a 3D integrated structure and fabricated by 3D printing technologies. A novel multi-axis robot 3D printing system was also utilized to increase the fabrication efficiency with the capability of printing on vertical and tilted surfaces seamlessly. Furthermore, the developed 3D neuromorphic system was easily adapted to the application of tactile sensing. A portable neuromorphic system was integrated with a tactile sensing system for the intelligent tactile sensing application of the humanoid robot. Finally, the bio-inspired reflex arc for the unconscious response was also demonstrated by training the neuromorphic tactile sensing system

Action Tubelet Detector for Spatio-Temporal Action Localization

International audienceCurrent state-of-the-art approaches for spatio-temporal action detection rely on detections at the frame level that are then linked or tracked across time. In this paper, we leverage the temporal continuity of videos instead of operating at the frame level. We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, ie., sequences of bounding boxes with associated scores. The same way state-of-the-art object detectors rely on anchor boxes, our ACT-detector is based on anchor cuboids. We build upon the state-of-the-art SSD framework. Convolutional features are extracted for each frame, while scores and regressions are based on the temporal stacking of these features, thus exploiting information from a sequence. Our experimental results show that leveraging sequences of frames significantly improves detection performance over using individual frames. The gain of our tubelet detector can be explained by both more relevant scores and more precise localization. Our ACT-detector outperforms the state of the art methods for frame-mAP and video-mAP on the J-HMDB and UCF-101 datasets, in particular at high overlap thresholds

Learning Pregrasp Manipulation of Objects from Ungraspable Poses

In robotic grasping, objects are often occluded in ungraspable configurations such that no pregrasp pose can be found, eg large flat boxes on the table that can only be grasped from the side. Inspired by humans' bimanual manipulation, eg one hand to lift up things and the other to grasp, we address this type of problems by introducing pregrasp manipulation - push and lift actions. We propose a model-free Deep Reinforcement Learning framework to train control policies that utilize visual information and proprioceptive states of the robot to autonomously discover robust pregrasp manipulation. The robot arm learns to first push the object towards a support surface and establishes a pivot to lift up one side of the object, thus creating a clearance between the object and the table for possible grasping solutions. Furthermore, we show the effectiveness of our proposed learning framework in training robust pregrasp policies that can directly transfer from simulation to real hardware through suitable design of training procedures, state, and action space. Lastly, we evaluate the effectiveness and the generalisation ability of the learned policies in real-world experiments, and demonstrate pregrasp manipulation of objects with various size, shape, weight, and surface friction.Comment: 8 pages open access version for ICRA2020 6 pages acceptance pape

Object Recognition and Localization : the Role of Tactile Sensors

Tactile sensors, because of their intrinsic insensitivity to lighting conditions and water turbidity, provide promising opportunities for augmenting the capabilities of vision sensors in applications involving object recognition and localization. This thesis presents two approaches for haptic object recognition and localization for ground and underwater environments. The first approach called Batch Ransac and Iterative Closest Point augmented Sequential Filter (BRICPSF) is based on an innovative combination of a sequential filter, Iterative-Closest-Point algorithm, and a feature-based Random Sampling and Consensus (RANSAC) algorithm for database matching. It can handle a large database of 3D-objects of complex shapes and performs a complete six-degree-of-freedom localization of static objects. The algorithms are validated by experimentation in simulation and using actual hardware. To our knowledge this is the first instance of haptic object recognition and localization in underwater environments. The second approach is biologically inspired, and provides a close integration between exploration and recognition. An edge following exploration strategy is developed that receives feedback from the current state of recognition. A recognition by parts approach is developed which uses BRICPSF for object part recognition. Object exploration is either directed to explore a part until it is successfully recognized, or is directed towards new parts to endorse the current recognition belief. This approach is validated by simulation experiments