Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks

One of the challenges in modeling cognitive events from electroencephalogram (EEG) data is finding representations that are invariant to inter- and intra-subject differences, as well as to inherent noise associated with such data. Herein, we propose a novel approach for learning such representations from multi-channel EEG time-series, and demonstrate its advantages in the context of mental load classification task. First, we transform EEG activities into a sequence of topology-preserving multi-spectral images, as opposed to standard EEG analysis techniques that ignore such spatial information. Next, we train a deep recurrent-convolutional network inspired by state-of-the-art video classification to learn robust representations from the sequence of images. The proposed approach is designed to preserve the spatial, spectral, and temporal structure of EEG which leads to finding features that are less sensitive to variations and distortions within each dimension. Empirical evaluation on the cognitive load classification task demonstrated significant improvements in classification accuracy over current state-of-the-art approaches in this field.Comment: To be published as a conference paper at ICLR 201

Robust Modeling of Epistemic Mental States

This work identifies and advances some research challenges in the analysis of facial features and their temporal dynamics with epistemic mental states in dyadic conversations. Epistemic states are: Agreement, Concentration, Thoughtful, Certain, and Interest. In this paper, we perform a number of statistical analyses and simulations to identify the relationship between facial features and epistemic states. Non-linear relations are found to be more prevalent, while temporal features derived from original facial features have demonstrated a strong correlation with intensity changes. Then, we propose a novel prediction framework that takes facial features and their nonlinear relation scores as input and predict different epistemic states in videos. The prediction of epistemic states is boosted when the classification of emotion changing regions such as rising, falling, or steady-state are incorporated with the temporal features. The proposed predictive models can predict the epistemic states with significantly improved accuracy: correlation coefficient (CoERR) for Agreement is 0.827, for Concentration 0.901, for Thoughtful 0.794, for Certain 0.854, and for Interest 0.913.Comment: Accepted for Publication in Multimedia Tools and Application, Special Issue: Socio-Affective Technologie

Physics-Based Approaches For Structural Health Monitoring And Nondestructive Evaluation With Ultrasonic Guided Waves

The engineering infrastructures have a growing demand for damage monitoring systems to avoid any potential risk of failure. Proper damage monitoring solutions are of a great interest to this growing demand. The structural health monitoring (SHM) and nondestructive evaluation (NDE) offer appropriate online and offline damage monitoring solutions for a variety of mechanical and civil infrastructures that includes unmanned aerial vehicles (UAV), spaceships, commercial aircraft, ground transportation, wind turbines, nuclear spent fuel storage tanks, bridges, naval ships, and submarines. The fundamentals of the ultrasonic SHM and NDE consist of multi-disciplinary fields. The dissertation addresses SHM and NDE using ultrasonic guided waves, with an emphasis on the development of an analytical solution for non-axisymmetric guided wave propagation, multiphysics simulation, and experimental study of acoustic emission from the structural fatigue damage. An analytical solution for non-axisymmetric coupled guided wave propagation in plate-like structures was developed based on the equations of motion and elasticity relations. A general non-axisymmetric solution of guided wave propagation inplateis needed to analyze the guided wave-scatter from non-axisymmetric damage as encountered in practice. Under non-axisymmetric conditions, the problem is highly coupled and no potential based analytical solution has been reported in the literature so far. Helmholtz decomposition theorem was applied to the Navier-Lame equations that yielded a set of four coupled partial differential equations in four unknowns, the scalar potential Φ and the three components of the vector potential Hr, Hz, HΘ. A fourth equation, the ‘gauge condition’ was then added to the decomposition. A particular interpretation of the gauge condition is proposed. Our proposed approach decouples the governing equations and reduced the number of unknowns from four to three thus allowing one to express the solution in an elegant straight-forward way. The Rayleigh-Lamb characteristic equations were recovered and a general normal-modes expression for the solution was obtained. A hybrid global analytical and local finite element method was used to solve a practical aerospace rivet hole crack detection. The scatter cube of complex-valued wave damage interaction coefficients (WDICs) was developed to analyze any rivet hole of a multiple-rivet-hole lap joint system. It had been shown that not all parameters such as actuator-sensor locations, and frequencies were equally sensitive to the damage scatter. The optimum combination of parameters could better detect the crack in the rivet hole. The simulated time domain signals were produced for the optimum combination of parameters. Multiphysics simulations for fatigue crack generated acoustic emission (AE) were performed and the results were validated by the experiments. A novel application of inexpensive piezoelectric wafer active sensors (PWAS) has been explored. It has been shown that PWAS transducers successfully captured the fatigue-crack generated acoustic emissions in the thin plate-like aerospace materials. PWAS performance was compared with existing commercial AE sensors. It was found that PWAS captured richer frequency content than the existing AE sensors. Various AE waveform signatures were found from the fatigue crack advancement during the fatigue load evolution. Some AE waveform signatures were found to be related to the fatigue-crack extension while some of them were related to the fatigue-crack fretting, rubbing, and clapping. This observation was confirmed viii by synchronizing the fatigue loading with AE measurement by the same AE instrument. The in-situ microscopic measurement was performed during fatigue loading in MTS which provided the insights of the AE waveform evolution. It was hypothesized that the crack length estimation could be related the AE waveform signatures. FEM simulations and experiments were conducted using laser Doppler vibrometer (LDV) to verify our hypothesis. Two case studies are discussed showing the implementation of SHM and NDE approach in practical applications: (1) horizontal crack detection, size, and shape estimation in stiffened structures, (2) impact damage detection in manufactured aerospace composite structures. The dissertation finishes with conclusions, major contributions, and suggestions for future work

Eigen-CAM: Class Activation Map using Principal Components

Deep neural networks are ubiquitous due to the ease of developing models and their influence on other domains. At the heart of this progress is convolutional neural networks (CNNs) that are capable of learning representations or features given a set of data. Making sense of such complex models (i.e., millions of parameters and hundreds of layers) remains challenging for developers as well as the end-users. This is partially due to the lack of tools or interfaces capable of providing interpretability and transparency. A growing body of literature, for example, class activation map (CAM), focuses on making sense of what a model learns from the data or why it behaves poorly in a given task. This paper builds on previous ideas to cope with the increasing demand for interpretable, robust, and transparent models. Our approach provides a simpler and intuitive (or familiar) way of generating CAM. The proposed Eigen-CAM computes and visualizes the principle components of the learned features/representations from the convolutional layers. Empirical studies were performed to compare the Eigen-CAM with the state-of-the-art methods (such as Grad-CAM, Grad-CAM++, CNN-fixations) by evaluating on benchmark datasets such as weakly-supervised localization and localizing objects in the presence of adversarial noise. Eigen-CAM was found to be robust against classification errors made by fully connected layers in CNNs, does not rely on the backpropagation of gradients, class relevance score, maximum activation locations, or any other form of weighting features. In addition, it works with all CNN models without the need to modify layers or retrain models. Empirical results show up to 12% improvement over the best method among the methods compared on weakly supervised object localization.Comment: 7 pages, 4 figure

An Intelligent System Approach to the Dynamic Hybrid Robot Control

The objective of this study was to solve the robot dynamic hybrid control problem using intelligent computational processes. In the course of problem- solving, biologically inspired models were used. This was because a robot can be seen as a physical intelligent system which interacts with the real world environment by means of its sensors and actuators. In the robot hybrid control method the neural networks, fuzzy logics and randomization strategies were used. To derive a complete intelligent state-of-the-art hybrid control system, several experiments were conducted in the study. Firstly an algorithm was formulated that can estimate the attracting basin boundary for a stable equilibrium point of a robot's kinematic nonlinear system. From this point the Artificial Neural Networks (ANN) based solution approach was verified for the inverse kinematics solution. Secondly, for the intelligent trajectory generation approach, the segmented tree neural networks for each link (inverse kinematics solution) and the randomness with fuzziness (coping the unstructured environment from the cost function) were used. A one-pass smoothing algorithm was used to generate a practical smooth trajectory path in near real time. Finally, for the hybrid control system the task was decomposed into several individual intelligent control agents, where the task space was split into the position-controlled subspaces, the force-controlled subspaces and the uncertain hyper plane identification subspaces. The problem was considered as a blind-tracking task by a human

Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures

Although speech and gesture recognition has been studied extensively, all the successful attempts of combining them in the unified framework were semantically motivated, e.g., keyword-gesture cooccurrence. Such formulations inherited the complexity of natural language processing. This paper presents a Bayesian formulation that uses a phenomenon of gesture and speech articulation for improving accuracy of automatic recognition of continuous coverbal gestures. The prosodic features from the speech signal were coanalyzed with the visual signal to learn the prior probability of co-occurrence of the prominent spoken segments with the particular kinematical phases of gestures. It was found that the above co-analysis helps in detecting and disambiguating visually small gestures, which subsequently improves the rate of continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected from the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.Comment: Alternative see: http://vision.cse.psu.edu/kettebek/academ/publications.ht

Using the Gauge Condition to Simplify The Elastodynamic Analysis of Guided Wave Propagation

In this article, gauge condition in elastodynamics is explored more to revive its potential capability of simplifying wave propagation problems in elastic medium. The inception of gauge condition in elastodynamics happens from the Navier-Lame equations upon application of Helmholtz theorem. In order to solve the elastic wave problems by potential function approach, the gauge condition provides the necessary conditions for the potential functions. The gauge condition may be considered as the superposition of the separate gauge conditions of Lamb waves and shear horizontal (SH) guided waves respectively, and thus, it may be resolved into corresponding gauges of Lamb waves and SH waves. The manipulation and proper choice of the gauge condition does not violate the classical solutions of elastic waves in plates; rather, it simplifies the problems. The gauge condition allows to obtain the analytical solution of complicated problems in a simplified manner