A neural network approach to audio-assisted movie dialogue detection

A novel framework for audio-assisted dialogue detection based on indicator functions and neural networks is investigated. An indicator function defines that an actor is present at a particular time instant. The cross-correlation function of a pair of indicator functions and the magnitude of the corresponding cross-power spectral density are fed as input to neural networks for dialogue detection. Several types of artificial neural networks, including multilayer perceptrons, voted perceptrons, radial basis function networks, support vector machines, and particle swarm optimization-based multilayer perceptrons are tested. Experiments are carried out to validate the feasibility of the aforementioned approach by using ground-truth indicator functions determined by human observers on 6 different movies. A total of 41 dialogue instances and another 20 non-dialogue instances is employed. The average detection accuracy achieved is high, ranging between 84.78%±5.499% and 91.43%±4.239%

Path Planning and Energy Efficiency of Heterogeneous Mobile Robots Using Cuckoo–Beetle Swarm Search Algorithms with Applications in UGV Obstacle Avoidance

In this paper, a new meta-heuristic path planning algorithm, the cuckoo–beetle swarm search (CBSS) algorithm, is introduced to solve the path planning problems of heterogeneous mobile robots. Traditional meta-heuristic algorithms, e.g., genetic algorithms (GA), particle swarm search (PSO), beetle swarm optimization (BSO), and cuckoo search (CS), have problems such as the tenancy to become trapped in local minima because of premature convergence and a weakness in global search capability in path planning. Note that the CBSS algorithm imitates the biological habits of cuckoo and beetle herds and thus has good robustness and global optimization ability. In addition, computer simulations verify the accuracy, search speed, energy efficiency and stability of the CBSS algorithm. The results of the real-world experiment prove that the proposed CBSS algorithm is much better than its counterparts. Finally, the CBSS algorithm is applied to 2D path planning and 3D path planning in heterogeneous mobile robots. In contrast to its counterparts, the CBSS algorithm is guaranteed to find the shortest global optimal path in different sizes and types of maps

Event Detection in Eye-Tracking Data for Use in Applications with Dynamic Stimuli

This doctoral thesis has signal processing of eye-tracking data as its main theme. An eye-tracker is a tool used for estimation of the point where one is looking. Automatic algorithms for classification of different types of eye movements, so called events, form the basis for relating the eye-tracking data to cognitive processes during, e.g., reading a text or watching a movie. The problems with the algorithms available today are that there are few algorithms that can handle detection of events during dynamic stimuli and that there is no standardized procedure for how to evaluate the algorithms. This thesis comprises an introduction and four papers describing methods for detection of the most common types of eye movements in eye-tracking data and strategies for evaluation of such methods. The most common types of eye movements are fixations, saccades, and smooth pursuit movements. In addition to these eye movements, the event post-saccadic oscillations, (PSO), is considered. The eye-tracking data in this thesis are recorded using both high- and low-speed eye-trackers. The first paper presents a method for detection of saccades and PSO. The saccades are detected using the acceleration signal and three specialized criteria based on directional information. In order to detect PSO, the interval after each saccade is modeled and the parameters of the model are used to determine whether PSO are present or not. The algorithm was evaluated by comparing the detection results to manual annotations and to the detection results of the most recent PSO detection algorithm. The results show that the algorithm is in good agreement with annotations, and has better performance than the compared algorithm. In the second paper, a method for separation of fixations and smooth pursuit movements is proposed. In the intervals between the detected saccades/PSO, the algorithm uses different spatial scales of the position signal in order to separate between the two types of eye movements. The algorithm is evaluated by computing five different performance measures, showing both general and detailed aspects of the discrimination performance. The performance of the algorithm is compared to the performance of a velocity and dispersion based algorithm, (I-VDT), to the performance of an algorithm based on principle component analysis, (I-PCA), and to manual annotations by two experts. The results show that the proposed algorithm performs considerably better than the compared algorithms. In the third paper, a method based on eye-tracking signals from both eyes is proposed for improved separation of fixations and smooth pursuit movements. The method utilizes directional clustering of the eye-tracking signals in combination with binary filters taking both temporal and spatial aspects of the eye-tracking signal into account. The performance of the method is evaluated using a novel evaluation strategy based on automatically detected moving objects in the video stimuli. The results show that the use of binocular information for separation of fixations and smooth pursuit movements is advantageous in static stimuli, without impairing the algorithm's ability to detect smooth pursuit movements in video and moving dot stimuli. The three first papers in this thesis are based on eye-tracking signals recorded using a stationary eye-tracker, while the fourth paper uses eye-tracking signals recorded using a mobile eye-tracker. In mobile eye-tracking, the user is allowed to move the head and the body, which affects the recorded data. In the fourth paper, a method for compensation of head movements using an inertial measurement unit, (IMU), combined with an event detector for lower sampling rate data is proposed. The event detection is performed by combining information from the eye-tracking signals with information about objects extracted from the scene video of the mobile eye-tracker. The results show that by introducing head movement compensation and information about detected objects in the scene video in the event detector, improved classification can be achieved. In summary, this thesis proposes an entire methodological framework for robust event detection which performs better than previous methods when analyzing eye-tracking signals recorded during dynamic stimuli, and also provides a methodology for performance evaluation of event detection algorithms

A comparison of post-saccadic oscillations in European-Born and China-Born British University Undergraduates

Previous research has revealed that people from different genetic, racial, biological, and/or cultural backgrounds may display fundamental differences in eye-tracking behavior. These differences may have a cognitive origin or they may be at a lower level within the neurophysiology of the oculomotor network, or they may be related to environment factors. In this paper we investigated one of the physiological aspects of eye movements known as post-saccadic oscillations and we show that this type of eye movement is very different between two different populations. We compared the post-saccadic oscillations recorded by a video-based eye tracker between two groups of participants: European-born and Chinese-born British students. We recorded eye movements from a group of 42 Caucasians defined as White British or White Europeans and 52 Chinese-born participants all with ages ranging from 18 to 36 during a prosaccade task. The post-saccadic oscillations were extracted from the gaze data which was compared between the two groups in terms of their first overshoot and undershoot. The results revealed that the shape of the post-saccadic oscillations varied significantly between the two groups which may indicate a difference in a multitude of genetic, cultural, physiologic, anatomical or environmental factors. We further show that the differences in the post-saccadic oscillations could influence the oculomotor characteristics such as saccade duration. We conclude that genetic, racial, biological, and/or cultural differences can affect the morphology of the eye movement data recorded and should be considered when studying eye movements and oculomotor fixation and saccadic behaviors

Spatio-temporal prediction of wind fields

Short-term wind and wind power forecasts are required for the reliable and economic operation of power systems with significant wind power penetration. This thesis presents new statistical techniques for producing forecasts at multiple locations using spatiotemporal information. Forecast horizons of up to 6 hours are considered for which statistical methods outperform physical models in general. Several methods for producing hourly wind speed and direction forecasts from 1 to 6 hours ahead are presented in addition to a method for producing five-minute-ahead probabilistic wind power forecasts. The former have applications in areas such as energy trading and defining reserve requirements, and the latter in power system balancing and wind farm control. Spatio-temporal information is captured by vector autoregressive (VAR) models that incorporate wind direction by modelling the wind time series using complex numbers. In a further development, the VAR coefficients are replaced with coefficient functions in order to capture the dependence of the predictor on external variables, such as the time of year or wind direction. The complex-valued approach is found to produce accurate speed predictions, and the conditional predictors offer improved performance with little additional computational cost. Two non-linear algorithms have been developed for wind forecasting. In the first, the predictor is derived from an ensemble of particle swarm optimised candidate solutions. This approach is low cost and requires very little training data but fails to capitalise on spatial information. The second approach uses kernelised forms of popular linear algorithms which are shown to produce more accurate forecasts than their linear equivalents for multi-step-ahead prediction. Finally, very-short-term wind power forecasting is considered. Five-minute-ahead parametric probabilistic forecasts are produced by modelling the predictive distribution as logit-normal and forecasting its parameters using a sparse-VAR (sVAR) approach. Development of the sVAR is motivated by the desire to produce forecasts on a large spatial scale, i.e. hundreds of locations, which is critical during periods of high instantaneous wind penetration.Short-term wind and wind power forecasts are required for the reliable and economic operation of power systems with significant wind power penetration. This thesis presents new statistical techniques for producing forecasts at multiple locations using spatiotemporal information. Forecast horizons of up to 6 hours are considered for which statistical methods outperform physical models in general. Several methods for producing hourly wind speed and direction forecasts from 1 to 6 hours ahead are presented in addition to a method for producing five-minute-ahead probabilistic wind power forecasts. The former have applications in areas such as energy trading and defining reserve requirements, and the latter in power system balancing and wind farm control. Spatio-temporal information is captured by vector autoregressive (VAR) models that incorporate wind direction by modelling the wind time series using complex numbers. In a further development, the VAR coefficients are replaced with coefficient functions in order to capture the dependence of the predictor on external variables, such as the time of year or wind direction. The complex-valued approach is found to produce accurate speed predictions, and the conditional predictors offer improved performance with little additional computational cost. Two non-linear algorithms have been developed for wind forecasting. In the first, the predictor is derived from an ensemble of particle swarm optimised candidate solutions. This approach is low cost and requires very little training data but fails to capitalise on spatial information. The second approach uses kernelised forms of popular linear algorithms which are shown to produce more accurate forecasts than their linear equivalents for multi-step-ahead prediction. Finally, very-short-term wind power forecasting is considered. Five-minute-ahead parametric probabilistic forecasts are produced by modelling the predictive distribution as logit-normal and forecasting its parameters using a sparse-VAR (sVAR) approach. Development of the sVAR is motivated by the desire to produce forecasts on a large spatial scale, i.e. hundreds of locations, which is critical during periods of high instantaneous wind penetration