Video Stabilisation Based on Spatial Transformer Networks

User-Generated Content is normally recorded with mobile phones by non-professionals, which leads to a low viewing experience due to artifacts such as jitter and blur. Other jittery videos are those recorded with mounted cameras or moving platforms. In these scenarios, Digital Video Stabilization (DVS) has been utilized, to create high quality, professional level videos. In the industry and academia, there are a number of traditional and Deep Learning (DL)-based DVS systems, however both approaches have limitations: the former struggles to extract and track features in a number of scenarios, and the latter struggles with camera path smoothing, a hard problem to define in this context. On the other hand, traditional methods have shown good performance in smoothing camera path whereas DL methods are effective in feature extraction, tracking, and motion parameter estimation. Hence, to the best of our knowledge the available DVS systems struggle to stabilize videos in a wide variety of scenarios, especially with high motion and certain scene content, such as textureless areas, dark scenes, close object, lack of depth, amongst others. Another challenge faced by current DVS implementations is the resulting artifacts that such systems add to the stabilized videos, degrading the viewing experience. These artifacts are mainly distortion, blur, zoom, and ghosting effects. In this thesis, we utilize the strengths of Deep Learning and traditional methods for video stabilization. Our approach is robust to a wide variety of scene content and camera motion, and avoids adding artifacts to the stabilized video. First, we provide a dataset and evaluation framework for Deep Learning-based DVS. Then, we present our image alignment module, which contains a Spatial Transformer Network (STN). Next, we leverage this module to propose a homography-based video stabilization system. Aiming at avoiding blur and distortion caused by homographies, our next proposal is a translation-based video stabilization method, which contains Exponential Weighted Moving Averages (EWMAs) to smooth the camera path. Finally, instead of using EWMAs, we study the utilization of filters in our approach. In this case, we compare a number of filters and choose the filters with best performance. Since the quality of experience of a viewer does not only consist of video stability, but also of blur and distortion, we consider it is a good trade off to allow some jitter left on the video while avoiding adding distortion and blur. In all three cases, we show that this approach pays off, since our systems ourperform the state-of-the-art proposals

Recent Advances in Signal Processing

The signal processing task is a very critical issue in the majority of new technological inventions and challenges in a variety of applications in both science and engineering fields. Classical signal processing techniques have largely worked with mathematical models that are linear, local, stationary, and Gaussian. They have always favored closed-form tractability over real-world accuracy. These constraints were imposed by the lack of powerful computing tools. During the last few decades, signal processing theories, developments, and applications have matured rapidly and now include tools from many areas of mathematics, computer science, physics, and engineering. This book is targeted primarily toward both students and researchers who want to be exposed to a wide variety of signal processing techniques and algorithms. It includes 27 chapters that can be categorized into five different areas depending on the application at hand. These five categories are ordered to address image processing, speech processing, communication systems, time-series analysis, and educational packages respectively. The book has the advantage of providing a collection of applications that are completely independent and self-contained; thus, the interested reader can choose any chapter and skip to another without losing continuity

Autocalibrating vision guided navigation of unmanned air vehicles via tactical monocular cameras in GPS denied environments

This thesis presents a novel robotic navigation strategy by using a conventional tactical monocular camera, proving the feasibility of using a monocular camera as the sole proximity sensing, object avoidance, mapping, and path-planning mechanism to fly and navigate small to medium scale unmanned rotary-wing aircraft in an autonomous manner. The range measurement strategy is scalable, self-calibrating, indoor-outdoor capable, and has been biologically inspired by the key adaptive mechanisms for depth perception and pattern recognition found in humans and intelligent animals (particularly bats), designed to assume operations in previously unknown, GPS-denied environments. It proposes novel electronics, aircraft, aircraft systems, systems, and procedures and algorithms that come together to form airborne systems which measure absolute ranges from a monocular camera via passive photometry, mimicking that of a human-pilot like judgement. The research is intended to bridge the gap between practical GPS coverage and precision localization and mapping problem in a small aircraft. In the context of this study, several robotic platforms, airborne and ground alike, have been developed, some of which have been integrated in real-life field trials, for experimental validation. Albeit the emphasis on miniature robotic aircraft this research has been tested and found compatible with tactical vests and helmets, and it can be used to augment the reliability of many other types of proximity sensors

Bayesian Methods for Gas-Phase Tomography

Gas-phase tomography refers to a set of techniques that determine the 2D or 3D distribution of a target species in a jet, plume, or flame using measurements of light, made around the boundary of a flow area. Reconstructed quantities may include the concentration of one or more species, temperature, pressure, and optical density, among others. Tomography is increasingly used to study fundamental aspects of turbulent combustion and monitor emissions for regulatory compliance. This thesis develops statistical methods to improve gas-phase tomography and reports two novel experimental applications. Tomography is an inverse problem, meaning that a forward model (calculating measurements of light for a known distribution of gas) is inverted to estimate the model parameters (transforming experimental data into a gas distribution). The measurement modality varies with the problem geometry and objective of the experiment. For instance, transmittance data from an array of laser beams that transect a jet may be inverted to recover 2D fields of concentration and temperature; and multiple high-resolution images of a flame, captured from different angles, are used to reconstruct wrinkling of the 3D reacting zone. Forward models for gas-phase tomography modalities share a common mathematical form, that of a Fredholm integral equation of the first-kind (IFK). The inversion of coupled IFKs is necessarily ill-posed, however, meaning that solutions are either unstable or non-unique. Measurements are thus insufficient in themselves to generate a realistic image of the gas and additional information must be incorporated into the reconstruction procedure. Statistical inversion is an approach to inverse problems in which the measurements, experimental parameters, and quantities of interest are treated as random variables, characterized by a probability distribution. These distributions reflect uncertainty about the target due to fluctuations in the flow field, noise in the data, errors in the forward model, and the ill-posed nature of reconstruction. The Bayesian framework for tomography features a likelihood probability density function (pdf), which describes the chance of observing a measurement for a given distribution of gas, and prior pdf, which assigns a relative plausibility to candidate distributions based on assumptions about the flow physics. Bayes’ equation updates information about the target in response to measurement data, combining the likelihood and prior functions to form a posterior pdf. The posterior is usually summarized by the maximum a posteriori (MAP) estimate, which is the most likely distribution of gas for a set of data, subject to the effects of noise, model errors, and prior information. The framework can be used to estimate credibility intervals for a reconstruction and the form of Bayes’ equation suggests procedures for improving gas tomography. The accuracy of reconstructions depends on the information content of the data, which is a function of the experimental design, as well as the specificity and validity of the prior. This thesis employs theoretical arguments and experimental measurements of scalar fluctuations to justify joint-normal likelihood and prior pdfs for gas-phase tomography. Three methods are introduced to improve each stage of the inverse problem: to develop priors, design optimal experiments, and select a discretization scheme. First, a self-similarity analysis of turbulent jets—common targets in gas tomography—is used to construct an advanced prior, informed by an estimate of the jet’s spatial covariance. Next, a Bayesian objective function is proposed to optimize beam positions in limited-data arrays, which are necessary in scenarios where optical access to the flow area is restricted. Finally, a Bayesian expression for model selection is derived from the joint-normal pdfs and employed to select a mathematical basis to reconstruct a flow. Extensive numerical evidence is presented to validate these methods. The dissertation continues with two novel experiments, conducted in a Bayesian way. Broadband absorption tomography is a new technique intended for quantitative emissions detection from spectrally-convolved absorption signals. Theoretical foundations for the diagnostic are developed and the results of a proof-of-concept emissions detection experiment are reported. Lastly, background-oriented schlieren (BOS) tomography is applied to combustion for the first time. BOS tomography employs measurements of beam steering to reconstruct a fluid’s optical density field, which can be used to infer temperature and density. The application of BOS tomography to flame imaging sets the stage for instantaneous 3D combustion thermometry. Numerical and experimental results reported in this thesis support a Bayesian approach to gas-phase tomography. Bayesian tomography makes the role of prior information explicit, which can be leveraged to optimize reconstructions and design better imaging systems in support of research on fluid flow and combustion dynamics

Directional Estimation for Robotic Beating Heart Surgery

In robotic beating heart surgery, a remote-controlled robot can be used to carry out the operation while automatically canceling out the heart motion. The surgeon controlling the robot is shown a stabilized view of the heart. First, we consider the use of directional statistics for estimation of the phase of the heartbeat. Second, we deal with reconstruction of a moving and deformable surface. Third, we address the question of obtaining a stabilized image of the heart

Directional Estimation for Robotic Beating Heart Surgery

In robotic beating heart surgery, a remote-controlled robot can be used to carry out the operation while automatically canceling out the heart motion. The surgeon controlling the robot is shown a stabilized view of the heart. First, we consider the use of directional statistics for estimation of the phase of the heartbeat. Second, we deal with reconstruction of a moving and deformable surface. Third, we address the question of obtaining a stabilized image of the heart

Dynamic Thermal Imaging for Intraoperative Monitoring of Neuronal Activity and Cortical Perfusion

Neurosurgery is a demanding medical discipline that requires a complex interplay of several neuroimaging techniques. This allows structural as well as functional information to be recovered and then visualized to the surgeon. In the case of tumor resections this approach allows more fine-grained differentiation of healthy and pathological tissue which positively influences the postoperative outcome as well as the patient's quality of life. In this work, we will discuss several approaches to establish thermal imaging as a novel neuroimaging technique to primarily visualize neural activity and perfusion state in case of ischaemic stroke. Both applications require novel methods for data-preprocessing, visualization, pattern recognition as well as regression analysis of intraoperative thermal imaging. Online multimodal integration of preoperative and intraoperative data is accomplished by a 2D-3D image registration and image fusion framework with an average accuracy of 2.46 mm. In navigated surgeries, the proposed framework generally provides all necessary tools to project intraoperative 2D imaging data onto preoperative 3D volumetric datasets like 3D MR or CT imaging. Additionally, a fast machine learning framework for the recognition of cortical NaCl rinsings will be discussed throughout this thesis. Hereby, the standardized quantification of tissue perfusion by means of an approximated heating model can be achieved. Classifying the parameters of these models yields a map of connected areas, for which we have shown that these areas correlate with the demarcation caused by an ischaemic stroke segmented in postoperative CT datasets. Finally, a semiparametric regression model has been developed for intraoperative neural activity monitoring of the somatosensory cortex by somatosensory evoked potentials. These results were correlated with neural activity of optical imaging. We found that thermal imaging yields comparable results, yet doesn't share the limitations of optical imaging. In this thesis we would like to emphasize that thermal imaging depicts a novel and valid tool for both intraoperative functional and structural neuroimaging