Video anatomy : spatial-temporal video profile

Indiana University-Purdue University Indianapolis (IUPUI)A massive amount of videos are uploaded on video websites, smooth video browsing, editing, retrieval, and summarization are demanded. Most of the videos employ several types of camera operations for expanding field of view, emphasizing events, and expressing cinematic effect. To digest heterogeneous videos in video websites and databases, video clips are profiled to 2D image scroll containing both spatial and temporal information for video preview. The video profile is visually continuous, compact, scalable, and indexing to each frame. This work analyzes the camera kinematics including zoom, translation, and rotation, and categorize camera actions as their combinations. An automatic video summarization framework is proposed and developed. After conventional video clip segmentation and video segmentation for smooth camera operations, the global flow field under all camera actions has been investigated for profiling various types of video. A new algorithm has been designed to extract the major flow direction and convergence factor using condensed images. Then this work proposes a uniform scheme to segment video clips and sections, sample video volume across the major flow, compute flow convergence factor, in order to obtain an intrinsic scene space less influenced by the camera ego-motion. The motion blur technique has also been used to render dynamic targets in the profile. The resulting profile of video can be displayed in a video track to guide the access to video frames, help video editing, and facilitate the applications such as surveillance, visual archiving of environment, video retrieval, and online video preview

Developing object detection, tracking and image mosaicing algorithms for visual surveillance

Visual surveillance systems are becoming increasingly important in the last decades due to proliferation of cameras. These systems have been widely used in scientific, commercial and end-user applications where they can store, extract and infer huge amount of information automatically without human help. In this thesis, we focus on developing object detection, tracking and image mosaicing algorithms for a visual surveillance system. First, we review some real-time object detection algorithms that exploit motion cue and enhance one of them that is suitable for use in dynamic scenes. This algorithm adopts a nonparametric probabilistic model over the whole image and exploits pixel adjacencies to detect foreground regions under even small baseline motion. Then we develop a multiple object tracking algorithm which utilizes this algorithm as its detection step. The algorithm analyzes multiple object interactions in a probabilistic framework using virtual shells to track objects in case of severe occlusions. The final part of the thesis is devoted to an image mosaicing algorithm that stitches ordered images to create a large and visually attractive mosaic for large sequence of images. The proposed mosaicing method eliminates nonlinear optimization techniques with the capability of real-time operation on large datasets. Experimental results show that developed algorithms work quite successfully in dynamic and cluttered environments with real-time performance

Parallax-Tolerant Image Stitching

Parallax handling is a challenging task for image stitch-ing. This paper presents a local stitching method to handle parallax based on the observation that input images do not need to be perfectly aligned over the whole overlapping re-gion for stitching. Instead, they only need to be aligned in a way that there exists a local region where they can be seam-lessly blended together. We adopt a hybrid alignment model that combines homography and content-preserving warp-ing to provide flexibility for handling parallax and avoiding objectionable local distortion. We then develop an efficient randomized algorithm to search for a homography, which, combined with content-preserving warping, allows for op-timal stitching. We predict how well a homography enables plausible stitching by finding a plausible seam and using the seam cost as the quality metric. We develop a seam finding method that estimates a plausible seam from only roughly aligned images by considering both geometric alignment and image content. We then pre-align input images using the optimal homography and further use content-preserving warping to locally refine the alignment. We finally compose aligned images together using a standard seam-cutting al-gorithm and a multi-band blending algorithm. Our exper-iments show that our method can effectively stitch images with large parallax that are difficult for existing methods. 1

Historical Land use/Land cover classification and its change detection mapping using Different Remotely Sensed Data from LANDSAT (MSS, TM and ETM+) and Terra (ASTER) sensors: a case study of the Euphrates River Basin in Syria with focus on agricultural irrigation projects

This thesis deals spatially and regionally with the natural boundaries of the Euphrates River Basin (ERB) in Syria. Scientifically, the research covers the application of remote sensing science (optical remote sensing: LANDSAT-MSS, TM, and ETM+; and TERRA: ASTER); and methodologically, in Land Use/Land Cover (LULC) classification and mapping, automatically and/or semi-automatically; in LULC-change detection; and finally in the mapping of historical irrigation and agricultural projects for the extraction of differing crop types and the estimation of their areas. With regard to time, the work is based on the years 1975, 1987, 2005 and 2007. Initially, preprocessing of the satellite data (geometric- and radiometric- processing, image enhancement, best bands composite selection, transformation, mosaicing and finally subsetting) was carried out. Then, the Land Use/Land Cover Classification System (LCCS) of the Food and Agriculture Organization (FAO) was chosen. The following steps were followed in LULC- classification and change detection mapping: visual interpretation in addition to digital image processing techniques; pixel-based classification methods; unsupervised classification: ISODATA-method; and supervised classification and multistage supervised approaches using the algorithms: Maximum Likelihood Classifier (MLC), Neural Network classifier (NN) and Support Vector Machines (SVM). These were trialed on a test area to determine the optimized classification approach/algorithm for application on the whole study area (ERB) based on the available imagery. Pre- and post- classification change detection methods (comparison approaches) were used to detect changes in land use/land cover-classes (for the years 1975, 1987 and 2007) in the study area. The remote sensing methods show a high potential in mapping historical and present land use/land cover classes and its changes over time. Significant results are also possible for agricultural crop classification in relatively large regional areas (the ERB in Syria is almost 50,335 km²). Change trends in the study area and period was characterized by land-intensive agricultural expansion. The rapid, more labor- and capital- intensive growth in the agricultural sector was enabled by the introduction of fertilizer, improved access to rural roads and markets, and the expansion of the government irrigation projects. Irrigated areas increased 148 % in the past 32 years from 249,681 ha in 1975 to 596,612 ha in 2007

The Impact of Sensor Characteristics and Data Availability on Remote Sensing Based Change Detection

Land cover and land use change are among the major drivers of global change. In a time of mounting challenges for sustainable living on our planet any research benefits from interdisciplinary collaborations to gain an improved understanding of the human-environment system and to develop suitable and improve existing measures of natural resource management. This includes comprehensive understanding of land cover and land use changes, which is fundamental to mitigate global change. Remote sensing technology is essential for the analyses of the land surface (and hence related changes) because it offers cost-effective ways of collecting data simultaneously over large areas. With increasing variety of sensors and better data availability, the application of remote sensing as a means to assist in modeling, to support monitoring, and to detect changes at various spatial and temporal scales becomes more and more feasible. The relationship between the nature of the changes on the land surface, the sensor properties, and the conditions at the time of acquisition influences the potential and quality of land cover and land use change detection. Despite the wealth of existing change detection research, there is a need for new methodologies in order to efficiently explore the huge amount of data acquired by remote sensing systems with different sensor characteristics. The research of this thesis provides solutions to two main challenges of remote sensing based change detection. First, geometric effects and distortions occur when using data taken under different sun-target-sensor geometries. These effects mainly occur if sun position and/or viewing angles differ between images. This challenge was met by developing a theoretical framework of bi-temporal change detection scenarios. The concept includes the quantification of distortions that can occur in unfavorable situations. The invention and application of a new method – the Robust Change Vector Analysis (RCVA) – reduced the detection of false changes due to these distortions. The quality and robustness of the RCVA were demonstrated in an example of bi-temporal cross-sensor change detection in an urban environment in Cologne, Germany. Comparison with a state-of-the-art method showed better performance of RCVA and robustness against thresholding. Second, this thesis provides new insights into how to optimize the use of dense time series for forest cover change detection. A collection of spectral indices was reviewed for their suitability to display forest structure, development, and condition at a study site on Vancouver Island, British Columbia, Canada. The spatio-temporal variability of the indices was analyzed to identify those indices, which are considered most suitable for forest monitoring based on dense time series. Amongst the indices, the Disturbance Index (DI) was found to be sensitive to the state of the forest (i.e., forest structure). The Normalized Difference Moisture Index (NDMI) was found to be spatio-temporally stable and to be the most sensitive index for changes in forest condition. Both indices were successfully applied to detect abrupt forest cover changes. Further, this thesis demonstrated that relative radiometric normalization can obscure actual seasonal variation and long-term trends of spectral signals and is therefore not recommended to be incorporated in the time series pre-processing of remotely-sensed data. The main outcome of this part of the presented research is a new method for detecting discontinuities in time series of spectral indices. The method takes advantage of all available information in terms of cloud-free pixels and hence increases the number of observations compared to most existing methods. Also, the first derivative of the time series was identified (together with the discontinuity measure) as a suitable variable to display and quantify the dynamic of dense Landsat time series that cannot be observed with less dense time series. Given that these discontinuities are predominantly related to abrupt changes, the presented method was successfully applied to clearcut harvest detection. The presented method detected major events of forest change at unprecedented temporal resolution and with high accuracy (93% overall accuracy). This thesis contributes to improved understanding of bi-temporal change detection, addressing image artifacts that result from flexible acquisition features of modern satellites (e.g., off-nadir capabilities). The demonstrated ability to efficiently analyze cross-sensor data and data taken under unfavorable conditions is increasingly important for the detection of many rapid changes, e.g., to assist in emergency response. This thesis further contributes to the optimized use of remotely sensed time series for improving the understanding, accuracy, and reliability of forest cover change detection. Additionally, the thesis demonstrates the usability of and also the necessity for continuity in medium spatial resolution satellite imagery, such as the Landsat data, for forest management. Constellations of recently launched (e.g., Landsat 8 OLI) and upcoming sensors (e.g., Sentinel-2) will deliver new opportunities to apply and extend the presented methodologies.Der Einfluss von Sensorcharakteristik und Datenverfügbarkeit auf die fernerkundungsbasierte Veränderungsdetektion Landbedeckungs- und Landnutzungswandel gehören zu den Haupttriebkräften des Globalen Wandels. In einer Zeit, in der ein nachhaltiges Leben auf unserem Planeten zu einer wachsenden Herausforderung wird, profitiert die Wissenschaft von interdisziplinärer Zusammenarbeit, um ein besseres Verständnis der Mensch-Umwelt-Beziehungen zu erlangen und um verbesserte Maßnahmen des Ressourcenmanagements zu entwickeln. Dazu gehört auch ein erweitertes Verständnis von Landbedeckungs- und Landnutzungswandel, das elementar ist, um dem Globalen Wandel zu begegnen. Die Fernerkundungstechnologie ist grundlegend für die Analyse der Landoberfläche und damit verknüpften Veränderungen, weil sie in der Lage ist, große Flächen gleichzeitig zu erfassen. Mit zunehmender Sensorenvielfalt und besserer Datenverfügbarkeit gewinnt Fernerkundung bei der Modellierung, beim Monitoring sowie als Mittel zur Erkennung von Veränderungen in verschiedenen räumlichen und zeitlichen Skalen zunehmend an Bedeutung. Das Wirkungsgeflecht zwischen der Art von Veränderungen der Landoberfläche, Sensoreigenschaften und Aufnahmebedingungen beeinflusst das Potenzial und die Qualität fernerkundungsbasierter Landbedeckungs- und Landnutzungsveränderungs-detektion. Trotz der Fülle an bestehenden Forschungsleistungen zur Veränderungsdetektion besteht ein dringender Bedarf an neuen Methoden, die geeignet sind, das große Aufkommen von Daten unterschiedlicher Sensoren effizient zu nutzen. Die in dieser Abschlussarbeit durchgeführte Forschung befasst sich mit zwei aktuellen Problemfeldern der fernerkundungsbasierten Veränderungsdetektion. Das erste sind die geometrischen Effekte und Verzerrungen, die auftreten, wenn Daten genutzt werden, die unter verschiedenen Sonne-Zielobjekt-Sensor-Geometrien aufgenommen wurden. Diese Effekte treten vor allem dann auf, wenn unterschiedliche Sonnenstände und/oder unterschiedliche Einfallswinkel der Satelliten genutzt werden. Der Herausforderung wurde begegnet, indem ein theoretisches Konzept von Szenarien dargelegt wurde, die bei der bi-temporalen Veränderungsdetektion auftreten können. Das Konzept beinhaltet die Quantifizierung der Verzerrungen, die in ungünstigen Fällen auftreten können. Um die Falscherkennung von Veränderungen in Folge der resultierenden Verzerrungen zu reduzieren, wurde eine neue Methode entwickelt – die Robust Change Vector Analysis (RCVA). Die Qualität der Methode wird an einem Beispiel der Veränderungsdetektion im urbanen Raum (Köln, Deutschland) aufgezeigt. Ein Vergleich mit einer anderen gängigen Methode zeigt bessere Ergebnisse für die neue RCVA und untermauert deren Robustheit gegenüber der Schwellenwertbestimmung. Die zweite Herausforderung, mit der sich die vorliegende Arbeit befasst, betrifft die optimierte Nutzung von dichten Zeitreihen zur Veränderungsdetektion von Wäldern. Eine Auswahl spektraler Indizes wurde hinsichtlich ihrer Tauglichkeit zur Erfassung von Waldstruktur, Waldentwicklung und Waldzustand in einem Untersuchungsgebiet auf Vancouver Island, British Columbia, Kanada, bewertet. Um die Einsatzmöglichkeiten der Indizes für dichte Zeitreihen bewerten zu können, wurde ihre raum-zeitliche Variabilität untersucht. Der Disturbance Index (DI) ist ein Index, der sensitiv für das Stadium eines Waldes ist (d. h. seine Struktur). DerNormalized Difference Moisture Index (NDMI) ist raum-zeitlich stabil und zudem am sensitivsten für Veränderungen des Waldzustands. Beide Indizes wurden erfolgreich zur Erkennung von abrupten Veränderungen getestet. In der vorliegenden Arbeit wird aufgezeigt, dass die relative radiometrische Normierung saisonale Variabilität und Langzeittrends von Zeitreihen spektraler Signale verzerrt. Die relative radiometrische Normierung wird daher nicht zur Vorprozessierung von Fernerkundungszeitreihen empfohlen. Das wichtigste Ergebnis dieser Studie ist eine neue Methode zur Erkennung von Diskontinuitäten in Zeitreihen spektraler Indizes. Die Methode nutzt alle wolkenfreien, ungestörten Beobachtungen (d. h. unabhängig von der Gesamtbewölkung in einem Bild) in einer Zeitreihe und erhöht dadurch die Anzahl an Beobachtungen im Vergleich zu anderen Methoden. Die erste Ableitung und die Messgröße zur Erfassung der Diskontinuitäten sind gut geeignet, um die Dynamik dichter Zeitreihen zu beschreiben und zu quantifizieren. Dies ist mit weniger dichten Zeitreihen nicht möglich. Da diese Diskontinuitäten im Untersuchungsgebiet üblicherweise abrupter Natur sind, ist die Methode gut geeignet, um Kahlschläge zu erfassen. Die hier dargelegte neue Methode detektiert Waldbedeckungsveränderungen mit einzigartiger zeitlicher Auflösung und hoher Genauigkeit (93% Gesamtgenauigkeit). Die vorliegende Arbeit trägt zu einem verbesserten Verständnis bi-temporaler Veränderungsdetektion bei, indem Bildartefakte berücksichtigt werden, die infolge der Flexibilität moderner Sensoren entstehen können. Die dargestellte Möglichkeit, Daten zu analysieren, die von unterschiedlichen Sensoren stammen und die unter ungünstigen Bedingungen aufgenommen wurden, wird zukünftig bei der Erfassung von schnellen Veränderungen an Bedeutung gewinnen, z. B. bei Katastropheneinsätzen. Ein weiterer Beitrag der vorliegenden Arbeit liegt in der optimierten Anwendung von Fernerkundungszeitreihen zur Verbesserung von Verständnis, Genauigkeit und Verlässlichkeit der Waldveränderungsdetektion. Des Weiteren zeigt die Arbeit den Nutzen und die Notwendigkeit der Fortführung von Satellitendaten mit mittlerer Auflösung (z. B. Landsat) für das Waldmanagement. Konstellationen kürzlich gestarteter (z. B. Landsat 8 OLI) und zukünftiger Sensoren (z. B. Sentinel-2) werden neue Möglichkeiten zur Anwendung und Optimierung der hier vorgestellten Methoden bieten

Self consistent bathymetric mapping from robotic vehicles in the deep ocean

Submitted In partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and Woods Hole Oceanographic Institution June 2005Obtaining accurate and repeatable navigation for robotic vehicles in the deep ocean is difficult and consequently a limiting factor when constructing vehicle-based bathymetric maps. This thesis presents a methodology to produce self-consistent maps and simultaneously improve vehicle position estimation by exploiting accurate local navigation and utilizing terrain relative measurements. It is common for errors in the vehicle position estimate to far exceed the errors associated with the acoustic range sensor. This disparity creates inconsistency when an area is imaged multiple times and causes artifacts that distort map integrity. Our technique utilizes small terrain "submaps" that can be pairwise registered and used to additionally constrain the vehicle position estimates in accordance with actual bottom topography. A delayed state Kalman filter is used to incorporate these sub-map registrations as relative position measurements between previously visited vehicle locations. The archiving of previous positions in a filter state vector allows for continual adjustment of the sub-map locations. The terrain registration is accomplished using a two dimensional correlation and a six degree of freedom point cloud alignment method tailored for bathymetric data. The complete bathymetric map is then created from the union of all sub-maps that have been aligned in a consistent manner. Experimental results from the fully automated processing of a multibeam survey over the TAG hydrothermal structure at the Mid-Atlantic ridge are presented to validate the proposed method.This work was funded by the CenSSIS ERC of the Nation Science Foundation under grant EEC-9986821 and in part by the Woods Hole Oceanographic Institution through a grant from the Penzance Foundation

Cartographie dense basée sur une représentation compacte RGB-D dédiée à la navigation autonome

Our aim is concentrated around building ego-centric topometric maps represented as a graph of keyframe nodes which can be efficiently used by autonomous agents. The keyframe nodes which combines a spherical image and a depth map (augmented visual sphere) synthesises information collected in a local area of space by an embedded acquisition system. The representation of the global environment consists of a collection of augmented visual spheres that provide the necessary coverage of an operational area. A "pose" graph that links these spheres together in six degrees of freedom, also defines the domain potentially exploitable for navigation tasks in real time. As part of this research, an approach to map-based representation has been proposed by considering the following issues : how to robustly apply visual odometry by making the most of both photometric and ; geometric information available from our augmented spherical database ; how to determine the quantity and optimal placement of these augmented spheres to cover an environment completely ; how tomodel sensor uncertainties and update the dense infomation of the augmented spheres ; how to compactly represent the information contained in the augmented sphere to ensure robustness, accuracy and stability along an explored trajectory by making use of saliency maps.Dans ce travail, nous proposons une représentation efficace de l’environnement adaptée à la problématique de la navigation autonome. Cette représentation topométrique est constituée d’un graphe de sphères de vision augmentées d’informations de profondeur. Localement la sphère de vision augmentée constitue une représentation égocentrée complète de l’environnement proche. Le graphe de sphères permet de couvrir un environnement de grande taille et d’en assurer la représentation. Les "poses" à 6 degrés de liberté calculées entre sphères sont facilement exploitables par des tâches de navigation en temps réel. Dans cette thèse, les problématiques suivantes ont été considérées : Comment intégrer des informations géométriques et photométriques dans une approche d’odométrie visuelle robuste ; comment déterminer le nombre et le placement des sphères augmentées pour représenter un environnement de façon complète ; comment modéliser les incertitudes pour fusionner les observations dans le but d’augmenter la précision de la représentation ; comment utiliser des cartes de saillances pour augmenter la précision et la stabilité du processus d’odométrie visuelle

Interdisciplinarity in the Age of the Triple Helix: a Film Practitioner's Perspective

This integrative chapter contextualises my research including articles I have published as well as one of the creative artefacts developed from it, the feature film The Knife That Killed Me. I review my work considering the ways in which technology, industry methods and academic practice have evolved as well as how attitudes to interdisciplinarity have changed, linking these to Etzkowitz and Leydesdorff’s ‘Triple Helix’ model (1995). I explore my own experiences and observations of opportunities and challenges that have been posed by the intersection of different stakeholder needs and expectations, both from industry and academic perspectives, and argue that my work provides novel examples of the applicability of the ‘Triple Helix’ to the creative industries. The chapter concludes with a reflection on the evolution and direction of my work, the relevance of the ‘Triple Helix’ to creative practice, and ways in which this relationship could be investigated further