3D Textured Surface Reconstruction from Endoscopic Video

Endoscopy enables high-resolution visualization of tissue texture and is a critical step in many clinical workflows, including diagnosis of infections, tumors or diseases and treatment planning for cancers. This includes my target problems of radiation treatment planning in the nasopharynx and pre-cancerous polyps screening and treatment in colonoscopy. However, an endoscopic video does not provide its information in 3D space, making it difficult to use for tumor localization, and it is inefficient to review. In addition, when there are incomplete camera observations of the organ surface, full surface coverage cannot be guaranteed in an endoscopic procedure, and unsurveyed regions can hardly be noticed in a continuous first-person perspective. This dissertation introduces a new imaging approach that we call endoscopography: an endoscopic video is reconstructed into a full 3D textured surface, which we call an endoscopogram. In this dissertation, I present two endoscopography techniques. One method is a combination of a frame-by-frame algorithmic 3D reconstruction method and a groupwise deformable surface registration method. My contribution is the innovative combination of the two methods that improves the temporal consistency of the frame-by-frame 3D reconstruction algorithm and eliminates the manual intervention that was needed in the deformable surface registration method. The combined method reconstructs an endoscopogram in an offline manner, and the information contained in the tissue texture in the endoscopogram can be transferred to a 3D image such as CT through a surface-to-surface registration. Then, through an interactive tool, the physician can draw directly on the endoscopogram surface to specify a tumor, which then can be automatically transferred to CT slices to aid tumor localization. The second method is a novel deep-learning-driven dense SLAM (simultaneous localization and mapping) system, called RNN-SLAM, that in real time can produce an endoscopogram with display of the unsurveyed regions. In particular, my contribution is the deep learning system in the RNN-SLAM, called RNN-DP. RNN-DP is a novel multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses.Doctor of Philosoph

WiFi-Based Human Activity Recognition Using Attention-Based BiLSTM

Recently, significant efforts have been made to explore human activity recognition (HAR) techniques that use information gathered by existing indoor wireless infrastructures through WiFi signals without demanding the monitored subject to carry a dedicated device. The key intuition is that different activities introduce different multi-paths in WiFi signals and generate different patterns in the time series of channel state information (CSI). In this paper, we propose and evaluate a full pipeline for a CSI-based human activity recognition framework for 12 activities in three different spatial environments using two deep learning models: ABiLSTM and CNN-ABiLSTM. Evaluation experiments have demonstrated that the proposed models outperform state-of-the-art models. Also, the experiments show that the proposed models can be applied to other environments with different configurations, albeit with some caveats. The proposed ABiLSTM model achieves an overall accuracy of 94.03%, 91.96%, and 92.59% across the 3 target environments. While the proposed CNN-ABiLSTM model reaches an accuracy of 98.54%, 94.25% and 95.09% across those same environments

Random Finite Sets Based Very Short-Term Solar Power Forecasting Through Cloud Tracking

Tracking clouds with a sky camera within a very short horizon below thirty seconds can be a solution to mitigate the effects of sunlight disruptions. A Probability Hypothesis Density (PHD) filter and a Cardinalised Probability Hypothesis Density (CPHD) filter were used on a set of pre-processed sky images. Both filters have been compared with the state-of-the-art methods for performance. It was found that both filters are suitable to perform very-short term irradiance forecasting

Trooppisten alkuperäismetsien monitorointi Taita Hillsin alueella digitaalisen ilmakuva-aineiston avulla

The loss and degradation of forest cover is currently a globally recognised problem. The fragmentation of forests is further affecting the biodiversity and well-being of the ecosystems also in Kenya. This study focuses on two indigenous tropical montane forests in the Taita Hills in southeastern Kenya. The study is a part of the TAITA-project within the Department of Geography in the University of Helsinki. The study forests, Ngangao and Chawia, are studied by remote sensing and GIS methods. The main data includes black and white aerial photography from 1955 and true colour digital camera data from 2004. This data is used to produce aerial mosaics from the study areas. The land cover of these study areas is studied by visual interpretation, pixel-based supervised classification and object-oriented supervised classification. The change of the forest cover is studied with GIS methods using the visual interpretations from 1955 and 2004. Furthermore, the present state of the study forests is assessed with leaf area index and canopy closure parameters retrieved from hemispherical photographs as well as with additional, previously collected forest health monitoring data. The canopy parameters are also compared with textural parameters from digital aerial mosaics. This study concludes that the classification of forest areas by using true colour data is not an easy task although the digital aerial mosaics are proved to be very accurate. The best classifications are still achieved with visual interpretation methods as the accuracies of the pixel-based and object-oriented supervised classification methods are not satisfying. According to the change detection of the land cover in the study areas, the area of indigenous woodland in both forests has decreased in 1955-2004. However in Ngangao, the overall woodland area has grown mainly because of plantations of exotic species. In general, the land cover of both study areas is more fragmented in 2004 than in 1955. Although the forest area has decreased, forests seem to have a more optimistic future than before. This is due to the increasing appreciation of the forest areas.Metsien väheneminen ja niiden laadun heikkeneminen on maailmanlaajuisesti tunnustettu ongelma. Metsien pirstoutuminen vaikuttaa biodiversiteettiin ja ekosysteemien hyvinvointiin myös Keniassa. Tämä tutkimus keskittyy kahden trooppisen alkuperäisvuoristometsän tutkimiseen Taita Hillsin alueella Kaakkois-Keniassa. Tutkimus on osa Helsingin yliopiston maantieteen laitoksen TAITA-projektia. Tutkimusmetsiä, Ngangaoa ja Chawiaa tutkitaan kaukokartoitus- ja paikkatietomenetelmien avulla. Tutkimuksen pääaineiston muodostavat mustavalkoiset ilmakuvat vuodelta 1955 ja digitaaliset oikeaväri-ilmakuvat vuodelta 2004. Näistä ilmakuvista muodostetaan ilmakuvamosaiikit tutkimusalueilta. Alueiden maanpeite luokitellaan kolmella metodilla: visuaalisella tulkinnalla, pikselipohjaisella ohjatulla luokituksella sekä objekti-orientoidulla ohjatulla luokituksella. Metsäpinta-alan muutosta vuosina 1955-2004 tutkitaan visuaalisten luokitusten perusteella käyttämällä paikkatietomenetelmiä. Tutkimusmetsien kuntoa arvioidaan lehtipinta-alaindeksin ja latvuksen sulkeituneisuuden avulla. Nämä parametrit saadaan käyttämällä hemisfäärisiä valokuvia. Lisäksi tutkimuksessa käytetään metsien kuntoa arvioivaa aiemmin kerättyä tutkimustietoa. Latvusparametreja verrataan digitaali-ilmakuvamosaiikeilta saatuihin tekstuurisiin parametreihin. Yhteenvetona voidaan sanoa, että metsäalueiden luokitus oikeaväri-ilmakuvia käyttämällä ei ole helppoa, vaikka itse digitaali-ilmakuvista tehdyt mosaiikit olisivat erittäin tarkkoja. Parhaat luokitustulokset saavutetaan edelleen visuaalisella tulkinnalla, sillä pikselipohjainen ja objekti-orientoitu ohjattu luokitus eivät saavuta tarpeeksi hyvää luotettavuutta. Tutkimusalueiden maanpeitteen muutostulkinnan mukaan alkuperäismetsän osuus on vähentynyt sekä Ngangaossa että Chawiassa 1955-2004. Ngangaossa metsän kokonaisala on kuitenkin lisääntynyt lähinnä eksoottisten puulajien istutusten vuoksi. Molempien tutkimusalueiden maanpeite on huomattavasti pirstoutuneempaa vuonna 2004 kuin vuonna 1955. Vaikka metsäala on pienentynyt, tutkimusmetsien tulevaisuus näyttää paremmalta kuin aiemmin. Tämä johtuu lähinnä kasvavasta metsien arvostuksesta

Investigation of Computer Vision Concepts and Methods for Structural Health Monitoring and Identification Applications

This study presents a comprehensive investigation of methods and technologies for developing a computer vision-based framework for Structural Health Monitoring (SHM) and Structural Identification (St-Id) for civil infrastructure systems, with particular emphasis on various types of bridges. SHM is implemented on various structures over the last two decades, yet, there are some issues such as considerable cost, field implementation time and excessive labor needs for the instrumentation of sensors, cable wiring work and possible interruptions during implementation. These issues make it only viable when major investments for SHM are warranted for decision making. For other cases, there needs to be a practical and effective solution, which computer-vision based framework can be a viable alternative. Computer vision based SHM has been explored over the last decade. Unlike most of the vision-based structural identification studies and practices, which focus either on structural input (vehicle location) estimation or on structural output (structural displacement and strain responses) estimation, the proposed framework combines the vision-based structural input and the structural output from non-contact sensors to overcome the limitations given above. First, this study develops a series of computer vision-based displacement measurement methods for structural response (structural output) monitoring which can be applied to different infrastructures such as grandstands, stadiums, towers, footbridges, small/medium span concrete bridges, railway bridges, and long span bridges, and under different loading cases such as human crowd, pedestrians, wind, vehicle, etc. Structural behavior, modal properties, load carrying capacities, structural serviceability and performance are investigated using vision-based methods and validated by comparing with conventional SHM approaches. In this study, some of the most famous landmark structures such as long span bridges are utilized as case studies. This study also investigated the serviceability status of structures by using computer vision-based methods. Subsequently, issues and considerations for computer vision-based measurement in field application are discussed and recommendations are provided for better results. This study also proposes a robust vision-based method for displacement measurement using spatio-temporal context learning and Taylor approximation to overcome the difficulties of vision-based monitoring under adverse environmental factors such as fog and illumination change. In addition, it is shown that the external load distribution on structures (structural input) can be estimated by using visual tracking, and afterward load rating of a bridge can be determined by using the load distribution factors extracted from computer vision-based methods. By combining the structural input and output results, the unit influence line (UIL) of structures are extracted during daily traffic just using cameras from which the external loads can be estimated by using just cameras and extracted UIL. Finally, the condition assessment at global structural level can be achieved using the structural input and output, both obtained from computer vision approaches, would give a normalized response irrespective of the type and/or load configurations of the vehicles or human loads

Service robotics and machine learning for close-range remote sensing

L'abstract è presente nell'allegato / the abstract is in the attachmen

Learning geometric and lighting priors from natural images

Comprendre les images est d’une importance cruciale pour une pléthore de tâches, de la composition numérique au ré-éclairage d’une image, en passant par la reconstruction 3D d’objets. Ces tâches permettent aux artistes visuels de réaliser des chef-d’oeuvres ou d’aider des opérateurs à prendre des décisions de façon sécuritaire en fonction de stimulis visuels. Pour beaucoup de ces tâches, les modèles physiques et géométriques que la communauté scientifique a développés donnent lieu à des problèmes mal posés possédant plusieurs solutions, dont généralement une seule est raisonnable. Pour résoudre ces indéterminations, le raisonnement sur le contexte visuel et sémantique d’une scène est habituellement relayé à un artiste ou un expert qui emploie son expérience pour réaliser son travail. Ceci est dû au fait qu’il est généralement nécessaire de raisonner sur la scène de façon globale afin d’obtenir des résultats plausibles et appréciables. Serait-il possible de modéliser l’expérience à partir de données visuelles et d’automatiser en partie ou en totalité ces tâches ? Le sujet de cette thèse est celui-ci : la modélisation d’a priori par apprentissage automatique profond pour permettre la résolution de problèmes typiquement mal posés. Plus spécifiquement, nous couvrirons trois axes de recherche, soient : 1) la reconstruction de surface par photométrie, 2) l’estimation d’illumination extérieure à partir d’une seule image et 3) l’estimation de calibration de caméra à partir d’une seule image avec un contenu générique. Ces trois sujets seront abordés avec une perspective axée sur les données. Chacun de ces axes comporte des analyses de performance approfondies et, malgré la réputation d’opacité des algorithmes d’apprentissage machine profonds, nous proposons des études sur les indices visuels captés par nos méthodes.Understanding images is needed for a plethora of tasks, from compositing to image relighting, including 3D object reconstruction. These tasks allow artists to realize masterpieces or help operators to safely make decisions based on visual stimuli. For many of these tasks, the physical and geometric models that the scientific community has developed give rise to ill-posed problems with several solutions, only one of which is generally reasonable. To resolve these indeterminations, the reasoning about the visual and semantic context of a scene is usually relayed to an artist or an expert who uses his experience to carry out his work. This is because humans are able to reason globally on the scene in order to obtain plausible and appreciable results. Would it be possible to model this experience from visual data and partly or totally automate tasks? This is the topic of this thesis: modeling priors using deep machine learning to solve typically ill-posed problems. More specifically, we will cover three research axes: 1) surface reconstruction using photometric cues, 2) outdoor illumination estimation from a single image and 3) camera calibration estimation from a single image with generic content. These three topics will be addressed from a data-driven perspective. Each of these axes includes in-depth performance analyses and, despite the reputation of opacity of deep machine learning algorithms, we offer studies on the visual cues captured by our methods