Wide baseline pose estimation from video with a density-based uncertainty model

International audienceRobust wide baseline pose estimation is an essential step in the deployment of smart camera networks. In this work, we highlight some current limitations of conventional strategies for relative pose estimation in difficult urban scenes. Then, we propose a solution which relies on an adaptive search of corresponding interest points in synchronized video streams which allows us to converge robustly toward a high-quality solution. The core idea of our algorithm is to build across the image space a nonstationary mapping of the local pose estimation uncertainty, based on the spatial distribution of interest points. Subsequently, the mapping guides the selection of new observations from the video stream in order to prioritize the coverage of areas of high uncertainty. With an additional step in the initial stage, the proposed algorithm may also be used for refining an existing pose estimation based on the video data; this mode allows for performing a data-driven self-calibration task for stereo rigs for which accuracy is critical, such as onboard medical or vehicular systems. We validate our method on three different datasets which cover typical scenarios in pose estimation. The results show a fast and robust convergence of the solution, with a significant improvement, compared to single image-based alternatives, of the RMSE of ground-truth matches, and of the maximum absolute error

Image synthesis based on a model of human vision

Modern computer graphics systems are able to construct renderings of such high quality that viewers are deceived into regarding the images as coming from a photographic source. Large amounts of computing resources are expended in this rendering process, using complex mathematical models of lighting and shading. However, psychophysical experiments have revealed that viewers only regard certain informative regions within a presented image. Furthermore, it has been shown that these visually important regions contain low-level visual feature differences that attract the attention of the viewer. This thesis will present a new approach to image synthesis that exploits these experimental findings by modulating the spatial quality of image regions by their visual importance. Efficiency gains are therefore reaped, without sacrificing much of the perceived quality of the image. Two tasks must be undertaken to achieve this goal. Firstly, the design of an appropriate region-based model of visual importance, and secondly, the modification of progressive rendering techniques to effect an importance-based rendering approach. A rule-based fuzzy logic model is presented that computes, using spatial feature differences, the relative visual importance of regions in an image. This model improves upon previous work by incorporating threshold effects induced by global feature difference distributions and by using texture concentration measures. A modified approach to progressive ray-tracing is also presented. This new approach uses the visual importance model to guide the progressive refinement of an image. In addition, this concept of visual importance has been incorporated into supersampling, texture mapping and computer animation techniques. Experimental results are presented, illustrating the efficiency gains reaped from using this method of progressive rendering. This visual importance-based rendering approach is expected to have applications in the entertainment industry, where image fidelity may be sacrificed for efficiency purposes, as long as the overall visual impression of the scene is maintained. Different aspects of the approach should find many other applications in image compression, image retrieval, progressive data transmission and active robotic vision

Combining Procedural and Hand Modeling Techniques for Creating Animated Digital 3D Natural Environments

This thesis focuses on a systematic solution for rendering 3D photorealistic natural environments using Maya\u27s procedural methods and ZBrush. The methods used in this thesis started with comparing two industry specific procedural applications, Vue and Maya\u27s Paint Effects, to determine which is better suited for applying animated procedural effects with the highest level of fidelity and expandability. Generated objects from Paint Effects contained the highest potential through object attributes, texturing and lighting. To optimize results further, compatibility with sculpting programs such as ZBrush are required to sculpt higher levels of detail. The final combination workflow produces results used in the short film Fall. The need for producing these effects is attributed to the growth of the visual effect industry\u27s ability to deliver realistic simulated complexities of nature and as such, the public\u27s insatiable need to see them on screen. Usually, however, the requirements for delivering a photorealistic digital environment fall under tight deadlines due to various phases of the visual effects project being interconnected across multiple production houses, thereby requiring the need for effective methods to deliver a high-end visual presentation. The use of a procedural system, such as an L-system, is often an initial step within a workflow leading toward creating photorealistic vegetation for visual effects environments. Procedure-based systems, such as Maya\u27s Paint Effects, feature robust controls that can generate many natural objects. A balance is thus created between being able to model objects quickly, but with limited detail, and control. Other methods outside this system must be used to achieve higher levels of fidelity through the use of attributes, expressions, lighting and texturing. Utilizing the procedural engine within Maya\u27s Paint Effects allows the beginning stages of modeling a 3D natural environment. ZBrush\u27s manual system approach can further bring the aesthetics to a much finer degree of fidelity. The benefit in leveraging both types of systems results in photorealistic objects that preserve all of the procedural and dynamic forces specified within the Paint Effects procedural engine

AI-Generated Images as Data Source: The Dawn of Synthetic Era

The advancement of visual intelligence is intrinsically tethered to the availability of large-scale data. In parallel, generative Artificial Intelligence (AI) has unlocked the potential to create synthetic images that closely resemble real-world photographs. This prompts a compelling inquiry: how much visual intelligence could benefit from the advance of generative AI? This paper explores the innovative concept of harnessing these AI-generated images as new data sources, reshaping traditional modeling paradigms in visual intelligence. In contrast to real data, AI-generated data exhibit remarkable advantages, including unmatched abundance and scalability, the rapid generation of vast datasets, and the effortless simulation of edge cases. Built on the success of generative AI models, we examine the potential of their generated data in a range of applications, from training machine learning models to simulating scenarios for computational modeling, testing, and validation. We probe the technological foundations that support this groundbreaking use of generative AI, engaging in an in-depth discussion on the ethical, legal, and practical considerations that accompany this transformative paradigm shift. Through an exhaustive survey of current technologies and applications, this paper presents a comprehensive view of the synthetic era in visual intelligence. A project associated with this paper can be found at https://github.com/mwxely/AIGS .Comment: 20 pages, 11 figure

Nebuliinivarianttien vaikutus nebuliini-aktiini-vuorovaikutukseen

Nemaline myopathy (NM) is a rare congenital disorder, the most common of congenital myopathies. It affects primarily the skeletal muscles and it is recognised by nemaline bodies in muscle tissue samples and muscle weakness. Mutation of eleven genes are known to lead to NM and the most frequent disease-causing variants are either recessive NEB variants or dominant ACTA1 variants. Variants in NEB are thought to be well tolerated and only 7% of them are hypothesized to be pathogenic. Over 200 pathogenic NEB-variants have been identified in Helsinki and the majority occurred in patients as a combination of two different variants. The missense variants were speculated to have a modifying effect on pathogenicity by affecting nebulin-actin or nebulin-tropomyosin interactions. Nebulin is a gigantic protein coded by NEB and is one of the largest proteins in vertebrates. It is located in the thin filament of the skeletal muscle sarcomere. Enclosed by terminal regions, nebulin has an extensive repetitive modular region that covers over 90% of the protein. The repetitive zone comprises of 26 modules called super repeats (SR). SRs consist of seven simple repeats. There are seven conserved SDXXYK actin-binding sites at each super repeat, one per simple repeat, and one conserved WLKGIGW tropomyosin-binding site. Due to its enormous size and highly repetitive sequence, nebulin is one of the least studied proteins in vivo, in vitro or in silico. In the NM patient database used for this study, there are 70 families with verified pathogenic mutations and in 30 of them, there were additional missense variants in NEB. These missense variants can be pathogenic modifying factors or have no impact on the phenotype. Seven missense variants were selected to study the effect of these mutations on actin-binding capacity compared to wild-type nebulin using the SR panel constructed previously by Laitila and Lehtonen. Also, due to the differences in actin-binding capacity of SRs compared to each other, one of the aims was to determine whether corresponding mutations in different SRs would have a similar or different effect on actin-binding capacity. For this aim, one missense mutation in the strongly actin-binding SR 1, and one in the weakly actin-binding SR 7 were selected from the NM database, and corresponding variants were created. Also, an in-frame deletion in SR7 found in the ExAC database and the corresponding mutation in SR1 were constructed for this study. The actin-binding strength was determined using actin co-sedimentation assay and actin affinity assay. The results for co-sedimentation assay indicate that missense variants can have an effect on nebulin-actin interactions and, therefore, can be a possible cause for NM. The corresponding mutations had no correlation in their effect on actin-binding strength, just the opposite. S1-m-2 decreased actin-binding strength of SR1 and S7-m-2 had no effect on SR7. Likewise, S7-m-1 and S7-del-1 decreased actin-binding strength of SR7 and corresponding mutations had no effect on SR1. The selected missense mutations found in NM patients in SRs 2 and 4 decreased actin-binding strength, if located at the actin-binding sites and in SR 10 increased the actin-binding strength, if located at the actin-binding site. The change in actin binding strength was defined as significant if the P-value was below 0.005. The more accurate affinity assay was performed as a trial only for S16 and S16-m-1, a variant at a tropomyosin-binding site close to an actin-binding site. It indicated a difference in actin-binding affinity missed by the actin co-sedimentation assay. The results are preliminary, but show big promise and should be optimized and implemented in the future missense mutation affinity studies. In an attempt to understand if the effect missense mutations have on nebulin-actin interaction is based on the change in nebulin structure, the 3D-structure of each produced fusion protein was predicted in silico. Considering that the variants were produced as GST-fusion proteins, the position and effect of GST in them is also a point of interest. In order to predict the structure of these large proteins, a combined approach was implemented using I-TASSER (Iterative Threading ASSEmbly Refinement) software. The software uses ab initio modeling, threading methods and atomic-level structure refinement to build an accurate 3D-model of a protein from sequence. According to the predicted 3D models of the fusion proteins, the GST-part of the proteins folds into a globular structure and acts as a core around which the nebulin fragments fold. The GST does not bind to actin and is positioned on the inside, which indicates minimal effect on nebulin-actin interaction, but may be a reason for an alternative nebulin fragment folding. The accuracy of the default set of programs in software does not give the definitive answer of the possible effect missense mutations can have on structural changes. However, I-TASSER approach for 3D-modeling is promising with further software optimization and can possibly serve as an effective bioinformatic tool in the future.Nemaliinimyopatia (NM) on harvinainen synnynnäinen sairaus, mutta yleisin syntyperäisistä lihastaudeista. Se vaikuttaa ensisijaisesti luustolihaksiin ja se tunnistetaan nemaliinikappaleista lihaskoepaloissa ja lihasheikkoudesta. Tällä hetkellä on tunnistettu mutaatioita 11 geenissä, jotka johtavat NM:aan ja yleisimmät sairautta aiheuttavat variantit ovat joko resessiiviset NEB-variantit tai dominantit ACTA1-variantit. NEB-varianttien uskotaan olevan hyvin siedettyjä ja vain 7% niistä oletetaan olevan patogeenisiä. Helsingissä on tunnistettu yli 200 patogeenistä NEB-varianttia ja suurin osa niistä esiintyi kahden eri variantin yhdistelmänä. Missense-variantteilla on ajateltu olevan modifioiva vaikutus patogeenisyyteen vaikuttamalla nebuliini-aktiini- ja nebuliini-tropomyosiini-vuorovaikutuksiin. Nebuliini on jättikokoinen proteiini, jonka koodaa NEB-geeni, ja se on yksi suurimmista selkärankaisten proteiineista. Se sijaitsee luustolihaksen sarkomeerissä, ohuessa säikeessä. Suurin osa proteiinista, yli 90%, on toistuvaa modulaarista aluetta, jonka päissä sijaitsevat terminaaliset alueet. Toistoalue koostuu 26 moduulista, joita kutsutaan supertoistoiksi (super repeat, SR). SR:t rakentuvat seitsemästä yksinkertaisesta toistosta. Jokaisessa supertoistossa on seitsemän konservoitunutta SDXXYK-aktiinisitoutumiskohtaa, yksi per yksinkertainen toisto, ja yksi konservoitunut WLKGIGW-tropomyosiinisitoutumiskohtaa. Valtavan kokonsa ja hyvin toistuvan sekvenssinsä vuoksi nebuliini on yksi vähiten tutkituista proteiineista in vivo, in vitro tai in silico. Tässä tutkimuksessa käytettiin NM-potilastietokantaa, jossa 70:ssä perheessä on vahvistettu olevan patogeeninen mutaatio ja niistä 30:ssä on lisäksi löydetty missense-variantteja NEB:ssa. Nämä missense-variantit voivat olla patogeenisyyttä muuntelevia tekijöitä tai voi olla, ettei niillä ole vaikutusta fenotyyppiin. Seitsemän missense-varianttia oli valittu ja mutaatioiden vaikutukset nebuliinin aktiinisitomiskykyyn verrattuna villityyppi-nebuliiniin oli tutkittu käyttäen aiemmin Laitilan ja Lehtosen kehittämää SR-paneelia. Lisäksi, toistojaksojen aktiinisitoutumiskyky vaihtelee toisiinsa nähden, joten yhtenä tavoitteista oli selvittää, onko vastaavilla mutaatioilla eri SR:ssa samanlainen vai erilainen vaikutus aktiinin sitomiskykyyn. Tähän tarkoitukseen oli valittu yksi missense-mutaatio vahvasti sitovasta SR 1:stä ja yksi heikosti sitovasta SR7:stä NM-potilastietokannasta, ja rakennettiin vastaavat variantit molemmissa SR:ssa. Lisäksi, perustuen ExAC-tietokannasta löydettyyn in-frame deleetioon SR 7:ssä, tutkimusta varten oli rakennettu SR 7-variantti ja sitä vastaava SR 1-variantti. Aktiinisitoutumisen vahvuus määriteltiin käyttäen aktiinin ko-sedimentaatioanalyysia ja aktiiniaffiniteettianalyysia. Kosedimentaaatio-analyysin tulokset viittaavat siihen, että missense-varianteilla voi olla vaikutus nebuliini-aktiini-vuorovaikutukseen ja siten voi mahdollisesti aiheuttaa NM:aa. Vastaavien mutaatioiden vaikutukset aktiinin sitomisvahvuuteen eivät korreloineet, vaan olivat päinvastaisia. S1-m-2 vähensi SR1:n aktiinin sitomisvahvuutta ja S7-m-2:lla ei ollut siihen vaikutusta. Samalla tavalla S7-m-1 ja S7-del-1 laskivat SR7:n aktiinin sitomisvahvuutta ja vastaavilla mutaatioilla ei ollut vaikutusta. Potilastietokannasta valitut missense-mutaatiot SR2:ssa ja SR4:ssa laskivat aktiinin sitomisvahvuutta, mikäli ne sijaitsivat aktiinin sitoutumiskohdissa ja SR10:ssä aktiinin sitoutumiskohdissa olevat mutaatiot lisäsivät aktiinin sitomisvahvuutta. Aktiinin sitoutumisvahvuuden muutos määriteltiin merkitseväksi, jos sen P-arvo oli alle 0.005. Tarkempi affiniteettianalyysi suoritettiin kokeiluna vain S16:lle ja S16-m-1:lle, tropomyosiinin sitoutumiskohdassa ja lähellä aktiinin sitoutumiskohtaa sijaitsevalle variantille SR16:ssa. Affiniteettianalyysin tulosten mukaan näiden kahden nebuliinifragmentin aktiiniaffiniteetissä on eroa, jota ei havaittu aktiinin kosedimentaatioanalyysissa. Alustavat tulokset näyttävät lupaavilta ja affiniteetinmääritysmenetelmä tulisi optimoida ja käyttää tulevissa missense-mutaatioiden affiniteettitutkimuksissa. Selvittääksemme johtuuko missense-mutaatioiden vaikutus nebuliini-aktiini-vuorovaikutukseen muutoksesta nebuliinin rakenteessa, jokaisen tuotetun GST-nebuliini-fuusioproteiinin 3D-rakenne oli ennustettu in silico. GST:n sijainti ja vaikutus fuusioproteiinin rakenteeseen olivat myös mielenkiinnon kohteena. Mallien ennustamiseen käytettiin I-TASSER (Iterative Threading ASSEmbly Refinement)-ohjelmistoa, jossa yhdistyvät kolme proteiinien mallintamisen lähestymistapaa: ab initio-mallinnus, laskostumisen tunnistusmenetelmät ja atomitason rakenteen tarkennus. Ohjelmisto rakentaa tarkan 3D-mallin aminohapposekvenssistä. Ennustettujen 3D-mallien mukaan, fuusioproteiinien GST-osa laskostuu pallomaiseksi rakenteeksi ja toimii ytimenä, jonka ympärille nebuliinifragmentit laskostuvat. GST ei sido aktiinia ja sijaitsee rakenteen keskustassa, mikä viittaa siihen, että sen vaikutus nebuliini-aktiini-vuorovaikutukseen on hyvin pieni tai olematon. On kuitenkin mahdollista, että GST:n läsnäolo johtaa vaihtoehtoiseen nebuliinifragmenttien laskostumiseen. Ohjelmiston tarkkuus ei oletusohjelmilla anna selkeää vastausta mahdollisista missense-mutaatioiden vaikutuksista proteiinirakenteeseen. Kuitenkin, I-TASSER-lähestymistapa nebuliinivarianttien 3D-mallintamiseen näyttää lupaavalta ohjelmiston edelleenoptimisaatiolla ja se voi mahdollisesti toimia tehokkaana bioinformaattisena työkaluna tulevaisuudessa

Real-time simulation and visualisation of cloth using edge-based adaptive meshes

Real-time rendering and the animation of realistic virtual environments and characters has progressed at a great pace, following advances in computer graphics hardware in the last decade. The role of cloth simulation is becoming ever more important in the quest to improve the realism of virtual environments. The real-time simulation of cloth and clothing is important for many applications such as virtual reality, crowd simulation, games and software for online clothes shopping. A large number of polygons are necessary to depict the highly exible nature of cloth with wrinkling and frequent changes in its curvature. In combination with the physical calculations which model the deformations, the effort required to simulate cloth in detail is very computationally expensive resulting in much diffculty for its realistic simulation at interactive frame rates. Real-time cloth simulations can lack quality and realism compared to their offline counterparts, since coarse meshes must often be employed for performance reasons. The focus of this thesis is to develop techniques to allow the real-time simulation of realistic cloth and clothing. Adaptive meshes have previously been developed to act as a bridge between low and high polygon meshes, aiming to adaptively exploit variations in the shape of the cloth. The mesh complexity is dynamically increased or refined to balance quality against computational cost during a simulation. A limitation of many approaches is they do not often consider the decimation or coarsening of previously refined areas, or otherwise are not fast enough for real-time applications. A novel edge-based adaptive mesh is developed for the fast incremental refinement and coarsening of a triangular mesh. A mass-spring network is integrated into the mesh permitting the real-time adaptive simulation of cloth, and techniques are developed for the simulation of clothing on an animated character

Systematic Visual Reasoning through Object-Centric Relational Abstraction

Human visual reasoning is characterized by an ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part on our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focused inputs. Here, we combine these two approaches, introducing Object-Centric Relational Abstraction (OCRA), a model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including a novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays

A PhD Dissertation on Road Topology Classification for Autonomous Driving

La clasificaci´on de la topolog´ıa de la carretera es un punto clave si queremos desarrollar sistemas de conducci´on aut´onoma completos y seguros. Es l´ogico pensar que la comprensi ´on de forma exhaustiva del entorno que rodea al vehiculo, tal como sucede cuando es un ser humano el que toma las decisiones al volante, es una condici´on indispensable si se quiere avanzar en la consecuci´on de veh´ıculos aut´onomos de nivel 4 o 5. Si el conductor, ya sea un sistema aut´onomo, como un ser humano, no tiene acceso a la informaci´on del entorno la disminuci´on de la seguridad es cr´ıtica y el accidente es casi instant´aneo i.e., cuando un conductor se duerme al volante. A lo largo de esta tesis doctoral se presentan sendos sistemas basados en deep leaning que ayudan al sistema de conducci´on aut´onoma a comprender el entorno en el que se encuentra en ese instante. El primero de ellos 3D-Deep y su optimizaci´on 3D-Deepest, es una nueva arquitectura de red para la segmentaci´on sem´antica de carretera en el que se integran fuentes de datos de diferente tipolog´ıa. La segmentaci´on de carretera es clave en un veh´ıculo aut´onomo, ya que es el medio por el que deber´ıa circular en el 99,9% de los casos. El segundo es un sistema de clasificaci´on de intersecciones urbanas mediante diferentes enfoques comprendidos dentro del metric-learning, la integraci´on temporal y la generaci´on de im´agenes sint´eticas. La seguridad es un punto clave en cualquier sistema aut´onomo, y si es de conducci´on a´un m´as. Las intersecciones son uno de los lugares dentro de las ciudades donde la seguridad es cr´ıtica. Los coches siguen trayectorias secantes y por tanto pueden colisionar, la mayor´ıa de ellas son usadas por los peatones para atravesar la v´ıa independientemente de si existen pasos de cebra o no, lo que incrementa de forma alarmante los riesgos de atropello y colisi´on. La implementaci´on de la combinaci´on de ambos sistemas mejora substancialmente la comprensi´on del entorno, y puede considerarse que incrementa la seguridad, allanando el camino en la investigaci´on hacia un veh´ıculo completamente aut´onomo.Road topology classification is a crucial point if we want to develop complete and safe autonomous driving systems. It is logical to think that a thorough understanding of the environment surrounding the ego-vehicle, as it happens when a human being is a decision-maker at the wheel, is an indispensable condition if we want to advance in the achievement of level 4 or 5 autonomous vehicles. If the driver, either an autonomous system or a human being, does not have access to the information of the environment, the decrease in safety is critical, and the accident is almost instantaneous, i.e., when a driver falls asleep at the wheel. Throughout this doctoral thesis, we present two deep learning systems that will help an autonomous driving system understand the environment in which it is at that instant. The first one, 3D-Deep and its optimization 3D-Deepest, is a new network architecture for semantic road segmentation in which data sources of different types are integrated. Road segmentation is vital in an autonomous vehicle since it is the medium on which it should drive in 99.9% of the cases. The second is an urban intersection classification system using different approaches comprised of metric-learning, temporal integration, and synthetic image generation. Safety is a crucial point in any autonomous system, and if it is a driving system, even more so. Intersections are one of the places within cities where safety is critical. Cars follow secant trajectories and therefore can collide; most of them are used by pedestrians to cross the road regardless of whether there are crosswalks or not, which alarmingly increases the risks of being hit and collision. The implementation of the combination of both systems substantially improves the understanding of the environment and can be considered to increase safety, paving the way in the research towards a fully autonomous vehicle

A comprehensive survey on recent deep learning-based methods applied to surgical data

Minimally invasive surgery is highly operator dependant with a lengthy procedural time causing fatigue to surgeon and risks to patients such as injury to organs, infection, bleeding, and complications of anesthesia. To mitigate such risks, real-time systems are desired to be developed that can provide intra-operative guidance to surgeons. For example, an automated system for tool localization, tool (or tissue) tracking, and depth estimation can enable a clear understanding of surgical scenes preventing miscalculations during surgical procedures. In this work, we present a systematic review of recent machine learning-based approaches including surgical tool localization, segmentation, tracking, and 3D scene perception. Furthermore, we provide a detailed overview of publicly available benchmark datasets widely used for surgical navigation tasks. While recent deep learning architectures have shown promising results, there are still several open research problems such as a lack of annotated datasets, the presence of artifacts in surgical scenes, and non-textured surfaces that hinder 3D reconstruction of the anatomical structures. Based on our comprehensive review, we present a discussion on current gaps and needed steps to improve the adaptation of technology in surgery.Comment: This paper is to be submitted to International journal of computer visio