Vision for Scene Understanding

This manuscript covers my recent research on vision algorithms for scene understanding, articulated in 3 research axes: 3D Vision, Weakly supervised vision, and Vision and physics. At the core of the most recent works is weakly-supervised learning and physics-embodied vision, which address short comings of supervised learning that requires large amount of data. The use of more physically grounded algorithms appears evidently beneficial as both robots and humans naturally evolve in a 3D physical world. On the other hand, accounting for physics knowledge reflects important cue about lighting and weather conditions of the scene central in my work. Physics-informed machine learning is not only beneficial for increased interpretability but also to compensate labels and data scarcity

A Pipeline for Modelling of Ice-Hockey Stick Shape Deformation Using Actual Shot Video

In Ice-Hockey, performance of the player’s shots depends on their skill level, body strength as well as stick’s construction and stiffness. In fact, research suggests that one of the primary reasons that the elite players generate much faster shots is their ability to flex their hockey stick. Thus, reconstructing the deformable 3D shape of the stick during the course of a player shot has important applications in performance analysis of ice-hockey stick. We present a new, low cost, portable system to acquire videos of a player shot and to automatically reconstruct the stick shape’s deformation in 3D. This thesis is a sub-part and contributes towards the ultimate goal of the pipeline in many different ways. First, designing a mobile stereovision setup and its calibration, capturing a lot of data acquisitions with different players shooting in different styles. Second, developing a two step pruning methodology to prune structurally thin and fast moving ice-hockey stick from noisy reconstructed point cloud in 3D. Third, automating the process of initial rigid alignment of the stick template in the noisy reconstruction. Forth, reducing the effect of noise by using medial axis approximation approach and suppressing the hand occlusion effect on the final template bending by a curve fitting approach. This pipeline is also robust against different ice-hockey sticks along with different players, shooting at different styles

Digital Fabrication Approaches for the Design and Development of Shape-Changing Displays

Interactive shape-changing displays enable dynamic representations of data and information through physically reconfigurable geometry. The actuated physical deformations of these displays can be utilised in a wide range of new application areas, such as dynamic landscape and topographical modelling, architectural design, physical telepresence and object manipulation. Traditionally, shape-changing displays have a high development cost in mechanical complexity, technical skills and time/finances required for fabrication. There is still a limited number of robust shape-changing displays that go beyond one-off prototypes. Specifically, there is limited focus on low-cost/accessible design and development approaches involving digital fabrication (e.g. 3D printing). To address this challenge, this thesis presents accessible digital fabrication approaches that support the development of shape-changing displays with a range of application examples – such as physical terrain modelling and interior design artefacts. Both laser cutting and 3D printing methods have been explored to ensure generalisability and accessibility for a range of potential users. The first design-led content generation explorations show that novice users, from the general public, can successfully design and present their own application ideas using the physical animation features of the display. By engaging with domain experts in designing shape-changing content to represent data specific to their work domains the thesis was able to demonstrate the utility of shape-changing displays beyond novel systems and describe practical use-case scenarios and applications through rapid prototyping methods. This thesis then demonstrates new ways of designing and building shape-changing displays that goes beyond current implementation examples available (e.g. pin arrays and continuous surface shape-changing displays). To achieve this, the thesis demonstrates how laser cutting and 3D printing can be utilised to rapidly fabricate deformable surfaces for shape-changing displays with embedded electronics. This thesis is concluded with a discussion of research implications and future direction for this work

Virtuaalse proovikabiini 3D kehakujude ja roboti juhtimisalgoritmide uurimine

Väitekirja elektrooniline versioon ei sisalda publikatsiooneVirtuaalne riiete proovimine on üks põhilistest teenustest, mille pakkumine võib suurendada rõivapoodide edukust, sest tänu sellele lahendusele väheneb füüsilise töö vajadus proovimise faasis ning riiete proovimine muutub kasutaja jaoks mugavamaks. Samas pole enamikel varem välja pakutud masinnägemise ja graafika meetoditel õnnestunud inimkeha realistlik modelleerimine, eriti terve keha 3D modelleerimine, mis vajab suurt kogust andmeid ja palju arvutuslikku ressurssi. Varasemad katsed on ebaõnnestunud põhiliselt seetõttu, et ei ole suudetud korralikult arvesse võtta samaaegseid muutusi keha pinnal. Lisaks pole varasemad meetodid enamasti suutnud kujutiste liikumisi realistlikult reaalajas visualiseerida. Käesolev projekt kavatseb kõrvaldada eelmainitud puudused nii, et rahuldada virtuaalse proovikabiini vajadusi. Välja pakutud meetod seisneb nii kasutaja keha kui ka riiete skaneerimises, analüüsimises, modelleerimises, mõõtmete arvutamises, orientiiride paigutamises, mannekeenidelt võetud 3D visuaalsete andmete segmenteerimises ning riiete mudeli paigutamises ja visualiseerimises kasutaja kehal. Selle projekti käigus koguti visuaalseid andmeid kasutades 3D laserskannerit ja Kinecti optilist kaamerat ning koostati nendest andmebaas. Neid andmeid kasutati välja töötatud algoritmide testimiseks, mis peamiselt tegelevad riiete realistliku visuaalse kujutamisega inimkehal ja suuruse pakkumise süsteemi täiendamisega virtuaalse proovikabiini kontekstis.Virtual fitting constitutes a fundamental element of the developments expected to rise the commercial prosperity of online garment retailers to a new level, as it is expected to reduce the load of the manual labor and physical efforts required. Nevertheless, most of the previously proposed computer vision and graphics methods have failed to accurately and realistically model the human body, especially, when it comes to the 3D modeling of the whole human body. The failure is largely related to the huge data and calculations required, which in reality is caused mainly by inability to properly account for the simultaneous variations in the body surface. In addition, most of the foregoing techniques cannot render realistic movement representations in real-time. This project intends to overcome the aforementioned shortcomings so as to satisfy the requirements of a virtual fitting room. The proposed methodology consists in scanning and performing some specific analyses of both the user's body and the prospective garment to be virtually fitted, modeling, extracting measurements and assigning reference points on them, and segmenting the 3D visual data imported from the mannequins. Finally, superimposing, adopting and depicting the resulting garment model on the user's body. The project is intended to gather sufficient amounts of visual data using a 3D laser scanner and the Kinect optical camera, to manage it in form of a usable database, in order to experimentally implement the algorithms devised. The latter will provide a realistic visual representation of the garment on the body, and enhance the size-advisor system in the context of the virtual fitting room under study

Instruction with 3D Computer Generated Anatomy

Research objectives. 1) To create an original and useful software application; 2) to investigate the utility of dyna-linking for teaching upper limb anatomy. Dyna-linking is an arrangement whereby interaction with one representation automatically drives the behaviour of another representation. Method. An iterative user-centred software development methodology was used to build, test and refine successive prototypes of an upper limb software tutorial. A randomised trial then tested the null hypothesis: There will be no significant difference in learning outcomes between participants using dyna-linked 2D and 3D representations of the upper limb and those using non dyna-linked representations. Data was analysed in SPSS using factorial analysis of variance (ANOVA). Results and analysis. The study failed to reject the null hypothesis as there was no signi cant di fference between experimental conditions. Post-hoc analysis revealed that participants with low prior knowledge performed significantly better (p = 0.036) without dyna-linking (mean gain = 7.45) than with dyna-linking (mean gain = 4.58). Participants with high prior knowledge performed equally well with or without dyna-linking. These findings reveal an aptitude by treatment interaction (ATI) whereby the effectiveness of dyna-linking varies according to learner ability. On average, participants using the non dyna-linked system spent 3 minutes and 4 seconds longer studying the tutorial. Participants using the non dyna-linked system clicked 30% more on the representations. Dyna-linking had a high perceived value in questionnaire surveys (n=48) and a focus group (n=7). Conclusion. Dyna-linking has a high perceived value but may actually over-automate learning by prematurely giving novice learners a fully worked solution. Further research is required to confirm if this finding is repeated in other domains, with different learners and more sophisticated implementations of dyna-linking

Computational methods for the analysis of functional 4D-CT chest images.

Medical imaging is an important emerging technology that has been intensively used in the last few decades for disease diagnosis and monitoring as well as for the assessment of treatment effectiveness. Medical images provide a very large amount of valuable information that is too huge to be exploited by radiologists and physicians. Therefore, the design of computer-aided diagnostic (CAD) system, which can be used as an assistive tool for the medical community, is of a great importance. This dissertation deals with the development of a complete CAD system for lung cancer patients, which remains the leading cause of cancer-related death in the USA. In 2014, there were approximately 224,210 new cases of lung cancer and 159,260 related deaths. The process begins with the detection of lung cancer which is detected through the diagnosis of lung nodules (a manifestation of lung cancer). These nodules are approximately spherical regions of primarily high density tissue that are visible in computed tomography (CT) images of the lung. The treatment of these lung cancer nodules is complex, nearly 70% of lung cancer patients require radiation therapy as part of their treatment. Radiation-induced lung injury is a limiting toxicity that may decrease cure rates and increase morbidity and mortality treatment. By finding ways to accurately detect, at early stage, and hence prevent lung injury, it will have significant positive consequences for lung cancer patients. The ultimate goal of this dissertation is to develop a clinically usable CAD system that can improve the sensitivity and specificity of early detection of radiation-induced lung injury based on the hypotheses that radiated lung tissues may get affected and suffer decrease of their functionality as a side effect of radiation therapy treatment. These hypotheses have been validated by demonstrating that automatic segmentation of the lung regions and registration of consecutive respiratory phases to estimate their elasticity, ventilation, and texture features to provide discriminatory descriptors that can be used for early detection of radiation-induced lung injury. The proposed methodologies will lead to novel indexes for distinguishing normal/healthy and injured lung tissues in clinical decision-making. To achieve this goal, a CAD system for accurate detection of radiation-induced lung injury that requires three basic components has been developed. These components are the lung fields segmentation, lung registration, and features extraction and tissue classification. This dissertation starts with an exploration of the available medical imaging modalities to present the importance of medical imaging in today’s clinical applications. Secondly, the methodologies, challenges, and limitations of recent CAD systems for lung cancer detection are covered. This is followed by introducing an accurate segmentation methodology of the lung parenchyma with the focus of pathological lungs to extract the volume of interest (VOI) to be analyzed for potential existence of lung injuries stemmed from the radiation therapy. After the segmentation of the VOI, a lung registration framework is introduced to perform a crucial and important step that ensures the co-alignment of the intra-patient scans. This step eliminates the effects of orientation differences, motion, breathing, heart beats, and differences in scanning parameters to be able to accurately extract the functionality features for the lung fields. The developed registration framework also helps in the evaluation and gated control of the radiotherapy through the motion estimation analysis before and after the therapy dose. Finally, the radiation-induced lung injury is introduced, which combines the previous two medical image processing and analysis steps with the features estimation and classification step. This framework estimates and combines both texture and functional features. The texture features are modeled using the novel 7th-order Markov Gibbs random field (MGRF) model that has the ability to accurately models the texture of healthy and injured lung tissues through simultaneously accounting for both vertical and horizontal relative dependencies between voxel-wise signals. While the functionality features calculations are based on the calculated deformation fields, obtained from the 4D-CT lung registration, that maps lung voxels between successive CT scans in the respiratory cycle. These functionality features describe the ventilation, the air flow rate, of the lung tissues using the Jacobian of the deformation field and the tissues’ elasticity using the strain components calculated from the gradient of the deformation field. Finally, these features are combined in the classification model to detect the injured parts of the lung at an early stage and enables an earlier intervention

Fluid structure interaction in bioinspired locomotion problems

Mención Internacional en el título de doctorNature offers a vast amount of examples of efficient locomotion. Millions of years of evolution have allowed animals —such as fish, insects and birds—, and even plants —such as winged-seeds or dandelions— to achieve outstanding locomotive skills. Therefore, it is not a surprise that scientists and engineers have tried to replicate the flight and swimming capabilities of the former examples in order to develop efficient aerial and nautical robots. In fact, these efforts have led to the design and development of several successful bioinspired robots. However, their performance is still far below their living counterparts. One of the main reasons is that the understanding of the physics underlying biological locomotion is still limited. This is due to the complexity of the problem under consideration: the locomotion of a body through a fluid medium. This can be considered fluid structure interaction (FSI) problem where the dynamics of the specimens is the result from the hydrodynamic interaction with the surrounding fluid, which in turn is modified by the motion of the specimens. Consequently, the resulting problem is highly nonlinear and complex from a mathematical standpoint. This dissertation attempts to contribute to further understand the fluid structure interactions in bioinspired locomotion problems. To that end, direct numerical simulations of several examples of bioinspired FSI problems are performed. These examples include the auto-rotation of a winged-seed, the flow interactions between the wings of a dragonfly, and the schooling patterns that emerge between two fish. In the first part of this dissertation, the algorithm which has been developed to perform part of the aforementioned studies is presented. The proposed algorithm allows the study of the FSI of systems of connected rigid bodies —which serve as a model for the actual specimens— immersed in an incompressible fluid. It is built based on a preexisting flow solver, coupled with a robotic algorithm for the computation of the dynamic equations of the bodies. The use of robotic algorithms endows the proposed methodology with a great fiexibility, allowing to simulate a large variety of problems with different geometries and configurations. The second part of the thesis is devoted to the analysis of the aforementioned examples. In this regard, we first consider the flight of a winged-seed. This is a very interesting, yet complex, problem of fluid-dynamic interaction; in which the auto-rotative motion is the result of a subtle equilibrium between the aerodynamic forces and the inertia properties of the winged-seed. In our study, the dynamics and the flow surrounding the auto-rotating seed are characterized in a range of Reynolds numbers, Re. Specifically, we focus on the study of the leading edge vortex (LEV) that is developed on the upper surface of the seed's wing as it auto-rotates. Our findings suggest that, in the explored range Re = [80 — 240], LEV's stability is not driven by vorticity transport along the spanwise direction nor viscous effects, as reported in the literature of rotating wings. Instead, fictitious accelerations (i.e., Coriolis and centrifugal accelerations) are the most suitable candidates to stabilize the LEV over the seed's wing. In the second example, we study the effect of the three-dimensional (3D) interactions in the performance of two tandem wings, resembling those of a dragonfly. To that end, the wings undergo a two-dimensional (2D) optimum kinematics which is a combination of heaving and pitching. We first analyze the effect of wings' aspect ratio, AR, by comparing the 3D and 2D simulations. The results show that 3D vertical interactions are detrimental for the thrust production of the hindwing, but they do not significantly affect the propulsive efficiency of the tandem arrangement. Next, a more realistic flapping kinematics of the 3D is considered and compared to the previous heaving kinematics. We find a decrease in the propulsive efficiency of the flapping wings compared to their heaving counterparts, which has been linked to a non-desired shedding of vorticity on the inboard region of the wings. The last bioinspired example corresponds to the collective motion of two self-propelled three-dimensional bodies. These bodies are idealized as rectangular, flat plates with flexibility along their chordwise direction, and that self-propels thanks to a prescribed vertical motion of their leading edges. We observe that tandem configurations emerge where both plates swim at a constant mean horizontal velocity and with a mean equilibrium horizontal distance. These configurations can be classified, attending to the resulting flow interactions, into compact and regular configurations. In the former, the performance of the upstream flapper is modified due to the close interaction with the downstream flapper. However, in the regular configurations, the performance of the upstream flapper is similar to that of an isolated flapper. Conversely, the performance of the downstream flapper is affected in both configurations by the interaction with the wake of the upstream flapper. We are able to link the changes in the downstream flapper's performance to its interaction with the vertical jet induced by vortex rings of the upstream flapper's wake. Finally, we propose a model to qualitatively predict the performance of a hypothetical downstream flapper based on the flow field of and isolated flapper, showing good agreement with the actual simulations.La naturaleza ofrece una gran cantidad de ejemplos de locomoción eficiente. Millones de años de evolución han permitido a animales —tales como peces, insectos o pájaros— e incluso plantas —como sainaras o dientes de león— lograr unas habilidades de lomoción excepcionales. Por lo tanto, no es una sorpresa que científicos e ingenieros hayan intentado replicar la capacidades de vuelo y nado de los anteriores ejemplos, con el objetivo de desarrollar robots aéreos y nadadores más eficientes. De hecho, estos esfuerzos han dado lugar al diseño y desarrollo exitoso de varios robots bioinsipirados. Sin embargo, el rendimiento de éstos es todavía muy inferior al de sus referentes biológicos. Una de las principales razones es que la comprensión de la física subyacente de la lomococión de sistemas biológicos es aún limitada. Esto es debido a la complejidad del problema, a saber, el movimiento de un cuerpo a través de un medio fluido. Este se puede considerar como un problema de interacción fluido estructura (FSI) donde la dinámica del espécimen es el resultado de la interacción fluidodinámica con el fluido de alrededor, el cual es a su vez modificado por el movimiento del cuerpo. Consecuentemente, el problema resultante es altamente no lineal y complejo desde un punto de vista matemático. Con esta disertación se pretende contribuir a una mayor comprensión de la interacción fluido estructura en problemas de locomoción bioninspirados. Con tal propósito, se han realizado simulaciones numéricas directas de varios ejemplos bioinspirados de interacción fluido estructura. Estos ejemplos incluyen la autorrotación de una sámara, las interaccionés fluidas entre las alas de una libélula y los patrones de nado que surgen entre dos peces. Durante la primera parte de esta disertación, se describe el algoritmo que ha sido desarrollado con el propósito de simular alguno de los problemas anteriormente citados. El algoritmo propuesto permite el estudio de la interacción fluido estructura de sistemas de cuerpos rígidos conectados —los cuales sirven como modelo de los especímenes reales— que están sumergidos en un fluido incompresible. Está construido sobre un solver fluido pre-existente, acoplado a un algoritmo robótico que se encarga de calcular las ecuaciones dinámicas de los cuerpos. El uso de algoritmos robóticos proporciona a la metodología propuesta una gran flexibilidad, permitiendo simular una gran variedad de problemas con diversas geometrías y configuraciones. La segunda parte de esta tesis está dedicada al análisis de los ejemplos mencionados anteriormente. En este respecto, consideramos primero el vuelo de una sámara, el cual es un problema muy interesante, aunque complejo, de interacción fluido dinámica en el cual el movimiento autorrotativo es el resultado de un sutil equilibrio entre las fuerzas aerodinámicas y las propiedades inerciales de la semilla. En nuestro estudio, caracterizamos la dinámica y el flujo alrededor de la semilla autorrotante en un rango de números de Reynolds, Re. En concreto, nos centramos en el estudio del vórtice del borde de ataque (LEV) que se forma en la parte superior del ala de la sámara cuando ésta autorrota. Nuestros hallazgos sugieren que, en el rango explorado de Re = [80 — 240], la estabilidad del LEV no se debe a un transporte de vorticidad a lo largo de la dirección de la envergadura del ala, ni a efectos viscosos —como se ha mencionado en la literatura de alas rotativas—, sino que las aceleraciones ficticias (es decir, las aceleraciones centrífugas y de Coriolis), son las candidatas más probables responsables de la estabilización del LEV. En el segundo ejemplo, se estudia el efecto de las interacciones tridimensionales (3D) en el rendimiento de dos alas en configuración tándem, basadas en las de una libélula. Para ello, se prescribe que el movimiento de las alas sea una combinación de cabeceo y oscilación vertical que es óptimo en 2 dimensiones (2D). Primero analizamos el efecto de la relación de aspecto de las alas, A% comparando los resultados de las simulaciones en 3D y en 2D. Los resultados revelan que las interacciones vorticales en 3D son perjudiciales para la generación de empuje del ala trasera, pero estas interacciones no afectan de forma significativa a la eficiencia propulsiva del conjunto. Posteriormente, se considera un movimiento de batimiento más realista de las alas, y se compara su eficiencia con la obtenida previamente para las alas en movimiento oscilatorio vertical. Se observa una menor eficiencia de las alas en batimiento en comparación con las mismas alas en movimiento oscilatorio vertical. Este deterioro es asociado a un desprendimiento de estructuras vorticales cerca de los bordes marginales de las alas en batimiento. El último ejemplo bioinspirado es el del movimiento colectivo de dos cuerpos tridimiensionales que se auto propulsan. Estos cuerpos se idealizan como placas planas rectangulares, siendo flexibles a lo largo de su cuerda, y que se auto propulsan gracias a un movimiento vertical impuesto de sus bordes de ataque. Los resultados muestran la aparición de configuraciones tándem donde sendas placas nadan con una velocidad inedia constante y separadas a una distancia de equilibrio que es también constante. Estas configuraciones son clasificadas —atendiendo a las interacciones fluidas— entre compactas y regulares. En las primeras, el rendimiento de la placa que nada aguas arriba (a la que llamaremos líder) se ve afectado por las interacciones cercanas con el cuerpo que nada aguas abajo (al que denominaremos seguidor). En cambio, en las configuraciones regulares el redimiento del líder es el mismo que el de una placa similar nadando de forma aislada. Por el contrario, el rendimiento del seguidor se ve afectado en ambas configuraciones debido a las interacciones con la estela del líder. Se ha podido relacionar estos cambios en la eficiencia del seguidor con la interacción con el chorro inducido por los anillos vorticales de la estela del líder. Finalmente, hemos propuesto un modelo que permite predecir, de forma cualitativa, el rendimiento de un seguidor hipotético basándonos en el campo fluido de una placa aislada. El modelo muestra una buena correlación con los datos obtenidos de las simulaciones numéricas.This thesis has been carried out in the Bioengineering and Aerospace Engineering Department at Universidad Carlos III de Madrid. The financial support has been provided by the Spanish Ministry of Economy and Competitiveness through grant DPI2016-76151-C2-2-R (AEI/FEDER, UE).Programa de Doctorado en Mecánica de Fluidos por la Universidad Carlos III de Madrid; la Universidad de Jaén; la Universidad de Zaragoza; la Universidad Nacional de Educación a Distancia; la Universidad Politécnica de Madrid y la Universidad Rovira i VirgiliPresidente: Francisco Javier Huera-Huarte.- Secretario: Javier Rodríguez Rodríguez.- Vocal: Ignazio María Viol

Medical Robotics

The first generation of surgical robots are already being installed in a number of operating rooms around the world. Robotics is being introduced to medicine because it allows for unprecedented control and precision of surgical instruments in minimally invasive procedures. So far, robots have been used to position an endoscope, perform gallbladder surgery and correct gastroesophogeal reflux and heartburn. The ultimate goal of the robotic surgery field is to design a robot that can be used to perform closed-chest, beating-heart surgery. The use of robotics in surgery will expand over the next decades without any doubt. Minimally Invasive Surgery (MIS) is a revolutionary approach in surgery. In MIS, the operation is performed with instruments and viewing equipment inserted into the body through small incisions created by the surgeon, in contrast to open surgery with large incisions. This minimizes surgical trauma and damage to healthy tissue, resulting in shorter patient recovery time. The aim of this book is to provide an overview of the state-of-art, to present new ideas, original results and practical experiences in this expanding area. Nevertheless, many chapters in the book concern advanced research on this growing area. The book provides critical analysis of clinical trials, assessment of the benefits and risks of the application of these technologies. This book is certainly a small sample of the research activity on Medical Robotics going on around the globe as you read it, but it surely covers a good deal of what has been done in the field recently, and as such it works as a valuable source for researchers interested in the involved subjects, whether they are currently “medical roboticists” or not