Shape from inconsistent silhouette: Reconstruction of objects in the presence of segmentation and camera calibration error

Silhouettes are useful features to reconstruct the object shape when the object is textureless or the shape classes of objects are unknown. In this dissertation, we explore the problem of reconstructing the shape of challenging objects from silhouettes under real-world conditions such as the presence of silhouette and camera calibration error. This problem is called the Shape from Inconsistent Silhouettes problem. A psuedo-Boolean cost function is formalized for this problem, which penalizes differences between the reconstruction images and the silhouette images, and the Shape from Inconsistent Silhouette problem is cast as a psuedo-Boolean minimization problem. We propose a memory and time efficient method to find a local minimum solution to the optimization problem, including heuristics that take into account the geometric nature of the problem. Our methods are demonstrated on a variety of challenging objects including humans and large, thin objects. We also compare our methods to the state-of-the-art by generating reconstructions of synthetic objects with induced error. ^ We also propose a method for correcting camera calibration error given silhouettes with segmentation error. Unlike other existing methods, our method allows camera calibration error to be corrected without camera placement constraints and allows for silhouette segmentation error. This is accomplished by a modified Iterative Closest Point algorithm which minimizes the difference between an initial reconstruction and the input silhouettes. We characterize the degree of error that can be corrected with synthetic datasets with increasing error, and demonstrate the ability of the camera calibration correction method in improving the reconstruction quality in several challenging real-world datasets

The Hubble Space Telescope Treasury Program on the Orion Nebula Cluster

The Hubble Space Telescope (HST) Treasury Program on the Orion Nebula Cluster has used 104 orbits of HST time to image the Great Orion Nebula region with the Advanced Camera for Surveys (ACS), the Wide-Field/Planetary Camera 2 (WFPC2) and the Near Infrared Camera and Multi Object Spectrograph (NICMOS) instruments in 11 filters ranging from the U-band to the H-band equivalent of HST. The program has been intended to perform the definitive study of the stellar component of the ONC at visible wavelengths, addressing key questions like the cluster IMF, age spread, mass accretion, binarity and cirumstellar disk evolution. The scanning pattern allowed to cover a contiguous field of approximately 600 square arcminutes with both ACS and WFPC2, with a typical exposure time of approximately 11 minutes per ACS filter, corresponding to a point source depth AB(F435W) = 25.8 and AB(F775W)=25.2 with 0.2 magnitudes of photometric error. We describe the observations, data reduction and data products, including images, source catalogs and tools for quick look preview. In particular, we provide ACS photometry for 3399 stars, most of them detected at multiple epochs, WFPC2 photometry for 1643 stars, 1021 of them detected in the U-band, and NICMOS JH photometry for 2116 stars. We summarize the early science results that have been presented in a number of papers. The final set of images and the photometric catalogs are publicly available through the archive as High Level Science Products at the STScI Multimission Archive hosted by the Space Telescope Science Institute.Comment: Accepted for publication on the Astrophysical Journal Supplement Series, March 27, 201

Parametric region-based foreround segmentation in planar and multi-view sequences

Foreground segmentation in video sequences is an important area of the image processing that attracts great interest among the scientist community, since it makes possible the detection of the objects that appear in the sequences under analysis, and allows us to achieve a correct performance of high level applications which use foreground segmentation as an initial step. The current Ph.D. thesis entitled Parametric Region-Based Foreground Segmentation in Planar and Multi-View Sequences details, in the following pages, the research work carried out within this eld. In this investigation, we propose to use parametric probabilistic models at pixel-wise and region level in order to model the di erent classes that are involved in the classi cation process of the di erent regions of the image: foreground, background and, in some sequences, shadow. The development is presented in the following chapters as a generalization of the techniques proposed for objects segmentation in 2D planar sequences to 3D multi-view environment, where we establish a cooperative relationship between all the sensors that are recording the scene. Hence, di erent scenarios have been analyzed in this thesis in order to improve the foreground segmentation techniques: In the first part of this research, we present segmentation methods appropriate for 2D planar scenarios. We start dealing with foreground segmentation in static camera sequences, where a system that combines pixel-wise background model with region-based foreground and shadow models is proposed in a Bayesian classi cation framework. The research continues with the application of this method to moving camera scenarios, where the Bayesian framework is developed between foreground and background classes, both characterized with region-based models, in order to obtain a robust foreground segmentation for this kind of sequences. The second stage of the research is devoted to apply these 2D techniques to multi-view acquisition setups, where several cameras are recording the scene at the same time. At the beginning of this section, we propose a foreground segmentation system for sequences recorded by means of color and depth sensors, which combines di erent probabilistic models created for the background and foreground classes in each one of the views, by taking into account the reliability that each sensor presents. The investigation goes ahead by proposing foreground segregation methods for multi-view smart room scenarios. In these sections, we design two systems where foreground segmentation and 3D reconstruction are combined in order to improve the results of each process. The proposals end with the presentation of a multi-view segmentation system where a foreground probabilistic model is proposed in the 3D space to gather all the object information that appears in the views. The results presented in each one of the proposals show that the foreground segmentation and also the 3D reconstruction can be improved, in these scenarios, by using parametric probabilistic models for modeling the objects to segment, thus introducing the information of the object in a Bayesian classi cation framework.La segmentaci on de objetos de primer plano en secuencias de v deo es una importante area del procesado de imagen que despierta gran inter es por parte de la comunidad cient ca, ya que posibilita la detecci on de objetos que aparecen en las diferentes secuencias en an alisis, y permite el buen funcionamiento de aplicaciones de alto nivel que utilizan esta segmentaci on obtenida como par ametro de entrada. La presente tesis doctoral titulada Parametric Region-Based Foreground Segmentation in Planar and Multi-View Sequences detalla, en las p aginas que siguen, el trabajo de investigaci on desarrollado en este campo. En esta investigaci on se propone utilizar modelos probabil sticos param etricos a nivel de p xel y a nivel de regi on para modelar las diferentes clases que participan en la clasi caci on de las regiones de la imagen: primer plano, fondo y en seg un que secuencias, las regiones de sombra. El desarrollo se presenta en los cap tulos que siguen como una generalizaci on de t ecnicas propuestas para la segmentaci on de objetos en secuencias 2D mono-c amara, al entorno 3D multi-c amara, donde se establece la cooperaci on de los diferentes sensores que participan en la grabaci on de la escena. De esta manera, diferentes escenarios han sido estudiados con el objetivo de mejorar las t ecnicas de segmentaci on para cada uno de ellos: En la primera parte de la investigaci on, se presentan m etodos de segmentaci on para escenarios monoc amara. Concretamente, se comienza tratando la segmentaci on de primer plano para c amara est atica, donde se propone un sistema completo basado en la clasi caci on Bayesiana entre el modelo a nivel de p xel de nido para modelar el fondo, y los modelos a nivel de regi on creados para modelar los objetos de primer plano y la sombra que cada uno de ellos proyecta. La investigaci on prosigue con la aplicaci on de este m etodo a secuencias grabadas mediante c amara en movimiento, donde la clasi caci on Bayesiana se plantea entre las clases de fondo y primer plano, ambas caracterizadas con modelos a nivel de regi on, con el objetivo de obtener una segmentaci on robusta para este tipo de secuencias. La segunda parte de la investigaci on, se centra en la aplicaci on de estas t ecnicas mono-c amara a entornos multi-vista, donde varias c amaras graban conjuntamente la misma escena. Al inicio de dicho apartado, se propone una segmentaci on de primer plano en secuencias donde se combina una c amara de color con una c amara de profundidad en una clasi caci on que combina los diferentes modelos probabil sticos creados para el fondo y el primer plano en cada c amara, a partir de la fi abilidad que presenta cada sensor. La investigaci on prosigue proponiendo m etodos de segmentaci on de primer plano para entornos multi-vista en salas inteligentes. En estos apartados se diseñan dos sistemas donde la segmentaci on de primer plano y la reconstrucci on 3D se combinan para mejorar los resultados de cada uno de estos procesos. Las propuestas fi nalizan con la presentaci on de un sistema de segmentaci on multi-c amara donde se centraliza la informaci on del objeto a segmentar mediante el diseño de un modelo probabil stico 3D. Los resultados presentados en cada uno de los sistemas, demuestran que la segmentacion de primer plano y la reconstrucci on 3D pueden verse mejorados en estos escenarios mediante el uso de modelos probabilisticos param etricos para modelar los objetos a segmentar, introduciendo as la informaci on disponible del objeto en un marco de clasi caci on Bayesiano

3D object reconstruction using computer vision : reconstruction and characterization applications for external human anatomical structures

Tese de doutoramento. Engenharia Informática. Faculdade de Engenharia. Universidade do Porto. 201

Virtual camera synthesis for soccer game replays

International audienceIn this paper, we present a set of tools developed during the creation of a platform that allows the automatic generation of virtual views in a live soccer game production. Observing the scene through a multi-camera system, a 3D approximation of the players is computed and used for the synthesis of virtual views. The system is suitable both for static scenes, to create bullet time effects, and for video applications, where the virtual camera moves as the game plays

Enhanced fish bending model for automatic tuna sizing using computer vision

[EN] This paper presents a non-invasive fully automatic procedure to obtain highly accurate fish length estimation in adult Bluefin Tuna, based on a stereoscopic vision system and a deformable model of the fish ventral silhouette. The present work takes a geometric tuna model, which was previously developed by the same authors to discriminate fish in 2D images, and proposes new models to enhance the capabilities of the automatic procedure, from fish discrimination to accurate 3D length estimation. Fish length information is an important indicator of the health of wild fish stocks and for predicting biomass using length-weight relations. The proposal pays special attention to parts of the fish silhouette that have special relevance for accurate length estimation. The models have been designed to best fit the rear part of the fish, in particular the caudal peduncle, and a width parameter has been added to better fit the silhouette. Moreover, algorithms have been developed to extract snout tip and caudal peduncle features, allowing better initialization of model parameters. Snout Fork Length (SFL) measurements using the different models are extracted from images recorded with a stereoscopic vision system in a sea cage containing 312 adult Atlantic Bluefin Tuna. The automatic measurements are compared with two ground truths: one configured with semiautomatic measurements of favourable selected samples and one with real SFL measurements of the tuna stock collected at harvesting. Comparison with the semiautomatic measurements demonstrates that the combination of improved geometric models and feature extraction algorithms delivers good results in terms of fish length estimation error (up to 90% of the samples bounded in a 3% error margin) and number of automatic measurements (up to 950 samples out of 1000). When compared with real SFL measurements of the tuna stock, the system provides a high number of automatic detections (up to 6706 in a video of 135¿min duration, i.e., 50 automatic measurements per minute of recording) and highly accurate length measurements, obtaining no statistically significant difference between automatic and real SFL frequency distributions. This procedure could be extended to other species to assess the size distribution of stocks, as discussed in the paper.This work was supported by funding from ACUSTUNA project ref. CTM2015-70446-R (MINECO/ERDF, EU). This project has been possible thanks to the collaboration of IEO (Spanish Oceanographic Institute). We acknowledge the assistance provided by the Spanish company Grup Balfego S.L. in supplying boats and divers to acquire underwater video in the Mediterranean Sea.Muñoz-Benavent, P.; Andreu García, G.; Valiente González, JM.; Atienza-Vanacloig, V.; Puig Pons, V.; Espinosa Roselló, V. (2018). Enhanced fish bending model for automatic tuna sizing using computer vision. Computers and Electronics in Agriculture. 150:52-61. https://doi.org/10.1016/j.compag.2018.04.005S526115

Vision-based traffic surveys in urban environments

This paper presents a state-of-the-art, vision-based vehicle detection and type classification to perform traffic surveys from a roadside closed-circuit television camera. Vehicles are detected using background subtraction based on a Gaussian mixture model that can cope with vehicles that become stationary over a significant period of time. Vehicle silhouettes are described using a combination of shape and appearance features using an intensity-based pyramid histogram of orientation gradients (HOG). Classification is performed using a support vector machine, which is trained on a small set of hand-labeled silhouette exemplars. These exemplars are identified using a model-based preclassifier that utilizes calibrated images mapped by Google Earth to provide accurately surveyed scene geometry matched to visible image landmarks. Kalman filters track the vehicles to enable classification by majority voting over several consecutive frames. The system counts vehicles and separates them into four categories: car, van, bus, and motorcycle (including bicycles). Experiments with real-world data have been undertaken to evaluate system performance and vehicle detection rates of 96.45% and classification accuracy of 95.70% have been achieved on this data.The authors gratefully acknowledge the Royal Borough of Kingston for providing the video data. S.A. Velastin is grateful to funding received from the Universidad Carlos III de Madrid, the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement nº 600371, el Ministerio de Economía y Competitividad (COFUND2013-51509) and Banco Santander

Single View Modeling and View Synthesis

This thesis develops new algorithms to produce 3D content from a single camera. Today, amateurs can use hand-held camcorders to capture and display the 3D world in 2D, using mature technologies. However, there is always a strong desire to record and re-explore the 3D world in 3D. To achieve this goal, current approaches usually make use of a camera array, which suffers from tedious setup and calibration processes, as well as lack of portability, limiting its application to lab experiments. In this thesis, I try to produce the 3D contents using a single camera, making it as simple as shooting pictures. It requires a new front end capturing device rather than a regular camcorder, as well as more sophisticated algorithms. First, in order to capture the highly detailed object surfaces, I designed and developed a depth camera based on a novel technique called light fall-off stereo (LFS). The LFS depth camera outputs color+depth image sequences and achieves 30 fps, which is necessary for capturing dynamic scenes. Based on the output color+depth images, I developed a new approach that builds 3D models of dynamic and deformable objects. While the camera can only capture part of a whole object at any instance, partial surfaces are assembled together to form a complete 3D model by a novel warping algorithm. Inspired by the success of single view 3D modeling, I extended my exploration into 2D-3D video conversion that does not utilize a depth camera. I developed a semi-automatic system that converts monocular videos into stereoscopic videos, via view synthesis. It combines motion analysis with user interaction, aiming to transfer as much depth inferring work from the user to the computer. I developed two new methods that analyze the optical flow in order to provide additional qualitative depth constraints. The automatically extracted depth information is presented in the user interface to assist with user labeling work. In this thesis, I developed new algorithms to produce 3D contents from a single camera. Depending on the input data, my algorithm can build high fidelity 3D models for dynamic and deformable objects if depth maps are provided. Otherwise, it can turn the video clips into stereoscopic video

Automatic Bluefin Tuna sizing using a stereoscopic vision system

[EN] This article presents a non-invasive fully automatic procedure for Bluefin Tuna sizing, based on a stereoscopic vision system and a deformable model of the fish ventral silhouette. An image processing procedure is performed on each video frame to extract individual fish, followed by a fitting proce- dure to adjust the fish model to the extracted targets, adapting it to the bending movements of the fish. The proposed system is able to give accu- rate measurements of tuna snout fork length (SFL) and widths at five predefined silhouette points without manual intervention. In this work, the system is used to study size evolution in adult Atlantic Bluefin Tuna (Thunnus Thynnus) over time in a growing farm. The dataset is composed of 12 pairs of videos, which were acquired once a month in 2015, between July and October, in three grow-out cages of tuna aquaculture facilities on the west Mediterranean coast. Each grow out cage contains between 300 and 650 fish on an approximate volume of 20 000 m3.Measurements were au- tomatically obtained for the 4 consecutive months after caging and suggest a fattening process: SFL shows an increase of just a few centimetres (2%) while themaximum width (A1)shows arelative increaseofmorethan20%,mostlyinthe first 2months in farm. Moreover, a linear relation (with co- efficient of determination R2> 0.98) between SFL and widths for each month is deduced, and a fattening factor (F) is introduced. The validity of the measurements is proved by comparing 15 780 SFL measurements, obtained with our automatic system in the last month, versus ground truth data of a high percentage of the stock under study (1143 out of 1579), obtaining no statistically significant difference. This procedure could be extended to other species to assess the size distribution of stocks, as discussed in the article.This work was supported by funding from ACUSTUNA project ref. CTM2015-70446-R (MINECO/ERDF, EU). This project has been possible thanks to the collaboration of IEO (Spanish Oceanographic Institute).Muñoz-Benavent, P.; Andreu García, G.; Valiente González, JM.; Atienza-Vanacloig, V.; Puig Pons, V.; Espinosa Roselló, V. (2018). Automatic Bluefin Tuna sizing using a stereoscopic vision system. ICES Journal of Marine Science. 75(1):390-401. https://doi.org/10.1093/icesjms/fsx151S39040175