Scalable software architecture for on-line multi-camera video processing

In this paper we present a scalable software architecture for on-line multi-camera video processing, that guarantees a good trade off between computational power, scalability and flexibility. The software system is modular and its main blocks are the Processing Units (PUs), and the Central Unit. The Central Unit works as a supervisor of the running PUs and each PU manages the acquisition phase and the processing phase. Furthermore, an approach to easily parallelize the desired processing application has been presented. In this paper, as case study, we apply the proposed software architecture to a multi-camera system in order to efficiently manage multiple 2D object detection modules in a real-time scenario. System performance has been evaluated under different load conditions such as number of cameras and image sizes. The results show that the software architecture scales well with the number of camera and can easily works with different image formats respecting the real time constraints. Moreover, the parallelization approach can be used in order to speed up the processing tasks with a low level of overhea

Accurate Calibration Scheme for a Multi-Camera Mobile Mapping System

Mobile mapping systems (MMS) are increasingly used for many photogrammetric and computer vision applications, especially encouraged by the fast and accurate geospatial data generation. The accuracy of point position in an MMS is mainly dependent on the quality of calibration, accuracy of sensor synchronization, accuracy of georeferencing and stability of geometric configuration of space intersections. In this study, we focus on multi-camera calibration (interior and relative orientation parameter estimation) and MMS calibration (mounting parameter estimation). The objective of this study was to develop a practical scheme for rigorous and accurate system calibration of a photogrammetric mapping station equipped with a multi-projective camera (MPC) and a global navigation satellite system (GNSS) and inertial measurement unit (IMU) for direct georeferencing. The proposed technique is comprised of two steps. Firstly, interior orientation parameters of each individual camera in an MPC and the relative orientation parameters of each cameras of the MPC with respect to the first camera are estimated. In the second step the offset and misalignment between MPC and GNSS/IMU are estimated. The global accuracy of the proposed method was assessed using independent check points. A correspondence map for a panorama is introduced that provides metric information. Our results highlight that the proposed calibration scheme reaches centimeter-level global accuracy for 3D point positioning. This level of global accuracy demonstrates the feasibility of the proposed technique and has the potential to fit accurate mapping purposes

Modelling and automated calibration of a general multi-projective camera

Recently, multi-projective cameras (MPCs), often based on frame-mounted multiple cameras with a small baseline and arbitrary overlap, have found a remarkable place in geomatics and vision-based applications. This paper outlines the geometric calibration of a general MPC by presenting a mathematical model that describes its unknown generic geometry. A modified bundle block adjustment is employed to calibrate an industrial-level 360° non-metric camera. The structure of any MPC can be retrieved as a calibration set of relative and interior orientation parameters (as well as the pose of the MPC shots) using a calibration room which has been accurately determined by close range photogrammetry. To demonstrate the efficiency and precision of the model, a Panono camera (an MPC with 36 individual cameras) was calibrated. After the adjustment, sub-pixel image residuals and acceptable object-space errors were observed.Peer reviewe

Accurate Calibration Scheme for a Multi-Camera Mobile Mapping System

Mobile mapping systems (MMS) are increasingly used for many photogrammetric and computer vision applications, especially encouraged by the fast and accurate geospatial data generation. The accuracy of point position in an MMS is mainly dependent on the quality of calibration, accuracy of sensor synchronization, accuracy of georeferencing and stability of geometric configuration of space intersections. In this study, we focus on multi-camera calibration (interior and relative orientation parameter estimation) and MMS calibration (mounting parameter estimation). The objective of this study was to develop a practical scheme for rigorous and accurate system calibration of a photogrammetric mapping station equipped with a multi-projective camera (MPC) and a global navigation satellite system (GNSS) and inertial measurement unit (IMU) for direct georeferencing. The proposed technique is comprised of two steps. Firstly, interior orientation parameters of each individual camera in an MPC and the relative orientation parameters of each cameras of the MPC with respect to the first camera are estimated. In the second step the offset and misalignment between MPC and GNSS/IMU are estimated. The global accuracy of the proposed method was assessed using independent check points. A correspondence map for a panorama is introduced that provides metric information. Our results highlight that the proposed calibration scheme reaches centimeter-level global accuracy for 3D point positioning. This level of global accuracy demonstrates the feasibility of the proposed technique and has the potential to fit accurate mapping purposes

Integration Of Multispectral Camera Systems For Enhanced Visualization Biological Studies Using UAS

The purpose of this research is to enhance visualization of warm-blooded animals and analyze the vegetation in which they are located using a camera system mounted on Unmanned Aircraft System (UAS). The results are coherently displayed in a single image at the same spatial location so that biologists will have accurate animal counts along with vegetation conditions. Application of aerial imagery has been used to analyze the vegetation health and determining the number of the animals by the wildlife service. However, a major obstacle of the research is to combine the two imaging systems to obtain the same spatial image with enhanced visualization. The two camera systems used in this research are the Tetracam ADC lite multispectral and the FLIR Photon 320 infrared camera. These two camera systems each have a different lens, field of view and sensor array size. The project involves the alignment of the two cameras to pixel level for the spectral image analysis. The spectral image analysis provides both vegetation information, such as Normalized Difference Vegetation Index (NDVI), along with enhanced visualization of warm-blooded targets. The system was miniaturized for the standalone payload for aerospace applications including UAS. The FLIR Photon 320 was used to capture the infrared image and a Tetracam ADC lite multispectral camera was used to capture the near infrared, red and green spectral band images. A laboratory experimental setup was designed to mechanically align the two camera systems to get close identical spatial imagery. Spatial registration of the two images was performed using reverse image warping method by finding affine transformation matrix using point correspondences. Both camera systems were calibrated using Camera Calibration Toolbox for Matlab to reduce any distortion due to the lenses. A single board computer is used to capture and store the image data from FLIR Photon 320 infrared camera while the Tetracam image data is stored internally on board. The image capture time was set by continuous timed delay triggering within the Tetracam camera. The single board computer follows the Tetracam signals and matches the FLIR Photon 320 still image acquisition time with the Tetracam ADC lite. Once the images are captured and stored by the camera systems, the files are downloaded and image processing is conducted. The data was analyzed to calculate the NDVI to observe the plant health. The infrared spectral band was used to identify the warm-blooded animals. In addition, various false color combinations of spectral bands and normalized difference ratios are processed to observe the visual enhancement capabilities on the vegetation and warm-blooded animal. It was determined that detected warm-blooded animal in the infrared band registered on top of NDVI image to show the vegetation health in a single image produced effective results. The combined image data from the FLIR Photon 320 and Tetracam ADC lite produced meaningful vegetation and animal information in the single image. This enhanced the capacity to identify and count animals while simultaneously characterize the vegetation environment, which is highly desired in ecosystem studie

Multi-Projective Camera-Calibration, Modeling, and Integration in Mobile-Mapping Systems

Optical systems are vital parts of most modern systems such as mobile mapping systems, autonomous cars, unmanned aerial vehicles (UAV), and game consoles. Multi-camera systems (MCS) are commonly employed for precise mapping including aerial and close-range applications. In the first part of this thesis a simple and practical calibration model and a calibration scheme for multi-projective cameras (MPC) is presented. The calibration scheme is enabled by implementing a camera test field equipped with a customized coded target as FGI’s camera calibration room. The first hypothesis was that a test field is necessary to calibrate an MPC. Two commercially available MPCs with 6 and 36 cameras were successfully calibrated in FGI’s calibration room. The calibration results suggest that the proposed model is able to estimate parameters of the MPCs with high geometric accuracy, and reveals the internal structure of the MPCs. In the second part, the applicability of an MPC calibrated by the proposed approach was investigated in a mobile mapping system (MMS). The second hypothesis was that a system calibration is necessary to achieve high geometric accuracies in a multi-camera MMS. The MPC model was updated to consider mounting parameters with respect to GNSS and IMU. A system calibration scheme for an MMS was proposed. The results showed that the proposed system calibration approach was able to produce accurate results by direct georeferencing of multi-images in an MMS. Results of geometric assessments suggested that a centimeter-level accuracy is achievable by employing the proposed approach. A novel correspondence map is demonstrated for MPCs that helps to create metric panoramas. In the third part, the problem of real-time trajectory estimation of a UAV equipped with a projective camera was studied. The main objective of this part was to address the problem of real-time monocular simultaneous localization and mapping (SLAM) of a UAV. An angular framework was discussed to address the gimbal lock singular situation. The results suggest that the proposed solution is an effective and rigorous monocular SLAM for aerial cases where the object is near-planar. In the last part, the problem of tree-species classification by a UAV equipped with two hyper-spectral an RGB cameras was studied. The objective of this study was to investigate different aspects of a precise tree-species classification problem by employing state-of-art methods. A 3D convolutional neural-network (3D-CNN) and a multi-layered perceptron (MLP) were proposed and compared. Both classifiers were highly successful in their tasks, while the 3D-CNN was superior in performance. The classification result was the most accurate results published in comparison to other works.Optiset kuvauslaitteet ovat keskeisessä roolissa moderneissa konenäköön perustuvissa järjestelmissä kuten autonomiset autot, miehittämättömät lentolaitteet (UAV) ja pelikonsolit. Tällaisissa sovelluksissa hyödynnetään tyypillisesti monikamerajärjestelmiä. Väitöskirjan ensimmäisessä osassa kehitetään yksinkertainen ja käytännöllinen matemaattinen malli ja kalibrointimenetelmä monikamerajärjestelmille. Koodatut kohteet ovat keinotekoisia kuvia, joita voidaan tulostaa esimerkiksi A4-paperiarkeille ja jotka voidaan mitata automaattisesti tietokonealgoritmeillä. Matemaattinen malli määritetään hyödyntämällä 3-ulotteista kamerakalibrointihuonetta, johon kehitetyt koodatut kohteet asennetaan. Kaksi kaupallista monikamerajärjestelmää, jotka muodostuvat 6 ja 36 erillisestä kamerasta, kalibroitiin onnistuneesti ehdotetulla menetelmällä. Tulokset osoittivat, että menetelmä tuotti tarkat estimaatit monikamerajärjestelmän geometrisille parametreille ja että estimoidut parametrit vastasivat hyvin kameran sisäistä rakennetta. Työn toisessa osassa tutkittiin ehdotetulla menetelmällä kalibroidun monikamerajärjestelmän mittauskäyttöä liikkuvassa kartoitusjärjestelmässä (MMS). Tavoitteena oli kehittää ja tutkia korkean geometrisen tarkkuuden kartoitusmittauksia. Monikameramallia laajennettiin navigointilaitteiston paikannus ja kallistussensoreihin (GNSS/IMU) liittyvillä parametreillä ja ehdotettiin järjestelmäkalibrointimenetelmää liikkuvalle kartoitusjärjestelmälle. Kalibroidulla järjestelmällä saavutettiin senttimetritarkkuus suorapaikannusmittauksissa. Työssä myös esitettiin monikuville vastaavuuskartta, joka mahdollistaa metristen panoraamojen luonnin monikamarajärjestelmän kuvista. Kolmannessa osassa tutkittiin UAV:​​n liikeradan reaaliaikaista estimointia hyödyntäen yhteen kameraan perustuvaa menetelmää. Päätavoitteena oli kehittää monokulaariseen kuvaamiseen perustuva reaaliaikaisen samanaikaisen paikannuksen ja kartoituksen (SLAM) menetelmä. Työssä ehdotettiin moniresoluutioisiin kuvapyramideihin ja eteneviin suorakulmaisiin alueisiin perustuvaa sovitusmenetelmää. Ehdotetulla lähestymistavalla pystyttiin alentamaan yhteensovittamisen kustannuksia sovituksen tarkkuuden säilyessä muuttumattomana. Kardaanilukko (gimbal lock) tilanteen käsittelemiseksi toteutettiin uusi kulmajärjestelmä. Tulokset osoittivat, että ehdotettu ratkaisu oli tehokas ja tarkka tilanteissa joissa kohde on lähes tasomainen. Suorituskyvyn arviointi osoitti, että kehitetty menetelmä täytti UAV:n reaaliaikaiselle reitinestimoinnille annetut aika- ja tarkkuustavoitteet. Työn viimeisessä osassa tutkittiin puulajiluokitusta käyttäen hyperspektri- ja RGB-kameralla varustettua UAV-järjestelmää. Tavoitteena oli tutkia uusien koneoppimismenetelmien käyttöä tarkassa puulajiluokituksessa ja lisäksi vertailla hyperspektri ja RGB-aineistojen suorituskykyä. Työssä verrattiin 3D-konvoluutiohermoverkkoa (3D-CNN) ja monikerroksista perceptronia (MLP). Molemmat luokittelijat tuottivat hyvän luokittelutarkkuuden, mutta 3D-CNN tuotti tarkimmat tulokset. Saavutettu tarkkuus oli parempi kuin aikaisemmat julkaistut tulokset vastaavilla aineistoilla. Hyperspektrisen ja RGB-datan yhdistelmä tuotti parhaan tarkkuuden, mutta myös RGB-kamera yksin tuotti tarkan tuloksen ja on edullinen ja tehokas aineisto monille luokittelusovelluksille

Selbstkalibrierung mobiler Multisensorsysteme mittels geometrischer 3D-Merkmale

Ein mobiles Multisensorsystem ermöglicht die effiziente, räumliche Erfassung von Objekten und der Umgebung. Die Kalibrierung des mobilen Multisensorsystems ist ein notwendiger Vorverarbeitungsschritt für die Sensordatenfusion und für genaue räumliche Erfassungen. Bei herkömmlichen Verfahren kalibrieren Experten das mobile Multisensorsystem in aufwändigen Prozeduren vor Verwendung durch Aufnahmen eines Kalibrierobjektes mit bekannter Form. Im Gegensatz zu solchen objektbasierten Kalibrierungen ist eine Selbstkalibrierung praktikabler, zeitsparender und bestimmt die gesuchten Parameter mit höherer Aktualität. Diese Arbeit stellt eine neue Methode zur Selbstkalibrierung mobiler Multisensorsysteme vor, die als Merkmalsbasierte Selbstkalibrierung bezeichnet wird. Die Merkmalsbasierte Selbstkalibrierung ist ein datenbasiertes, universelles Verfahren, das für eine beliebige Kombination aus einem Posenbestimmungssensor und einem Tiefensensor geeignet ist. Die fundamentale Annahme der Merkmalsbasierten Selbstkalibrierung ist, dass die gesuchten Parameter am besten bestimmt sind, wenn die erfasste Punktwolke die höchstmögliche Qualität hat. Die Kostenfunktion, die zur Bewertung der Qualität verwendet wird, basiert auf Geometrischen 3D-Merkmalen, die wiederum auf den lokalen Nachbarschaften jedes Punktes basieren. Neben der detaillierten Analyse unterschiedlicher Aspekte der Selbstkalibrierung, wie dem Einfluss der Systemposen auf das Ergebnis, der Eignung verschiedener Geometrischer 3D-Merkmale für die Selbstkalibrierung und dem Konvergenzradius des Verfahrens, wird die Merkmalsbasierte Selbstkalibrierung anhand eines synthethischen und dreier realer Datensätze evaluiert. Diese Datensätze wurden dabei mit unterschiedlichen Sensoren und in unterschiedlichen Umgebungen aufgezeichnet. Die Experimente zeigen die vielseitige Einsetzbarkeit der Merkmalsbasierten Selbstkalibrierung hinsichtlich der Sensoren und der Umgebungen. Die Ergebnisse werden stets mit einer geeigneten objektbasierten Kalibrierung aus der Literatur und einer weiteren, nachimplementierten Selbstkalibrierung verglichen. Verglichen mit diesen Verfahren erzielt die Merkmalsbasierte Selbstkalibrierung bessere oder zumindest vergleichbare Genauigkeiten für alle Datensätze. Die Genauigkeit und Präzision der Merkmalsbasierten Selbstkalibrierung entspricht dem aktuellen Stand der Forschung. Für den Datensatz, der die höchsten Sensorgenauigkeiten aufweist, werden beispielsweise die Parameter der relativen Translation zwischen dem Rigid Body eines Motion Capture Systems und einem Laserscanner mit einer Genauigkeit von ca. $1\,\mathrm{cm}$ bestimmt, obwohl die Distanzmessgenauigkeit dieses Laserscanners nur $3\,\mathrm{cm}$ beträgt

ViRoom - Low Cost Synchronized Multicamera System and its Self-Calibration

This paper presents a multicamera Visual Room (ViRoom)