A pipeline for automated processing of declassified Corona KH-4 (1962-1972) stereo imagery

This study was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (XDA20100300) and the Swiss National Science Foundation (200021E 177652/1) within the framework of the DFG Research Unit GlobalCDA (FOR2630).The Corona KH-4 reconnaissance satellite missions acquired panoramic stereo imagery with high spatial resolution of 1.8–7.5m from 1962-1972. The potential of 800,000+ declassified Corona images has not been leveraged due to the complexities arising from handling of panoramic imaging geometry, film distortions and limited availability of the metadata required for georeferencing of the Corona imagery. This paper presents the Corona Stereo Pipeline (CoSP): A pipeline for processing of Corona KH-4 stereo panoramic imagery. CoSP utilizes deep learning based feature matcher SuperGlue to automatically match features point between Corona KH-4 images and recent satellite imagery to generate Ground Control Points (GCPs). To model the imaging geometry and the scanning motion of the panoramic KH-4 cameras, a rigorous camera model consisting of modified collinearity equations with time-dependent exterior orientation parameters is employed. Using the entire frame of the Corona image, bundle adjustment with well-distributed GCPs results in an average standard deviation or σ0 of less than two pixels. We evaluate fiducial marks on the Corona films and show that pre-processing the Corona images to compensate for film bending improves the 3D reconstruction accuracy. The distortion pattern of image residuals of GCPs and y-parallax in epipolar resampled images suggest that film distortions due to long-term storage likely cause systematic deviations of up to six pixels. Compared to the SRTM DEM, the Corona DEM computed using CoSP achieved a Normalized Median Absolute Deviation of elevation differences of ≈ 4m over an area of approx. 4000km2 after a tile-based fine coregistration of the DEMs. We further assess CoSP on complex scenes involving high relief and glacierized terrain and show that the resulting DEMs can be used to compute long-term glacier elevation changes over large areas.PostprintPeer reviewe

Vulnerable road users and connected autonomous vehicles interaction: a survey

There is a group of users within the vehicular traffic ecosystem known as Vulnerable Road Users (VRUs). VRUs include pedestrians, cyclists, motorcyclists, among others. On the other hand, connected autonomous vehicles (CAVs) are a set of technologies that combines, on the one hand, communication technologies to stay always ubiquitous connected, and on the other hand, automated technologies to assist or replace the human driver during the driving process. Autonomous vehicles are being visualized as a viable alternative to solve road accidents providing a general safe environment for all the users on the road specifically to the most vulnerable. One of the problems facing autonomous vehicles is to generate mechanisms that facilitate their integration not only within the mobility environment, but also into the road society in a safe and efficient way. In this paper, we analyze and discuss how this integration can take place, reviewing the work that has been developed in recent years in each of the stages of the vehicle-human interaction, analyzing the challenges of vulnerable users and proposing solutions that contribute to solving these challenges.This work was partially funded by the Ministry of Economy, Industry, and Competitiveness of Spain under Grant: Supervision of drone fleet and optimization of commercial operations flight plans, PID2020-116377RB-C21.Peer ReviewedPostprint (published version

Challenges of multi-view satellite stereo reconstruction pipelines and some contributions on key stages.

Satellite imagery is quickly gaining in importance, with Earth observation satellites producing daily images from all the points of the globe, both commercially and freely available. In this thesis we concentrate on surface reconstruction from visible light satellite images through stereo-vision. Given two images of a scene from different known viewpoints, the objective of stereo is to estimate the most likely 3D shape or depth that explains those images. When more than two images are available, multi-view stereo (MVS) can be applied working by pairs and integrating the reconstructions (pair-wise MVS) or deriving a reconstruction from all the images at a time (true MVS). In the case of satellite images, MVS has traditionally been performed with pair-wise approaches where the multiple views are treated by pairs doing traditional two-view stereo and then aggregating the digital surface models (DSM) from the pair-wise reconstructions to get the final result. Several well established commercial and open-source solutions organize their working pipelines in this way. This solutions mostly rely on classic stereo algorithms while deep learning (DL) alternatives are slowly being adapted to work in the pipelines. But the DL based approaches have not still clearly outperformed the traditional pipelines and there is room for much more work in this yet open area. A crucial issue that complicates the advance in this field is the scarce public datasets with well curated ground-truth. In this thesis a set of methods from different approaches of pair-wise and true MVS were evaluated and compared. For the comparison, classic and deep learning methods were adapted to work with satellite images and to correctly interface with S2P, a modular satellite stereo pipeline. The results obtained with deep learning methods showed the potential of using this kind of algorithms on satellite images as a step in a classic pipeline or as an end-to-end MVS solution. Considering pair-wise MVS, besides the stereo matching, two other steps are crucial to achieve a good reconstruction: (a) the selection of the most appropriate pairs, and (b) the fusion of the DSMs reconstructed from the pairs. For pair selection, a novel strategy based on the simulation of satellite images was devised and can order the pairs in a more consistent way than commonly used heuristics. For the simulation of images, a tool that can generate views from an artificial 3D scene was developed. Regarding the fusion of DSMs, an iterative scheme based on the bilateral filtering was conceived showing to be a robust and performant method. Improvements in other stages of the baseline stereo pipeline and the processing and analysis of point clouds were also part of the topics addressed during the thesis.Los satélites que toman imágenes de la Tierra son cada vez más numerosos, produciendo imágenes diarias de todos los puntos del globo, tanto gratuitas como de pago. En esta tesis nos concentramos en la reconstrucción de superficies a partir de imágenes de satélite de luz visible a través de estereovisión. Dadas dos imágenes de una escena desde diferentes puntos de vista conocidos, el objetivo del estéreo es estimar la forma o profundidad 3D más probable que explica esas imágenes. Cuando hay más de dos imágenes disponibles, se puede aplicar el estéreo multivista (MVS) trabajando por pares e integrando las reconstrucciones (MVS por pares)o derivando una reconstrucción de todas las imágenes a la vez (MVS “real”). En el caso de las imágenes de satélite, el MVS se ha realizado tradicionalmente con enfoques por pares, en los que las múltiples vistas se tratan por pares realizando estéreo tradicional de dos vistas y luego fusionando los modelos digitales de superficie (DSM) de las reconstrucciones por pares para obtener el resultado final.Varias soluciones comerciales y de código abierto bien establecidas organizan sus pipelines de trabajo de este modo. Estas soluciones se basan principalmente en algoritmos de estéreo clásicos, mientras que las alternativas de aprendizaje profundo(AP) se están adaptando poco a poco para funcionar en los pipelines. Pero los resultados de los métodos basados en AP no han superado claramente a los de los pipelines tradicionales y queda mucho por hacer en este campo aún abierto. Una cuestión crucial que complica el avance en este campo es la escasez de conjuntos de datos públicos con altura conocida. En la tesis se evaluaron y compararon un conjunto de métodos de diferentes enfoques de MVS por pares y real. Para la comparación, se adaptaron métodos clásicos y de aprendizaje profundo para trabajar con imágenes de satélite y para interactuar correctamente con S2P, un pipeline modular de estereo satelital. Los resultados obtenidos con los métodos de aprendizaje profundo mostraron el potencial del uso de este tipo de algoritmos en imágenes de satélite como un paso en un pipeline estéreo clásico o como una solución MVS de extremo a extremo. Si se considera el MVS por pares, además del matching estéreo, hay otros dos pasos cruciales para lograr una buena reconstrucción: (a) la selección de los pares más apropiados, y (b) la fusión de los DSMs reconstruidos a partir de los pares. Para la selección de pares, se concibió una estrategia novedosa basada en la simulación de imágenes de satélite que puede ordenar los pares de forma más consistente que las heurísticas utilizadas habitualmente. Para la simulación de imágenes, se desarrolló una herramienta que puede generar vistas a partir de una escena 3D artificial. En cuanto a la fusión de DSMs, se desarrolló un esquema iterativo basado en el filtrado bilateral que demostró ser un método robusto. Las mejoras en otras etapas del pipeline estéreo satelital y el procesamiento de nubes de puntos también formaron parte de los temas abordados durante la tesis

Electronics for Sensors

The aim of this Special Issue is to explore new advanced solutions in electronic systems and interfaces to be employed in sensors, describing best practices, implementations, and applications. The selected papers in particular concern photomultiplier tubes (PMTs) and silicon photomultipliers (SiPMs) interfaces and applications, techniques for monitoring radiation levels, electronics for biomedical applications, design and applications of time-to-digital converters, interfaces for image sensors, and general-purpose theory and topologies for electronic interfaces

UAV or Drones for Remote Sensing Applications in GPS/GNSS Enabled and GPS/GNSS Denied Environments

The design of novel UAV systems and the use of UAV platforms integrated with robotic sensing and imaging techniques, as well as the development of processing workflows and the capacity of ultra-high temporal and spatial resolution data, have enabled a rapid uptake of UAVs and drones across several industries and application domains.This book provides a forum for high-quality peer-reviewed papers that broaden awareness and understanding of single- and multiple-UAV developments for remote sensing applications, and associated developments in sensor technology, data processing and communications, and UAV system design and sensing capabilities in GPS-enabled and, more broadly, Global Navigation Satellite System (GNSS)-enabled and GPS/GNSS-denied environments.Contributions include:UAV-based photogrammetry, laser scanning, multispectral imaging, hyperspectral imaging, and thermal imaging;UAV sensor applications; spatial ecology; pest detection; reef; forestry; volcanology; precision agriculture wildlife species tracking; search and rescue; target tracking; atmosphere monitoring; chemical, biological, and natural disaster phenomena; fire prevention, flood prevention; volcanic monitoring; pollution monitoring; microclimates; and land use;Wildlife and target detection and recognition from UAV imagery using deep learning and machine learning techniques;UAV-based change detection

Architectures and synchronization techniques for distributed satellite systems: a survey

Cohesive Distributed Satellite Systems (CDSSs) is a key enabling technology for the future of remote sensing and communication missions. However, they have to meet strict synchronization requirements before their use is generalized. When clock or local oscillator signals are generated locally at each of the distributed nodes, achieving exact synchronization in absolute phase, frequency, and time is a complex problem. In addition, satellite systems have significant resource constraints, especially for small satellites, which are envisioned to be part of the future CDSSs. Thus, the development of precise, robust, and resource-efficient synchronization techniques is essential for the advancement of future CDSSs. In this context, this survey aims to summarize and categorize the most relevant results on synchronization techniques for Distributed Satellite Systems (DSSs). First, some important architecture and system concepts are defined. Then, the synchronization methods reported in the literature are reviewed and categorized. This article also provides an extensive list of applications and examples of synchronization techniques for DSSs in addition to the most significant advances in other operations closely related to synchronization, such as inter-satellite ranging and relative position. The survey also provides a discussion on emerging data-driven synchronization techniques based on Machine Learning (ML). Finally, a compilation of current research activities and potential research topics is proposed, identifying problems and open challenges that can be useful for researchers in the field.This work was supported by the Luxembourg National Research Fund (FNR), through the CORE Project COHEsive SATellite (COHESAT): Cognitive Cohesive Networks of Distributed Units for Active and Passive Space Applications, under Grant FNR11689919.Award-winningPostprint (published version

Practical Processing Techniques for Magnetic 3D Motion Tracking

Tohoku University北村喜文課

Learning the surroundings: 3D scene understanding from omnidirectional images

Las redes neuronales se han extendido por todo el mundo, siendo utilizadas en una gran variedad de aplicaciones. Estos métodos son capaces de reconocer música y audio, generar textos completos a partir de ideas simples u obtener información detallada y relevante de imágenes y videos. Las posibilidades que ofrecen las redes neuronales y métodos de aprendizaje profundo son incontables, convirtiéndose en la principal herramienta de investigación y nuevas aplicaciones en nuestra vida diaria. Al mismo tiempo, las imágenes omnidireccionales se están extendiendo dentro de la industria y nuestra sociedad, causando que la visión omnidireccional gane atención. A partir de imágenes 360 capturamos toda la información que rodea a la cámara en una sola toma.La combinación del aprendizaje profundo y la visión omnidireccional ha atraído a muchos investigadores. A partir de una única imagen omnidireccional se obtiene suficiente información del entorno para que una red neuronal comprenda sus alrededores y pueda interactuar con el entorno. Para aplicaciones como navegación y conducción autónoma, el uso de cámaras omnidireccionales proporciona información en torno del robot, person o vehículo, mientras que las cámaras convencionales carecen de esta información contextual debido a su reducido campo de visión. Aunque algunas aplicaciones pueden incluir varias cámaras convencionales para aumentar el campo de visión del sistema, tareas en las que el peso es importante (P.ej. guiado de personas con discapacidad visual o navegación de drones autónomos), un número reducido de dispositivos es altamente deseable.En esta tesis nos centramos en el uso conjunto de cámaras omnidireccionales, aprendizaje profundo, geometría y fotometría. Evaluamos diferentes enfoques para tratar con imágenes omnidireccionales, adaptando métodos a los modelos de proyección omnidireccionales y proponiendo nuevas soluciones para afrontar los retos de este tipo de imágenes. Para la comprensión de entornos interiores, proponemos una nueva red neuronal que obtiene segmentación semántica y mapas de profundidad de forma conjunta a partir de un único panoramaequirectangular. Nuestra red logra, con un nuevo enfoque convolucional, aprovechar la información del entorno proporcionada por la imagen panorámica y explotar la información combinada de semántica y profundidad. En el mismo tema, combinamos aprendizaje profundo y soluciones geométricas para recuperar el diseño estructural, junto con su escala, de entornos de interior a partir de un único panorama no central. Esta combinación de métodos proporciona una implementación rápida, debido a la red neuronal, y resultados precisos, gracias a lassoluciones geométricas. Además, también proponemos varios enfoques para la adaptación de redes neuronales a la distorsión de modelos de proyección omnidireccionales para la navegación y la adaptación del dominio soluciones previas. En términos generales, esta tesis busca encontrar soluciones novedosas e innovadoras para aprovechar las ventajas de las cámaras omnidireccionales y superar los desafíos que plantean.Neural networks have become widespread all around the world and are used for many different applications. These new methods are able to recognize music and audio, generate full texts from simple ideas and obtain detailed and relevant information from images and videos. The possibilities of neural networks and deep learning methods are uncountable, becoming the main tool for research and new applications in our daily-life. At the same time, omnidirectional and 360 images are also becoming widespread in industry and in consumer society, causing omnidirectional computer vision to gain attention. From 360 images, we capture all the information surrounding the camera in a single shot. The combination of deep learning methods and omnidirectional computer vision have attracted many researchers to this new field. From a single omnidirectional image, we obtain enough information of the environment to make a neural network understand its surroundings and interact with the environment. For applications such as navigation and autonomous driving, the use of omnidirectional cameras provide information all around the robot, person or vehicle, while conventional perspective cameras lack this context information due to their narrow field of view. Even if some applications can include several conventional cameras to increase the system's field of view, tasks where weight is more important (i.e. guidance of visually impaired people or navigation of autonomous drones), the less cameras we need to include, the better. In this thesis, we focus in the joint use of omnidirectional cameras, deep learning, geometry and photometric methods. We evaluate different approaches to handle omnidirectional images, adapting previous methods to the distortion of omnidirectional projection models and also proposing new solutions to tackle the challenges of this kind of images. For indoor scene understanding, we propose a novel neural network that jointly obtains semantic segmentation and depth maps from single equirectangular panoramas. Our network manages, with a new convolutional approach, to leverage the context information provided by the panoramic image and exploit the combined information of semantics and depth. In the same topic, we combine deep learning and geometric solvers to recover the scaled structural layout of indoor environments from single non-central panoramas. This combination provides a fast implementation, thanks to the learning approach, and accurate result, due to the geometric solvers. Additionally, we also propose several approaches of network adaptation to the distortion of omnidirectional projection models for outdoor navigation and domain adaptation of previous solutions. All in all, this thesis looks for finding novel and innovative solutions to take advantage of omnidirectional cameras while overcoming the challenges they pose.<br /

Applications of satellite technology to broadband ISDN networks

Two satellite architectures for delivering broadband integrated services digital network (B-ISDN) service are evaluated. The first is assumed integral to an existing terrestrial network, and provides complementary services such as interconnects to remote nodes as well as high-rate multicast and broadcast service. The interconnects are at a 155 Mbs rate and are shown as being met with a nonregenerative multibeam satellite having 10-1.5 degree spots. The second satellite architecture focuses on providing private B-ISDN networks as well as acting as a gateway to the public network. This is conceived as being provided by a regenerative multibeam satellite with on-board ATM (asynchronous transfer mode) processing payload. With up to 800 Mbs offered, higher satellite EIRP is required. This is accomplished with 12-0.4 degree hopping beams, covering a total of 110 dwell positions. It is estimated the space segment capital cost for architecture one would be about $190M whereas the second architecture would be about$250M. The net user cost is given for a variety of scenarios, but the cost for 155 Mbs services is shown to be about $15-22/minute for 25 percent system utilization