Situational awareness is the key for maximizing the return of any operation, both for scientific and engineering assessments. SPACECLICK and INAF propose the OMNICAM, a novel camera system which can capture a 360° context view and an optically magnified portion of the same panorama with a single lens and sensor. The Bifocal Panoramic Lens (BPL) (designed by INAF researchers) is a lens capable of warping a panoramic Field-of-View (FoV) of 360° x 100° and simultaneously a round FoV of 20° in high-resolution. While the panoramic portion is always available for continuous monitoring surrounding environment, assets and ongoing operations, additionally the OMNICAM makes use of a pan/tilt mechanism to provide users of orientation capabilities for the magnified portion of the FoV

PERNECHELE, Claudio

POZZOBON, Riccardo

Tomasin, Marco

Turchi, Leonardo

OA@INAF - Istituto Nazionale di Astrofisica

2022Publication Year2023-02-06T15:15:14ZAcceptance in OA@INAFOMNICAM: Bifocal Panoramic Camera for Human and Robotic ExplorationTitleTurchi, Leonardo; POZZOBON, Riccardo; Tomasin, Marco; PERNECHELE, ClaudioAuthorshttp://hdl.handle.net/20.500.12386/33192Handle73rd International Astronautical Congress (IAC), Paris, France, 18-22 September 2022.  Copyright ©2022 by the International Astronautical Federation (IAF). All rights reserved. IAC-22,A3,IP,37,x72100         Page 1 of 5  IAC-22,A3,IP,37,x72100  OMNICAM: Bifocal Panoramic Camera for Human and Robotic Exploration  L. Turchia* , R. Pozzobonb , M. Tomasinc , C. Pernechelec  a Spaceclick S.r.l. Milan, Italy, info@spaceclick.com, b University of Padova, Dept. Of Geoscience, c INAF National Institute for Astrophysics, Padova, * Corresponding Author   Abstract Situational awareness is the key for maximizing the return of any operation, both for scientific and engineering assessments. SPACECLICK and INAF propose the OMNICAM, a novel camera system which can capture a 360° context view and an optically magnified portion of the same panorama with a single lens and sensor. The Bifocal Panoramic Lens (BPL) (designed by INAF researchers) is a lens capable of warping a panoramic Field-of-View (FoV) of 360° x 100° and simultaneously a round FoV of 20° in high-resolution. While the panoramic portion is always available for continuous monitoring surrounding environment, assets and ongoing operations, additionally the OMNICAM makes use of a pan/tilt mechanism to provide users of orientation capabilities for the magnified portion of the FoV. The small number of systems in OMNICAM leads to an overall reduction of weight and power consumption/complexity of redundant electronics as well as possible weak points. An ad-hoc developed software allows wireless remote control from ground control rooms and takes care of automatic de-warping of the captured frames, for producing fully nominal (flat) pictures of the surroundings, and additional pictures for the magnified pointed areas. The OMNICAM prototype has been field tested in the ESA PANGAEA geological field training for astronauts in November 2021, where was used for remote monitoring of field science operations. The concept has been proposed to ESA to support the next human exploration challenges, due to the uniqueness of such concept of camera system setup which can capture such extended FoV with a single camera sensor.   Keywords: Panoramic camera, optical zoom, situational awareness, surface exploration, photogrammetry, extended reality.     Acronyms/Abbreviations AR: Augmented Reality BPL: Bifocal Panoramic Lens CDF: (ESA) Concurrent Design Facility FoV: Field of View EVA: Extra Vehicular Activities ISRU: In-Situ Resource Utilization  INAF: Italian National Institute for Astrophysics  VR: Virtual Reality   1. Introduction Upcoming human and robotic planetary surface exploration will deal with harsh environments (e.g. impact craters, lava tubes, lava rilles, and rough surfaces over vast regions). Modern technology can be used by mission operations to maximize the return of any operation both in scientific (e.g. geological tasks such as documentation and sampling) and engineering (e.g. assembly, repairing, survey of equipment) assessments while minimising the risk of failure. The Artemis landing sites are in fact planned at locations with extreme environmental conditions that have never been visited during the Apollo missions. There are technological challenges to face in order to tailor modern technology to the requirements of these mission operations. High level of situational awareness is of high interest for the crew and the scientific backroom. Both can benefit from a camera feed which can enhance remote support by ground control rooms. OMNICAM is an innovative optical device capable to record a panoramic field (360°x100°) and capture simultaneously a magnified view of a portion of it, using a single image sensor.  2. OMNICAM’s Design The OMNICAM is comprised of software and software. The first includes the imaging device, the mechanical mount, mechanisms and motors, the computational board, and the batteries. The latter is a custom-made command and control software for real time interaction, preview streaming, and remote image snapshot capabilities. OMNICAM makes use of only one sensor and one lens to capture both the surroundings and an optically magnified portion of the panorama.   The concept is comprised of the following components:  - Bifocal Panoramic Lens (BPL) - Image sensor 2/3” format @ 5 Mpx - Two axes single motor for pan & tilt movement 73rd International Astronautical Congress (IAC), Paris, France, 18-22 September 2022.  Copyright ©2022 by the International Astronautical Federation (IAF). All rights reserved. IAC-22,A3,IP,37,x72100         Page 2 of 5 - Readout and conditioning electronics - Remote control software  2.1 Bifocal Panoramic Lens The BPL is a wide-angle lens which combines the capabilities of the fisheye and the omnidirectional lenses. The total field of view of the lens results in 360 degrees in azimuth angle and up to 270 degrees in zenithal angle. A sketch of how it works is shown in the Fig. 1.    Fig. 1. (top) The BPL optical diagram, (bottom) example of a frame/snapshot as it appears RAW in the imaging device.  On the lens focal plane an image sensor based on the Sony IMX264 CMOS (color) chip 2/3 inch format and resolution 5 Mpx takes place. It is useful underline in this context that a space COTS image sensor based on this Sony chip is present on the market with a 70 Krad TID. So all the tests developed on our earth-based system any be easily mirrored into a space qualified device.  2.2 Electronics & Software The electronics include the camera sensor, the computational & networking wireless boards, batteries, and servomotor controllers which are contained in the enclosure. Solar panels were then integrated on the outer surface of the enclosure to extend the use of OMNICAM as shown in Fig. 2.    Fig.2: OMNICAM deployed in the field in an ESA Analogue Astronaut Training Campaign.  The ground personnel gets a live feed from the field, and can orientate the optical zoom on specific portions of the landscape for recording high resolution pictures. An ad-hoc software de-warps the captured frames and renders flat pictures both panoramic and magnified of the areas of interest. The same software enables the ground to control the servomotors to pan/tilt and provide users orientation capabilities for the magnified portion of the FoV [1]. The integrated system and software have been developed by Spaceclick [2].  3. Applications For sustainable and scalable exploration missions engineering and scientific instruments will need to be interconnected for easy use by astronauts in Extra Vehicular Activities (EVA) or ground through remote control. OMNICAM’s are envisaged to be deployed either on tripod-like supports, on external infrastructure (e.g. habitats, landers), or anchored to rovers. In both cases, the system embeds a networking board and could also act as disruption tolerant data retransmission station. The OMNICAM has been field tested in the ESA PANGAEA geological field training for astronauts in November 2021, where allowed for remote monitoring of field science operations (Fig. 3) [3].  During operations, field pictures and videos are usually taken though standard wide cameras (containing one sensor) or double-fisheye 360° cameras (containing two sensors). Magnification of portions of the landscape 73rd International Astronautical Congress (IAC), Paris, France, 18-22 September 2022.  Copyright ©2022 by the International Astronautical Federation (IAF). All rights reserved. IAC-22,A3,IP,37,x72100         Page 3 of 5 in those frames requires a very high-resolution camera, in order not to lose too much in definition when digitally zooming-in. Using typical optical zoom would require the deployment of a different camera system, losing the panoramic capabilities. Monitoring of assets, or operational areas (e. g. live monitoring of a crater in shadow, from the rim) usually require orientating a dedicated camera, which can lead to losing situational awareness on other spots. Leveraging the OMNICAM, the BPL can be controlled from ground to focus on an optically magnified portion of the surroundings, with objective providing situational awareness and remote assets monitoring for human or robotics operations.  3.1 OMNICAM Tests in analogue field campaigns      Field training and technology testing on Earth is being carried out in places with similar, relevant geological features and operational conditions. This will help astronauts in performing geological investigations on the Moon and to maintain a meaningful level of situational awareness with science backrooms. Spaceclick contributes to supporting the ESA CAVES underground astronaut training, ESA PANGAEA [4] geological field training course and ESA PANGAEA-X test campaigns, which provide astronauts of the scientific and operational knowledge and practical skills to be effective field scientists during future planetary exploration missions to the Moon and Mars. During these campaigns, visual monitoring of the area surrounding astronauts’ activities is key for scientific documentation and for receiving effective ground support [5][6][7]. Magnification of portions of the landscape in those frames requires a very high-resolution camera, in order not to lose too much in definition when digitally zooming-in and therefore preserve fine details useful for science feedback from the ground support.  Using typical optical zoom would require the deployment of a different camera system, losing the panoramic capabilities and possibly adding failure points by the addition of sensors, lenses and redundant moving parts. Monitoring of assets, or operational areas (e.g. live monitoring of a crater in shadow, from the rim) usually require to orientate a dedicated camera, which can lead to losing situational awareness on other spots.  In the session, we could test the OMNICAM in two operational outdoor activities, in a lunar analogue environment. The ground personnel could get live feed from the field, and orientate the optical zoom on specific portions of the landscape for recording high resolution pictures of sampling activities. In the field the OMNICAM was also acting as a network retransmission station, since including a network node of compatible with the configuration used in ESA PANGAEA. During operations the ground could benefit from the feed effectively monitoring the overall operations environment. A GUI was developed in order Fig.3: (above) The field tests of the OMNICAM. The prototype was mounted on a tripod nearby the crew, simulating a rover (A). The panoramic picture showing the surroundings, including the crew operating and a portable cart in green (B), the optically enlarged portion for monitoring sampling activities (C). Both the (B) and (C) where captured via a single high-resolution snapshot and split/de-warped by the onboard software. (below) An artistic conceptual representation of what the OMNICAM could provide during space operations, from monitoring of assets, infrastructure, external payloads, to scientific and geological operations. D . 73rd International Astronautical Congress (IAC), Paris, France, 18-22 September 2022.  Copyright ©2022 by the International Astronautical Federation (IAF). All rights reserved. IAC-22,A3,IP,37,x72100         Page 4 of 5 to take control during operations of the servomotors which helped to monitor the crew field operations. The manual possibility to adjust the exposure time, in order to compensate the illumination conditions depending on the magnified portion orientation was also extremely helpful and will also impact possible operations on the Moon surface as well, where lighting conditions transition between lit and shaded areas is often extreme. Finally, the  possibility to snap timestamped pictures of the whole setting from ground used to further create activity logs proved to be extremely helpful.    Fig.4: OMNICAM integrated in a rover part of a monitoring, mapping and sampling swarm, and tested at the ESA-ESRIC robotic challenge 2021.  3.2 OMNICAM at the ESA-ESRIC Robotic Challenge 2021 A prototype of the OMNICAM has been tested on the ESA-ESRIC robotic challenge for prospecting lunar resources, where was mounted on rovers and interfaced with their communications system. The participation to the challenge was done in partnership between Spaceclick and the MIST Lab, of the Polytechnique Montréal, Canada. The setup envisaged the deployment of three rovers capable of mapping, monitoring, and sampling. The monitoring portion was delegated to the OMNICAM (Fig. 4). Despite some hiccups with the network configuration of two rovers, the ground team (remotely controlling the rovers and other instruments from a time delayed control room), could successfully control and acquire snapshots from the field through the OMNICAM.  3.4 Photogrammetric Rendering During our field tests we realized we could leverage from a series of snapshots taken at different locations, to dynamically render and provide to the ground team a 3D version of the environment surrounding the camera.  At the same time, optical zoom periscope can be pointed to a specific asset (or outcrop, sample, etc.) for scientific assessments and documentation. Images taken by the OMNICAM’s at different points of view can be used indeed in combination with another Spaceclick project (MapNCloud, under development) aiming to enable high-resolution and fast photogrammetric distance rendering out of simple pictures, hence providing 3D objects for VR/AR applications.  OMNICAM are envisaged to be deployed either on tripod-like supports or anchored to rovers. In both cases, the system embeds a networking board and could also act as disruption tolerant data retransmission station. In this setup, the utilization of multiple OMNICAMs deployed in the field, would hence provide spatial information from multiple point-of-views. This would allow an overlapping of 360° panoramic FoV for the surroundings. We are currently working in improving the quality of the 3D rendering and looking at utilisation of Machine Learning to identify and track personnel and assets in the field environment, via optical trilateration.  \ Fig. 5: (Left) Deployment of a set of OMNICAMs for situational awareness and concurrent 3D photogrammetric reconstruction of a site of interest through the orientable magnifier. (Right) Example of a 3D reconstruction of a boulder [AS17-146 of Apollo 17 – credits: NASA]  As shown in the Fig. 5, the image from multiple point of views can be used to track and monitor any object of interest: in orange we have an object of interest to which the four OMINCAM’s collectively focus on without any loss of resolution. The optical magnification would allow for detailed views in full resolution and the combination of real time photogrammetry would benefit the tracking and archival of any object of interest placed in the operational environment.  3.5 Future Applications The OMNICAM is an innovative camera concept designed to support lunar human exploration activities. The camera lens, sensor and boards employed are compatible with space grading, and we are working and looking for opportunities for rising the current TRL 3 of 73rd International Astronautical Congress (IAC), Paris, France, 18-22 September 2022.  Copyright ©2022 by the International Astronautical Federation (IAF). All rights reserved. IAC-22,A3,IP,37,x72100         Page 5 of 5 the system to higher in the next. The concept has been presented to ASI (Italian Space Agency) and ESA, and we hope will be considered as additional device for any suite of tools targeting lunar science operations improvement, such as the Electronic Field Book [6].  A modified version of the BPL has also taken part an ESA CDF study concerning design of robotics lunar caves exploration [8][9][10] and another adaptation has been considered for a satellite star-tracker [11].  4. Conclusions  To our knowledge there is no similar camera system setup which is capable of capturing such extended FoV with a single camera sensor, the OMNICAM results being therefore a unique device in the international context. The novelty of panoramic camera which uses optical zoom to perform situational awareness while including a few components which can be space and rad-graded makes OMNICAM a great candidate to support  and enhance planetary surface exploration carried machines and astronauts.    Acknowledgements We would like to thank Sébastien Ruhlmann, Leonardo Facchini and Jesús Asenjo Terrón for their work on the prototypes and for the dedication during field testing of the OMNICAM concept. References [1] Pernechele C. (2016), “Hyper hemispheric lens” Opt Expr., 24(5), 5014, doi: 10.1364/OE.24.005014. [2] Spaceclick S.r.l. Milan, Italy https://www.spaceclick.com/ [3] ESA CAVES & PANGAEA https://www.esa.int/Science_Exploration/Human_and_Robotic_Exploration/CAVES_and_Pangaea/What_is_Pangaea  [4] Sauro, F., et al. (2018) in EGU, General Assembly 2018. Vienna. [5] Sauro, F., et al., “Volcanic Caves of Lanzarote: A Natural Laboratory for Understanding Volcano-Speleogenetic Processes and Planetary Caves, in Lanzarote and Chinijo Islands Geopark: From Earth to Space”, 2019, Springer. p. 125-142.   [6]  Turchi, L, et al. (2021) “The Electronic FieldBook: A system for supporting distributed field science operations during astronaut training and human planetary exploration”, Planetary and Space Science. [7]  Turchi, L, et al. (2021) “The Electronic FieldBook (EFB) as a Tool for Caves Science Operations”, CREG Journal, British Cave Research Association. [8] Pernechele, C., et al., “Bifocal panoramic stereoscopic camera for lunar exploration”, EPSC 2020. [9] Borrmann, D., et al., “Lunar caves exploration with the Daedalus spherical robot”, 52th LPSC, 2548, #2073, 2021. [10] Pernechele, C., et al., “Daedalus-CAM: an immersive stereoscopic camera to explore lunar caves”, NESF-ELS, 2021. [11] Opromolla, R., et al., “A new star tracker concept for satellite attitude determination based on a multi-purpose panoramic camera”, Acta Astronautica.     

OMNICAM: Bifocal Panoramic Camera for Human and Robotic Exploration

https://openaccess.inaf.it/bitstream/20.500.12386/33192/1/OMNICAM%20-%20IAC.pdf

OMNICAM: Bifocal Panoramic Camera for Human and Robotic Exploration

Abstract

Similar works

Full text

Available Versions

OA@INAF - Istituto Nazionale di Astrofisica