It has long been recognized as gathering of people is one of the major risk factor
in spreading of viral epidemics. Social distancing is then one of the most simple and powerful
system to mitigate the spread of infections.We explore here the possibility of monitoring
public people’s gathering by using a novel bifocal omnidirectional lens designed by INAF
jointly with deep learning-based algorithms. The paper briefly describe how the lens works,
the applied deep learning algorithms and the preliminary results of the trials

Bondanza, I.

Capucci, A

Grandis, N.

Lessio, L.

Paoletti, L.

Pernechele, C.

Sparc, S.

Tomasin, M.

Yusupova, R.

OA@INAF - Istituto Nazionale di Astrofisica

2022Publication Year2023-07-12T15:38:31ZAcceptance in OA@INAFOmnidirectional people's gathering monitoring by using deep learning algorithmsTitlePernechele, C.; Grandis, N.; Bondanza, I.; Capucci, A; Lessio, L.; et al.Authorshttp://hdl.handle.net/20.500.12386/34273HandleMEMORIE DELLA SOCIETA ASTRONOMICA ITALIANAJournal93NumberMem. S.A.It. Vol. 93, 91© SAIt 2022 Memorie dellaOmnidirectional people’s gathering monitoringby using deep learning algorithmsClaudio Pernechele1, Nicola Grandis2, Irene Bondanza2, Alessandro Capucci2,Luigi Lessio1, Lorenzo Paoletti1, Silviu Sparc2, Marco Tomasin1, Renata Yusupova21 Istituto Nazionale di Astrofisica–OAPD, Vicolo dell’Osservatorio, 5, 35122, Padova2 ASC27 srl, Lungotevere Michelangelo, 9, 00192, RomaReceived: 14 June 2021; Accepted: 3 May 2022Abstract. It has long been recognized as gathering of people is one of the major risk factorin spreading of viral epidemics. Social distancing is then one of the most simple and power-ful system to mitigate the spread of infections. We explore here the possibility of monitoringpublic people’s gathering by using a novel bifocal omnidirectional lens designed by INAFjointly with deep learning-based algorithms. The paper briefly describe how the lens works,the applied deep learning algorithms and the preliminary results of the trials.1. IntroductionPanoramic cameras have been realized in manydifferent fashions for more than one century.Panoramic images can be created in a varietyof ways: the oldest and most popular is one oftaking a set of pictures around the observer andstitch them in a post-processing phase. Threemain classes of panoramic cameras can be dis-tinguished:– Panoramic vision systems based on mov-ing cameras;– Panoramic vision that make use of a set ofindependent lenses (and sensors) looking atdifferent points on the surrounding space;– Motionless panoramic cameras (omnidi-rectional lens).A pan-tilt mechanism is usually the mainpart of the first family. A standard perspectivecamera pans and tilts capturing a number ofperspective shots of the scene. A panoramicview is then created composing such images.The pan-tilt camera requires a significantamount of time and it has to stop while cap-turing the shots that are used to compose thepanoramic image; this represents a great draw-back: a moving camera is not able to fur-nish a simultaneous vision of the surround-ing environment and dynamic information arelost. For this reason, the system is useful forstatic scenery (panoramic photography, art di-agnostic, architectural view, etc...). Panoramicsensors with multiple cameras (i. e., the sec-ond class) offer a simultaneous vision of thepanoramic space and have satisfactory spatialresolution but they are rather expensive, bulkyand, usually, require a complicated calibra-tion. To the third class belong the omnidirec-tional cameras. A motionless panoramic cam-era has many advantages with respect to theother classes: it makes use of only one detector(on the contrary of class two) and doesn’t makeuse of moving motor (as in the first class); allof these aspects reflects on lower power con-92 Pernechele et al.: People’s gathering monitoringTable 1. Panoramic cameras classesLens Class Dynamic Complexity Size Post Spatialscene Processing ResolutionMultiple Lenses Yes High Large Hard HighMotorized lens No Medium Medium Hard HighOmnidirectional Yes Low Small Easy Lowsumption, lower cost, lower weight, smallerbulky and, furthermore, reduce possible fail-ure points. The use of this family is fruitfulwhen the recording of a dynamic scene, com-bined with lightness, is necessary (as in roboticand anti-collision systems, self-driving, scenemonitoring etc...). An evident drawback of thisfamily of panoramic camera is one of furnish-ing a relative low spatial resolution informa-tion of the surrounding environment. These pa-rameters are summarized in the table 1. We de-scribe here how the use of an omnidirectionalcamera, based on a novel lens designed by theINAF researchers, may be useful in monitoringpeople’s gathering. An omnidirectinal camerahas many advantages in solving these tasks:– It does not use motorized parts (powerconsuming, possible point failure, mainte-nence cost);– It avoids the use of multiple cameras(complex placing and maintenance, postprocessing image stitching, heavy datastream);– The apparent weakness of its intrinsic lowresolution turn out as an advantage, due tothe privacy issue.2. Bifocal Panoramic LensThe bifocal panoramic lens (BPL) is an opti-cal device which records a panoramic field ofview (360◦ × 100◦) and, simultaneously, an en-larged part of this panoramic field. It has beendesigned and patented by INAF researchers(Pernechele 2016). A sketch of how it worksis shown in the figure 1. The panoramic field isimaged with a donut-like shape, while the cen-ter of the donut is filled with the enlarging partof the lens, using a flat folding mirror settled inthe lens frontal optics. In this case one obtains amagnification of 3×with respect to the paraxialfocal length. We designed the panoramic armwith a focal length of 2 mm (working at F/3.5),while the frontal optics, used for magnifyingthe field, has a focal length of 6 mm (at F/3.5too).Fig. 1. A graphical explanation of how the BPLworks.2.1. BPL Mapping FunctionThe way in which the 3D object space ismapped onto the 2D focal plane is describedby the lens mapping function. With referenceto the figure 2, let be the zenith angle Z mea-Pernechele et al.: People’s gathering monitoring 93sured from zenith, down to the maximum fieldof view (Zmax). The angular field distributionfunction in the focal plane ψ(Z) is, in general,not linear with respect to Z and an amount ofoptical distortion would be present. The focallength f is changing along the field. In par-ticular, f = f (Z) (with f (0) the paraxial focallength) and this causes the deformation of theimage.Fig. 2. BPL sky mapping function.In the case of a fisheye Zmax = 90◦, whilein the BPL it may go up to 135◦. Hereinafterwe designate with f the paraxial focal length,with f = f (0). The space mapping functionR(Z) = f (Z) may have different forms. The”perfect” undistorted map of the object space isone where R = f tan(Z). In this way every pointin the space is mapped maintaining the sameangular distribution into the focal plane. Thelens works like a pinhole camera and objectstraight lines remain straight (distortion-free)in the focal plane. This function is known asthe ”perspective projection”. It is well knownthat it has no practical sense for wide anglelenses, because the focal plane would be in-finitely extended and the entrance pupil (thepin-hole) would be completely obscured forZ = 90◦. Equidistant (linear scaled) projec-tion is one where R = f Z: it maintains angulardistances. This lens projection scheme is alsoknow as F-Theta. The most general projectionfunction has the form:R(Z) = fψ(Z) = f k1sin(k2Z). (1)Within this class of mapping functions,high compression of the marginal objects ispresent. Among those there is the equisolid an-gle projection (k1 = 2, k2 = 0.5), which main-tains surface relations: each pixel in the detec-tor subtends an equal solid angle, i. e. an equalarea on the unit sphere. Finally we cite the or-thographic projection (k1 = k2 = 1) whereR = f sin(Z). This mapping function maintainsplanar illuminance. Here the marginal fieldsare extremely compressed at the focal planeand it make sense only for Z < 90◦. The pro-jection function of the BPL lies in the midwaybetween the equisolid angle and orthographicprojections (black continuous line in Figure 2),with f = 2mm, k1 = 1.48 and k2 = 0.64. Thoseparameters are necessary during the image dis-tortion corrections (the unwarping process).2.2. BPL ApplicationsDue to its optical multi-functionalities (ex-tremely large field for contextual imaging andsimultaneous higher resolution for scientificpurposes) jointly with its compactness and mo-tionless fashion, the BPL may find many ap-plications in space applications. We proposeits applications as multi-functional tracker formini- and micro-satellites (Pernechele et al.2018; Opromolla et al. 2017), stereoscopic94 Pernechele et al.: People’s gathering monitoringauto-driving and simultaneous scientific capa-bilities for planetary rovers (Pernechele et al.2020), as an immersive stereo-camera for ex-ploring lunar caves (Borrmann et al. 2021;Pozzobon et al. 2021; Pernechele et al. 2021)and as an ibrid GNSS-optical navigation sys-tem for placing seismometers in lunar experi-ments (Harms et al. 2021).Fig. 3. The manufactured lens.3. The PANCAMA remote panoramic monitoring system(PANCAM) based on the bifocal panoramiclens has been assembled following the ar-chitecture outlined in figure 4. The imagefrom a CMOS sensor chip (Sony IMX264,5Mpx, 2/3 inch format) is acquired by anmicro-PC (Odroid-XU4, octacore CPUs). Themicro-PC also processes the image unwarping,in accordance with the equation (1) of theprevious section. The frame transfer travels@USB3 speed. The unwarped images are sentby a client wi-fi toward the remote wi-fi server(master). The ongoing images may be seen bya remote (wi-fi connected) devices, such e.g.smartphones, pc tablets, etc. . . by means ofan dedicated user interface. Images are alsosaved at the ACQ board-level onto a microSDflash memory at a frame rate of 1 fps. Thesystem is supplied by a DC-DC converterwhich provides 1A @12V. As an example, theimage seen in the central panel of figure 1 hasbeen acquired by this PANCAM system.Fig. 4. The PANCAM architecture.4. System TrialIn order to test the deep learning algorithmon realistic BPL-based PANCAM acquisitions,we record an images stream in an urban envi-ronment.4.1. Image AcquisitionThe system has been tested in a public areas(Piazza dei Signori in the city of Padova down-town). A ”frog view” of the area, with thePANCAM settled in the place, is shown in fig-ure 5.The site environment shows a high con-trast between sun illuminated and shaded ar-eas, typical for extremely wide angle lenses.Fig. 5. The PANCAM settled in the place.Pernechele et al.: People’s gathering monitoring 95By running the DL algorithm described in sec-tion 4 it was possible to identify the personsin the shaded region, despite of the low levelof signal in the scene. Another important re-sult has regarded the identification target al-gorithm. Indeed, it works also in the original(before to apply the unwarping algorithm) im-age. If confirmed in future trials, this fact couldmake not necessary the application of the time-consuming equation (1) in section 2.1. An ex-ample is shown in figure 6, where a personhas been identified in the shaded area (red andgreen boxes in, respectively, original and un-warped images), although surrounded by a se-ries of small trees, which are very similar inshape and dimension. Discriminate a personfrom such similar silhouettes is a valid testshowing the goodness of the DL running algo-rithm.Fig. 6. Example of DL algorithm results.Top:original image; bottom: unwarped image.4.2. Deep Learning AlgorithmThe acquired images are then been processedusing the ”SafePeople” software developed bythe ASC27 company (www.asc27.com). Thissoftware, based on artificial intelligence mod-els, has been originally developed for differentpurposes.Fig. 7. Image rotated of 90 degrees (upper pan-els), resulting into four copies and (bottompanels) preprocessing of each images.In this case we have used models for ObjectDetection to identify people and models for so-96 Pernechele et al.: People’s gathering monitoringFig. 8. Resulting image as the concatenation ofthe four images of Fig. 7.cial distancing monitoring and prediction. Forpeople object detection models we adopted twodifferent approaches. In this respect, due to thedistortion of the original images, the algorithmneed to take into account different angles whendetecting targets.In the first approach, the original imagefrom the lens is used and multiple rotationson the image of 90° degrees each are applied(see upper panels in figure 7). Therefore, foreach image, we use four rotated copies of it inoredere to detect objects. Moreover, as in eachrotated image the relevant part is the upperone, each rotated image is sliced into sixteensub-windows and the detection algorithm isapplied for the most relevant portions (see bot-tom panels in figure 7). As some sub-windowsmay overlap, it is applied a post processingalgorithm called Non Maximum Suppression(NMS). This is a standard technique utilizedto concatenate all the detected bounding boxes(figure 8).The second approach is based on a customSSD (Single Shot MultiBox Detector) networkand can be used both with original and un-warped images, by applying the algorithm onthe entire image. Regardless of the type of im-ages, an annotator has defined bounding boxesfor the targets to be detected, people in thiscase.Fig. 9. Original images with people detection.An important aspect is the unconventionalimages that the bifocal panoramic lens cap-tures, therefore the annotator must processthem taking into account which targets arePernechele et al.: People’s gathering monitoring 97Table 2. AP: mean precision of a class, mAP: mean of APs of all classes, Speed V100: averagedspeed performances using a V100 GPU machine.Model mAPval 0.5:0.95 mAPtest 0.5:0.95 mAPval 0.5 Speed V100 (ms)MD01A 34.2 35.8 54.9 3.8MD01B 41.3 43.2 62.8 4.2MD01C 44.8 47.5 65.5 5.3MD01D 49.5 50.1 67.2 8.0more relevant and which are confusing for thealgorithm. Furthermore, regarding the originalimages, objects may not reflect the canonicalproportion because of the lens distortion.Given the labelled dataset, the detection al-gorithm is trained with a portion of it and thenthe remaining part of the dataset is used to testthe algorithm. In the figure [9] some samplesof the detected images are showed, taken fromthe dataset that contains the original images.Accuracy and performance of the custom ob-ject detection models can be improved withfurther works and training but, actually, theyalready shown a good quality.After the identification of people throughobject detection, the information of the relativebounding boxes are processed by a network forsocial distancing, in order to identify possibledangerous situations.In table 2 the performances of the differentmodels, in terms of precision, are shown.5. ConclusionsIn order to test an omnidirectional people’sgathering monitoring system based on deeplearning algorithms, we developed and testeda panoramic camera (PANCAM) mounting abifocal panoramic lens (designed by INAF).Trials running on a realistic environment, asquare in the Padova downtown, show that, de-spite of the high contrast of the scene, the sys-tem works appropriately in achieving this task.The PANCAM is a compact optical systemand may monitor extremely large areas, avoid-ing the use of any motorized element. Thismakes the PANCAM system a valid alterna-tive to most common multi-camera and pan-tilt video surveillance systems. Moreover, theintrinsic low resolution of the PANCAM, turnout into a positive fact, saving from privacy is-sues.Acknowledgements. We are grateful to INAF andMUR for support this job by means of dedicatedcovid-contrast funds.ReferencesBorrmann, D., et al. 2021, LPSC, vol. 2548 (id.2073).Harms, J., et al. 2021, APJ, doi:10.3847/1538-4357/abe5a7Opromolla, R., et al. 2017, Acta Astronautica,doi: 10.1016/j.actaastro.2017.08.020, 2017.Pernechele, C., 2016, Optics Express, doi:10.1364/OE.24.005014Pernechele, C., et al. 2018, Adv. in Space Res.,doi:10.1016/j.asr.2018.02.025, 2018.Pernechele, C., et al. 2020, EPSC, doi: 10.5194/epsc2020-557.Pernechele, C., et al. 2021, NESF ELS, inpress.Pozzobon, R., et al. 2021, LPSC, vol. 2548 (id.1886).Simioni, E., et al. 2020, ISPR, doi: 10.5194/isprs-archives-XLIII-B3-2020-1151-2020.

Omnidirectional people's gathering monitoring by using deep learning algorithms

https://openaccess.inaf.it/bitstream/20.500.12386/34273/1/2022MmSAI..1...91P.pdf

Omnidirectional people's gathering monitoring by using deep learning algorithms

Abstract

Similar works

Full text

Available Versions

OA@INAF - Istituto Nazionale di Astrofisica