Recent studies conducted by some insurance companies highlighted that the most part of drivers do not know the meaning of the dashboard lights. This leads drivers to be dangerous for others and themselves. Hence, the need to provide drivers with tools that support them to always be aware of the state of their cars. This paper proposes a system for mobile devices that uses augmented reality to be able to give information on a particular dashboard lights. The system is implemented by combining the use of Simultaneous Localization and Mapping (SLAM) maps with central moment computation widely used in computer vision. Preliminary research results show that the proposed system achieves its goal enables a a real-time visual feedback

ERRA, UGO

Francesco Carotenuto

Vittorio Scarano

English

F. Carotenuto

U. Erra

SCARANO, Vittorio

Archivio della Ricerca - Università di Salerno

SLAM Map Application for Tracking Lights on Car Dashboards

Ugo Erra

Crossref

Archivio della Ricerca - Università della Basilicata

SLAM Map Application for Tracking Lightson Car DashboardsFrancesco Carotenuto1(B), Ugo Erra2, and Vittorio Scarano11 Dipartimento di Informatica, Università di Salerno, Fisciano, Italyfrancesco.carotenuto@gmail.com, vitsca@unisa.it2 Dipartimento di Matematica, Informatica ed Economia,Università della Basilicata, Potenza, Italyugo.erra@unibas.itAbstract. Recent studies conducted by some insurance companies high-lighted that the most part of drivers do not know the meaning of thedashboard lights. This leads drivers to be dangerous for others and them-selves. Hence, the need to provide drivers with tools that support them toalways be aware of the state of their cars. This paper proposes a systemfor mobile devices that uses augmented reality to be able to give infor-mation on a particular dashboard lights. The system is implemented bycombining the use of Simultaneous Localization and Mapping (SLAM)maps with central moment computation widely used in computer vision.Preliminary research results show that the proposed system achieves itsgoal enables a a real-time visual feedback.1 IntroductionRecent studies carried out of some insurance agencies [6] [8] have highlighted thatthe most part of drivers do not know the car components, and, in particular, themeaning of the dashboard lights. This lack of knowledge of drivers is dangerousbecause unknown car malfunctioning can lead to incidents and, as a consequence,to be injured, or, in the worst cases, to dead. For this reason, car manufacturersprovide drivers with a printed maintenance guide. However, it is tedious andtime consuming to leaf through a long printed car maintenance guide and, as aconsequence, several drivers give up keeping cars in an unsafe state. To avoid thisdangerous behavior, some applications mobile augmented reality[4] have beenimplemented to replace printed guides and to allow drivers to obtain informationin a simpler way. Usually in these applications the user points with the cameraof the mobile device a component such as a car wheel and some informationabout it are shown, such as instructions to change a wheel.However, a feature which lacks to these applications is the tracking of thelights on the dashboard. In order to take full advantage of this feature, it shouldbe always performed even when the car stops in an area not served by any wire-less connection. Moreover, systems providing this functionality should have twoimportant requirements: working in any type of lighting condition and recog-nizing the lights even if partially occluded by steering wheel as illustrated inFigure 1.c© Springer International Publishing Switzerland 2014L.T. De Paolis and A. Mongelli (Eds.): AVR 2014, LNCS 8853, pp. 413–420, 2014.DOI: 10.1007/978-3-319-13969-2 31414 F. Carotenuto et al.(a) Nothing (b) Partial (c) FullFig. 1. Examples of possible occlusions of the dashboard caused by the position of thesteering wheelThe aim of this paper is to enrich the state of the art with a system basedon augmented reality capable of giving information about dashboard lights in asimple and fast way and, at the same, satisfying the above mentioned require-ments.The idea is to associate with each light a relative position with respect to amarker (such as a portion of the speedometer). On the basis of these positionsare derived from the frame of the segments that triggered the tracking. Withinthese segments, of fixed size, there should be a particular indicator. To detectthe presence of this indicator, we proceed to search for a centroid located in aposition close to the center of the segment.In particular, the proposed system combines the use of Simultaneous Local-ization and Mapping (SLAM) maps, implemented through Metaio framework,with central moment computation obtained through OpenCV library.The paper is structured as follows. Section 2 presents the background knowl-edge including description of SLAM framework implemented by Metaio andstudy of the moments and, in particular, of the central moment, performed byOpenCV. Section 3 presents the general architecture of the system and the stepsthat implement the lights tracking. Before concluding in Section 5, system testsare presented in Section 4.2 Background InformationThis section presents the two principal methodologies on which proposed systemis based: SLAM and moment computation. SLAM [7] allows to realize a mappingof a 3D environment and can support a fast and stable tracking [3]. The producedmaps are typically used by robots to be able to move in an environment or simplyto give them an idea of their position within the environment. There are multipleways to be able to trace the maps SLAM, some of these methods make use ofcertain devices such as GPS [1], laser and sonar[2]. These last two ones areusually installed on self-propelled devices.However, it is also possible to construct a SLAM map by using a mobile deviceequipped with a camera [11]. In this way, it is possible to recover a particularSLAM Map Application for Tracking Lights on Car Dashboards 415scene, identify a set of points that are placed on a 3D coordinate system. In thiswork, we exploits Metaio SDK. It provides a set of APIs for the developmentof augmented reality applications. In particular, it provides functionalities fordisplaying multimedia content, but especially features that allow to make thetracking using maps SLAM. For the creation of the maps, SLAM Metaio SDK[5] uses an application called Toolbox available for both Android and iOS.The second principal methodology is the moment computation. The momentis a particular average intensity of pixels in an image. It is defined as:Mpq =∫ +∞−∞∫ +∞−∞xpyqf(x, y)dxdy (1)In particular, this work considers the study of the central moments, as they forma particular feature of an object represented in an image independently of itslocation within the image itself. In the case of a binary image the coordinates ofthe central moment is obtained with:C = (M10M00,M01M00) (2)where M00, M10, M01 represent, respectively, the area of the object and thecenter of mass of the object on x and y axis. In this work, we OpenCV [9] asa library that provides functionalities to support applications that make useof Computer Vision. In particular, OpenCV is used to calculate the momentswithin the image that are the basis for the calculation of the central moment.3 System OverviewThis section presents the proposed system aimed at giving drivers informationabout dashboard lights. In order archive its aim, the system uses the slam maps,together with some pre-processed information. In particular, the system is com-posed of two modules: the first devoted to the acquisition of such information,called Training, and the second one devoted to the tracking, named Detection.The training module works to acquire the SLAM maps that make up the dash-board and some information that uniquely characterize the lights inside thedashboard. Instead, the detection module uses the SLAM maps and informationproduced by the training component in order to track dashboard lights.The two components are used by different actors: training is used by carmaker whereas detection is used by the driver. Hereafter, more details aboutsystem components are given.3.1 TrainingThe training module of the system is illustrated in in Figure 2. From the dia-gram it is possible to note that before the training, there is a pre-training which416 F. Carotenuto et al.Tracking a MapInsertion of the position of lightLightsPosition DBStudingmorfology of dashboardCreatingmaps of dashboardLoadingMapsPre-Training Phase Training Maps the dashboardFig. 2. Steps of training moduleconsists of two steps. The first step is devoted to the study of the morphology andarrangement of the lights on the dashboard. This step is useful to understandhow many maps are needed to keep track of all lights. Using the information,we proceed to the creation of the SLAM maps using the application provided byToolbox as shown in Figure 3.Fig. 3. SLAM map of a dashboard regionSubsequently, the generated maps are loaded into the training module asinput. Using these maps, the user performs the tracking by pointing the camera ofthe device in the dashboard areas of interest. During the tracking an augmentedobject (AO) appears on the device screen. This augmented object can serve as a3D placeholder to fix the position of the lights during the tracking as illustratedin Figure 4. Once the augmented object is placed on the light the user store itsposition in a database. Associated to the position of the light position furtherinformation can be inserted such as light name and more important a descriptionof the meaning of the light. This process continues until all the 3D positions ofall the dashboard lights are stored.SLAM Map Application for Tracking Lights on Car Dashboards 417Fig. 4. Inserting of the positions of the light using training module3.2 DetectionAs illustrated in Figure 5, the detection module needs certain information suchas those relating to the SLAM maps and the database with the positions of lightscreated in the previous phase.Tracking a MapSearch lights LightsPosition DBLoading Maps And DataDetectionMaps the dashboardActive AOAO DBPositions of the lights on the mapArray of lights onFig. 5. Steps detection moduleIn addition to this infor-mation, further data arerequired as the database ofaugmented objects(AO DB)that should be displayed inthe case of tracking of oneor more lights on. The AOassociated with a single lightis a tooltip which shows anicon and a brief descriptionFigure 6.The first step performedby detection module is to loadall the positions of the lightsin the form of fictitious AOs. These objects are never displayed during the track-ing of the portion of the dashboard where the user is pointing the camera forthe light, but they only serve to obtain the position of the light in the scene. Indetail, the detection module performs the tracking of a portion of the dashboardin order to obtain all the stored 3D position with the associated SLAM map.These positions, are mapped from 3D points in the world to 2D points using thecamera device coordinates system.Since the resolution and the aspect ratio of the display differs from those ofthe camera it is necessary to make the rescaling of the coordinate obtained in theprevious step. After calculating all the coordinates of the lights in the area, weproceed with the capturing of the frame of the camera that triggered the tracking.Once obtained coordinates and the frame, it is possible to search the lights onin accordance with the steps described in Figure 7. The first step is to obtain418 F. Carotenuto et al.Fig. 6. AO visible on light onsegments of the image of a fixed size n×n whose the point at bottom-left is oneof the obtained coordinates. Each one of these segments should contain a light.Fig. 7. Phases of searchFor each of these segments, we calcu-late the center of mass of the objectspresent in the segment. Ideally, thecenter of mass of the light should be inthe coordinated C = (n/2, n/2). Sincethe front shot can translate some cen-timeters, then it is necessary to definea range within which the system canobtain that there is a light on. Thisrange is defined in an interval n/2−δ ≤ Cx ≤ n/2 + δ and n/2 − δ ≤Cy ≤ n/2 + δ.The values of δ and n must bedefined in order to avoid false posi-tives and false negatives. A false pos-itive occurs when the system detects the presence of a ligth on even if it is off,conversely as regards the false negative.High values of δ and n can lead to increase the number of false positives,whereas, small values increase the number of false negatives. In this work, thevalues of δ and n are defined empirically, taking as reference the resolution ofthe camera 320 × 240.Whenever it finds a light on in the dashboard is inserted a new element inan array given in input to the last step of the Detection component. At the end,this last step views AO associated with the name of the light in the array.4 Experiments and ResultsThis section presents preliminary experiments performed to test the proposedsystem. The carried out experiments take into account different occlusions andlight conditions.SLAM Map Application for Tracking Lights on Car Dashboards 419The lighting condition is considered to be primarily related to the maximumillumination of the instrument panel lighting, obtained using the mobile devicesflash while shooting the scene. To better evaluate the robustness of the approachmade and taking account of the absence of the flash from some mobile devices,such as tablet it was thought to perform tests also considering the condition ofnatural light. For natural light means the only light coming from the windowsand windshield of the vehicle.In particular, we consider four combinations of occlusions and lightingconditions:– Scenario 1 (S1): absence of occlusion with full light.– Scenario 2 (S2): absence of occlusion with natural light.– Scenario 3 (S3): presence of occlusion with full light.– Scenario 4 (S4): presence of occlusion with natural lightMoreover, since the system should be able to detect one or more indicatorssite in the same are, we consider two cases: situations with one light and duallights adjacent. By summarizing, combining the defined modes with the afore-mentioned combinations of lighting and occlusion, we perform 8 tests. Eachperformed test is repeated five times and with different mobile devices.The hardware configuration was based on a Samsung Galaxy Tab 2 GT-P7100, dual core 1GHz Tegra 2, 1GB of RAM, camera 3Mpixel with flash anda Motorola Moto G, quad core 1.2Ghz Snapdragon 200, 1GB of RAM, camera5Mpixel with flash.The evaluation of the proposed system is carried out by using the followingscale:– yes: tracking is performed within 30 seconds of the first shot dashboard.– partial : tracking is performed within 45 seconds.– none: tracking fails.Table 1. Summary table of the tests performedConfiguration S1 S2 S3 S4Single Light yes yes yes yesMultiple Light yes partial yes partialThe Table 1 shows the result of the performed experiments. As shown, theproposed system yields good performance obtaining the highest score in the 75%of the performed tests. We point out that these tests are preliminary and we aregoing to perform further experiments considering several users and a full dayusage.5 Conclusions and Future WorksThis work is aimed at creating a system for mobile devices that helps the driverto better understand the instrumentation and controls as an easily and more420 F. Carotenuto et al.appealing alternative to the car guide. As shown by the preliminary experiments,the proposed system yields good performance.However, the proposed system tends to generate false positives due to thepresence of unexpected occlusion objects within the camera view frustum. Asfor instance, an object with a color similar to light on can be detected as anobjects with a contour and consequently possess a center of mass.In the future, to reduce false positives we think to add a pre-processing stepon the frame before breaking it up into segments. For example, this step couldconsist in the application of techniques for color segmentation [10] useful toremove all objects that do not match the color of the lights. In this way, wethink to minimize the number of calculated centroids, and as consequence toincrease the accuracy of the proposed system.References1. Carlson, J.D.: Mapping Large, Urban Environments with GPS-Aided SLAM. PhDthesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (August2010)2. Choi, J.W., Ahn, S., Chung, W.K.: Robust sonar feature detection for the slam ofmobile robot. In: 2005 IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS 2005), pp. 3415–3420 (August 2005)3. Davison, A.J., Mayol, W.W., Murray, D.W.: Real-time localization and mappingwith wearable active vision. In: Proceedings of the Second IEEE and ACM Inter-national Symposium on Mixed and Augmented Reality, pp. 18–27 (October 2003)4. Fiat. Ciao fiat. http://www.fiat.it/ciao-fiat5. Metaio GmB. Metaio sdk. http://www.metaio.com6. Horton, P.: Dashboard warning lights confuse drivers (July 2013). http://www.lv.com/about-us/press/article/dashboard-warning-lights-confuse-drivers7. Leonard, J., Durrant-Whyte, H., Cox, I.J.: Dynamic map building for autonomousmobile robot. In: Proceedings of the IEEE International Workshop on IntelligentRobots and Systems, IROS 1990, Towards a New Frontier of Applications, vol.1,pp. 89–96 (July 1990)8. Megna, M.: Warning! drivers’ knowledge of dashboard lights is runningon empty (November 2013). http://www.insurance.com/auto-insurance/safety/what-does-dashboard-light-mean.html9. OpenCV. Opencv. http://www.opencv.org10. Skarbek, W., Koschan, A.: Colour image segmentation - a survey (1994)11. Ventura, J., Arth, C., Reitmayr, G., Schmalstieg, D.: Global localization frommonocular slam on a mobile phone. In: IEEE Virtual Reality (March 2014)

SLAM map application for tracking lights on car dashboards

https://iris.unibas.it/bitstream/11563/98095/2/avr2014.pdf

SLAM map application for tracking lights on car dashboards

Abstract

Similar works

Full text

Available Versions

Archivio della Ricerca - Università di Salerno

Crossref

Archivio della Ricerca - Università della Basilicata