Bodin, Bruno

Kelly, Paul H. J.

Nardi, Luigi

O'Boyle, Michael

Wagstaff, Harry

Bruno Bodin

Luigi Nardi

Harry Wagstaff

Paul H J Kelly

Michael O'Boyle

Crossref

Algorithmic Performance-Accuracy Trade-off in 3D Vision Applications

Simultaneous Localisation And Mapping (SLAM) is a key component of robotics and augmented reality (AR) systems. While a large number of SLAM algorithms have been presented, there has been little effort to unify the interface of such algorithms, or to perform a holistic comparison of their capabilities. This is particularly true when it comes to evaluate the potential trade-offs between computation speed, accuracy, and power consumption. SLAMBench is a benchmarking framework to evaluate existing and future SLAM systems, both open and closed source, over an extensible list of datasets, while using a comparable and clearly specified list of performance metrics. SLAMBench is a publicly-available software framework which represents a starting point for quantitative, comparable and validatable experimental research to investigate trade-offs in performance, accuracy and energy consumption across SLAM systems. In this poster we give an overview of SLAMBench and in particular we show how this framework can be used within Design Space Exploration and large-scale performance evaluation on mobile phones

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Algorithmic Performance-Accuracy Trade-off in 3D VisionApplicationsCitation for published version:Bodin, B, Nardi, L, Wagstaff, H, Kelly, PHJ & O'Boyle, M 2018, Algorithmic Performance-Accuracy Trade-offin 3D Vision Applications. in 2018 IEEE International Symposium on Performance Analysis of Systems andSoftware. IEEE, Belfast, UK, pp. 123-124, 2018 IEEE International Symposium on Performance Analysis ofSystems and Software, Belfast, United Kingdom, 2/04/18. DOI: 10.1109/ISPASS.2018.00024Digital Object Identifier (DOI):10.1109/ISPASS.2018.00024Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:2018 IEEE International Symposium on Performance Analysis of Systems and SoftwareGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 20191Algorithmic Performance-Accuracy Trade-off in 3D Vision ApplicationsBruno Bodin†, Luigi Nardi¶, Harry Wagstaff†, Paul H. J. Kelly‡ & Michael O’Boyle†Abstract— Simultaneous Localisation And Mapping (SLAM)is a key component of robotics and augmented reality (AR)systems. While a large number of SLAM algorithms have beenpresented, there has been little effort to unify the interfaceof such algorithms, or to perform a holistic comparison oftheir capabilities. This is particularly true when it comes toevaluate the potential trade-offs between computation speed,accuracy, and power consumption. SLAMBench is a bench-marking framework to evaluate existing and future SLAMsystems, both open and closed source, over an extensible listof datasets, while using a comparable and clearly specified listof performance metrics. SLAMBench is a publicly-availablesoftware framework which represents a starting point forquantitative, comparable and validatable experimental researchto investigate trade-offs in performance, accuracy and energyconsumption across SLAM systems. In this poster we give anoverview of SLAMBench and in particular we show how thisframework can be used within Design Space Exploration andlarge-scale performance evaluation on mobile phones.SLAMBENCH: PERFORMANCE AND ACCURACYBENCHMARKING METHODOLOGY FOR SLAMSimultaneous Localisation And Mapping (SLAM) is akey component in robotics that constructs a map of anunknown environment while simultaneously keeping track ofthe robot’s location within it. SLAMBench [1] is an open-source benchmark based on the SLAM system KinectFu-sion [2] that produces dense 3D model of an arbitrary sceneusing an RGB-D camera. SLAMBench provides implemen-tations of KinectFusion using popular languages, such asCUDA, OpenCL, OpenMP and C++. Figure 1 shows thegraphical interface of SLAMBench. The two frames on thetop left of the interface are the RGB and Depth frame of theKinect sensor. The bottom left is the tracking status of thealgorithm and the right frame is the current map generatedby the SLAM system. SLAMBench supports research inhardware accelerators and software tools by enablingthe comparison across algorithms, implementations, anddatasets, of performance, energy-consumption, and ac-curacy of the generated 3D model in the context of aknown ground-truth.HYPERMAPPER: CO-DESIGN EXPLORATION OFSLAMBENCH USING MACHINE LEARNINGWe examine how SLAMBench can be mapped to powerconstrained embedded systems [3], [4]. Key to our approachis the idea of incremental co-design exploration, whereoptimization choices that concern the domain layer areincrementally explored together with low-level compiler andarchitecture choices. The goal of this exploration is to reduce† School of Informatics, University of Edinburgh, UK‡ Department of Computing, Imperial College London, UK¶ Stanford University, USAFig. 1. The SLAMBench Graphical User Interface provides real-time statusof the different performance metrics such as speed, power, and accuracy.execution time while minimizing power and meeting ourquality of result objective. As the design space is too largeto exhaustively evaluate, we use active learning based ona random forest predictor to find good designs. Figure 2shows an overview of this learning process. We show thatour approach can, for the first time, achieve dense 3Dmapping and tracking in the real-time range within a1W power budget on the Odroid XU3 embedded device.This is a 4.8x execution time improvement and a 2.8xpower reduction compared to the state-of-the-art.RELATED WORKComputer vision research has traditionally focused on op-timising the accuracy of algorithms. In autonomous driving,for example, the KITTI benchmark suite [5] provides dataand evaluation criteria for the stereo, optical flow, visualodometry and 3D object recognition. The ICL-NUIM dataset[6] and TUM RGB-D benchmark [7] aim to benchmark theaccuracy of visual odometry and SLAM algorithms.An important early benchmark suite for performance eval-uation entirely dedicated to computer vision is SD-VBS [8].SD-VBS provides single-threaded C and MATLAB imple-mentations of 28 commonly used computer vision kernelsthat are combined to build 9 high-level vision applica-tions. Another contribution at such performance evaluation isMEVBench [9], which focuses on a set of visual recognitionapplications including face detection, feature classification,object tracking and feature extraction. It provides single andmultithreaded C++ implementations for some of the kernelswith a special emphasis on low-power embedded systems.While such efforts are a step in the right direction, they do2AlgorithmicConﬁgurationParameters[    ] AccuracyRuntimePower[   ]Random samplesMACHINELEARNINGPREDICTIVEMODELRun new samplesActive LearningLearning0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45Runtim e  (se c)0.0350.0400.0450.0500.055Max ATE (m)Accuracy lim it = 0.05mDe fault configurationActive  le arningRandom  sam plingBest conﬁgurationsVolume resolution < 96Compute size ratio < 3 Compute size ratio > 6Volume resolution >192Mu distance< 0.05 Compute size ratio < 3Accurate (Max ATE < 5 cm)Fast (Speed > 30 FPS)Power eﬃcient (consumption < 3W)++OR++OR ORYES NOKnowledge+Fig. 2. Design space exploration methodology for algorithmic parameters. The first step performs random sampling of the space. Then a predictive modelis built to guide the exploration. Finally this model can be used to understand the impact of parameters on the different performance metrics.0 2 4 6 8 10 12 14Speed-upFig. 3. The OpenCL KinectFusion has been run on 83 smart-phones and tablets from the market using an Android application. One the left can be seena screenshot of the Android application running. On the right is the speed-up result. For each device, we computed the speedup of the configuration wefound for the ODROID-XU3 with HyperMapper.not provide the software tools for accuracy verification andexploitation of hardware accelerators or graphics processorunits (GPUs). Nor do they enable investigation of energyconsumption, performance and accuracy envelopes for 3Dscene reconstruction algorithms across a range of hardwaretargets.A key feature of SLAMBench is that it is designed ontop of the recently-proposed ICL-NUIM accuracy benchmark[6], and thus supports wider research in hardware and soft-ware. The quantitative evaluation of solution accuracy intoSLAMBench enables algorithmic research to be performed.AN EVALUATION OF SLAMBENCH AND HYPERMAPPERACROSS A WIDE RANGE OF MOBILE PHONESThe SLAMBench framework and more specifically itsvarious KinectFusion [2] implementations has been ported toAndroid [10]. More than 1000 downloads have been madesince its official release on the Google Play store. Thissuccess allowed us to crowdsource data from more than 100different mobile phones. Figure 3 summarizes the perfor-mance results from the collected data. We now plan to usethis data to analyse the performance of KinectFusion on thoseplatforms, and to provide techniques to optimise KinectFu-sion performance depending of the targeted platform. Webelieve that by combining the potential of HyperMapper andthe data collected on Android, we could train a decisionmachine for mobile phones.REFERENCES[1] L. Nardi, B. Bodin, M. Z. Zia, J. Mawer, A. Nisbet, P. H. J. Kelly, A. J.Davison, M. Luja´n, M. F. P. O’Boyle, G. Riley et al., “IntroducingSLAMBench, a performance and accuracy benchmarking methodol-ogy for SLAM,” in IEEE Intl. Conf. on Robotics and Automation(ICRA), May 2015.[2] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim,A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon,“KinectFusion: Real-time dense surface mapping and tracking,” inISMAR. IEEE, 2011.[3] B. Bodin, L. Nardi, M. Z. Zia, H. Wagstaff, G. Sreekar Shenoy,M. Emani, J. Mawer, C. Kotselidis, A. Nisbet, M. Lujan et al., “In-tegrating algorithmic parameters into benchmarking and design spaceexploration in 3d scene understanding,” in Proceedings of the 2016International Conference on Parallel Architectures and Compilation,ser. PACT ’16. New York, NY, USA: ACM, 2016, pp. 57–69.[4] L. Nardi, B. Bodin, S. Saeedi, E. Vespa, A. J. Davison, and P. H. Kelly,“Algorithmic performance-accuracy trade-off in 3d vision applicationsusing hypermapper,” in iWAPT. IEEE, 2017.[5] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomousdriving? The KITTI vision benchmark suite,” in CVPR, 2012.[6] A. Handa, T. Whelan, J. McDonald, and A. Davison, “A Benchmarkfor RGB-D Visual Odometry, 3D Reconstruction and SLAM,” inICRA, 2014.[7] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “Abenchmark for the evaluation of RGB-D SLAM systems,” in IROS,2012.[8] S. K. Venkata, I. Ahn, D. Jeon, A. Gupta, C. Louie, S. Garcia,S. Belongie, and M. B. Taylor, “SD-VBS: The San Diego visionbenchmark suite,” in IISWC, 2009.[9] J. Clemons, H. Zhu, S. Savarese, and T. Austin, “MEVBench: Amobile computer vision benchmarking suite,” in IISWC, 2011.[10] “Slambench for android,” https://play.google.com/store/apps/details?id=project.pamela.slambench.

https://www.pure.ed.ac.uk/ws/files/61165706/Algorithmic_Performance_Accuracy_Trade_off_in_3D_Vision_Applications.pdf

Algorithmic Performance-Accuracy Trade-off in 3D Vision Applications

Abstract

Similar works

Full text

Available Versions

Crossref

Edinburgh Research Explorer