Benchmarking of localization solutions : guidelines for the selection of evaluation points

Indoor localization solutions are key enablers for next-generation indoor navigation and track and tracing solutions. As a result, an increasing number of different localization algorithms have been proposed and evaluated in scientific literature. However, many of these publications do not accurately substantiate the used evaluation methods. In particular, many authors utilize a different number of evaluation points, but they do not (i) analyze if the number of used evaluation points is sufficient to accurately evaluate the performance of their solutions and (ii) report on the uncertainty of the published results. To remedy this, this paper evaluates the influence of the selection of evaluation points. Based on statistical parameters such as the standard error of the mean value, an estimator is defined that can be used to quantitatively analyze the impact of the number of used evaluation points on the confidence interval of the mean value of the obtained results. This estimator is used to estimate the uncertainty of the presented accuracy results, and can be used to identify if more evaluations are required. To validate the proposed estimator, two different localization algorithms are evaluated in different testbeds and using different types of technology, showing that the number of required evaluation points does indeed vary significantly depending on the evaluated solution. (C) 2017 Elsevier B.V. All rights reserved

LocaRDS: A Localization Reference Data Set

The use of wireless signals for the purposes of localization enables a host of applications relating to the determination and verification of the positions of network participants ranging from radar to satellite navigation. Consequently, this has been a longstanding interest of theoretical and practical research in mobile networks and many solutions have been proposed in the scientific literature. However, it is hard to assess the performance of these in the real world and, more importantly, to compare their advantages and disadvantages in a controlled scientific manner. With this work, we attempt to improve the current state of art methodology in localization research and to place it on a solid scientific grounding for future investigations. Concretely, we developed LocaRDS, an open reference data set of real-world crowdsourced flight data featuring more than 222 million measurements from over 50 million transmissions recorded by 323 sensors. We demonstrate how we can verify the quality of LocaRDS measurements so that it can be used to test, analyze and directly compare different localization methods. Finally, we provide an example implementation for the aircraft localization problem and a discussion of possible metrics for use with LocaRDS