A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL–ETRI competition

Pre-print versionThis paper presents results from comparing different Wi-Fi fingerprinting algorithms on the same private dataset. The algorithms where realized by independent teams in the frame of the off-site track of the EvAAL-ETRI Indoor Localization Competition which was part of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015). Competitors designed and validated their algorithms against the publicly available UJIIndoorLoc database which contains a huge reference- and validation data set. All competing systems were evaluated using the mean error in positioning, with penalties, using a private test dataset. The authors believe that this is the first work in which Wi-Fi fingerprinting algorithm results delivered by several independent and competing teams are fairly compared under the same evaluation conditions. The analysis also comprises a combined approach: Results indicate that the competing systems where complementary, since an ensemble that combines three competing methods reported the overall best results.We would like to thank Francesco Potortì, Paolo Barsocchi, Michele Girolami and Kyle O’Keefe for their valuable help in organizing and spread the EVAALETRI competition and the off-site track. We would also like to thank the TPC members Machaj Juraj, Christos Laoudias, Antoni Pérez-Navarro and Robert Piché for their valuable comments, suggestions and reviews. Parts of this work were funded in the frame of the Spanish Ministry of Economy and Competitiveness through the “Metodologiías avanzadas para el diseño, desarrollo, evaluación e integración de algoritmos de localización en interiores” project (Proyectos I+D Excelencia, código TIN2015-70202-P) and the “Red de Posicionamiento y Navegación en Interiores” network (Redes de Excelencia, código TEC2015-71426- REDT). Parts of this work were funded in the frame of the German federal Ministry of Education and Research programme "FHprofUnt2013" under contract 03FH035PB3 (Project SPIRIT).info:eu-repo/semantics/acceptedVersio

Evaluating indoor positioning systems in a shopping mall : the lessons learned from the IPIN 2018 competition

The Indoor Positioning and Indoor Navigation (IPIN) conference holds an annual competition in which indoor localization systems from different research groups worldwide are evaluated empirically. The objective of this competition is to establish a systematic evaluation methodology with rigorous metrics both for real-time (on-site) and post-processing (off-site) situations, in a realistic environment unfamiliar to the prototype developers. For the IPIN 2018 conference, this competition was held on September 22nd, 2018, in Atlantis, a large shopping mall in Nantes (France). Four competition tracks (two on-site and two off-site) were designed. They consisted of several 1 km routes traversing several floors of the mall. Along these paths, 180 points were topographically surveyed with a 10 cm accuracy, to serve as ground truth landmarks, combining theodolite measurements, differential global navigation satellite system (GNSS) and 3D scanner systems. 34 teams effectively competed. The accuracy score corresponds to the third quartile (75th percentile) of an error metric that combines the horizontal positioning error and the floor detection. The best results for the on-site tracks showed an accuracy score of 11.70 m (Track 1) and 5.50 m (Track 2), while the best results for the off-site tracks showed an accuracy score of 0.90 m (Track 3) and 1.30 m (Track 4). These results showed that it is possible to obtain high accuracy indoor positioning solutions in large, realistic environments using wearable light-weight sensors without deploying any beacon. This paper describes the organization work of the tracks, analyzes the methodology used to quantify the results, reviews the lessons learned from the competition and discusses its future

Multiple simultaneous Wi-Fi measurements in fingerprinting indoor positioning

The accuracy of fingerprinting-based positioning methods accuracy is limited by the fluctuations in the radio signal intensity mainly due to reflections, refractions, and multipath interference, among other factors. We consider that the fluctuations (often modelled as a Gaussian process for simplification purposes) can be minimized by exploiting the richness of multiple signals collected simultaneously through independent network interfaces. This paper introduces an analysis of Wi-Fi signals' statistics using simultaneous measurements which shows that RSSI values obtained from independent devices are not highly correlated. The low correlation between Wi-Fi interfaces might be exploited to improve the positioning accuracy. The validation of the proposed fingerprinting approach in a real scenario shows that the mean and maximum error in positioning can be reduced by more than 40% when five Wi-Fi interfaces are simultaneously used for fingerprinting.This work has been supported by COMPETE: POCI-01- 0145-FEDER-007043 and FCT—Fundação para a Ciência e Tecnologia within the scope of project UID/CEC/00319/2013, by the Portugal Incentive System for Research and Technological Development in the scope of the projects in co- promotion no 002814/2015 (iFACTORY 2015-2018), and by the José Castillejo mobility grant (CAS16/00072).info:eu-repo/semantics/publishedVersio

Situation Goodness Method for Weighted Centroid-Based Wi-Fi APs Localization

Knowing the location of Wi-Fi antennas may be critical for indoor localization. However, in a real environment, their positions may be unknown since they can be managed by external entities. This paper introduces a new method for evaluating the suitability of using the weighted centroid method for the 2D localization of a Wi-Fi AP. The method is based on the idea that the weighted centroid method provides its best results when there are fingerprints taken around the AP. In order to find the probability of being in the presence of such situations, a natural neighbor interpolation method is used to find the regions with the highest signal strengths. A geometrical method is then used to characterize that probability based on the distribution of those regions in relation to the AP position estimation given by the weighted centroid method. The paper describes the testing location and the used Wi-Fi fingerprints database. That database is used to create new databases that recreate different sampling possibilities through a samples deletion strategy. The original database and the newly created ones are then used to evaluate the localization results of several AP localization methods and the new method proposed in this paper. The evaluation results have shown that the proposed method is able to provide a proper probability for the suitability of using the weighted centroid method for localizing a Wi-Fi AP

Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned From the IPIN 2018 Competition

The Indoor Positioning and Indoor Navigation (IPIN) conference holds an annual competition in which indoor localization systems from different research groups worldwide are evaluated empirically. The objective of this competition is to establish a systematic evaluation methodology with rigorous metrics both for real-time (on-site) and post-processing (off-site) situations, in a realistic environment unfamiliar to the prototype developers. For the IPIN 2018 conference, this competition was held on September 22nd, 2018, in Atlantis, a large shopping mall in Nantes (France). Four competition tracks (two on-site and two off-site) were designed. They consisted of several 1 km routes traversing several floors of the mall. Along these paths, 180 points were topographically surveyed with a 10 cm accuracy, to serve as ground truth landmarks, combining theodolite measurements, differential global navigation satellite system (GNSS) and 3D scanner systems. 34 teams effectively competed. The accuracy score corresponds to the third quartile (75 th percentile) of an error metric that combines the horizontal positioning error and the floor detection. The best results for the on-site tracks showed an accuracy score of 11.70 m (Track 1) and 5.50 m (Track 2), while the best results for the off-site tracks showed an accuracy score of 0.90 m (Track 3) and 1.30 m (Track 4). These results showed that it is possible to obtain high accuracy indoor positioning solutions in large, realistic environments using wearable light-weight sensors without deploying any beacon. This paper describes the organization work of the tracks, analyzes the methodology used to quantify the results, reviews the lessons learned from the competition and discusses its future

Indoor Positioning and Navigation

In recent years, rapid development in robotics, mobile, and communication technologies has encouraged many studies in the field of localization and navigation in indoor environments. An accurate localization system that can operate in an indoor environment has considerable practical value, because it can be built into autonomous mobile systems or a personal navigation system on a smartphone for guiding people through airports, shopping malls, museums and other public institutions, etc. Such a system would be particularly useful for blind people. Modern smartphones are equipped with numerous sensors (such as inertial sensors, cameras, and barometers) and communication modules (such as WiFi, Bluetooth, NFC, LTE/5G, and UWB capabilities), which enable the implementation of various localization algorithms, namely, visual localization, inertial navigation system, and radio localization. For the mapping of indoor environments and localization of autonomous mobile sysems, LIDAR sensors are also frequently used in addition to smartphone sensors. Visual localization and inertial navigation systems are sensitive to external disturbances; therefore, sensor fusion approaches can be used for the implementation of robust localization algorithms. These have to be optimized in order to be computationally efficient, which is essential for real-time processing and low energy consumption on a smartphone or robot

Off-line evaluation of mobile-centric indoor positioning systems: the experiences from the 2017 IPIN competition

The development of indoor positioning solutions using smartphones is a growing activity with an enormous potential for everyday life and professional applications. The research activities on this topic concentrate on the development of new positioning solutions that are tested in specific environments under their own evaluation metrics. To explore the real positioning quality of smartphone-based solutions and their capabilities for seamlessly adapting to different scenarios, it is needed to find fair evaluation frameworks. The design of competitions using extensive pre-recorded datasets is a valid way to generate open data for comparing the different solutions created by research teams. In this paper, we discuss the details of the 2017 IPIN indoor localization competition, the different datasets created, the teams participating in the event, and the results they obtained. We compare these results with other competition-based approaches (Microsoft and Perf-loc) and on-line evaluation web sites. The lessons learned by organising these competitions and the benefits for the community are addressed along the paper. Our analysis paves the way for future developments on the standardization of evaluations and for creating a widely-adopted benchmark strategy for researchers and companies in the field.We would like to thank Topcon Corporation for sponsoring the competition track with an award for the winning team. We are also grateful to Francesco Potorti, Sangjoon Park, Hideo Makino, Nobuo Kawaguchi, Takeshi Kurata and Jesus Urena for their invaluable help in organizing and promoting the IPIN competition and conference. Many thanks to Raul Montoliu, Emilio Sansano, Marina Granel and Luis Alisandra for collecting the databases in the UJITI building. Parts of this work were carried out with the financial support received from projects and grants: REPNIN network (TEC2015-71426-REDT), LORIS (TIN2012-38080-C04-04), TARSIUS (TIN2015-71564-C4-2-R (MINECO/FEDER)), SmartLoc (CSIC-PIE Ref. 201450E011), "Metodologias avanzadas para el diseno, desarrollo, evaluacion e integracion de algoritmos de localizacion en interiores" (TIN2015-70202-P), GEO-C (Project ID: 642332, H2020-MSCA-ITN-2014-Marie Sklodowska-Curie Action: Innovative Training Networks), and financial support from the Ministry of Science and Technology, Taiwan (106-3114-E-007-005 and 105-2221-E-155-013-MY3). The HFTS team has been supported in the frame of the German Federal Ministry of Education and Research programme "FHprofUnt2013" under contract 03FH035PB3 (Project SPIRIT). The UMinho team has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT-Fundacao para a Ciencia e Tecnologia within the Project Scope: UID/CEC/00319/2013. G.M. Mendoza-Silva gratefully acknowledges funding from grant PREDOC/2016/55 by Universitat Jaume I.info:eu-repo/semantics/publishedVersio