EvAAL: Evaluating AAL Systems through Competitive Benchmarking

Owing to the complexity of Ambient Assisted Living (AAL) systems and platforms, the evaluation of AAL solutions is a complex task that will challenge researchers for years to come. However, the analysis and comparison of proposed solutions is paramount to enable us to assess research results in this area. We have thus organized an international contest called EvAAL: Evaluating AAL Systems through Competitive Benchmarking. Its aims are to raise interest within the research and developer communities in the multidisciplinary research fields enabling AAL, and to create benchmarks for the evaluation and comparison of AAL systems

Competitive Live Evaluation of Activity-recognition Systems

In order to ensure the validity and usability of activity recognition approaches, an agreement on a set of standard evaluation methods is needed. Due to the diversity of the sensors and other hardware employed, designing and accepting standard tests is a difficult task. This article presents an initiative to evaluate activity recognition systems: a living-lab evaluation established through an annual competition − EvAAL-AR (Evaluating Ambient Assisted Living Systems through Competitive Benchmarking − Activity Recognition). In the competition, each team brings their own activity-recognition system, which is evaluated live on the same activity scenario performed by an actor. The evaluation criteria attempt to capture the practical usability: recognition accuracy, user acceptance, recognition delay, installation complexity, and interoperability with ambient assisted living systems. The article also presents the competing systems with emphasis on two best-performing ones: (i) a system that achieved the best recognition accuracy, and (ii) a system that was evaluated as the best overall. Finally, the article presents lessons learned from the competition and ideas for future development of the competition and of the activity recognition field in general

Analysis of the Behavior of an Indoor Position System Based on Fingerprints and IEEE 802.15.4

This paper presents an analysis of the behaviour of an indoor position system based on fingerprints and IEEE 802.15.4 that has been adapted to be tested in a competition, called EvAAL (Evaluating AAL Systems through Competitive Benchmarking), hold both in Madrid (tests) and Eindhoven (results). The objectives of this analysis are to determine the best algorithm that should have been applied in order to obtain the best results in the competition to use them in other environments. Among the different combinations that can be applied, i.e., the way the signature database is filled in and the algorithm uses to determine the closest location point, the best results are obtained using a global signature database where each signature entry is calculated by the medium of samples signatures database and the closest location is determined by a centroid algorithm with the parameter c set to 1.3. In this way, the error made improves the one obtained in the EvAAL, which is reduced by 100 centimetres.Junta de Andalucía p08-TIC-363

A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL–ETRI competition

Pre-print versionThis paper presents results from comparing different Wi-Fi fingerprinting algorithms on the same private dataset. The algorithms where realized by independent teams in the frame of the off-site track of the EvAAL-ETRI Indoor Localization Competition which was part of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015). Competitors designed and validated their algorithms against the publicly available UJIIndoorLoc database which contains a huge reference- and validation data set. All competing systems were evaluated using the mean error in positioning, with penalties, using a private test dataset. The authors believe that this is the first work in which Wi-Fi fingerprinting algorithm results delivered by several independent and competing teams are fairly compared under the same evaluation conditions. The analysis also comprises a combined approach: Results indicate that the competing systems where complementary, since an ensemble that combines three competing methods reported the overall best results.We would like to thank Francesco Potortì, Paolo Barsocchi, Michele Girolami and Kyle O’Keefe for their valuable help in organizing and spread the EVAALETRI competition and the off-site track. We would also like to thank the TPC members Machaj Juraj, Christos Laoudias, Antoni Pérez-Navarro and Robert Piché for their valuable comments, suggestions and reviews. Parts of this work were funded in the frame of the Spanish Ministry of Economy and Competitiveness through the “Metodologiías avanzadas para el diseño, desarrollo, evaluación e integración de algoritmos de localización en interiores” project (Proyectos I+D Excelencia, código TIN2015-70202-P) and the “Red de Posicionamiento y Navegación en Interiores” network (Redes de Excelencia, código TEC2015-71426- REDT). Parts of this work were funded in the frame of the German federal Ministry of Education and Research programme "FHprofUnt2013" under contract 03FH035PB3 (Project SPIRIT).info:eu-repo/semantics/acceptedVersio

Evaluating the use of robots to enlarge AAL services

We introduce robots as a tools to enhance Ambient Assisted Living (AAL) services. Robots are a unique opportunity to create new systems to cooperate in reaching better living conditions. Robots offer the possibility of richer interaction with humans, and can perform actions to actively change the environment. The current state-of-art includes skills in various areas, including advanced interaction (natural language, visual attention, object recognition, intention learning), navigation (map learning, obstacle avoidance), manipulation (grasping, use of tools), and cognitive architectures to handle highly unpredictable environments. From our experience in several robotics projects and principally in the RoboCup@Home competition, a new set of evaluation methods is proposed to assess the maturity of the required skills. Such comparison should ideally enable the abstraction from the particular robotic platform and concentrate on the easy comparison of skills. The validity of that low-level skills can be then scaled to more complex tasks, that are composed by several skills. Our conclusion is that effective evaluation methods can be designed with the objective of enabling robots to enlarge AAL services.This research was partly supported by the PATRICIA project (TIN2012-38416-C03-01), MANIPlus project (201350E102), Spanish Ministry of Economy and Competitiveness, and European Found for Regional Development (FEDER).Peer Reviewe

Human activity recognition using multisensor data fusion based on Reservoir Computing

Activity recognition plays a key role in providing activity assistance and care for users in smart homes. In this work, we present an activity recognition system that classifies in the near real-time a set of common daily activities exploiting both the data sampled by sensors embedded in a smartphone carried out by the user and the reciprocal Received Signal Strength (RSS) values coming from worn wireless sensor devices and from sensors deployed in the environment. In order to achieve an effective and responsive classification, a decision tree based on multisensor data-stream is applied fusing data coming from embedded sensors on the smartphone and environmental sensors before processing the RSS stream. To this end, we model the RSS stream, obtained from a Wireless Sensor Network (WSN), using Recurrent Neural Networks (RNNs) implemented as efficient Echo State Networks (ESNs), within the Reservoir Computing (RC) paradigm. We targeted the system for the EvAAL scenario, an international competition that aims at establishing benchmarks and evaluation metrics for comparing Ambient Assisted Living (AAL) solutions. In this paper, the performance of the proposed activity recognition system is assessed on a purposely collected real-world dataset, taking also into account a competitive neural network approach for performance comparison. Our results show that, with an appropriate configuration of the information fusion chain, the proposed system reaches a very good accuracy with a low deployment cost

Requirements and metrics for location and tracking for ambient assisted living

Location and tracking services and technologies are becoming fundamental components for supporting healthcare solutions. They facilitate patients’ tracking and monitoring processes and also allow for better and long-term daily activity recognition. Various location and tracking services have been developed, over the last years, to provide real time localization for different applications. However, most of these services are not designed particularly to comply with all the requirements of Ambient Assisted Living (AAL) and, as a result, they reduce the viability of adopting AAL services as an alternative for continuous healthcare services. In this paper we set out the general requirements for location and tracking services for AAL. The requirements are extracted from a typical scenario of AAL. From the scenario, we define the requirements and also we identify a set of metrics to be used as evaluation criteria. If the identified requirements and metrics are adopted widely, potential location and tracking services will fit the real needs of AAL, and thus will increase the accessibility to AAL services by a larger sector of people. Moreover, in the paper, we evaluate two of the existing location techniques through the use of the proposed metrics. The aim is to asses to which level these solutions fulfill the identified requirements.This work was supported by the FEDER program through the COMPETE and the Portuguese Science and Technology Foundation (FCT), within the context of the AAL4ALL (COMPETE 13852) and FCOMP-01-FEDER-0124-022674 projects

The smartphone-based offline indoor location competition at IPIN 2016: analysis and future work

This paper presents the analysis and discussion of the off-site localization competition track, which took place during the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016). Five international teams proposed different strategies for smartphone-based indoor positioning using the same reference data. The competitors were provided with several smartphone-collected signal datasets, some of which were used for training (known trajectories), and others for evaluating (unknown trajectories). The competition permits a coherent evaluation method of the competitors' estimations, where inside information to fine-tune their systems is not offered, and thus provides, in our opinion, a good starting point to introduce a fair comparison between the smartphone-based systems found in the literature. The methodology, experience, feedback from competitors and future working lines are described.We would like to thank Tecnalia Research & Innovation Foundation for sponsoring the competition track with an award for the winning team. We are also grateful to Francesco Potortì, Sangjoon Park, Jesús Ureña and Kyle O’Keefe for their invaluable help in promoting the IPIN competition and conference. Parts of this work was carried out with the financial support received from projects and grants: LORIS (TIN2012-38080-C04-04), TARSIUS (TIN2015-71564-C4-2-R (MINECO/FEDER)), SmartLoc (CSIC-PIE Ref.201450E011), “Metodologías avanzadas para el diseño, desarrollo, evaluación e integración de algoritmos de localización en interiores” (TIN2015-70202-P), REPNIN network (TEC2015-71426-REDT) and the José Castillejo mobility grant (CAS16/00072). The HFTS team has been supported in the frame of the German Federal Ministry of Education and Research programme “FHprofUnt2013” under contract 03FH035PB3 (Project SPIRIT). The UMinho team has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT — Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2013.info:eu-repo/semantics/publishedVersio