Fostering Progress in Performance Evaluation and Benchmarking of Robotic and Automation Systems

We have shared benchmarks for many engineering systems and products in the market that can be used to compare solutions and systems. We can compare cars in terms of maximum speed, acceleration, and maximum torque; computers in terms of flops, random access memory, and hard disk capacity; and smartphones in terms of battery life and screen dimensions. We also have shared usability metrics based on human factors, which are used to compare the ease of use of different software interfaces. When we come to the evaluation and the comparison of how intelligent, robust, adaptive, and antifragile the behaviors of robots are in performing a given set of tasks, such as daily life activities with daily life objects such as in a kitchen or a hospital room, we are in trouble

Robotic Services for New Paradigm Smart Cities Based on Decentralized Technologies

This article describes different methods of organizing robotic services for smart cities using secure encrypted decentralized technologies and market mechanisms—as opposed to models based on centralized solutions based (or not) on using cloud services and stripping citizens of the control of their own data. The basis of the proposed methods is the Ethereum decentralized computer with the mechanism of smart contracts. In this work, special attention is paid to the integration of technical and economic information into one network of transactions, which allows creating a unified way of interaction between robots—the robot economy. Three possible scenarios of robotic services for smart cities based on the economy of robots are presented: unmanned aerial vehicles (UAVs), environmental monitoring, and smart factories. In order to demonstrate the feasibility of the proposed scenarios, three experiments are presented and discussed. Our work shows that the Ethereum network can provide, through smart contracts and their ability to activate programs to interact with the physical world, an effective and practical way to manage robot services for smart cities

Can we Reproduce it? Toward the Implementation of good Experimental Methodology in Interdisciplinary Robotics Research

Lier F, Lücking P, de Leeuw J, Wachsmuth S, Šabanović S, Goldstone R. Can we Reproduce it? Toward the Implementation of good Experimental Methodology in Interdisciplinary Robotics Research. In: Bonsignorio FP, ed. ICRA 2017 Workshop on Reproducible Research in Robotics: Current Status and Road Ahead. Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE Xplore. IEEE; 2017

Measuring progress in robotics: benchmarking and the ‘measure-target confusion’

While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment