Measuring progress in robotics: Benchmarking and the ‘measure-target confusion’

While it is often said that robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion between a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment

An atomic clock with 10−1810^{-18} instability

Atomic clocks have been transformational in science and technology, leading to innovations such as global positioning, advanced communications, and tests of fundamental constant variation. Next-generation optical atomic clocks can extend the capability of these timekeepers, where researchers have long aspired toward measurement precision at 1 part in $\bm{10^{18}}$. This milestone will enable a second revolution of new timing applications such as relativistic geodesy, enhanced Earth- and space-based navigation and telescopy, and new tests on physics beyond the Standard Model. Here, we describe the development and operation of two optical lattice clocks, both utilizing spin-polarized, ultracold atomic ytterbium. A measurement comparing these systems demonstrates an unprecedented atomic clock instability of $\bm{1.6\times 10^{-18}}$ after only $\bm{7}$ hours of averaging

A Time Like No Other: Charting the Course of the Next Revolution - A Summary of the Boston Indicators Report 2004-2006

Summarizes findings from the Boston Indicators Project, a long-term research study of the city's economic, social, and technical progress across ten sectors

The Silicon Meta-shell X-ray Mirror Technology Development Roadmap for the Lynx Mission

This document presents a roadmap for advancing the silicon meta-shell optics (SMO). It describes an overall strategy and key technical elements to be developed to meet the four-fold Lynx requirements: (1) angular resolution, (2) effective area, (3) mass, and (4) production schedule and cost. It also describes the building and testing of an engineering unit whose successful completion will retire all risks, technical, logistical, schedule, and cost, associated with building and delivering a mirror assembly for the Lynx mission. All of this work, designed to advance this technology to TRL 6, will be completed by Preliminary Design Review (PDR) to ensure that the flight mirror assembly production process will be but a repetition of a set of well-defined and mature steps, leading to on-time and on-budget delivery of a mirror assembly for the Lynx mission

Near-Field Scanning Microwave Microscopy in the Single Photon Regime

The microwave properties of nano-scale structures are important in a wide variety of applications in quantum technology. Here we describe a low-power cryogenic near-field scanning microwave microscope (NSMM) which maintains nano-scale dielectric contrast down to the single microwave photon regime, up to $10^{9}$ times lower power than in typical NSMMs. We discuss the remaining challenges towards developing nano-scale NSMM for quantum coherent interaction with two-level systems as an enabling tool for the development of quantum technologies in the microwave regime