LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture

An end-to-end review of gaze estimation and its interactive applications on handheld mobile devices

In recent years we have witnessed an increasing number of interactive systems on handheld mobile devices which utilise gaze as a single or complementary interaction modality. This trend is driven by the enhanced computational power of these devices, higher resolution and capacity of their cameras, and improved gaze estimation accuracy obtained from advanced machine learning techniques, especially in deep learning. As the literature is fast progressing, there is a pressing need to review the state of the art, delineate the boundary, and identify the key research challenges and opportunities in gaze estimation and interaction. This paper aims to serve this purpose by presenting an end-to-end holistic view in this area, from gaze capturing sensors, to gaze estimation workflows, to deep learning techniques, and to gaze interactive applications.PostprintPeer reviewe

A NEW HANDHELD SCANNER FOR 3D SURVEY OF SMALL ARTIFACTS: THE STONEX F6

Movable heritage preserved in our museums are an invaluable evidence of our past. In order to properly respond to the need of 3D documentation of these significant assets, in the last few years both range-based and image-based solutions have been developed by researchers operating in the framework of Geomatics with a special focus on reaching a high level of detail and on texture radiometric quality, taking into consideration the intrinsic fragility of these kinds of objects which during the survey require a contactless approach. During the presented research a collection of architectural models representing ancient Nubian temples from “Museo Egizio di Torino” had been digitalized using different techniques; in particular, the wooden maquette of the temple of El-Hilla has been acquired using a new structured light handheld laser scanner, the Stonex F6 SR, and applying a close-range photogrammetric approach. In this paper a comparison between the two approaches is proposed as regards acquisition workflow, final results and suitability as regards digitisation of objects belonging to movable heritage and museum collections

Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -- the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.Comment: 13 pages, 9 figure

Assessment of aerial thermography as a method of in situ measurement of radiant heat transfer in urban public spaces

Una propuesta de nuevas estrategias para la mejora del medio ambiente urbano, usando termografía aérea para el cálculo de la temperatura media radianteUrban public spaces are an essential part of the urban environment, supporting social relationships and pro- moting a healthy lifestyle among citizens. However, the high value of urban land has led to an over-urbanisation of cities, increasing urban heat stress and decreasing the number and size of public spaces. Rising air temper- atures in cities – known as the urban heat island effect (UHI) - combined with global warming, make public spaces less comfortable. For these reasons, there has been a growing concern to improve the thermal comfort of urban spaces. Thermal radiation is a determining factor in urban thermal comfort and is normally summarised in a value called mean radiant temperature (TMRT). In the past, conventional methods have been used to calculate it, such as net radiometers and globe thermometers. In recent years, the scientific community has used ground- based handheld thermal cameras for its quantification. However, there is a lack of literature on the use of aerial thermography for this purpose (i.e. an unmanned aerial vehicle (UAV) equipped with a thermal infrared device). Given this gap in the literature and the advantages in time, versatility and accuracy of these systems, this paper presents a new method for assessing the measurement of radiant heat transfer in a pedestrian urban space using aerial thermography. From the surface temperatures of the infrared imagery collected by the UAV, TMRT was estimated at multiple points in a pedestrian area of a subtropical city (Huelva, Spain) during a typical summer day. In order to verify accuracy of the proposed method to estimate the TMRT, a microclimate urban simulation was carried out using ENVI-met v5. The comparative analysis of the measured and simulated dataset verified the applicability of aerial thermography for the measurement of radiant heat transfer (with R2 values of 0.98 for the data set and 0.8 for the data of each time period). To conclude, new strategies were proposed to improve urban thermal comfort and to make cities more sustainable.Funding for open access charge: Universidad de Huelva/CBUA. Proyecto SALTES (P20_00730): Smartgrid with reconfigurable Architecture for testing controL Techniques and Energy Storage priority. Programa Operativo FEDER 2014-2020 Junta de Andalucia