Instrumentation, Data, And Algorithms For Visually Understanding Haptic Surface Properties

Autonomous robots need to efficiently walk over varied surfaces and grasp diverse objects. We hypothesize that the association between how such surfaces look and how they physically feel during contact can be learned from a database of matched haptic and visual data recorded from various end-effectors\u27 interactions with hundreds of real-world surfaces. Testing this hypothesis required the creation of a new multimodal sensing apparatus, the collection of a large multimodal dataset, and development of a machine-learning pipeline. This thesis begins by describing the design and construction of the Portable Robotic Optical/Tactile ObservatioN PACKage (PROTONPACK, or Proton for short), an untethered handheld sensing device that emulates the capabilities of the human senses of vision and touch. Its sensory modalities include RGBD vision, egomotion, contact force, and contact vibration. Three interchangeable end-effectors (a steel tooling ball, an OptoForce three-axis force sensor, and a SynTouch BioTac artificial fingertip) allow for different material properties at the contact point and provide additional tactile data. We then detail the calibration process for the motion and force sensing systems, as well as several proof-of-concept surface discrimination experiments that demonstrate the reliability of the device and the utility of the data it collects. This thesis then presents a large-scale dataset of multimodal surface interaction recordings, including 357 unique surfaces such as furniture, fabrics, outdoor fixtures, and items from several private and public material sample collections. Each surface was touched with one, two, or three end-effectors, comprising approximately one minute per end-effector of tapping and dragging at various forces and speeds. We hope that the larger community of robotics researchers will find broad applications for the published dataset. Lastly, we demonstrate an algorithm that learns to estimate haptic surface properties given visual input. Surfaces were rated on hardness, roughness, stickiness, and temperature by the human experimenter and by a pool of purely visual observers. Then we trained an algorithm to perform the same task as well as infer quantitative properties calculated from the haptic data. Overall, the task of predicting haptic properties from vision alone proved difficult for both humans and computers, but a hybrid algorithm using a deep neural network and a support vector machine achieved a correlation between expected and actual regression output between approximately ρ = 0.3 and ρ = 0.5 on previously unseen surfaces

Touch and Go: Learning from Human-Collected Vision and Touch

The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.Comment: Accepted by NeurIPS 2022 Track of Datasets and Benchmark

Proceedings of the 5th international conference on disability, virtual reality and associated technologies (ICDVRAT 2004)

The proceedings of the conferenc

An assessment of cortical function in expert and novice surgeons

Imperial Users onl

Pilot study for subgroup classification for autism spectrum disorder based on dysmorphology and physical measurements in Chinese children

Poster Sessions: 157 - Comorbid Medical Conditions: abstract 157.058 58BACKGROUND: Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder affecting individuals along a continuum of severity in communication, social interaction and behaviour. The impact of ASD significantly varies amongst individuals, and the cause of ASD can originate broadly between genetic and environmental factors. Objectives: Previous ASD researches indicate that early identification combined with a targeted treatment plan involving behavioural interventions and multidisciplinary therapies can provide substantial improvement for ASD patients. Currently there is no cure for ASD, and the clinical variability and uncertainty of the disorder still remains. Hence, the search to unravel heterogeneity within ASD by subgroup classification may provide clinicians with a better understanding of ASD and to work towards a more definitive course of action. METHODS: In this study, a norm of physical measurements including height, weight, head circumference, ear length, outer and inner canthi, interpupillary distance, philtrum, hand and foot length was collected from 658 Typical Developing (TD) Chinese children aged 1 to 7 years (mean age of 4.19 years). The norm collected was compared against 80 ASD Chinese children aged 1 to 12 years (mean age of 4.36 years). We then further attempted to find subgroups within ASD based on identifying physical abnormalities; individuals were classified as (non) dysmorphic with the Autism Dysmorphology Measure (ADM) from physical examinations of 12 body regions. RESULTS: Our results show that there were significant differences between ASD and TD children for measurements in: head circumference (p=0.009), outer (p=0.021) and inner (p=0.021) canthus, philtrum length (p=0.003), right (p=0.023) and left (p=0.20) foot length. Within the 80 ASD patients, 37(46%) were classified as dysmorphic (p=0.00). CONCLUSIONS: This study attempts to identify subgroups within ASD based on physical measurements and dysmorphology examinations. The information from this study seeks to benefit ASD community by identifying possible subtypes of ASD in Chinese population; in seek for a more definitive diagnosis, referral and treatment plan.published_or_final_versio