Control and Guidance of Low-Cost Robots via Gesture Perception for Monitoring Activities in the Home

This paper describes the development of a low-cost mini-robot that is controlled by visual gestures. The prototype allows a person with disabilities to perform visual inspections indoors and in domestic spaces. Such a device could be used as the operator's eyes obviating the need for him to move about. The robot is equipped with a motorised webcam that is also controlled by visual gestures. This camera is used to monitor tasks in the home using the mini-robot while the operator remains quiet and motionless. The prototype was evaluated through several experiments testing the ability to use the mini-robot’s kinematics and communication systems to make it follow certain paths. The mini-robot can be programmed with specific orders and can be tele-operated by means of 3D hand gestures to enable the operator to perform movements and monitor tasks from a distance.The research that yielded these results has received funding from the projects DPI2012-32390 and PROMETEO/2013/085

Gemini Telepresence Robot System Design: A Low-Cost Solution for Manipulation and Enhanced Perception of Telepresence Robots

Current telepresence robots are costly and only allow the operator to see the environment on a 2D screen and move around on a wheelbase. Thus, these telepresence devices are severely limited because of the high barrier of entry, and the operator is unable to manipulate objects or easily perceive the world in 3D. Therefore, to address these gaps in capabilities, Gemini, an open-source telepresence humanoid robot and interface station, was designed to grant the operator the ability to manipulate objects, expand the human interface by putting the user in the 3D world with the use of a virtual reality (VR) headset, and be low-cost. The simplistic, low-cost, and intuitive controls of Gemini promote early adoption by businesses and medical personnel to grant increased telepresence needs. In addition, this platform can be utilized by robotics enthusiasts and university researchers studying humanoid robotics or human-robot interaction. This paper presents an overview of the Gemini robot’s mechanical, electrical, and programmatic systems. Upon completion of this study, it was found that Gemini was able to grant the ability to manipulate objects, increase user perception with intuitive controls, in addition to costing approximately 30% less than commercial telepresence robots. Furthermore, the paper is concluded with remarks on future iterations of the project

Instrumentation, Data, And Algorithms For Visually Understanding Haptic Surface Properties

Autonomous robots need to efficiently walk over varied surfaces and grasp diverse objects. We hypothesize that the association between how such surfaces look and how they physically feel during contact can be learned from a database of matched haptic and visual data recorded from various end-effectors\u27 interactions with hundreds of real-world surfaces. Testing this hypothesis required the creation of a new multimodal sensing apparatus, the collection of a large multimodal dataset, and development of a machine-learning pipeline. This thesis begins by describing the design and construction of the Portable Robotic Optical/Tactile ObservatioN PACKage (PROTONPACK, or Proton for short), an untethered handheld sensing device that emulates the capabilities of the human senses of vision and touch. Its sensory modalities include RGBD vision, egomotion, contact force, and contact vibration. Three interchangeable end-effectors (a steel tooling ball, an OptoForce three-axis force sensor, and a SynTouch BioTac artificial fingertip) allow for different material properties at the contact point and provide additional tactile data. We then detail the calibration process for the motion and force sensing systems, as well as several proof-of-concept surface discrimination experiments that demonstrate the reliability of the device and the utility of the data it collects. This thesis then presents a large-scale dataset of multimodal surface interaction recordings, including 357 unique surfaces such as furniture, fabrics, outdoor fixtures, and items from several private and public material sample collections. Each surface was touched with one, two, or three end-effectors, comprising approximately one minute per end-effector of tapping and dragging at various forces and speeds. We hope that the larger community of robotics researchers will find broad applications for the published dataset. Lastly, we demonstrate an algorithm that learns to estimate haptic surface properties given visual input. Surfaces were rated on hardness, roughness, stickiness, and temperature by the human experimenter and by a pool of purely visual observers. Then we trained an algorithm to perform the same task as well as infer quantitative properties calculated from the haptic data. Overall, the task of predicting haptic properties from vision alone proved difficult for both humans and computers, but a hybrid algorithm using a deep neural network and a support vector machine achieved a correlation between expected and actual regression output between approximately ρ = 0.3 and ρ = 0.5 on previously unseen surfaces

Videos in Context for Telecommunication and Spatial Browsing

The research presented in this thesis explores the use of videos embedded in panoramic imagery to transmit spatial and temporal information describing remote environments and their dynamics. Virtual environments (VEs) through which users can explore remote locations are rapidly emerging as a popular medium of presence and remote collaboration. However, capturing visual representation of locations to be used in VEs is usually a tedious process that requires either manual modelling of environments or the employment of specific hardware. Capturing environment dynamics is not straightforward either, and it is usually performed through specific tracking hardware. Similarly, browsing large unstructured video-collections with available tools is difficult, as the abundance of spatial and temporal information makes them hard to comprehend. At the same time, on a spectrum between 3D VEs and 2D images, panoramas lie in between, as they offer the same 2D images accessibility while preserving 3D virtual environments surrounding representation. For this reason, panoramas are an attractive basis for videoconferencing and browsing tools as they can relate several videos temporally and spatially. This research explores methods to acquire, fuse, render and stream data coming from heterogeneous cameras, with the help of panoramic imagery. Three distinct but interrelated questions are addressed. First, the thesis considers how spatially localised video can be used to increase the spatial information transmitted during video mediated communication, and if this improves quality of communication. Second, the research asks whether videos in panoramic context can be used to convey spatial and temporal information of a remote place and the dynamics within, and if this improves users' performance in tasks that require spatio-temporal thinking. Finally, the thesis considers whether there is an impact of display type on reasoning about events within videos in panoramic context. These research questions were investigated over three experiments, covering scenarios common to computer-supported cooperative work and video browsing. To support the investigation, two distinct video+context systems were developed. The first telecommunication experiment compared our videos in context interface with fully-panoramic video and conventional webcam video conferencing in an object placement scenario. The second experiment investigated the impact of videos in panoramic context on quality of spatio-temporal thinking during localization tasks. To support the experiment, a novel interface to video-collection in panoramic context was developed and compared with common video-browsing tools. The final experimental study investigated the impact of display type on reasoning about events. The study explored three adaptations of our video-collection interface to three display types. The overall conclusion is that videos in panoramic context offer a valid solution to spatio-temporal exploration of remote locations. Our approach presents a richer visual representation in terms of space and time than standard tools, showing that providing panoramic contexts to video collections makes spatio-temporal tasks easier. To this end, videos in context are suitable alternative to more difficult, and often expensive solutions. These findings are beneficial to many applications, including teleconferencing, virtual tourism and remote assistance

An Investigation into the Feasibility of Remote Object Manipulation Using Consumer Virtual Reality Hardware and Decoupled Mapping Control

With the occurrence of the COVID-19 pandemic, many health workers in hospitals, nursing homes and quarantine facilities were put at increased health risk in their workplace. One effective solution to reduce this risk is to use Telerobotics which enables health workers to carry out their tasks remotely. Such a system must be able to perform a wide range of tasks in an unpredictable environment. This research paper will focus on the pick and place of small objects, which is one of the most common tasks in health care settings. This research designs a prototype telerobotic system using an innovative and new control method to remotely pick and place small objects and will perform the following test: An operator will use a virtual reality headset to remotely control a robot arm located more than 44 kilometres away from the operator and pick several common everyday objects and place them into a container. By developing this proof of concept prototype, this research will help to accelerate the adoption of telerobotic technologies in health care and other industries