Possibilities of man-machine interaction through the perception of human gestures

A mesura que les màquines s'utilitzen interaccionant cada cop més amb les persones, la necessitat d'interfícies més amigables esdevé una necessitat creixent. La comunicació oral persona-màquina com una forma d'interacció utilitzant el llenguatge natural és cada vegada més usual. La interpretació dels gestos humans pot, en certes aplicacions, complementar aquesta comunicació oral. Aquest article descriu un sistema d'interpretació dels gestos basat en la visió per computador. El procés d'interpretació realitza la detecció i seguiment d'un operador humà, i a partir dels seus moviments interpreta un conjunt específic d'ordres gestuals, en temps real.As man-machine interaction grows there is an increasing need for friendly interfaces. Human-machine oral communication as a means of natural language interaction is becoming quite common. Interpretation of human gestures can, in some applications, complement such communication. This article describes an interpretation of gestures procedure. The system is based on a computer vision system for the detection and tracking of a human operator and the interpretation of a specific set of human gestures in real time

A snake-based scheme for path planning and control with constraints by distributed visual sensors

YesThis paper proposes a robot navigation scheme using wireless visual sensors deployed in an environment. Different from the conventional autonomous robot approaches, the scheme intends to relieve massive on-board information processing required by a robot to its environment so that a robot or a vehicle with less intelligence can exhibit sophisticated mobility. A three-state snake mechanism is developed for coordinating a series of sensors to form a reference path. Wireless visual sensors communicate internal forces with each other along the reference snake for dynamic adjustment, react to repulsive forces from obstacles, and activate a state change in the snake body from a flexible state to a rigid or even to a broken state due to kinematic or environmental constraints. A control snake is further proposed as a tracker of the reference path, taking into account the robot’s non-holonomic constraint and limited steering power. A predictive control algorithm is developed to have an optimal velocity profile under robot dynamic constraints for the snake tracking. They together form a unified solution for robot navigation by distributed sensors to deal with the kinematic and dynamic constraints of a robot and to react to dynamic changes in advance. Simulations and experiments demonstrate the capability of a wireless sensor network to carry out low-level control activities for a vehicle.Royal Society, Natural Science Funding Council (China

Multi-scale metrology for automated non-destructive testing systems

This thesis was previously held under moratorium from 5/05/2020 to 5/05/2022The use of lightweight composite structures in the aerospace industry is now commonplace. Unlike conventional materials, these parts can be moulded into complex aerodynamic shapes, which are diffcult to inspect rapidly using conventional Non-Destructive Testing (NDT) techniques. Industrial robots provide a means of automating the inspection process due to their high dexterity and improved path planning methods. This thesis concerns using industrial robots as a method for assessing the quality of components with complex geometries. The focus of the investigations in this thesis is on improving the overall system performance through the use of concepts from the field of metrology, specifically calibration and traceability. The use of computer vision is investigated as a way to increase automation levels by identifying a component's type and approximate position through comparison with CAD models. The challenges identified through this research include developing novel calibration techniques for optimising sensor integration, verifying system performance using laser trackers, and improving automation levels through optical sensing. The developed calibration techniques are evaluated experimentally using standard reference samples. A 70% increase in absolute accuracy was achieved in comparison to manual calibration techniques. Inspections were improved as verified by a 30% improvement in ultrasonic signal response. A new approach to automatically identify and estimate the pose of a component was developed specifically for automated NDT applications. The method uses 2D and 3D camera measurements along with CAD models to extract and match shape information. It was found that optical large volume measurements could provide suffciently high accuracy measurements to allow ultrasonic alignment methods to work, establishing a multi-scale metrology approach to increasing automation levels. A classification framework based on shape outlines extracted from images was shown to provide over 88% accuracy on a limited number of samples.The use of lightweight composite structures in the aerospace industry is now commonplace. Unlike conventional materials, these parts can be moulded into complex aerodynamic shapes, which are diffcult to inspect rapidly using conventional Non-Destructive Testing (NDT) techniques. Industrial robots provide a means of automating the inspection process due to their high dexterity and improved path planning methods. This thesis concerns using industrial robots as a method for assessing the quality of components with complex geometries. The focus of the investigations in this thesis is on improving the overall system performance through the use of concepts from the field of metrology, specifically calibration and traceability. The use of computer vision is investigated as a way to increase automation levels by identifying a component's type and approximate position through comparison with CAD models. The challenges identified through this research include developing novel calibration techniques for optimising sensor integration, verifying system performance using laser trackers, and improving automation levels through optical sensing. The developed calibration techniques are evaluated experimentally using standard reference samples. A 70% increase in absolute accuracy was achieved in comparison to manual calibration techniques. Inspections were improved as verified by a 30% improvement in ultrasonic signal response. A new approach to automatically identify and estimate the pose of a component was developed specifically for automated NDT applications. The method uses 2D and 3D camera measurements along with CAD models to extract and match shape information. It was found that optical large volume measurements could provide suffciently high accuracy measurements to allow ultrasonic alignment methods to work, establishing a multi-scale metrology approach to increasing automation levels. A classification framework based on shape outlines extracted from images was shown to provide over 88% accuracy on a limited number of samples

Improving the safety and efficiency of rail yard operations using robotics

textSignificant efforts have been expended by the railroad industry to make operations safer and more efficient through the intelligent use of sensor data. This work proposes to take the technology one step further to use this data for the control of physical systems designed to automate hazardous railroad operations, particularly those that require humans to interact with moving trains. To accomplish this, application specific requirements must be established to design self-contained machine vision and robotic solutions to eliminate the risks associated with existing manual operations. Present-day rail yard operations have been identified as good candidates to begin development. Manual uncoupling, in particular, of rolling stock in classification yards has been investigated. To automate this process, an intelligent robotic system must be able to detect, track, approach, contact, and manipulate constrained objects on equipment in motion. This work presents multiple prototypes capable of autonomously uncoupling full-scale freight cars using feedback from its surrounding environment. Geometric image processing algorithms and machine learning techniques were implemented to accurately identify cylindrical objects in point clouds generated in real-vi time. Unique methods fusing velocity and vision data were developed to synchronize a pair of moving rigid bodies in real-time. Multiple custom end-effectors with in-built compliance and fault tolerance were designed, fabricated, and tested for grasping and manipulating cylindrical objects. Finally, an event-driven robotic control application was developed to safely and reliably uncouple freight cars using data from 3D cameras, velocity sensors, force/torque transducers, and intelligent end-effector tooling. Experimental results in a lab setting confirm that modern robotic and sensing hardware can be used to reliably separate pairs of rolling stock up to two miles per hour. Additionally, subcomponents of the autonomous pin-pulling system (APPS) were designed to be modular to the point where they could be used to automate other hazardous, labor-intensive tasks found in U.S. classification yards. Overall, this work supports the deployment of autonomous robotic systems in semi-unstructured yard environments to increase the safety and efficiency of rail operations.Mechanical Engineerin

Real-Time, Multiple Pan/Tilt/Zoom Computer Vision Tracking and 3D Positioning System for Unmanned Aerial System Metrology

The study of structural characteristics of Unmanned Aerial Systems (UASs) continues to be an important field of research for developing state of the art nano/micro systems. Development of a metrology system using computer vision (CV) tracking and 3D point extraction would provide an avenue for making these theoretical developments. This work provides a portable, scalable system capable of real-time tracking, zooming, and 3D position estimation of a UAS using multiple cameras. Current state-of-the-art photogrammetry systems use retro-reflective markers or single point lasers to obtain object poses and/or positions over time. Using a CV pan/tilt/zoom (PTZ) system has the potential to circumvent their limitations. The system developed in this paper exploits parallel-processing and the GPU for CV-tracking, using optical flow and known camera motion, in order to capture a moving object using two PTU cameras. The parallel-processing technique developed in this work is versatile, allowing the ability to test other CV methods with a PTZ system using known camera motion. Utilizing known camera poses, the object\u27s 3D position is estimated and focal lengths are estimated for filling the image to a desired amount. This system is tested against truth data obtained using an industrial system

Mobile Robots Navigation

Mobile robots navigation includes different interrelated activities: (i) perception, as obtaining and interpreting sensory information; (ii) exploration, as the strategy that guides the robot to select the next direction to go; (iii) mapping, involving the construction of a spatial representation by using the sensory information perceived; (iv) localization, as the strategy to estimate the robot position within the spatial map; (v) path planning, as the strategy to find a path towards a goal location being optimal or not; and (vi) path execution, where motor actions are determined and adapted to environmental changes. The book addresses those activities by integrating results from the research work of several authors all over the world. Research cases are documented in 32 chapters organized within 7 categories next described