Neuromorphic event-based slip detection and suppression in robotic grasping and manipulation

Slip detection is essential for robots to make robust grasping and fine manipulation. In this paper, a novel dynamic vision-based finger system for slip detection and suppression is proposed. We also present a baseline and feature based approach to detect object slips under illumination and vibration uncertainty. A threshold method is devised to autonomously sample noise in real-time to improve slip detection. Moreover, a fuzzy based suppression strategy using incipient slip feedback is proposed for regulating the grip force. A comprehensive experimental study of our proposed approaches under uncertainty and system for high-performance precision manipulation are presented. We also propose a slip metric to evaluate such performance quantitatively. Results indicate that the system can effectively detect incipient slip events at a sampling rate of 2kHz ($\Delta t = 500\mu s$) and suppress them before a gross slip occurs. The event-based approach holds promises to high precision manipulation task requirement in industrial manufacturing and household services.Comment: 18 pages, 14 figure

Object grasping and safe manipulation using friction-based sensing.

This project provides a solution for slippage prevention in industrial robotic grippers for the purpose of safe object manipulation. Slippage sensing is performed using novel friction-based sensors, with customisable slippage sensitivity and complemented by an effective slippage prediction strategy. The outcome is a reliable and affordable slippage prevention technology

Model-Based Systems Engineering Applied to the Detection and Correction of Object Slippage Within a Dexterous Robotic Hand from the Laboratory to Simulation

Now more than ever, it is important to have the ability to replicate robotic tasks in simulation and be able to validate the simulation against stakeholder requirements and verify the simulation against simulation requirements. In a previous study, a five-fingered robotic hand, the Shadow Dexterous Hand, with haptic BioTac SP sensors attached was used to detect the moment of slip of an object from the robotic hand while weight was continuously being added and stop the object from falling from the grasp while not overcorrecting. This work was accomplished by Dr. Zhenyu Lin, Dr. John S. Baras, and the author in the Autonomy Robotics Cognition Laboratory at the University of Maryland. This thesis will present the use of Model-Based System Engineering techniques to replicate the detection and correction of object slippage by a five-fingered robotic hand using force feedback control in simulation

Objekt-Manipulation und Steuerung der Greifkraft durch Verwendung von Taktilen Sensoren

This dissertation describes a new type of tactile sensor and an improved version of the dynamic tactile sensing approach that can provide a regularly updated and accurate estimate of minimum applied forces for use in the control of gripper manipulation. The pre-slip sensing algorithm is proposed and implemented into two-finger robot gripper. An algorithm that can discriminate between types of contact surface and recognize objects at the contact stage is also proposed. A technique for recognizing objects using tactile sensor arrays, and a method based on the quadric surface parameter for classifying grasped objects is described. Tactile arrays can recognize surface types on contact, making it possible for a tactile system to recognize translation, rotation, and scaling of an object independently.Diese Dissertation beschreibt eine neue Art von taktilen Sensoren und einen verbesserten Ansatz zur dynamischen Erfassung von taktilen daten, der in regelmäßigen Zeitabständen eine genaue Bewertung der minimalen Greifkraft liefert, die zur Steuerung des Greifers nötig ist. Ein Berechnungsverfahren zur Voraussage des Schlupfs, das in einen Zwei-Finger-Greifarm eines Roboters eingebaut wurde, wird vorgestellt. Auch ein Algorithmus zur Unterscheidung von verschiedenen Oberflächenarten und zur Erkennung von Objektformen bei der Berührung wird vorgestellt. Ein Verfahren zur Objekterkennung mit Hilfe einer Matrix aus taktilen Sensoren und eine Methode zur Klassifikation ergriffener Objekte, basierend auf den Daten einer rechteckigen Oberfläche, werden beschrieben. Mit Hilfe dieser Matrix können unter schiedliche Arten von Oberflächen bei Berührung erkannt werden, was es für das Tastsystem möglich macht, Verschiebung, Drehung und Größe eines Objektes unabhängig voneinander zu erkennen

Adaptive robust interaction control for low-cost robotic grasping

Robotic grasping is a challenging area in the field of robotics. When a gripper starts interacting with an object to perform a grasp, the mechanical properties of the object (stiffness and damping) will play an important role. A gripper which is stable in isolated conditions, can become unstable when coupled to an object. This can lead to the extreme condition where the gripper becomes unstable and generates excessive or insufficient grip force resulting in the grasped object either being crushed, or falling and breaking. In addition to the stability issue, grasp maintenance is one of the most important requirements of any grasp where it guarantees a secure grasp in the presence of any unknown disturbance. The term grasp maintenance refers to the reaction of the controller in the presence of external disturbances, trying to prevent any undesired slippage. To do so, the controller continuously adjusts the grip force. This is a challenging task as it requires an accurate model of the friction and object’s weight to estimate a sufficient grip force to stop the object from slipping while incurring minimum deformation. Unfortunately, in reality, there is no solution which is able to obtain the mechanical properties, frictional coefficient and weight of an object before establishing a mechanical interaction with it. External disturbance forces are also stochastic meaning they are impossible to predict. This thesis addresses both of the problems mentioned above by:Creating a novel variable stiffness gripper, capable of grasping unknown objects, mainly those found in agricultural or food manufacturing companies. In addition to the stabilisation effect of the introduced variable stiffness mechanism, a novel force control algorithm has been designed that passively controls the grip force in variable stiffness grippers. Due to the passive nature of the suggested controller, it completely eliminates the necessity for any force sensor. The combination of both the proposed variable stiffness gripper and the passivity based control provides a unique solution for the stable grasp and force control problem in tendon driven, angular grippers.Introducing a novel active multi input-multi output slip prevention algorithm. The algorithm developed provides a robust control solution to endow direct drive parallel jaw grippers with the capability to stop held objects from slipping while incurring minimum deformation; this can be done without any prior knowledge of the object’s friction and weight. The large number of experiments provided in this thesis demonstrate the robustness of the proposed controller when controlling parallel jaw grippers in order to quickly grip, lift and place a broad range of objects firmly without dropping or crushing them. This is particularly useful for teleoperation and nuclear decommissioning tasks where there is often no accurate information available about the objects to be handled. This can mean that pre-programming of the gripper is required for each different object and for high numbers of objects this is impractical and overly time-consuming. A robust controller, which is able to compensate for any uncertainties regarding the object model and any unknown external disturbances during grasping, is implemented. This work has advanced the state of the art in the following two main areas: Direct impedance modulation for stable grasping in tendon driven, angular grippers. Active MIMO slip prevention grasp control for direct drive parallel jaw grippers

Grasp and stress analysis of an underactuated finger for proprioceptive tactile sensing

This paper presents the design and evaluation of a new sensorized underactuated self-adaptive finger. The design incorporates a two-degrees-of-freedom link-driven underactuated mechanism with an embedded load cell for contact force measurement and a trimmer potentiometer for acquiring joint variables. The utilization of proprioceptive (internal) sensors results in tactile-like sensations in the finger without compromising the size and complexity of the proposed design. To obtain an optimum finger design, the placement of the load cell is analyzed using finite element method. The design of the finger features a particular rounded shape of the distal phalanx and specific size ratio between the phalanxes to enable both precision and power grasps. A quantitative evaluation of the grasp efficiency by constructing a grasp wrench space is provided. The effectiveness of the proposed design is verified through experimental results that demonstrate the grasp external wrench tolerance, shape adaptability, and tactile capability. All CAD files and ROS package for the proposed underactuated design can be found on https://github.com/mahyaret

Robotic Manipulation of Environmentally Constrained Objects Using Underactuated Hands

Robotics for agriculture represents the ultimate application of one of our society\u27s latest and most advanced innovations to its most ancient and vital industry. Over the course of history, mechanization and automation have increased crop output several orders of magnitude, enabling a geometric growth in population and an increase in quality of life across the globe. As a challenging step, manipulating objects in harvesting automation is still under investigation in literature. Harvesting or the process of gathering ripe crops can be described as breaking environmentally constrained objects into two or more pieces at the desired locations. In this thesis, the problem of purposefully failing (breaking) or yielding objects by a robotic gripper is investigated. A failure task is first formulated using mechanical failure theories. Next, a grasp quality measure is presented to characterize a suitable grasp configuration and systematically control the failure behavior of the object. This approach combines the failure task and the capability of the gripper for wrench insertion. The friction between the object and the gripper is used to formulate the capability of the gripper for wrench insertion. A new method inspired by the human pre-manipulation process is introduced to utilize the gripper itself as the measurement tool and obtain a friction model. The developed friction model is capable of capturing the anisotropic behavior of materials which is the case for most fruits and vegetables.The limited operating space for harvesting process, the vulnerability of agricultural products and clusters of crops demand strict conditions for the manipulation process. This thesis presents a new sensorized underactuated self-adaptive finger to address the stringent conditions in the agricultural environment. This design incorporates link-driven underactuated mechanism with an embedded load cell for contact force measurement and a trimmer potentiometer for acquiring joint variables. The integration of these sensors results in tactile-like sensations in the finger without compromising the size and complexity of the proposed design. To obtain an optimum finger design, the placement of the load cell is analyzed using Finite Element Method (FEM). The design of the finger features a particular round shape of the distal phalanx and specific size ratio between the phalanxes to enable both precision and power grasps. A quantitative evaluation of the grasp efficiency by constructing a grasp wrench space is also provided. The effectiveness of the proposed designs and theories are verified through real-time experiments. For conducting the experiments in real-time, a software/hardware platform capable of dataset management is crucial. In this thesis, a new comprehensive software interface for integration of industrial robots with peripheral tools and sensors is designed and developed. This software provides a real-time low-level access to the manipulator controller. Furthermore, Data Acquisition boards are integrated into the software which enables Rapid Prototyping methods. Additionally, Hardware-in-the-loop techniques can be implemented by adding the complexity of the plant under control to the test platform. The software is a collection of features developed and distributed under GPL V3.0

Planning, Monitoring and Learning with Safety and Temporal Constraints for Robotic Systems

In this thesis, we address the problem of planning, monitoring and learning in robotic systems, while considering the safety and time constraints. Motion and action planning for robotic systems is important for real, physical world applications. Robots are capable of performing repetitive tasks at speeds and accuracies that far exceed those of human operators and are widely used in manufacturing, medical fields and even transportation. Planning commonly refers to a process of converting high-level task specifications into low-level control commands that can be executed on the system of interest. Time behavior is a most important issue for the autonomous systems of interest, and it is critical for many robotic tasks. Most state of the art methods, however, are not capable of providing the framework needed for the autonomous systems to plan under finite time constraints. Safety and time constraints are two important aspects for the plan. We are interested in the safety of the plan, such as ``Can the robot reach the goal without collision?''. We are also interested in the time constraints for the plan, such as ``Can the robot finish this task after 3 minutes but no later than 5 minutes?''. These type of tasks are important to understand in robot search and rescue or cooperative robotic production line. In this thesis, we address these problems by two different approaches, the first one is a timed automata based approach, which focuses on a more high-level, abstracted result with less computational requirement. The other one involves converting the problem into a mixed integer linear programming (MILP) with more low-level control details but requires higher computational power. Both methods are able to automatically generate a plan that are guaranteed to be correct. The robotic systems may behave differently in runtime and not able to execute the task perfectly as planned. Given that a robotic system is naturally cyber-physical, and malfunctions can have safety consequences, monitoring the system’s behavior at runtime can be key to safe operation. Therefore, it is important to consider both time and space tolerances during the planning phase, and also design runtime monitors for error detection and possible self-correction. We provide an optimization-based formulation which takes the tolerances into account, and we have designed runtime monitors to monitor the status of the systems, as well as an event-triggered model predictive controller for self-correction. Learning is another very important aspect for the robotics field. We hope to only provide the robot with high-level task specifications, and the robot learns to accomplish the task. Thus, in the next part of this thesis, we discussed how the robot could learn to accomplish task specified by metric interval temporal logic, and how the robot could replan and self-correct if the initial plan fails to execute correctly. As the field of robotics is expanding from the fixed environment of a production line to complex human environments, robots are required to perform increasingly human-like manipulation tasks. Thus, for the last aspect of the thesis, we considered a manipulation task with dexterous robotic hand - Shadow Hand. We collected the multimodal haptic-vision dataset, and proposed the framework of self-assurance slippage detection and correction. We provided the simulation and also real-world implementation with a UR10 and Shadowhand robotic system

Control techniques for mechatronic assisted surgery

The treatment response for traumatic head injured patients can be improved by using an autonomous robotic system to perform basic, time-critical emergency neurosurgery, reducing costs and saving lives. In this thesis, a concept for a neurosurgical robotic system is proposed to perform three specific emergency neurosurgical procedures; they are the placement of an intracranial pressure monitor, external ventricular drainage, and the evacuation of chronic subdural haematoma. The control methods for this system are investigated following a curiosity led approach. Individual problems are interpreted in the widest sense and solutions posed that are general in nature. Three main contributions result from this approach: 1) a clinical evidence based review of surgical robotics and a methodology to assist in their evaluation, 2) a new controller for soft-grasping of objects, and 3) new propositions and theorems for chatter suppression sliding mode controllers. These contributions directly assist in the design of the control system of the neurosurgical robot and, more broadly, impact other areas outside the narrow con nes of the target application. A methodology for applied research in surgical robotics is proposed. The methodology sets out a hierarchy of criteria consisting of three tiers, with the most important being the bottom tier and the least being the top tier. It is argued that a robotic system must adhere to these criteria in order to achieve acceptability. Recent commercial systems are reviewed against these criteria, and are found to conform up to at least the bottom and intermediate tiers. However, the lack of conformity to the criteria in the top tier, combined with the inability to conclusively prove increased clinical benefit, particularly symptomatic benefit, is shown to be hampering the potential of surgical robotics in gaining wide establishment. A control scheme for soft-grasping objects is presented. Grasping a soft or fragile object requires the use of minimum contact force to prevent damage or deformation. Without precise knowledge of object parameters, real-time feedback control must be used to regulate the contact force and prevent slip. Moreover, the controller must be designed to have good performance characteristics to rapidly modulate the fingertip contact force in response to a slip event. A fuzzy sliding mode controller combined with a disturbance observer is proposed for contact force control and slip prevention. The robustness of the controller is evaluated through both simulation and experiment. The control scheme was found to be effective and robust to parameter uncertainty. When tested on a real system, however, chattering phenomena, well known to sliding mode research, was induced by the unmodelled suboptimal components of the system (filtering, backlash, and time delays). This reduced the controller performance. The problem of chattering and potential solutions are explored. Real systems using sliding mode controllers, such as the control scheme for soft-grasping, have a tendency to chatter at high frequencies. This is caused by the sliding mode controller interacting with un-modelled parasitic dynamics at the actuator-input and sensor-output of the plant. As a result, new chatter-suppression sliding mode controllers have been developed, which introduce new parameters into the system. However, the effect any particular choice of parameters has on system performance is unclear, and this can make tuning the parameters to meet a set of performance criteria di cult. In this thesis, common chatter-suppression sliding mode control strategies are surveyed and simple design and estimation methods are proposed. The estimation methods predict convergence, chattering amplitude, settling time, and maximum output bounds (overshoot) using harmonic linearizations and invariant ellipsoid sets