Learning-based robotic manipulation for dynamic object handling : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Mechatronic Engineering at the School of Food and Advanced Technology, Massey University, Turitea Campus, Palmerston North, New Zealand

Figures are re-used in this thesis with permission of their respective publishers or under a Creative Commons licence.Recent trends have shown that the lifecycles and production volumes of modern products are shortening. Consequently, many manufacturers subject to frequent change prefer flexible and reconfigurable production systems. Such schemes are often achieved by means of manual assembly, as conventional automated systems are perceived as lacking flexibility. Production lines that incorporate human workers are particularly common within consumer electronics and small appliances. Artificial intelligence (AI) is a possible avenue to achieve smart robotic automation in this context. In this research it is argued that a robust, autonomous object handling process plays a crucial role in future manufacturing systems that incorporate robotics—key to further closing the gap between manual and fully automated production. Novel object grasping is a difficult task, confounded by many factors including object geometry, weight distribution, friction coefficients and deformation characteristics. Sensing and actuation accuracy can also significantly impact manipulation quality. Another challenge is understanding the relationship between these factors, a specific grasping strategy, the robotic arm and the employed end-effector. Manipulation has been a central research topic within robotics for many years. Some works focus on design, i.e. specifying a gripper-object interface such that the effects of imprecise gripper placement and other confounding control-related factors are mitigated. Many universal robotic gripper designs have been considered, including 3-fingered gripper designs, anthropomorphic grippers, granular jamming end-effectors and underactuated mechanisms. While such approaches have maintained some interest, contemporary works predominantly utilise machine learning in conjunction with imaging technologies and generic force-closure end-effectors. Neural networks that utilise supervised and unsupervised learning schemes with an RGB or RGB-D input make up the bulk of publications within this field. Though many solutions have been studied, automatically generating a robust grasp configuration for objects not known a priori, remains an open-ended problem. An element of this issue relates to a lack of objective performance metrics to quantify the effectiveness of a solution—which has traditionally driven the direction of community focus by highlighting gaps in the state-of-the-art. This research employs monocular vision and deep learning to generate—and select from—a set of hypothesis grasps. A significant portion of this research relates to the process by which a final grasp is selected. Grasp synthesis is achieved by sampling the workspace using convolutional neural networks trained to recognise prospective grasp areas. Each potential pose is evaluated by the proposed method in conjunction with other input modalities—such as load-cells and an alternate perspective. To overcome human bias and build upon traditional metrics, scores are established to objectively quantify the quality of an executed grasp trial. Learning frameworks that aim to maximise for these scores are employed in the selection process to improve performance. The proposed methodology and associated metrics are empirically evaluated. A physical prototype system was constructed, employing a Dobot Magician robotic manipulator, vision enclosure, imaging system, conveyor, sensing unit and control system. Over 4,000 trials were conducted utilising 100 objects. Experimentation showed that robotic manipulation quality could be improved by 10.3% when selecting to optimise for the proposed metrics—quantified by a metric related to translational error. Trials further demonstrated a grasp success rate of 99.3% for known objects and 98.9% for objects for which a priori information is unavailable. For unknown objects, this equated to an improvement of approximately 10% relative to other similar methodologies in literature. A 5.3% reduction in grasp rate was observed when removing the metrics as selection criteria for the prototype system. The system operated at approximately 1 Hz when contemporary hardware was employed. Experimentation demonstrated that selecting a grasp pose based on the proposed metrics improved grasp rates by up to 4.6% for known objects and 2.5% for unknown objects—compared to selecting for grasp rate alone. This project was sponsored by the Richard and Mary Earle Technology Trust, the Ken and Elizabeth Powell Bursary and the Massey University Foundation. Without the financial support provided by these entities, it would not have been possible to construct the physical robotic system used for testing and experimentation. This research adds to the field of robotic manipulation, contributing to topics on grasp-induced error analysis, post-grasp error minimisation, grasp synthesis framework design and general grasp synthesis. Three journal publications and one IEEE Xplore paper have been published as a result of this research

A METHODOLOGY FOR AUTONOMOUS ROOF BOLT INSTALLATION USING INDUSTRIAL ROBOTICS

The mining sector is currently in the stage of adopting more automation, and with it, robotics. Autonomous bolting in underground environments remains a hot topic for the mining industry. Roof bolter operators are exposed to hazardous conditions due to their proximity to the unsupported roof, loose bolts, and heavy spinning mass. Prolonged exposure to the risk inevitably leads to accidents and injuries. The current thesis presents the development of a robotic assembly capable of carrying out the entire sequence of roof bolting operations in full and partial autonomous sensor-driven rock bolting operations to achieve a high-impact health and safety intervention for equipment operators. The automation of a complete cycle of drill steel positioning, drilling, bolt orientation and placement, resin placement, and bolt securing is discussed using an anthropomorphic robotic arm.A human-computer interface is developed to enable the interaction of the operators with the machines. Collision detection techniques will have to be implemented to minimize the impact after an unexpected collision has occurred. A robust failure-detection protocol is developed to check the vital parameters of robot operations continuously. This unique approach to automation of small materials handling is described with lessons learned. A user-centered GUI has been developed that allows for a human user to control and monitor the autonomous roof bolter. Preliminary tests have been conducted in a mock mine to evaluate the developed system\u27s performance. In addition, a number of different scenarios simulating typical missions that a roof bolter needs to undertake in an underground coal mine were tested

Human factors in space telepresence

The problems of interfacing a human with a teleoperation system, for work in space are discussed. Much of the information presented here is the result of experience gained by the M.I.T. Space Systems Laboratory during the past two years of work on the ARAMIS (Automation, Robotics, and Machine Intelligence Systems) project. Many factors impact the design of the man-machine interface for a teleoperator. The effects of each are described in turn. An annotated bibliography gives the key references that were used. No conclusions are presented as a best design, since much depends on the particular application desired, and the relevant technology is swiftly changing

NASA space station automation: AI-based technology review

Research and Development projects in automation for the Space Station are discussed. Artificial Intelligence (AI) based automation technologies are planned to enhance crew safety through reduced need for EVA, increase crew productivity through the reduction of routine operations, increase space station autonomy, and augment space station capability through the use of teleoperation and robotics. AI technology will also be developed for the servicing of satellites at the Space Station, system monitoring and diagnosis, space manufacturing, and the assembly of large space structures

Control techniques for mechatronic assisted surgery

The treatment response for traumatic head injured patients can be improved by using an autonomous robotic system to perform basic, time-critical emergency neurosurgery, reducing costs and saving lives. In this thesis, a concept for a neurosurgical robotic system is proposed to perform three specific emergency neurosurgical procedures; they are the placement of an intracranial pressure monitor, external ventricular drainage, and the evacuation of chronic subdural haematoma. The control methods for this system are investigated following a curiosity led approach. Individual problems are interpreted in the widest sense and solutions posed that are general in nature. Three main contributions result from this approach: 1) a clinical evidence based review of surgical robotics and a methodology to assist in their evaluation, 2) a new controller for soft-grasping of objects, and 3) new propositions and theorems for chatter suppression sliding mode controllers. These contributions directly assist in the design of the control system of the neurosurgical robot and, more broadly, impact other areas outside the narrow con nes of the target application. A methodology for applied research in surgical robotics is proposed. The methodology sets out a hierarchy of criteria consisting of three tiers, with the most important being the bottom tier and the least being the top tier. It is argued that a robotic system must adhere to these criteria in order to achieve acceptability. Recent commercial systems are reviewed against these criteria, and are found to conform up to at least the bottom and intermediate tiers. However, the lack of conformity to the criteria in the top tier, combined with the inability to conclusively prove increased clinical benefit, particularly symptomatic benefit, is shown to be hampering the potential of surgical robotics in gaining wide establishment. A control scheme for soft-grasping objects is presented. Grasping a soft or fragile object requires the use of minimum contact force to prevent damage or deformation. Without precise knowledge of object parameters, real-time feedback control must be used to regulate the contact force and prevent slip. Moreover, the controller must be designed to have good performance characteristics to rapidly modulate the fingertip contact force in response to a slip event. A fuzzy sliding mode controller combined with a disturbance observer is proposed for contact force control and slip prevention. The robustness of the controller is evaluated through both simulation and experiment. The control scheme was found to be effective and robust to parameter uncertainty. When tested on a real system, however, chattering phenomena, well known to sliding mode research, was induced by the unmodelled suboptimal components of the system (filtering, backlash, and time delays). This reduced the controller performance. The problem of chattering and potential solutions are explored. Real systems using sliding mode controllers, such as the control scheme for soft-grasping, have a tendency to chatter at high frequencies. This is caused by the sliding mode controller interacting with un-modelled parasitic dynamics at the actuator-input and sensor-output of the plant. As a result, new chatter-suppression sliding mode controllers have been developed, which introduce new parameters into the system. However, the effect any particular choice of parameters has on system performance is unclear, and this can make tuning the parameters to meet a set of performance criteria di cult. In this thesis, common chatter-suppression sliding mode control strategies are surveyed and simple design and estimation methods are proposed. The estimation methods predict convergence, chattering amplitude, settling time, and maximum output bounds (overshoot) using harmonic linearizations and invariant ellipsoid sets

A Hierarchical Architecture for Flexible Human-Robot Collaboration

This thesis is devoted to design a software architecture for Human- Robot Collaboration (HRC), to enhance the robots\u2019 abilities for working alongside humans. We propose FlexHRC, a hierarchical and flexible human-robot cooperation architecture specifically designed to provide collaborative robots with an extended degree of autonomy when supporting human operators in tasks with high-variability. Along with FlexHRC, we have introduced novel techniques appropriate for three interleaved levels, namely perception, representation, and action, each one aimed at addressing specific traits of humanrobot cooperation tasks. The Industry 4.0 paradigm emphasizes the crucial benefits that collaborative robots could bring to the whole production process. In this context, a yet unreached enabling technology is the design of robots able to deal at all levels with humans\u2019 intrinsic variability, which is not only a necessary element to a comfortable working experience for humans but also a precious capability for efficiently dealing with unexpected events. Moreover, a flexible assembly of semi-finished products is one of the expected features of next-generation shop-floor lines. Currently, such flexibility is placed on the shoulders of human operators, who are responsible for product variability, and therefore they are subject to potentially high stress levels and cognitive load when dealing with complex operations. At the same time, operations in the shop-floor are still very structured and well-defined. Collaborative robots have been designed to allow for a transition of such burden from human operators to robots that are flexible enough to support them in high-variability tasks while they unfold. As mentioned before, FlexHRC architecture encompasses three perception, action, and representation levels. The perception level relies on wearable sensors for human action recognition and point cloud data for perceiving the object in the scene. The action level embraces four components, the robot execution manager for decoupling action planning from robot motion planning and mapping the symbolic actions to the robot controller command interface, a task Priority framework to control the robot, a differential equation solver to simulate and evaluate the robot behaviour on-the-fly, and finally a random-based method for the robot path planning. The representation level depends on AND/OR graphs for the representation of and the reasoning upon human-robot cooperation models online, a task manager to plan, adapt, and make decision for the robot behaviors, and a knowledge base in order to store the cooperation and workspace information. We evaluated the FlexHRC functionalities according to the application desired objectives. This evaluation is accompanied with several experiments, namely collaborative screwing task, coordinated transportation of the objects in cluttered environment, collaborative table assembly task, and object positioning tasks. The main contributions of this work are: (i) design and implementation of FlexHRC which enables the functional requirements necessary for the shop-floor assembly application such as task and team level flexibility, scalability, adaptability, and safety just a few to name, (ii) development of the task representation, which integrates a hierarchical AND/OR graph whose online behaviour is formally specified using First Order Logic, (iii) an in-the-loop simulation-based decision making process for the operations of collaborative robots coping with the variability of human operator actions, (iv) the robot adaptation to the human on-the-fly decisions and actions via human action recognition, and (v) the predictable robot behavior to the human user thanks to the task priority based control frame, the introduced path planner, and the natural and intuitive communication of the robot with the human

Design and Prototyping of a 3DOF Worm-drive Robot Arm

PEARLDOI for this publication Labels There are no labels on this output. Add labels Many designs for robot arms exist. Here we present an affordable revolute arm, capable of executing simple pick-and-place tasks. The arm employs a double parallelogram structure, which ensures its endpoint angle in the plane of the upper arm remains fixed without the need for additional actuation. Its limbs are fabricated from circular tubes made from bonded carbon fiber, to ensure low moving mass while maintaining high rigidity. All custom structural elements of the arm are produced via 3D printing. We employ worm-drive DC motor actuation to ensure that stationary configurations are maintained without the necessity of continuous motor power. Our discussion encompasses an analysis of the arm’s kinematics. A simulation of the arm’s operation was carried out in MATLAB, revealing key operational metrics. In conclusion, we achieved extrinsic endpoint position tracking by implementing its inverse kinematics and PID control using a microcontroller. We also demonstrate the arm’s functionality through simple movement tracking and object manipulation tasks

Human-centered Electric Prosthetic (HELP) Hand

Through a partnership with Indian non-profit Bhagwan Mahaveer Viklang Sahayata Samiti, we designed a functional, robust, and and low cost electrically powered prosthetic hand that communicates with unilateral, transradial, urban Indian amputees through a biointerface. The device uses compliant tendon actuation, a small linear servo, and a wearable garment outfitted with flex sensors to produce a device that, once placed inside a prosthetic glove, is anthropomorphic in both look and feel. The prosthesis was developed such that future groups can design for manufacturing and distribution in India