The Cluttered Environment Picking Benchmark (CEPB) for Advanced Warehouse Automation

Autonomous and reliable robotic grasping is a desirable functionality in robotic manipulation and is still an open problem. Standardized benchmarks are important tools for evaluating and comparing robotic grasping and manipulation systems among different research groups and also for sharing with the community the best practices to learn from errors. An ideal benchmarking protocol should encompass the different aspects underpinning grasp execution, including the mechatronic design of grippers, planning, perception, and control to give information on each aspect and the overall problem. This article gives an overview of the benchmarks, datasets, and competitions that have been proposed and adopted in the last few years and presents a novel benchmark with protocols for different tasks that evaluate both the single components of the system and the system as a whole, introducing an evaluation metric that allows for a fair comparison in highly cluttered scenes taking into account the difficulty of the clutter. A website dedicated to the benchmark containing information on the different tasks, maintaining the leaderboards, and serving as a contact point for the community is also provided

Variable autonomy assignment algorithms for human-robot interactions.

As robotic agents become increasingly present in human environments, task completion rates during human-robot interaction has grown into an increasingly important topic of research. Safe collaborative robots executing tasks under human supervision often augment their perception and planning capabilities through traded or shared control schemes. However, such systems are often proscribed only at the most abstract level, with the meticulous details of implementation left to the designer\u27s prerogative. Without a rigorous structure for implementing controls, the work of design is frequently left to ad hoc mechanism with only bespoke guarantees of systematic efficacy, if any such proof is forthcoming at all. Herein, I present two quantitatively defined models for implementing sliding-scale variable autonomy, in which levels of autonomy are determined by the relative efficacy of autonomous subroutines. I experimentally test the resulting Variable Autonomy Planning (VAP) algorithm and against a traditional traded control scheme in a pick-and-place task, and apply the Variable Autonomy Tasking algorithm to the implementation of a robot performing a complex sanitation task in real-world environs. Results show that prioritizing autonomy levels with higher success rates, as encoded into VAP, allows users to effectively and intuitively select optimal autonomy levels for efficient task completion. Further, the Pareto optimal design structure of the VAP+ algorithm allows for significant performance improvements to be made through intervention planning based on systematic input determining failure probabilities through sensorized measurements. This thesis describes the design, analysis, and implementation of these two algorithms, with a particular focus on the VAP+ algorithm. The core conceit is that they are methods for rigorously defining locally optimal plans for traded control being shared between a human and one or more autonomous processes. It is derived from an earlier algorithmic model, the VAP algorithm, developed to address the issue of rigorous, repeatable assignment of autonomy levels based on system data which provides guarantees on basis of the failure-rate sorting of paired autonomous and manual subtask achievement systems. Using only probability ranking to define levels of autonomy, the VAP algorithm is able to sort modules into optimizable ordered sets, but is limited to only solving sequential task assignments. By constructing a joint cost metric for the entire plan, and by implementing a back-to-front calculation scheme for this metric, it is possible for the VAP+ algorithm to generate optimal planning solutions which minimize the expected cost, as amortized over time, funds, accuracy, or any metric combination thereof. The algorithm is additionally very efficient, and able to perform on-line assessments of environmental changes to the conditional probabilities associated with plan choices, should a suitable model for determining these probabilities be present. This system, as a paired set of two algorithms and a design augmentation, form the VAP+ algorithm in full

Robot Assisted Object Manipulation for Minimally Invasive Surgery

Robotic systems have an increasingly important role in facilitating minimally invasive surgical treatments. In robot-assisted minimally invasive surgery, surgeons remotely control instruments from a console to perform operations inside the patient. However, despite the advanced technological status of surgical robots, fully autonomous systems, with decision-making capabilities, are not yet available. In 2017, a structure to classify the research efforts toward autonomy achievable with surgical robots was proposed by Yang et al. Six different levels were identified: no autonomy, robot assistance, task autonomy, conditional autonomy, high autonomy, and full autonomy. All the commercially available platforms in robot-assisted surgery is still in level 0 (no autonomy). Despite increasing the level of autonomy remains an open challenge, its adoption could potentially introduce multiple benefits, such as decreasing surgeons’ workload and fatigue and pursuing a consistent quality of procedures. Ultimately, allowing the surgeons to interpret the ample and intelligent information from the system will enhance the surgical outcome and positively reflect both on patients and society. Three main aspects are required to introduce automation into surgery: the surgical robot must move with high precision, have motion planning capabilities and understand the surgical scene. Besides these main factors, depending on the type of surgery, there could be other aspects that might play a fundamental role, to name some compliance, stiffness, etc. This thesis addresses three technological challenges encountered when trying to achieve the aforementioned goals, in the specific case of robot-object interaction. First, how to overcome the inaccuracy of cable-driven systems when executing fine and precise movements. Second, planning different tasks in dynamically changing environments. Lastly, how the understanding of a surgical scene can be used to solve more than one manipulation task. To address the first challenge, a control scheme relying on accurate calibration is implemented to execute the pick-up of a surgical needle. Regarding the planning of surgical tasks, two approaches are explored: one is learning from demonstration to pick and place a surgical object, and the second is using a gradient-based approach to trigger a smoother object repositioning phase during intraoperative procedures. Finally, to improve scene understanding, this thesis focuses on developing a simulation environment where multiple tasks can be learned based on the surgical scene and then transferred to the real robot. Experiments proved that automation of the pick and place task of different surgical objects is possible. The robot was successfully able to autonomously pick up a suturing needle, position a surgical device for intraoperative ultrasound scanning and manipulate soft tissue for intraoperative organ retraction. Despite automation of surgical subtasks has been demonstrated in this work, several challenges remain open, such as the capabilities of the generated algorithm to generalise over different environment conditions and different patients

Artificial Intelligence for Digitalization in Agriculture: Considerations for the Development of a Fruit Detection System

The adoption of digital solutions is gradually diffusing also in the realm of agriculture, due to the valuable contributions that innovative technologies can bring to a distressed sector. Among these, the application of Artificial Intelligence based fruit detection systems is receiving increasing interest, given the reliance that many technological agricultural applications have on detection tasks to execute their functions, as well as the usefulness such solutions can have in improving several activities: once they track down fruits on a tree, they are able to provide for a quality analysis of the fruits, thus rendering information over maturity level or presence of diseases, for yield estimates ahead of time or for the implementation of intelligent robots able to automatically collect fruits or perform agrochemicals spraying. Nonetheless, the development of an AI based fruit detection system is a non-trivial process since it requires many accurate and pondered considerations over intricate technological aspects relating to data requirements, feature extraction, existing models, necessary hardware configurations, as well as over the socio-economic context. Through an analysis of these elements based on relevant literature, the present elaborate aims to provide therefore a comprehensive understanding of the broader implications that arise during the conception, design, and integration phases of AI technologies for fruit detection tasks, encouraging the necessity of an holistic perspective for informed decision-making processes that could actually result beneficial for agricultural practices

Ultra high frequency (UHF) radio-frequency identification (RFID) for robot perception and mobile manipulation

Personal robots with autonomy, mobility, and manipulation capabilities have the potential to dramatically improve quality of life for various user populations, such as older adults and individuals with motor impairments. Unfortunately, unstructured environments present many challenges that hinder robot deployment in ordinary homes. This thesis seeks to address some of these challenges through a new robotic sensing modality that leverages a small amount of environmental augmentation in the form of Ultra High Frequency (UHF) Radio-Frequency Identification (RFID) tags. Previous research has demonstrated the utility of infrastructure tags (affixed to walls) for robot localization; in this thesis, we specifically focus on tagging objects. Owing to their low-cost and passive (battery-free) operation, users can apply UHF RFID tags to hundreds of objects throughout their homes. The tags provide two valuable properties for robots: a unique identifier and receive signal strength indicator (RSSI, the strength of a tag's response). This thesis explores robot behaviors and radio frequency perception techniques using robot-mounted UHF RFID readers that enable a robot to efficiently discover, locate, and interact with UHF RFID tags applied to objects and people of interest. The behaviors and algorithms explicitly rely on the robot's mobility and manipulation capabilities to provide multiple opportunistic views of the complex electromagnetic landscape inside a home environment. The electromagnetic properties of RFID tags change when applied to common household objects. Objects can have varied material properties, can be placed in diverse orientations, and be relocated to completely new environments. We present a new class of optimization-based techniques for RFID sensing that are robust to the variation in tag performance caused by these complexities. We discuss a hybrid global-local search algorithm where a robot employing long-range directional antennas searches for tagged objects by maximizing expected RSSI measurements; that is, the robot attempts to position itself (1) near a desired tagged object and (2) oriented towards it. The robot first performs a sparse, global RFID search to locate a pose in the neighborhood of the tagged object, followed by a series of local search behaviors (bearing estimation and RFID servoing) to refine the robot's state within the local basin of attraction. We report on RFID search experiments performed in Georgia Tech's Aware Home (a real home). Our optimization-based approach yields superior performance compared to state of the art tag localization algorithms, does not require RF sensor models, is easy to implement, and generalizes to other short-range RFID sensor systems embedded in a robot's end effector. We demonstrate proof of concept applications, such as medication delivery and multi-sensor fusion, using these techniques. Through our experimental results, we show that UHF RFID is a complementary sensing modality that can assist robots in unstructured human environments.PhDCommittee Chair: Kemp, Charles C.; Committee Member: Abowd, Gregory; Committee Member: Howard, Ayanna; Committee Member: Ingram, Mary Ann; Committee Member: Reynolds, Matt; Committee Member: Tentzeris, Emmanoui

Automated manufacturing of smart tunnel segments

Tunnels, essential infrastructures, require regular inspections and maintenance to ensure their prolonged service life. While conventional methods heavily rely on expert human manpower, modern tunnel structural monitoring techniques, such as sensor-based Structural Health Monitoring (SHM), are increasingly utilized in both existing and newly constructed tunnels. Despite providing valuable insights into post-construction structural behaviour, these methods often overlook the behaviour of individual precast elements, such as tunnel segments, before their installation. This thesis explores the concept of smart tunnel segments instrumented by robotic means to address this gap. In this project lab-scale tunnel segments were instrumented using a 6-axis robotic arm making them smart enabling their properties to be tracked from manufacturing through the operational phase of the tunnel. The research involves a comprehensive review of current tunnel instrumentation practices, identifying structural strains as the most monitored parameters. Vibrating Wire Strain Gauges (VWSGs) were identified as the most suitable sensors for this application due to their compatibility with a modular system and superior long-term properties, especially when embedded in concrete. Furthermore, the study identifies untapped potential in fully automated precast factories and proposes repurposing certain features of industrial robots to deploy VWSGs nodes via robotic pick-and-place. Through a novel evaluation framework, the research demonstrates the effectiveness of automated sensor deployment by robots. This includes the robotic installation of a pair of embedded VWSGs in lab-scale tunnel segments, thereby rendering them "smart," and subjecting them to repetitive flexural loadings to evaluate their performance and accuracy. The calculated strain transfer exhibits consistent and repeatable behaviour across segments. Finally, the thesis outlines the economic justification for smart segments, which outperform traditional on-site wired and wireless alternatives, thereby contributing to a more comprehensive and cost-effective tunnel maintenance strategyTunnels, essential infrastructures, require regular inspections and maintenance to ensure their prolonged service life. While conventional methods heavily rely on expert human manpower, modern tunnel structural monitoring techniques, such as sensor-based Structural Health Monitoring (SHM), are increasingly utilized in both existing and newly constructed tunnels. Despite providing valuable insights into post-construction structural behaviour, these methods often overlook the behaviour of individual precast elements, such as tunnel segments, before their installation. This thesis explores the concept of smart tunnel segments instrumented by robotic means to address this gap. In this project lab-scale tunnel segments were instrumented using a 6-axis robotic arm making them smart enabling their properties to be tracked from manufacturing through the operational phase of the tunnel. The research involves a comprehensive review of current tunnel instrumentation practices, identifying structural strains as the most monitored parameters. Vibrating Wire Strain Gauges (VWSGs) were identified as the most suitable sensors for this application due to their compatibility with a modular system and superior long-term properties, especially when embedded in concrete. Furthermore, the study identifies untapped potential in fully automated precast factories and proposes repurposing certain features of industrial robots to deploy VWSGs nodes via robotic pick-and-place. Through a novel evaluation framework, the research demonstrates the effectiveness of automated sensor deployment by robots. This includes the robotic installation of a pair of embedded VWSGs in lab-scale tunnel segments, thereby rendering them "smart," and subjecting them to repetitive flexural loadings to evaluate their performance and accuracy. The calculated strain transfer exhibits consistent and repeatable behaviour across segments. Finally, the thesis outlines the economic justification for smart segments, which outperform traditional on-site wired and wireless alternatives, thereby contributing to a more comprehensive and cost-effective tunnel maintenance strateg