A Multirobots Teleoperated Platform for Artificial Intelligence Training Data Collection in Minimally Invasive Surgery

Dexterity and perception capabilities of surgical robots may soon be improved by cognitive functions that can support surgeons in decision making and performance monitoring, and enhance the impact of automation within the operating rooms. Nowadays, the basic elements of autonomy in robotic surgery are still not well understood and their mutual interaction is unexplored. Current classification of autonomy encompasses six basic levels: Level 0: no autonomy; Level 1: robot assistance; Level 2: task autonomy; Level 3: conditional autonomy; Level 4: high autonomy. Level 5: full autonomy. The practical meaning of each level and the necessary technologies to move from one level to the next are the subject of intense debate and development. In this paper, we discuss the first outcomes of the European funded project Smart Autonomous Robotic Assistant Surgeon (SARAS). SARAS will develop a cognitive architecture able to make decisions based on pre-operative knowledge and on scene understanding via advanced machine learning algorithms. To reach this ambitious goal that allows us to reach Level 1 and 2, it is of paramount importance to collect reliable data to train the algorithms. We will present the experimental setup to collect the data for a complex surgical procedure (Robotic Assisted Radical Prostatectomy) on very sophisticated manikins (i.e. phantoms of the inflated human abdomen). The SARAS platform allows the main surgeon and the assistant to teleoperate two independent two-arm robots. The data acquired with this platform (videos, kinematics, audio) will be used in our project and will be released (with annotations) for research purposes

Gesture Recognition and Control for Semi-Autonomous Robotic Assistant Surgeons

The next stage for robotics development is to introduce autonomy and cooperation with human agents in tasks that require high levels of precision and/or that exert considerable physical strain. To guarantee the highest possible safety standards, the best approach is to devise a deterministic automaton that performs identically for each operation. Clearly, such approach inevitably fails to adapt itself to changing environments or different human companions. In a surgical scenario, the highest variability happens for the timing of different actions performed within the same phases. This thesis explores the solutions adopted in pursuing automation in robotic minimally-invasive surgeries (R-MIS) and presents a novel cognitive control architecture that uses a multi-modal neural network trained on a cooperative task performed by human surgeons and produces an action segmentation that provides the required timing for actions while maintaining full phase execution control via a deterministic Supervisory Controller and full execution safety by a velocity-constrained Model-Predictive Controller