Vision Based Automatic Calibration of Microrobotic System

During the last decade, the advancement of microrobotics has provided a powerful tool for micromanipulation in various fields including living cell manipulation, MEMS/MOEMS assembly, and micro-/nanoscale material characterization. Several dexterous micromanipulation systems have been developed and demonstrated. Nowadays, the research on micromanipulation has shifted the scope from the conceptual system development to the industrial applications. Consequently, the future development of this field lies on the industrial applicability of systems that aims to convert the micromanipulation technique to the mass manufacturing process. In order to achieve this goal, the automatic microrobotic system, as the core in the process chain, plays a significant role. This thesis focuses on the calibration procedure of the positioning control, which is one of the fundamental issues during the automatic microrobotic system development. A novel vision based procedure for three dimensional (3D) calibrations of micromanipulators is proposed. Two major issues in the proposed calibration approach - vision system calibration and manipulator kinematic calibration - are investigated in details in this thesis. For the stereo vision measurement system, the calibration principle and algorithm are presented. Additionally, the manipulator kinematic calibration is carried out in four steps: kinematic modeling, data acquisition, parameter estimation, and compensation implementation. The procedures are presented with two typical models: the matrix model and the polynomial model. Finally, verification and evaluation experiments are conducted on the microrobotic fiber characterization platform in the Micro- and Nano Systems Research Group (MST) at Tampere University of Technology. The results demonstrate that the proposed calibration models are able to reduce the prediction error below 2.59 micrometers. With those models, the pose error, compensated by the feed-forward compensator, can be reduced to be smaller than 5 µm. The proposed approach also demonstrates the feasibility in calibrating the decoupled motions, by reducing the undesired movement from 28 µm to 8 µm (For 4800 µm desired movement)

Workshop on "Control issues in the micro / nano - world".

International audienceDuring the last decade, the need of systems with micro/nanometers accuracy and fast dynamics has been growing rapidly. Such systems occur in applications including 1) micromanipulation of biological cells, 2) micrassembly of MEMS/MOEMS, 3) micro/nanosensors for environmental monitoring, 4) nanometer resolution imaging and metrology (AFM and SEM). The scale and requirement of such systems present a number of challenges to the control system design that will be addressed in this workshop. Working in the micro/nano-world involves displacements from nanometers to tens of microns. Because of this precision requirement, environmental conditions such as temperature, humidity, vibration, could generate noise and disturbance that are in the same range as the displacements of interest. The so-called smart materials, e.g., piezoceramics, magnetostrictive, shape memory, electroactive polymer, have been used for actuation or sensing in the micro/nano-world. They allow high resolution positioning as compared to hinges based systems. However, these materials exhibit hysteresis nonlinearity, and in the case of piezoelectric materials, drifts (called creep) in response to constant inputs In the case of oscillating micro/nano-structures (cantilever, tube), these nonlinearities and vibrations strongly decrease their performances. Many MEMS and NEMS applications involve gripping, feeding, or sorting, operations, where sensor feedback is necessary for their execution. Sensors that are readily available, e.g., interferometer, triangulation laser, and machine vision, are bulky and expensive. Sensors that are compact in size and convenient for packaging, e.g., strain gage, piezoceramic charge sensor, etc., have limited performance or robustness. To account for these difficulties, new control oriented techniques are emerging, such as[d the combination of two or more ‘packageable' sensors , the use of feedforward control technique which does not require sensors, and the use of robust controllers which account the sensor characteristics. The aim of this workshop is to provide a forum for specialists to present and overview the different approaches of control system design for the micro/nano-world and to initiate collaborations and joint projects

Novel estimation and control techniques in micromanipulation using vision and force feedback

With the recent advances in the fields of micro and nanotechnology, there has been growing interest for complex micromanipulation and microassembly strategies. Despite the fact that many commercially available micro devices such as the key components in automobile airbags, ink-jet printers and projection display systems are currently produced in a batch technique with little assembly, many other products such as read/write heads for hard disks and fiber optics assemblies require flexible precision assemblies. Furthermore, many biological micromanipulations such as invitro-fertilization, cell characterization and treatment rely on the ability of human operators. Requirement of high-precision, repeatable and financially viable operations in these tasks has given rise to the elimination of direct human involvement, and autonomy in micromanipulation and microassembly. In this thesis, a fully automated dexterous micromanipulation strategy based on vision and force feedback is developed. More specifically, a robust vision based control architecture is proposed and implemented to compensate errors due to the uncertainties about the position, behavior and shape of the microobjects to be manipulated. Moreover, novel estimators are designed to identify the system and to characterize the mechanical properties of the biological structures through a synthesis of concepts from the computer vision, estimation and control theory. Estimated mechanical parameters are utilized to reconstruct the imposed force on a biomembrane and to provide the adequate information to control the position, velocity and acceleration of the probe without damaging the cell/tissue during an injection task

Vision Based Automatic Calibration of Microrobotic System

During the last decade, the advancement of microrobotics has provided a powerful tool for micromanipulation in various fields including living cell manipulation, MEMS/MOEMS assembly, and micro-/nanoscale material characterization. Several dexterous micromanipulation systems have been developed and demonstrated. Nowadays, the research on micromanipulation has shifted the scope from the conceptual system development to the industrial applications. Consequently, the future development of this field lies on the industrial applicability of systems that aims to convert the micromanipulation technique to the mass manufacturing process. In order to achieve this goal, the automatic microrobotic system, as the core in the process chain, plays a significant role. This thesis focuses on the calibration procedure of the positioning control, which is one of the fundamental issues during the automatic microrobotic system development. A novel vision based procedure for three dimensional (3D) calibrations of micromanipulators is proposed. Two major issues in the proposed calibration approach - vision system calibration and manipulator kinematic calibration - are investigated in details in this thesis. For the stereo vision measurement system, the calibration principle and algorithm are presented. Additionally, the manipulator kinematic calibration is carried out in four steps: kinematic modeling, data acquisition, parameter estimation, and compensation implementation. The procedures are presented with two typical models: the matrix model and the polynomial model. Finally, verification and evaluation experiments are conducted on the microrobotic fiber characterization platform in the Micro- and Nano Systems Research Group (MST) at Tampere University of Technology. The results demonstrate that the proposed calibration models are able to reduce the prediction error below 2.59 micrometers. With those models, the pose error, compensated by the feed-forward compensator, can be reduced to be smaller than 5 µm. The proposed approach also demonstrates the feasibility in calibrating the decoupled motions, by reducing the undesired movement from 28 µm to 8 µm (For 4800 µm desired movement)

Challenges in flexible microsystem manufacturing : fabrication, robotic assembly, control, and packaging.

Microsystems have been investigated with renewed interest for the last three decades because of the emerging development of microelectromechanical system (MEMS) technology and the advancement of nanotechnology. The applications of microrobots and distributed sensors have the potential to revolutionize micro and nano manufacturing and have other important health applications for drug delivery and minimal invasive surgery. A class of microrobots studied in this thesis, such as the Solid Articulated Four Axis Microrobot (sAFAM) are driven by MEMS actuators, transmissions, and end-effectors realized by 3-Dimensional MEMS assembly. Another class of microrobots studied here, like those competing in the annual IEEE Mobile Microrobot Challenge event (MMC) are untethered and driven by external fields, such as magnetic fields generated by a focused permanent magnet. A third class of microsystems studied in this thesis includes distributed MEMS pressure sensors for robotic skin applications that are manufactured in the cleanroom and packaged in our lab. In this thesis, we discuss typical challenges associated with the fabrication, robotic assembly and packaging of these microsystems. For sAFAM we discuss challenges arising from pick and place manipulation under microscopic closed-loop control, as well as bonding and attachment of silicon MEMS microparts. For MMC, we discuss challenges arising from cooperative manipulation of microparts that advance the capabilities of magnetic micro-agents. Custom microrobotic hardware configured and demonstrated during this research (such as the NeXus microassembly station) include micro-positioners, microscopes, and controllers driven via LabVIEW. Finally, we also discuss challenges arising in distributed sensor manufacturing. We describe sensor fabrication steps using clean-room techniques on Kapton flexible substrates, and present results of lamination, interconnection and testing of such sensors are presented

Modeling and experimental validation of a parallel microrobot for biomanipulation

The main purpose of this project is the development of a commercial micropositioner's (SmarPod 115.25, SmarAct GmbH) geometrical model. SmarPod is characterized by parallel kinematics and is employed for precise and accurate sample's positioning under SEM microscope, being vacuum-compatible, for various applications. Geometrical modeling represents the preliminar step to fully understand, and possibly improve, robot's closed loop behaviour in terms of task's quality precision, when enterprises does not provide sufficient documentation. The robotic system, in fact, represents in this case a "black box" from which it's possible to extract information. This step is essential in order to improve, consequently, the reliability of bio-microsystem manipulation and characterization. Disposing of a detailed microrobot's model becomes essential to deal with the typical lack of sensing at microscale, as it allows a 3D precise and adequate reconstruction, realized through proper softwares, of the manipulation set-up. The roles of Virtual Reality (VR) and of simulations, carried out, in this case, in Blender environment, are asserted as well as an essential helping tool in mycrosystem's task planning. Blender is a professional free and open-source 3D computer graphics software and it is proven to be a basic instrument to validate microrobot's model, even to simplify it in case of complex system's geometries

Image-Guided Robot-Assisted Techniques with Applications in Minimally Invasive Therapy and Cell Biology

There are several situations where tasks can be performed better robotically rather than manually. Among these are situations (a) where high accuracy and robustness are required, (b) where difficult or hazardous working conditions exist, and (c) where very large or very small motions or forces are involved. Recent advances in technology have resulted in smaller size robots with higher accuracy and reliability. As a result, robotics is fi nding more and more applications in Biomedical Engineering. Medical Robotics and Cell Micro-Manipulation are two of these applications involving interaction with delicate living organs at very di fferent scales.Availability of a wide range of imaging modalities from ultrasound and X-ray fluoroscopy to high magni cation optical microscopes, makes it possible to use imaging as a powerful means to guide and control robot manipulators. This thesis includes three parts focusing on three applications of Image-Guided Robotics in biomedical engineering, including: Vascular Catheterization: a robotic system was developed to insert a catheter through the vasculature and guide it to a desired point via visual servoing. The system provides shared control with the operator to perform a task semi-automatically or through master-slave control. The system provides control of a catheter tip with high accuracy while reducing X-ray exposure to the clinicians and providing a more ergonomic situation for the cardiologists. Cardiac Catheterization: a master-slave robotic system was developed to perform accurate control of a steerable catheter to touch and ablate faulty regions on the inner walls of a beating heart in order to treat arrhythmia. The system facilitates touching and making contact with a target point in a beating heart chamber through master-slave control with coordinated visual feedback. Live Neuron Micro-Manipulation: a microscope image-guided robotic system was developed to provide shared control over multiple micro-manipulators to touch cell membranes in order to perform patch clamp electrophysiology. Image-guided robot-assisted techniques with master-slave control were implemented for each case to provide shared control between a human operator and a robot. The results show increased accuracy and reduced operation time in all three cases

Design, evaluation, and control of nexus: a multiscale additive manufacturing platform with integrated 3D printing and robotic assembly.

Additive manufacturing (AM) technology is an emerging approach to creating three-dimensional (3D) objects and has seen numerous applications in medical implants, transportation, aerospace, energy, consumer products, etc. Compared with manufacturing by forming and machining, additive manufacturing techniques provide more rapid, economical, efficient, reliable, and complex manufacturing processes. However, additive manufacturing also has limitations on print strength and dimensional tolerance, while traditional additive manufacturing hardware platforms for 3D printing have limited flexibility. In particular, part geometry and materials are limited to most 3D printing hardware. In addition, for multiscale and complex products, samples must be printed, fabricated, and transferred among different additive manufacturing platforms in different locations, which leads to high cost, long process time, and low yield of products. This thesis investigates methods to design, evaluate, and control the NeXus, which is a novel custom robotic platform for multiscale additive manufacturing with integrated 3D printing and robotic assembly. NeXus can be used to prototype miniature devices and systems, such as wearable MEMS sensor fabrics, microrobots for wafer-scale microfactories, tactile robot skins, next generation energy storage (solar cells), nanostructure plasmonic devices, and biosensors. The NeXus has the flexibility to fixture, position, transport, and assemble components across a wide spectrum of length scales (Macro-Meso-Micro-Nano, 1m to 100nm) and provides unparalleled additive process capabilities such as 3D printing through both aerosol jetting and ultrasonic bonding and forming, thin-film photonic sintering, fiber loom weaving, and in-situ Micro-Electro-Mechanical System (MEMS) packaging and interconnect formation. The NeXus system has a footprint of around 4m x 3.5m x 2.4m (X-Y-Z) and includes two industrial robotic arms, precision positioners, multiple manipulation tools, and additive manufacturing processes and packaging capabilities. The design of the NeXus platform adopted the Lean Robotic Micromanufacturing (LRM) design principles and simulation tools to mitigate development risks. The NeXus has more than 50 degrees of freedom (DOF) from different instruments, precise evaluation of the custom robots and positioners is indispensable before employing them in complex and multiscale applications. The integration and control of multi-functional instruments is also a challenge in the NeXus system due to different communication protocols and compatibility. Thus, the NeXus system is controlled by National Instruments (NI) LabVIEW real-time operating system (RTOS) with NI PXI controller and a LabVIEW State Machine User Interface (SMUI) and was programmed considering the synchronization of various instruments and sequencing of additive manufacturing processes for different tasks. The operation sequences of each robot along with relevant tools must be organized in safe mode to avoid crashes and damage to tools during robots’ motions. This thesis also describes two demonstrators that are realized by the NeXus system in detail: skin tactile sensor arrays and electronic textiles. The fabrication process of the skin tactile sensor uses the automated manufacturing line in the NeXus with pattern design, precise calibration, synchronization of an Aerosol Jet printer, and a custom positioner. The fabrication process for electronic textiles is a combination of MEMS fabrication techniques in the cleanroom and the collaboration of multiple NeXus robots including two industrial robotic arms and a custom high-precision positioner for the deterministic alignment process

単一運動性微生物の刺激応答計測のためのマイクロロボティックプラットホーム

九州工業大学博士学位論文 学位記番号:生工博甲第355号 学位授与年月日:令和元年9月20日1 Introduction|2 Observation Platform|3 Stimulation Platform|4 Application to Actual Motile Microorganisms|5 Conclusion九州工業大学令和元年