Computer Vision Measurements for Automated Microrobotic Paper Fiber Studies

The mechanical characterization of paper fibers and paper fiber bonds determines the key parameters affecting the mechanical properties of paper. Although bulk measurements from test sheets can give average values, they do not yield any real fiber-level data. The current, state-of-the-art methods for fiberlevel measurements are slow and laborious, requiring delicate manual handling of microscopic samples. There are commercial microrobotic actuators that allow automated or tele-operated manipulation of microscopic objects such as fibers, but it is challenging to acquire the data needed to guide such demanding manipulation. This thesis presents a solution to the illumination problem and computer vision algorithms for obtaining the required data. The solutions are designed for a microrobotic platform that comprises actuators for manipulating the fibers and one or two microscope cameras for visual feedback.The algorithms have been developed both for wet fibers, which can be treated as 2D objects, and for dry fibers and fiber bonds, which are treated as 3D objects. The major innovations in the algorithms are the rules for the micromanipulation of the curly fiber strands and the automated 3D measurements of microscale objects with random geometries. The solutions are validated by imaging and manipulation experiments with wet and dry paper fibers and dry paper fiber bonds. In the imaging experiments, the results are compared with the reference data obtained either from an experienced human or another imaging device. The results show that these solutions provide morphological data about the fibers which is accurate and precise enough to enable automated fiber manipulation. Although this thesis is focused on the manipulation of paper fibers and paper fiber bonds, both the illumination solution and the computer vision algorithms are applicable to other types of fibrous materials

Measurement of Z-Directional Individual Fibre-Fibre Bond Strength and Microfibril Angle Using Microrobotics

The use of microrobotics in high throughput and precise characterization of objects at microscale has been noticeably increased during recent years. Microrobotics has provided a significant added value to multiple realms e.g. biomedical research, bio-based industry, microassembly of miniature products, etc. Recently, the use of microrobotic technology in paper industry has been also commenced for measuring properties at the single fibre level. There is a large interest in the measurement of different loading modes of individual fibre-fibre bonds in pulp and paper/board industry. Among the four different modes of loading, it would be desirable for papermaking companies and paper converting companies to obtain the Z-directional strength of pulp and paper. Indeed, the Z-directional properties affect compressive properties, and accordingly the performance of structural paperboard products. Several methods have been developed to measure the Z-directional strength at a handsheet level; however, there is not any reported device capable of the Z-directional fibre-fibre bond strength measurement at a fibre level. This thesis work presents a novel method for the experimental evaluation of the Z-directional bond strength using microrobotics and a Polyvinylidene fluoride (PVDF) film microforce sensor. Due to the special dynamics of PVDF microforce sensors, the effect of the deformation rate on the performance of the sensor is studied. The Z-directional fibre-fibre bond strength experiments have been performed successfully for unrefined and refined bleached softwood Kraft pulp fibres. Besides, paper scientists are interested in microfibril angle changes during and after application of the Z-directional force. Indeed, there is interest in simultaneous measurement of microfibril angle and mechanical properties such as Z-directional bond strength. To address this need, a microfibril angle measurements system based on microscopic transmission ellipsometry is developed and integrated to the microrobotic platform. The results from both Z-directional bond strength and microfibril angle measurement are promising. In summary, the first concept for simultaneous measurement of microfibril angle and mechanical properties such as Z-directional bond strength at the individual fibre level is developed during this thesis work which has a high practical impact on the fibre characterization research field

Design and implementation of rotational degrees of freedom into microrobotics platform

The strength of the individual paper fiber bonds (IPFB) is the key parameter which determines the mechanical quality of paper hand sheets. Currently, most of the strength measurements are still done on hand-sheet level because of the absence of high throughput IPFB strength measurement tools. Micro and Nanosystems research group of Tampere University of Technology recognized the demand for an IPFB characterization system and built a microrobotics platform. However, the current configuration of the platform is not able to rotate the microgripper which limits the measurements such as Z-directional bond breaking and shear mode bond breaking. Moreover, this configuration is not capable of dealing with twisted fibers. This thesis addresses these problems and introduces addition of two more degrees of rotation to the current platform. This modification of microrobotic platform will enable the bond strength measurement of IPFBs in desired pure modes which will enhance the paper fiber scientist`s understanding of IPFBs breaking process. Bond strength measurement with the current platform provides data that is a combination of normal and shear forces which is not desired. After the modifications provided by this thesis, the microrobotic platform will be able to separate the shear force and the normal force during shear mode bond breaking. In the Z-directional bond strength measurement, it is essential to know which fiber is on the top whereas the platform does not fulfill this requirement. The rotation of the microgripper and thus, the fibers will reveal the orientation of the IPFBs. Moreover, the rotation of the microgripper enables the user to untwist the twisted fibers by rotating from one end while the other end is fixed with another microgripper. Forward kinematics of the modified system is calculated through Matlab and compared with the real system. The errors between the ideal system and real system are reduced significantly by modifying the parameters in the overall transformation matrix which ensures that the modified microrobotic platform is now capable of solving all three problems discussed above. Maximum errors are decreased 90.65% (from 107 micrometers to 10 micrometers) at the X-axis, 82.47% (from 97 micrometers to 17 micrometers) at the Y-axis and 87.17% (from 195 micrometers to 25 micrometers) at the Z-axis

Vision Based Automatic Calibration of Microrobotic System

During the last decade, the advancement of microrobotics has provided a powerful tool for micromanipulation in various fields including living cell manipulation, MEMS/MOEMS assembly, and micro-/nanoscale material characterization. Several dexterous micromanipulation systems have been developed and demonstrated. Nowadays, the research on micromanipulation has shifted the scope from the conceptual system development to the industrial applications. Consequently, the future development of this field lies on the industrial applicability of systems that aims to convert the micromanipulation technique to the mass manufacturing process. In order to achieve this goal, the automatic microrobotic system, as the core in the process chain, plays a significant role. This thesis focuses on the calibration procedure of the positioning control, which is one of the fundamental issues during the automatic microrobotic system development. A novel vision based procedure for three dimensional (3D) calibrations of micromanipulators is proposed. Two major issues in the proposed calibration approach - vision system calibration and manipulator kinematic calibration - are investigated in details in this thesis. For the stereo vision measurement system, the calibration principle and algorithm are presented. Additionally, the manipulator kinematic calibration is carried out in four steps: kinematic modeling, data acquisition, parameter estimation, and compensation implementation. The procedures are presented with two typical models: the matrix model and the polynomial model. Finally, verification and evaluation experiments are conducted on the microrobotic fiber characterization platform in the Micro- and Nano Systems Research Group (MST) at Tampere University of Technology. The results demonstrate that the proposed calibration models are able to reduce the prediction error below 2.59 micrometers. With those models, the pose error, compensated by the feed-forward compensator, can be reduced to be smaller than 5 µm. The proposed approach also demonstrates the feasibility in calibrating the decoupled motions, by reducing the undesired movement from 28 µm to 8 µm (For 4800 µm desired movement)

Vision Based Automatic Calibration of Microrobotic System

During the last decade, the advancement of microrobotics has provided a powerful tool for micromanipulation in various fields including living cell manipulation, MEMS/MOEMS assembly, and micro-/nanoscale material characterization. Several dexterous micromanipulation systems have been developed and demonstrated. Nowadays, the research on micromanipulation has shifted the scope from the conceptual system development to the industrial applications. Consequently, the future development of this field lies on the industrial applicability of systems that aims to convert the micromanipulation technique to the mass manufacturing process. In order to achieve this goal, the automatic microrobotic system, as the core in the process chain, plays a significant role. This thesis focuses on the calibration procedure of the positioning control, which is one of the fundamental issues during the automatic microrobotic system development. A novel vision based procedure for three dimensional (3D) calibrations of micromanipulators is proposed. Two major issues in the proposed calibration approach - vision system calibration and manipulator kinematic calibration - are investigated in details in this thesis. For the stereo vision measurement system, the calibration principle and algorithm are presented. Additionally, the manipulator kinematic calibration is carried out in four steps: kinematic modeling, data acquisition, parameter estimation, and compensation implementation. The procedures are presented with two typical models: the matrix model and the polynomial model. Finally, verification and evaluation experiments are conducted on the microrobotic fiber characterization platform in the Micro- and Nano Systems Research Group (MST) at Tampere University of Technology. The results demonstrate that the proposed calibration models are able to reduce the prediction error below 2.59 micrometers. With those models, the pose error, compensated by the feed-forward compensator, can be reduced to be smaller than 5 µm. The proposed approach also demonstrates the feasibility in calibrating the decoupled motions, by reducing the undesired movement from 28 µm to 8 µm (For 4800 µm desired movement)

Micro/nanoscale magnetic robots for biomedical applications

Magnetic small-scale robots are devices of great potential for the biomedical field because of the several benefits of this method of actuation. Recent work on the development of these devices has seen tremendous innovation and refinement toward ​improved performance for potential clinical applications. This review briefly details recent advancements in small-scale robots used for biomedical applications, covering their design, fabrication, applications, and demonstration of ability, and identifies the gap in studies and the difficulties that have persisted in the optimization of the use of these devices. In addition, alternative biomedical applications are also suggested for some of the technologies that show potential for other functions. This study concludes that although the field of small-scale robot research is highly innovative ​there is need for more concerted efforts to improve functionality and reliability of these devices particularly in clinical applications. Finally, further suggestions are made toward ​the achievement of commercialization for these devices

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications

Robotic micromanipulation for microassembly : modelling by sequencial function chart and achievement by multiple scale visual servoings.

International audienceThe paper investigates robotic assembly by focusing on the manipulation of microparts. This task is formalized through the notion of basic tasks which are organized in a logical sequence represented by a function chart and interpreted as the model of the behavior of the experimental setup. The latter includes a robotic system, a gripping system, an imaging system, and a clean environment. The imaging system is a photon videomicroscope able to work at multiple scales. It is modelled by a linear projective model where the relation between the scale factor and the magnification or zoom is explicitly established. So, the usual visual control law is modified in order to take into account this relation. The manipulation of some silicon microparts (400 μm×400 μm×100 μm) by means of a distributed robotic system (xyθ system, ϕz system), a two-finger gripping system and a controllable zoom and focus videomicroscope shows the relevance of the concepts. The 30 % of failure rate comes mainly from the physical phenomena (electrostatic and capillary forces) instead of the accuracy of control or the occultations of microparts

Microdevices and Microsystems for Cell Manipulation

Microfabricated devices and systems capable of micromanipulation are well-suited for the manipulation of cells. These technologies are capable of a variety of functions, including cell trapping, cell sorting, cell culturing, and cell surgery, often at single-cell or sub-cellular resolution. These functionalities are achieved through a variety of mechanisms, including mechanical, electrical, magnetic, optical, and thermal forces. The operations that these microdevices and microsystems enable are relevant to many areas of biomedical research, including tissue engineering, cellular therapeutics, drug discovery, and diagnostics. This Special Issue will highlight recent advances in the field of cellular manipulation. Technologies capable of parallel single-cell manipulation are of special interest

Design, evaluation, and control of nexus: a multiscale additive manufacturing platform with integrated 3D printing and robotic assembly.

Additive manufacturing (AM) technology is an emerging approach to creating three-dimensional (3D) objects and has seen numerous applications in medical implants, transportation, aerospace, energy, consumer products, etc. Compared with manufacturing by forming and machining, additive manufacturing techniques provide more rapid, economical, efficient, reliable, and complex manufacturing processes. However, additive manufacturing also has limitations on print strength and dimensional tolerance, while traditional additive manufacturing hardware platforms for 3D printing have limited flexibility. In particular, part geometry and materials are limited to most 3D printing hardware. In addition, for multiscale and complex products, samples must be printed, fabricated, and transferred among different additive manufacturing platforms in different locations, which leads to high cost, long process time, and low yield of products. This thesis investigates methods to design, evaluate, and control the NeXus, which is a novel custom robotic platform for multiscale additive manufacturing with integrated 3D printing and robotic assembly. NeXus can be used to prototype miniature devices and systems, such as wearable MEMS sensor fabrics, microrobots for wafer-scale microfactories, tactile robot skins, next generation energy storage (solar cells), nanostructure plasmonic devices, and biosensors. The NeXus has the flexibility to fixture, position, transport, and assemble components across a wide spectrum of length scales (Macro-Meso-Micro-Nano, 1m to 100nm) and provides unparalleled additive process capabilities such as 3D printing through both aerosol jetting and ultrasonic bonding and forming, thin-film photonic sintering, fiber loom weaving, and in-situ Micro-Electro-Mechanical System (MEMS) packaging and interconnect formation. The NeXus system has a footprint of around 4m x 3.5m x 2.4m (X-Y-Z) and includes two industrial robotic arms, precision positioners, multiple manipulation tools, and additive manufacturing processes and packaging capabilities. The design of the NeXus platform adopted the Lean Robotic Micromanufacturing (LRM) design principles and simulation tools to mitigate development risks. The NeXus has more than 50 degrees of freedom (DOF) from different instruments, precise evaluation of the custom robots and positioners is indispensable before employing them in complex and multiscale applications. The integration and control of multi-functional instruments is also a challenge in the NeXus system due to different communication protocols and compatibility. Thus, the NeXus system is controlled by National Instruments (NI) LabVIEW real-time operating system (RTOS) with NI PXI controller and a LabVIEW State Machine User Interface (SMUI) and was programmed considering the synchronization of various instruments and sequencing of additive manufacturing processes for different tasks. The operation sequences of each robot along with relevant tools must be organized in safe mode to avoid crashes and damage to tools during robots’ motions. This thesis also describes two demonstrators that are realized by the NeXus system in detail: skin tactile sensor arrays and electronic textiles. The fabrication process of the skin tactile sensor uses the automated manufacturing line in the NeXus with pattern design, precise calibration, synchronization of an Aerosol Jet printer, and a custom positioner. The fabrication process for electronic textiles is a combination of MEMS fabrication techniques in the cleanroom and the collaboration of multiple NeXus robots including two industrial robotic arms and a custom high-precision positioner for the deterministic alignment process