A machine vision system for quality grading of painted slates

The major aim of this chapter is to detail the technology associated with a novel industrial inspection system that is able to robustly identify the visual defects present on the surface of painted slates. The development of a real-time automated slate inspection system proved to be a challenging task since the surface of the slate is painted with glossy dark colours, the slate is characterised by depth profile non-uniformities and it is transported at the inspection line via high-speed conveyors. In order to implement an industrial compliant system, in our design we had to devise a large number of novel solutions including the development of a full customised illumination set-up and the development of flexible image-processing procedures that can accommodate the large spectrum of visual defects that can be present on the slate surface and the vibrations generated by the slate transport system. The developed machine vision system has been subjected to a thorough robustness evaluation and the reported experimental results indicate that the proposed solution can be used to replace the manual procedure that is currently used to grade the painted slates in manufacturing environments

Doctor of Philosophy

dissertationThe optimization of novel stretchable fingernail sensors for detecting fingertip touch force direction is introduced. The fingernail sensor uses optical reflectance photoplethysmography to measure the change in blood perfusion in the fingernail bed when the finger pad touches a surface with various forces. This "fingernail sensing" technique involves mounting an array of LEDs (Light Emitting Diodes) and photodetectors on the fingernail surface to detect changes in the reflection intensity as a function of applied force. The intensity changes correspond to changes in blood volume underneath the fingernail and allow for fingertip force detection without haptic obstruction, which has several applications in the area of human-machine interaction. This dissertation experimentally determines the optimal optical parameters for the transmittance of light through the human fingernail bed. Specifically, the effect of varying the wavelength and optical path length on light transmittance through the nail bed are thoroughly investigated. Light transmittance through the human fingernail is optimized when using green light (525nm) and when placing optoelectronic pairs as close together as possible. The optimal locations of the optoelectronic devices are predicted by introducing an optical model that describes light transmittance between an LED and a photodiode in the fingernail area based on optical experimentation. A reduced configuration is derived from the optimal optoelectronic locations in order to facilitate iv the fabrication of the optimized fingernail sensor without significantly compromising the recognition accuracy. This results in an overall force direction recognition accuracy of 95%. Using novel fabrication techniques, we successfully build a stretchable fingernail sensor prototype, which fully conforms to the two-dimensional fingernail surface and is independent of its geometry. Namely, we overcome the challenges of patterning conductive lines on a stretchable substrate, and embedding rigid optical components in a stretchable platform while maintaining electrical conductivity. A finite element analysis is conducted to optimize the electrical contact resistance between the optoelectronic components and underlying stretchable conductors, as a function of the bending curvature and substrate thickness. The functionality of the stretchable sensor is tested in relation to the design parameters. Finally, applications and potential impacts of this work are discussed

X-ray based machine vision system for distal locking of intramedullary nails

In surgical procedures for femoral shaft fracture treatment, current techniques for locking the distal end of intramedullary nails, using two screws, rely heavily on the use of two-dimensional X-ray images to guide three-dimensional bone drilling processes. Therefore, a large number of X-ray images are required, as the surgeon uses his/her skills and experience to locate the distal hole axes on the intramedullary nail. The long-term effects of X-ray radiation and their relation to different types of cancer still remain uncertain. Therefore, there is a need to develop a surgical technique that can limit the use of X-rays during the distal locking procedure. A Robotic-Assisted Orthopaedic Surgery System has been developed at Loughborough University named Loughborough Orthopaedic Assistant System (LOAS) to assist orthopaedic surgeons during distal-locking of intramedullary nails. It uses a calibration frame and a C-arm X-ray unit. The system simplifies the current approach as it uses only two near-orthogonal X-ray images to determine the drilling trajectory of the distal-locking holes, thereby considerably reducing irradiation to both the surgeon and patient. The LOAS differs from existing computer-assisted orthopaedic surgery systems, as it eliminates the need for optical tracking equipment which tends to clutter the operating theatre environment and requires care in maintaining the line of sight. Additionally use of optical tracking equipment makes such systems an expensive method for surgical guidance in distal-locking of intramedullary nails. This study is specifically concerned with the improvements of the existing system. [Continues.

Human behavior understanding for worker-centered intelligent manufacturing

“In a worker-centered intelligent manufacturing system, sensing and understanding of the worker’s behavior are the primary tasks, which are essential for automatic performance evaluation & optimization, intelligent training & assistance, and human-robot collaboration. In this study, a worker-centered training & assistant system is proposed for intelligent manufacturing, which is featured with self-awareness and active-guidance. To understand the hand behavior, a method is proposed for complex hand gesture recognition using Convolutional Neural Networks (CNN) with multiview augmentation and inference fusion, from depth images captured by Microsoft Kinect. To sense and understand the worker in a more comprehensive way, a multi-modal approach is proposed for worker activity recognition using Inertial Measurement Unit (IMU) signals obtained from a Myo armband and videos from a visual camera. To automatically learn the importance of different sensors, a novel attention-based approach is proposed to human activity recognition using multiple IMU sensors worn at different body locations. To deploy the developed algorithms to the factory floor, a real-time assembly operation recognition system is proposed with fog computing and transfer learning. The proposed worker-centered training & assistant system has been validated and demonstrated the feasibility and great potential for applying to the manufacturing industry for frontline workers. Our developed approaches have been evaluated: 1) the multi-view approach outperforms the state-of-the-arts on two public benchmark datasets, 2) the multi-modal approach achieves an accuracy of 97% on a worker activity dataset including 6 activities and achieves the best performance on a public dataset, 3) the attention-based method outperforms the state-of-the-art methods on five publicly available datasets, and 4) the developed transfer learning model achieves a real-time recognition accuracy of 95% on a dataset including 10 worker operations”--Abstract, page iv

Analysis and Observations from the First Amazon Picking Challenge

This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge