Computer vision reading on stickers and direct part marking on horticultural products : challenges and possible solutions

Traceability of products from production to the consumer has led to a technological advancement in product identification. There has been development from the use of traditional one-dimensional barcodes (EAN-13, Code 128, etc.) to 2D (two-dimensional) barcodes such as QR (Quick Response) and Data Matrix codes. Over the last two decades there has been an increased use of Radio Frequency Identification (RFID) and Direct Part Marking (DPM) using lasers for product identification in agriculture. However, in agriculture there are still considerable challenges to adopting barcodes, RFID and DPM technologies, unlike in industry where these technologies have been very successful. This study was divided into three main objectives. Firstly, determination of the effect of speed, dirt, moisture and bar width on barcode detection was carried out both in the laboratory and a flower producing company, Brandkamp GmbH. This study developed algorithms for automation and detection of Code 128 barcodes under rough production conditions. Secondly, investigations were carried out on the effect of low laser marking energy on barcode size, print growth, colour and contrast on decoding 2D Data Matrix codes printed directly on apples. Three different apple varieties (Golden Delicious, Kanzi and Red Jonaprince) were marked with various levels of energy and different barcode sizes. Image processing using Halcon 11.0.1 (MvTec) was used to evaluate the markings on the apples. Finally, the third objective was to evaluate both algorithms for 1D and 2D barcodes. According to the results, increasing the speed and angle of inclination of the barcode decreased barcode recognition. Also, increasing the dirt on the surface of the barcode resulted in decreasing the successful detection of those barcodes. However, there was 100% detection of the Code 128 barcode at the company’s production speed (0.15 m/s) with the proposed algorithm. Overall, the results from the company showed that the image-based system has a future prospect for automation in horticultural production systems. It overcomes the problem of using laser barcode readers. The results for apples showed that laser energy, barcode size, print growth, type of product, contrast between the markings and the colour of the products, the inertia of the laser system and the days of storage all singularly or in combination with each other influence the readability of laser Data Matrix codes and implementation on apples. There was poor detection of the Data Matrix code on Kanzi and Red Jonaprince due to the poor contrast between the markings on their skins. The proposed algorithm is currently working successfully on Golden Delicious with 100% detection for 10 days using energy 0.108 J mm-2 and a barcode size of 10 × 10 mm2. This shows that there is a future prospect of not only marking barcodes on apples but also on other agricultural products for real time production

Comparison of manual and automatic barcode detection in rough horticultural production systems

Automation of production in the nurseries of flower producing companies using barcode scanners have been attempted but with little success. Stationary laser barcode scanners which have been used for automation have failed due to the close proximity between the barcode and the scanner, and factors such as speed, angle of inclination of the barcode, damage to the barcode and dirt on the barcode. Furthermore, laser barcode scanners are still being used manually in the nurseries making work laborious and time consuming, thereby leading to reduced productivity. Therefore, an automated image-based barcode detection system to help solve the aforementioned problems was proposed. Experiments were conducted under different situations with clean and artificially soiled Code 128 barcodes in both the laboratory and under real production conditions in a flower producing company. The images were analyzed with a specific algorithm developed with the software tool Halcon. Overall the results from the company showed that the image-based system has a future prospect for automation in the nursery

Geometric, Semantic, and System-Level Scene Understanding for Improved Construction and Operation of the Built Environment

Recent advances in robotics and enabling fields such as computer vision, deep learning, and low-latency data passing offer significant potential for developing efficient and low-cost solutions for improved construction and operation of the built environment. Examples of such potential solutions include the introduction of automation in environment monitoring, infrastructure inspections, asset management, and building performance analyses. In an effort to advance the fundamental computational building blocks for such applications, this dissertation explored three categories of scene understanding capabilities: 1) Localization and mapping for geometric scene understanding that enables a mobile agent (e.g., robot) to locate itself in an environment, map the geometry of the environment, and navigate through it; 2) Object recognition for semantic scene understanding that allows for automatic asset information extraction for asset tracking and resource management; 3) Distributed coupling analysis for system-level scene understanding that allows for discovery of interdependencies between different built-environment processes for system-level performance analyses and response-planning. First, this dissertation advanced Simultaneous Localization and Mapping (SLAM) techniques for convenient and low-cost locating capabilities compared with previous work. To provide a versatile Real-Time Location System (RTLS), an occupancy grid mapping enhanced visual SLAM (vSLAM) was developed to support path planning and continuous navigation that cannot be implemented directly on vSLAM’s original feature map. The system’s localization accuracy was experimentally evaluated with a set of visual landmarks. The achieved marker position measurement accuracy ranges from 0.039m to 0.186m, proving the method’s feasibility and applicability in providing real-time localization for a wide range of applications. In addition, a Self-Adaptive Feature Transform (SAFT) was proposed to improve such an RTLS’s robustness in challenging environments. As an example implementation, the SAFT descriptor was implemented with a learning-based descriptor and integrated into a vSLAM for experimentation. The evaluation results on two public datasets proved the feasibility and effectiveness of SAFT in improving the matching performance of learning-based descriptors for locating applications. Second, this dissertation explored vision-based 1D barcode marker extraction for automated object recognition and asset tracking that is more convenient and efficient than the traditional methods of using barcode or asset scanners. As an example application in inventory management, a 1D barcode extraction framework was designed to extract 1D barcodes from video scan of a built environment. The performance of the framework was evaluated with video scan data collected from an active logistics warehouse near Detroit Metropolitan Airport (DTW), demonstrating its applicability in automating inventory tracking and management applications. Finally, this dissertation explored distributed coupling analysis for understanding interdependencies between processes affecting the built environment and its occupants, allowing for accurate performance and response analyses compared with previous research. In this research, a Lightweight Communications and Marshalling (LCM)-based distributed coupling analysis framework and a message wrapper were designed. This proposed framework and message wrapper were tested with analysis models from wind engineering and structural engineering, where they demonstrated the abilities to link analysis models from different domains and reveal key interdependencies between the involved built-environment processes.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155042/1/lichaox_1.pd

Objects extraction and recognition for camera-based interaction : heuristic and statistical approaches

In this thesis, heuristic and probabilistic methods are applied to a number of problems for camera-based interactions. The goal is to provide solutions for a vision based system that is able to extract and analyze interested objects in camera images and to use that information for various interactions for mobile usage. New methods and new attempts of combination of existing methods are developed for different applications, including text extraction from complex scene images, bar code reading performed by camera phones, and face/facial feature detection and facial expression manipulation. The application-driven problems of camera-based interaction can not be modeled by a uniform and straightforward model that has very strong simplifications of reality. The solutions we learned to be efficient were to apply heuristic but easy of implementation approaches at first to reduce the complexity of the problems and search for possible means, then use developed statistical learning approaches to deal with the remaining difficult but well-defined problems and get much better accuracy. The process can be evolved in some or all of the stages, and the combination of the approaches is problem-dependent. Contribution of this thesis resides in two aspects: firstly, new features and approaches are proposed either as heuristics or statistical means for concrete applications; secondly engineering design combining seveal methods for system optimization is studied. Geometrical characteristics and the alignment of text, texture features of bar codes, and structures of faces can all be extracted as heuristics for object extraction and further recognition. The boosting algorithm is one of the proper choices to perform probabilistic learning and to achieve desired accuracy. New feature selection techniques are proposed for constructing the weak learner and applying the boosting output in concrete applications. Subspace methods such as manifold learning algorithms are introduced and tailored for facial expression analysis and synthesis. A modified generalized learning vector quantization method is proposed to deal with the blurring of bar code images. Efficient implementations that combine the approaches in a rational joint point are presented and the results are illustrated.reviewe

Low Complexity Image Recognition Algorithms for Handheld devices

Content Based Image Retrieval (CBIR) has gained a lot of interest over the last two decades. The need to search and retrieve images from databases, based on information (“features”) extracted from the image itself, is becoming increasingly important. CBIR can be useful for handheld image recognition devices in which the image to be recognized is acquired with a camera, and thus there is no additional metadata associated to it. However, most CBIR systems require large computations, preventing their use in handheld devices. In this PhD work, we have developed low-complexity algorithms for content based image retrieval in handheld devices for camera acquired images. Two novel algorithms, ‘Color Density Circular Crop’ (CDCC) and ‘DCT-Phase Match’ (DCTPM), to perform image retrieval along with a two-stage image retrieval algorithm that combines CDCC and DCTPM, to achieve the low complexity required in handheld devices are presented. The image recognition algorithms run on a handheld device over a large database with fast retrieval time besides having high accuracy, precision and robustness to environment variations. Three algorithms for Rotation, Scale, and Translation (RST) compensation for images were also developed in this PhD work to be used in conjunction with the two-stage image retrieval algorithm. The developed algorithms are implemented, using a commercial fixed-point Digital Signal Processor (DSP), into a device, called ‘PictoBar’, in the domain of Alternative and Augmentative Communication (AAC). The PictoBar is intended to be used in the field of electronic aid for disabled people, in areas like speech rehabilitation therapy, education etc. The PictoBar is able to recognize pictograms and pictures contained in a database. Once an image is found in the database, a corresponding associated speech message is played. A methodology for optimal implementation and systematic testing of the developed image retrieval algorithms on a fixed point DSP is also established as part of this PhD work

Biometric Systems

Biometric authentication has been widely used for access control and security systems over the past few years. The purpose of this book is to provide the readers with life cycle of different biometric authentication systems from their design and development to qualification and final application. The major systems discussed in this book include fingerprint identification, face recognition, iris segmentation and classification, signature verification and other miscellaneous systems which describe management policies of biometrics, reliability measures, pressure based typing and signature verification, bio-chemical systems and behavioral characteristics. In summary, this book provides the students and the researchers with different approaches to develop biometric authentication systems and at the same time includes state-of-the-art approaches in their design and development. The approaches have been thoroughly tested on standard databases and in real world applications

Robustness of Double Random Phase Encoding spread-space spread-spectrum watermarking technique

In this paper the robustness of a recently proposed image watermarking scheme, namely the Double Random Phase Encoding spread-space spread-spectrum watermarking (DRPE SS-SS) technique, is investigated. The watermark, which is chosen to be in the form of a digital barcode image, is numerically encrypted using a simulation of the optical DRPE process. This produces a random complex image, which is then processed to form a real valued random image with a low number of quantization levels. This signal is added to the host image. Extraction of the barcode, involves applying an inverse DRPE process to the watermarked image followed by low pass filtering. This algorithm is designed to utilize the capability of the DRPE to reversibly spread the energy of the watermarking information in both the space and spatial frequency domains. In this way the energy of the watermark in any spatial or spatial frequency bin is very small. To test robustness several common geometric transformations and signal processing operations are performed using both informed and blind detections for different barcode widths and different quantization levels. The results presented indicate that while the DRPE SS-SS method is robust to scaling, and JPEG compression distortion, it is especially robust to spatial cropping and both low and high pass filtering. Both random-watermark and random-host false positive cases are examined. The uniqueness of the watermark is demonstrated, and it is shown that the DRPE SS-SS has very low false positive errors, and that the larger the barcode width, the lower the false positive rate. Finally the effects of both printing and scanning are examined

Information embedding and retrieval in 3D printed objects

Deep learning and convolutional neural networks have become the main tools of computer vision. These techniques are good at using supervised learning to learn complex representations from data. In particular, under limited settings, the image recognition model now performs better than the human baseline. However, computer vision science aims to build machines that can see. It requires the model to be able to extract more valuable information from images and videos than recognition. Generally, it is much more challenging to apply these deep learning models from recognition to other problems in computer vision. This thesis presents end-to-end deep learning architectures for a new computer vision field: watermark retrieval from 3D printed objects. As it is a new area, there is no state-of-the-art on many challenging benchmarks. Hence, we first define the problems and introduce the traditional approach, Local Binary Pattern method, to set our baseline for further study. Our neural networks seem useful but straightfor- ward, which outperform traditional approaches. What is more, these networks have good generalization. However, because our research field is new, the problems we face are not only various unpredictable parameters but also limited and low-quality training data. To address this, we make two observations: (i) we do not need to learn everything from scratch, we know a lot about the image segmentation area, and (ii) we cannot know everything from data, our models should be aware what key features they should learn. This thesis explores these ideas and even explore more. We show how to use end-to-end deep learning models to learn to retrieve watermark bumps and tackle covariates from a few training images data. Secondly, we introduce ideas from synthetic image data and domain randomization to augment training data and understand various covariates that may affect retrieve real-world 3D watermark bumps. We also show how the illumination in synthetic images data to effect and even improve retrieval accuracy for real-world recognization applications

Digital Twin Framework and Auto-Linking for Management of Legacy Assets

The integration of Digital Twins (DTs) for facility and asset management is becoming increasingly common with the advancement of sensor and modelling technology. A digital twin is defined as a responsive and virtual representation of a physical system or sub-system. While traditionally used at the design stage, there are many benefits to incorporating this technology during operation and maintenance of assets. The application area examined in this thesis relates to Nuclear Power Plant (NPP) facilities with focus on legacy assets. Integration of DTs for facility management (FM) of NPP facilities should improve the efficiency of many processes involved in operations and maintenance of assets. DTs are also advantageous in this application due to their use of virtual representation that can be used for training, scenario modelling, and problem solving with minimal asset down time. To streamline the integration of DTs into current FM standard procedures, the development of a comprehensive DT framework that defines levels of DT development within each NPP subsystem, identifies workflows, and details support documents is required. This thesis aims to develop a DT framework that can be applied to varying levels of legacy assets within an NPP. The framework will guide operators and engineers through the decision-making process that is involved when implementing DTs. The presented DT framework incorporates a layered approach to demonstrate the connections between the physical environment and the virtual digital twin. The underlying levels include perception, communication, and action, each of which represent individual processes that contribute to a functional virtual model. This DT framework uses imagery, 3D scanning, operational data, and asset management software in an organized manner to streamline information integration for digital twin construction. Most of the technology and software included in the framework are commonly used in current practices, which reduces the initial cost and training investment. Since the DT framework presented was specifically developed for NPP FM application, the levels of DT were redefined to accommodate existing NPP FM practices. The support documents and workflows were also adapted according to current practices to simplify the employment of this technology. Implementing DTs within the presented application of a legacy NPP requires a high level of effort that includes digitizing records, creating models of equipment and processes, and linking these digital representations in a functional way within the overall framework. Some of this effort can be automated to improve overall efficiency and reduce time efforts. Auto-linking was developed as a tool that automates steps in the process of identifying physical asset labels within an image and 3D scan of a space within the facility including multiple physical assets such as mechanical, electrical, plumbing equipment, valves, instruments and other related assets. Those asset labels, the information they contain, and the 3D point cloud segments and images related to each tag’s asset must be linked to their respective asset’s information in an asset database that is used for NPP asset management. The 3D point cloud segments, images, and digital database serve as part of the foundation for the NPP DT being constructed within the framework presented. The Auto-linking algorithm uses a deep-learning based approach to object (tag) detection, employs optical character recognition (OCR) to read the asset tag’s unique identifier, then sorts this information into a format that directly relates it, via the asset database, to each 3D point cloud of each scene in which the asset exists. Through Auto-linking, the procedure of labelling legacy assets throughout the facility and documenting existing assets for a digital database is accelerated compared to the current manual procedures. This thesis identifies the challenges faced in FM for NPP facilities and legacy assets and demonstrates how and where DTs could be implemented for improved efficiency. The primary contributions of this thesis include the framework and support tools that can be used for some tasks required for implementing DTs for NPP FM, as well as the Auto-linking algorithm for automated asset management database development or enrichment. While these tools were developed specifically for NPP application, other application areas and adaptation for future work is also highlighted throughout the thesis