A sample weight and adaboost CNN-based coarse to fine classification of fruit and vegetables at a supermarket self-checkout

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. The physical features of fruit and vegetables make the task of vision-based classification of fruit and vegetables challenging. The classification of fruit and vegetables at a supermarket self-checkout poses even more challenges due to variable lighting conditions and human factors arising from customer interactions with the system along with the challenges associated with the colour, texture, shape, and size of a fruit or vegetable. Considering this complex application, we have proposed a progressive coarse to fine classification technique to classify fruit and vegetables at supermarket checkouts. The image and weight of fruit and vegetables have been obtained using a prototype designed to simulate the supermarket environment, including the lighting conditions. The weight information is used to change the coarse classification of 15 classes down to three, which are further used in AdaBoost-based Convolutional Neural Network (CNN) optimisation for fine classification. The training samples for each coarse class are weighted based on AdaBoost optimisation, which are updated on each iteration of a training phase. Multi-class likelihood distribution obtained by the fine classification stage is used to estimate a final classification with a softmax classifier. GoogleNet, MobileNet, and a custom CNN have been used for AdaBoost optimisation, with promising classification results

안경에서 기계적으로 증폭된 힘 측정을 통한 측두근 활동의 감지

학위논문 (박사)-- 서울대학교 대학원 공과대학 기계항공공학부, 2017. 8. 이건우.Recently, the form of a pair of glasses is broadly utilized as a wearable device that provides the virtual and augmented reality in addition to its natural functionality as a visual aid. These approaches, however, have lacked the use of its inherent kinematic structure, which is composed of both the temple and the hinge. When we equip the glasses, the force is concentrated at the hinge, which connects the head piece and the temple, from the law of the lever. In addition, since the temple passes through a temporalis muscle, chewing and wink activity, anatomically activated by the contraction and relaxation of the temporalis muscle, can be detected from the mechanically amplified force measurement at the hinge. This study presents a new and effective method for automatic and objective measurement of the temporalis muscle activity through the natural-born lever mechanism of the glasses. From the implementation of the load cell-integrated wireless circuit module inserted into the both hinges of a 3D printed glasses frame, we developed the system that responds to the temporalis muscle activity persistently regardless of various form factor different from each person. This offers the potential to improve previous studies by avoiding the morphological, behavioral, and environmental constraints of using skin-attached, proximity, and sound sensors. In this study, we collected data featured as sedentary rest, chewing, walking, chewing while walking, talking and wink from 10-subject user study. The collected data were transferred to a series of 84-dimentional feature vectors, each of which was composed of the statistical features of both temporal and spectral domain. These feature vectors, then, were used to define a classifier model implemented by the support vector machine (SVM) algorithm. The model classified the featured activities (chewing, wink, and physical activity) as the average F1 score of 93.7%. This study provides a novel approach on the monitoring of ingestive behavior (MIB) in a non-intrusive and un-obtrusive manner. It supplies the possibility to apply the MIB into daily life by distinguishing the food intake from the other physical activities such as walking, talking, and wink with higher accuracy and wearability. Furthermore, through applying this approach to a sensor-integrated hair band, it can be potentially used for the medical monitoring of the sleep bruxism or temporomandibular dysfunction.Abstract Chapter 1. Introduction 1.1. Motivation 1.1.1. Law of the Lever 1.1.2. Lever Mechanism in Human Body 1.1.3. Mechanical Advantage in Auditory Ossicle 1.1.4. Mechanical Advantage in Glasses 1.2. Background 1.2.1. Biological Information from Temporalis Muscle 1.2.2. Detection of Temporalis Muscle Activity 1.2.3. Monitoring of Ingestive Behavior 1.3. Research Scope and Objectives Chapter 2. Proof-of-Concept Validation 2.1. Experimental Apparatus 2.2. Measurement Results 2.3. Discussion Chapter 3. Implementation of GlasSense 3.1. Hardware Prototyping 3.1.1. Preparation 3.1.2. Load Cell-Integrated Circuit Module 3.1.3. 3D Printed Frame of Glasses 3.1.4. Hardware Integration 3.2. Data Acquisition System 3.2.1. Wireless Data Transmission 3.2.2. Data Collecting Module Chapter 4. Data Collection through User Study 4.1. Preparation for Experiment 4.2. Activity Recording Chapter 5. Feature Extraction 5.1. Signal Preprocessing and Segmentation 5.1.1. Temporal Frame 5.1.2. Spectral Frame 5.2. Feature Extraction 5.2.1. Temporal Features 5.2.2. Spectral Features 5.2.3. Feature Vector Generation Chapter 6. Classification of Featured Activity 6.1. Support Vector Machine (SVM) 6.2. Design of Classifier Model 6.2.1. Grid-Search 6.2.2. Cross-Validation 6.3. Classification Result 6.4. Performance Improvement 6.5. Discussion Chapter 7. Conclusions Bibliography 초록Docto

Computer vision based classification of fruits and vegetables for self-checkout at supermarkets

The field of machine learning, and, in particular, methods to improve the capability of machines to perform a wider variety of generalised tasks are among the most rapidly growing research areas in today’s world. The current applications of machine learning and artificial intelligence can be divided into many significant fields namely computer vision, data sciences, real time analytics and Natural Language Processing (NLP). All these applications are being used to help computer based systems to operate more usefully in everyday contexts. Computer vision research is currently active in a wide range of areas such as the development of autonomous vehicles, object recognition, Content Based Image Retrieval (CBIR), image segmentation and terrestrial analysis from space (i.e. crop estimation). Despite significant prior research, the area of object recognition still has many topics to be explored. This PhD thesis focuses on using advanced machine learning approaches to enable the automated recognition of fresh produce (i.e. fruits and vegetables) at supermarket self-checkouts. This type of complex classification task is one of the most recently emerging applications of advanced computer vision approaches and is a productive research topic in this field due to the limited means of representing the features and machine learning techniques for classification. Fruits and vegetables offer significant inter and intra class variance in weight, shape, size, colour and texture which makes the classification challenging. The applications of effective fruit and vegetable classification have significant importance in daily life e.g. crop estimation, fruit classification, robotic harvesting, fruit quality assessment, etc. One potential application for this fruit and vegetable classification capability is for supermarket self-checkouts. Increasingly, supermarkets are introducing self-checkouts in stores to make the checkout process easier and faster. However, there are a number of challenges with this as all goods cannot readily be sold with packaging and barcodes, for instance loose fresh items (e.g. fruits and vegetables). Adding barcodes to these types of items individually is impractical and pre-packaging limits the freedom of choice when selecting fruits and vegetables and creates additional waste, hence reducing customer satisfaction. The current situation, which relies on customers correctly identifying produce themselves leaves open the potential for incorrect billing either due to inadvertent error, or due to intentional fraudulent misclassification resulting in financial losses for the store. To address this identified problem, the main goals of this PhD work are: (a) exploring the types of visual and non-visual sensors that could be incorporated into a self-checkout system for classification of fruits and vegetables, (b) determining a suitable feature representation method for fresh produce items available at supermarkets, (c) identifying optimal machine learning techniques for classification within this context and (d) evaluating our work relative to the state-of-the-art object classification results presented in the literature. An in-depth analysis of related computer vision literature and techniques is performed to identify and implement the possible solutions. A progressive process distribution approach is used for this project where the task of computer vision based fruit and vegetables classification is divided into pre-processing and classification techniques. Different classification techniques have been implemented and evaluated as possible solution for this problem. Both visual and non-visual features of fruit and vegetables are exploited to perform the classification. Novel classification techniques have been carefully developed to deal with the complex and highly variant physical features of fruit and vegetables while taking advantages of both visual and non-visual features. The capability of classification techniques is tested in individual and ensemble manner to achieved the higher effectiveness. Significant results have been obtained where it can be concluded that the fruit and vegetables classification is complex task with many challenges involved. It is also observed that a larger dataset can better comprehend the complex variant features of fruit and vegetables. Complex multidimensional features can be extracted from the larger datasets to generalise on higher number of classes. However, development of a larger multiclass dataset is an expensive and time consuming process. The effectiveness of classification techniques can be significantly improved by subtracting the background occlusions and complexities. It is also worth mentioning that ensemble of simple and less complicated classification techniques can achieve effective results even if applied to less number of features for smaller number of classes. The combination of visual and nonvisual features can reduce the struggle of a classification technique to deal with higher number of classes with similar physical features. Classification of fruit and vegetables with similar physical features (i.e. colour and texture) needs careful estimation and hyper-dimensional embedding of visual features. Implementing rigorous classification penalties as loss function can achieve this goal at the cost of time and computational requirements. There is a significant need to develop larger datasets for different fruit and vegetables related computer vision applications. Considering more sophisticated loss function penalties and discriminative hyper-dimensional features embedding techniques can significantly improve the effectiveness of the classification techniques for the fruit and vegetables applications

Brain Tumor Diagnosis Support System: A decision Fusion Framework

An important factor in providing effective and efficient therapy for brain tumors is early and accurate detection, which can increase survival rates. Current image-based tumor detection and diagnosis techniques are heavily dependent on interpretation by neuro-specialists and/or radiologists, making the evaluation process time-consuming and prone to human error and subjectivity. Besides, widespread use of MR spectroscopy requires specialized processing and assessment of the data and obvious and fast show of the results as photos or maps for routine medical interpretative of an exam. Automatic brain tumor detection and classification have the potential to offer greater efficiency and predictions that are more accurate. However, the performance accuracy of automatic detection and classification techniques tends to be dependent on the specific image modality and is well known to vary from technique to technique. For this reason, it would be prudent to examine the variations in the execution of these methods to obtain consistently high levels of achievement accuracy. Designing, implementing, and evaluating categorization software is the goal of the suggested framework for discerning various brain tumor types on magnetic resonance imaging (MRI) using textural features. This thesis introduces a brain tumor detection support system that involves the use of a variety of tumor classifiers. The system is designed as a decision fusion framework that enables these multi-classifier to analyze medical images, such as those obtained from magnetic resonance imaging (MRI). The fusion procedure is ground on the Dempster-Shafer evidence fusion theory. Numerous experimental scenarios have been implemented to validate the efficiency of the proposed framework. Compared with alternative approaches, the outcomes show that the methodology developed in this thesis demonstrates higher accuracy and higher computational efficiency