深層学習による弱教師あり領域分割とその食事画像への応用

電気通信大学201

ResSeg: Residual encoder-decoder convolutional neural network for food segmentation

This paper presents the implementation and evaluation of different convolutional neural network architectures focused on food segmentation. To perform this task, it is proposed the recognition of 6 categories, among which are the main food groups (protein, grains, fruit, vegetables) and two additional groups, rice and drink or juice. In addition, to make the recognition more complex, it is decided to test the networks with food dishes already started, i.e. during different moments, from its serving to its finishing, in order to verify the capability to see when there is no more food on the plate. Finally, a comparison is made between the two best resulting networks, a SegNet with architecture VGG-16 and a network proposed in this work, called Residual Segmentation Convolutional Neural Network or ResSeg, with which accuracies greater than 90% and interception-over-union greater than 75% were obtained. This demonstrates the ability, not only of SegNet architectures for food segmentation, but the use of residual layers to improve the contour of the segmentation and segmentation of complex distribution or initiated of food dishes, opening the field of application of this type of networks to be implemented in feeding assistants or in automated restaurants, including also for dietary control for the amount of food consumed

GourmetNet: Food Segmentation Using Multi-Scale Waterfall Features With Spatial and Channel Attention

Deep learning and Computer vision are extensively used to solve problems in wide range of domains from automotive and manufacturing to healthcare and surveillance. Research in deep learning for food images is mainly limited to food identification and detection. Food segmentation is an important problem as the first step for nutrition monitoring, food volume and calorie estimation. This research is intended to expand the horizons of deep learning and semantic segmentation by proposing a novel single-pass, end-to-end trainable network for food segmentation. Our novel architecture incorporates both channel attention and spatial attention information in an expanded multi-scale feature representation using the WASPv2 module. The refined features will be processed with the advanced multi-scale waterfall module that combines the benefits of cascade filtering and pyramid representations without requiring a separate decoder or postprocessing

Analysis & Numerical Simulation of Indian Food Image Classification Using Convolutional Neural Network

Recognition of Indian food can be assumed to be a fine-grained visual task owing to recognition property of various food classes. It is therefore important to provide an optimized approach to segmentation and classification for different applications based on food recognition. Food computation mainly utilizes a computer science approach which needs food data from various data outlets like real-time images, social flat-forms, food journaling, food datasets etc, for different modalities. In order to consider Indian food images for a number of applications we need a proper analysis of food images with state-of-art-techniques. The appropriate segmentation and classification methods are required to forecast the relevant and upgraded analysis. As accurate segmentation lead to proper recognition and identification, in essence we have considered segmentation of food items from images. Considering the basic convolution neural network (CNN) model, there are edge and shape constraints that influence the outcome of segmentation on the edge side. Approaches that can solve the problem of edges need to be developed; an edge-adaptive As we have solved the problem of food segmentation with CNN, we also have difficulty in classifying food, which has been an important area for various types of applications. Food analysis is the primary component of health-related applications and is needed in our day to day life. It has the proficiency to directly predict the score function from image pixels, input layer to produce the tensor outputs and convolution layer is used for self- learning kernel through back-propagation. In this method, feature extraction and Max-Pooling is considered with multiple layers, and outputs are obtained using softmax functionality. The proposed implementation tests 92.89% accuracy by considering some data from yummly dataset and by own prepared dataset. Consequently, it is seen that some more improvement is needed in food image classification. We therefore consider the segmented feature of EA-CNN and concatenated it with the feature of our custom Inception-V3 to provide an optimized classification. It enhances the capacity of important features for further classification process. In extension we have considered south Indian food classes, with our own collected food image dataset and got 96.27% accuracy. The obtained accuracy for the considered dataset is very well in comparison with our foregoing method and state-of-the-art techniques.

Camera-Based Meal Type and Weight Estimation in Self-Service Lunch Line Restaurants

Individuals, restaurant owners and health organizations are all interested in accurate information about food intake, but collecting the information in sufficiently automated way remains a practical challenge. In controlled environments, such as lunch line restaurants, the food intake can be estimated by measuring the portions upon purchase and separately monitoring the food waste, but even this requires often complicated setups such as repeated weighing of the plate after every meal component. In this work we explore the feasibility of using a combination of ceiling-mounted cameras and computer vision for estimating both the types and weights of individual food items the customers are taking in a lunch line restaurant. We describe the imaging system and weighing- based sensing for obtaining ground truth training data, and develop and evaluate deep learning models for the computer vision tasks. We demonstrate high accuracy especially in meal type identification and hence validate the feasibility of the approach. We release the annotated dataset for further development of improved methods.Peer reviewe

Hyperspectral Image Analysis of Food for Nutritional Intake

The primary object of this dissertation is to investigate the application of hyperspectral technology to accommodate for the growing demand in the automatic dietary assessment applications. Food intake is one of the main factors that contribute to human health. In other words, it is necessary to get information about the amount of nutrition and vitamins that a human body requires through a daily diet. Manual dietary assessments are time-consuming and are also not precise enough, especially when the information is used for the care and treatment of hospitalized patients. Moreover, the data must be analyzed by nutritional experts. Therefore, researchers have developed various semiautomatic or automatic dietary assessment systems; most of them are based on the conventional color images such as RGB. The main disadvantage of such systems is their inability to differentiate foods of similar color or same ingredients in various colors, or different forms such as cooked or mixed forms. Although adding features such as shape, size and texture improve the overall performance, they are sensitive to changes in the illumination, rotation, scale, etc. A balance between quality and quantity of features representation, and system efficiency must also be considered. Hyperspectral technology combines conventional imaging technology with spectroscopy in a three-dimensional data-cube to obtain both the spatial and spectral information of the objects. However, the high dimensionality of hyperspectral data in addition to the redundancy between spectral bands limits performance, especially in online or onboard data processing applications. Thus, various features selection/extraction are also used to select the optimal feature subsets. The results are promising and verify the feasibility of using hyperspectral technology in dietary assessment applications

Deep learning for food instance segmentation

Food object detection and instance segmentation are critical in many applications such as dietary management or food intake monitoring. Food image recognition poses different challenges, such as the existence of a large number of classes, a high inter-class similarity, and high intra-class variance. This, along with the traditional problems associated with object detection and instance segmentation make this a very complex computer vision task. Real-world food datasets generally suffer from long-tailed and fine-grained distributions. However, the recent literature fails to address food detection in this regard. In this research, we propose a novel two-stage object detector, which we call Strong LOng-tailed Food object Detection and instance Segmentation (SLOF-DS), to tackle the long-tailed nature of food images. In addition, a multi-task based framework, which exploits different sources of prior information, was proposed to improve the classification of fine-grained classes. Lastly, we also propose a new module based on Graph Neural Neworks, we call Graph Confidence Propagation (GCP) that additionally improves the performance of both object detection and instance segmentation modules by combining all the model outputs considering the global image context. Exhaustive quantitive and qualitative analysis performed on two open source food benchmarks, namely the UECFood-256 (object detection) and the AiCrowd Food Recognition Challenge 2022 dataset (instance segmentation) using different baseline algorithms prove the robust improvements introduced by the different components proposed in this thesis. More concretely, we outperformed the state-of-the-art performance on both public datasets

Deep learning in food category recognition

Integrating artificial intelligence with food category recognition has been a field of interest for research for the past few decades. It is potentially one of the next steps in revolutionizing human interaction with food. The modern advent of big data and the development of data-oriented fields like deep learning have provided advancements in food category recognition. With increasing computational power and ever-larger food datasets, the approach’s potential has yet to be realized. This survey provides an overview of methods that can be applied to various food category recognition tasks, including detecting type, ingredients, quality, and quantity. We survey the core components for constructing a machine learning system for food category recognition, including datasets, data augmentation, hand-crafted feature extraction, and machine learning algorithms. We place a particular focus on the field of deep learning, including the utilization of convolutional neural networks, transfer learning, and semi-supervised learning. We provide an overview of relevant studies to promote further developments in food category recognition for research and industrial applicationsMRC (MC_PC_17171)Royal Society (RP202G0230)BHF (AA/18/3/34220)Hope Foundation for Cancer Research (RM60G0680)GCRF (P202PF11)Sino-UK Industrial Fund (RP202G0289)LIAS (P202ED10Data Science Enhancement Fund (P202RE237)Fight for Sight (24NN201);Sino-UK Education Fund (OP202006)BBSRC (RM32G0178B8