Real-time food intake classification and energy expenditure estimation on a mobile device

© 2015 IEEE.Assessment of food intake has a wide range of applications in public health and life-style related chronic disease management. In this paper, we propose a real-time food recognition platform combined with daily activity and energy expenditure estimation. In the proposed method, food recognition is based on hierarchical classification using multiple visual cues, supported by efficient software implementation suitable for realtime mobile device execution. A Fischer Vector representation together with a set of linear classifiers are used to categorize food intake. Daily energy expenditure estimation is achieved by using the built-in inertial motion sensors of the mobile device. The performance of the vision-based food recognition algorithm is compared to the current state-of-the-art, showing improved accuracy and high computational efficiency suitable for realtime feedback. Detailed user studies have also been performed to demonstrate the practical value of the software environment

Automatic Food Intake Assessment Using Camera Phones

Obesity is becoming an epidemic phenomenon in most developed countries. The fundamental cause of obesity and overweight is an energy imbalance between calories consumed and calories expended. It is essential to monitor everyday food intake for obesity prevention and management. Existing dietary assessment methods usually require manually recording and recall of food types and portions. Accuracy of the results largely relies on many uncertain factors such as user\u27s memory, food knowledge, and portion estimations. As a result, the accuracy is often compromised. Accurate and convenient dietary assessment methods are still blank and needed in both population and research societies. In this thesis, an automatic food intake assessment method using cameras, inertial measurement units (IMUs) on smart phones was developed to help people foster a healthy life style. With this method, users use their smart phones before and after a meal to capture images or videos around the meal. The smart phone will recognize food items and calculate the volume of the food consumed and provide the results to users. The technical objective is to explore the feasibility of image based food recognition and image based volume estimation. This thesis comprises five publications that address four specific goals of this work: (1) to develop a prototype system with existing methods to review the literature methods, find their drawbacks and explore the feasibility to develop novel methods; (2) based on the prototype system, to investigate new food classification methods to improve the recognition accuracy to a field application level; (3) to design indexing methods for large-scale image database to facilitate the development of new food image recognition and retrieval algorithms; (4) to develop novel convenient and accurate food volume estimation methods using only smart phones with cameras and IMUs. A prototype system was implemented to review existing methods. Image feature detector and descriptor were developed and a nearest neighbor classifier were implemented to classify food items. A reedit card marker method was introduced for metric scale 3D reconstruction and volume calculation. To increase recognition accuracy, novel multi-view food recognition algorithms were developed to recognize regular shape food items. To further increase the accuracy and make the algorithm applicable to arbitrary food items, new food features, new classifiers were designed. The efficiency of the algorithm was increased by means of developing novel image indexing method in large-scale image database. Finally, the volume calculation was enhanced through reducing the marker and introducing IMUs. Sensor fusion technique to combine measurements from cameras and IMUs were explored to infer the metric scale of the 3D model as well as reduce noises from these sensors

Food Recognition and Volume Estimation in a Dietary Assessment System

Recently obesity has become an epidemic and one of the most serious worldwide public health concerns of the 21st century. Obesity diminishes the average life expectancy and there is now convincing evidence that poor diet, in combination with physical inactivity are key determinants of an individual s risk of developing chronic diseases such as cancer, cardiovascular disease or diabetes. Assessing what people eat is fundamental to establishing the link between diet and disease. Food records are considered the best approach for assessing energy intake. However, this method requires literate and highly motivated subjects. This is a particular problem for adolescents and young adults who are the least likely to undertake food records. The ready access of the majority of the population to mobile phones (with integrated camera, improved memory capacity, network connectivity and faster processing capability) has opened up new opportunities for dietary assessment. The dietary information extracted from dietary assessment provide valuable insights into the cause of diseases that greatly helps practicing dietitians and researchers to develop subsequent approaches for mounting intervention programs for prevention. In such systems, the camera in the mobile phone is used for capturing images of food consumed and these images are then processed to automatically estimate the nutritional content of the food. However, food objects are deformable objects that exhibit variations in appearance, shape, texture and color so the food classification and volume estimation in these systems suffer from lower accuracy. The improvement of the food recognition accuracy and volume estimation accuracy are challenging tasks. This thesis presents new techniques for food classification and food volume estimation. For food recognition, emphasis was given to texture features. The existing food recognition techniques assume that the food images will be viewed at similar scales and from the same viewpoints. However, this assumption fails in practical applications, because it is difficult to ensure that a user in a dietary assessment system will put his/her camera at the same scale and orientation to capture food images as that of the target food images in the database. A new scale and rotation invariant feature generation approach that applies Gabor filter banks is proposed. To obtain scale and rotation invariance, the proposed approach identifies the dominant orientation of the filtered coefficient and applies a circular shifting operation to place this value at the first scale of dominant direction. The advantages of this technique are it does not require the scale factor to be known in advance and it is scale/and rotation invariant separately and concurrently. This approach is modified to achieve improved accuracy by applying a Gaussian window along the scale dimension which reduces the impact of high and low frequencies of the filter outputs enabling better matching between the same classes. Besides automatic classification, semi automatic classification and group classification are also considered to have an idea about the improvement. To estimate the volume of a food item, a stereo pair is used to recover the structure as a 3D point cloud. A slice based volume estimation approach is proposed that converts the 3D point cloud to a series of 2D slices. The proposed approach eliminates the problem of knowing the distance between two cameras with the help of disparities and depth information from a fiducial marker. The experimental results show that the proposed approach can provide an accurate estimate of food volume

Food Object Recognition Using a Mobile Device: State of the Art

In this paper nine mobile food recognition systems are described based on their system architecture and their core properties (the core properties and experimental results are shown on the last page). While the mobile hardware increased its power through the years (2009 - 2013) and the food detection algorithms got optimized, still there was no uniform approach to the question of food detection. Also, some system used additional information for better detection, like voice data, OCR and bounding boxes. Three systems included a volume estimation feature. First five systems were implemented on a client-server architecture, while the last three took advantage of the available hardware in later years and proposed a client only based architecture

Food Object Recognition Using a Mobile Device: State of the Art

In this paper nine mobile food recognition systems are described based on their system architecture and their core properties (the core properties and experimental results are shown on the last page). While the mobile hardware increased its power through the years (2009 - 2013) and the food detection algorithms got optimized, still there was no uniform approach to the question of food detection. Also, some system used additional information for better detection, like voice data, OCR and bounding boxes. Three systems included a volume estimation feature. First five systems were implemented on a client-server architecture, while the last three took advantage of the available hardware in later years and proposed a client only based architecture

A Survey on Automated Food Monitoring and Dietary Management Systems

Healthy diet with balanced nutrition is key to the prevention of life-threatening diseases such as obesity, cardiovascular disease, and cancer. Recent advances in smartphone and wearable sensor technologies have led to a proliferation of food monitoring applications based on automated food image processing and eating episode detection, with the goal to conquer drawbacks of the traditional manual food journaling that is time consuming, inaccurate, underreporting, and low adherent. In order to provide users feedback with nutritional information accompanied by insightful dietary advice, various techniques in light of the key computational learning principles have been explored. This survey presents a variety of methodologies and resources on this topic, along with unsolved problems, and closes with a perspective and boarder implications of this field

Technology-assisted dietary assessment

Dietary intake provides valuable insights for mounting intervention programs for prevention of disease. With growing concern for adolescent obesity, the need to accurately measure diet becomes imperative. Assessment among adolescents is problematic as this group has irregular eating patterns and have less enthusiasm for recording food intake. Preliminary studies among adolescents suggest that innovative use of technology may improve the accuracy of diet information from young people. In this paper, we propose a novel food record method using a mobile device that will provide an accurate account of daily food and nutrient intake among adolescents. Our approach includes the use of image analysis tools for identification and quantification of food consumption. Images obtained before and after food is consumed can be used to estimate the diet of an individual. In this paper we describe our initial results and indicate the potential of the proposed system

Dietary assessment and obesity aviodance system based on vision: A review

Using technology for food objects recognition and estimation of its calories is very useful to spread food culture and awareness among people in the age of obesity due to the bad habits of food consumption and wide range of inappropriate food products.Image based sensing of such system is very promising with the large expanding of camera embedded portable devices such as smartphones, PC tablets, and laptops.In the past decade, researchers have been working on developing a reliable image based system for food recognition and calories estimation.Different approaches have tackled the system from different aspects.This paper reviews the state of the art of this interesting application, and presents its experimental results.Future work of research is presented in order to guide new researchers toward potential tracks to create more maturity and reliability to this application