Eating and Exercise Detection with Continuous Glucose Monitors

Eating and exercise detection using continuous glucose monitor (CGM) signals is key to provide recommendations for a healthy lifestyle. However, this can be challenging given imbalanced data and other contexts. Previous works have used accelerometers, gyroscopes, glucose monitors, and other sensors but not necessarily all three plus others combined. Therefore, I aim to build a model by testing various techniques and testing glucose along with different statistical body measurements, such as electrodermal activity, heart rate, blood volume, accelerometer, gyroscope, etc. A sliding window is used to extract statistical measures from each body measurement, such as standard deviation, mean, and range to look for patterns correlated to eating and exercise. I select an extreme gradient boosted decision tree algorithm with Synthetic Minority Oversampling Technique. I compare the performance of just solely using glucose and then adding more sensory data and discovered that there is not consistent change in performance. I also adjusted the window and overlap to compare eating detection performance and found that there is not a concrete impact on the performance. Furthermore, I performed exercise detection and compare with and without CGM. There appears to be no significant performance difference with or without glucose. In addition to eating detection, I also examine for correlation between glucose variation and exercise moments. I finally conclude that it is not feasibly possible to detect eating with my current methods. However, for exercise detection, I can produce better detection results compared to eating, but my current method for detecting correlations between glucose levels and exercise moments can be later improved

Embedding a Grid of Load Cells into a Dining Table for Automatic Monitoring and Detection of Eating Events

This dissertation describes a “smart dining table” that can detect and measure consumption events. This work is motivated by the growing problem of obesity, which is a global problem and an epidemic in the United States and Europe. Chapter 1 gives a background on the economic burden of obesity and its comorbidities. For the assessment of obesity, we briefly describe the classic dietary assessment tools and discuss their drawback and the necessity of using more objective, accurate, low-cost, and in-situ automatic dietary assessment tools. We explain in short various technologies used for automatic dietary assessment such as acoustic-, motion-, or image-based systems. This is followed by a literature review of prior works related to the detection of weights and locations of objects sitting on a table surface. Finally, we state the novelty of this work. In chapter 2, we describe the construction of a table that uses an embedded grid of load cells to sense the weights and positions of objects. The main challenge is aligning the tops of adjacent load cells to within a few micrometer tolerance, which we accomplish using a novel inversion process during construction. Experimental tests found that object weights distributed across 4 to 16 load cells could be measured with 99.97±0.1% accuracy. Testing the surface for flatness at 58 points showed that we achieved approximately 4.2±0.5 um deviation among adjacent 2x2 grid of tiles. Through empirical measurements we determined that the table has a 40.2 signal-to-noise ratio when detecting the smallest expected intake amount (0.5 g) from a normal meal (approximate total weight is 560 g), indicating that a tiny amount of intake can be detected well above the noise level of the sensors. In chapter 3, we describe a pilot experiment that tests the capability of the table to monitor eating. Eleven human subjects were video recorded for ground truth while eating a meal on the table using a plate, bowl, and cup. To detect consumption events, we describe an algorithm that analyzes the grid of weight measurements in the format of an image. The algorithm segments the image into multiple objects, tracks them over time, and uses a set of rules to detect and measure individual bites of food and drinks of liquid. On average, each meal consisted of 62 consumption events. Event detection accuracy was very high, with an F1-score per subject of 0.91 to 1.0, and an F1 score per container of 0.97 for the plate and bowl, and 0.99 for the cup. The experiment demonstrates that our device is capable of detecting and measuring individual consumption events during a meal. Chapter 4 compares the capability of our new tool to monitor eating against previous works that have also monitored table surfaces. We completed a literature search and identified the three state-of-the-art methods to be used for comparison. The main limitation of all previous methods is that they used only one load cell for monitoring, so only the total surface weight can be analyzed. To simulate their operations, the weights of our grid of load cells were summed up to use the 2D data as 1D. Data were prepared according to the requirements of each method. Four metrics were used to evaluate the comparison: precision, recall, accuracy, and F1-score. Our method scored the highest in recall, accuracy, and F1-score; compared to all other methods, our method scored 13-21% higher for recall, 8-28% higher for accuracy, and 10-18% higher for F1-score. For precision, our method scored 97% that is just 1% lower than the highest precision, which was 98%. In summary, this dissertation describes novel hardware, a pilot experiment, and a comparison against current state-of-the-art tools. We also believe our methods could be used to build a similar surface for other applications besides monitoring consumption

DETECTION OF HEALTH-RELATED BEHAVIOURS USING HEAD-MOUNTED DEVICES

The detection of health-related behaviors is the basis of many mobile-sensing applications for healthcare and can trigger other inquiries or interventions. Wearable sensors have been widely used for mobile sensing due to their ever-decreasing cost, ease of deployment, and ability to provide continuous monitoring. In this dissertation, we develop a generalizable approach to sensing eating-related behavior. First, we developed Auracle, a wearable earpiece that can automatically detect eating episodes. Using an off-the-shelf contact microphone placed behind the ear, Auracle captures the sound of a person chewing as it passes through the head. This audio data is then processed by a custom circuit board. We collected data with 14 participants for 32 hours in free-living conditions and achieved accuracy exceeding 92.8% and F1 score exceeding77.5% for eating detection with 1-minute resolution. Second, we adapted Auracle for measuring children’s eating behavior, and improved the accuracy and robustness of the eating-activity detection algorithms. We used this improved prototype in a laboratory study with a sample of 10 children for 60 total sessions and collected 22.3 hours of data in both meal and snack scenarios. Overall, we achieved 95.5% accuracy and 95.7% F1 score for eating detection with 1-minute resolution. Third, we developed a computer-vision approach for eating detection in free-living scenarios. Using a miniature head-mounted camera, we collected data with 10 participants for about 55 hours. The camera was fixed under the brim of a cap, pointing to the mouth of the wearer and continuously recording video (but not audio) throughout their normal daily activity. We evaluated performance for eating detection using four different Convolutional Neural Network (CNN) models. The best model achieved 90.9% accuracy and 78.7%F1 score for eating detection with 1-minute resolution. Finally, we validated the feasibility of deploying the 3D CNN model in wearable or mobile platforms when considering computation, memory, and power constraints

Objectively measured eating behaviors and their relation to food intake in school and hospital settings

Introduction: The measurement of food intake (what and how much we eat and drink) is of great importance due to its involvement in three great challenges facing humanity: 1) obesity/overnutrition, 2) undernutrition and 3) climate change, as well as their related health consequences. However, measuring food and energy intake in humans is complicated since traditional self-reported methods have systematic bias while traditional objective laboratory methods have generalizability and upscaling issues. Therefore, novel methods to measure food and energy intake in humans have often been requested. A plethora of factors have been associated with variation in food intake in humans. For example, internal behavioral factors such as eating rate, internal disease conditions such as Parkinson’s disease (PD) as well as external environmental factors such as food proximity are notable ones. These factors have mainly been investigated by use of the traditional methods listed above. Aims: The overarching aim with this thesis was to use novel technological tools (i.e., portable food scales and video cameras) to measure and explain variance in food intake and body mass index in school, hospital and free-living settings. Aims in school setting: To explain variance in food mass intake during school lunch with objectively measured eating behaviors (how a person eats), the proximity to food and subjective appetite measures. To assess the test-retest reliability of objectively measured food mass intake and eating rate during school lunch. To assess the concurrent validity of self-reported eating rate. Aims in hospital setting: To compare energy intake among healthy controls, early and advanced PD patients and to investigate the association between clinical features of PD as well as objective eating behaviors with energy intake during a hospital lunch. Aim in free-living setting: To distinguish differences in BMI z-scores (BMIz) among self-reported eating rate categories in populations of Swedish and Greek high school students. Methods: School studies Settings: The data collection was conducted in the school lunch cafeteria environment at a high school in central area of Stockholm, Sweden. Study design: A cross-sectional study design was used to explain variance in food intake and to investigate the association between objectively measured eating rate and food intake. An experimental study design was used to investigate the effects of food proximity and a repeat-measures study design was used to assess the test-retest reliability of objectively measured food mass intake and eating rate. Participants: Six high school classes including 187 students were invited to participate in monitored school lunches during 2015-2017. Out of these, 114 unique students provided complete meal data and 103 with a mean (SD) age of 16.7 (0.6) and BMIz of -0.07 (1.05)were included in the food intake variance analysis. All 114 participants (with a mean (SD) age of 16.5 (0.8) and BMIz 0.04 (1.01)) were included in the association between eating rate and BMIz. Out of the 114 unique participants, 50 students came for a repeated meal and provided complete data for test-retest analyses. Study procedures: The lunch study was conducted during normal school lunch hours (11.30-13.00). The students who participated in the snack experiment came back at 15.30 for the one-hour experimental snack session with snack foods, either a) close to the table where they were sitting (proximal condition) or further away from them (distal condition). Served food: During school lunches, usual lunch food at the included school (beef/vegetable patties, brown sauce, potatoes, fish, variety of vegetables, water/milk) was served in a buffet-like setting. For the snack experiment, chocolate lentils, crackers and grapes were served ad libitum. Hospital study Settings: The data collection was conducted in a dedicated room at the Department of Neurology of the Technical University Dresden (TUD), Germany. Study design: A cross-sectional study design was used. Participants: 64 participants (n = 23 healthy controls, n = 20 early and n = 21 advanced PD patients) with a mean (SD) age of 62.4 (7.8) and BMI 27.2 (4.3) were included. Study procedures: Study participants had a medical evaluation before they ate their lunch meal during normal lunch hours (11.00-15.00). Served food: A standardized meal (200g sausages, 400g potato salad, 200g apple mash and 500ml of water) was served to all participants. Free-living study Settings: A smartphone application was developed to gather self-reported eating rate and BMIz. Study design: A cross-sectional study design was used. Participants: Students from multiple high schools in Sweden (n = 748) and Greece (n = 1084) were recruited through school supported actions (n = 1832 in total, mean (SD) age of 15.8 (0.9), BMIz 0.47 (1.41)) that included self-reported measures of weight, height and eating rate. Study procedures: Students who chose to participate downloaded the study mobile application and self-reported their data. Data sources and measurements In the school and hospital setting, weight and height scales were used to measure participants body weight and height, and food mass and energy intake were measured with portable food scales. Video cameras were used to record the meals and eating behaviors were annotated onto the videos in computer software. In the free-living setting, students self-reported their age, weight, height, and their speed of eating in comparison to others at their own discretion. Results: Reliability and validity: In the school setting, there was no significant systematic change in mean food mass intake from lunch 1 to lunch 2 (-7.5g, 95% confidence interval: -43.1g to +28.0g). The intraclass correlation between food mass intake during lunch 1 vs. lunch 2 was 0.74 (95% confidence interval 0.58 to 0.84). There was a significant systematic change in eating rate (g/min) from lunch 1 to lunch 2 (+4.4 g/min, 95% confidence interval: +0.7 g/min to +8.1 g/min). The intraclass correlation between eating rate during lunch 1 vs. lunch 2 was 0.73 (95% confidence interval 0.59 to 0.85). When comparing the objective eating rate among the three categories of self-reported eating rate (slow, intermediate, and fast), a significant difference between the groups was obtained [F(2, 111) = 7.104, P = 0.001, partial η2 = 0.113]. Bonferroni post hoc comparisons showed that students who self-reported eating slower than others had significantly lower eating rate (-13.7g/min, 95% confidence interval: -22.5g/min to -4.84g/min) compared to students who self-reported eating faster than others. The weighted Kappa value for self-reported eating rate categories versus objectively established eating rate categories was 0.31 (P < 0.001). Main results: School: Eating rate, number of spoonfuls, sex, number of food additions and food taste (explanatory power in that order) were all significant explanatory variables for variance in food mass intake during school lunch, while BMI and change in fullness were not significant (effect size: adjusted R squared = 0.766 for the total model). There was a significant “large” (R = 0.667) correlation between objectively measured eating rate and food mass intake during school lunch. When dividing students into tertiles of eating rate (slow, intermediate and fast eaters), a significant difference in food mass intake between the three groups was found [F(2, 111) = 30.578, P < 0.001, partial η2 = 0.355]. Bonferroni post hoc comparisons showed that students in the “slow” objective eating rate tertile were eating 133 grams less food (95% confidence intervals = -210g to -56g) than students in the “intermediate” objective eating rate tertile, and 247 grams less (95% confidence intervals = -324g to -170g) than students in the “fast” eating rate tertile. Students who were participating in the distal snack food condition were eating significantly less energy from snacks than students in the proximal condition (mean difference = -222.7 kcal 95% confidence intervals: -428.3 kcal to -17.2 kcal). Hospital: Advanced PD patients consumed significantly less energy during lunch compared to both early PD patients (b = -202.7 kcal, 95% confidence interval: -329.2 kcal to -76.2 kcal) and healthy controls (b = -162.1 kcal, 95% confidence interval: -285.7 kcal to -38.4 kcal) when controlling for sex. Free-living: Self-reported eating rate was found to be a significant explanatory variable for variation in self-reported BMI z-scores [F(2, 1829) = 9.724, P < 0.001, partial η2 = 0.011]. Bonferroni post hoc test showed that students who self-reported eating slower than others had 0.23 units lower BMI z-scores (95% confidence intervals: -0.43 to, -0.03) than students who self-reported intermediate eating rate, and 0.37 units lower (95% confidence intervals: -0.57 to -0.17) than students who self-reported eating faster than others. Outcome synthesis: Overall, eating behaviors were the most powerful explanatory variables, while desire to eat and food taste were the most powerful self-reported variables for food and energy intake variance when controlling for sex in the included studies. Advanced PD status (hospital study) as well as the food proximity (snack experiment) were also powerful explanatory variables, while PD-related symptomatology as well as self-reported eating rate, hunger, change in fullness and BMI had low or no explanatory power. Conclusions: Objectively measured single-meal food mass intake and eating rate could be used to rank individuals in comparison to their peers. Subjective eating rate could be used to distinguish groups with slow and fast eating rates in large scale studies but should not be used on an individual level. The outcomes of this thesis suggest that objectively measured eating behaviors and subjective factors such as food taste and desire to eat, as well as the external condition proximity to food, are all powerful explanatory factors for variance in food mass and energy intake and might be potential targets in future interventions that aim to modify food intake. Additionally, advanced PD condition was associated with lower energy intake. Potential interventions mentioned above might be helpful in this patient group to normalize their energy intake and reduce their risk of undernutrition. Furthermore, the results suggest that novel methods that objectively measure eating behaviors could be utilized in larger-scale nutrition research. Further technological developments of these methods could also give real-time feedback on targeted eating behaviors that are related to food intake, thus ultimately reducing the risk of diseases related to over- and undernutrition

Sensing with Earables: A Systematic Literature Review and Taxonomy of Phenomena

Earables have emerged as a unique platform for ubiquitous computing by augmenting ear-worn devices with state-of-the-art sensing. This new platform has spurred a wealth of new research exploring what can be detected on a wearable, small form factor. As a sensing platform, the ears are less susceptible to motion artifacts and are located in close proximity to a number of important anatomical structures including the brain, blood vessels, and facial muscles which reveal a wealth of information. They can be easily reached by the hands and the ear canal itself is affected by mouth, face, and head movements. We have conducted a systematic literature review of 271 earable publications from the ACM and IEEE libraries. These were synthesized into an open-ended taxonomy of 47 different phenomena that can be sensed in, on, or around the ear. Through analysis, we identify 13 fundamental phenomena from which all other phenomena can be derived, and discuss the different sensors and sensing principles used to detect them. We comprehensively review the phenomena in four main areas of (i) physiological monitoring and health, (ii) movement and activity, (iii) interaction, and (iv) authentication and identification. This breadth highlights the potential that earables have to offer as a ubiquitous, general-purpose platform

Sensing with Earables: A Systematic Literature Review and Taxonomy of Phenomena

Earables have emerged as a unique platform for ubiquitous computing by augmenting ear-worn devices with state-of-the-art sensing. This new platform has spurred a wealth of new research exploring what can be detected on a wearable, small form factor. As a sensing platform, the ears are less susceptible to motion artifacts and are located in close proximity to a number of important anatomical structures including the brain, blood vessels, and facial muscles which reveal a wealth of information. They can be easily reached by the hands and the ear canal itself is affected by mouth, face, and head movements. We have conducted a systematic literature review of 271 earable publications from the ACM and IEEE libraries. These were synthesized into an open-ended taxonomy of 47 different phenomena that can be sensed in, on, or around the ear. Through analysis, we identify 13 fundamental phenomena from which all other phenomena can be derived, and discuss the different sensors and sensing principles used to detect them. We comprehensively review the phenomena in four main areas of (i) physiological monitoring and health, (ii) movement and activity, (iii) interaction, and (iv) authentication and identification. This breadth highlights the potential that earables have to offer as a ubiquitous, general-purpose platform

Objective quantification and analysis of eating behaviour associated with obesity development - from lab to real-life

Introduction: The last four decades have seen a marked increase in childhood and adult obesity prevalence, attributed to an “obesogenic” environment. Several genetical, environmental and behavioural factors have been identified that increase the risk of obesity, but treatment outcomes are usually modest and the risk of relapse high. One limitation responsible for these moderate results could be methodological, with researchers questioning both the external validity of eating behaviour measures in the laboratory (controlled) and the internal validity of eating behaviour measures in free-living (real-life) settings. Technological advances could solve some of these issues, allowing for accurate methods, similar to those used in controlled settings, to be used in real- life. Deploying accurate methods in both controlled and real-life settings would in turn enable the estimation of external validity, determining the limits of generalization between settings. In turn enabling the deployment of these methods in settings which allow large scale screening, for early identification of individuals at risk of becoming obese. Aim: The overarching aim of the thesis was to: i) evaluate the stability of human eating behaviour and ii) investigate the usability and feasibility of methods developed for controlled settings, when deployed in semi-controlled and real-life settings. Paper I – Determine if individuals maintain their eating behaviour, in relation to the group, despite experimental manipulations to meal conditions (i.e., unit sizes and serving occasion). Paper II – Feasibility of employing novel technology for baseline eating behaviour collection in adolescents eating school lunches in a school cafeteria setting (semi-controlled). Paper III – Feasibility of employing novel technology in an experimental manipulation study, to determine the effect of proximity in a semi-controlled school setting. Paper IV – By use of novel technology, examine the maintenance of eating behaviours in adolescents, from semi-controlled to real-life settings, both at group- and individual-level. Methods: Paper I – Three randomised crossover studies, of which two compared eating behaviour across different unit sizes, while one compared eating behaviour between lunch and dinner in healthy young adults. Performed in a controlled setting, employing traditional laboratory methods. Paper II – An observational study of healthy adolescents, performed at lunch in a school cafeteria, employing traditional laboratory methods in a semi-controlled setting. Paper III – A randomised experimental study of healthy adolescents, performed in a semi- controlled, comparing the eating behaviour between two groups seated at different proximity to food items. Paper IV – An observational study on eating behaviour of healthy adolescents, divided into two parts; i) collection of eating behaviour data, performed at lunch in a school cafeteria, using a similar protocol to that of Paper II and ii) collection of eating behaviour data by the participants in real-life settings, using the same devices as in the controlled setting. Results: In all papers the distribution of eating behaviour values between individuals were large. In Paper I, the largest increase in unit size significantly increased meal duration and chews and while there was a trend for both increased meal duration and number of chews the larger the food unit sizes were, it did not lead to a significant reduction in food intake. Meanwhile, the correlation coefficient of all eating behaviours across all conditions was high (except for number of bites between the largest and smallest food unit size condition). In Paper II, male participants ate significantly more, mediated by significantly larger bites. The bite sizes of both men and women were reduced as the meal progressed. In Paper III, increased distance to food led to a significant reduction in intake, caused by individuals taking less chocolate. In Paper IV, there was no significant difference in eating behaviour characteristics between the semi- controlled and real-life meals. In addition, the correlation coefficient of food intake and eating rate was high between settings, while the correlation of meal duration was low. Also, on an individual level, 50%, 32% and 27% of the food intake, eating rate and meal duration measures, respectively, from the semi-controlled meal fell within the confidence interval of the real-life meals. In the semi-controlled and real-life settings (Papers II-IV), the agreement between subjective and objective eating behaviour measures were very low. Meanwhile, in both semi- controlled and real-life settings the method could be deployed within the time schedule imposed by the school, with high data retention. Also, participants rated the comfortability participating in the semi-controlled and real-life settings very high and the usability of the system as “Good” or higher. Conclusions: Human eating behaviour appears stable in comparison to the group when unit size and serving occasion is manipulated in a controlled setting and when eating in different settings (semi- controlled and real-life). Suggesting generalisations can be made between settings and conditions and that risk behaviours may be measured in settings other than real-life, at least on group level. However, although individual prediction rates of eating behaviour characteristics from semi-controlled setting to real-life settings appears higher than subjective ratings, they are still too low for use in the design of tailored interventions. In addition, compared to controlled studies, the method allowed recruitment of a younger age group, since it enabled measurements in a different location. The thesis also provides evidence that the employed methods are usable, feasible and acceptable, with high data retention in adolescent users, in semi-controlled and real-life settings. Methods similar to the ones used in this thesis can provide previously unattainable information (primarily temporal) in settings that are less controlled than the laboratory, such as semi-controlled and real-life