Neurosymbolic AI approach to Attribution in Large Language Models

Attribution in large language models (LLMs) remains a significant challenge, particularly in ensuring the factual accuracy and reliability of the generated outputs. Current methods for citation or attribution, such as those employed by tools like Perplexity.ai and Bing Search-integrated LLMs, attempt to ground responses by providing real-time search results and citations. However, so far, these approaches suffer from issues such as hallucinations, biases, surface-level relevance matching, and the complexity of managing vast, unfiltered knowledge sources. While tools like Perplexity.ai dynamically integrate web-based information and citations, they often rely on inconsistent sources such as blog posts or unreliable sources, which limits their overall reliability. We present that these challenges can be mitigated by integrating Neurosymbolic AI (NesyAI), which combines the strengths of neural networks with structured symbolic reasoning. NesyAI offers transparent, interpretable, and dynamic reasoning processes, addressing the limitations of current attribution methods by incorporating structured symbolic knowledge with flexible, neural-based learning. This paper explores how NesyAI frameworks can enhance existing attribution models, offering more reliable, interpretable, and adaptable systems for LLMs

“How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma

Background: In the traditional asthma management protocol, a child meets with a clinician infrequently, once in 3 to 6 months, and is assessed using the Asthma Control Test questionnaire. This information is inadequate for timely determination of asthma control, compliance, precise diagnosis of the cause, and assessing the effectiveness of the treatment plan. The continuous monitoring and improved tracking of the child’s symptoms, activities, sleep, and treatment adherence can allow precise determination of asthma triggers and a reliable assessment of medication compliance and effectiveness. Digital phenotyping refers to moment-by-moment quantification of the individual-level human phenotype in situ using data from personal digital devices, in particular, mobile phones. The kHealth kit consists of a mobile app, provided on an Android tablet, that asks timely and contextually relevant questions related to asthma symptoms, medication intake, reduced activity because of symptoms, and nighttime awakenings; a Fitbit to monitor activity and sleep; a Microlife Peak Flow Meter to monitor the peak expiratory flow and forced exhaled volume in 1 second; and a Foobot to monitor indoor air quality. The kHealth cloud stores personal health data and environmental data collected using Web services. The kHealth Dashboard interactively visualizes the collected data. Objective: The objective of this study was to discuss the usability and feasibility of collecting clinically relevant data to help clinicians diagnose or intervene in a child’s care plan by using the kHealth system for continuous and comprehensive monitoring of child’s symptoms, activity, sleep pattern, environmental triggers, and compliance. The kHealth system helps in deriving actionable insights to help manage asthma at both the personal and cohort levels. The Digital Phenotype Score and Controller Compliance Score introduced in the study are the basis of ongoing work on addressing personalized asthma care and answer questions such as, “How can I help my child better adhere to care instructions and reduce future exacerbation?” Methods: The Digital Phenotype Score and Controller Compliance Score summarize the child’s condition from the data collected using the kHealth kit to provide actionable insights. The Digital Phenotype Score formalizes the asthma control level using data about symptoms, rescue medication usage, activity level, and sleep pattern. The Compliance Score captures how well the child is complying with the treatment protocol. We monitored and analyzed data for 95 children, each recruited for a 1- or 3-month-long study. The Asthma Control Test scores obtained from the medical records of 57 children were used to validate the asthma control levels calculated using the Digital Phenotype Scores. Results: At the cohort level, we found asthma was very poorly controlled in 37% (30/82) of the children, not well controlled in 26% (21/82), and well controlled in 38% (31/82). Among the very poorly controlled children (n=30), we found 30% (9/30) were highly compliant toward their controller medication intake—suggesting a re-evaluation for change in medication or dosage—whereas 50% (15/30) were poorly compliant and candidates for a more timely intervention to improve compliance to mitigate their situation. We observed a negative Kendall Tau correlation between Asthma Control Test scores and Digital Phenotype Score as −0.509 (P\u3c.01). Conclusions: kHealth kit is suitable for the collection of clinically relevant information from pediatric patients. Furthermore, Digital Phenotype Score and Controller Compliance Score, computed based on the continuous digital monitoring, provide the clinician with timely and detailed evidence of a child’s asthma-related condition when compared with the Asthma Control Test scores taken infrequently during clinic visits

Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes

As people become more aware of their food choices, food computation models have become increasingly popular in assisting people in maintaining healthy eating habits. For example, food recommendation systems analyze recipe instructions to assess nutritional contents and provide recipe recommendations. The recent and remarkable successes of generative AI methods, such as auto-regressive large language models, can lead to robust methods for a more comprehensive understanding of recipes for healthy food recommendations beyond surface-level nutrition content assessments. In this study, we explore the use of generative AI methods to extend current food computation models, primarily involving the analysis of nutrition and ingredients, to also incorporate cooking actions (e.g., add salt, fry the meat, boil the vegetables, etc.). Cooking actions are notoriously hard to model using statistical learning methods due to irregular data patterns - significantly varying natural language descriptions for the same action (e.g., marinate the meat vs. marinate the meat and leave overnight) and infrequently occurring patterns (e.g., add salt occurs far more frequently than marinating the meat). The prototypical approach to handling irregular data patterns is to increase the volume of data that the model ingests by orders of magnitude. Unfortunately, in the cooking domain, these problems are further compounded with larger data volumes presenting a unique challenge that is not easily handled by simply scaling up. In this work, we propose novel aggregation-based generative AI methods, Cook-Gen, that reliably generate cooking actions from recipes, despite difficulties with irregular data patterns, while also outperforming Large Language Models and other strong baselines

Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes

As people become more aware of their food choices, food computation models have become increasingly popular in assisting people in maintaining healthy eating habits. For example, food recommendation systems analyze recipe instructions to assess nutritional contents and provide recipe recommendations. The recent and remarkable successes of generative AI methods, such as auto-regressive large language models, can lead to robust methods for a more comprehensive understanding of recipes for healthy food recommendations beyond surface-level nutrition content assessments. In this study, we explore the use of generative AI methods to extend current food computation models, primarily involving the analysis of nutrition and ingredients, to also incorporate cooking actions (e.g., add salt, fry the meat, boil the vegetables, etc.). Cooking actions are notoriously hard to model using statistical learning methods due to irregular data patterns - significantly varying natural language descriptions for the same action (e.g., marinate the meat vs. marinate the meat and leave overnight) and infrequently occurring patterns (e.g., add salt occurs far more frequently than marinating the meat). The prototypical approach to handling irregular data patterns is to increase the volume of data that the model ingests by orders of magnitude. Unfortunately, in the cooking domain, these problems are further compounded with larger data volumes presenting a unique challenge that is not easily handled by simply scaling up. In this work, we propose novel aggregation-based generative AI methods, Cook-Gen, that reliably generate cooking actions from recipes, despite difficulties with irregular data patterns, while also outperforming Large Language Models and other strong baselines

Ki-Cook: Clustering Multimodal Cooking Representations Through Knowledge-infused Learning

Cross-modal recipe retrieval has gained prominence due to its ability to retrieve a text representation given an image representation and vice versa. Clustering these recipe representations based on similarity is essential to retrieve relevant information about unknown food images. Existing studies cluster similar recipe representations in the latent space based on class names. Due to inter-class similarity and intraclass variation, associating a recipe with a class name does not provide sufficient knowledge about recipes to determine similarity. However, recipe title, ingredients, and cooking actions provide detailed knowledge about recipes and are a better determinant of similar recipes. In this study, we utilized this additional knowledge of recipes, such as ingredients and recipe title, to identify similar recipes, emphasizing attention especially on rare ingredients. To incorporate this knowledge, we propose a knowledge-infused multimodal cooking representation learning network, Ki-Cook, built on the procedural attribute of the cooking process. To the best of our knowledge, this is the first study to adopt a comprehensive recipe similarity determinant to identify and cluster similar recipe representations. The proposed network also incorporates ingredient images to learn multimodal cooking representation. Since the motivation for clustering similar recipes is to retrieve relevant information for an unknown food image, we evaluated the ingredient retrieval task. We performed an empirical analysis to establish that our proposed model improves the Coverage of Ground Truth by 12% and the Intersection Over Union by 10% compared to the baseline models. On average, the representations learned by our model contain an additional 15.33% of rare ingredients compared to the baseline models. Owing to this difference, our qualitative evaluation shows a 39% improvement in clustering similar recipes in the latent space compared to the baseline models, with an inter-annotator agreement of the Fleiss kappa score of 0.35

Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study

Background: Asthma is a chronic pulmonary disease with multiple triggers. It can be managed by strict adherence to an asthma care plan and by avoiding these triggers. Clinicians cannot continuously monitor their patients’ environment and their adherence to an asthma care plan, which poses a significant challenge for asthma management. Objective: In this study, pediatric patients were continuously monitored using low-cost sensors to collect asthma-relevant information. The objective of this study was to assess whether kHealth kit, which contains low-cost sensors, can identify personalized triggers and provide actionable insights to clinicians for the development of a tailored asthma care plan. Methods: The kHealth asthma kit was developed to continuously track the symptoms of asthma in pediatric patients and monitor the patients’ environment and adherence to their care plan for either 1 or 3 months. The kit consists of an Android app–based questionnaire to collect information on asthma symptoms and medication intake, Fitbit to track sleep and activity, the Peak Flow meter to monitor lung functions, and Foobot to monitor indoor air quality. The data on the patient’s outdoor environment were collected using third-party Web services based on the patient’s zip code. To date, 107 patients consented to participate in the study and were recruited from the Dayton Children’s Hospital, of which 83 patients completed the study as instructed. Results: Patient-generated health data from the 83 patients who completed the study were included in the cohort-level analysis. Of the 19% (16/83) of patients deployed in spring, the symptoms of 63% (10/16) and 19% (3/16) of patients suggested pollen and particulate matter (PM2.5), respectively, to be their major asthma triggers. Of the 17% (14/83) of patients deployed in fall, symptoms of 29% (4/17) and 21% (3/17) of patients suggested pollen and PM2.5, respectively, to be their major triggers. Among the 28% (23/83) of patients deployed in winter, PM2.5 was identified as the major trigger for 83% (19/23) of patients. Similar correlations were not observed between asthma symptoms and factors such as ozone level, temperature, and humidity. Furthermore, 1 patient from each season was chosen to explain, in detail, his or her personalized triggers by observing temporal associations between triggers and asthma symptoms gathered using the kHealth asthma kit. Conclusions: The continuous monitoring of pediatric asthma patients using the kHealth asthma kit generates insights on the relationship between their asthma symptoms and triggers across different seasons. This can ultimately inform personalized asthma management and intervention plans

Knowledge-enabled Personalized Dashboard for Asthma Management in Children

Introduction: Childhood Asthma is a significant public health concern worldwide. Effective management of childhood asthma requires close monitoring of disease triggers, medication compliance and symptom control. The recent growth of the Internet of Things (IoT) based devices has enabled continuous monitoring of patients. kHealth-Asthma is a knowledge-enabled semantic framework consisting of IoT enabled sensors to record patient symptoms, medication usage and their environment. For each patient, 29 diverse parameters with 1852 data points are collected daily. kHealthDash platform enables real-time visual analysis at an individual and cohort level over such high volume, high variety data. Methods: The kHealth kit was given to 100 asthmatic children (5 to 17 years of age) for a period of one or three months each. The kit consists of an Android app-based questionnaire to record symptoms and medication usage, Fitbit to track activity and sleep, peak flow meter to measure PEF and FEV1, Foobot to monitor indoor air quality and web services to obtain outdoor environmental observations. Data collected are pushed to a private cloud storage in near real-time and visualized using kHealthDash. Five healthcare providers evaluated the effectiveness of kHealthDash by answering questions on data interpretation. Results: Providers reported that analyzing data with kHealthDash was 65% easier than using data in tabular format. The System Usability Score for kHealthDash is 80.5 (\u3e68.5 - threshold), implying that kHealthDash is a user-friendly interface. Conclusion: kHealthDash integrates and visualizes multimodal data and holds promise to aid the clinicians in better decision making for asthma management

“How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma

Background: In the traditional asthma management protocol, a child meets with a clinician infrequently, once in 3 to 6 months, and is assessed using the Asthma Control Test questionnaire. This information is inadequate for timely determination of asthma control, compliance, precise diagnosis of the cause, and assessing the effectiveness of the treatment plan. The continuous monitoring and improved tracking of the child’s symptoms, activities, sleep, and treatment adherence can allow precise determination of asthma triggers and a reliable assessment of medication compliance and effectiveness. Digital phenotyping refers to moment-by-moment quantification of the individual-level human phenotype in situ using data from personal digital devices, in particular, mobile phones. The kHealth kit consists of a mobile app, provided on an Android tablet, that asks timely and contextually relevant questions related to asthma symptoms, medication intake, reduced activity because of symptoms, and nighttime awakenings; a Fitbit to monitor activity and sleep; a Microlife Peak Flow Meter to monitor the peak expiratory flow and forced exhaled volume in 1 second; and a Foobot to monitor indoor air quality. The kHealth cloud stores personal health data and environmental data collected using Web services. The kHealth Dashboard interactively visualizes the collected data. Objective: The objective of this study was to discuss the usability and feasibility of collecting clinically relevant data to help clinicians diagnose or intervene in a child’s care plan by using the kHealth system for continuous and comprehensive monitoring of child’s symptoms, activity, sleep pattern, environmental triggers, and compliance. The kHealth system helps in deriving actionable insights to help manage asthma at both the personal and cohort levels. The Digital Phenotype Score and Controller Compliance Score introduced in the study are the basis of ongoing work on addressing personalized asthma care and answer questions such as, “How can I help my child better adhere to care instructions and reduce future exacerbation?” Methods: The Digital Phenotype Score and Controller Compliance Score summarize the child’s condition from the data collected using the kHealth kit to provide actionable insights. The Digital Phenotype Score formalizes the asthma control level using data about symptoms, rescue medication usage, activity level, and sleep pattern. The Compliance Score captures how well the child is complying with the treatment protocol. We monitored and analyzed data for 95 children, each recruited for a 1- or 3-month-long study. The Asthma Control Test scores obtained from the medical records of 57 children were used to validate the asthma control levels calculated using the Digital Phenotype Scores. Results: At the cohort level, we found asthma was very poorly controlled in 37% (30/82) of the children, not well controlled in 26% (21/82), and well controlled in 38% (31/82). Among the very poorly controlled children (n=30), we found 30% (9/30) were highly compliant toward their controller medication intake—suggesting a re-evaluation for change in medication or dosage—whereas 50% (15/30) were poorly compliant and candidates for a more timely intervention to improve compliance to mitigate their situation. We observed a negative Kendall Tau correlation between Asthma Control Test scores and Digital Phenotype Score as −0.509 (P\u3c.01). Conclusions: kHealth kit is suitable for the collection of clinically relevant information from pediatric patients. Furthermore, Digital Phenotype Score and Controller Compliance Score, computed based on the continuous digital monitoring, provide the clinician with timely and detailed evidence of a child’s asthma-related condition when compared with the Asthma Control Test scores taken infrequently during clinic visits