Building a foundation model for neuroscience

The brain’s complexity enables its remarkable functions, but this very complexity makes it hard to understand. Current methodologies for recording brain activity often provide narrow views of the brain's function, limited by the constraints of current recording technology and the structured nature of the standard neuroscience experiment. This fragmentation of datasets has hampered the development of robust and comprehensive computational models of brain function that generalize across diverse conditions, tasks, and individuals. Our work is motivated by the need for a large-scale foundation model in neuroscience--one that can go beyond the limitations of single-dataset approaches and offer a fuller, more comprehensive picture of brain function. In this thesis, we propose novel methodologies and frameworks aimed at addressing the challenges of building such a model. We discuss three main contributions. The first contribution is towards building scalable and unified approaches for training on diverse neural datasets. The second contribution aims to develop self-supervised methods for understanding dynamics of behavior at multiple timescales. The third contribution is to develop methods for building invariances in neural data to further our understanding of the brain.Ph.D.Machine Learnin

Computational Sensemaking for Embodied Co-Creative Artificial Intelligence

This research lies at the intersection of human-AI interaction, creative collaboration, embodied cognition, and social cognition. Its central aim is to advance the understanding of how AI agents can actively participate in creative processes traditionally dominated by human-human interactions, such as dance and drawing. Grounded in theories of embodiment and intersubjectivity—which prioritize sensory engagement and interaction over abstract cognition—this work explores the complexities of co-creativity and social cognition through the lens of sensemaking within AI systems. The research investigates how AI systems can engage in co-creative processes by leveraging sensemaking patterns, both descriptively and generatively. It explores how theories of embodiment and sensemaking enhance our understanding of co-creativity, how sensemaking patterns can be analyzed and integrated into co-creative systems, and how design considerations for future co-creative AI systems can be developed. The study employs a mixed-methods approach, combining qualitative and quantitative analyses through empirical studies, interviews, video coding, thematic analysis, surveys, speculative design, and self-reflective exercises. Key contributions include a new perspective on computational co-creativity that integrates theories from human-computer interaction, embodiment, and social cognition. The dissertation introduces the Observable Creative Sensemaking (OCSM) framework, a method for quantifying sensemaking in embodied creative improvisation. It demonstrates how OCSM can be used descriptively to compare different co-creative interactions and applied as a generative model to guide real-time improvisation. Additionally, the development of two co-creative AI systems—Drawcto, a multi-agent drawing application, and LuminAI, an embodied improvisational dance system—highlights the practical application of these theoretical frameworks and models in real-world AI systems.Ph.D.Digital Medi

Steering the EdTech Ship

Interview portion of Lost in the Stacks, episode 625. Features interview with Warren Goetzel, discussing his role in leading educational technology initiatives at Georgia Tech. Warren is Director of Academic Technology and Engagement for Georgia Tech’s Office of Information Technology & Director of External and Faculty Engagement with the Center for 21st Century Universities (C21U).Interview portion of Lost in the Stacks, episode 625. Features interview with Warren Goetzel, discussing his role in leading educational technology initiatives at Georgia Tech. Warren is Director of Academic Technology and Engagement for Georgia Tech’s Office of Information Technology & Director of External and Faculty Engagement with the Center for 21st Century Universities (C21U)

Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer

In this dissertation, we propose a Bayesian adaptive learning framework by focusing our attention on estimating a manageable number of latent variables of deep neural networks (DNNs).The deep latent variables here refer to the unobservable representations of data, and they usually correspond to intermediate hidden embedding outputs from a specific layer of DNNs. Within this framework, we explore two Bayesian estimation techniques: Variational Bayes (VB) and Maximum A Posteriori (MAP). The VB approach utilizes variational inference to approximate the entire posterior distribution and introduces Gaussian mean-field variational inference and empirical Bayes to handle different parallel data scenarios across domains. The MAP approach, on the other hand, focuses on point estimation of latent variables using Gaussian-based and Dirichlet-based distribution assumptions.Experimental validations on acoustic adaptation tasks demonstrate the superiority of the proposed Bayesian adaptation approaches over other knowledge transfer methods. Furthermore, these Bayesian adaptation techniques have been successfully applied to large pre-trained models like Wav2Vec2 and Whisper, showcasing considerable advantages and affirming their efficacy in complex acoustic settings, thus advancing effective model adaptation strategies.Ph.D.Electrical and Computer Engineerin

"LNP Biodistribution data to support manuscript entitled 'Sensitizing solid tumors to CAR-mediated cytotoxicity by lipid nanoparticle delivery of synthetic antigens'"

Dataset for Biodistribution NatCancer. Organs were harvested 24 hrs following intratumoral injection of nLuc-LNPs and immediately placed on ice cold 1x PBS-/- until imaging. Organs were treated with Nano-Glo Live Cell Assay System initially diluted 20x in provided buffer and then further diluted 50x in 1xPBS-/- before use. Prior to IVIS imaging, organs were dipped in the diluted substrate for 5 min at 37ºC. Excess liquid was removed by tapping the tissue on a KimWipe prior to placing on black construction paper for imaging

Autonomous Robot-Enabled Data Collection, Classification, and Processing Method for Real-Time Construction Schedule Updating

Despite significant advancements in the manufacturing sector, the construction industry has been lagging in adopting computer-aided and robotic technologies. This has resulted in inefficient and unproductive construction procedures not being fully addressed. Construction progress tracking is an essential part of project management and keeps the project’s shareholders updated and confident about finishing the project on time within the specified budget and quality. Monitoring, data collection, and updating the schedule are some of project management's most time-consuming and labor-intensive tasks. Modern technologies such as robots and computer vision present a potential solution to this issue. Although previous research enhanced data collection and data processing separately, a considerable amount of manual effort is needed to classify and transform collected data to the processing stages. This research presents a novel framework to address the gap, utilizing autonomous four-legged robots to capture images from indoor building construction, classify data for creating quantified material lists, generate as-built models, and compare them with original models to enable real-time construction schedules updates. The proposed method offers a streamlined, optimized, and productive inspection and data collection approach, potentially reducing time and labor requirements. By providing decision-makers with live updates, construction project managers may reduce overruns, meet deadlines, and reach the project’s ultimate goals. The study highlights the technical details of the proposed procedure, discusses the potential benefits of adopting quadrupeds in the construction industry, and presents a direction for future research and development in this area.M.S.Building Construction and Facility Managemen

Resilient Autonomy for Extreme and Uncertain Environments

Presented on March 26, 2025 at 12:15 p.m. in the Klaus Advanced Computing Building, Room 1116.Masayoshi Tomizuka is the Cheryl and John Neerhout, Jr. Distinguished Professor and a Distinguished Professor of Mechanical Engineering at the University of California, Berkeley. His research includes adaptive control, computer-aided manufacturing, control systems and theory, digital control, dynamic systems, manufacturing, and mechanical vibrations.Runtime: 48:54 minutesRobustness in AI and robotics shows great promise if systems can transition from the lab to real-world environments, moving beyond the single-operator per robot paradigm. However, the unstructured nature of the real-world demands nuanced decision-making from robots. In this talk, Prof. Scherer will outline approaches, progress, and results in multi-modal sensing, nuanced perception inputs, navigation in difficult terrain, and extensions to multi-robot teams, as well as future research directions

Deep Reinforcement Learning Framework for Autonomous Surface Vehicles in Environmental Cleanup

The water pollution from floating plastics poses significant environmental threats that require efficient solutions. ASV presents a promising solution to address this challenge. However, deploying DRL for ASV control in real-world environmental missions is underexplored due to simulation limitations and the sim-to-real gap. This thesis presents a DRL framework for ASVs focused on environmental missions, explicitly targeting the autonomous collection of floating waste. An open-source, highly parallelized hydrodynamics and buoyancy simulation environment is developed to facilitate large-scale training. By integrating system identification with domain randomization, we reduce the sim-to-real gap, enhancing the robustness and energy efficiency of the trained agents. The proposed approach is validated through simulation and real-world experiments, demonstrating improved task completion times and reduced energy consumption. Task experiments show that our approach reduces energy consumption by 13.1%, while reducing task completion time by 7.4%. These findings, supported by sharing our open-source implementation, have the potential to impact the efficiency and versatility of ASVs, contributing to environmental preservation efforts. This thesis incorporates and expands on work previously published in a paper presented at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in 2024. Significant portions of the content have been reused and adapted to fit the comprehensive format and depth required for the thesis.M.S.Computer Scienc

Life after Stroke

This short documentary was created as a course requirement in HTS 3086 – Sociology of Medicine and Health under the supervision of Dr. Jennifer Singh.Runtime: 07:34 minutesThis documentary explores the illness experience of the stigma associated with the physical and social challenges after experiencing a stroke

Multimodal assessment of neuropsychiatric disorders using audiovisual recordings

Over one billion people worldwide live with a neuropsychiatric disorder, yet most do not have access to adequate diagnosis and care. Accurate, fast, and accessible detection of those disorders is critical to early and effective interventions. Over the last decade, digitally administered assessments have emerged as one of the most promising approaches. Moreover, the increasing use of telemedicine in psychiatry and neurology in recent years presented an unprecedented opportunity to use audiovisual data for accessible neuropsychiatric assessments without the limitation of geographical location and specialized hardware. This dissertation describes the use of low-cost audiovisual data collected from in-lab and remote mobile devices to assess neuropsychiatric conditions by extracting and combining various behavioral and physiological indicators. First, we showed that facial and speech emotions can be effectively estimated from audiovisual data collected in interviews and used for major depressive disorder evaluation. Then, we presented the automated assessment of cognitive impairment using facial emotions and viewing behaviors recognized from videos passively collected from a mobile device. Lastly, we further improved the scalability and accessibility by extracting facial, vocal, linguistic, and cardiovascular features from audiovisual data collected remotely from heterogeneous mobile devices and validating them in both clinician-rated and self-rated mental health condition evaluation.Ph.D.Machine Learnin