Generative Models for Art and Society

Text-to-image models have demonstrated remarkable capabilities in producing high-fidelity images from natural language prompts. The widespread application and increasing accessibility of pioneering models, such as Stable Diffusion, have gained significant attention regarding the impact of generated images on representations in downstream tasks. Concurrently, ethical considerations on text-to-image generation have emerged especially regarding gender bias. This paper presents three projects that explore generative models on their capabilities and bias. The first project leverages Stable Diffusion to disentangle content and style in art paintings, paving the way for applying the generative model to digital humanities. The second project evaluates gender bias in text-to-image generation, analyzing its origins and manifestations in generated images. The third project presents a survey on societal bias evaluation in generative models, targeting to synthesize current research and provide insights into future directions. Through these projects, we aim to contribute to the growing body of knowledge on the applications and potential societal impacts of text-to-image generation, fostering a more nuanced understanding of their capabilities and limitations

Statistically Principled Deep Learning for SAR Image Segmentation

This paper proposes a novel approach for Synthetic Aperture Radar (SAR) image segmentation by incorporating known statistical properties of SAR into deep learning models. We generate synthetic data using the Generalized Gamma distribution, modify the U-Net architecture to encompass statistical moments, and employ stochastic distance losses for improved segmentation performance. Evaluation against traditional methods will reveal the potential of this approach to advance SAR image analysis, with broader applications in environmental monitoring and general image segmentation tasks

Knowledge-Powered Recommendation for an Improved Diet Water Footprint

According to WWF, 1.1 billion people lack access to water, and 2.7 billion experience water scarcity at least one month a year. By 2025, two-thirds of the world's population may be facing water shortages. This highlights the urgency of managing water usage efficiently, especially in water-intensive sectors like food. This paper proposes a recommendation engine, powered by knowledge graphs, aiming to facilitate sustainable and healthy food consumption. The engine recommends ingredient substitutes in user recipes that improve nutritional value and reduce environmental impact, particularly water footprint. The system architecture includes source identification, information extraction, schema alignment, knowledge graph construction, and user interface development. The research offers a promising tool for promoting healthier eating habits and contributing to water conservation efforts

Is Federated Learning Still Alive in the Foundation Model Era?

Federated learning (FL) has arisen as an alternative to collecting large amounts of data in a central place to train a machine learning (ML) model. FL is privacy-friendly, allowing multiple parties to collaboratively train an ML model without exchanging or transmitting their training data. For this purpose, an aggregator iteratively coordinates the training process among parties, and parties simply share with the aggregator model updates, which contain information pertinent to the model such as neural network weights. Besides privacy, generalization has been another key driver for FL: parties who do not have enough data to train a good performing model by themselves can now engage in FL to obtain an ML model suitable for their tasks. Products and real applications in the industry and consumer space have demonstrated the power of this learning paradigm. Recently, foundation models have taken the AI community by storm, promising to solve the shortage of labeled data. A foundation model is a powerful model that can be recycled for a variety of use cases by applying techniques such as zero-shot learning and full or parameter-efficient fine tuning. The premise is that the amount of data required to fine tune a foundation model for a new task is much smaller than fully training a traditional model from scratch. The reason why this is the case is that a good foundation model has already learned relevant general representations, and thus, adapting it to a new task only requires a minimal number of additional samples. This raises the question: Is FL still alive in the era of foundation models? In this talk, I will address this question. I will present some use cases where FL is very much alive. In these use cases, finding a foundation model with a desired representation is difficult if not impossible. With this pragmatic point of view, I hope to shed some light into a real use case where disparate private data is available in isolation at different parties and where labels may be located at a single party that doesn’t have any other information, making it impossible for a single party to train a model on its own. Furthermore, in some vertically-partitioned scenarios, cleaning data is not an option due to privacy-related reasons and it is not clear how to apply foundation models. Finally, I will also go over a few other requirements that are often overlooked, such as unlearning of data and its implications for the lifecycle management of FL and systems based on foundation models

You Can Have Your Cake and Eat It Too: Ensuring Practical Robustness and Privacy in Federated Learning

Inherently, federated learning (FL) robustness is very challenging to guarantee, especially when trying to maintain privacy. Compared to standard ML settings, FL's open training process allows for malicious clients to easily go under the radar. Alongside this, malicious clients can easily collude to attack the training process continuously, and without detection. FL models are also still susceptible to attacks on standard ML training procedures. This massive attack surface makes balancing the tradeoff between utility, practicality, robustness, and privacy extremely challenging. While there have been proposed defenses to attacks using popular privacy-preserving primitives, such as fully homomorphic encryption, they often face trouble balancing an all-important question that is present in all privacy-preserving systems: How much utility and practicality am I willing to give up to ensure privacy and robustness? In this work, we discuss a practical approach towards secure and robust FL and the challenges that face this field of emerging research

Operational Environments at the Extreme Tactical Edge

You can’t get more “on the tactical edge” than in space. No other operational domain suffers from the combinations of distance from the operator, harsh environments, unreachable assets with aging hardware, and increadably long communications as space systems. The complexity of developing and deploying AI solutions in satellites and probes is far more difficult than deploying similar AI on Earth. This talk explores some of the considerations involved in deploying AI and machine learning (ML) in the space domain

Advancing Neuro-Inspired Lifelong Learning for Edge with Co-Design

Lifelong learning, which refers to an agent's ability to continuously learn and enhance its performance over its lifespan, is a significant challenge in artificial intelligence (AI), that biological systems tackle efficiently. This challenge is further exacerbated when AI is deployed in untethered environments with strict energy and latency constraints. We take inspiration from neural plasticity and investigate how to leverage and build energy-efficient lifelong learning machines. Specifically, we study how a combination of neural plasticity mechanisms, namely neuromodulation, synaptic consolidation, and metaplasticity, enhance the continual learning capabilities of AI models. We further co-design architectures that leverage compute-in-memory topologies and sparse spike-based communication with quantization for the edge. Aspects of this co-design can be transferred to federated lifelong learning scenarios

Reconciling Privacy and Byzantine-robustness in Federated Learning

In this talk, we will discuss how to make federated learning secure for the server and private for the clients simultaneously. Most prior efforts fall into either of the two categories. At one end of the spectrum, some work uses techniques like secure aggregation to hide the individual client’s updates and only reveal the aggregated global update to a malicious server that strives to infer the clients’ privacy from their updates. At the other end of the spectrum, some work uses Byzantine-robust FL protocols to suppress the influence of malicious clients’ updates. We present a protocol that offers bidirectional defense to simultaneously combat against the malicious centralized server and Byzantine malicious clients. Our protocol also improves the dimension dependence and achieve a near-optimal statistical rate for strongly convex cases

Human-like Learning in Temporally Structured Environments

Natural environments have correlations at a wide range of timescales. Human cognition is tuned to this temporal structure, as seen by power laws of learning and memory, and by spacing effects whereby the intervals between repeated training data affect how long knowledge is retained. Machine learning is instead dominated by batch iid training or else relatively simple nonstationarity assumptions such as random walks or discrete task sequences. The main contributions of our work are: (1) We develop a Bayesian model formalizing the brain's inductive bias for temporal structure and show our model accounts for key features of human learning and memory. (2) We translate the model into a new gradient-based optimization technique for neural networks that endows them with human-like temporal inductive bias and improves their performance in realistic nonstationary tasks. Our technical approach is founded on Bayesian inference over 1/f noise, a statistical signature of many natural environments with long-range, power law correlations. We derive a new closed-form solution to this problem by treating the state of the environment as a sum of processes on different timescales and applying an extended Kalman filter to learn all timescales jointly. We then derive a variational approximation of this model for training neural networks, which can be used as a drop-in replacement for standard optimizers in arbitrary architectures. Our optimizer decomposes each weight in the network as a sum of subweights with different learning and decay rates and tracks their joint uncertainty. Thus knowledge becomes distributed across timescales, enabling rapid adaptation to task changes while retaining long-term knowledge and avoiding catastrophic interference. Simulations show improved performance in environments with realistic multiscale nonstationarity. Finally, we present simulations showing our model gives essentially parameter-free fits of learning, forgetting, and spacing effects in human data. We then explore the analogue of human spacing effects in a deep net trained in a structured environment where tasks recur at different rates and compare the model's behavioral properties to those of people

Remote Possibilities: Where There Is a WIL, Is There a Way? AI Education for Remote Learners in a New Era of Work-Integrated-Learning

Increasing diversity in educational settings is challenging in part due to the lack of access to resources for non-traditional learners in remote communities. Post-pandemic platforms designed specifically for remote and hybrid learning---supporting team-based collaboration online---are positioned to bridge this gap. Our work combines the use of these new platforms with co-creation and collaboration tools for AI assisted remote Work-Integrated-Learning (WIL) opportunities, including efforts in community and with the public library system. This paper outlines some of our experiences to date, and proposes methods to further integrate AI education into community-driven applications for remote WIL