171 research outputs found
Uncertainty estimation under model misspecification in neural network regression
Although neural networks are powerful function approximators, the underlying modelling assumptions ultimately define the likelihood and thus the model class they are parameterizing. In classification, these assumptions are minimal as the commonly employed softmax is capable of representing any discrete distribution over a finite set of outcomes. In regression, however, restrictive assumptions on the type of continuous distribution to be realized are typically placed, like the dominant choice of training via mean-squared error and its underlying Gaussianity assumption. Recently, modelling advances allow to be agnostic to the type of continuous distribution to be modelled, granting regression the flexibility of classification models. While past studies stress the benefit of such flexible regression models in terms of performance, here we study the effect of the model choice on uncertainty estimation. We highlight that under model misspecification, aleatoric uncertainty is not properly captured, and that a Bayesian treatment of a misspecified model leads to unreliable epistemic uncertainty estimates. Overall, our study provides an overview on how modelling choices in regression may influence uncertainty estimation and thus any downstream decision making process
Diagnosing and Augmenting Feature Representations in Correctional Inverse Reinforcement Learning
Robots have been increasingly better at doing tasks for humans by learning
from their feedback, but still often suffer from model misalignment due to
missing or incorrectly learned features. When the features the robot needs to
learn to perform its task are missing or do not generalize well to new
settings, the robot will not be able to learn the task the human wants and,
even worse, may learn a completely different and undesired behavior. Prior work
shows how the robot can detect when its representation is missing some feature
and can, thus, ask the human to be taught about the new feature; however, these
works do not differentiate between features that are completely missing and
those that exist but do not generalize to new environments. In the latter case,
the robot would detect misalignment and simply learn a new feature, leading to
an arbitrarily growing feature representation that can, in turn, lead to
spurious correlations and incorrect learning down the line. In this work, we
propose separating the two sources of misalignment: we propose a framework for
determining whether a feature the robot needs is incorrectly learned and does
not generalize to new environment setups vs. is entirely missing from the
robot's representation. Once we detect the source of error, we show how the
human can initiate the realignment process for the model: if the feature is
missing, we follow prior work for learning new features; however, if the
feature exists but does not generalize, we use data augmentation to expand its
training and, thus, complete the correction. We demonstrate the proposed
approach in experiments with a simulated 7DoF robot manipulator and physical
human corrections.Comment: 8 pages, 4 figure
Uncertainty Minimization in Robotic 3D Mapping Systems Operating in Dynamic Large-Scale Environments
This dissertation research is motivated by the potential and promise of 3D sensing technologies in safety and security applications. With specific focus on unmanned robotic mapping to aid clean-up of hazardous environments, under-vehicle inspection, automatic runway/pavement inspection and modeling of urban environments, we develop modular, multi-sensor, multi-modality robotic 3D imaging prototypes using localization/navigation hardware, laser range scanners and video cameras.
While deploying our multi-modality complementary approach to pose and structure recovery in dynamic real-world operating conditions, we observe several data fusion issues that state-of-the-art methodologies are not able to handle. Different bounds on the noise model of heterogeneous sensors, the dynamism of the operating conditions and the interaction of the sensing mechanisms with the environment introduce situations where sensors can intermittently degenerate to accuracy levels lower than their design specification. This observation necessitates the derivation of methods to integrate multi-sensor data considering sensor conflict, performance degradation and potential failure during operation.
Our work in this dissertation contributes the derivation of a fault-diagnosis framework inspired by information complexity theory to the data fusion literature. We implement the framework as opportunistic sensing intelligence that is able to evolve a belief policy on the sensors within the multi-agent 3D mapping systems to survive and counter concerns of failure in challenging operating conditions. The implementation of the information-theoretic framework, in addition to eliminating failed/non-functional sensors and avoiding catastrophic fusion, is able to minimize uncertainty during autonomous operation by adaptively deciding to fuse or choose believable sensors. We demonstrate our framework through experiments in multi-sensor robot state localization in large scale dynamic environments and vision-based 3D inference. Our modular hardware and software design of robotic imaging prototypes along with the opportunistic sensing intelligence provides significant improvements towards autonomous accurate photo-realistic 3D mapping and remote visualization of scenes for the motivating applications
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Foundation models are first pre-trained on vast unsupervised datasets and
then fine-tuned on labeled data. Reinforcement learning, notably from human
feedback (RLHF), can further align the network with the intended usage. Yet the
imperfections in the proxy reward may hinder the training and lead to
suboptimal results; the diversity of objectives in real-world tasks and human
opinions exacerbate the issue. This paper proposes embracing the heterogeneity
of diverse rewards by following a multi-policy strategy. Rather than focusing
on a single a priori reward, we aim for Pareto-optimal generalization across
the entire space of preferences. To this end, we propose rewarded soup, first
specializing multiple networks independently (one for each proxy reward) and
then interpolating their weights linearly. This succeeds empirically because we
show that the weights remain linearly connected when fine-tuned on diverse
rewards from a shared pre-trained initialization. We demonstrate the
effectiveness of our approach for text-to-text (summarization, Q&A, helpful
assistant, review), text-image (image captioning, text-to-image generation,
visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the
alignment of deep models, and how they interact with the world in all its
diversity
Towards Safe Artificial General Intelligence
The field of artificial intelligence has recently experienced a
number of breakthroughs thanks to progress in deep learning and
reinforcement learning. Computer algorithms now outperform humans
at Go, Jeopardy, image classification, and lip reading, and are
becoming very competent at driving cars and interpreting natural
language. The rapid development has led many to conjecture that
artificial intelligence with greater-than-human ability on a wide
range of tasks may not be far. This in turn raises concerns
whether we know how to control such systems, in case we were to
successfully build them.
Indeed, if humanity would find itself in conflict with a system
of much greater intelligence than itself, then human society
would likely lose. One way to make sure we avoid such a conflict
is to ensure that any future AI system with potentially
greater-than-human-intelligence has goals that are aligned with
the goals of the rest of humanity. For example, it should not
wish to kill humans or steal their resources.
The main focus of this thesis will therefore be goal alignment,
i.e. how to design artificially intelligent agents with goals
coinciding with the goals of their designers. Focus will mainly
be directed towards variants of reinforcement learning, as
reinforcement learning currently seems to be the most promising
path towards powerful artificial intelligence. We identify and
categorize goal misalignment problems in reinforcement learning
agents as designed today, and give examples of how these agents
may cause catastrophes in the future. We also suggest a number of
reasonably modest modifications that can be used to avoid or
mitigate each identified misalignment problem. Finally, we also
study various choices of decision algorithms, and conditions for
when a powerful reinforcement learning system will permit us to
shut it down.
The central conclusion is that while reinforcement learning
systems as designed today are inherently unsafe to scale to human
levels of intelligence, there are ways to potentially address
many of these issues without straying too far from the currently
so successful reinforcement learning paradigm. Much work remains
in turning the high-level proposals suggested in this thesis into
practical algorithms, however
Dynamic Data Assimilation
Data assimilation is a process of fusing data with a model for the singular purpose of estimating unknown variables. It can be used, for example, to predict the evolution of the atmosphere at a given point and time. This book examines data assimilation methods including Kalman filtering, artificial intelligence, neural networks, machine learning, and cognitive computing
Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability
A trustworthy reinforcement learning algorithm should be competent in solving
challenging real-world problems, including {robustly} handling uncertainties,
satisfying {safety} constraints to avoid catastrophic failures, and
{generalizing} to unseen scenarios during deployments. This study aims to
overview these main perspectives of trustworthy reinforcement learning
considering its intrinsic vulnerabilities on robustness, safety, and
generalizability. In particular, we give rigorous formulations, categorize
corresponding methodologies, and discuss benchmarks for each perspective.
Moreover, we provide an outlook section to spur promising future directions
with a brief discussion on extrinsic vulnerabilities considering human
feedback. We hope this survey could bring together separate threads of studies
together in a unified framework and promote the trustworthiness of
reinforcement learning.Comment: 36 pages, 5 figure
- …