4 research outputs found
Measuring Value Understanding in Language Models through Discriminator-Critique Gap
Recent advancements in Large Language Models (LLMs) have heightened concerns
about their potential misalignment with human values. However, evaluating their
grasp of these values is complex due to their intricate and adaptable nature.
We argue that truly understanding values in LLMs requires considering both
"know what" and "know why". To this end, we present the Value Understanding
Measurement (VUM) framework that quantitatively assesses both "know what" and
"know why" by measuring the discriminator-critique gap related to human values.
Using the Schwartz Value Survey, we specify our evaluation values and develop a
thousand-level dialogue dataset with GPT-4. Our assessment looks at both the
value alignment of LLM's outputs compared to baseline answers and how LLM
responses align with reasons for value recognition versus GPT-4's annotations.
We evaluate five representative LLMs and provide strong evidence that the
scaling law significantly impacts "know what" but not much on "know why", which
has consistently maintained a high level. This may further suggest that LLMs
might craft plausible explanations based on the provided context without truly
understanding their inherent value, indicating potential risks
Zero-shot Preference Learning for Offline RL via Optimal Transport
Preference-based Reinforcement Learning (PbRL) has demonstrated remarkable
efficacy in aligning rewards with human intentions. However, a significant
challenge lies in the need of substantial human labels, which is costly and
time-consuming. Additionally, the expensive preference data obtained from prior
tasks is not typically reusable for subsequent task learning, leading to
extensive labeling for each new task. In this paper, we propose a novel
zero-shot preference-based RL algorithm that leverages labeled preference data
from source tasks to infer labels for target tasks, eliminating the requirement
for human queries. Our approach utilizes Gromov-Wasserstein distance to align
trajectory distributions between source and target tasks. The solved optimal
transport matrix serves as a correspondence between trajectories of two tasks,
making it possible to identify corresponding trajectory pairs between tasks and
transfer the preference labels. However, learning directly from inferred labels
that contains a fraction of noisy labels will result in an inaccurate reward
function, subsequently affecting policy performance. To this end, we introduce
Robust Preference Transformer, which models the rewards as Gaussian
distributions and incorporates reward uncertainty in addition to reward mean.
The empirical results on robotic manipulation tasks of Meta-World and Robomimic
show that our method has strong capabilities of transferring preferences
between tasks and learns reward functions from noisy labels robustly.
Furthermore, we reveal that our method attains near-oracle performance with a
small proportion of scripted labels
PiCor: Multi-Task Deep Reinforcement Learning with Policy Correction
Multi-task deep reinforcement learning (DRL) ambitiously aims to train a general agent that masters multiple tasks simultaneously. However, varying learning speeds of different tasks compounding with negative gradients interference makes policy learning inefficient. In this work, we propose PiCor, an efficient multi-task DRL framework that splits learning into policy optimization and policy correction phases. The policy optimization phase improves the policy by any DRL algothrim on the sampled single task without considering other tasks. The policy correction phase first constructs an adaptive adjusted performance constraint set. Then the intermediate policy learned by the first phase is constrained to the set, which controls the negative interference and balances the learning speeds across tasks. Empirically, we demonstrate that PiCor outperforms previous methods and significantly improves sample efficiency on simulated robotic manipulation and continuous control tasks. We additionally show that adaptive weight adjusting can further improve data efficiency and performance