3 research outputs found
Heterogeneous Value Evaluation for Large Language Models
The emergent capabilities of Large Language Models (LLMs) have made it
crucial to align their values with those of humans. Current methodologies
typically attempt alignment with a homogeneous human value and requires human
verification, yet lack consensus on the desired aspect and depth of alignment
and resulting human biases. In this paper, we propose A2EHV, an Automated
Alignment Evaluation with a Heterogeneous Value system that (1) is automated to
minimize individual human biases, and (2) allows assessments against various
target values to foster heterogeneous agents. Our approach pivots on the
concept of value rationality, which represents the ability for agents to
execute behaviors that satisfy a target value the most. The quantification of
value rationality is facilitated by the Social Value Orientation framework from
social psychology, which partitions the value space into four categories to
assess social preferences from agents' behaviors. We evaluate the value
rationality of eight mainstream LLMs and observe that large models are more
inclined to align neutral values compared to those with strong personal values.
By examining the behavior of these LLMs, we contribute to a deeper
understanding of value alignment within a heterogeneous value system.Comment: Our full prompts are released in the repo:
https://github.com/zowiezhang/A2E
MetaGPT: Meta Programming for Multi-Agent Collaborative Framework
Recently, remarkable progress has been made in automated task-solving through
the use of multi-agents driven by large language models (LLMs). However,
existing works primarily focuses on simple tasks lacking exploration and
investigation in complicated tasks mainly due to the hallucination problem.
This kind of hallucination gets amplified infinitely as multiple intelligent
agents interact with each other, resulting in failures when tackling
complicated problems.Therefore, we introduce MetaGPT, an innovative framework
that infuses effective human workflows as a meta programming approach into
LLM-driven multi-agent collaboration. In particular, MetaGPT first encodes
Standardized Operating Procedures (SOPs) into prompts, fostering structured
coordination. And then, it further mandates modular outputs, bestowing agents
with domain expertise paralleling human professionals to validate outputs and
reduce compounded errors. In this way, MetaGPT leverages the assembly line work
model to assign diverse roles to various agents, thus establishing a framework
that can effectively and cohesively deconstruct complex multi-agent
collaborative problems. Our experiments conducted on collaborative software
engineering tasks illustrate MetaGPT's capability in producing comprehensive
solutions with higher coherence relative to existing conversational and
chat-based multi-agent systems. This underscores the potential of incorporating
human domain knowledge into multi-agents, thus opening up novel avenues for
grappling with intricate real-world challenges. The GitHub repository of this
project is made publicly available on: https://github.com/geekan/MetaGP
Data-Adaptive M-Estimators for Robust Regression via Bi-Level Optimization
M-estimators are widely used in robust regression to handle heavy-tailed data corrupted by outliers. Although they have been applied to a plethora of real scenarios, it remains a challenge to practitioners how to set the tuning parameters. Often, it is set by manual tuning or according to the asymptotic efficiency rule, being sub-optimal for a real dataset with finite size. In this paper, we explore a data-driven paradigm where the optimal tuning parameters are determined by the dataset itself. Specifically, we treat the tuning parameters as hyper-parameters in robust regression, formulate the tuning problem via a novel bi-level optimization framework, and solve the regression model parameters and the tuning parameters in a joint manner. To solve this problem efficiently, especially when using neural network as the regression model, we further employ an online approximation strategy to iteratively optimize the model parameters and the tuning parameters with a proven sub-linear convergence rate. Moreover, our proposed framework is generic for any parametric regression model and M-estimator with differentiable loss function. We instantiate this framework with two popular M-estimators (Huber’s and Tukey’s) and derive the corresponding data-adaptive M-estimators. In the experiment, we present positive simulation results compared with various salient benchmarks