Heterogeneous Value Evaluation for Large Language Models

The emergent capabilities of Large Language Models (LLMs) have made it crucial to align their values with those of humans. Current methodologies typically attempt alignment with a homogeneous human value and requires human verification, yet lack consensus on the desired aspect and depth of alignment and resulting human biases. In this paper, we propose A2EHV, an Automated Alignment Evaluation with a Heterogeneous Value system that (1) is automated to minimize individual human biases, and (2) allows assessments against various target values to foster heterogeneous agents. Our approach pivots on the concept of value rationality, which represents the ability for agents to execute behaviors that satisfy a target value the most. The quantification of value rationality is facilitated by the Social Value Orientation framework from social psychology, which partitions the value space into four categories to assess social preferences from agents' behaviors. We evaluate the value rationality of eight mainstream LLMs and observe that large models are more inclined to align neutral values compared to those with strong personal values. By examining the behavior of these LLMs, we contribute to a deeper understanding of value alignment within a heterogeneous value system.Comment: Our full prompts are released in the repo: https://github.com/zowiezhang/A2E

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework

Recently, remarkable progress has been made in automated task-solving through the use of multi-agents driven by large language models (LLMs). However, existing works primarily focuses on simple tasks lacking exploration and investigation in complicated tasks mainly due to the hallucination problem. This kind of hallucination gets amplified infinitely as multiple intelligent agents interact with each other, resulting in failures when tackling complicated problems.Therefore, we introduce MetaGPT, an innovative framework that infuses effective human workflows as a meta programming approach into LLM-driven multi-agent collaboration. In particular, MetaGPT first encodes Standardized Operating Procedures (SOPs) into prompts, fostering structured coordination. And then, it further mandates modular outputs, bestowing agents with domain expertise paralleling human professionals to validate outputs and reduce compounded errors. In this way, MetaGPT leverages the assembly line work model to assign diverse roles to various agents, thus establishing a framework that can effectively and cohesively deconstruct complex multi-agent collaborative problems. Our experiments conducted on collaborative software engineering tasks illustrate MetaGPT's capability in producing comprehensive solutions with higher coherence relative to existing conversational and chat-based multi-agent systems. This underscores the potential of incorporating human domain knowledge into multi-agents, thus opening up novel avenues for grappling with intricate real-world challenges. The GitHub repository of this project is made publicly available on: https://github.com/geekan/MetaGP

Data-Adaptive M-Estimators for Robust Regression via Bi-Level Optimization

M-estimators are widely used in robust regression to handle heavy-tailed data corrupted by outliers. Although they have been applied to a plethora of real scenarios, it remains a challenge to practitioners how to set the tuning parameters. Often, it is set by manual tuning or according to the asymptotic efficiency rule, being sub-optimal for a real dataset with finite size. In this paper, we explore a data-driven paradigm where the optimal tuning parameters are determined by the dataset itself. Specifically, we treat the tuning parameters as hyper-parameters in robust regression, formulate the tuning problem via a novel bi-level optimization framework, and solve the regression model parameters and the tuning parameters in a joint manner. To solve this problem efficiently, especially when using neural network as the regression model, we further employ an online approximation strategy to iteratively optimize the model parameters and the tuning parameters with a proven sub-linear convergence rate. Moreover, our proposed framework is generic for any parametric regression model and M-estimator with differentiable loss function. We instantiate this framework with two popular M-estimators (Huber’s and Tukey’s) and derive the corresponding data-adaptive M-estimators. In the experiment, we present positive simulation results compared with various salient benchmarks