199,585 research outputs found
Evaluating Agents using Social Choice Theory
We argue that many general evaluation problems can be viewed through the lens
of voting theory. Each task is interpreted as a separate voter, which requires
only ordinal rankings or pairwise comparisons of agents to produce an overall
evaluation. By viewing the aggregator as a social welfare function, we are able
to leverage centuries of research in social choice theory to derive principled
evaluation frameworks with axiomatic foundations. These evaluations are
interpretable and flexible, while avoiding many of the problems currently
facing cross-task evaluation. We apply this Voting-as-Evaluation (VasE)
framework across multiple settings, including reinforcement learning, large
language models, and humans. In practice, we observe that VasE can be more
robust than popular evaluation frameworks (Elo and Nash averaging), discovers
properties in the evaluation data not evident from scores alone, and can
predict outcomes better than Elo in a complex seven-player game. We identify
one particular approach, maximal lotteries, that satisfies important
consistency properties relevant to evaluation, is computationally efficient
(polynomial in the size of the evaluation data), and identifies game-theoretic
cycles
From source to target and back: symmetric bi-directional adaptive GAN
The effectiveness of generative adversarial approaches in producing images
according to a specific style or visual domain has recently opened new
directions to solve the unsupervised domain adaptation problem. It has been
shown that source labeled images can be modified to mimic target samples making
it possible to train directly a classifier in the target domain, despite the
original lack of annotated data. Inverse mappings from the target to the source
domain have also been evaluated but only passing through adapted feature
spaces, thus without new image generation. In this paper we propose to better
exploit the potential of generative adversarial networks for adaptation by
introducing a novel symmetric mapping among domains. We jointly optimize
bi-directional image transformations combining them with target self-labeling.
Moreover we define a new class consistency loss that aligns the generators in
the two directions imposing to conserve the class identity of an image passing
through both domain mappings. A detailed qualitative and quantitative analysis
of the reconstructed images confirm the power of our approach. By integrating
the two domain specific classifiers obtained with our bi-directional network we
exceed previous state-of-the-art unsupervised adaptation results on four
different benchmark datasets
- …