On Implicit Bias in Overparameterized Bilevel Optimization

Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems consist of two nested sub-problems, called the outer and inner problems, respectively. In practice, often at least one of these sub-problems is overparameterized. In this case, there are many ways to choose among optima that achieve equivalent objective values. Inspired by recent studies of the implicit bias induced by optimization algorithms in single-level optimization, we investigate the implicit bias of gradient-based algorithms for bilevel optimization. We delineate two standard BLO methods -- cold-start and warm-start -- and show that the converged solution or long-run behavior depends to a large degree on these and other algorithmic choices, such as the hypergradient approximation. We also show that the inner solutions obtained by warm-start BLO can encode a surprising amount of information about the outer objective, even when the outer parameters are low-dimensional. We believe that implicit bias deserves as central a role in the study of bilevel optimization as it has attained in the study of single-level neural net optimization.Comment: ICML 202

An investigation of iterated multi-agent belief change

Multi-agent belief change is an area concerned with the belief dynamics of a network of communicating agents. A network is represented by a graph, where vertices represent agents that share information via a process of minimizing disagreements between themselves. Previous work by Delgrande, Lang, and Schaub addressed belief change through global minimization, with a weak notion of distance between agents. We extend it by applying iterative procedures that take distance into account. We have identified two approaches to iteration: in the first, a vertex incorporates information from its immediate neighbours only; in the second, a vertex incorporates information from progressively more distant neighbours. Our research has both theoretical and practical contributions: first, we define the iterative approaches, find relationships between them, and investigate their logical properties; then, we introduce a software system called Equibel that implements both the global and iterative approaches, using Answer Set Programming and Python

An Implementation of Consistency-Based Multi-Agent Belief Change using ASP

To appearInternational audienceThis paper presents an implementation of a general frameworkfor consistency-based belief change using Answer Set Programming(ASP). We describe Equibel, a software system for working with beliefchange operations on arbitrary graph topologies. The system has an ASPcomponent that performs a core maximization procedure, and a Pythoncomponent that performs additional processing on the output of theASP solver. The Python component also provides an interactive interfacethat allows users to create a graph, set formulas at nodes, performbelief change operations, and query the resulting graph