Bilevel Parameter Learning for Higher-Order Total Variation Regularisation Models.

We consider a bilevel optimisation approach for parameter learning in higher-order total variation image reconstruction models. Apart from the least squares cost functional, naturally used in bilevel learning, we propose and analyse an alternative cost based on a Huber-regularised TV seminorm. Differentiability properties of the solution operator are verified and a first-order optimality system is derived. Based on the adjoint information, a combined quasi-Newton/semismooth Newton algorithm is proposed for the numerical solution of the bilevel problems. Numerical experiments are carried out to show the suitability of our approach and the improved performance of the new cost functional. Thanks to the bilevel optimisation framework, also a detailed comparison between TGV 2 and ICTV is carried out, showing the advantages and shortcomings of both regularisers, depending on the structure of the processed images and their noise level.King Abdullah University of Science and Technology (KAUST) (Grant ID: KUKI1-007-43), Engineering and Physical Sciences Research Council (Grant IDs: Nr. EP/J009539/1 “Sparse & Higher-order Image Restoration” and Nr. EP/M00483X/1 “Efficient computational tools for inverse imaging problems”), Escuela Politécnica Nacional de Quito (Grant ID: PIS 12-14, MATHAmSud project SOCDE “Sparse Optimal Control of Differential Equations”), Leverhulme Trust (project on “Breaking the non-convexity barrier”), SENESCYT (Ecuadorian Ministry of Higher Education, Science, Technology and Innovation) (Prometeo Fellowship)This is the final version of the article. It first appeared from Springer via http://dx.doi.org/10.1007/s10851-016-0662-

Continual Invariant Risk Minimization

Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations. IRM was introduced by Arjovsky et al. (2019) and extended by Ahuja et al. (2020). IRM assumes that all environments are available to the learning system at the same time. With this work, we generalize the concept of IRM to scenarios where environments are observed sequentially. We show that existing approaches, including those designed for continual learning, fail to identify the invariant features and models across sequentially presented environments. We extend IRM under a variational Bayesian and bilevel framework, creating a general approach to continual invariant risk minimization. We also describe a strategy to solve the optimization problems using a variant of the alternating direction method of multiplier (ADMM). We show empirically using multiple datasets and with multiple sequential environments that the proposed methods outperform or is competitive with prior approaches.Comment: Shorter version of this paper was presented at RobustML workshop of ICLR 202