101 research outputs found
Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers
Building on recent progress at the intersection of combinatorial optimization
and deep learning, we propose an end-to-end trainable architecture for deep
graph matching that contains unmodified combinatorial solvers. Using the
presence of heavily optimized combinatorial solvers together with some
improvements in architecture design, we advance state-of-the-art on deep graph
matching benchmarks for keypoint correspondence. In addition, we highlight the
conceptual advantages of incorporating solvers into deep learning
architectures, such as the possibility of post-processing with a strong
multi-graph matching solver or the indifference to changes in the training
setting. Finally, we propose two new challenging experimental setups. The code
is available at https://github.com/martius-lab/blackbox-deep-graph-matchingComment: ECCV 2020 conference pape
Backpropagation through Combinatorial Algorithms: Identity with Projection Works
Embedding discrete solvers as differentiable layers has given modern deep
learning architectures combinatorial expressivity and discrete reasoning
capabilities. The derivative of these solvers is zero or undefined, therefore a
meaningful replacement is crucial for effective gradient-based learning. Prior
works rely on smoothing the solver with input perturbations, relaxing the
solver to continuous problems, or interpolating the loss landscape with
techniques that typically require additional solver calls, introduce extra
hyper-parameters, or compromise performance. We propose a principled approach
to exploit the geometry of the discrete solution space to treat the solver as a
negative identity on the backward pass and further provide a theoretical
justification. Our experiments demonstrate that such a straightforward
hyper-parameter-free approach is able to compete with previous more complex
methods on numerous experiments such as backpropagation through discrete
samplers, deep graph matching, and image retrieval. Furthermore, we substitute
the previously proposed problem-specific and label-dependent margin with a
generic regularization procedure that prevents cost collapse and increases
robustness.Comment: The first two authors contributed equall
Rethinking and Benchmarking Predict-then-Optimize Paradigm for Combinatorial Optimization Problems
Numerous web applications rely on solving combinatorial optimization
problems, such as energy cost-aware scheduling, budget allocation on web
advertising, and graph matching on social networks. However, many optimization
problems involve unknown coefficients, and improper predictions of these
factors may lead to inferior decisions which may cause energy wastage,
inefficient resource allocation, inappropriate matching in social networks,
etc. Such a research topic is referred to as "Predict-Then-Optimize (PTO)"
which considers the performance of prediction and decision-making in a unified
system. A noteworthy recent development is the end-to-end methods by directly
optimizing the ultimate decision quality which claims to yield better results
in contrast to the traditional two-stage approach. However, the evaluation
benchmarks in this field are fragmented and the effectiveness of various models
in different scenarios remains unclear, hindering the comprehensive assessment
and fast deployment of these methods. To address these issues, we provide a
comprehensive categorization of current approaches and integrate existing
experimental scenarios to establish a unified benchmark, elucidating the
circumstances under which end-to-end training yields improvements, as well as
the contexts in which it performs ineffectively. We also introduce a new
dataset for the industrial combinatorial advertising problem for inclusive
finance to open-source. We hope the rethinking and benchmarking of PTO could
facilitate more convenient evaluation and deployment, and inspire further
improvements both in the academy and industry within this field
Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach
We propose a fully differentiable architecture for simultaneous semantic and
instance segmentation (a.k.a. panoptic segmentation) consisting of a
convolutional neural network and an asymmetric multiway cut problem solver. The
latter solves a combinatorial optimization problem that elegantly incorporates
semantic and boundary predictions to produce a panoptic labeling. Our
formulation allows to directly maximize a smooth surrogate of the panoptic
quality metric by backpropagating the gradient through the optimization
problem. Experimental evaluation shows improvement by backpropagating through
the optimization problem w.r.t. comparable approaches on Cityscapes and COCO
datasets. Overall, our approach shows the utility of using combinatorial
optimization in tandem with deep learning in a challenging large scale
real-world problem and showcases benefits and insights into training such an
architecture.Comment: To be presented at NeurIPS 202
Unsupervised Deep Graph Matching Based on Cycle Consistency
We contribute to the sparsely populated area of unsupervised deep graph
matching with application to keypoint matching in images. Contrary to the
standard \emph{supervised} approach, our method does not require ground truth
correspondences between keypoint pairs. Instead, it is self-supervised by
enforcing consistency of matchings between images of the same object category.
As the matching and the consistency loss are discrete, their derivatives cannot
be straightforwardly used for learning. We address this issue in a principled
way by building our method upon the recent results on black-box differentiation
of combinatorial solvers. This makes our method exceptionally flexible, as it
is compatible with arbitrary network architectures and combinatorial solvers.
Our experimental evaluation suggests that our technique sets a new
state-of-the-art for unsupervised graph matching.Comment: 12 pages, 5 figures, 3 paper
Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities
Decision-focused learning (DFL) is an emerging paradigm in machine learning
which trains a model to optimize decisions, integrating prediction and
optimization in an end-to-end system. This paradigm holds the promise to
revolutionize decision-making in many real-world applications which operate
under uncertainty, where the estimation of unknown parameters within these
decision models often becomes a substantial roadblock. This paper presents a
comprehensive review of DFL. It provides an in-depth analysis of the various
techniques devised to integrate machine learning and optimization models,
introduces a taxonomy of DFL methods distinguished by their unique
characteristics, and conducts an extensive empirical evaluation of these
methods proposing suitable benchmark dataset and tasks for DFL. Finally, the
study provides valuable insights into current and potential future avenues in
DFL research.Comment: Experimental Survey and Benchmarkin
QuAnt: Quantum Annealing with Learnt Couplings
Modern quantum annealers can find high-quality solutions to combinatorialoptimisation objectives given as quadratic unconstrained binary optimisation(QUBO) problems. Unfortunately, obtaining suitable QUBO forms in computervision remains challenging and currently requires problem-specific analyticalderivations. Moreover, such explicit formulations impose tangible constraintson solution encodings. In stark contrast to prior work, this paper proposes tolearn QUBO forms from data through gradient backpropagation instead of derivingthem. As a result, the solution encodings can be chosen flexibly and compactly.Furthermore, our methodology is general and virtually independent of thespecifics of the target problem type. We demonstrate the advantages of learntQUBOs on the diverse problem types of graph matching, 2D point cloud alignmentand 3D rotation estimation. Our results are competitive with the previousquantum state of the art while requiring much fewer logical and physicalqubits, enabling our method to scale to larger problems. The code and the newdataset will be open-sourced.<br
LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching
Obtaining large pre-trained models that can be fine-tuned to new tasks with
limited annotated samples has remained an open challenge for medical imaging
data. While pre-trained deep networks on ImageNet and vision-language
foundation models trained on web-scale data are prevailing approaches, their
effectiveness on medical tasks is limited due to the significant domain shift
between natural and medical images. To bridge this gap, we introduce LVM-Med,
the first family of deep networks trained on large-scale medical datasets. We
have collected approximately 1.3 million medical images from 55 publicly
available datasets, covering a large number of organs and modalities such as
CT, MRI, X-ray, and Ultrasound. We benchmark several state-of-the-art
self-supervised algorithms on this dataset and propose a novel self-supervised
contrastive learning algorithm using a graph-matching formulation. The proposed
approach makes three contributions: (i) it integrates prior pair-wise image
similarity metrics based on local and global information; (ii) it captures the
structural constraints of feature embeddings through a loss function
constructed via a combinatorial graph-matching objective; and (iii) it can be
trained efficiently end-to-end using modern gradient-estimation techniques for
black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream
medical tasks ranging from segmentation and classification to object detection,
and both for the in and out-of-distribution settings. LVM-Med empirically
outperforms a number of state-of-the-art supervised, self-supervised, and
foundation models. For challenging tasks such as Brain Tumor Classification or
Diabetic Retinopathy Grading, LVM-Med improves previous vision-language models
trained on 1 billion masks by 6-7% while using only a ResNet-50.Comment: Update Appendi
- …