The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification

We improve the effectiveness of propagation- and linear-optimization-based neural network verification algorithms with a new tightened convex relaxation for ReLU neurons. Unlike previous single-neuron relaxations which focus only on the univariate input space of the ReLU, our method considers the multivariate input space of the affine pre-activation function preceding the ReLU. Using results from submodularity and convex geometry, we derive an explicit description of the tightest possible convex relaxation when this multivariate input is over a box domain. We show that our convex relaxation is significantly stronger than the commonly used univariate-input relaxation which has been proposed as a natural convex relaxation barrier for verification. While our description of the relaxation may require an exponential number of inequalities, we show that they can be separated in linear time and hence can be efficiently incorporated into optimization algorithms on an as-needed basis. Based on this novel relaxation, we design two polynomial-time algorithms for neural network verification: a linear-programming-based algorithm that leverages the full power of our relaxation, and a fast propagation algorithm that generalizes existing approaches. In both cases, we show that for a modest increase in computational effort, our strengthened relaxation enables us to verify a significantly larger number of instances compared to similar algorithms

Geometrische Interpretationen und Algorithmische Verifikation von exakten Lösungen in Compressed Sensing

In an era dominated by the topic big data, in which everyone is confronted with spying scandals, personalized advertising, and retention of data, it is not surprising that a topic as compressed sensing is of such a great interest. Further the field of compressed sensing is very interesting for problems in signal- and image processing. Similarly, the question arises how many measurements are necessarily required to capture and represent high-resolution signal or objects. In the thesis at hand, the applicability of three of the most applied optimization problems with linear restrictions in compressed sensing is studied. These are basis pursuit, analysis l1-minimization und isotropic total variation minimization. Unique solutions of basis pursuit and analysis l1-minimization are considered and, on the basis of their characterizations, methods are designed which verify whether a given vector can be reconstructed exactly by basis pursuit or analysis l1-minimization. Further, a method is developed which guarantees that a given vector is the unique solution of isotropic total variation minimization. In addition, results on experiments for all three methods are presented where the linear restrictions are given as a random matrix and as a matrix which models the measurement process in computed tomography. Furthermore, in the present thesis geometrical interpretations are presented. By considering the theory of convex polytopes, three geometrical objects are examined and placed within the context of compressed sensing. The result is a comprehensive study of the geometry of basis pursuit which contains many new insights to necessary geometrical conditions for unique solutions and an explicit number of equivalence classes of unique solutions. The number of these equivalence classes itself is strongly related to the number of unique solutions of basis pursuit for an arbitrary matrix. In addition, the question is addressed for which linear restrictions do exist the most unique solutions of basis pursuit. For this purpose, upper bounds are developed and explicit restrictions are given under which the most vectors can be reconstructed via basis pursuit.In Zeiten von Big Data, in denen man nahezu täglich mit Überwachungsskandalen, personalisierter Werbung und Vorratsdatenspeicherung konfrontiert wird, ist es kein Wunder dass ein Forschungsgebiet wie Compressed Sensing von so grossem Interesse ist. Es stellt sich die Frage, wie viele Messungen tatsächlich nötig sind, um ein Signal oder ein Objekt hochaufgelöst darstellen zu können. In der vorliegenden Arbeit wird die Anwendungsmöglichkeit von drei in Compressed Sensing verwendeten Optimierungsprobleme mit linearen Nebenbedingungen untersucht. Hierbei handelt es sich namentlich um Basis Pursuit, Analysis l1-Minimierung und Isotropic Total Variation. Es werden eindeutige Lösungen von Basis Pursuit und der Analysis l1-Minimierung betrachtet, um auf der Grundlage ihrer Charakterisierungen Methoden vorzustellen, die Verifizieren ob ein gegebener Vektor exakt durch Basis Pursuit oder der Analysis l1-Minimierung rekonstruiert werden kann. Für Isotropic Total Variation werden hinreichende Bedingungen aufgestellt, die garantieren, dass ein gegebener Vektor die eindeutige Lösung von Isotropic Total Variation ist. Darüber hinaus werden Ergebnisse zu Experimenten mit Zufallsmatrizen als linearen Nebenbedingungen sowie Ergebnisse zu Experimenten mit Matrizen vorgestellt, die den Aufnahmeprozess bei Computertomographie simulieren. Weiterhin werden in der vorliegenden Arbeit verschiedene geometrische Interpretationen von Basis Pursuit vorgestellt. Unter Verwendung der konvexen Polytop-Theorie werden drei unterschiedliche geometrische Objekte untersucht und in den Zusammenhang mit Compressed Sensing gestellt. Das Ergebnis ist eine umfangreiche Studie der Geometrie von Basis Pursuit mit vielen neuen Einblicken in notwendige geometrische Bedingungen für eindeutige Lösungen und in die explizite Anzahl von Äquivalenzklassen eindeutiger Lösungen. Darüber hinaus wird der Frage nachgegangen, unter welchen linearen Nebenbedingungen die meisten eindeutigen Lösungen existieren. Zu diesem Zweck werden obere Schranken entwickelt, sowie explizite Nebenbedingungen genannt unter denen die meisten Vektoren exakt rekonstruiert werden können

Markov decision processes with uncertain parameters

Markov decision processes model stochastic uncertainty in systems and allow one to construct strategies which optimize the behaviour of a system with respect to some reward function. However, the parameters for this uncertainty, that is, the probabilities inside a Markov decision model, are derived from empirical or expert knowledge and are themselves subject to uncertainties such as measurement errors or limited expertise. This work considers second-order uncertainty models for Markov decision processes and derives theoretical and practical results. Among other models, this work considers two main forms of uncertainty. One form is a set of discrete scenarios with a prior probability distribution and the task to maximize the expected reward under the given probability distribution. Another form of uncertainty is a continuous uncertainty set of scenarios and the task to compute a policy that optimizes the rewards in the optimistic and pessimistic cases. The work provides two kinds of results. First, we establish complexity-theoretic hardness results for the considered optimization problems. Second, we design heuristics for some of the problems and evaluate them empirically. In the first class of results, we show that additional model uncertainty makes the optimization problems harder to solve, as they add an additional party with own optimization goals. In the second class of results, we show that even if the discussed problems are hard to solve in theory, we can come up with efficient heuristics that can solve them adequately well for practical applications

Representational Capabilities of Feed-forward and Sequential Neural Architectures

Despite the widespread empirical success of deep neural networks over the past decade, a comprehensive understanding of their mathematical properties remains elusive, which limits the abilities of practitioners to train neural networks in a principled manner. This dissertation provides a representational characterization of a variety of neural network architectures, including fully-connected feed-forward networks and sequential models like transformers. The representational capabilities of neural networks are most famously characterized by the universal approximation theorem, which states that sufficiently large neural networks can closely approximate any well-behaved target function. However, the universal approximation theorem applies exclusively to two-layer neural networks of unbounded size and fails to capture the comparative strengths and weaknesses of different architectures. The thesis addresses these limitations by quantifying the representational consequences of random features, weight regularization, and model depth on feed-forward architectures. It further investigates and contrasts the expressive powers of transformers and other sequential neural architectures. Taken together, these results apply a wide range of theoretical tools—including approximation theory, discrete dynamical systems, and communication complexity—to prove rigorous separations between different neural architectures and scaling regimes