On Correctness of Automatic Differentiation for Non-Differentiable Functions

Differentiation lies at the core of many machine-learning algorithms, and is well-supported by popular autodiff systems, such as TensorFlow and PyTorch. Originally, these systems have been developed to compute derivatives of differentiable functions, but in practice, they are commonly applied to functions with non-differentiabilities. For instance, neural networks using ReLU define non-differentiable functions in general, but the gradients of losses involving those functions are computed using autodiff systems in practice. This status quo raises a natural question: are autodiff systems correct in any formal sense when they are applied to such non-differentiable functions? In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in often-used informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly non-differentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions

BOOSTING THERMAL STABILITY OF PEROVSKITE SOLAR CELLS BY INTRODUCING NEW HTL

2

On Correctness of Automatic Differentiation for Non-Differentiable Functions

International audienceDifferentiation lies at the core of many machine-learning algorithms, and is wellsupported by popular autodiff systems, such as TensorFlow and PyTorch. Originally, these systems have been developed to compute derivatives of differentiable functions, but in practice, they are commonly applied to functions with non-differentiabilities. For instance, neural networks using ReLU define nondifferentiable functions in general, but the gradients of losses involving those functions are computed using autodiff systems in practice. This status quo raises a natural question: are autodiff systems correct in any formal sense when they are applied to such non-differentiable functions? In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in oftenused informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly nondifferentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions

열 소산을 통해 높은 열 안정성을 보이는 페로브스카이트 태양전지

2

Towards Verified Stochastic Variational Inference for Probabilistic Programs

International audienceProbabilistic programming is the idea of writing models from statistics and machine learning using program notations and reasoning about these models using generic inference engines. Recently its combination with deep learning has been explored intensely, which led to the development of so called deep probabilistic programming languages, such as Pyro, Edward and ProbTorch. At the core of this development lie inference engines based on stochastic variational inference algorithms. When asked to find information about the posterior distribution of a model written in such a language, these algorithms convert this posterior-inference query into an optimisation problem and solve it approximately by a form of gradient ascent or descent. In this paper, we analyse one of the most fundamental and versatile variational inference algorithms, called score estimator or REINFORCE, using tools from denotational semantics and program analysis. We formally express what this algorithm does on models denoted by programs, and expose implicit assumptions made by the algorithm on the models. The violation of these assumptions may lead to an undefined optimisation objective or the loss of convergence guarantee of the optimisation process. We then describe rules for proving these assumptions, which can be automated by static program analyses. Some of our rules use nontrivial facts from continuous mathematics, and let us replace requirements about integrals in the assumptions, such as integrability of functions defined in terms of programs' denotations, by conditions involving differentiation or boundedness, which are much easier to prove automatically (and manually). Following our general methodology, we have developed a static program analysis for the Pyro programming language that aims at discharging the assumption about what we call model-guide support match. Our analysis is applied to the eight representative model-guide pairs from the Pyro webpage, which include sophisticated neural network models such as AIR. It finds a bug in one of these cases, reveals a non-standard use of an inference engine in another, and shows that the assumptions are met in the remaining six cases

Search for intermediate-mass black hole binaries in the third observing run of Advanced LIGO and Advanced Virgo

International audienceIntermediate-mass black holes (IMBHs) span the approximate mass range 100−105 M⊙, between black holes (BHs) that formed by stellar collapse and the supermassive BHs at the centers of galaxies. Mergers of IMBH binaries are the most energetic gravitational-wave sources accessible by the terrestrial detector network. Searches of the first two observing runs of Advanced LIGO and Advanced Virgo did not yield any significant IMBH binary signals. In the third observing run (O3), the increased network sensitivity enabled the detection of GW190521, a signal consistent with a binary merger of mass ∼150 M⊙ providing direct evidence of IMBH formation. Here, we report on a dedicated search of O3 data for further IMBH binary mergers, combining both modeled (matched filter) and model-independent search methods. We find some marginal candidates, but none are sufficiently significant to indicate detection of further IMBH mergers. We quantify the sensitivity of the individual search methods and of the combined search using a suite of IMBH binary signals obtained via numerical relativity, including the effects of spins misaligned with the binary orbital axis, and present the resulting upper limits on astrophysical merger rates. Our most stringent limit is for equal mass and aligned spin BH binary of total mass 200 M⊙ and effective aligned spin 0.8 at 0.056 Gpc−3 yr−1 (90% confidence), a factor of 3.5 more constraining than previous LIGO-Virgo limits. We also update the estimated rate of mergers similar to GW190521 to 0.08 Gpc−3 yr−1.Key words: gravitational waves / stars: black holes / black hole physicsCorresponding author: W. Del Pozzo, e-mail: [email protected]† Deceased, August 2020