Speeding Up BatchBALD: A k-BALD Family of Approximations for Active Learning

Active learning is a powerful method for training machine learning models with limited labeled data. One commonly used technique for active learning is BatchBALD, which uses Bayesian neural networks to find the most informative points to label in a pool set. However, BatchBALD can be very slow to compute, especially for larger datasets. In this paper, we propose a new approximation, k-BALD, which uses k-wise mutual information terms to approximate BatchBALD, making it much less expensive to compute. Results on the MNIST dataset show that k-BALD is significantly faster than BatchBALD while maintaining similar performance. Additionally, we also propose a dynamic approach for choosing k based on the quality of the approximation, making it more efficient for larger datasets.Comment: 5 pages, workshop preprin

Black-Box Batch Active Learning for Regression

Batch active learning is a popular approach for efficiently training machine learning models on large, initially unlabelled datasets by repeatedly acquiring labels for batches of data points. However, many recent batch active learning methods are white-box approaches and are often limited to differentiable parametric models: they score unlabeled points using acquisition functions based on model embeddings or first- and second-order derivatives. In this paper, we propose black-box batch active learning for regression tasks as an extension of white-box approaches. Crucially, our method only relies on model predictions. This approach is compatible with a wide range of machine learning models, including regular and Bayesian deep learning models and non-differentiable models such as random forests. It is rooted in Bayesian principles and utilizes recent kernel-based approaches. This allows us to extend a wide range of existing state-of-the-art white-box batch active learning methods (BADGE, BAIT, LCMD) to black-box models. We demonstrate the effectiveness of our approach through extensive experimental evaluations on regression datasets, achieving surprisingly strong performance compared to white-box approaches for deep learning models.Comment: 12 pages + 11 pages appendi

Inverse problems for abstract evolution equations II: higher order differentiability for viscoelasticity

Abstract. In this follow-up of [Inverse Problems 32 (2016) 085001] we generalize our previous abstract results so that they can be applied to the viscoelastic wave equation which serves as a forward model for full waveform inversion (FWI) in seismic imaging including dispersion and attenuation. FWI is the nonlinear inverse problem of identifying parameter functions of the viscoelastic wave equation from measurements of the reflected wave field. Here we rigorously derive rather explicit analytic expressions for the Fréchet derivative and its adjoint (adjoint state method) of the underlying parameter-to-solution map. These quantities enter crucially Newton-like gradient decent solvers for FWI. Moreover, we provide the second Fréchet derivative and a related adjoint as ingredients to second degree solvers

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not

The paper 'Deep Learning on a Data Diet' by Paul et al. (2021) introduces two innovative metrics for pruning datasets during the training of neural networks. While we are able to replicate the results for the EL2N score at epoch 20, the same cannot be said for the GraNd score at initialization. The GraNd scores later in training provide useful pruning signals, however. The GraNd score at initialization calculates the average gradient norm of an input sample across multiple randomly initialized models before any training has taken place. Our analysis reveals a strong correlation between the GraNd score at initialization and the input norm of a sample, suggesting that the latter could have been a cheap new baseline for data pruning. Unfortunately, neither the GraNd score at initialization nor the input norm surpasses random pruning in performance. This contradicts one of the findings in Paul et al. (2021). We were unable to reproduce their CIFAR-10 results using both an updated version of the original JAX repository and in a newly implemented PyTorch codebase. An investigation of the underlying JAX/FLAX code from 2021 surfaced a bug in the checkpoint restoring code that was fixed in April 2021 (https://github.com/google/flax/commit/28fbd95500f4bf2f9924d2560062fa50e919b1a5).Comment: 5 page

A scattering problem for a local pertubation of an open periodic waveguide

In this paper we consider the propagation of waves in an open waveguide in $\mathbb{R}^2$ where the index of refraction is a local perturbation of a function which is periodic along the axis of the waveguide (which we choose to be the $x_1$−axis) and equal to one for $|x_2| > h_0$ for some $h_0 > 0$. Motivated by the limiting absorption principle (proven in [17] for the case of an open waveguide in the half space $\mathbb{R} \times (0, \infty))$ we formulate a radiation condition which allows the existence of propagating modes and prove uniqueness, existence, and stability of a solution under the assumption that no bound states exist. In the second part we determine the order of decay of the radiating part of the solution in the direction of the layer and in the direction orthogonal to it. Finally, we show that it satisfies the classical Sommerfeld radiation condition and allows the definition of a far field pattern

On the scattering of a plane wave by a perturbed open periodic waveguide

We consider the scattering of a plane wave by a locally perturbed periodic (with respect to $x_1$) medium. If there is no perturbation it is usually assumed that the scattered wave is quasi-periodic with the same parameter as the incident plane wave. As it is well known, one can show existence under this condition but not necessarily uniqueness. Uniqueness fails for certain incident directions (if the wavenumber is kept fixed), and it is not clear which additional condition has to be assumed in this case. In this paper we will analyze three concepts. For the Limiting Absorption Principle (LAP) we replace the refractive index $n = n(x)$ by $n(x) + i\varepsilon$ in a layer of finite width and consider the limiting case $\varepsilon\to0$. This will give an unsatisfactory condition. In a second approach we require continuity of the field with respect to the incident direction. This will give the same satisfactory condition as the third approach where we approximate the incident plane wave by an incident point source and let the location of the source tend to infinity

The limiting absorption principle and a radiation condition for the scattering by a periodic layer

Scattering of time-harmonic waves from periodic structures at some fixed real-valued wave number becomes analytically difficult whenever there arise surface waves: These non-zero solutions to the homogeneous scattering problem physically correspond to modes propagating along the periodic structure and clearly imply non-uniqueness of any solution to the scattering problem. In this paper, we consider a medium that is defined in the upper two-dimensional half-space by a penetrable and periodic contrast. We prove that there is a so-called limiting absorption solution to the associated scattering problem. By definition, such a solution is the limit of a sequence of unique solutions for artificial complex-valued wave numbers tending to the above-mentioned real-valued wave number. Our method of proof seems to be new: By the Floquet-Bloch transform we first reduce the scattering problem to a finite-dimensional one that is set in the linear space spanned by all surface waves. In this space, we then compute explicitly which modes propagate along the periodic structure to the left or to the right. This finally yields a representation for our limiting absorption solution which leads to a proper extension of the well known upward propagating radiation condition. Finally, we prove uniqueness of a solution under this radiation condition

Direct and inverse interaction problems with bi-periodic interfaces between acoustic and elastic waves

Consider a time-harmonic acoustic plane wave incident onto a doubly periodic (biperiodic) surface from above. The medium above the surface is supposed to be filled with homogeneous compressible inviscid fluid with a constant mass density, whereas the region below is occupied by an isotropic and linearly elastic solid body characterized by the Lam'e constants. This paper is concerned with direct (or forward) and inverse fluid-solid interaction (FSI) problems with unbounded bi-periodic interfaces between acoustic and elastic waves. We present a variational approach to the forward interaction problem with Lipschitz interfaces. Existence of quasi-periodic solutions in Sobolev spaces is established at arbitrary frequency of incidence, while uniqueness is proved only for small frequencies or for all frequencies excluding a discrete set. Concerning the inverse problem, we show that the factorization method by Kirsch (1998) is applicable to the FSI problem in periodic structures. A computational criterion and a uniqueness result are justified for precisely characterizing the elastic body by utilizing the scattered acoustic near field measured in the fluid