22 research outputs found

    Small Transformers Compute Universal Metric Embeddings

    Full text link
    We study representations of data from an arbitrary metric space X\mathcal{X} in the space of univariate Gaussian mixtures with a transport metric (Delon and Desolneux 2020). We derive embedding guarantees for feature maps implemented by small neural networks called \emph{probabilistic transformers}. Our guarantees are of memorization type: we prove that a probabilistic transformer of depth about nlog‚Ā°(n)n\log(n) and width about n2n^2 can bi-H\"{o}lder embed any nn-point dataset from X\mathcal{X} with low metric distortion, thus avoiding the curse of dimensionality. We further derive probabilistic bi-Lipschitz guarantees, which trade off the amount of distortion and the probability that a randomly chosen pair of points embeds with that distortion. If X\mathcal{X}'s geometry is sufficiently regular, we obtain stronger, bi-Lipschitz guarantees for all points in the dataset. As applications, we derive neural embedding guarantees for datasets from Riemannian manifolds, metric trees, and certain types of combinatorial graphs. When instead embedding into multivariate Gaussian mixtures, we show that probabilistic transformers can compute bi-H\"{o}lder embeddings with arbitrarily small distortion.Comment: 42 pages, 10 Figures, 3 Table

    FunkNN: Neural Interpolation for Functional Generation

    Full text link
    Can we build continuous generative models which generalize across scales, can be evaluated at any coordinate, admit calculation of exact derivatives, and are conceptually simple? Existing MLP-based architectures generate worse samples than the grid-based generators with favorable convolutional inductive biases. Models that focus on generating images at different scales do better, but employ complex architectures not designed for continuous evaluation of images and derivatives. We take a signal-processing perspective and treat continuous image generation as interpolation from samples. Indeed, correctly sampled discrete images contain all information about the low spatial frequencies. The question is then how to extrapolate the spectrum in a data-driven way while meeting the above design criteria. Our answer is FunkNN -- a new convolutional network which learns how to reconstruct continuous images at arbitrary coordinates and can be applied to any image dataset. Combined with a discrete generative model it becomes a functional generator which can act as a prior in continuous ill-posed inverse problems. We show that FunkNN generates high-quality continuous images and exhibits strong out-of-distribution performance thanks to its patch-based design. We further showcase its performance in several stylized inverse problems with exact spatial derivatives.Comment: 17 pages, 13 figure

    Joint Cryo-ET Alignment and Reconstruction with Neural Deformation Fields

    Full text link
    We propose a framework to jointly determine the deformation parameters and reconstruct the unknown volume in electron cryotomography (CryoET). CryoET aims to reconstruct three-dimensional biological samples from two-dimensional projections. A major challenge is that we can only acquire projections for a limited range of tilts, and that each projection undergoes an unknown deformation during acquisition. Not accounting for these deformations results in poor reconstruction. The existing CryoET software packages attempt to align the projections, often in a workflow which uses manual feedback. Our proposed method sidesteps this inconvenience by automatically computing a set of undeformed projections while simultaneously reconstructing the unknown volume. We achieve this by learning a continuous representation of the undeformed measurements and deformation parameters. We show that our approach enables the recovery of high-frequency details that are destroyed without accounting for deformations

    Differentiable Uncalibrated Imaging

    Full text link
    We propose a differentiable imaging framework to address uncertainty in measurement coordinates such as sensor locations and projection angles. We formulate the problem as measurement interpolation at unknown nodes supervised through the forward operator. To solve it we apply implicit neural networks, also known as neural fields, which are naturally differentiable with respect to the input coordinates. We also develop differentiable spline interpolators which perform as well as neural networks, require less time to optimize and have well-understood properties. Differentiability is key as it allows us to jointly fit a measurement representation, optimize over the uncertain measurement coordinates, and perform image reconstruction which in turn ensures consistent calibration. We apply our approach to 2D and 3D computed tomography and show that it produces improved reconstructions compared to baselines that do not account for the lack of calibration. The flexibility of the proposed framework makes it easy to apply to almost arbitrary imaging problems

    Manifold Rewiring for Unlabeled Imaging

    Full text link
    Geometric data analysis relies on graphs that are either given as input or inferred from data. These graphs are often treated as "correct" when solving downstream tasks such as graph signal denoising. But real-world graphs are known to contain missing and spurious links. Similarly, graphs inferred from noisy data will be perturbed. We thus define and study the problem of graph denoising, as opposed to graph signal denoising, and propose an approach based on link-prediction graph neural networks. We focus in particular on neighborhood graphs over point clouds sampled from low-dimensional manifolds, such as those arising in imaging inverse problems and exploratory data analysis. We illustrate our graph denoising framework on regular synthetic graphs and then apply it to single-particle cryo-EM where the measurements are corrupted by very high levels of noise. Due to this degradation, the initial graph is contaminated by noise, leading to missing or spurious edges. We show that our proposed graph denoising algorithm improves the state-of-the-art performance of multi-frequency vector diffusion maps

    Overnight consolidation aids the transfer of statistical knowledge from the medial temporal lobe to the striatum

    Get PDF
    Sleep is important for abstraction of the underlying principles (or gist) which bind together conceptually related stimuli, but little is known about the neural correlates of this process. Here, we investigate this issue using overnight sleep monitoring and functional magnetic resonance imaging (fMRI). Participants were exposed to a statistically structured sequence of auditory tones then tested immediately for recognition of short sequences which conformed to the learned statistical pattern. Subsequently, after consolidation over either 30min or 24h, they performed a delayed test session in which brain activity was monitored with fMRI. Behaviorally, there was greater improvement across 24h than across 30min, and this was predicted by the amount of slow wave sleep (SWS) obtained. Functionally, we observed weaker parahippocampal responses and stronger striatal responses after sleep. Like the behavioral result, these differences in functional response were predicted by the amount of SWS obtained. Furthermore, connectivity between striatum and parahippocampus was weaker after sleep, whereas connectivity between putamen and planum temporale was stronger. Taken together, these findings suggest that abstraction is associated with a gradual shift from the hippocampal to the striatal memory system and that this may be mediated by SWS

    Microscopie computationnelle

    Get PDF
    The contributions of this thesis are numerical and theoretical tools for the resolution of blind inverse problems in imaging. We first focus in the case where the observation operator is unknown (e.g. microscopy, astronomy, photography). A very popular approach consists in estimating this operator from an image containing point sources (microbeads or fluorescent proteins in microscopy, stars in astronomy). Such an observation provides a measure of the impulse response of the degradation operator at several points in the field of view. Processing this observation requires robust tools that can rapidly use the data. We propose a toolbox that estimates a degradation operator from an image containing point sources. The estimated operator has the property that at any location in the field of view, its impulse response is expressed as a linear combination of elementary estimated functions. This makes it possible to estimate spatially invariant (convolution) and variant (product-convolution expansion) operators. An important specificity of this toolbox is its high level of automation: only a small number of easily accessible parameters allows to cover a large majority of practical cases. The size of the point source (e.g. bead), the background and the noise are also taken in consideration in the estimation. This tool, coined PSF-estimator, comes in the form of a module for the Fiji software, and is based on a parallelized implementation in C++. The operators generated by an optical system are usually changing for each experiment, which ideally requires a calibration of the system before each acquisition. To overcome this, we propose to represent an optical system not by a single operator (e.g. convolution blur with a fixed kernel for different experiments), but by subspace of operators. This set allows to represent all the possible states of a microscope. We introduce a method for estimating such a subspace from a collection of low rank operators (such as those estimated by the toolbox PSF-Estimator). We show that under reasonable assumptions, this subspace is low-dimensional and consists of low rank elements. In a second step, we apply this process in microscopy on large fields of view and with spatially varying operators. This implementation is possible thanks to the use of additional methods to process real images (e.g. background, noise, discretization of the observation).The construction of an operator subspace is only one step in the resolution of blind inverse problems. It is then necessary to identify the degradation operator in this set from a single observed image. In this thesis, we provide a mathematical framework to this operator identification problem in the case where the original image is constituted of point sources. Theoretical conditions arise from this work, allowing a better understanding of the conditions under which this problem can be solved. We illustrate how this formal study allows the resolution of a blind deblurring problem on a microscopy example.[...]Les travaux pr√©sent√©s de cette th√®se visent √† proposer des outils num√©riques et th√©oriques pour la r√©solution de probl√®mes inverses en imagerie. Nous nous int√©ressons particuli√®rement au cas o√Ļ l'op√©rateur d'observation (e.g. flou) n'est pas connu. Les r√©sultats principaux de cette th√®se s'articulent autour de l'estimation et l'identification de cet op√©rateur d'observation. Une approche pl√©biscit√©e pour estimer un op√©rateur de d√©gradation consiste √† observer un √©chantillon contenant des sources ponctuelles (microbilles en microscopie, √©toiles en astronomie). Une telle acquisition fournit une mesure de la r√©ponse impulsionnelle de l'op√©rateur en plusieurs points du champ de vue. Le traitement de cette observation requiert des outils robustes pouvant utiliser rapidement les donn√©es rencontr√©es en pratique. Nous proposons une bo√ģte √† outils qui estime un op√©rateur de d√©gradation √† partir d'une image contenant des sources ponctuelles. L'op√©rateur estim√© √† la propri√©t√© qu'en tout point du champ de vue, sa r√©ponse impulsionnelle s'exprime comme une combinaison lin√©aire de fonctions √©l√©mentaires. Cela permet d'estimer des op√©rateurs invariants (convolutions) et variants (d√©veloppement en convolution-produit) spatialement. Une sp√©cificit√© importante de cette bo√ģte √† outils est son caract√®re automatique : seul un nombre r√©duit de param√®tres facilement accessibles permettent de couvrir une grande majorit√© des cas pratiques. La taille de la source ponctuelle (e.g. bille), le fond et le bruit sont √©galement pris en compte dans l'estimation. Cet outil se pr√©sente sous la forme d'un module appel√© PSF-Estimator pour le logiciel Fiji, et repose sur une impl√©mentation parall√©lis√©e en C++. En r√©alit√©, les op√©rateurs mod√©lisant un syst√®me optique varient d'une exp√©rience √† une autre, ce qui, dans l'id√©al, n√©cessite une calibration du syst√®me avant chaque acquisition. Pour pallier √† cela, nous proposons de repr√©senter un syst√®me optique non pas par un unique op√©rateur de d√©gradation, mais par un sous-espace d'op√©rateurs. Cet ensemble doit permettre de repr√©senter chaque op√©rateur g√©n√©r√© par un microscope. Nous introduisons une m√©thode d'estimation d'un tel sous-espace √† partir d'une collection d'op√©rateurs de faible rang (comme ceux estim√©s par la bo√ģte √† outils PSF-Estimator). Nous montrons que sous des hypoth√®ses raisonnables, ce sous-espace est de faible dimension et est constitu√© d'√©l√©ments de faible rang. Dans un second temps, nous appliquons ce proc√©d√© en microscopie sur de grands champs de vue et avec des op√©rateurs variant spatialement. Cette mise en Ňďuvre est possible gr√Ęce √† l'utilisation de m√©thodes compl√©mentaires pour traiter des images r√©elles (e.g. le fond, le bruit, la discr√©tisation de l'observation). La construction d'un sous-espace d'op√©rateurs n'est qu'une √©tape dans l'√©talonnage de syst√®mes optiques et la r√©solution de probl√®mes inverses. [...

    Computational microscopy

    No full text
    Les travaux pr√©sent√©s de cette th√®se visent √† proposer des outils num√©riques et th√©oriques pour la r√©solution de probl√®mes inverses en imagerie. Nous nous int√©ressons particuli√®rement au cas o√Ļ l'op√©rateur d'observation (e.g. flou) n'est pas connu. Les r√©sultats principaux de cette th√®se s'articulent autour de l'estimation et l'identification de cet op√©rateur d'observation. Une approche pl√©biscit√©e pour estimer un op√©rateur de d√©gradation consiste √† observer un √©chantillon contenant des sources ponctuelles (microbilles en microscopie, √©toiles en astronomie). Une telle acquisition fournit une mesure de la r√©ponse impulsionnelle de l'op√©rateur en plusieurs points du champ de vue. Le traitement de cette observation requiert des outils robustes pouvant utiliser rapidement les donn√©es rencontr√©es en pratique. Nous proposons une bo√ģte √† outils qui estime un op√©rateur de d√©gradation √† partir d'une image contenant des sources ponctuelles. L'op√©rateur estim√© √† la propri√©t√© qu'en tout point du champ de vue, sa r√©ponse impulsionnelle s'exprime comme une combinaison lin√©aire de fonctions √©l√©mentaires. Cela permet d'estimer des op√©rateurs invariants (convolutions) et variants (d√©veloppement en convolution-produit) spatialement. Une sp√©cificit√© importante de cette bo√ģte √† outils est son caract√®re automatique : seul un nombre r√©duit de param√®tres facilement accessibles permettent de couvrir une grande majorit√© des cas pratiques. La taille de la source ponctuelle (e.g. bille), le fond et le bruit sont √©galement pris en compte dans l'estimation. Cet outil se pr√©sente sous la forme d'un module appel√© PSF-Estimator pour le logiciel Fiji, et repose sur une impl√©mentation parall√©lis√©e en C++. En r√©alit√©, les op√©rateurs mod√©lisant un syst√®me optique varient d'une exp√©rience √† une autre, ce qui, dans l'id√©al, n√©cessite une calibration du syst√®me avant chaque acquisition. Pour pallier √† cela, nous proposons de repr√©senter un syst√®me optique non pas par un unique op√©rateur de d√©gradation, mais par un sous-espace d'op√©rateurs. Cet ensemble doit permettre de repr√©senter chaque op√©rateur g√©n√©r√© par un microscope. Nous introduisons une m√©thode d'estimation d'un tel sous-espace √† partir d'une collection d'op√©rateurs de faible rang (comme ceux estim√©s par la bo√ģte √† outils PSF-Estimator). Nous montrons que sous des hypoth√®ses raisonnables, ce sous-espace est de faible dimension et est constitu√© d'√©l√©ments de faible rang. Dans un second temps, nous appliquons ce proc√©d√© en microscopie sur de grands champs de vue et avec des op√©rateurs variant spatialement. Cette mise en Ňďuvre est possible gr√Ęce √† l'utilisation de m√©thodes compl√©mentaires pour traiter des images r√©elles (e.g. le fond, le bruit, la discr√©tisation de l'observation). La construction d'un sous-espace d'op√©rateurs n'est qu'une √©tape dans l'√©talonnage de syst√®mes optiques et la r√©solution de probl√®mes inverses. [...]The contributions of this thesis are numerical and theoretical tools for the resolution of blind inverse problems in imaging. We first focus in the case where the observation operator is unknown (e.g. microscopy, astronomy, photography). A very popular approach consists in estimating this operator from an image containing point sources (microbeads or fluorescent proteins in microscopy, stars in astronomy). Such an observation provides a measure of the impulse response of the degradation operator at several points in the field of view. Processing this observation requires robust tools that can rapidly use the data. We propose a toolbox that estimates a degradation operator from an image containing point sources. The estimated operator has the property that at any location in the field of view, its impulse response is expressed as a linear combination of elementary estimated functions. This makes it possible to estimate spatially invariant (convolution) and variant (product-convolution expansion) operators. An important specificity of this toolbox is its high level of automation: only a small number of easily accessible parameters allows to cover a large majority of practical cases. The size of the point source (e.g. bead), the background and the noise are also taken in consideration in the estimation. This tool, coined PSF-estimator, comes in the form of a module for the Fiji software, and is based on a parallelized implementation in C++. The operators generated by an optical system are usually changing for each experiment, which ideally requires a calibration of the system before each acquisition. To overcome this, we propose to represent an optical system not by a single operator (e.g. convolution blur with a fixed kernel for different experiments), but by subspace of operators. This set allows to represent all the possible states of a microscope. We introduce a method for estimating such a subspace from a collection of low rank operators (such as those estimated by the toolbox PSF-Estimator). We show that under reasonable assumptions, this subspace is low-dimensional and consists of low rank elements. In a second step, we apply this process in microscopy on large fields of view and with spatially varying operators. This implementation is possible thanks to the use of additional methods to process real images (e.g. background, noise, discretization of the observation).The construction of an operator subspace is only one step in the resolution of blind inverse problems. It is then necessary to identify the degradation operator in this set from a single observed image. In this thesis, we provide a mathematical framework to this operator identification problem in the case where the original image is constituted of point sources. Theoretical conditions arise from this work, allowing a better understanding of the conditions under which this problem can be solved. We illustrate how this formal study allows the resolution of a blind deblurring problem on a microscopy example.[...

    DEEPBLUR: BLIND IDENTIFICATION OF SPACE VARIANT PSF

    No full text
    We propose to train a neural network to estimate space varying blur operators from a single blurry image. The key assumption is that the operator lives in a subset of a known subspace, which is a reasonable assumption in many microscopes. We detail a specific sampling procedure of the subset to train a Resnet architecture. This allows a fast estimation. We finally illustrate the performance of the network on de-blurring problems. Index Terms-blur identification, neural network, non-uniform blur, blind debluring, blind inverse problem

    Deep-Blur : Blind Identification and Deblurring with Convolutional Neural Networks

    No full text
    We propose a neural network architecture and a training procedure to estimate blurring operators and deblur images from a single degraded image. The key assumption is that the forward operators can be parameterized by a lowdimensional vector. The considered models include a description of the point spread function with Zernike polynomials in the pupil plane or product-convolution expansions, which allow to tackle space varying operators. Numerical experiments reveal that the proposed method recovers the blur parameters robustly even for large noise levels. This estimate can then used as an input of an unrolled neural network to deblur the image. It is trained with a specific sampling procedure adapted to a family of parameterized operators. This strategy provides a fast and accurate blind inverse problem solver, requiring nearly no human interaction once the operator parameterization has been set up. It adapts to arbirtrary noise levels and considerably improves alternative blind deblurring softwares in the examples considered in this paper, especially in the high noise regimes.
    corecore