Hamiltonian Variational Auto-Encoder

Variational Auto-Encoders (VAEs) have become very popular techniques to perform inference and learning in latent variable models as they allow us to leverage the rich representational power of neural networks to obtain flexible approximations of the posterior of latent variables as well as tight evidence lower bounds (ELBOs). Combined with stochastic variational inference, this provides a methodology scaling to large datasets. However, for this methodology to be practically efficient, it is necessary to obtain low-variance unbiased estimators of the ELBO and its gradients with respect to the parameters of interest. While the use of Markov chain Monte Carlo (MCMC) techniques such as Hamiltonian Monte Carlo (HMC) has been previously suggested to achieve this [23, 26], the proposed methods require specifying reverse kernels which have a large impact on performance. Additionally, the resulting unbiased estimator of the ELBO for most MCMC kernels is typically not amenable to the reparameterization trick. We show here how to optimally select reverse kernels in this setting and, by building upon Hamiltonian Importance Sampling (HIS) [17], we obtain a scheme that provides low-variance unbiased estimators of the ELBO and its gradients using the reparameterization trick. This allows us to develop a Hamiltonian Variational Auto-Encoder (HVAE). This method can be reinterpreted as a target-informed normalizing flow [20] which, within our context, only requires a few evaluations of the gradient of the sampled likelihood and trivial Jacobian calculations at each iteration.Comment: Accepted as a poster in the proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS

Multiple congenital bilateral trigger digits in a 2-year-old child: case report

Trigger finger is a rare condition in children. In this paper, we report on a 2-year-old boy with multiple congenital bilateral trigger digits. The patient had no history of perinatal trauma, viral or bacterial infections, or metabolic disorders. The patient was treated with physiotherapy for one year. At the one-year follow-up, the boy presented with six trigger fingers (3 on the right hand, 3 on the left hand). Neither thumb was involved. The six trigger fingers were treated surgically: first, the right-hand trigger fingers and, six months later, those of the left hand. After each operation, a 4-week brace in extension was applied to the operated hand. The symptoms were completely resolved after surgical treatment. Many authors have recommended surgical release for the treatment of trigger finger in children; empirical treatment with physiotherapy may be an option when symptoms present or appear at an older age

Post-traumatic malunion of the proximal phalanx of the finger. Medium-term results in 24 cases treated by "in situ" osteotomy

We report the clinical and radiographic medium-term results obtained for 20 patients (24 fingers) treated surgically for post-traumatic malunion of the proximal phalanx of the finger. In all cases we performed a corrective osteoclasia or osteotomy at the site of malunion, followed by miniplate and screw fixation or by screw fixation only. The corrective osteoclasia was performed when malalignment was addressed within six weeks after injury. Two patients who had two fractures underwent additional surgery (tenolysis and/or capsulolysis) to improve function and ROM. At the final follow-up, at a mean of 24 months after corrective surgery, good or excellent clinical and radiographic results were obtained in all the patients. The pseudoclaw deformity disappeared in all cases in which a volar angulation deformity was present. An average improvement of about 30% in the range of motion of the MP and PIP joints was observed; only 4 patients complained of mild pain at the maximum degrees of articular excursion of the MP and PIP joints. All the patients presented an improvement in grip strength. The mean DASH score in our series was 5 points. In two of the four cases treated by an intra-articular corrective osteotomy, mild radiographic signs of osteoarthritis at the MP joint were present. The data for this study confirm that "in situ" osteotomy stabilized by miniplates and/or screws is an effective procedure to correct post-traumatic malunions of the proximal phalanges of the fingers

Relating Regularization and Generalization through the Intrinsic Dimension of Activations

Given a pair of models with similar training set performance, it is natural to assume that the model that possesses simpler internal representations would exhibit better generalization. In this work, we provide empirical evidence for this intuition through an analysis of the intrinsic dimension (ID) of model activations, which can be thought of as the minimal number of factors of variation in the model's representation of the data. First, we show that common regularization techniques uniformly decrease the last-layer ID (LLID) of validation set activations for image classification models and show how this strongly affects generalization performance. We also investigate how excessive regularization decreases a model's ability to extract features from data in earlier layers, leading to a negative effect on validation accuracy even while LLID continues to decrease and training accuracy remains near-perfect. Finally, we examine the LLID over the course of training of models that exhibit grokking. We observe that well after training accuracy saturates, when models ``grok'' and validation accuracy suddenly improves from random to perfect, there is a co-occurent sudden drop in LLID, thus providing more insight into the dynamics of sudden generalization.Comment: NeurIPS 2022 OPT and HITY workshop

CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds

Precision measurements and new physics searches at the Large Hadron Collider require efficient simulations of particle propagation and interactions within the detectors. The most computationally expensive simulations involve calorimeter showers. Advances in deep generative modelling - particularly in the realm of high-dimensional data - have opened the possibility of generating realistic calorimeter showers orders of magnitude more quickly than physics-based simulation. However, the high-dimensional representation of showers belies the relative simplicity and structure of the underlying physical laws. This phenomenon is yet another example of the manifold hypothesis from machine learning, which states that high-dimensional data is supported on low-dimensional manifolds. We thus propose modelling calorimeter showers first by learning their manifold structure, and then estimating the density of data across this manifold. Learning manifold structure reduces the dimensionality of the data, which enables fast training and generation when compared with competing methods.Comment: Accepted to the Machine Learning and the Physical Sciences Workshop at NeurIPS 202

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.Comment: NeurIPS 2023. 53 pages, 29 figures, 12 tables. Code at https://github.com/layer6ai-labs/dgm-eval, reviews at https://openreview.net/forum?id=08zf7kTOo

Performance of serum-supplemented and serum-free media in IFNγ Elispot Assays for human T cells

The choice of serum for supplementation of media for T cell assays and in particular, Elispot has been a major challenge for assay performance, standardization, optimization, and reproducibility. The Assay Working Group of the Cancer Vaccine Consortium (CVC-CRI) has recently identified the choice of serum to be the leading cause for variability and suboptimal performance in large international Elispot proficiency panels. Therefore, a serum task force was initiated to compare the performance of commercially available serum-free media to laboratories’ own medium/serum combinations. The objective of this project was to investigate whether a serum-free medium exists that performs as well as lab-own serum/media combinations with regard to antigen-specific responses and background reactivity in Elispot. In this way, a straightforward solution could be provided to address the serum challenge. Eleven laboratories tested peripheral blood mononuclear cells (PBMC) from four donors for their reactivity against two peptide pools, following their own Standard Operating Procedure (SOP). Each laboratory performed five simultaneous experiments with the same SOP, the only difference between the experiments was the medium used. The five media were lab-own serum-supplemented medium, AIM-V, CTL, Optmizer, and X-Vivo. The serum task force results demonstrate compellingly that serum-free media perform as well as qualified medium/serum combinations, independent of the applied SOP. Recovery and viability of cells are largely unaffected by serum-free conditions even after overnight resting. Furthermore, one serum-free medium was identified that appears to enhance antigen-specific IFNγ-secretion