Learning efficient image representations: Connections between statistics and neuroscience

This thesis summarizes different works developed in the framework of analyzing the relation between image processing, statistics and neuroscience. These relations are analyzed from the efficient coding hypothesis point of view (H. Barlow [1961] and Attneave [1954]). This hypothesis suggests that the human visual system has been adapted during the ages in order to process the visual information in an efficient way, i.e. taking advantage of the statistical regularities of the visual world. Under this classical idea different works in different directions are developed. One direction is analyzing the statistical properties of a revisited, extended and fitted classical model of the human visual system. No statistical information is used in the model. Results show that this model obtains a representation with good statistical properties, which is a new evidence in favor of the efficient coding hypothesis. From the statistical point of view, different methods are proposed and optimized using natural images. The models obtained using these statistical methods show similar behavior to the human visual system, both in the spatial and color dimensions, which are also new evidences of the efficient coding hypothesis. Applications in image processing are an important part of the Thesis. Statistical and neuroscience based methods are employed to develop a wide set of image processing algorithms. Results of these methods in denoising, classification, synthesis and quality assessment are comparable to some of the most successful current methods

Validação de heterogeneidade estrutural em dados de Crio-ME por comitês de agrupadores

Orientadores: Fernando José Von Zuben, Rodrigo Villares PortugalDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: Análise de Partículas Isoladas é uma técnica que permite o estudo da estrutura tridimensional de proteínas e outros complexos macromoleculares de interesse biológico. Seus dados primários consistem em imagens de microscopia eletrônica de transmissão de múltiplas cópias da molécula em orientações aleatórias. Tais imagens são bastante ruidosas devido à baixa dose de elétrons utilizada. Reconstruções 3D podem ser obtidas combinando-se muitas imagens de partículas em orientações similares e estimando seus ângulos relativos. Entretanto, estados conformacionais heterogêneos frequentemente coexistem na amostra, porque os complexos moleculares podem ser flexíveis e também interagir com outras partículas. Heterogeneidade representa um desafio na reconstrução de modelos 3D confiáveis e degrada a resolução dos mesmos. Entre os algoritmos mais populares usados para classificação estrutural estão o agrupamento por k-médias, agrupamento hierárquico, mapas autoorganizáveis e estimadores de máxima verossimilhança. Tais abordagens estão geralmente entrelaçadas à reconstrução dos modelos 3D. No entanto, trabalhos recentes indicam ser possível inferir informações a respeito da estrutura das moléculas diretamente do conjunto de projeções 2D. Dentre estas descobertas, está a relação entre a variabilidade estrutural e manifolds em um espaço de atributos multidimensional. Esta dissertação investiga se um comitê de algoritmos de não-supervisionados é capaz de separar tais "manifolds conformacionais". Métodos de "consenso" tendem a fornecer classificação mais precisa e podem alcançar performance satisfatória em uma ampla gama de conjuntos de dados, se comparados a algoritmos individuais. Nós investigamos o comportamento de seis algoritmos de agrupamento, tanto individualmente quanto combinados em comitês, para a tarefa de classificação de heterogeneidade conformacional. A abordagem proposta foi testada em conjuntos sintéticos e reais contendo misturas de imagens de projeção da proteína Mm-cpn nos estados "aberto" e "fechado". Demonstra-se que comitês de agrupadores podem fornecer informações úteis na validação de particionamentos estruturais independetemente de algoritmos de reconstrução 3DAbstract: Single Particle Analysis is a technique that allows the study of the three-dimensional structure of proteins and other macromolecular assemblies of biological interest. Its primary data consists of transmission electron microscopy images from multiple copies of the molecule in random orientations. Such images are very noisy due to the low electron dose employed. Reconstruction of the macromolecule can be obtained by averaging many images of particles in similar orientations and estimating their relative angles. However, heterogeneous conformational states often co-exist in the sample, because the molecular complexes can be flexible and may also interact with other particles. Heterogeneity poses a challenge to the reconstruction of reliable 3D models and degrades their resolution. Among the most popular algorithms used for structural classification are k-means clustering, hierarchical clustering, self-organizing maps and maximum-likelihood estimators. Such approaches are usually interlaced with the reconstructions of the 3D models. Nevertheless, recent works indicate that it is possible to infer information about the structure of the molecules directly from the dataset of 2D projections. Among these findings is the relationship between structural variability and manifolds in a multidimensional feature space. This dissertation investigates whether an ensemble of unsupervised classification algorithms is able to separate these "conformational manifolds". Ensemble or "consensus" methods tend to provide more accurate classification and may achieve satisfactory performance across a wide range of datasets, when compared with individual algorithms. We investigate the behavior of six clustering algorithms both individually and combined in ensembles for the task of structural heterogeneity classification. The approach was tested on synthetic and real datasets containing a mixture of images from the Mm-cpn chaperonin in the "open" and "closed" states. It is shown that cluster ensembles can provide useful information in validating the structural partitionings independently of 3D reconstruction methodsMestradoEngenharia de ComputaçãoMestre em Engenharia Elétric

Efficient Human Activity Recognition in Large Image and Video Databases

Vision-based human action recognition has attracted considerable interest in recent research for its applications to video surveillance, content-based search, healthcare, and interactive games. Most existing research deals with building informative feature descriptors, designing efficient and robust algorithms, proposing versatile and challenging datasets, and fusing multiple modalities. Often, these approaches build on certain conventions such as the use of motion cues to determine video descriptors, application of off-the-shelf classifiers, and single-factor classification of videos. In this thesis, we deal with important but overlooked issues such as efficiency, simplicity, and scalability of human activity recognition in different application scenarios: controlled video environment (e.g.~indoor surveillance), unconstrained videos (e.g.~YouTube), depth or skeletal data (e.g.~captured by Kinect), and person images (e.g.~Flicker). In particular, we are interested in answering questions like (a) is it possible to efficiently recognize human actions in controlled videos without temporal cues? (b) given that the large-scale unconstrained video data are often of high dimension low sample size (HDLSS) nature, how to efficiently recognize human actions in such data? (c) considering the rich 3D motion information available from depth or motion capture sensors, is it possible to recognize both the actions and the actors using only the motion dynamics of underlying activities? and (d) can motion information from monocular videos be used for automatically determining saliency regions for recognizing actions in still images

Spin-resolved microscopy of strongly correlated fermionic many-body states

Mit ultrakalte Gasen in optischen Gittern lassen sich stark wechselwirkende Quantenvielteilchensysteme auf der Ebene einzelner Spins untersuchen. Diese Doktorarbeit fasst den Aufbau und die ersten Ergebnisse eines Quantengasmikroskops mit fermionischen Li-6 Atomen zusammen. Wir konnten erstmals antiferromagnetische Spinkorrelationen in Hubbard-Systemen beobachten und in eindimensionalen Systeme die Trennung von Ladungs- und Spinfreiheitsgraden mit Korrelationen nachweisen, die im thermischen Gleichgewicht gemessen wurden. Die Grundlage für diese Experimente ist das Quantengasmikroskop, welches während der Doktorarbeit geplant und aufgebaut wurde. Die Bilder, die man damit nehmen kann, sind Momentaufnahmen eines Quantenvielteilchensystems, auf denen man alle Atome einzeln auf ihren jeweiligen Gitterplätzen erkennen kann. Wir produzieren ein ultrakaltes Quantengas mit Standardverfahren wie Laserkühlung, optischen Fallen und Verdunstungskühlung und laden es in eine einzelne Ebene eines dreidimensionalen optischen Gitters. Vor dem Abbilden wird jedes Atom entsprechend seines Spin mit einem Stern-Gerlach-Magnetfeld um einen halben Gitterplatz nach links oder rechts verschoben. Zur Abbildung der Atome messen wir die Fluoreszenz eines Raman-Seitenbandkühlprozesses, welcher in einem zusätzlichen sehr tiefen optischen Gitter abläuft, und können so 97% der Atome erfolgreich detektieren. Ein Kapitel dieser Arbeit widmet sich den Details dieses Prozesses und kann die verbliebenen Verluste durch eine Nichtgleichgewichtsverteilung der lokalen Anregungen erklären. Die Messung der Dichteverteilung der stark wechelwirkenden Atome im inhomogenen Gitter erlaubt es die Zustandsgleichung des Fermi-Hubbard-Models zu bestimmen. Dabei beobachten wir die starke Unterdrückung der Kompressibilität in der Mott-Isolator-Phase. Mit unseren hochaufgelösten Bildern können wir auch die Dichtekorrelationen des Systems messen und so das Fluktuations-Dissipations-Theorem bestätigen, welches die Kompressibilität in Beziehung zu der Summe aller Dichtefluktuationen setzt. Für eine Entropie pro Teilchen von weniger als log(2) kB zeigt der Mott-Isolator antiferromagnetische Spinkorrelationen aufgrund der Austauschwechselwirkung. In eindimensionalen Spinketten konnten wir diese magnetische Ordnung bis zu einer Distanz von vier Gitterplätzen direkt messen. Die Stärke der beobachteten Korrelationen stimmt sehr gut überein mit Quanten-Monte-Carlo-Rechnungen bei einer Temperatur von einem Achtel der Bandbreite, welches einer Entropie von 0.4 kB pro Atom entspricht. Für Spinketten mit weniger als einem Atom pro Gitterplatz sehen wir eine charakteristische Verschiebung der Spinkorrelationen zu größeren Wellenlängen, die der einer Luttinger Flüssigkeit entspricht. Besonders interessante physikalische Phänomene treten auf, wenn man den Spinfreiheitsgrad mit der Bewegung der Atome koppelt. In eindimensionalen Systemen tritt hier die Spin-Ladungs-Trennung auf, die einem Loch eine freie Bewegung durch eine Spinkette ermöglicht. Allerdings scheint diese Delokalisierung zu einer Reduktionen der magnetischen Ordnung zu führen, da die Position der Teilchen nun fluktuiert. Normale Zweipunktkorrelatoren messen so kleinere Werte. Allerdings konnten wir zeigen, dass die Spins um einzelne Löcher herum primär antiparallel ausgerichtet sind und so nachweisen, dass die Spinordnung fast unabhängig vom Grad der Dotierung ist, wenn man die Lochposition mitberücksichtigt. Diese Messungen lassen sich als erster direkten Nachweis von Spin-Ladungs-Trennung durch Gleichgewichtskorrelationen interpretieren. Diese Arbeit umfasst die ersten Messungen von Spin-Loch-Korrelationen mit ultrakalten Atomen und sie stellt damit einen wichtigen Schritt auf dem Weg zu Quantensimulationen dotierter Antiferromagneten dar, die einen Beitrag zum Verständnis von Hochtemperatursupraleitern leisten könnten.Ultracold fermionic atoms in optical lattices allows to simulate the behavior of electrons in strongly correlated materials. In this thesis, we demonstrate the preparation and site- and spin-resolved imaging of Hubbard systems with fermionic Li-6 atoms. We realize and measure strong antiferromagnetic spin correlations and study their amplitude for various temperatures, interactions and dopings. In one-dimensional systems we observe spin-charge separation signatures by measuring equilibrium correlations for spin and density. The basis for these measurements is a quantum gas microscope for fermionic Li-6 atoms, which was built during this PhD thesis. It allows to take snapshots of the quantum many-body system with single-atom and single-site resolution. Using standard techniques of laser cooling, optical trapping, and evaporative cooling, ultracold Fermi gases are prepared and loaded into a single plane of a three-dimensional optical lattice of tunable geometry. The spin of each atom is converted to a spatial information via a local Stern-Gerlach splitting. The imaging is performed by collection of fluorescence light from Raman sideband cooling in an additional, deep optical lattice. A detailed analysis of this cooling process, which explains our imaging fidelity of 97% from the non-thermal distribution of excitations is presented. A study of the density distribution of the strongly interacting atoms in the lattice allows to derive the equatioän of state of the Fermi-Hubbard model, which shows a strongly reduced compressibility in the Mott insulating regime. From the high-resolution images we can, in addition, extract all density correlations. This allows us to experimentally confirm the fluctuation-dissipation theorem linking the compressibility to the sum of all density fluctuations. At entropies below log(2) kB per particle, antiferromagnetic correlations arise from exchange interactions in a Mott insulator. We directly observe magnetic correlations up to four sites in one-dimensional spin chains. The measured antiferromagnetic spin correlations agree well with quantum Monte-Carlo calculations at temperatures of 1/8 of the band width, which corresponds to an entropy per particle of only 0.4 kB. At fillings below one atom per site, we observe characteristic oscillations of the spin correlations vs density as predicted by Luttinger liquid theory. Interesting physics arises when one couples the spin degree of freedom with the motion of the quantum particles. In one dimension, the phenomenon of spin-charge separation allows the holes to propagate through a spin chain without energy cost. Their motion, however, hides the magnetic order from local observables. Thanks to our simultaneous imaging of spins and holes, we can directly study the spin alignment around individual holes. We reveal spin correlations which are almost fully independent of the degree of hole doping with string spin-density correlation functions. These measurement are the first experimental observation of spin-charge separation in equilibrium correlation measurements. This work demonstrates the experimental study of doped quantum magnetism with individual spin resolution and paves the way for quantum simulations of doped two-dimensional antiferromagnets relevant to high temperature superconductivity

Graduate School: Course Decriptions, 1972-73

Official publication of Cornell University V.64 1972/7

Quantum simulation in strongly correlated optical lattices

An outstanding problem in physics is how to understand strongly interacting quantum many-body systems such as the quark-gluon plasma, neutron stars, superfluid 4He, and the high-temperature superconducting cuprates. The physics approach to this problem is to reduce these complex systems to minimal models that are believed to retain relevant phenomenology. For example, the Hubbard model — the focus of this thesis — describes quantum particles tunneling between sites of a lattice with on-site interactions. The Hubbard model is conjectured to describe the low-energy charge and spin properties of high-temperature superconducting cuprates. Thus far, there are no analytic solutions to the Hubbard model, and numerical calculations are difficult and even impossible in some regimes (e.g., the Fermi-Hubbard model away from half-filling). Therefore, whether the Hubbard model is a minimal model for the cuprates remains unresolved. In the face of these difficulties, a new approach has emerged — quantum simulation. The premise of quantum simulation is to perform experiments on a quantum system that is well-described by the model we are trying to study, has tunable parameters, and is easily probed. Ultracold atoms trapped in optical lattices are an ideal candidate for quantum simulation of the Hubbard models. This thesis describes work on two such systems: a 87Rb (boson) optical lattice experiment in the group of Brian DeMarco at the University of Illinois to simulate Bose-Hubbard physics, and a 40K (fermion) optical lattice experiment in the group of Joseph Thywissen at the University of Toronto to simulate Fermi-Hubbard physics. My work on the 87Rb apparatus focuses on three main topics: simulating the Bose-Hubbard (BH) model out of equilibrium, developing thermometry probes, and developing impurity probes using a 3D spin-dependent lattice. Theoretical techniques (e.g., QMC) are adept at describing the equilibrium properties of the BH model, but the dynamics are unknown — simulation is able to bridge this gap. We perform two experiments to simulate the BH model out of equilibrium. In the first experiment, published in Ref. [1], we measure the decay rate of the center-of-mass velocity for a Bose-Einstein condensate trapped in a cubic lattice. We explore this dissipation for different Bose-Hubbard parameters (corresponding to different lattice depths) and temperatures. We observe a decay rate that asymptotes to a finite value at zero temperature, which we interpret as evidence of intrinsic decay due to quantum tunneling of phase slips. The decay rate exponentially increases with temperature, which is consistent with a cross-over from quantum tunneling to thermal activation. While phase slips are a well-known dissipation mechanism in superconductors, numerous effects prevent unambiguous detection of quantum phase slips. Therefore, our measurement is among the strongest evidence for quantum tunneling of phase slips. In a second experiment, published in Ref. [2] with theory collaborators at Cornell University, we investigate condensate fraction evolution during fast (i.e., millisecond) ramps of the lattice potential depth. These ramps simulate the BH model with time-dependent parameters. We determine that interactions lead to significant condensate fraction redistribution during these ramps, in agreement with mean-field calculations. This result clarifies adiabatic timescales for the lattice gas and strongly constrains bandmapping as an equilibrium probe. Another part of this thesis work involves developing thermometry techniques for the lattice gas. These techniques are important because the ability to measure temperature is required for quantum simulation and to evaluate in-lattice cooling schemes. In work published in Ref. [3], we explore measuring temperature by directly fitting the quasimomentum distribution of a thermal lattice gas. We attempt to obtain quasimomentum distributions by bandmapping, a process in which the lattice depth is reduced slowly compared to the bandgap but fast with respect to all other timescales. We find that these temperature measurements fail when the thermal energy is comparable to the bandwidth of the lattice. This failure results from two main causes. First, the quasimomentum distribution is an insensitive probe at high temperatures because the band is occupied (i.e., additional thermal energy cannot be accommodated in the kinetic energy degrees of freedom). Second, the bandmapping process does not produce accurate quasimomentum distributions because of smoothing at the Brillouin zone edge. We determine that measuring temperature using the in-situ width overcomes these issues. The in-situ width does not asymptote to a finite value as temperature increases, and the in-situ width can be measured directly without using a mapping procedure. In a second experiment, we investigate using condensate fraction (obtained from the time-of-flight momentum distribution) as an indirect means to measure temperature in the superfluid regime of the BH model. Since no standard fitting procedure exists for the lattice time-of-flight distributions, we define and test a procedure as part of this work. We measure condensate fraction for a range of lattice depths varying from deep in the superfluid regime to lattice depths proximate to the Mott-insulator transition. We also vary the entropy per particle, which is measured in the harmonic trap before adiabatically loading into the lattice. As expected, the condensate fraction increases as entropy decreases, and the condensate fraction decreases at high lattice depths (due to quantum depletion). We compare our experimental results to condensate fraction predicted by the non-interacting, Hartree-Fock-Bogoliubov-Popov, and site-decoupled-mean-field theories. Theory and experiment disagree, which motivates several future extensions to this work, including calculating condensate fraction (and testing our fit procedure) using quantum Monte Carlo numerics, and experimentally and theoretically investigating the dynamics of the lattice load process (for the finite-temperature strongly correlated regime). Finally, we develop impurity probes for the Bose-Hubbard model by employing a spin-dependent lattice. A primary accomplishment of this thesis work was to develop the first 3D spin-dependent lattice in the strongly correlated regime (published in Ref. [4]). The spin-dependent lattice depth is proportional to |gFmF|, enabling the creation of mixtures of atoms trapped in the lattice (nonzero mF) co-trapped with atoms that do not experience the lattice (mF = 0). We use the non-lattice atoms as an impurity probe. We investigate using the impurity to probe the lattice temperature, and we determine that thermalization between the impurity and lattice gas is suppressed for larger lattice depths. Using a comparison to a Fermi’s golden rule calculation of the collisional energy exchange rate, we determine that this effect is consistent with suppression of energy-exchanging collisions by a mismatch between the impurity and lattice gas dispersion. While this result invalidates the concept of an impurity thermometer, it paves the way for a unique cooling scheme that relies on inter-species thermal isolation. We also explore impurity transport through the lattice gas. In other preliminary measurements, we also identify the decay rate of the center-of-mass motion as a prospective impurity probe. A separate aspect of this thesis work is the design and construction of a new 40K apparatus for single-site imaging of atoms to simulate the 2D Fermi-Hubbard model. The main component of this apparatus is high resolution fluorescence imaging on the 4S-5P transition of K at 404.5nm. Fluorescence imaging using this transition has two advantages over imaging on the standard D2 transition at 767nm: a smaller wavelength and therefore higher resolution, and a lower Doppler temperature limit which enables longer imaging times. To validate this approach, we demonstrate the first 40K magneto-optical trap (MOT) using the 404.5nm transition

On unifying sparsity and geometry for image-based 3D scene representation

Demand has emerged for next generation visual technologies that go beyond conventional 2D imaging. Such technologies should capture and communicate all perceptually relevant three-dimensional information about an environment to a distant observer, providing a satisfying, immersive experience. Camera networks offer a low cost solution to the acquisition of 3D visual information, by capturing multi-view images from different viewpoints. However, the camera's representation of the data is not ideal for common tasks such as data compression or 3D scene analysis, as it does not make the 3D scene geometry explicit. Image-based scene representations fundamentally require a multi-view image model that facilitates extraction of underlying geometrical relationships between the cameras and scene components. Developing new, efficient multi-view image models is thus one of the major challenges in image-based 3D scene representation methods. This dissertation focuses on defining and exploiting a new method for multi-view image representation, from which the 3D geometry information is easily extractable, and which is additionally highly compressible. The method is based on sparse image representation using an overcomplete dictionary of geometric features, where a single image is represented as a linear combination of few fundamental image structure features (edges for example). We construct the dictionary by applying a unitary operator to an analytic function, which introduces a composition of geometric transforms (translations, rotation and anisotropic scaling) to that function. The advantage of this approach is that the features across multiple views can be related with a single composition of transforms. We then establish a connection between image components and scene geometry by defining the transforms that satisfy the multi-view geometry constraint, and obtain a new geometric multi-view correlation model. We first address the construction of dictionaries for images acquired by omnidirectional cameras, which are particularly convenient for scene representation due to their wide field of view. Since most omnidirectional images can be uniquely mapped to spherical images, we form a dictionary by applying motions on the sphere, rotations, and anisotropic scaling to a function that lives on the sphere. We have used this dictionary and a sparse approximation algorithm, Matching Pursuit, for compression of omnidirectional images, and additionally for coding 3D objects represented as spherical signals. Both methods offer better rate-distortion performance than state of the art schemes at low bit rates. The novel multi-view representation method and the dictionary on the sphere are then exploited for the design of a distributed coding method for multi-view omnidirectional images. In a distributed scenario, cameras compress acquired images without communicating with each other. Using a reliable model of correlation between views, distributed coding can achieve higher compression ratios than independent compression of each image. However, the lack of a proper model has been an obstacle for distributed coding in camera networks for many years. We propose to use our geometric correlation model for distributed multi-view image coding with side information. The encoder employs a coset coding strategy, developed by dictionary partitioning based on atom shape similarity and multi-view geometry constraints. Our method results in significant rate savings compared to independent coding. An additional contribution of the proposed correlation model is that it gives information about the scene geometry, leading to a new camera pose estimation method using an extremely small amount of data from each camera. Finally, we develop a method for learning stereo visual dictionaries based on the new multi-view image model. Although dictionary learning for still images has received a lot of attention recently, dictionary learning for stereo images has been investigated only sparingly. Our method maximizes the likelihood that a set of natural stereo images is efficiently represented with selected stereo dictionaries, where the multi-view geometry constraint is included in the probabilistic modeling. Experimental results demonstrate that including the geometric constraints in learning leads to stereo dictionaries that give both better distributed stereo matching and approximation properties than randomly selected dictionaries. We show that learning dictionaries for optimal scene representation based on the novel correlation model improves the camera pose estimation and that it can be beneficial for distributed coding