Image Understanding by Socializing the Semantic Gap

Several technological developments like the Internet, mobile devices and Social Networks have spurred the sharing of images in unprecedented volumes, making tagging and commenting a common habit. Despite the recent progress in image analysis, the problem of Semantic Gap still hinders machines in fully understand the rich semantic of a shared photo. In this book, we tackle this problem by exploiting social network contributions. A comprehensive treatise of three linked problems on image annotation is presented, with a novel experimental protocol used to test eleven state-of-the-art methods. Three novel approaches to annotate, under stand the sentiment and predict the popularity of an image are presented. We conclude with the many challenges and opportunities ahead for the multimedia community

Phenotype Recognition with Combined Features and Random Subspace Classifier Ensemble

<p>Abstract</p> <p>Background</p> <p>Automated, image based high-content screening is a fundamental tool for discovery in biological science. Modern robotic fluorescence microscopes are able to capture thousands of images from massively parallel experiments such as RNA interference (RNAi) or small-molecule screens. As such, efficient computational methods are required for automatic cellular phenotype identification capable of dealing with large image data sets. In this paper we investigated an efficient method for the extraction of quantitative features from images by combining second order statistics, or Haralick features, with curvelet transform. A random subspace based classifier ensemble with multiple layer perceptron (MLP) as the base classifier was then exploited for classification. Haralick features estimate image properties related to second-order statistics based on the grey level co-occurrence matrix (GLCM), which has been extensively used for various image processing applications. The curvelet transform has a more sparse representation of the image than wavelet, thus offering a description with higher time frequency resolution and high degree of directionality and anisotropy, which is particularly appropriate for many images rich with edges and curves. A combined feature description from Haralick feature and curvelet transform can further increase the accuracy of classification by taking their complementary information. We then investigate the applicability of the random subspace (RS) ensemble method for phenotype classification based on microscopy images. A base classifier is trained with a RS sampled subset of the original feature set and the ensemble assigns a class label by majority voting.</p> <p>Results</p> <p>Experimental results on the phenotype recognition from three benchmarking image sets including HeLa, CHO and RNAi show the effectiveness of the proposed approach. The combined feature is better than any individual one in the classification accuracy. The ensemble model produces better classification performance compared to the component neural networks trained. For the three images sets HeLa, CHO and RNAi, the Random Subspace Ensembles offers the classification rates 91.20%, 98.86% and 91.03% respectively, which compares sharply with the published result 84%, 93% and 82% from a multi-purpose image classifier WND-CHARM which applied wavelet transforms and other feature extraction methods. We investigated the problem of estimation of ensemble parameters and found that satisfactory performance improvement could be brought by a relative medium dimensionality of feature subsets and small ensemble size.</p> <p>Conclusions</p> <p>The characteristics of curvelet transform of being multiscale and multidirectional suit the description of microscopy images very well. It is empirically demonstrated that the curvelet-based feature is clearly preferred to wavelet-based feature for bioimage descriptions. The random subspace ensemble of MLPs is much better than a number of commonly applied multi-class classifiers in the investigated application of phenotype recognition.</p

Model-driven and Data-driven Approaches for some Object Recognition Problems

Recognizing objects from images and videos has been a long standing problem in computer vision. The recent surge in the prevalence of visual cameras has given rise to two main challenges where, (i) it is important to understand different sources of object variations in more unconstrained scenarios, and (ii) rather than describing an object in isolation, efficient learning methods for modeling object-scene `contextual' relations are required to resolve visual ambiguities. This dissertation addresses some aspects of these challenges, and consists of two parts. First part of the work focuses on obtaining object descriptors that are largely preserved across certain sources of variations, by utilizing models for image formation and local image features. Given a single instance of an object, we investigate the following three problems. (i) Representing a 2D projection of a 3D non-planar shape invariant to articulations, when there are no self-occlusions. We propose an articulation invariant distance that is preserved across piece-wise affine transformations of a non-rigid object `parts', under a weak perspective imaging model, and then obtain a shape context-like descriptor to perform recognition; (ii) Understanding the space of `arbitrary' blurred images of an object, by representing an unknown blur kernel of a known maximum size using a complete set of orthonormal basis functions spanning that space, and showing that subspaces resulting from convolving a clean object and its blurred versions with these basis functions are equal under some assumptions. We then view the invariant subspaces as points on a Grassmann manifold, and use statistical tools that account for the underlying non-Euclidean nature of the space of these invariants to perform recognition across blur; (iii) Analyzing the robustness of local feature descriptors to different illumination conditions. We perform an empirical study of these descriptors for the problem of face recognition under lighting change, and show that the direction of image gradient largely preserves object properties across varying lighting conditions. The second part of the dissertation utilizes information conveyed by large quantity of data to learn contextual information shared by an object (or an entity) with its surroundings. (i) We first consider a supervised two-class problem of detecting lane markings from road video sequences, where we learn relevant feature-level contextual information through a machine learning algorithm based on boosting. We then focus on unsupervised object classification scenarios where, (ii) we perform clustering using maximum margin principles, by deriving some basic properties on the affinity of `a pair of points' belonging to the same cluster using the information conveyed by `all' points in the system, and (iii) then consider correspondence-free adaptation of statistical classifiers across domain shifting transformations, by generating meaningful `intermediate domains' that incrementally convey potential information about the domain change

Speech Recognition

Chapters in the first part of the book cover all the essential speech processing techniques for building robust, automatic speech recognition systems: the representation for speech signals and the methods for speech-features extraction, acoustic and language modeling, efficient algorithms for searching the hypothesis space, and multimodal approaches to speech recognition. The last part of the book is devoted to other speech processing applications that can use the information from automatic speech recognition for speaker identification and tracking, for prosody modeling in emotion-detection systems and in other speech processing applications that are able to operate in real-world environments, like mobile communication services and smart homes

ELUCIDATION OF THE MECHANISM OF ACTION OF A RESPIRATORY SYNCYTIAL VIRUS SUBUNIT VACCINE CANDIDATE CONTAINING A POLYMER-BASED COMBINATION ADJUVANT

Human respiratory syncytial virus (RSV) is the primary cause of respiratory illnesses in infants, young children, elderly and immunocompromised individuals. Supportive care is the mainstay of RSV treatment. Currently no licensed vaccine against RSV is available. We have developed a subunit RSV vaccine candidate (ΔF/TriAdj) consisting of a truncated version of the RSV fusion protein (ΔF) formulated with a combination adjuvant (TriAdj) comprised of low molecular weight (LMW) polyinosinic:polycytidylic acid [poly(I:C)], an innate defense regulator (IDR) peptide and poly[di(sodium carboxylatoethylphenoxy)]-phosphazene (PCEP). We previously demonstrated the safety and protective efficacy of ΔF/TriAdj in several animal models. The overall objective of this thesis was to elucidate the mechanism of action of ΔF/TriAdj in BALB/c mice. First, we determined that ΔF/TriAdj when delivered intranasally plays a crucial role in stimulating innate immune responses in both upper and lower respiratory tracts of immunized mice as demonstrated by local production of cytokines, chemokines and interferons, as well as infiltration and activation of immune cells. Innate activation subsequently led to robust adaptive immunity and protection against RSV. Next, we elucidated the mechanisms of action of ΔF/TriAdj at the cell-signaling level in macrophages. Macrophages responded directly to in vitro stimulation with ΔF/TriAdj with induction of both endosomal and cytosolic pattern recognition receptors (PRRs). Based on inhibition studies, we determined that multiple signal transduction pathways are involved in ΔF/TriAdj-mediated activation of macrophages. Finally, we conducted a comprehensive chemical isotope labeling liquid chromatography-mass spectrometry (CIL LC-MS) analysis of the lung tissues from vaccinated and unvaccinated, RSV-infected mice as well as healthy controls, to understand the underlying mechanisms of action of ΔF/TriAdj at the further downstream metabolomic level. Metabolomic profiling revealed alterations of tryptophan metabolism (including kynurenine pathway), biosynthesis of amino acids (including arginine biosynthesis), urea cycle and tyrosine metabolism due to RSV infection. Interestingly, ΔF/TriAdj was found to a play a critical role in modulating alterations in the concentrations of the metabolites of the above-mentioned pathways in response to RSV infection. Ultimately, information on the mechanism of action of this RSV vaccine candidate may serve to identify potential biomarkers for immunogenicity and protective efficacy of ΔF/TriAdj in future