Structured Learning with Parsimony in Measurements and Computations: Theory, Algorithms, and Applications

University of Minnesota Ph.D. dissertation. July 2018. Major: Electrical Engineering. Advisor: Jarvis Haupt. 1 computer file (PDF); xvi, 289 pages.In modern ``Big Data'' applications, structured learning is the most widely employed methodology. Within this paradigm, the fundamental challenge lies in developing practical, effective algorithmic inference methods. Often (e.g., deep learning) successful heuristic-based approaches exist but theoretical studies are far behind, limiting understanding and potential improvements. In other settings (e.g., recommender systems) provably effective algorithmic methods exist, but the sheer sizes of datasets can limit their applicability. This twofold challenge motivates this work on developing new analytical and algorithmic methods for structured learning, with a particular focus on parsimony in measurements and computation, i.e., those requiring low storage and computational costs. Toward this end, we make efforts to investigate the theoretical properties of models and algorithms that present significant improvement in measurement and computation requirement. In particular, we first develop randomized approaches for dimensionality reduction on matrix and tensor data, which allow accurate estimation and inference procedures using significantly smaller data sizes that only depend on the intrinsic dimension (e.g., the rank of matrix/tensor) rather than the ambient ones. Our next effort is to study iterative algorithms for solving high dimensional learning problems, including both convex and nonconvex optimization. Using contemporary analysis techniques, we demonstrate guarantees of iteration complexities that are analogous to the low dimensional cases. In addition, we explore the landscape of nonconvex optimizations that exhibit computational advantages over their convex counterparts and characterize their properties from a general point of view in theory

Towards Lifelong Reasoning with Sparse and Compressive Memory Systems

Humans have a remarkable ability to remember information over long time horizons. When reading a book, we build up a compressed representation of the past narrative, such as the characters and events that have built up the story so far. We can do this even if they are separated by thousands of words from the current text, or long stretches of time between readings. During our life, we build up and retain memories that tell us where we live, what we have experienced, and who we are. Adding memory to artificial neural networks has been transformative in machine learning, allowing models to extract structure from temporal data, and more accurately model the future. However the capacity for long-range reasoning in current memory-augmented neural networks is considerably limited, in comparison to humans, despite the access to powerful modern computers. This thesis explores two prominent approaches towards scaling artificial memories to lifelong capacity: sparse access and compressive memory structures. With sparse access, the inspection, retrieval, and updating of only a very small subset of pertinent memory is considered. It is found that sparse memory access is beneficial for learning, allowing for improved data-efficiency and improved generalisation. From a computational perspective - sparsity allows scaling to memories with millions of entities on a simple CPU-based machine. It is shown that memory systems that compress the past to a smaller set of representations reduce redundancy and can speed up the learning of rare classes and improve upon classical data-structures in database systems. Compressive memory architectures are also devised for sequence prediction tasks and are observed to significantly increase the state-of-the-art in modelling natural language

PERICLES Deliverable 4.3:Content Semantics and Use Context Analysis Techniques

The current deliverable summarises the work conducted within task T4.3 of WP4, focusing on the extraction and the subsequent analysis of semantic information from digital content, which is imperative for its preservability. More specifically, the deliverable defines content semantic information from a visual and textual perspective, explains how this information can be exploited in long-term digital preservation and proposes novel approaches for extracting this information in a scalable manner. Additionally, the deliverable discusses novel techniques for retrieving and analysing the context of use of digital objects. Although this topic has not been extensively studied by existing literature, we believe use context is vital in augmenting the semantic information and maintaining the usability and preservability of the digital objects, as well as their ability to be accurately interpreted as initially intended.PERICLE

Large-scale interactive exploratory visual search

Large scale visual search has been one of the challenging issues in the era of big data. It demands techniques that are not only highly effective and efficient but also allow users conveniently express their information needs and refine their intents. In this thesis, we focus on developing an exploratory framework for large scale visual search. We also develop a number of enabling techniques in this thesis, including compact visual content representation for scalable search, near duplicate video shot detection, and action based event detection. We propose a novel scheme for extremely low bit rate visual search, which sends compressed visual words consisting of vocabulary tree histogram and descriptor orientations rather than descriptors. Compact representation of video data is achieved through identifying keyframes of a video which can also help users comprehend visual content efficiently. We propose a novel Bag-of-Importance model for static video summarization. Near duplicate detection is one of the key issues for large scale visual search, since there exist a large number nearly identical images and videos. We propose an improved near-duplicate video shot detection approach for more effective shot representation. Event detection has been one of the solutions for bridging the semantic gap in visual search. We particular focus on human action centred event detection. We propose an enhanced sparse coding scheme to model human actions. Our proposed approach is able to significantly reduce computational cost while achieving recognition accuracy highly comparable to the state-of-the-art methods. At last, we propose an integrated solution for addressing the prime challenges raised from large-scale interactive visual search. The proposed system is also one of the first attempts for exploratory visual search. It provides users more robust results to satisfy their exploring experiences

Structure-oriented prediction in complex networks

Complex systems are extremely hard to predict due to its highly nonlinear interactions and rich emergent properties. Thanks to the rapid development of network science, our understanding of the structure of real complex systems and the dynamics on them has been remarkably deepened, which meanwhile largely stimulates the growth of effective prediction approaches on these systems. In this article, we aim to review different network-related prediction problems, summarize and classify relevant prediction methods, analyze their advantages and disadvantages, and point out the forefront as well as critical challenges of the field

Quantitative Character and the Composite Account of Phenomenal Content

I advance an account of quantitative character, a species of phenomenal character that presents as an intensity (cf. a quality) and includes experience dimensions such as loudness, pain intensity, and visual pop-out. I employ psychological and neuroscientific evidence to demonstrate that quantitative characters are best explained by attentional processing, and hence that they do not represent external qualities. Nonetheless, the proposed account of quantitative character is conceived as a compliment to the reductive intentionalist strategy toward qualitative states; I argue that an account of perceptual experience that combines a tracking account of qualitative character with my functionalist proposal of quantitative character permits replies to some notoriously difficult problems for tracking representationalism without sacrificing its chief virtues

Ergonomic Evaluation of Manually Operated Valves.

Due to the large number of facilities that produce oil and gas with numerous valves used, a review of valve handwheel operation was of interest. Review of literature on two handed handwheel tasks yielded very little and raised questions about what amount of torque a user could exert on a handwheel. The objectives of this research were to determine the effects of height of exertion, orientation of the handwheel and type of handwheel while making two-handed torque exertions, test a custom designed handwheel against commonly distributed handwheels and to determine the effects of gloves commonly used in the oil and gas industry on two-handed torque generation capability. The objectives were addressed through two experiments. In experiment one, it was found that the custom designed handwheel allowed for the generation of significantly different and greater torque than the two industry handwheels, that the height of exertion was significant only in the case of the overhead height, and that the vertical orientation of the handwheel allowed for more torque generation. A second analysis was conducted without the overhead height data which indicated that the custom designed handwheel remained significantly better than the other two handwheels, and that the vertical orientation allowed for the generation of the most torque in the majority of configurations. These results also indicated that floor height is significantly different from the other three heights. The development of a predictive equation for torque capability based on task, anthropometric and strength factors was not as robust as deemed necessary due to the small sample size. The results of the second experiment indicated that no one glove type was significantly different from the other, but that the cotton glove with plastic dotting did allow for the generation of greater torque than the bare-handed condition which was greater than the leather gloved condition. It was also found that the quarter arc handwheel allowed for the generation of significantly greater torque than did the circular handwheel. The effect of gender was also seen with females generating 46.78% of the torque of males

A Molecular Dynamics Study of the Structure-Dynamics Relationships of Supercooled Liquids and Glasses

Central to the field of condensed matter physics is a decades old outstanding problem in the study of glasses – namely explaining the extreme slowing of dynamics in a liquid as it is supercooled towards the so-called glass transition. Efforts to universally describe the stretched relaxation processes and heterogeneous dynamics that characteristically develop in supercooled liquids remain divided in both their approaches and successes. Towards this end, a consensus on the role that atomic and molecular structures play in the liquid is even more tenuous. However, mounting material science research efforts have culminated to reveal that the vast diversity of metallic glass species and their properties are rooted in an equally-broad set of structural archetypes. Herein lies the motivation of this dissertation: the detailed information available regarding the structure-property relationships of metallic glasses provides a new context in which one can study the evolution of a supercooled liquid by utilizing a structural motif that is known to dominate the glass. Cu_64 Zr_36 is a binary alloy whose good glass-forming ability and simple composition makes it a canonical material to both empirical and numerical studies. Here, we perform classical molecular dynamics simulations and conduct a comprehensive analysis of the dynamical regimes of liquid Cu_64 Zr_36, while focusing on the roles played by atomic icosahedral ordering – a structural motif which ultimately percolates the glass’ structure. Large data analysis techniques are leveraged to obtain uniquely detailed structural and dynamical information in this context. In doing so, we develop the first account of the origin of icosahedral order in this alloy, revealing deep connections between this incipient structural ordering, frustration-limited domain theory, and recent important empirical findings that are relevant to the nature of metallic liquids at large. Furthermore, important dynamical landmarks such as the breakdown of the Stokes-Einstein relationship, the decoupling of particle diffusivities, and the development of general “glassy” relaxation features are found to coincide with successive manifestation of icosahedral ordering that arise as the liquid is supercooled. Remarkably, we detect critical-like features in the growth of the icosahedron network, with signatures that suggest that a liquid-liquid phase transition may occur in the deeply supercooled regime to precede glass formation. Such a transition is predicted to occur in many supercooled liquids, although explicit evidence of this phenomenon in realistic systems is scarce. Ultimately this work concludes that icosahedral order characterizes all dynamical regimes of Cu_64 Zr_36, demonstrating the importance and utility of studying supercooled liquids in the context of locally-preferred structure. More broadly, it serves to confirm and inform recent theoretical and empirical findings that are central to understanding the physics underlying the glass transition