16 research outputs found

    Matrix completion of noisy graph signals via proximal gradient minimization

    Get PDF
    ©2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper takes on the problem of recovering the missing entries of an incomplete matrix, which is known as matrix completion, when the columns of the matrix are signals that lie on a graph and the available observations are noisy. We solve a version of the problem regularized with the Laplacian quadratic form by means of the proximal gradient method, and derive theoretical bounds on the recovery error. Moreover, in order to speed up the convergence of the proximal gradient, we propose an initialization method that utilizes the structural information contained in the Laplacian matrix of the graph.Peer ReviewedPostprint (author's final draft

    Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

    Full text link
    Transformer-based Large Language Models (LLMs) have been applied in diverse areas such as knowledge bases, human interfaces, and dynamic agents, and marking a stride towards achieving Artificial General Intelligence (AGI). However, current LLMs are predominantly pretrained on short text snippets, which compromises their effectiveness in processing the long-context prompts that are frequently encountered in practical scenarios. This article offers a comprehensive survey of the recent advancement in Transformer-based LLM architectures aimed at enhancing the long-context capabilities of LLMs throughout the entire model lifecycle, from pre-training through to inference. We first delineate and analyze the problems of handling long-context input and output with the current Transformer-based models. We then provide a taxonomy and the landscape of upgrades on Transformer architecture to solve these problems. Afterwards, we provide an investigation on wildly used evaluation necessities tailored for long-context LLMs, including datasets, metrics, and baseline models, as well as optimization toolkits such as libraries, frameworks, and compilers to boost the efficacy of LLMs across different stages in runtime. Finally, we discuss the challenges and potential avenues for future research. A curated repository of relevant literature, continuously updated, is available at https://github.com/Strivin0311/long-llms-learning.Comment: 40 pages, 3 figures, 4 table

    Energy-Aware Data Movement In Non-Volatile Memory Hierarchies

    Get PDF
    While technology scaling enables increased density for memory cells, the intrinsic high leakage power of conventional CMOS technology and the demand for reduced energy consumption inspires the use of emerging technology alternatives such as eDRAM and Non-Volatile Memory (NVM) including STT-MRAM, PCM, and RRAM. The utilization of emerging technology in Last Level Cache (LLC) designs which occupies a signifcant fraction of total die area in Chip Multi Processors (CMPs) introduces new dimensions of vulnerability, energy consumption, and performance delivery. To be specific, a part of this research focuses on eDRAM Bit Upset Vulnerability Factor (BUVF) to assess vulnerable portion of the eDRAM refresh cycle where the critical charge varies depending on the write voltage, storage and bit-line capacitance. This dissertation broaden the study on vulnerability assessment of LLC through investigating the impact of Process Variations (PV) on narrow resistive sensing margins in high-density NVM arrays, including on-chip cache and primary memory. Large-latency and power-hungry Sense Amplifers (SAs) have been adapted to combat PV in the past. Herein, a novel approach is proposed to leverage the PV in NVM arrays using Self-Organized Sub-bank (SOS) design. SOS engages the preferred SA alternative based on the intrinsic as-built behavior of the resistive sensing timing margin to reduce the latency and power consumption while maintaining acceptable access time. On the other hand, this dissertation investigates a novel technique to prioritize the service to 1) Extensive Read Reused Accessed blocks of the LLC that are silently dropped from higher levels of cache, and 2) the portion of the working set that may exhibit distant re-reference interval in L2. In particular, we develop a lightweight Multi-level Access History Profiler to effciently identify ERRA blocks through aggregating the LLC block addresses tagged with identical Most Signifcant Bits into a single entry. Experimental results indicate that the proposed technique can reduce the L2 read miss ratio by 51.7% on average across PARSEC and SPEC2006 workloads. In addition, this dissertation will broaden and apply advancements in theories of subspace recovery to pioneer computationally-aware in-situ operand reconstruction via the novel Logic In Interconnect (LI2) scheme. LI2 will be developed, validated, and re?ned both theoretically and experimentally to realize a radically different approach to post-Moore\u27s Law computing by leveraging low-rank matrices features offering data reconstruction instead of fetching data from main memory to reduce energy/latency cost per data movement. We propose LI2 enhancement to attain high performance delivery in the post-Moore\u27s Law era through equipping the contemporary micro-architecture design with a customized memory controller which orchestrates the memory request for fetching low-rank matrices to customized Fine Grain Reconfigurable Accelerator (FGRA) for reconstruction while the other memory requests are serviced as before. The goal of LI2 is to conquer the high latency/energy required to traverse main memory arrays in the case of LLC miss, by using in-situ construction of the requested data dealing with low-rank matrices. Thus, LI2 exchanges a high volume of data transfers with a novel lightweight reconstruction method under specific conditions using a cross-layer hardware/algorithm approach

    New and Provable Results for Network Inference Problems and Multi-agent Optimization Algorithms

    Get PDF
    abstract: Our ability to understand networks is important to many applications, from the analysis and modeling of biological networks to analyzing social networks. Unveiling network dynamics allows us to make predictions and decisions. Moreover, network dynamics models have inspired new ideas for computational methods involving multi-agent cooperation, offering effective solutions for optimization tasks. This dissertation presents new theoretical results on network inference and multi-agent optimization, split into two parts - The first part deals with modeling and identification of network dynamics. I study two types of network dynamics arising from social and gene networks. Based on the network dynamics, the proposed network identification method works like a `network RADAR', meaning that interaction strengths between agents are inferred by injecting `signal' into the network and observing the resultant reverberation. In social networks, this is accomplished by stubborn agents whose opinions do not change throughout a discussion. In gene networks, genes are suppressed to create desired perturbations. The steady-states under these perturbations are characterized. In contrast to the common assumption of full rank input, I take a laxer assumption where low-rank input is used, to better model the empirical network data. Importantly, a network is proven to be identifiable from low rank data of rank that grows proportional to the network's sparsity. The proposed method is applied to synthetic and empirical data, and is shown to offer superior performance compared to prior work. The second part is concerned with algorithms on networks. I develop three consensus-based algorithms for multi-agent optimization. The first method is a decentralized Frank-Wolfe (DeFW) algorithm. The main advantage of DeFW lies on its projection-free nature, where we can replace the costly projection step in traditional algorithms by a low-cost linear optimization step. I prove the convergence rates of DeFW for convex and non-convex problems. I also develop two consensus-based alternating optimization algorithms --- one for least square problems and one for non-convex problems. These algorithms exploit the problem structure for faster convergence and their efficacy is demonstrated by numerical simulations. I conclude this dissertation by describing future research directions.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

    Image Reconstruction and Motion Compensation Methods for Fast MRI Chaoping

    Get PDF

    Image Reconstruction and Motion Compensation Methods for Fast MRI Chaoping

    Get PDF

    A Study of Computationally Efficient Advanced Battery Management: Modeling, Identification, Estimation and Control

    Get PDF
    Lithium-ion batteries (LiBs) are a revolutionary technology for energy storage. They have become a dominant power source for consumer electronics and are rapidly penetrating into the sectors of electrified transportation and renewable energies, due to the high energy/power density, long cycle life and low memory effect. With continuously falling prices, they will become more popular in foreseeable future. LiBs demonstrate complex dynamic behaviors and are vulnerable to a number of operating problems including overcharging, overdischarging and thermal runaway. Hence, battery management systems (BMSs) are needed in practice to extract full potential from them and ensure their operational safety. Recent years have witnessed a growing amount of research on BMSs, which usually involves topics such as dynamic modeling, parameter identification, state estimation, cell balancing, optimal charging, thermal management, and fault detection. A common challenge for them is computational efficiency since BMSs typically run on embedded systems with limited computing and memory capabilities. Inspired by the challenge, this dissertation aims to address a series of problems towards advancing BMSs with low computational complexity but still high performance. Specifically, the efforts will focus on novel battery modeling and parameter identification (Chapters 2 and 3), highly efficient optimal charging control (Chapter 4) and spatio-temporal temperature estimation of LiB packs (Chapter 5). The developed new LiB models and algorithms can hopefully find use in future LiB systems to improve their performance, while offering insights into some key challenges in the field of BMSs. The research will also entail the development of some fundamental technical approaches concerning parameter identification, model predictive control and state estimation, which have a prospect of being applied to dynamic systems in various other problem domains

    The 1981 NASA/ASEE Summer Faculty Fellowship Program: Research reports

    Get PDF
    Research reports related to spacecraft industry technological advances, requirements, and applications were considered. Some of the topic areas addressed were: (1) Fabrication, evaluation, and use of high performance composites and ceramics, (2) antenna designs, (3) electronics and microcomputer applications and mathematical modeling and programming techniques, (4) design, fabrication, and failure detection methods for structural materials, components, and total systems, and (5) chemical studies of bindary organic mixtures and polymer synthesis. Space environment parameters were also discussed
    corecore