Performance Analysis of l_0 Norm Constraint Least Mean Square Algorithm

As one of the recently proposed algorithms for sparse system identification, $l_0$ norm constraint Least Mean Square ($l_0$-LMS) algorithm modifies the cost function of the traditional method with a penalty of tap-weight sparsity. The performance of $l_0$-LMS is quite attractive compared with its various precursors. However, there has been no detailed study of its performance. This paper presents all-around and throughout theoretical performance analysis of $l_0$-LMS for white Gaussian input data based on some reasonable assumptions. Expressions for steady-state mean square deviation (MSD) are derived and discussed with respect to algorithm parameters and system sparsity. The parameter selection rule is established for achieving the best performance. Approximated with Taylor series, the instantaneous behavior is also derived. In addition, the relationship between $l_0$-LMS and some previous arts and the sufficient conditions for $l_0$-LMS to accelerate convergence are set up. Finally, all of the theoretical results are compared with simulations and are shown to agree well in a large range of parameter setting.Comment: 31 pages, 8 figure

Exact and approximate polynomial decomposition methods for signal processing applications

Signal processing is a discipline in which functional composition and decomposition can potentially be utilized in a variety of creative ways. From an analysis point of view, further insight can be gained into existing signal processing systems and techniques by reinterpreting them in terms of functional composition. From a synthesis point of view, functional composition offers new algorithms and techniques with modular structure. Moreover, computations can be performed more efficiently and data can be represented more compactly in information systems represented in the context of a compositional structure. Polynomials are ubiquitous in signal processing in the form of z-transforms. In this paper, we summarize the fundamentals of functional composition and decomposition for polynomials from the perspective of exploiting them in signal processing. We compare exact polynomial decomposition algorithms for sequences that are exactly decomposable when expressed as a polynomial, and approximate decomposition algorithms for those that are not exactly decomposable. Furthermore, we identify efficiencies in using exact decomposition techniques in the context of signal processing and introduce a new approximate polynomial decomposition technique based on the use of Structured Total Least Norm (STLN) formulation.Texas Instruments Leadership University Consortium ProgramBose (Firm

Polynomial decomposition algorithms in signal processing

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 107-109).Polynomial decomposition has attracted considerable attention in computational mathematics. In general, the field identifies polynomials f(x) and g(x) such that their composition f(g(x)) equals or approximates a given polynomial h(x). Despite potentially promising applications, polynomial decomposition has not been significantly utilized in signal processing. This thesis studies the sensitivities of polynomial composition and decomposition to explore their robustness in potential signal processing applications and develops effective polynomial decomposition algorithms to be applied in a signal processing context. First, we state the problems of sensitivity, exact decomposition, and approximate decomposition. After that, the sensitivities of the composition and decomposition operations are theoretically derived from the perspective of robustness. In particular, we present and validate an approach to decrease certain sensitivities by using equivalent compositions, and a practical rule for parameter selection is proposed to get to a point that is near the minimum of these sensitivities. Then, new algorithms are proposed for the exact decomposition problems, and simulations are performed to make comparison with existing approaches. Finally, existing and new algorithms for the approximate decomposition problems are presented and evaluated using numerical simulations.by Guolong Su.S.M

QueryForm: A Simple Zero-shot Form Entity Query Framework

Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities. We present a novel query-based framework, QueryForm, that extracts entity values from form-like documents in a zero-shot fashion. QueryForm contains a dual prompting mechanism that composes both the document schema and a specific entity type into a query, which is used to prompt a Transformer model to perform a single entity extraction task. Furthermore, we propose to leverage large-scale query-entity pairs generated from form-like webpages with weak HTML annotations to pre-train QueryForm. By unifying pre-training and fine-tuning into the same query-based framework, QueryForm enables models to learn from structured documents containing various entities and layouts, leading to better generalization to target document types without the need for target-specific training data. QueryForm sets new state-of-the-art average F1 score on both the XFUND (+4.6%~10.1%) and the Payment (+3.2%~9.5%) zero-shot benchmark, with a smaller model size and no additional image input.Comment: Accepted to Findings of ACL 202

LMDX: Language Model-based Document Information Extraction and Localization

Large Language Models (LLM) have revolutionized Natural Language Processing (NLP), improving state-of-the-art on many existing tasks and exhibiting emergent capabilities. However, LLMs have not yet been successfully applied on semi-structured document information extraction, which is at the core of many document processing workflows and consists of extracting key entities from a visually rich document (VRD) given a predefined target schema. The main obstacles to LLM adoption in that task have been the absence of layout encoding within LLMs, critical for a high quality extraction, and the lack of a grounding mechanism ensuring the answer is not hallucinated. In this paper, we introduce Language Model-based Document Information Extraction and Localization (LMDX), a methodology to adapt arbitrary LLMs for document information extraction. LMDX can do extraction of singular, repeated, and hierarchical entities, both with and without training data, while providing grounding guarantees and localizing the entities within the document. In particular, we apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks, setting a new state-of-the-art and showing how LMDX enables the creation of high quality, data-efficient parsers

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.Comment: Accepted to ACL 202

Service-Oriented Node Scheduling Scheme for Wireless Sensor Networks Using Markov Random Field Model

Future wireless sensor networks are expected to provide various sensing services and energy efficiency is one of the most important criterions. The node scheduling strategy aims to increase network lifetime by selecting a set of sensor nodes to provide the required sensing services in a periodic manner. In this paper, we are concerned with the service-oriented node scheduling problem to provide multiple sensing services while maximizing the network lifetime. We firstly introduce how to model the data correlation for different services by using Markov Random Field (MRF) model. Secondly, we formulate the service-oriented node scheduling issue into three different problems, namely, the multi-service data denoising problem which aims at minimizing the noise level of sensed data, the representative node selection problem concerning with selecting a number of active nodes while determining the services they provide, and the multi-service node scheduling problem which aims at maximizing the network lifetime. Thirdly, we propose a Multi-service Data Denoising (MDD) algorithm, a novel multi-service Representative node Selection and service Determination (RSD) algorithm, and a novel MRF-based Multi-service Node Scheduling (MMNS) scheme to solve the above three problems respectively. Finally, extensive experiments demonstrate that the proposed scheme efficiently extends the network lifetime.This work is supported by the National Science Foundation of China under Grand No. 61370210 and the Development Foundation of Educational Committee of Fujian Province under Grand No. 2012JA12027.Cheng, H.; Su, Z.; Lloret, J.; Chen, G. (2014). Service-Oriented Node Scheduling Scheme for Wireless Sensor Networks Using Markov Random Field Model. Sensors. 14(11):20940-20962. https://doi.org/10.3390/s141120940S2094020962141

Sensitivity of polynomial composition and decomposition for signal processing applications

Polynomial composition is well studied in mathematics but has only been exploited indirectly and informally in signal processing. Potential future application of polynomial composition for filter implementation and data representation is dependent on its robustness both in forming higher degree polynomials from ones of lower degree and in exactly or approximately decomposing a polynomial into a composed form. This paper addresses robustness in this context, developing sensitivity bounds for both polynomial composition and decomposition and illustrates the sensitivity through simulations. It also demonstrates that sensitivity can be reduced by exploiting composition with first order polynomials and commutative polynomials.Texas Instruments Leadership University Consortium ProgramBose (Firm

Composition structures for system representation

Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Cataloged from student-submitted PDF version of thesis.Includes bibliographical references (pages 171-183).This thesis discusses parameter estimation algorithms for a number of structures for system representation that can be interpreted as different types of composition. We refer to the term composition as the systematic replacement of elements in an object by other object modules, where the objects can be functions that have a single or multiple input variables as well as operators that work on a set of signals of interest. In general, composition structures can be regarded as an important class of constrained parametric representations, which are widely used in signal processing. Different types of composition are considered in this thesis, including multivariate function composition, operator composition that naturally corresponds to cascade systems, and modular composition that we refer to as the replacement of each delay element in a system block diagram with an identical copy of another system module. There are a number of potential advantages of the use of composition structures in signal processing, such as reduction of the total number of independent parameters that achieves representational and computational efficiency, modular structures that benefit hardware implementation, and the ability to form more sophisticated models that can represent significantly larger classes of systems or functions. The first part of this thesis considers operator composition, which is an alternative interpretation of the class of cascade systems that has been widely studied in signal processing. As an important class of linear time-invariant (LTI) systems, we develop new algorithms to approximate a two-dimensional (2D) finite impulse response (FIR) filter as a cascade of a pair of 2D FIR filters with lower orders, which can gain computational efficiency. For nonlinear systems with a cascade structure, we generalize a two-step parameter estimation algorithm for the Hammerstein model, and propose a generalized all-pole modeling technique with the cascade of multiple nonlinear memoryless functions and LTI subsystems. The second part of this thesis discusses modular composition, which replaces each delay element in a FIR filter with another subsystem. As an example, we propose the modular Volterra system where the subsystem has the form of the Volterra series. Given statistical information between input and output signals, an algorithm is proposed to estimate the coefficients of the FIR filter and the kernels of the Volterra subsystem, under the assumption that the coefficients of the nonlinear kernels have sufficiently small magnitude. The third part of this thesis focuses on composition of multivariate functions. In particular, we consider two-level Boolean functions in the conjunctive or disjunctive normal forms, which can be considered as the composition of one-level multivariate Boolean functions that take the logical conjunction (or disjunction) over a subset of binary input variables. We propose new optimization-based approaches for learning a two-level Boolean function from a training dataset for classification purposes, with the joint criteria of accuracy and simplicity of the learned function.by Guolong Su.Ph. D