16 research outputs found

    LIPIcs, Volume 261, ICALP 2023, Complete Volume

    Get PDF
    LIPIcs, Volume 261, ICALP 2023, Complete Volum

    Exposing Attention Glitches with Flip-Flop Language Modeling

    Full text link
    Why do large language models sometimes output factual inaccuracies and exhibit erroneous reasoning? The brittleness of these models, particularly when executing long chains of reasoning, currently seems to be an inevitable price to pay for their advanced capabilities of coherently synthesizing knowledge, pragmatics, and abstract thought. Towards making sense of this fundamentally unsolved problem, this work identifies and analyzes the phenomenon of attention glitches, in which the Transformer architecture's inductive biases intermittently fail to capture robust reasoning. To isolate the issue, we introduce flip-flop language modeling (FFLM), a parametric family of synthetic benchmarks designed to probe the extrapolative behavior of neural language models. This simple generative task requires a model to copy binary symbols over long-range dependencies, ignoring the tokens in between. We find that Transformer FFLMs suffer from a long tail of sporadic reasoning errors, some of which we can eliminate using various regularization techniques. Our preliminary mechanistic analyses show why the remaining errors may be very difficult to diagnose and resolve. We hypothesize that attention glitches account for (some of) the closed-domain hallucinations in natural LLMs.Comment: v2: NeurIPS 2023 camera-ready + data releas

    Attribute Exploration of Gene Regulatory Processes

    Get PDF
    This thesis aims at the logical analysis of discrete processes, in particular of such generated by gene regulatory networks. States, transitions and operators from temporal logics are expressed in the language of Formal Concept Analysis. By the attribute exploration algorithm, an expert or a computer program is enabled to validate a minimal and complete set of implications, e.g. by comparison of predictions derived from literature with observed data. Here, these rules represent temporal dependencies within gene regulatory networks including coexpression of genes, reachability of states, invariants or possible causal relationships. This new approach is embedded into the theory of universal coalgebras, particularly automata, Kripke structures and Labelled Transition Systems. A comparison with the temporal expressivity of Description Logics is made. The main theoretical results concern the integration of background knowledge into the successive exploration of the defined data structures (formal contexts). Applying the method a Boolean network from literature modelling sporulation of Bacillus subtilis is examined. Finally, we developed an asynchronous Boolean network for extracellular matrix formation and destruction in the context of rheumatoid arthritis.Comment: 111 pages, 9 figures, file size 2.1 MB, PhD thesis University of Jena, Germany, Faculty of Mathematics and Computer Science, 2011. Online available at http://www.db-thueringen.de/servlets/DocumentServlet?id=1960

    Interval Groupoids

    Get PDF
    This book introduces several new classes of groupoid, like polynomial groupoids, matrix groupoids, interval groupoids,polynomial interval groupoids, matrix interval groupoids and their neutrosophic analogues. Interval groupoid happens to be the first non-associative structure constructed using intervals built using Zn or Z or Q or R or Z+ \cup {0} or Q+ \cup {0} and so on. This book has five chapters. Chapter one is introductory in nature. In chapter two new classes of groupoids and interval groupoids are defined and described. The analogous neutrosophic study is carried out in chapter three. The applications of this new structure is given in chapter four. The final chapter suggests more than 200 problems. This book has given 77 new definitions, 426 examples of these new notions and over 150 theorems.Comment: 240 page

    SMARANDACHE SPECIAL DEFINITE ALGEBRAIC STRUCTURES

    Get PDF
    Introducing the notion of Smarandache special definite algebraic structures, also called equivalently as Smarandache definite special algebraic structures. These new structures are defined as those strong algebraic structures which have in them a proper subset which is a weak algebraic structure. For instance, the existence of a semigroup in a group or a semifield in a field or a semiring in a ring. It is interesting to note that these concepts cannot be defined when the algebraic structure has finite cardinality i.e., when the algebraic structure has finite number of elements in it

    Mathematical linguistics

    Get PDF
    but in fact this is still an early draft, version 0.56, August 1 2001. Please d

    N-ALGEBRAIC STRUCTURES AND S-N-ALGEBRAIC STRUCTURES

    Get PDF
    In this book, for the first time we introduce the notions of Ngroups, N-semigroups, N-loops and N-groupoids. We also define a mixed N-algebraic structure. We expect the reader to be well versed in group theory and have at least basic knowledge about Smarandache groupoids, Smarandache loops, Smarandache semigroups and bialgebraic structures and Smarandache bialgebraic structures. The book is organized into six chapters. The first chapter gives the basic notions of S-semigroups, S-groupoids and S-loops thereby making the book self-contained. Chapter two introduces N-groups and their Smarandache analogues. In chapter three, Nloops and Smarandache N-loops are introduced and analyzed. Chapter four defines N-groupoids and S-N-groupoids. Since the N-semigroup structures are sandwiched between groups and groupoids, the study can be carried out without any difficulty. Mixed N-algebraic structures and S-mixed algebraic structures are given in chapter five. Some problems are suggested in chapter six. It is pertinent to mention that several exercises and problems (Some in the form of proof to the theorems are given in all the chapters.) A reader who attempts to solve them will certainly gain a sound knowledge about these concepts. We have given 50 problems for the reader to solve in chapter 6. The main aim of this book is to introduce new concepts and explain them with examples there by encouraging young mathematics to pursue research in this direction. Several theorems based on the definition can be easily proved with simple modification. Innovative readers can take up that job
    corecore