16 research outputs found
LIPIcs, Volume 261, ICALP 2023, Complete Volume
LIPIcs, Volume 261, ICALP 2023, Complete Volum
Exposing Attention Glitches with Flip-Flop Language Modeling
Why do large language models sometimes output factual inaccuracies and
exhibit erroneous reasoning? The brittleness of these models, particularly when
executing long chains of reasoning, currently seems to be an inevitable price
to pay for their advanced capabilities of coherently synthesizing knowledge,
pragmatics, and abstract thought. Towards making sense of this fundamentally
unsolved problem, this work identifies and analyzes the phenomenon of attention
glitches, in which the Transformer architecture's inductive biases
intermittently fail to capture robust reasoning. To isolate the issue, we
introduce flip-flop language modeling (FFLM), a parametric family of synthetic
benchmarks designed to probe the extrapolative behavior of neural language
models. This simple generative task requires a model to copy binary symbols
over long-range dependencies, ignoring the tokens in between. We find that
Transformer FFLMs suffer from a long tail of sporadic reasoning errors, some of
which we can eliminate using various regularization techniques. Our preliminary
mechanistic analyses show why the remaining errors may be very difficult to
diagnose and resolve. We hypothesize that attention glitches account for (some
of) the closed-domain hallucinations in natural LLMs.Comment: v2: NeurIPS 2023 camera-ready + data releas
Attribute Exploration of Gene Regulatory Processes
This thesis aims at the logical analysis of discrete processes, in particular
of such generated by gene regulatory networks. States, transitions and
operators from temporal logics are expressed in the language of Formal Concept
Analysis. By the attribute exploration algorithm, an expert or a computer
program is enabled to validate a minimal and complete set of implications, e.g.
by comparison of predictions derived from literature with observed data. Here,
these rules represent temporal dependencies within gene regulatory networks
including coexpression of genes, reachability of states, invariants or possible
causal relationships. This new approach is embedded into the theory of
universal coalgebras, particularly automata, Kripke structures and Labelled
Transition Systems. A comparison with the temporal expressivity of Description
Logics is made. The main theoretical results concern the integration of
background knowledge into the successive exploration of the defined data
structures (formal contexts). Applying the method a Boolean network from
literature modelling sporulation of Bacillus subtilis is examined. Finally, we
developed an asynchronous Boolean network for extracellular matrix formation
and destruction in the context of rheumatoid arthritis.Comment: 111 pages, 9 figures, file size 2.1 MB, PhD thesis University of
Jena, Germany, Faculty of Mathematics and Computer Science, 2011. Online
available at http://www.db-thueringen.de/servlets/DocumentServlet?id=1960
Interval Groupoids
This book introduces several new classes of groupoid, like polynomial
groupoids, matrix groupoids, interval groupoids,polynomial interval groupoids,
matrix interval groupoids and their neutrosophic analogues.
Interval groupoid happens to be the first non-associative structure
constructed using intervals built using Zn or Z or Q or R or Z+ \cup {0} or Q+
\cup {0} and so on.
This book has five chapters. Chapter one is introductory in nature. In
chapter two new classes of groupoids and interval groupoids are defined and
described. The analogous neutrosophic study is carried out in chapter three.
The applications of this new structure is given in chapter four. The final
chapter suggests more than 200 problems. This book has given 77 new
definitions, 426 examples of these new notions and over 150 theorems.Comment: 240 page
SMARANDACHE SPECIAL DEFINITE ALGEBRAIC STRUCTURES
Introducing the notion of Smarandache special definite algebraic structures, also called equivalently
as Smarandache definite special algebraic structures. These new structures are defined as those strong algebraic structures which have in them a proper subset which is a weak algebraic structure. For instance, the existence of a semigroup in a group or a semifield in a field or a semiring in a ring. It is interesting to note that these concepts cannot be defined when the algebraic structure has finite cardinality i.e., when the algebraic structure has finite number of elements in it
Mathematical linguistics
but in fact this is still an early draft, version 0.56, August 1 2001. Please d
N-ALGEBRAIC STRUCTURES AND S-N-ALGEBRAIC STRUCTURES
In this book, for the first time we introduce the notions of Ngroups, N-semigroups, N-loops and N-groupoids. We also define a mixed N-algebraic structure. We expect the reader to be well versed in group theory and have at least basic knowledge about Smarandache groupoids, Smarandache loops, Smarandache semigroups and bialgebraic structures and Smarandache bialgebraic structures. The book is organized into six chapters. The first chapter gives the basic notions of S-semigroups, S-groupoids and S-loops thereby making the book self-contained. Chapter two introduces N-groups and their Smarandache analogues. In chapter three, Nloops and Smarandache N-loops are introduced and analyzed. Chapter four defines N-groupoids and S-N-groupoids. Since the N-semigroup structures are sandwiched between groups and groupoids, the study can be carried out without any difficulty. Mixed N-algebraic structures and S-mixed algebraic structures are given in chapter five. Some problems are suggested in chapter six. It is pertinent to mention that several exercises and problems (Some in the form of proof to the theorems are given in all the chapters.) A reader who attempts to solve them will certainly gain a sound knowledge about these concepts. We have given 50 problems for the reader to solve in chapter 6. The main aim of this book is to introduce new concepts and explain them with examples there by encouraging young mathematics to pursue research in this direction. Several theorems based on the definition can be easily proved with simple modification. Innovative readers can take up that job