Deep Thinking Systems: Logical Extrapolation with Recurrent Neural Networks

Deep neural networks are powerful machines for visual pattern recognition, but reasoning tasks that are easy for humans are still be difficult for neural models. Humans possess the ability to extrapolate reasoning strategies learned on simple problems to solve harder examples, often by thinking for longer. We study neural networks that have exactly this capability. By employing recurrence, we build neural networks that can expend more computation when needed. Using several datasets designed specifically for studying generalization from easy problems to harder test samples, we show that our recurrent networks can extrapolate from easy training data to much harder examples at test time, and they do so with many more iterations of a recurrent block of layers than are used during training

Linear systems with sign-observations

Abstract. This paper deals with systems that are obtained from linear time-invariant continuousor discrete-time devices followed by a function that just provides the sign of each output. Such systems appear naturally in the study of quantized observations as well as in signal processing and neural network theory. Results are given on observability, minimal realizations, and other systemtheoretic concepts. Certain major differences exist with the linear case, and other results generalize in a surprisingly straightforward manner

Using Fourier-Neural Recurrent Networks to Fit Sequential Input/Output Data

This paper suggests the use of Fourier-type activation functions in fully recurrent neural networks. The main theoretical advantage is that, in principle, the problem of recovering internal coefficients from input/output data is solvable in closed form. 1 Introduction Neural networks provide a useful approach to parallel computation. The subclass of recurrent architectures is characterized by the inclusion of feedback loops in the information flow among processing units. With feedback, one may exploit context-sensitivity and memory, characteristics essential in sequence processing as well as in the modeling and control of processes involving dynamical elements. Recent theoretical results about neural networks have established their universality as models for systems approximation as well as analog computing devices (see e.g. [14, 12]). The use of recurrent networks has been proposed in areas as varied as the design of control laws for robotic manipulators, in speech recognition, speak..

Observability of Linear Systems with Saturated Outputs

In this paper, we present necessary and sufficient conditions for observability of the class of output-saturated systems. These are linear systems whose output passes through a saturation function before it can be measured. 1 Introduction The question of observability for time-invariant linear systems is certainly a well understood problem. But what happens when the output is not fully available? That is, instead of measuring Cx, we can only measure oe(Cx), where oe is some nonlinear function. If the nonlinearity oe is not injective, it is no longer obvious from the observability matrix [C 0 A 0 C 0 \Delta \Delta \Delta (A n\Gamma1 ) 0 C 0 ] 0 (prime indicates transpose), whether or not the state can be "observed" from the output. In [3], we answered this question in the case in which oe provided the sign of the output of the linear system. That model was motivated by quantization and pattern recognition. In this paper, we will look at continuous-time sytems in which th..