Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency

In this paper, we propose a novel neural network structure, namely \emph{feedforward sequential memory networks (FSMN)}, to model long-term dependency in time series without using recurrent feedback. The proposed FSMN is a standard fully-connected feedforward neural network equipped with some learnable memory blocks in its hidden layers. The memory blocks use a tapped-delay line structure to encode the long context information into a fixed-size representation as short-term memory mechanism. We have evaluated the proposed FSMNs in several standard benchmark tasks, including speech recognition and language modelling. Experimental results have shown FSMNs significantly outperform the conventional recurrent neural networks (RNN), including LSTMs, in modeling sequential signals like speech or language. Moreover, FSMNs can be learned much more reliably and faster than RNNs or LSTMs due to the inherent non-recurrent model structure.Comment: 11 pages, 5 figure

Incremental Training of Deep Convolutional Neural Networks

We propose an incremental training method that partitions the original network into sub-networks, which are then gradually incorporated in the running network during the training process. To allow for a smooth dynamic growth of the network, we introduce a look-ahead initialization that outperforms the random initialization. We demonstrate that our incremental approach reaches the reference network baseline accuracy. Additionally, it allows to identify smaller partitions of the original state-of-the-art network, that deliver the same final accuracy, by using only a fraction of the global number of parameters. This allows for a potential speedup of the training time of several factors. We report training results on CIFAR-10 for ResNet and VGGNet

Plan2Vec: Unsupervised Representation Learning by Latent Plans

In this paper we introduce plan2vec, an unsupervised representation learning approach that is inspired by reinforcement learning. Plan2vec constructs a weighted graph on an image dataset using near-neighbor distances, and then extrapolates this local metric to a global embedding by distilling path-integral over planned path. When applied to control, plan2vec offers a way to learn goal-conditioned value estimates that are accurate over long horizons that is both compute and sample efficient. We demonstrate the effectiveness of plan2vec on one simulated and two challenging real-world image datasets. Experimental results show that plan2vec successfully amortizes the planning cost, enabling reactive planning that is linear in memory and computation complexity rather than exhaustive over the entire state space.Comment: code available at https://geyang.github.io/plan2ve

LookAhead: Augmenting Crowdsourced Website Reputation Systems With Predictive Modeling

Unsafe websites consist of malicious as well as inappropriate sites, such as those hosting questionable or offensive content. Website reputation systems are intended to help ordinary users steer away from these unsafe sites. However, the process of assigning safety ratings for websites typically involves humans. Consequently it is time consuming, costly and not scalable. This has resulted in two major problems: (i) a significant proportion of the web space remains unrated and (ii) there is an unacceptable time lag before new websites are rated. In this paper, we show that by leveraging structural and content-based properties of websites, it is possible to reliably and efficiently predict their safety ratings, thereby mitigating both problems. We demonstrate the effectiveness of our approach using four datasets of up to 90,000 websites. We use ratings from Web of Trust (WOT), a popular crowdsourced web reputation system, as ground truth. We propose a novel ensemble classification technique that makes opportunistic use of available structural and content properties of webpages to predict their eventual ratings in two dimensions used by WOT: trustworthiness and child safety. Ours is the first classification system to predict such subjective ratings and the same approach works equally well in identifying malicious websites. Across all datasets, our classification performs well with average F$_1$-score in the 74--90\% range.Comment: 12 page

A Full-Bandwidth Audio Codec With Low Complexity And Very Low Delay

We propose an audio codec that addresses the low-delay requirements of some applications such as network music performance. The codec is based on the modified discrete cosine transform (MDCT) with very short frames and uses gain-shape quantization to preserve the spectral envelope. The short frame sizes required for low delay typically hinder the performance of transform codecs. However, at 96 kbit/s and with only 4 ms algorithmic delay, the proposed codec out-performs the ULD codec operating at the same rate. The total complexity of the codec is small, at only 17 WMOPS for real-time operation at 48 kHz.Comment: 5 pages, Proceedings of EUSIPCO 200

On the Effect of Quantum Interaction Distance on Quantum Addition Circuits

We investigate the theoretical limits of the effect of the quantum interaction distance on the speed of exact quantum addition circuits. For this study, we exploit graph embedding for quantum circuit analysis. We study a logical mapping of qubits and gates of any $\Omega(\log n)$-depth quantum adder circuit for two $n$-qubit registers onto a practical architecture, which limits interaction distance to the nearest neighbors only and supports only one- and two-qubit logical gates. Unfortunately, on the chosen $k$-dimensional practical architecture, we prove that the depth lower bound of any exact quantum addition circuits is no longer $\Omega(\log {n})$, but $\Omega(\sqrt[k]{n})$. This result, the first application of graph embedding to quantum circuits and devices, provides a new tool for compiler development, emphasizes the impact of quantum computer architecture on performance, and acts as a cautionary note when evaluating the time performance of quantum algorithms.Comment: accepted for ACM Journal on Emerging Technologies in Computing System

Asynchronous Transmission over Single-User State-Dependent Channels

Several channels with asynchronous side information are introduced. We first consider single-user state-dependent channels with asynchronous side information at the transmitter. It is assumed that the state information sequence is a possibly delayed version of the state sequence, and that the encoder and the decoder are aware of the fact that the state information might be delayed. It is additionally assumed that an upper bound on the delay is known to both encoder and decoder, but other than that, they are ignorant of the actual delay. We consider both the causal and the noncausal cases and present achievable rates for these channels, and the corresponding coding schemes. We find the capacity of the asynchronous Gel'fand-Pinsker channel with feedback. Finally, we consider a memoryless state dependent channel with asynchronous side information at both the transmitter and receiver, and establish a single-letter expression for its capacity.Comment: The paper "On channels with asynchronous side information" was split into two separate papers: the enclosed paper which considers only point-to-point channels and an additional paper named "On the multiple access channel with asynchronous cognition" which discusses the multiuser setup

The Porosity of Additive Noise Sequences

Consider a binary additive noise channel with noiseless feedback. When the noise is a stationary and ergodic process $\mathbf{Z}$, the capacity is $1-\mathbb{H}(\mathbf{Z})$ ($\mathbb{H}(\cdot)$ denoting the entropy rate). It is shown analogously that when the noise is a deterministic sequence $z^\infty$, the capacity under finite-state encoding and decoding is $1-\bar{\rho}(z^\infty)$, where $\bar{\rho}(\cdot)$ is Lempel and Ziv's finite-state compressibility. This quantity is termed the \emph{porosity} $\underline{\sigma}(\cdot)$ of an individual noise sequence. A sequence of schemes are presented that universally achieve porosity for any noise sequence. These converse and achievability results may be interpreted both as a channel-coding counterpart to Ziv and Lempel's work in universal source coding, as well as an extension to the work by Lomnitz and Feder and Shayevitz and Feder on communication across modulo-additive channels. Additionally, a slightly more practical architecture is suggested that draws a connection with finite-state predictability, as introduced by Feder, Gutman, and Merhav.Comment: 22 pages, 9 figure

A real-time interpolator for parametric curves

Driven by the ever increasing need for the high-speed high-accuracy machining of freeform surfaces, the interpolators for parametric curves become highly desirable, as they can eliminate the feedrate and acceleration fluctuation due to the discontinuity in the first derivatives along the linear tool path. The interpolation for parametric curves is essentially an optimization problem, and it is extremely difficult to get the time-optimal solution. This paper presents a novel real-time interpolator for parametric curves (RTIPC), which provides a near time-optimal solution. It limits the machine dynamics (axial velocities, axial accelerations and jerk) and contour error through feedrate lookahead and acceleration lookahead operations, meanwhile, the feedrate is maintained as high as possible with minimum fluctuation. The lookahead length is dynamically adjusted to minimize the computation load. And the numerical integration error is considered during the lookahead calculation. Two typical parametric curves are selected for both numerical simulation and experimental validation, a cubic phase plate freeform surface is also machined. The numerical simulation is performed using the software (open access information is in the Acknowledgment section) that implements the proposed RTIPC, the results demonstrate the effectiveness of the RTIPC. The real-time performance of the RTIPC is tested on the in-house developed controller, which shows satisfactory efficiency. Finally, machining trials are carried out in comparison with the industrial standard linear interpolator and the state-of-the-art Position-Velocity-Time (PVT) interpolator, the results show the significant advantages of the RTIPC in coding, productivity and motion smoothness

Symmetric measures via moments

Algebraic tools in statistics have recently been receiving special attention and a number of interactions between algebraic geometry and computational statistics have been rapidly developing. This paper presents another such connection, namely, one between probabilistic models invariant under a finite group of (non-singular) linear transformations and polynomials invariant under the same group. Two specific aspects of the connection are discussed: generalization of the (uniqueness part of the multivariate) problem of moments and log-linear, or toric, modeling by expansion of invariant terms. A distribution of minuscule subimages extracted from a large database of natural images is analyzed to illustrate the above concepts.Comment: Published in at http://dx.doi.org/10.3150/07-BEJ6144 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm