969 research outputs found
Learning compositional functions via multiplicative weight updates
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam—a multiplicative version of the Adam optimiser—and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology
Learning compositional functions via multiplicative weight updates
Compositionality is a basic structural feature of both biological and
artificial neural networks. Learning compositional functions via gradient
descent incurs well known problems like vanishing and exploding gradients,
making careful learning rate tuning essential for real-world applications. This
paper proves that multiplicative weight updates satisfy a descent lemma
tailored to compositional functions. Based on this lemma, we derive Madam -- a
multiplicative version of the Adam optimiser -- and show that it can train
state of the art neural network architectures without learning rate tuning. We
further show that Madam is easily adapted to train natively compressed neural
networks by representing their weights in a logarithmic number system. We
conclude by drawing connections between multiplicative weight updates and
recent findings about synapses in biology
Holographic Embeddings of Knowledge Graphs
Learning embeddings of entities and relations is an efficient and versatile
method to perform machine learning on relational data such as knowledge graphs.
In this work, we propose holographic embeddings (HolE) to learn compositional
vector space representations of entire knowledge graphs. The proposed method is
related to holographic models of associative memory in that it employs circular
correlation to create compositional representations. By using correlation as
the compositional operator HolE can capture rich interactions but
simultaneously remains efficient to compute, easy to train, and scalable to
very large datasets. In extensive experiments we show that holographic
embeddings are able to outperform state-of-the-art methods for link prediction
in knowledge graphs and relational learning benchmark datasets.Comment: To appear in AAAI-1
Differential Privacy for Relational Algebra: Improving the Sensitivity Bounds via Constraint Systems
Differential privacy is a modern approach in privacy-preserving data analysis
to control the amount of information that can be inferred about an individual
by querying a database. The most common techniques are based on the
introduction of probabilistic noise, often defined as a Laplacian parametric on
the sensitivity of the query. In order to maximize the utility of the query, it
is crucial to estimate the sensitivity as precisely as possible.
In this paper we consider relational algebra, the classical language for
queries in relational databases, and we propose a method for computing a bound
on the sensitivity of queries in an intuitive and compositional way. We use
constraint-based techniques to accumulate the information on the possible
values for attributes provided by the various components of the query, thus
making it possible to compute tight bounds on the sensitivity.Comment: In Proceedings QAPL 2012, arXiv:1207.055
- …