64,729 research outputs found
Smooth Monotonic Networks
Monotonicity constraints are powerful regularizers in statistical modelling.
They can support fairness in computer supported decision making and increase
plausibility in data-driven scientific models. The seminal min-max (MM) neural
network architecture ensures monotonicity, but often gets stuck in undesired
local optima during training because of vanishing gradients. We propose a
simple modification of the MM network using strictly-increasing smooth
non-linearities that alleviates this problem. The resulting smooth min-max
(SMM) network module inherits the asymptotic approximation properties from the
MM architecture. It can be used within larger deep learning systems trained
end-to-end. The SMM module is considerably simpler and less computationally
demanding than state-of-the-art neural networks for monotonic modelling. Still,
in our experiments, it compared favorably to alternative neural and non-neural
approaches in terms of generalization performance
Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons
A lossy data compression scheme for uniformly biased Boolean messages is
investigated via statistical mechanics techniques. We utilize tree-like
committee machine (committee tree) and tree-like parity machine (parity tree)
whose transfer functions are non-monotonic. The scheme performance at the
infinite code length limit is analyzed using the replica method. Both committee
and parity treelike networks are shown to saturate the Shannon bound. The AT
stability of the Replica Symmetric solution is analyzed, and the tuning of the
non-monotonic transfer function is also discussed.Comment: 29 pages, 7 figure
Certified Monotonic Neural Networks
Learning monotonic models with respect to a subset of the inputs is a
desirable feature to effectively address the fairness, interpretability, and
generalization issues in practice. Existing methods for learning monotonic
neural networks either require specifically designed model structures to ensure
monotonicity, which can be too restrictive/complicated, or enforce monotonicity
by adjusting the learning process, which cannot provably guarantee the learned
model is monotonic on selected features. In this work, we propose to certify
the monotonicity of the general piece-wise linear neural networks by solving a
mixed integer linear programming problem.This provides a new general approach
for learning monotonic neural networks with arbitrary model structures. Our
method allows us to train neural networks with heuristic monotonicity
regularizations, and we can gradually increase the regularization magnitude
until the learned network is certified monotonic. Compared to prior works, our
approach does not require human-designed constraints on the weight space and
also yields more accurate approximation. Empirical studies on various datasets
demonstrate the efficiency of our approach over the state-of-the-art methods,
such as Deep Lattice Networks
Constrained Monotonic Neural Networks
Wider adoption of neural networks in many critical domains such as finance
and healthcare is being hindered by the need to explain their predictions and
to impose additional constraints on them. Monotonicity constraint is one of the
most requested properties in real-world scenarios and is the focus of this
paper. One of the oldest ways to construct a monotonic fully connected neural
network is to constrain signs on its weights. Unfortunately, this construction
does not work with popular non-saturated activation functions as it can only
approximate convex functions. We show this shortcoming can be fixed by
constructing two additional activation functions from a typical unsaturated
monotonic activation function and employing each of them on the part of
neurons. Our experiments show this approach of building monotonic neural
networks has better accuracy when compared to other state-of-the-art methods,
while being the simplest one in the sense of having the least number of
parameters, and not requiring any modifications to the learning procedure or
post-learning steps. Finally, we prove it can approximate any continuous
monotone function on a compact subset of
- …