How degenerate is the parametrization of neural networks with the ReLU activation function?

Neural network training is usually accomplished by solving a non-convex optimization problem using stochastic gradient descent. Although one optimizes over the networks parameters, the main loss function generally only depends on the realization of the neural network, i.e. the function it computes. Studying the optimization problem over the space of realizations opens up new ways to understand neural network training. In particular, usual loss functions like mean squared error and categorical cross entropy are convex on spaces of neural network realizations, which themselves are non-convex. Approximation capabilities of neural networks can be used to deal with the latter non-convexity, which allows us to establish that for sufficiently large networks local minima of a regularized optimization problem on the realization space are almost optimal. Note, however, that each realization has many different, possibly degenerate, parametrizations. In particular, a local minimum in the parametrization space needs not correspond to a local minimum in the realization space. To establish such a connection, inverse stability of the realization map is required, meaning that proximity of realizations must imply proximity of corresponding parametrizations. We present pathologies which prevent inverse stability in general, and, for shallow networks, proceed to establish a restricted space of parametrizations on which we have inverse stability w.r.t. to a Sobolev norm. Furthermore, we show that by optimizing over such restricted sets, it is still possible to learn any function which can be learned by optimization over unrestricted sets.Comment: Accepted at NeurIPS 201

PILCO: A Model-Based and Data-Efficient Approach to Policy Search

In this paper, we introduce PILCO, a practical, data-efficient model-based policy search method. PILCO reduces model bias, one of the key problems of model-based reinforcement learning, in a principled way. By learning a probabilistic dynamics model and explicitly incorporating model uncertainty into long-term planning, PILCO can cope with very little data and facilitates learning from scratch in only a few trials. Policy evaluation is performed in closed form using state-of-the-art approximate inference. Furthermore, policy gradients are computed analytically for policy improvement. We report unprecedented learning efficiency on challenging and high-dimensional control tasks. Copyright 2011 by the author(s)/owner(s)

Designing Volumetric Truss Structures

We present the first algorithm for designing volumetric Michell Trusses. Our method uses a parametrization approach to generate trusses made of structural elements aligned with the primary direction of an object's stress field. Such trusses exhibit high strength-to-weight ratios. We demonstrate the structural robustness of our designs via a posteriori physical simulation. We believe our algorithm serves as an important complement to existing structural optimization tools and as a novel standalone design tool itself

Spatial Coordination Strategies in Future Ultra-Dense Wireless Networks

Ultra network densification is considered a major trend in the evolution of cellular networks, due to its ability to bring the network closer to the user side and reuse resources to the maximum extent. In this paper we explore spatial resources coordination as a key empowering technology for next generation (5G) ultra-dense networks. We propose an optimization framework for flexibly associating system users with a densely deployed network of access nodes, opting for the exploitation of densification and the control of overhead signaling. Combined with spatial precoding processing strategies, we design network resources management strategies reflecting various features, namely local vs global channel state information knowledge exploitation, centralized vs distributed implementation, and non-cooperative vs joint multi-node data processing. We apply these strategies to future UDN setups, and explore the impact of critical network parameters, that is, the densification levels of users and access nodes as well as the power budget constraints, to users performance. We demonstrate that spatial resources coordination is a key factor for capitalizing on the gains of ultra dense network deployments.Comment: An extended version of a paper submitted to ISWCS'14, Special Session on Empowering Technologies of 5G Wireless Communication