25,868 research outputs found
On the Global Convergence Rates of Softmax Policy Gradient Methods
We make three contributions toward better understanding policy gradient
methods in the tabular setting. First, we show that with the true gradient,
policy gradient with a softmax parametrization converges at a rate,
with constants depending on the problem and initialization. This result
significantly expands the recent asymptotic convergence results. The analysis
relies on two findings: that the softmax policy gradient satisfies a
\L{}ojasiewicz inequality, and the minimum probability of an optimal action
during optimization can be bounded in terms of its initial value. Second, we
analyze entropy regularized policy gradient and show that it enjoys a
significantly faster linear convergence rate toward softmax optimal
policy. This result resolves an open question in the recent literature.
Finally, combining the above two results and additional new lower
bound results, we explain how entropy regularization improves policy
optimization, even with the true gradient, from the perspective of convergence
rate. The separation of rates is further explained using the notion of
non-uniform \L{}ojasiewicz degree. These results provide a theoretical
understanding of the impact of entropy and corroborate existing empirical
studies.Comment: 64 pages, 5 figures. Published in ICML 202
The NOW dilemma
Energy efficiency has found its place at the very core of the discussion in Architecture and Urban Planning. Research & Development, Political Agendas and Education Curriculums are increasingly driven by the need to reach a fair balance between the way we inhabit the world and the energy we require for it. After many decades neglecting this discussion a growing awareness about the carrying capacity of our environment is being brought to actual policies on the built environment. The dominant tendency today privileges economic growth, thus being the maximization of performed labor per energy unit its ultimate goal. Renewal energy sources and energy efficiency are means for, on the one hand, an alternative to finite fossil fuel sources and, on the other hand, the optimization in the use of energy. Very little attention has been paid, however, to a more profound paradigm shift in economy. Some authors, however, have also claimed replacing the myth of economic growth by a more steady-state development as a solution for the current sustainability conundrum. The question is whether withholding the use of energy might be an alternative to its hi-tech optimization. Some of the contemporary authors who have discussed the issue in recent energy crisis are recounted here for a wider and holistic understanding of the problem.Peer ReviewedPostprint (published version
- …