Convergence Analysis of the Approximate Newton Method for Markov Decision Processes

Recently two approximate Newton methods were proposed for the optimisation of Markov Decision Processes. While these methods were shown to have desirable properties, such as a guarantee that the preconditioner is negative-semidefinite when the policy is $\log$-concave with respect to the policy parameters, and were demonstrated to have strong empirical performance in challenging domains, such as the game of Tetris, no convergence analysis was provided. The purpose of this paper is to provide such an analysis. We start by providing a detailed analysis of the Hessian of a Markov Decision Process, which is formed of a negative-semidefinite component, a positive-semidefinite component and a remainder term. The first part of our analysis details how the negative-semidefinite and positive-semidefinite components relate to each other, and how these two terms contribute to the Hessian. The next part of our analysis shows that under certain conditions, relating to the richness of the policy class, the remainder term in the Hessian vanishes in the vicinity of a local optimum. Finally, we bound the behaviour of this remainder term in terms of the mixing time of the Markov chain induced by the policy parameters, where this part of the analysis is applicable over the entire parameter space. Given this analysis of the Hessian we then provide our local convergence analysis of the approximate Newton framework.Comment: This work has been removed because a more recent piece (A Gauss-Newton method for Markov Decision Processes, T. Furmston & G. Lever) of work has subsumed i

Approximate Newton Methods for Policy Search in Markov Decision Processes

Approximate Newton methods are standard optimization tools which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, while alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton methods for policy optimization in Markov decision processes (MDPs). We first analyse the structure of the Hessian of the total expected reward, which is a standard objective function for MDPs. We show that, like the gradient, the Hessian exhibits useful structure in the context of MDPs and we use this analysis to motivate two Gauss-Newton methods for MDPs. Like the Gauss- Newton method for non-linear least squares, these methods drop certain terms in the Hessian. The approximate Hessians possess desirable properties, such as negative definiteness, and we demonstrate several important performance guarantees including guaranteed ascent directions, invariance to affine transformation of the parameter space and convergence guarantees. We finally provide a unifying perspective of key policy search algorithms, demonstrating that our second Gauss- Newton algorithm is closely related to both the EM-algorithm and natural gradient ascent applied to MDPs, but performs significantly better in practice on a range of challenging domains

A proposed taxonomy of contracts

We propose a classification of contracts based on their real-life usage. Recognizing that there are a number of different and overlapping ways of classifying contracts, we can identify at least seven ways of doing this: 1) by the subject matter, 2) by the way the contract is made, 3) by the function of contracts, 4) by the time-horizon, 5) by the ability to re-negotiate terms, 6) by the involvement of consumers, 7) by the existence of mutual trust. The proposed taxonomy draws our attention to a hitherto neglected area of contract scholarship by revealing an underlying order in which multiple elements in a contract are related to each other

Origin of Jeofail

In their interesting note on the origin of Jeofail ,\u27 Doctors Baker and Arnold suggest that the word is derived from jeu-faille (= game-fail) and say that A \u27game-fail\u27 in chess was presumably a stalemate; neither party could win, so the game failed or ended. 2 Since it has long been known that jeopardy has a chess origin3 (either from the old French jeu parti or the Latin jocus partitus = game in the balance and hence an uncertain chance) this explanation has an obvious attraction. Indeed in view of the alphabetical work habits of-lexicographers it is surprising that the suggestion has not been made before. (The two words are consecutive in the Oxford Dictionary of English Etymology). 4 Nevertheless the argument based on the analogy with chess is not free from difficulties. The definition of stalemate is inelegant for the modern version of the game but much more important it is very far from clear that in 1378 the result of a stalemate was that the game was drawn

Contract and Tort after Denning

Contract and Tort after Dennin

How “Fast” is Fast Furniture?

This paper explores the emerging concept of fast furniture, a rapidly growing sector characterized by quick production, low costs, and short product life cycles. Despite its substantial environmental impact, fast furniture remains underexplored in academic literature. Drawing parallels to fast fashion in its focus on trends, disposability, and mass production, this study examines the intersections of consumer behaviour, industry practices, and sustainability challenges within the context of fast furniture. Through a mixed-methods approach, combining a quantitative/qualitative survey of UK consumers and interviews with industry professionals, the study reveals significant insights into consumer perceptions, motivations, and the role of fashion-driven consumption in shaping the furniture market. Key findings indicate that while consumers increasingly engage with trend-driven furniture purchases, many are unaware of the term "fast furniture." Moreover, despite their significant market share, brands like IKEA are not strongly associated with the "fast" model by consumers, who instead view their products as affordable and functional, yet temporary. Industry professionals, meanwhile, emphasize that the "fastness" of furniture is determined largely by consumer choices, not necessarily the manufacturing process. This research stems from a larger doctoral study that contributes to the growing body of knowledge on sustainable consumption, advocating for a broader understanding of fast furniture as a consumer-driven phenomenon rather than an industry-defined product category. Future research is suggested to further explore the global dynamics of fast furniture consumption, the role of consumer education, and sustainability initiatives within the industry

A Significance Test for Inferring Affiliation Networks from Spatio-Temporal Data.

Scientists have long been interested in studying social structures within groups of gregarious animals. However, obtaining evidence about interactions between members of a group is difficult. Recent technologies, such as Global Positioning System technology, have made it possible to obtain a vast wealth of animal movement data, but inferring the underlying (latent) social structure of the group from such data remains an important open problem. While intuitively appealing measures of social interaction exist in the literature, they typically lack formal statistical grounding. In this article, we provide a statistical approach to the problem of inferring the social structure of a group from the movement patterns of its members. By constructing an appropriate null model, we are able to construct a significance test to detect meaningful affiliations between members of the group. We demonstrate our method on large-scale real-world data sets of positional data of flocks of Merino sheep, Ovis aries