12,769 research outputs found
Policy Space Diversity for Non-Transitive Games
Policy-Space Response Oracles (PSRO) is an influential algorithm framework
for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games.
Many previous studies have been trying to promote policy diversity in PSRO. A
major weakness in existing diversity metrics is that a more diverse (according
to their diversity metrics) population does not necessarily mean (as we proved
in the paper) a better approximation to a NE. To alleviate this problem, we
propose a new diversity metric, the improvement of which guarantees a better
approximation to a NE. Meanwhile, we develop a practical and well-justified
method to optimize our diversity metric using only state-action samples. By
incorporating our diversity regularization into the best response solving in
PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We
present the convergence property of PSD-PSRO. Empirically, extensive
experiments on various games demonstrate that PSD-PSRO is more effective in
producing significantly less exploitable policies than state-of-the-art PSRO
variants
An Investigation into the Transition Experiences of A-Level Students and Staff in a Further Education College
Post-16 transition is an inevitability in the life of every student. Yet post-16 educational transition in England has to date been under-researched. The age of participation in education in the United Kingdom was raised to 18 years in 2015, which along with concurrent reductions in funding have had ramifications for Post-16 transition provision and experiences. Transition experience impacts on students’ future engagement with education and long-term economic wellbeing. The present research investigated the transition experience of four first year A-Level students in a Further Education college, and five members of staff. Data were collected from all participants, using Interpretative Phenomenological Analysis (IPA), via semistructured interviews in the first term. Data were further collected from the student participants in the second term, using semi-structured interviews and a form of photo elicitation. From the analysis, three super-ordinate themes were highlighted from the student participants: Relationships and Interactions, Influences which Impact on Transition and Transition in Learning. The three super-ordinate themes from the staff participants were: Teacher practices, Management and Transition Provision. The implications from the findings indicate that having a longer-term and well considered plan towards Higher Education and career pathways can enhance the transition experience of all students, particularly those who find the process confusing or difficult. The overarching conclusion is that transition can be a challenging process which requires greater attention from strategic leadership and the inclusion of both staff and students in the decision-making process
Reinforcement learning in large state action spaces
Reinforcement learning (RL) is a promising framework for training intelligent agents which learn to optimize long term utility by directly interacting with the environment. Creating RL methods which scale to large state-action spaces is a critical problem towards ensuring real world deployment of RL systems. However, several challenges limit the applicability of RL to large scale settings. These include difficulties with exploration, low sample efficiency, computational intractability, task constraints like decentralization and lack of guarantees about important properties like performance, generalization and robustness in potentially unseen scenarios.
This thesis is motivated towards bridging the aforementioned gap. We propose several principled algorithms and frameworks for studying and addressing the above challenges RL. The proposed methods cover a wide range of RL settings (single and multi-agent systems (MAS) with all the variations in the latter, prediction and control, model-based and model-free methods, value-based and policy-based methods). In this work we propose the first results on several different problems: e.g. tensorization of the Bellman equation which allows exponential sample efficiency gains (Chapter 4), provable suboptimality arising from structural constraints in MAS(Chapter 3), combinatorial generalization results in cooperative MAS(Chapter 5), generalization results on observation shifts(Chapter 7), learning deterministic policies in a probabilistic RL framework(Chapter 6). Our algorithms exhibit provably enhanced performance and sample efficiency along with better scalability. Additionally, we also shed light on generalization aspects of the agents under different frameworks. These properties have been been driven by the use of several advanced tools (e.g. statistical machine learning, state abstraction, variational inference, tensor theory).
In summary, the contributions in this thesis significantly advance progress towards making RL agents ready for large scale, real world applications
Unstable Periodic Orbits: a language to interpret the complexity of chaotic systems
Unstable periodic orbits (UPOs), exact periodic solutions of the evolution equation, offer a very
powerful framework for studying chaotic dynamical systems, as they allow one to dissect their
dynamical structure. UPOs can be considered the skeleton of chaotic dynamics, its essential
building blocks. In fact, it is possible to prove that in a chaotic system, UPOs are dense in
the attractor, meaning that it is always possible to find a UPO arbitrarily near any chaotic
trajectory. We can thus think of the chaotic trajectory as being approximated by different
UPOs as it evolves in time, jumping from one UPO to another as a result of their instability.
In this thesis we provide a contribution towards the use of UPOs as a tool to understand and
distill the dynamical structure of chaotic dynamical systems. We will focus on two models,
characterised by different properties, the Lorenz-63 and Lorenz-96 model.
The process of approximation of a chaotic trajectory in terms of UPOs will play a central role
in our investigation. In fact, we will use this tool to explore the properties of the attractor of
the system under the lens of its UPOs.
In the first part of the thesis we consider the Lorenz-63 model with the classic parameters’ value.
We investigate how a chaotic trajectory can be approximated using a complete set of UPOs
up to symbolic dynamics’ period 14. At each instant in time, we rank the UPOs according to
their proximity to the position of the orbit in the phase space. We study this process from
two different perspectives. First, we find that longer period UPOs overwhelmingly provide the
best local approximation to the trajectory. Second, we construct a finite-state Markov chain
by studying the scattering of the trajectory between the neighbourhood of the various UPOs.
Each UPO and its neighbourhood are taken as a possible state of the system. Through the
analysis of the subdominant eigenvectors of the corresponding stochastic matrix we provide a
different interpretation of the mixing processes occurring in the system by taking advantage of
the concept of quasi-invariant sets.
In the second part of the thesis we provide an extensive numerical investigation of the variability
of the dynamical properties across the attractor of the much studied Lorenz ’96 dynamical
system. By combining the Lyapunov analysis of the tangent space with the study of the
shadowing of the chaotic trajectory performed by a very large set of unstable periodic orbits,
we show that the observed variability in the number of unstable dimensions, which shows a
serious breakdown of hyperbolicity, is associated with the presence of a substantial number of
finite-time Lyapunov exponents that fluctuate about zero also when very long averaging times
are considered
Aspect-Driven Structuring of Historical Dutch Newspaper Archives
Digital libraries oftentimes provide access to historical newspaper archives
via keyword-based search. Historical figures and their roles are particularly
interesting cognitive access points in historical research. Structuring and
clustering news articles would allow more sophisticated access for users to
explore such information. However, real-world limitations such as the lack of
training data, licensing restrictions and non-English text with OCR errors make
the composition of such a system difficult and cost-intensive in practice. In
this work we tackle these issues with the showcase of the National Library of
the Netherlands by introducing a role-based interface that structures news
articles on historical persons. In-depth, component-wise evaluations and
interviews with domain experts highlighted our prototype's effectiveness and
appropriateness for a real-world digital library collection.Comment: TPDL2023, Full Paper, 16 page
Chaos persists in large-scale multi-agent learning despite adaptive learning rates
Multi-agent learning is intrinsically harder, more unstable and unpredictable
than single agent optimization. For this reason, numerous specialized
heuristics and techniques have been designed towards the goal of achieving
convergence to equilibria in self-play. One such celebrated approach is the use
of dynamically adaptive learning rates. Although such techniques are known to
allow for improved convergence guarantees in small games, it has been much
harder to analyze them in more relevant settings with large populations of
agents. These settings are particularly hard as recent work has established
that learning with fixed rates will become chaotic given large enough
populations.In this work, we show that chaos persists in large population
congestion games despite using adaptive learning rates even for the ubiquitous
Multiplicative Weight Updates algorithm, even in the presence of only two
strategies. At a technical level, due to the non-autonomous nature of the
system, our approach goes beyond conventional period-three techniques Li-Yorke
by studying fundamental properties of the dynamics including invariant sets,
volume expansion and turbulent sets. We complement our theoretical insights
with experiments showcasing that slight variations to system parameters lead to
a wide variety of unpredictable behaviors.Comment: 30 pages, 6 figure
The home and mental health: an exploration of perceptions, connections and attachments to social housing for residents and housing professionals in Manchester, United Kingdom
This doctoral study is an exploration of the home environment, mental health, and the role of housing associations in supporting resident welfare. The home is significant, having the ability to provide protection and privacy, a secure base to develop relationships, build identity and feel safe from the outside world. However, many individuals are unable to experience these protective factors within their home setting for multiple reasons.
This thesis is framed within the context of the Devolution Agenda in Manchester, where there has been an emphasis on the integration of health and social care services (such as social housing providers) in efforts to tackle mental ill health and persistent health inequalities. This thesis undertook a multi-method approach that consists of a qualitative analysis of two focus groups with a housing provider and seven semi-structured interviews with residents within Gorton, an area located North of Manchester, UK. The research was conducted in two phases to capture the perceptions and experiences of socially housed residents, alongside investigating the understandings and professional realities of housing providers.
My research is informed by a critical realist approach, where I apply concepts such as ontological security, place, social capital, and stigma, to make sense of interactions between the home and mental health. The findings in this thesis highlight how the home environment can interact with mental health outcomes through various psychological, social, and environmental dimensions. Being able to establish and maintain a sense of home was important for residents, and this provided a source of security, self-esteem, and autonomy. Residents demonstrated complex articulations of place, where the importance of social networks, local value systems and a sense of belonging all reflected the deep connections and meanings attached to the home. Physical deterioration, the closure of local amenities and fragmented relationships with the service provider undermined the mental wellbeing of residents. The findings from this thesis identified a disconnect between how residents experienced their home and community and how the housing provider interprets residents support needs and the delegation of resources. In this thesis I highlight how integrated health approaches require further consideration to ensure services are needs-sensitive and reflective of residents’ priorities. Through bringing together the perspectives of residents and
housing professionals, this research makes a unique contribution to the integration of health approaches, the less tangible aspects of the home setting, and mental wellbeing
Knowledge Graph Building Blocks: An easy-to-use Framework for developing FAIREr Knowledge Graphs
Knowledge graphs and ontologies provide promising technical solutions for
implementing the FAIR Principles for Findable, Accessible, Interoperable, and
Reusable data and metadata. However, they also come with their own challenges.
Nine such challenges are discussed and associated with the criterion of
cognitive interoperability and specific FAIREr principles (FAIR + Explorability
raised) that they fail to meet. We introduce an easy-to-use, open source
knowledge graph framework that is based on knowledge graph building blocks
(KGBBs). KGBBs are small information modules for knowledge-processing, each
based on a specific type of semantic unit. By interrelating several KGBBs, one
can specify a KGBB-driven FAIREr knowledge graph. Besides implementing semantic
units, the KGBB Framework clearly distinguishes and decouples an internal
in-memory data model from data storage, data display, and data access/export
models. We argue that this decoupling is essential for solving many problems of
knowledge management systems. We discuss the architecture of the KGBB Framework
as we envision it, comprising (i) an openly accessible KGBB-Repository for
different types of KGBBs, (ii) a KGBB-Engine for managing and operating FAIREr
knowledge graphs (including automatic provenance tracking, editing changelog,
and versioning of semantic units); (iii) a repository for KGBB-Functions; (iv)
a low-code KGBB-Editor with which domain experts can create new KGBBs and
specify their own FAIREr knowledge graph without having to think about semantic
modelling. We conclude with discussing the nine challenges and how the KGBB
Framework provides solutions for the issues they raise. While most of what we
discuss here is entirely conceptual, we can point to two prototypes that
demonstrate the principle feasibility of using semantic units and KGBBs to
manage and structure knowledge graphs
Characterising Modal Formulas with Examples
We study the existence of finite characterisations for modal formulas. A
finite characterisation of a modal formula is a finite collection of
positive and negative examples that distinguishes from every other,
non-equivalent modal formula, where an example is a finite pointed Kripke
structure. This definition can be restricted to specific frame classes and to
fragments of the modal language: a modal fragment admits finite
characterisations with respect to a frame class if every formula
has a finite characterisation with respect to consting of
examples that are based on frames in . Finite characterisations are useful
for illustration, interactive specification, and debugging of formal
specifications, and their existence is a precondition for exact learnability
with membership queries. We show that the full modal language admits finite
characterisations with respect to a frame class only when the modal logic
of is locally tabular. We then study which modal fragments, freely
generated by some set of connectives, admit finite characterisations. Our main
result is that the positive modal language without the truth-constants
and admits finite characterisations w.r.t. the class of all frames. This
result is essentially optimal: finite characterizability fails when the
language is extended with the truth constant or with all but very
limited forms of negation.Comment: Expanded version of material from Raoul Koudijs's MSc thesis (2022
- …