13,670 research outputs found
Active Coverage for PAC Reinforcement Learning
Collecting and leveraging data with good coverage properties plays a crucial
role in different aspects of reinforcement learning (RL), including reward-free
exploration and offline learning. However, the notion of "good coverage" really
depends on the application at hand, as data suitable for one context may not be
so for another. In this paper, we formalize the problem of active coverage in
episodic Markov decision processes (MDPs), where the goal is to interact with
the environment so as to fulfill given sampling requirements. This framework is
sufficiently flexible to specify any desired coverage property, making it
applicable to any problem that involves online exploration. Our main
contribution is an instance-dependent lower bound on the sample complexity of
active coverage and a simple game-theoretic algorithm, CovGame, that nearly
matches it. We then show that CovGame can be used as a building block to solve
different PAC RL tasks. In particular, we obtain a simple algorithm for PAC
reward-free exploration with an instance-dependent sample complexity that, in
certain MDPs which are "easy to explore", is lower than the minimax one. By
further coupling this exploration algorithm with a new technique to do implicit
eliminations in policy space, we obtain a computationally-efficient algorithm
for best-policy identification whose instance-dependent sample complexity
scales with gaps between policy values.Comment: Accepted at COLT 202
An investigation of entorhinal spatial representations in self-localisation behaviours
Spatial-modulated cells of the medial entorhinal cortex (MEC) and neighbouring cortices are thought to provide the neural substrate for self-localisation behaviours. These cells include grid cells of the MEC which are thought to compute path integration operations to update self-location estimates. In order to read this grid code, downstream cells are thought to reconstruct a positional estimate as a simple rate-coded representation of space.
Here, I show the coding scheme of grid cell and putative readout cells recorded from mice performing a virtual reality (VR) linear location task which engaged mice in both beaconing and path integration behaviours. I found grid cells can encode two unique coding schemes on the linear track, namely a position code which reflects periodic grid fields anchored to salient features of the track and a distance code which reflects periodic grid fields without this anchoring. Grid cells were found to switch between these coding schemes within sessions. When grid cells were encoding position, mice performed better at trials that required path integration but not on trials that required beaconing. This result provides the first mechanistic evidence linking grid cell activity to path integration-dependent behaviour.
Putative readout cells were found in the form of ramp cells which fire proportionally as a function of location in defined regions of the linear track. This ramping activity was found to be primarily explained by track position rather than other kinematic variables like speed and acceleration. These representations were found to be maintained across both trial types and outcomes indicating they likely result from recall of the track structure.
Together, these results support the functional importance of grid and ramp cells for self-localisation behaviours. Future investigations will look into the coherence between these two neural populations, which may together form a complete neural system for coding and decoding self-location in the brain
Recommended from our members
Landfill site trees: Potential source or sink of greenhouse gases?
Tree stems can transport greenhouse gases (GHGs) produced belowground to the atmosphere. Previous studies in natural wetland and upland ecosystems have quantified tree stem fluxes of methane (CH4), carbon dioxide (CO2) and nitrous oxide (N2O). However, tree stem GHG fluxes have not previously been measured in the context of managed environments. The work presented in this thesis aimed to quantify GHG fluxes from tree stems on closed landfill sites.
To investigate the potential for trees growing on closed landfill sites to act as conduits for GHGs produced belowground to the atmosphere, GHG fluxes were measured from tree stem and soil surfaces. In situ measurements from a closed landfill site in the UK were examined for spatial and temporal patterns and evaluated against data from a comparable non-landfill area. Measurements were also conducted from landfill sites in the UK with varying management practices and different tree species present. The resulting flux values were scaled up to estimate the magnitude of tree stem GHG fluxes from closed landfills at a national level.
The findings presented here show evidence of tree mediated GHG transport on closed landfill sites and temporal variations in fluxes from tree stems were also observed, with generally higher fluxes in the summer months. Stem CH4 fluxes varied between trees growing on landfill sites with different management practices. Additionally, stem N2O fluxes displayed spatial patterns, with decreasing emissions at increased height from the forest floor, indicating an underground source. Evidence suggested that GHG fluxes from closed landfills are influenced by factors including the quantity of GHG produced in the waste (linked to the age of the site), the susceptibility of the area to waterlogging and landfill management techniques put in place upon closure (for example, clay caps, cover soils and gas extraction). Upscaled CH4 and N2O flux values from tree stems on closed landfill sites corresponded to less than 1% of the total CH4 and N2O emissions reported from UK landfills in 2020.
Overall, results indicated that measuring soil fluxes alone from forested landfill sites would result in an underestimation of the total surface fluxes. However, the emission rates from tree stems on closed landfills observed in this thesis do not exceed those in natural ecosystems. Therefore, with careful planning and management, the recommendation is that trees can be planted on closed landfill sites in the UK without emitting atypical levels of GHGs. However, including gas fluxes from tree stems on closed landfills would increase the accuracy of GHG budgets at national and global levels
Optimal neighbourhood selection in structural equation models
We study the optimal sample complexity of neighbourhood selection in linear
structural equation models, and compare this to best subset selection (BSS) for
linear models under general design. We show by example that -- even when the
structure is \emph{unknown} -- the existence of underlying structure can reduce
the sample complexity of neighbourhood selection. This result is complicated by
the possibility of path cancellation, which we study in detail, and show that
improvements are still possible in the presence of path cancellation. Finally,
we support these theoretical observations with experiments. The proof
introduces a modified BSS estimator, called klBSS, and compares its performance
to BSS. The analysis of klBSS may also be of independent interest since it
applies to arbitrary structured models, not necessarily those induced by a
structural equation model. Our results have implications for structure learning
in graphical models, which often relies on neighbourhood selection as a
subroutine
FAMO: Fast Adaptive Multitask Optimization
One of the grand enduring goals of AI is to create generalist agents that can
learn multiple different tasks from diverse data via multitask learning (MTL).
However, gradient descent (GD) on the average loss across all tasks may yield
poor multitask performance due to severe under-optimization of certain tasks.
Previous approaches that manipulate task gradients for a more balanced loss
decrease require storing and computing all task gradients (O(K) space and time
where K is the number of tasks), limiting their use in large-scale scenarios.
In this work, we introduce Fast Adaptive Multitask Optimization (FAMO), a
dynamic weighting method that decreases task losses in a balanced way using
O(1) space and time. We conduct an extensive set of experiments covering
multi-task supervised and reinforcement learning problems. Our results indicate
that FAMO achieves comparable or superior performance to state-of-the-art
gradient manipulation techniques while offering significant improvements in
space and computational efficiency. Code is available at
https://github.com/Cranial-XIX/FAMO
Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning
Advances in deep learning have greatly improved structure prediction of
molecules. However, many macroscopic observations that are important for
real-world applications are not functions of a single molecular structure, but
rather determined from the equilibrium distribution of structures. Traditional
methods for obtaining these distributions, such as molecular dynamics
simulation, are computationally expensive and often intractable. In this paper,
we introduce a novel deep learning framework, called Distributional Graphormer
(DiG), in an attempt to predict the equilibrium distribution of molecular
systems. Inspired by the annealing process in thermodynamics, DiG employs deep
neural networks to transform a simple distribution towards the equilibrium
distribution, conditioned on a descriptor of a molecular system, such as a
chemical graph or a protein sequence. This framework enables efficient
generation of diverse conformations and provides estimations of state
densities. We demonstrate the performance of DiG on several molecular tasks,
including protein conformation sampling, ligand structure sampling,
catalyst-adsorbate sampling, and property-guided structure generation. DiG
presents a significant advancement in methodology for statistically
understanding molecular systems, opening up new research opportunities in
molecular science.Comment: 80 pages, 11 figure
optimización da planificación de adquisición de datos LIDAR cara ó modelado 3D de interiores
The main objective of this doctoral thesis is the design, validation and implementation of methodologies that allow the geometric and topological modelling of navigable spaces, whether inside buildings or urban environments, to be integrated into three-dimensional geographic information systems (GIS-3D).
The input data of this work will consist mainly of point clouds (which can be classified) acquired by LiDAR systems both indoors and outdoors. In addition, the use of BIM infrastructure models and cadastral maps is proposed depending on their availability.
Point clouds provide a large amount of environmental information with high accuracy compared to data offered by other acquisition technologies. However, the lack of data structure and volume requires a great deal of processing effort. For this reason, the first step is to structure the data by dividing the input cloud into simpler entities that facilitate subsequent processes. For this first division, the physical elements present in the cloud will be considered, since they can be walls in the case of interior environments or kerbs in the case of exteriors.
In order to generate navigation routes adapted to different mobile agents, the next objective will try to establish a semantic subdivision of space according to the functionalities of space. In the case of internal environments, it is possible to use BIM models to evaluate the results and the use of cadastral maps that support the division of the urban environment.
Once the navigable space is divided, the design of topologically coherent navigation networks will be parameterized both geometrically and topologically. For this purpose, several spatial discretization techniques, such as 3D tessellations, will be studied to facilitate the establishment of topological relationships, adjacency, connectivity and inclusion between subspaces.
Based on the geometric characterization and the topological relations established in the previous phase, the creation of three-dimensional navigation networks with multimodal support will be addressed and different levels of detail will be considered according to the mobility specifications of each agent and its purpose.
Finally, the possibility of integrating the networks generated in a GIS-3D visualization system will be considered. For the correct visualization, the level of detail can be adjusted according to geometry and semantics. Aspects such as the type of user or transport, mobility, rights of access to spaces, etc. They must be considered at all times.El objetivo principal de esta tesis doctoral es el diseño, la validación y la implementación de metodologías que permitan el modelado geométrico y topológico de espacios navegables, ya sea de interiores de edificios o entornos urbanos, para integrarse en sistemas de información geográfica tridimensional (SIG). -3D).
Los datos de partida de este trabajo consistirán principalmente en nubes de puntos (que pueden estar clasificados) adquiridas por sistemas LiDAR tanto en interiores como en exteriores. Además, se propone el uso de modelos BIM de infraestructuras y mapas catastrales en función de su disponibilidad.
Las nubes de puntos proporcionan una gran cantidad de información del entorno con gran precisión con respecto a los datos ofrecidos por otras tecnologías de adquisición. Sin embargo, la falta de estructura de datos y su volumen requiere un gran esfuerzo de procesamiento. Por este motivo, el primer paso que se debe realizar consiste en estructurar los datos dividiendo la nube de entrada en entidades más simples que facilitan los procesos posteriores. Para esta primera división se considerarán los elementos físicos presentes en la nube, ya que pueden ser paredes en el caso de entornos interiores o bordillos en el caso de los exteriores.
Con el propósito de generar rutas de navegación adaptadas a diferentes agentes móviles, el próximo objetivo intentará establecer una subdivisión semántica del espacio de acuerdo con las funcionalidades del espacio. En el caso de entornos internos, es posible utilizar modelos BIM para evaluar los resultados y el uso de mapas catastrales que sirven de apoyo en la división del entorno urbano.
Una vez que se divide el espacio navegable, se parametrizará tanto geométrica como topológicamente al diseño de redes de navegación topológicamente coherentes. Para este propósito, se estudiarán varias técnicas de discretización espacial, como las teselaciones 3D, para facilitar el establecimiento de relaciones topológicas, la adyacencia, la conectividad y la inclusión entre subespacios.
A partir de la caracterización geométrica y las relaciones topológicas establecidas en la fase anterior, se abordará la creación de redes de navegación tridimensionales con soporte multimodal y se considerarán diversos niveles de detalle según las especificaciones de movilidad de cada agente y su propósito.
Finalmente, se contemplará la posibilidad de integrar las redes generadas en un sistema de visualización tridimensional 3D SIG 3D. Para la correcta visualización, el nivel de detalle se puede ajustar en función de la geometría y la semántica. Aspectos como el tipo de usuario o transporte, movilidad, derechos de acceso a espacios, etc. Deben ser considerados en todo momento.O obxectivo principal desta tese doutoral é o deseño, validación e implementación de metodoloxías que permitan o modelado xeométrico e topolóxico de espazos navegables, ben sexa de interiores de edificios ou de entornos urbanos, ca fin de seren integrados en Sistemas de Información Xeográfica tridimensionais (SIX-3D).
Os datos de partida deste traballo constarán principalmente de nubes de puntos (que poden estar clasificadas) adquiridas por sistemas LiDAR tanto en interiores como en exteriores. Ademáis plantease o uso de modelos BIM de infraestruturas e mapas catastrais dependendo da súa dispoñibilidade.
As nubes de puntos proporcionan unha gran cantidade de información do entorno cunha gran precisión respecto os datos que ofrecen outras tecnoloxías de adquisición. Sen embargo, a falta de estrutura dos datos e a seu volume esixe un amplo esforzo de procesado. Por este motivo o primeiro paso a levar a cabo consiste nunha estruturación dos datos mediante a división da nube de entrada en entidades máis sinxelas que faciliten os procesos posteriores. Para esta primeira división consideraranse elementos físicos presentes na nube como poden ser paredes no caso de entornos interiores ou bordillos no caso de exteriores.
Coa finalidade de xerar rutas de navegación adaptadas a distintos axentes móbiles, o seguinte obxectivo tratará de establecer unha subdivisión semántica do espazo de acordo as funcionalidades do espazo. No caso de entornos interiores plantease a posibilidade de empregar modelos BIM para avaliar os resultados e o uso de mapas catastrais que sirvan de apoio na división do entorno urbano.
Unha vez divido o espazo navigable parametrizarase tanto xeométricamente como topolóxicamene de cara ao deseño de redes de navegación topolóxicamente coherentes. Para este fin estudaranse varias técnicas de discretización de espazos como como son as teselacións 3D co obxectivo de facilitar establecer relacións topolóxicas, de adxacencia, conectividade e inclusión entre subespazos.
A partir da caracterización xeométrica e das relación topolóxicas establecidas na fase previa abordarase a creación de redes de navegación tridimensionais con soporte multi-modal e considerando varios niveis de detalle de acordo as especificacións de mobilidade de cada axente e a súa finalidade.
Finalmente comtemplarase a posibilidade de integrar as redes xeradas nun sistema SIX 3D visualización tridimensional. Para a correcta visualización o nivel de detalle poderá axustarse en base a xeometría e a semántica. Aspectos como o tipo de usuario ou transporte, mobilidade, dereitos de acceso a espazos, etc. deberán ser considerados en todo momento
Advances in Methane Production from Coal, Shale and Other Tight Rocks
This collection reports on the state of the art in fundamental discipline application in hydrocarbon production and associated challenges in geoengineering activities. Zheng et al. (2022) report an NMR-based method for multiphase methane characterization in coals. Wang et al. (2022) studied the genesis of bedding fractures in Ordovician to Silurian marine shale in the Sichuan basin. Kang et al. (2022) proposed research focusing on the prediction of shale gas production from horizontal wells. Liang et al. (2022) studied the pore structure of marine shale by adsorption method in terms of molecular interaction. Zhang et al. (2022) focus on the coal measures sandstones in the Xishanyao Formation, southern Junggar Basin, and the sandstone diagenetic characteristics are fully revealed. Yao et al. (2022) report the source-to-sink system in the Ledong submarine channel and the Dongfang submarine fan in the Yinggehai Basin, South China Sea. There are four papers focusing on the technologies associated with hydrocarbon productions. Wang et al. (2022) reported the analysis of pre-stack inversion in a carbonate karst reservoir. Chen et al. (2022) conducted an inversion study on the parameters of cascade coexisting gas-bearing reservoirs in coal measures in Huainan. To ensure the safety CCS, Zhang et al (2022) report their analysis of available conditions for InSAR surface deformation monitoring. Additionally, to ensure production safety in coal mines, Zhang et al. (2022) report the properties and application of gel materials for coal gangue control
Convex Optimization for Machine Learning
This book covers an introduction to convex optimization, one of the powerful and tractable optimization problems that can be efficiently solved on a computer. The goal of the book is to
help develop a sense of what convex optimization is, and how it can be used in a widening array of practical contexts with a particular emphasis on machine learning.
The first part of the book covers core concepts of convex sets, convex functions, and related basic definitions that serve understanding convex optimization and its corresponding models. The second part deals with one very useful theory, called duality, which enables us to: (1) gain algorithmic insights; and (2) obtain an approximate solution to non-convex optimization problems which are often difficult to solve. The last part focuses on modern applications in machine learning and deep learning.
A defining feature of this book is that it succinctly relates the “story” of how convex optimization plays a role, via historical examples and trending machine learning applications. Another key feature is that it includes programming implementation of a variety of machine learning algorithms inspired by optimization fundamentals, together with a brief tutorial of the used programming tools. The implementation is based on Python, CVXPY, and TensorFlow.
This book does not follow a traditional textbook-style organization, but is streamlined via a series of lecture notes that are intimately related, centered around coherent themes and concepts. It serves as a textbook mainly for a senior-level undergraduate course, yet is also suitable for a first-year graduate course. Readers benefit from having a good background in linear algebra, some exposure to probability, and basic familiarity with Python
- …