43 research outputs found
Deep Policies for Width-Based Planning in Pixel Domains
Width-based planning has demonstrated great success in recent years due to
its ability to scale independently of the size of the state space. For example,
Bandres et al. (2018) introduced a rollout version of the Iterated Width
algorithm whose performance compares well with humans and learning methods in
the pixel setting of the Atari games suite. In this setting, planning is done
on-line using the "screen" states and selecting actions by looking ahead into
the future. However, this algorithm is purely exploratory and does not leverage
past reward information. Furthermore, it requires the state to be factored into
features that need to be pre-defined for the particular task, e.g., the B-PROST
pixel features. In this work, we extend width-based planning by incorporating
an explicit policy in the action selection mechanism. Our method, called
-IW, interleaves width-based planning and policy learning using the
state-actions visited by the planner. The policy estimate takes the form of a
neural network and is in turn used to guide the planning step, thus reinforcing
promising paths. Surprisingly, we observe that the representation learned by
the neural network can be used as a feature space for the width-based planner
without degrading its performance, thus removing the requirement of pre-defined
features for the planner. We compare -IW with previous width-based methods
and with AlphaZero, a method that also interleaves planning and learning, in
simple environments, and show that -IW has superior performance. We also
show that -IW algorithm outperforms previous width-based methods in the
pixel setting of Atari games suite.Comment: In Proceedings of the 29th International Conference on Automated
Planning and Scheduling (ICAPS 2019). arXiv admin note: text overlap with
arXiv:1806.0589
Hierarchical Width-Based Planning and Learning
Width-based search methods have demonstrated state-of-the-art performance in
a wide range of testbeds, from classical planning problems to image-based
simulators such as Atari games. These methods scale independently of the size
of the state-space, but exponentially in the problem width. In practice,
running the algorithm with a width larger than 1 is computationally
intractable, prohibiting IW from solving higher width problems. In this paper,
we present a hierarchical algorithm that plans at two levels of abstraction. A
high-level planner uses abstract features that are incrementally discovered
from low-level pruning decisions. We illustrate this algorithm in classical
planning PDDL domains as well as in pixel-based simulator domains. In classical
planning, we show how IW(1) at two levels of abstraction can solve problems of
width 2. For pixel-based domains, we show how in combination with a learned
policy and a learned value function, the proposed hierarchical IW can
outperform current flat IW-based planners in Atari games with sparse rewards
Colaboración de la Universidad de Barcelona en las excavaciones del poblado ibérico de Molí d'Espígol, en Tornabous
La Internet del Futur: Anàlisi retrospectiva, reflexió i posicionament estratègic
Aquest treball té com a objectiu analitzar la situació actual de la Internet, per proposar noves alternatives i dibuixar un possible camí cap a la Internet del Futur. Aquesta anàlisi vol ser un punt de partida, un document de reflexió, que posicioni als agents locals i nacionals (entorns de la recerca, innovació i producció, i usuaris) en una situació capdavantera en la construcció del nou model econòmic i social del segle XXI.
El treball consta de dos grans apartats, una introducció descriptiva i conceptual del paper que juga la Internet i la seva projecció futura, i una segona part on es desdibuixen en mes profunditat les bases tecnològiques de la possible Internet del futur a partir de la realitat actual.Preprin
State of emergency medicine in Spain
Spain has universal public health care coverage. Emergency care provisions are offered to patients in different modalities and levels according to the characteristics of the medical complaint: at primary care centers (PCC), in an extrahospital setting by emergency medical services (EMS) and at hospital emergency departments (ED). We have more than 3,000 PCCs, which are run by family doctors (general practitioners) and pediatricians. On average, there is 1 PCC for every 15,000 to 20,000 inhabitants, and every family doctor is in charge of 1,500 to 2,000 citizens, although less populated zones tend to have lower ratios. Doctors spend part of their duty time in providing emergency care to their own patients. While not fully devoted to emergency medicine (EM) practice, they do manage minor emergencies. However, Spanish EMSs contribute hugely to guarantee population coverage in all situations. These EMS are run by EM technicians (EMT), nurses and doctors, who usually work exclusively in the emergency arena. EDs dealt with more than 25 million consultations in 2008, which implies, on average, that one out of two Spaniards visited an ED during this time. They are usually equipped with a wide range of diagnostic tools, most including ultrasonography and computerized tomography scans. The academic and training background of doctors working in the ED varies: nearly half lack any structured specialty residence training, but many have done specific master or postgraduate studies within the EM field. The demand for emergency care has grown at an annual rate of over 4% during the last decade. This percentage, which was greater than the 2% population increase during the same period, has outpaced the growth in ED capacity. Therefore, Spanish EDs become overcrowded when the system exerts minimal stress. Despite the high EM caseload and the potential severity of the conditions, training in EM is still unregulated in Spain. However, in April 2009 the Spanish Minister of Health announced the imminent approval of an EM specialty, allowing the first EM resident to officially start in 2011. Spanish emergency physicians look forward to the final approval, which will complete the modernization of emergency health care provision in Spain
Treatment with tocilizumab or corticosteroids for COVID-19 patients with hyperinflammatory state: a multicentre cohort study (SAM-COVID-19)
Objectives: The objective of this study was to estimate the association between tocilizumab or corticosteroids and the risk of intubation or death in patients with coronavirus disease 19 (COVID-19) with a hyperinflammatory state according to clinical and laboratory parameters.
Methods: A cohort study was performed in 60 Spanish hospitals including 778 patients with COVID-19 and clinical and laboratory data indicative of a hyperinflammatory state. Treatment was mainly with tocilizumab, an intermediate-high dose of corticosteroids (IHDC), a pulse dose of corticosteroids (PDC), combination therapy, or no treatment. Primary outcome was intubation or death; follow-up was 21 days. Propensity score-adjusted estimations using Cox regression (logistic regression if needed) were calculated. Propensity scores were used as confounders, matching variables and for the inverse probability of treatment weights (IPTWs).
Results: In all, 88, 117, 78 and 151 patients treated with tocilizumab, IHDC, PDC, and combination therapy, respectively, were compared with 344 untreated patients. The primary endpoint occurred in 10 (11.4%), 27 (23.1%), 12 (15.4%), 40 (25.6%) and 69 (21.1%), respectively. The IPTW-based hazard ratios (odds ratio for combination therapy) for the primary endpoint were 0.32 (95%CI 0.22-0.47; p < 0.001) for tocilizumab, 0.82 (0.71-1.30; p 0.82) for IHDC, 0.61 (0.43-0.86; p 0.006) for PDC, and 1.17 (0.86-1.58; p 0.30) for combination therapy. Other applications of the propensity score provided similar results, but were not significant for PDC. Tocilizumab was also associated with lower hazard of death alone in IPTW analysis (0.07; 0.02-0.17; p < 0.001).
Conclusions: Tocilizumab might be useful in COVID-19 patients with a hyperinflammatory state and should be prioritized for randomized trials in this situatio
Width-Based Planning and Learning
Optimal sequential decision making is a fundamental problem to many diverse fields. In recent
years, Reinforcement Learning (RL) methods have experienced unprecedented success, largely
enabled by the use of deep learning models, reaching human-level performance in several
domains, such as the Atari video games or the ancient game of Go. In contrast to the RL
approach in which the agent learns a policy from environment interaction samples, ignoring the
structure of the problem, the planning approach for decision making assumes known models for
the agent's goals and domain dynamics, and focuses on determining how the agent should
behave to achieve its objectives. Current planners are able to solve problem instances involving
huge state spaces by precisely exploiting the problem structure that is defined in the
state-action model.
In this work we combine the two approaches, leveraging fast and compact policies from learning
methods and the capacity to perform lookaheads in combinatorial problems from planning
methods. In particular, we focus on a family of planners called width-based planners, that has
demonstrated great success in recent years due to its ability to scale independently of the size
of the state space. The basic algorithm, Iterated Width (IW), was originally proposed for
classical planning problems, where a model for state transitions and goals, represented by sets
of atoms, is fully determined. Nevertheless, width-based planners do not require a fully defined
model of the environment, and can be used with simulators. For instance, they have been
recently applied in pixel domains such as the Atari games.
Despite its success, IW is purely exploratory, and does not leverage past reward information.
Furthermore, it requires the state to be factored into features that need to be pre-defined for the
particular task. Moreover, running the algorithm with a width larger than 1 in practice is usually
computationally intractable, prohibiting IW from solving higher width problems.
We begin this dissertation by studying the complexity of width-based methods when the state
space is defined by multivalued features, as in the RL setting, instead of Boolean atoms. We
provide a tight upper bound on the amount of nodes expanded by IW, as well as overall
algorithmic complexity results. In order to deal with more challenging problems (i.e., those with a
width higher than 1), we present a hierarchical algorithm that plans at two levels of abstraction.
A high-level planner uses abstract features that are incrementally discovered from low-level
pruning decisions. We illustrate this algorithm in classical planning PDDL domains as well as in
pixel-based simulator domains. In classical planning, we show how IW(1) at two levels of
abstraction can solve problems of width 2.
To leverage past reward information, we extend width-based planning by incorporating an
explicit policy in the action selection mechanism. Our method, called π-IW, interleaves
width-based planning and policy learning using the state-actions visited by the planner. The
policy estimate takes the form of a neural network and is in turn used to guide the planning step,
thus reinforcing promising paths. Notably, the representation learned by the neural network can
be used as a feature space for the width-based planner without degrading its performance, thus
removing the requirement of pre-defined features for the planner. We compare π-IW with
previous width-based methods and with AlphaZero, a method that also interleaves planning and
learning, in simple environments, and show that π-IW has superior performance. We also show
that the π-IW algorithm outperforms previous width-based methods in the pixel setting of Atari
games suite. Finally, we show that the proposed hierarchical IW can be seamlessly integrated
with our policy learning scheme, resulting in an algorithm that outperforms flat IW-based
planners in Atari games with sparse rewards.La presa seqüencial de decisions òptimes és un problema fonamental en diversos camps. En
els últims anys, els mètodes d'aprenentatge per reforç (RL) han experimentat un èxit sense
precedents, en gran part gràcies a l'ús de models d'aprenentatge profund, aconseguint un
rendiment a nivell humà en diversos dominis, com els videojocs d'Atari o l'antic joc de Go. En
contrast amb l'enfocament de RL, on l'agent aprèn una política a partir de mostres d'interacció
amb l'entorn, ignorant l'estructura del problema, l'enfocament de planificació assumeix models
coneguts per als objectius de l'agent i la dinàmica del domini, i es basa en determinar com ha
de comportar-se l'agent per aconseguir els seus objectius. Els planificadors actuals són
capaços de resoldre problemes que involucren grans espais d'estats precisament explotant
l'estructura del problema, definida en el model estat-acció.
En aquest treball combinem els dos enfocaments, aprofitant polítiques ràpides i compactes dels
mètodes d'aprenentatge i la capacitat de fer cerques en problemes combinatoris dels mètodes
de planificació. En particular, ens enfoquem en una família de planificadors basats en el width
(ample), que han tingut molt èxit en els últims anys gràcies a que la seva escalabilitat és
independent de la mida de l'espai d'estats. L'algorisme bàsic, Iterated Width (IW), es va
proposar originalment per problemes de planificació clàssica, on el model de transicions d'estat
i objectius ve completament determinat, representat per conjunts d'àtoms. No obstant, els
planificadors basats en width no requereixen un model de l'entorn completament definit i es
poden utilitzar amb simuladors. Per exemple, s'han aplicat recentment a dominis gràfics com
els jocs d'Atari.
Malgrat el seu èxit, IW és un algorisme purament exploratori i no aprofita la informació de
recompenses anteriors. A més, requereix que l'estat estigui factoritzat en característiques, que
han de predefinirse per a la tasca en concret. A més, executar l'algorisme amb un width
superior a 1 sol ser computacionalment intractable a la pràctica, el que impedeix que IW
resolgui problemes de width superior.
Comencem aquesta tesi estudiant la complexitat dels mètodes basats en width quan l'espai
d'estats està definit per característiques multivalor, com en els problemes de RL, en lloc d'àtoms
booleans. Proporcionem un límit superior més precís en la quantitat de nodes expandits per IW,
així com resultats generals de complexitat algorísmica. Per fer front a problemes més
complexos (és a dir, aquells amb un width superior a 1), presentem un algorisme jeràrquic que
planifica en dos nivells d'abstracció. El planificador d'alt nivell utilitza característiques abstractes
que es van descobrint gradualment a partir de decisions de poda en l'arbre de baix nivell.
Il·lustrem aquest algorisme en dominis PDDL de planificació clàssica, així com en dominis de
simuladors gràfics. En planificació clàssica, mostrem com IW(1) en dos nivells d'abstracció pot
resoldre problemes de width 2.
Per aprofitar la informació de recompenses passades, incorporem una política explícita en el
mecanisme de selecció d'accions. El nostre mètode, anomenat π-IW, intercala la planificació
basada en width i l'aprenentatge de la política usant les accions visitades pel planificador.
Representem la política amb una xarxa neuronal que, al seu torn, s'utilitza per guiar la
planificació, reforçant així camins prometedors. A més, la representació apresa per la xarxa
neuronal es pot utilitzar com a característiques per al planificador sense degradar el seu
rendiment, eliminant així el requisit d'usar característiques predefinides. Comparem π-IW amb
mètodes anteriors basats en width i amb AlphaZero, un mètode que també intercala planificació
i aprenentatge, i mostrem que π-IW té un rendiment superior en entorns simples. També
mostrem que l'algorisme π-IW supera altres mètodes basats en width en els jocs d'Atari.
Finalment, mostrem que el mètode IW jeràrquic proposat pot integrar-se fàcilment amb el nostre
esquema d'aprenentatge de la política, donant com a resultat un algorisme que supera els
planificadors no jeràrquics basats en IW en els jocs d'Atari amb recompenses distants.La toma secuencial de decisiones óptimas es un problema fundamental en diversos campos.
En los últimos años, los métodos de aprendizaje por refuerzo (RL) han experimentado un éxito
sin precedentes, en gran parte gracias al uso de modelos de aprendizaje profundo, alcanzando
un rendimiento a nivel humano en varios dominios, como los videojuegos de Atari o el antiguo
juego de Go. En contraste con el enfoque de RL, donde el agente aprende una política a partir
de muestras de interacción con el entorno, ignorando la estructura del problema, el enfoque de
planificación asume modelos conocidos para los objetivos del agente y la dinámica del dominio,
y se basa en determinar cómo debe comportarse el agente para lograr sus objetivos. Los
planificadores actuales son capaces de resolver problemas que involucran grandes espacios de
estados precisamente explotando la estructura del problema, definida en el modelo
estado-acción.
En este trabajo combinamos los dos enfoques, aprovechando políticas rápidas y compactas de
los métodos de aprendizaje y la capacidad de realizar búsquedas en problemas combinatorios
de los métodos de planificación. En particular, nos enfocamos en una familia de planificadores
basados en el width (ancho), que han demostrado un gran éxito en los últimos años debido a
que su escalabilidad es independiente del tamaño del espacio de estados. El algoritmo básico,
Iterated Width (IW), se propuso originalmente para problemas de planificación clásica, donde el
modelo de transiciones de estado y objetivos viene completamente determinado, representado
por conjuntos de átomos. Sin embargo, los planificadores basados en width no requieren un
modelo del entorno completamente definido y se pueden utilizar con simuladores. Por ejemplo,
se han aplicado recientemente en dominios gráficos como los juegos de Atari.
A pesar de su éxito, IW es un algoritmo puramente exploratorio y no aprovecha la información
de recompensas anteriores. Además, requiere que el estado esté factorizado en
características, que deben predefinirse para la tarea en concreto. Además, ejecutar el algoritmo
con un width superior a 1 suele ser computacionalmente intratable en la práctica, lo que impide
que IW resuelva problemas de width superior.
Empezamos esta tesis estudiando la complejidad de los métodos basados en width cuando el
espacio de estados está definido por características multivalor, como en los problemas de RL,
en lugar de átomos booleanos. Proporcionamos un límite superior más preciso en la cantidad
de nodos expandidos por IW, así como resultados generales de complejidad algorítmica. Para
hacer frente a problemas más complejos (es decir, aquellos con un width superior a 1),
presentamos un algoritmo jerárquico que planifica en dos niveles de abstracción. El planificador
de alto nivel utiliza características abstractas que se van descubriendo gradualmente a partir de
decisiones de poda en el árbol de bajo nivel. Ilustramos este algoritmo en dominios PDDL de
planificación clásica, así como en dominios de simuladores gráficos. En planificación clásica,
mostramos cómo IW(1) en dos niveles de abstracción puede resolver problemas de width 2.
Para aprovechar la información de recompensas pasadas, incorporamos una política explícita
en el mecanismo de selección de acciones. Nuestro método, llamado π-IW, intercala la
planificación basada en width y el aprendizaje de la política usando las acciones visitadas por el
planificador. Representamos la política con una red neuronal que, a su vez, se utiliza para guiar
la planificación, reforzando así caminos prometedores. Además, la representación aprendida
por la red neuronal se puede utilizar como características para el planificador sin degradar su
rendimiento, eliminando así el requisito de usar características predefinidas. Comparamos π-IW
con métodos anteriores basados en width y con AlphaZero, un método que también intercala
planificación y aprendizaje, y mostramos que π-IW tiene un rendimiento superior en entornos
simples. También mostramos que el algoritmo π-IW supera otros métodos basados en width en
los juegos de Atari. Finalmente, mostramos que el IW jerárquico propuesto puede integrarse
fácilmente con nuestro esquema de aprendizaje de la política, dando como resultado un
algoritmo que supera a los planificadores no jerárquicos basados en IW en los juegos de Atari
con recompensas distantes