Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning

Recent advances in combining deep neural network architectures with reinforcement learning techniques have shown promising potential results in solving complex control problems with high dimensional state and action spaces. Inspired by these successes, in this paper, we build two kinds of reinforcement learning algorithms: deep policy-gradient and value-function based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The policy-gradient based agent maps its observation directly to the control signal, however the value-function based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Our methods show promising results in a traffic network simulated in the SUMO traffic simulator, without suffering from instability issues during the training process

Utilising Assured Multi-Agent Reinforcement Learning within safety-critical scenarios

Multi-agent reinforcement learning allows a team of agents to learn how to work together to solve complex decision-making problems in a shared environment. However, this learning process utilises stochastic mechanisms, meaning that its use in safety-critical domains can be problematic. To overcome this issue, we propose an Assured Multi-Agent Reinforcement Learning (AMARL) approach that uses a model checking technique called quantitative verification to provide formal guarantees of agent compliance with safety, performance, and other non-functional requirements during and after the reinforcement learning process. We demonstrate the applicability of our AMARL approach in three different patrolling navigation domains in which multi-agent systems must learn to visit key areas by using different types of reinforcement learning algorithms (temporal difference learning, game theory, and direct policy search). Furthermore, we compare the effectiveness of these algorithms when used in combination with and without our approach. Our extensive experiments with both homogeneous and heterogeneous multi-agent systems of different sizes show that the use of AMARL leads to safety requirements being consistently satisfied and to better overall results than standard reinforcement learning

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology

Scientific advances in diabetes: the impact of the innovative medicines initiative

Tese de mestrado, Regulação e Avaliação do Medicamento e Produtos de Saúde, 2020, Universidade de Lisboa, Faculdade de Farmácia.A Iniciativa sobre Medicamentos Inovadores é uma parceria público-privada entre a Comunidade Europeia, representada pela Comissão Europeia, e a Indústria Farmacêutica, representada pela Federação Europeia da Indústria Farmacêutica. Esta Iniciativa de Tecnologia Conjunta tem como objetivos acelerar o processo de investigação e desenvolvimento de medicamentos inovadores, bem como gerar novos conhecimentos científicos que promovam a integração da medicina personalizada nas doenças prioritárias definidas pela Organização Mundial de Saúde. Atualmente, no âmbito desta iniciativa foram estabelecidos dois programas, sendo que o primeiro (IMI1) decorreu entre 2008 e 2013 e teve um orçamento de 2 mil milhões de euros, enquanto que o segundo programa (IMI2) está em decurso desde 2014 e terminará em 2020 e o orçamento disponibilizado foi de 3.276 mil milhões de euros. A diabetes mellitus é uma das doenças prioritárias indicadas pela Organização Mundial de Saúde alvo de financiamento pelos programas IMI. As principais justificações para este facto prendem-se com os dados epidemiológicos da doença. No decorrer dos anos, verificou-se um aumento exponencial da taxa de prevalência desta doença a nível mundial. Esta evidência é suportada pelo facto de, entre o período de 1980 e 2014, esta taxa ter sofrido um aumento de 4.7% para 8.7%, o quadruplo do valor, em adultos com idade igual ou superior a 18 anos e também por as estimativas a 20 anos, realizadas pela Organização Mundial de Saúde, indicarem que o número total de casos existentes corresponderá a mais de 20% da população universal. Simultaneamente, constatou-se um crescimento progressivo tanto da taxa de mortalidade por diabetes bem como dos custos de saúde acarretados por esta doença. No que diz respeito à taxa de mortalidade, em 2016, a diabetes foi considerada a sétima principal causa de morte no mundo. Em termos de impacto económico, a diabetes e as complicações decorrentes desta doença, como é o caso das doenças cardiovasculares, nefropatia diabética e retinopatia diabética, impõem um grande peso económico para os sistemas de saúde. A nível mundial, os custos anuais provocados pela diabetes, entre o ano de 2007 e 2019, aumentaram de 232 mil milhões de dólares para 760 mil milhões de dólares, o que equivale a incremento de 528 mil milhões de dólares em 12 anos. Na área da Diabetes, o principal objetivo dos programas IMI1 e IMI2 é o de reduzir a tendência crescente observada na taxa de prevalência desta doença. De forma a atingir esta meta, os dois programas supramencionados primaram o financiamento de projetos cujo intuito consistia no desenvolvimento do conhecimento, medicamentos, métodos, ferramentas e modelos que facilitassem a implementação da medicina personalizada, como modelo de prática médica corrente, em doentes com diabetes. Até outubro de 2019, os programas IMI financiaram treze projetos para a área da Diabetes & Doenças Metabólicas, nomeadamente SUMMIT, IMIDIA, DIRECT, StemBANCC, EMIF, EBiSC, INNODIA, RHAPSODY, BEAT-DKD, LITMUS, Hypo-RESOLVE, IM2PACT e CARDIATEAM. Entre estes, o INNODIA tinha como objetivo a diabetes tipo 1, o DIRECT, EMIF e RHAPSODY tinham como foco a diabetes tipo 2, o SUMMIT, BEAT-DKD, LITMUS, Hypo-RESOLVE e CARDIATEAM estavam associados às complicações da diabetes e os restantes projetos, o StemBANCC, EBiSC, IMIDIA e IMI2PACT, estavam orientados para o desenvolvimento da vertente científica. Em geral, um investimento monetário total na ordem dos €447 249 438 foi realizado pelo IMI na área da Diabetes. Todavia, a deteção da lacuna existente na integração dos resultados produzidos pelos diferentes projetos, impulsionou a elaboração da presente dissertação intitulada de “Scientific Advances in Diabetes: The Impact of the Innovative Medicines Initiative”, ou seja, Avanços Científicos na Área da Diabetes: Impacto da Iniciativa sobre Medicamentos Inovadores. Os principais objetivos estabelecidos para esta dissertação foram os de recolher os artigos publicados pelos projetos financiados e sistematizá-los nos eixos de investigação definidos na agenda estratégica do programa IMI2, mais concretamente: 1) identificação de alvos e biomarcadores, 2) novos paradigmas de ensaios clínicos, 3) medicamentos inovadores e 4) programas de adesão terapêutica centrados nos doentes. A metodologia de investigação aplicada nesta dissertação consistiu numa revisão de literatura, tendo-se utilizado como fontes de dados as páginas eletrónicas oficiais de cada projeto, o contacto com os coordenadores e co-coordenadores dos projetos e a base de dados europeia Cordis. No geral, um total de 662 citações foram identificadas, das quais 185 foram incluídas na análise realizada neste trabalho. Através da sistematização e integração dos artigos recolhidos nos projetos financiados pelo IMI, averiguou-se que para o eixo de identificação de alvos e biomarcadores, os outcomes relevantes responderam a cinco das recomendações definidas na agenda estratégica do programa IMI2, nomeadamente: 1) identificar e validar marcadores biológicos, ferramentas e ensaios, 2) identificar as determinantes que justificam a variabilidade interindividual, 3) compreender os mecanismos moleculares subjacentes à doença, 4) desenvolver uma plataforma de ensaios pré-clínicos e 5) estabelecer modelos de sistemas. De um modo geral, um vasto número de biomarcadores, ferramentas, fatores responsáveis pela heterogeneidade da população, incluindo marcadores genéticos, e mecanismos relevantes foram identificados para a diabetes tipo 1 pelo INNODIA, para a diabetes tipo 2 pelo SUMMIT, IMIDIA, DIRECT e EMIF, para as células beta pancreáticas pelo IMIDIA e RHAPSODY, para a nefropatia diabética pelo SUMMIT e BEAT-DKD, e para as doenças cardiovasculares e retinopatia diabética pelo SUMMIT. Suplementarmente, um conjunto de ferramentas e ensaios foram desenvolvidos pelos projetos StemBANCC, EBiSC e IMIDIA com o intuito de impulsionar avanços na área investigacional. Ainda neste eixo foram propostos dois modelos de estratificação dos doentes, um relativo ao controlo glicémico em doentes com diabetes tipo 1 estabelecido pelo INNODIA e outro correspondente à identificação dos subtipos de doentes com diabetes desenvolvido pela parceria BEAT-DKD/RHAPSODY. Relativamente ao eixo de ensaios clínicos, os dados analisados compreendiam propostas de novos parâmetros clínicos e de desenhos de ensaios, sendo que estes resultados visavam espelharem com maior precisão as características da subpopulação com diabetes em teste. Os dados incluídos neste eixo foram obtidos a partir dos projetos SUMMIT, DIRECT e BEAT-DKD. No que concerne ao eixo de medicamentos inovadores, as informações recolhidas dos artigos publicados pelos projetos SUMMIT, IMIDIA, DIRECT, StemBANNC, EMIF, INNODIA e BEAT-DKD consistiam na identificação de novos potenciais alvos terapêuticos bem como no desenvolvimento de novos agentes terapêuticos, ambos com a finalidade de tratar ou prevenir tanto a diabetes como as complicações associadas a esta doença. Foi ainda proposta uma nova abordagem de produção de células estaminais pluripotentes humanas em larga escala pelo StemBANCC. No que tange aos programas de maximização de resultados de saúde benéficos centrados no doente com diabetes, dois novos modelos preditivos foram desenvolvidos e validados pelo projeto DIRECT, permitindo a sua utilização como ferramentas de diagnóstico por médicos especialistas. Adicionalmente, esta dissertação tem como objetivo apresentar uma proposta de visão de complementaridade entre os treze projetos financiados pelo IMI, realçando as possíveis estratégias a adotar para a integração da medicina personalizada na prática clínica. Esta abordagem engloba a criação de indicadores biológicos e genéticos que facilitem a identificação dos indivíduos com risco elevado de desenvolver diabetes, a inclusão de ferramentas que possibilitem o diagnóstico precoce dos doentes e, por último, a seleção do tratamento apropriado às características do indivíduo, ou seja o que evidencie ser mais eficaz e seguro, suportado em modelos de estratificação de doentes, tentando desta forma retardar a progressão da doença, assim como prevenir o desenvolvimento das complicações relacionadas com a progressão da doença.Innovative Medicines Initiative (IMI) is a public-private partnership between the European Community, represented by the European Commission, and the European Federation of Pharmaceutical Industries and Associations. This joint undertaking aims at accelerating the medicines development process and generating new scientific knowledge to promote the implementation of personalized medicine for priority diseases established by the World Health Organization. Currently, two IMI programmes have been undertaken, the first one (IMI1) was carried out from 2008 until 2013 and had a budget of €2 billion, and the second one (IMI2) was developed from 2014 up to 2020 and the budget committed was up to €3.276 billion. Diabetes Mellitus is one of the World Health Organization’s priority diseases under research by the IMI programmes, mainly due to the exponential increase of its global prevalence over the years. Between 1980 and 2014, this rate quadrupled from 4.7% to 8.7% in adults aged 18 years and older and the 20 years- World Health Organization’s projections indicate that it could reach more than 20% of the population. Simultaneously, the mortality rate and the healthcare costs associated with diabetes have been increasing. Regarding mortality, diabetes was the seventh leading cause of death in 2016. In terms of economic impact, currently, diabetes and its related complications, such as cardiovascular diseases, diabetic kidney disease and diabetic retinopathy, represent a significant economic burden on the healthcare systems. Worldwide, the estimated annual costs of diabetes have increased from $232 billion to$760 billion, between 2007 and 2019. In the Diabetes field, the main aim of IMI1 and IMI2 programmes is to shorten the prevalence of this disease, through the development of knowledge and methods that enable the implementation of personalized treatment for diabetic patients. Up to October of 2019, thirteen projects were funded by IMI for Diabetes & Metabolic disorders, more precisely SUMMIT, IMIDIA, DIRECT, StemBANCC, EMIF, EBiSC, INNODIA, RHAPSODY, BEAT-DKD, LITMUS, Hypo-RESOLVE, IM2PACT, and CARDIATEAM. Of these, INNODIA aimed at type 1 diabetes, DIRECT, EMIF and RHAPSODY were focused on type 2 diabetes, SUMMIT, BEAT-DKD, LITMUS, Hypo- RESOLVE and CARDIATEAM were related to complications of diabetes, and the remaining projects, namely StemBANCC, EBiSC, IMIDIA and IMI2PACT, were directed to scientific research. In general, a total of €447 249 438 was spent by IMI in the area of Diabetes. However, there is a substantial lack of integration of achievements between the different projects, which prompted the development of this dissertation: “Scientific Advances in Diabetes: The Impact of the Innovative Medicines Initiative”. This dissertation’ objectives were to collect the data of the funded-projects and integrate them into the following research axes: 1) target and biomarker identification, 2) innovative clinical trials paradigms, 3) innovative medicines, and 4) patient-tailored adherence programmes. The research methodology applied was a literature review and the data sources used were the official project’s websites, contacts with the project’s coordinators and co-coordinator and the CORDIS database. From the 662 citations identified, 185 were included. Through the integration of the data collected from IMI-funded projects, it was verified that for Target and Biomarker identification, the main achievements were in order to 1) identify and validate biological markers, tools and assays, 2) identify determinants of inter-individual variability, 3) understand the molecular mechanisms underlying the disease, 4) develop a platform of pre-clinical assays, and 5) develop systems’ models. Therefore, several biomarkers, tools, inter-individual variability factors, including genetic markers, and relevant pathways were proposed for type 1 diabetes by INNODIA, for type 2 diabetes by SUMMIT, IMIDIA, DIRECT and EMIF, for pancreatic β-cells by IMIDIA and RHAPSODY, for diabetic kidney disease by SUMMIT and BEAT-DKD, and for cardiovascular diseases and diabetic retinopathy by SUMMIT. Moreover, new tools and assays to improve research field were developed by StemBANCC, EBiSC and IMIDIA. Also, two models for patients’ stratification were proposed, one related to glycaemic control in patients with type 1 diabetes established by INNODIA, and another corresponding to the identification of subtypes of diabetes patients developed by BEAT-DKD/RHAPSODY. Regarding the clinical trials, the data collected SUMMIT, DIRECT and BEAT-DKD corresponds to new clinical endpoints and trial designs to accurately reflect the characteristics of the diabetic subpopulation under test. In terms of innovative medicines, information retrieved by SUMMIT, IMIDIA, DIRECT, StemBANNC, EMIF, INNODIA and BEAT-DKD consists on the identification of new therapeutic targets and the development of agents with the purpose of treatment and prevent diabetes and its related complications. Furthermore, a new approach for the large-scale production of human pluripotent stem cells was proposed by StemBANCC. Concerning the maximization of beneficial health patient-centred outcomes, two novel predictive models were developed and validated by DIRECT for diabetes to be used as screening tools by doctors. In addition, this dissertation intends to present a joint vision of the IMI-projects with strategies for integrating personalized medicine into healthcare practice. This approach involves the creation of biological and genetic indicators that can be used to identify individuals at high risk of developing diabetes, the adoption of tools that allow early diagnosis and, lastly, the selection of appropriate treatment, i.e. the safest and most effective, supported by patient stratification models, in order to prevent/delay the development of diabetic complications

Proceedings, MSVSCC 2018

Proceedings of the 12th Annual Modeling, Simulation & Visualization Student Capstone Conference held on April 19, 2018 at VMASC in Suffolk, Virginia. 155 pp

X-Machines for Agent-Based Modeling

This book discusses various aspects of agent-based modeling and simulation using FLAME (Flexible Large-scale Agent-Based Modeling Environment) which is a popular agent-based modeling environment that enables automatic parallelization of models. Along with a focus on the software engineering principles in building agent-based models, the book comprehensively discusses how models can be written for various domains including biology, economics and social networks. The book also includes examples to guide readers on how to write their own models