Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning

Recent advances in combining deep neural network architectures with reinforcement learning techniques have shown promising potential results in solving complex control problems with high dimensional state and action spaces. Inspired by these successes, in this paper, we build two kinds of reinforcement learning algorithms: deep policy-gradient and value-function based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The policy-gradient based agent maps its observation directly to the control signal, however the value-function based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Our methods show promising results in a traffic network simulated in the SUMO traffic simulator, without suffering from instability issues during the training process

Queue Length Simulations in a Finite Single-line Queueing System with Repeated Calls

2000 Mathematics Subject Classification: 60K25.Simulated results about the queue length and the server state in a finite single server queuing system with repeated calls are presented. Formulas for the basic probability characteristics of the corresponding distributions are obtained in previous papers of the author. The numerical values computed according to these formulas are compared with the simulated results. Empirical mean values of the idle period are obtained as well

Optimized R functions for analysis of ecological community data using the R virtual laboratory (RvLab)

Background: Parallel data manipulation using R has previously been addressed by members of the R community, however most of these studies produce ad hoc solutions that are not readily available to the average R user. Our targeted users, ranging from the expert ecologist/microbiologists to computational biologists, often experience difficulties in finding optimal ways to exploit the full capacity of their computational resources. In addition, improving performance of commonly used R scripts becomes increasingly difficult especially with large datasets. Furthermore, the implementations described here can be of significant interest to expert bioinformaticians or R developers. Therefore, our goals can be summarized as: (i) description of a complete methodology for the analysis of large datasets by combining capabilities of diverse R packages, (ii) presentation of their application through a virtual R laboratory (RvLab) that makes execution of complex functions and visualization of results easy and readily available to the end-user. New information: In this paper, the novelty stems from implementations of parallel methodologies which rely on the processing of data on different levels of abstraction and the availability of these processes through an integrated portal. Parallel implementation R packages, such as the pbdMPI (Programming with Big Data – Interface to MPI) package, are used to implement Single Program Multiple Data (SPMD) parallelization on primitive mathematical operations, allowing for interplay with functions of the vegan package. The dplyr and RPostgreSQL R packages are further integrated offering connections to dataframe like objects (databases) as secondary storage solutions whenever memory demands exceed available RAM resources. The RvLab is running on a PC cluster, using version 3.1.2 (2014-10-31) on a x86_64-pc-linux-gnu (64-bit) platform, and offers an intuitive virtual environmet interface enabling users to perform analysis of ecological and microbial communities based on optimized vegan functions. A beta version of the RvLab is available after registration at: https://portal.lifewatchgreece.eu

Standard and retrial queueing systems: a comparative analysis

We describe main models and results of a new branch of the queueing theory, theory of retrial queues, which is characterized by the following basic assumption: a customer who cannot get service (due to finite capacity of the system, balking, impatience, etc.)leaves the service area, but after some random delay returns to the system again. Emphasis is done on comparison with standard queues with waiting line and queues with losses. We give a survey of main results for both single server M/G/1 type and multiserver M/M/c type retrial queues and discuss similarities and differences between the retrial queues and their standard counterparts. We demonstrate that although retrial queues are closely connected with these standard queueing models they, however, ossess unique distinguished features. We also mention some open problems.We describe main models and results of a new branch of the queueing theory, theory of retrial queues, which is characterized by the following basic assumption: a customer who cannot get service (due to finite capacity of the system, balking, impatience, etc.)leaves the service area, but after some random delay returns to the system again. Emphasis is done on comparison with standard queues with waiting line and queues with losses. We give a survey of main results for both single server M/G/1 type and multiserver M/M/c type retrial queues and discuss similarities and differences between the retrial queues and their standard counterparts. We demonstrate that although retrial queues are closely connected with these standard queueing models they, however, ossess unique distinguished features. We also mention some open problems

Leveraging Multi-Perspective A priori Knowledge in Predictive Business Process Monitoring

Äriprotsesside ennestusseire on valdkond, mis on pühendunud käimasolevate äriprotsesside tuleviku ennustamisele kasutades selleks minevikus sooritatud äriprotsesside kohta käivaid andmeid. Valdav osa uurimustööst selles valdkonnas keskendub ainult seda tüüpi andmetele, jättes tähelepanuta täiendavad teadmised (a priori teadmised) protsessi teostumise kohta tulevikus. Hiljuti pakuti välja lähenemine, mis võimaldab a priori teadmisi kasutada LTL-reeglite näol. Kuid tõsiasjana on antud tehnika limiteeritud äriprotsessi kontroll-voole, jättes välja võimaluse väljendada a priori teadmisi, mis puudutavad lisaks kontrollvoole ka informatsiooni protsessis leiduvate atribuutide kohta (multiperspektiivsed a priori teadmised). Me pakume välja lahenduse, mis võimaldab seda tüüpi teadmiste kasutuse, tehes multiperspektiivseid ennustusi käimasoleva äriprotsessi kohta. Tulemused, milleni jõuti rakendades väljapakutud tehnikat 20-le tehisärilogile ning ühele elulisele ärilogile, näitavad, et meie lähenemine suudab pakkuda konkurentsivõimelisi ennustusi.Predictive business process monitoring is an area dedicated to exploiting past process execution data in order to predict the future unfolding of a currently executed business process instance. Most of the research done in this domain focuses on exploiting the past process execution data only, leaving neglected additional a priori knowledge that might become available at runtime. Recently, an approach was proposed, which allows to leverage a priori knowledge on the control flow in the form of LTL-rules. However, cases exist in which more granular a priori knowledge becomes available about perspectives that go be-yond the pure control flow like data, time and resources (multiperspective a priori knowledge). In this thesis, we propose a technique that enables to leverage multi-perspective a priori knowledge when making predictions of complex sequences, i.e., sequences of events with a subset of the data attributes attached to them. The results, obtained by applying the proposed technique to 20 synthetic logs and 1 real life log, show that the proposed technique is able to overcome state-of-the-art approaches by successfully leveraging multiperspective a priori knowledge

Scheduling a Make-To-Stock Queue: Index Policies and Hedging Points

A single machine produces several different classes of items in a make-to-stock mode. We consider the problem of scheduling the machine to regulate finished goods inventory, minimizing holding and backorder or holding and lost sales costs. Demands are Poisson, service times are exponentially distributed, and there are no delays or costs associated with switching products. A scheduling policy dictates whether the machine is idle or busy, and specifies the job class to serve in the latter case. Since the optimal solution can only be numerically computed for problems with several products, our goal is to develop effective policies that are computationally tractable for a large number of products. We develop index policies to decide which class to serve, including Whittle's "restless bandit" index, which possesses a certain asymptotic optimality. Several idleness policies, which are characterized by hedging points, are derived, and the best policy is obtained from a heavy traffic diffusion approximation. Nine sample problems are considered in a numerical study, and the average suboptimality of the best policy is less than 3%