ViZDoom Competitions: Playing Doom from Pixels

This paper presents the first two editions of Visual Doom AI Competition, held in 2016 and 2017. The challenge was to create bots that compete in a multi-player deathmatch in a first-person shooter (FPS) game, Doom. The bots had to make their decisions based solely on visual information, i.e., a raw screen buffer. To play well, the bots needed to understand their surroundings, navigate, explore, and handle the opponents at the same time. These aspects, together with the competitive multi-agent aspect of the game, make the competition a unique platform for evaluating the state of the art reinforcement learning algorithms. The paper discusses the rules, solutions, results, and statistics that give insight into the agents' behaviors. Best-performing agents are described in more detail. The results of the competition lead to the conclusion that, although reinforcement learning can produce capable Doom bots, they still are not yet able to successfully compete against humans in this game. The paper also revisits the ViZDoom environment, which is a flexible, easy to use, and efficient 3D platform for research for vision-based reinforcement learning, based on a well-recognized first-person perspective game Doom

Optimal algorithmic trading and market microstructure

The efficient frontier is a core concept in Modern Portfolio Theory. Based on this idea, we will construct optimal trading curves for different types of portfolios. These curves correspond to the algorithmic trading strategies that minimize the expected transaction costs, i.e. the joint effect of market impact and market risk. We will study five portfolio trading strategies. For the first three (single-asset, general multi-asseet and balanced portfolios) we will assume that the underlyings follow a Gaussian diffusion, whereas for the last two portfolios we will suppose that there exists a combination of assets such that the corresponding portfolio follows a mean-reverting dynamics. The optimal trading curves can be computed by solving an N-dimensional optimization problem, where N is the (pre-determined) number of trading times. We will solve the recursive algorithm using the "shooting method", a numerical technique for differential equations. This method has the advantage that its corresponding equation is always one-dimensional regardless of the number of trading times N. This novel approach could be appealing for high-frequency traders and electronic brokers.quantitative finance; optimal trading; algorithmic trading; systematic trading; market microstructure

A TRAINING AND FEEDBACK SYSTEM FOR ARCHERS

INTRODUCTION Archery is a skill that demands utmost precision and repeatability of the motion. Minute deviations from the individual pattern are often below the JND threshold. Nevertheless they influence the score. More specifically, archers display radial or ulnar abduction of the bow-hand while aiming. This leads to changes in the energy stored in the upper and lower bow-limbs and thus to vertical deviations from the target. Also, a tremor of the drawing hand may occur. Furthermore, some archers tend to slowly rotate the bow about an axis parallel to the arrow. Since the arrow is well below the line of sight, lateral deviations occur even when the point of aim is dead on. While beginners have rather large and random inconsistencies, skilled archers have intermittent problems. The ability to control and reproduce the wrist position and the tilt of the bow is performance relevant. Small deviations may not be observable by the coach. Hence there is potential for a dedicated 'archery measurement' system that delivers fast and objective supplementary information for coaches and archers. The aim was to develop a precise and 'easy to use' tool to provide feedback. DESIGN AND FEATURES OF THE SYSTEM A HQY'T competition bow was instrumented with strain gauges above and below the bowgrip. The signals are pre-amplified and fed to an amplifier with gain and offset adjustment to accommodate bow-limbs of different stiffness. The difference between the forces F(a) and F(b) is supplied. The tilt of the bow is measured with a silicon damped inclinometer. The analog data are digitally converted using a National Instruments PC-LPM-16 board. Based on the LAB VIEW0 software an application was programmed to fulfill all user requirements regarding the data acquisition and processing as well as ease of operation and stability. In addition to the forces F1, AF2, F and the tilt angle a, two channels may be user defined to record for example the M. pectoralis EMG and an accelerometer signal to determine release and clicker times. The measuring module features programmable gain and offset, on-line help, visual data inspection as well as zoom of the display and data storage. During the data acquisition, an sound can be activated at a specified set point and in an adjustable window. This sound represents a real time feedback on whether the predefined levels are reached within the selected range. Different frequencies are used to distinguish 'force' and 'angle' information. Data can be recalled, viewed, compared and exported from a second software module. SYSTEM PERFORMANCE EVALUATION The feedback system uses up to date hardware and software technology. Implementation of bows with different strength and arrow length is possible. The software is flexible enough to accommodate future user requirements. The accuracy and reliability of the system was determined using a force vs. displacement measuring device that pulls the string to draw length without applying a torque to the handle. The system is now used in the training environment of international caliber Archers. RESULTS Archers have individual draw, anchor, aim and release patterns. Success depends on the reproducibility of the motion. Highly skilled archers have excellent inter-shot stability of the measured parameters. Deviations occur intermittently and can be recorded and analyzed with the present system. A long-term study was undertaken to establish the effect of the feedback with decreasing variability on the score for different performance levels

Time since discharge of 9mm cartridges by headspace analysis, part 2: Ageing study and estimation of the time since discharge using multivariate regression.

Estimating the time since discharge of spent cartridges can be a valuable tool in the forensic investigation of firearm-related crimes. To reach this aim, it was previously proposed that the decrease of volatile organic compounds released during discharge is monitored over time using non-destructive headspace extraction techniques. While promising results were obtained for large-calibre cartridges (e.g., shotgun shells), handgun calibres yielded unsatisfying results. In addition to the natural complexity of the specimen itself, these can also be attributed to some selective choices in the methods development. Thus, the present series of papers aimed to systematically evaluate the potential of headspace analysis to estimate the time since discharge of cartridges through the use of more comprehensive analytical and interpretative techniques. Following the comprehensive optimisation and validation of an exhaustive headspace sorptive extraction (HSSE) method in the first part of this work, the present paper addresses the application of chemometric tools in order to systematically evaluate the potential of applying headspace analysis to estimate the time since discharge of 9mm Geco cartridges. Several multivariate regression and pre-treatment methods were tested and compared to univariate models based on non-linear regression. Random forests (RF) and partial least squares (PLS) proceeded by pairwise log-ratios normalisation (PLR) showed the best results, and allowed to estimate time since discharge up to 48h of ageing and to differentiate recently fired from older cartridges (e.g., less than 5h compared to more than 1-2 days). The proposed multivariate approaches showed significant improvement compared to univariate models. The effects of storage conditions were also tested and results demonstrated that temperature, humidity and cartridge position should be taken into account when estimating the time since discharge

Reallocating resources to focused factories: a case study in chemotherapy

This study investigates the expected service performance associated with a proposal to reallocate resources from a centralized chemotherapy department to a breast cancer focused factory. Using a slotted queueing model we show that a decrease in performance is expected and calculate the amount of additional resources required to offset these losses. The model relies solely on typical outpatient scheduling system data, making the methodology easy to replicate in other outpatient clinic settings. Finally, the paper highlights important factors to consider when assigning capacity to focused factories. These considerations are generally relevant to other resource allocation decisions

Reallocating resources to focused factories: a case study in chemotherapy

This study investigates the expected service performance associated with a proposal to reallocate resources from a centralized chemotherapy department to a breast cancer focused factory. Using a slotted queueing model we show that a decrease in performance is expected and calculate the amount of additional resources required to offset these losses. The model relies solely on typical outpatient scheduling system data, making the methodology easy to replicate in other outpatient clinic settings. Finally, the paper highlights important factors to consider when assigning capacity to focused factories. These considerations are generally relevant to other resource allocation decisions

Building a Panel Survey on Health, Aging and Retirement in Europe

Ageing is one of the greatest social and economic challenges of the 21st century in Europe. SHARE, a EU-sponsored project that will build up a Survey of Health, Aging and Retirement in Europe, will be a fundamental resource for science and public policy to help mastering this unprecedented challenge. The main aim of SHARE is to create a pan-European interdisciplinary panel data set covering persons aged 50 and over. The project brings together many disciplines, including epidemiology, sociology, statistics, psychology, demography, and economics. Scientists from some 15 countries work on feasibility studies, experiments, and instrument development, culminating in a survey of about 22.000 individuals. The multidisciplinary nature of the data will provide new insights in the complex interactions between economic, health, psychological and social factors determining the quality of life of the elderly.

Teaching Modeling to Engineers in an Undergraduate Simulation Course

A significant challenge in teaching simulation to undergraduate students is to find a way to allow them to model a real world referent system within time and student skill constraints. Several research sources highlight not only the important challenge of model development (Garcia and Ceneno, 2009, Tako, 2011) but also the increased need for model development instruction among engineers (Grasas et. al., 2013, Saltzman and Roeder, 2013). One approach to this challenge is to use a general purpose discrete event simulation software package within the course, but this presents two challenges. Teaching the package to the students takes significant time, and the package introduces limitations which may restrict their ability to model certain real-world referents, particularly in the engineering domain. A conceptual approach to solving this problem is to use a model development paradigm that abstracts away the interface to the simulation infrastructure while still allowing the students to use the full expressive nature of a programming language. Two undergraduate courses at the United States Military Academy employed this strategy via the Discrete Events Specification System – Distributed Modeling Framework (DEVS-DMF) (Kewley et. al, 2016). The DEVS abstraction allowed students to think about their model as a simple state change function with defined inputs and outputs, and DMF allowed them to program in a cloud-based Jupyter Notebook using the Python language. Students in a combat modeling course employed a variety of models to understand drone jamming, and students in an engineering capstone project employed models to account for human factors in rifle marksmanship. The effectiveness of this approach was assessed through student grades, exit-interviews, and course-end surveys. These assessments showed an increased understanding of the model development process, and students also reported greater ownership of their models. However, this experiment also highlighted some weaknesses in their understanding of underlying methodologies and programming skills

Application of Fuzzy State Aggregation and Policy Hill Climbing to Multi-Agent Systems in Stochastic Environments

Reinforcement learning is one of the more attractive machine learning technologies, due to its unsupervised learning structure and ability to continually even as the operating environment changes. Applying this learning to multiple cooperative software agents (a multi-agent system) not only allows each individual agent to learn from its own experience, but also opens up the opportunity for the individual agents to learn from the other agents in the system, thus accelerating the rate of learning. This research presents the novel use of fuzzy state aggregation, as the means of function approximation, combined with the policy hill climbing methods of Win or Lose Fast (WoLF) and policy-dynamics based WoLF (PD-WoLF). The combination of fast policy hill climbing (PHC) and fuzzy state aggregation (FSA) function approximation is tested in two stochastic environments; Tileworld and the robot soccer domain, RoboCup. The Tileworld results demonstrate that a single agent using the combination of FSA and PHC learns quicker and performs better than combined fuzzy state aggregation and Q-learning lone. Results from the RoboCup domain again illustrate that the policy hill climbing algorithms perform better than Q-learning alone in a multi-agent environment. The learning is further enhanced by allowing the agents to share their experience through a weighted strategy sharing