8,863 research outputs found

    Model Diagnostics meets Forecast Evaluation: Goodness-of-Fit, Calibration, and Related Topics

    Get PDF
    Principled forecast evaluation and model diagnostics are vital in fitting probabilistic models and forecasting outcomes of interest. A common principle is that fitted or predicted distributions ought to be calibrated, ideally in the sense that the outcome is indistinguishable from a random draw from the posited distribution. Much of this thesis is centered on calibration properties of various types of forecasts. In the first part of the thesis, a simple algorithm for exact multinomial goodness-of-fit tests is proposed. The algorithm computes exact pp-values based on various test statistics, such as the log-likelihood ratio and Pearson\u27s chi-square. A thorough analysis shows improvement on extant methods. However, the runtime of the algorithm grows exponentially in the number of categories and hence its use is limited. In the second part, a framework rooted in probability theory is developed, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. Based on a general notion of conditional T-calibration, the thesis introduces population versions of T-reliability diagrams and revisits a score decomposition into measures of miscalibration, discrimination, and uncertainty. Stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, a universal coefficient of determination is introduced that nests and reinterprets the classical R2R^2 in least squares regression. In the third part, probabilistic top lists are proposed as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The probabilistic top list functional is elicited by strictly consistent evaluation metrics, based on symmetric proper scoring rules, which admit comparison of various types of predictions

    TkT: Automatic Inference of Timed and Extended Pushdown Automata

    Get PDF
    To mitigate the cost of manually producing and maintaining models capturing software specifications, specification mining techniques can be exploited to automatically derive up-to-date models that faithfully represent the behavior of software systems. So far, specification mining solutions focused on extracting information about the functional behavior of the system, especially in the form of models that represent the ordering of the operations. Well-known examples are finite state models capturing the usage protocol of software interfaces and temporal rules specifying relations among system events. Although the functional behavior of a software system is a primary aspect of concern, there are several other non-functional characteristics that must be typically addressed jointly with the functional behavior of a software system. Efficiency is one of the most relevant characteristics. In fact, an application delivering the right functionalities inefficiently has a big chance to not satisfy the expectation of its users. Interestingly, the timing behavior is strongly dependent on the functional behavior of a software system. For instance, the timing of an operation depends on the functional complexity and size of the computation that is performed. Consequently, models that combine the functional and timing behaviors, as well as their dependencies, are extremely important to precisely reason on the behavior of software systems. In this paper, we address the challenge of generating models that capture both the functional and timing behavior of a software system from execution traces. The result is the Timed k-Tail (TkT) specification mining technique, which can mine finite state models that capture such an interplay: the functional behavior is represented by the possible order of the events accepted by the transitions, while the timing behavior is represented through clocks and clock constraints of different nature associated with transitions. Our empirical evaluation with several libraries and applications show that TkT can generate accurate models, capable of supporting the identification of timing anomalies due to overloaded environment and performance faults. Furthermore, our study shows that TkT outperforms state-of-the-art techniques in terms of scalability and accuracy of the mined models

    How to Be a God

    Get PDF
    When it comes to questions concerning the nature of Reality, Philosophers and Theologians have the answers. Philosophers have the answers that can’t be proven right. Theologians have the answers that can’t be proven wrong. Today’s designers of Massively-Multiplayer Online Role-Playing Games create realities for a living. They can’t spend centuries mulling over the issues: they have to face them head-on. Their practical experiences can indicate which theoretical proposals actually work in practice. That’s today’s designers. Tomorrow’s will have a whole new set of questions to answer. The designers of virtual worlds are the literal gods of those realities. Suppose Artificial Intelligence comes through and allows us to create non-player characters as smart as us. What are our responsibilities as gods? How should we, as gods, conduct ourselves? How should we be gods

    Graphical scaffolding for the learning of data wrangling APIs

    Get PDF
    In order for students across the sciences to avail themselves of modern data streams, they must first know how to wrangle data: how to reshape ill-organised, tabular data into another format, and how to do this programmatically, in languages such as Python and R. Despite the cross-departmental demand and the ubiquity of data wrangling in analytical workflows, the research on how to optimise the instruction of it has been minimal. Although data wrangling as a programming domain presents distinctive challenges - characterised by on-the-fly syntax lookup and code example integration - it also presents opportunities. One such opportunity is how tabular data structures are easily visualised. To leverage the inherent visualisability of data wrangling, this dissertation evaluates three types of graphics that could be employed as scaffolding for novices: subgoal graphics, thumbnail graphics, and parameter graphics. Using a specially built e-learning platform, this dissertation documents a multi-institutional, randomised, and controlled experiment that investigates the pedagogical effects of these. Our results indicate that the graphics are well-received, that subgoal graphics boost the completion rate, and that thumbnail graphics improve navigability within a command menu. We also obtained several non-significant results, and indications that parameter graphics are counter-productive. We will discuss these findings in the context of general scaffolding dilemmas, and how they fit into a wider research programme on data wrangling instruction

    Investigative Methods: An NCRM Innovation Collection

    Get PDF
    This Innovation Collection on investigative methods brings together investigators working in different domains, sectors, and on different topics of interest to help capture the breadth, scope and relevance of investigative practices over 10 substantive chapters. Each of the papers presents a different investigative method or set of methods and, through case studies, attempts to demonstrate their value. All the contributions, in different ways and for different purposes, seek to reconstruct acts, events, practices, biographies and/or milieux, to which the researchers in question lack direct access, but which they want to reconstruct via the traces those phenomena leave behind, traces themselves often produced as part of the phenomena under investigation. These include reports of methods used in investigations on: - The use of force by state actors, including into police violence, military decisions to attack civilians, the provenance of munitions used to attack civilians, and the use and abuse of tear gas; - Networks of far-right discourse, and its links to criminal attacks and state-leveraged misinformation campaigns; - Archives to establish the penal biographies of convicts and the historical practices of democratic petitioning; - Corporate structures and processes that enable tax avoidance and an avoidance of legal responsibilities to workers and the environment. A working principle of the collection is that investigative methods may be considered, alongside creative, qualitative, quantitative, digital, participatory and mixed methods, a distinct yet complementary style of research

    Secure authentication and key agreement via abstract multi-agent interaction

    Get PDF
    Authentication and key agreement are the foundation for secure communication over the Internet. Authenticated Key Exchange (AKE) protocols provide methods for communicating parties to authenticate each other, and establish a shared session key by which they can encrypt messages in the session. Within the category of AKE protocols, symmetric AKE protocols rely on pre-shared master keys for both services. These master keys can be transformed after each session in a key-evolving scheme to provide the property of forward secrecy, whereby the compromise of master keys does not allow for the compromise of past session keys. This thesis contributes a symmetric AKE protocol named AMI (Authentication via Multi-Agent Interaction). The AMI protocol is a novel formulation of authentication and key agreement as a multi-agent system, where communicating parties are treated as autonomous agents whose behavior within the protocol is governed by private agent models used as the master keys. Parties interact repeatedly using their behavioral models for authentication and for agreeing upon a unique session key per communication session. These models are evolved after each session to provide forward secrecy. The security of the multi-agent interaction process rests upon the difficulty of modeling an agent's decisions from limited observations about its behavior, a long-standing problem in AI research known as opponent modeling. We conjecture that it is difficult to efficiently solve even by a quantum computer, since the problem is fundamentally one of missing information rather than computational hardness. We show empirically that the AMI protocol achieves high accuracy in correctly identifying legitimate agents while rejecting different adversarial strategies from the security literature. We demonstrate the protocol's resistance to adversarial agents which utilize random, replay, and maximum-likelihood estimation (MLE) strategies to bypass the authentication test. The random strategy chooses actions randomly without attempting to mimic a legitimate agent. The replay strategy replays actions previously observed by a legitimate client. The MLE strategy estimates a legitimate agent model using previously observed interactions, as an attempt to solve the opponent modeling problem. This thesis also introduces a reinforcement learning approach for efficient multi-agent interaction and authentication. This method trains an authenticating server agent's decision model to take effective probing actions which decrease the number of interactions in a single session required to successfully reject adversarial agents. We empirically evaluate the number of interactions required for a trained server agent to reject an adversarial agent, and show that using the optimized server leads to a much more sample-efficient interaction process than a server agent selecting actions by a uniform-random behavioral policy. Towards further research on and adoption of the AMI protocol for authenticated key-exchange, this thesis also contributes an open-source application written in Python, PyAMI. PyAMI consists of a multi-agent system where agents run on separate virtual machines, and communicate over low-level network sockets using TCP. The application supports extending the basic client-server setting to a larger multi-agent system for group authentication and key agreement, providing two such architectures for different deployment scenarios

    The stuff of strategy: the potential of the material turn in strategy studies

    Get PDF
    This thesis explores the potential of the material turn in strategy studies to explore how nonhuman ‘things’ contribute to strategy production. Drawing on the ontologies and methodologies of the strategy-as-practice and actor-network theory domains, the empirical research informing this thesis is an immersive mixed methods ethnographic study in a higher education school of art and design conducted over a period of 24 months. The study combines observation and a qualitative interview protocol to build four explorative case study narratives that consider various aspects of material agency in strategy production. Analysis and discussion inform a re-theorising of strategy production that foregrounds the agency of materials beyond that of human intent, providing a counterpoint to prevailing approaches that centre the affordances ‘things’ offer to human action and suggesting instead a novel extension to strategy studies that emphasises emancipatory critique of normative organisational practices and ontologies

    At the intersection between machine learning and econometrics: theory and applications

    Get PDF
    In the present work, we introduce theoretical and application novelties at the intersection between machine learning and econometrics in social and health sciences. In particular, Part 1 delves into optimizing the data collection process in a specific statistical model, commonly used in econometrics, employing an optimization criterion inspired by machine learning, namely, the generalization error conditioned on the training input data. In the first Chapter, we analyze and optimize the trade-off between sample size, the precision of supervision on a variation of the unbalanced fixed effects panel data model. In the second Chapter we extend the analysis to the Fixed Effects GLS (FEGLS) case in order to account for the heterogeneity in the data associated with different units, for which correlated measurement errors corrupt distinct observations related to the same unit. In Part 2, we introduce\ud applications of innovative econometrics and machine learning techniques. In the third Chapter we propose a novel methodology to explore the effect of market size on market innovation in the Pharmaceutical industry. Finally, in the fourth Chapter, we innovate the literature on the economic complexity of countries through machine learning. The Dissertation contributes to the literature on machine learning and applied econometrics mainly by: (i) extending the current framework to novel scenarios and applications (Chapter 1 - Chapter 2); (ii) developing a novel econometric methodology to assess long-debated issues in literature (Chapter 3); (iii) constructing a novel index of economic complexity through machine learning (Chapter 4)

    Flexographic printed nanogranular LBZA derived ZnO gas sensors: Synthesis, printing and processing

    Get PDF
    Within this document, investigations of the processes towards the production of a flexographic printed ZnO gas sensor for breath H2 analysis are presented. Initially, a hexamethylenetetramine (HMTA) based, microwave assisted, synthesis method of layered basic zinc acetate (LBZA) nanomaterials was investigated. Using the synthesised LBZA, a dropcast nanogranular ZnO gas sensor was produced. The testing of the sensor showed high sensitivity towards hydrogen with response (Resistanceair/ Resistancegas) to 200 ppm H2 at 328 °C of 7.27. The sensor is highly competitive with non-catalyst surface decorated sensors and sensitive enough to measure current H2 guideline thresholds for carbohydrate malabsorption (Positive test threshold: 20 ppm H2, Predicted response: 1.34). Secondly, a novel LBZA synthesis method was developed, replacing the HMTA by NaOH. This resulted in a large yield improvement, from a [OH-] conversion of 4.08 at% to 71.2 at%. The effects of [OH-]/[Zn2+] ratio, microwave exposure and transport to nucleation rate ratio on purity, length, aspect ratio and polydispersity were investigated in detail. Using classical nucleation theory, analysis of the basal layer charge symmetries, and oriented attachment theory, a dipole-oriented attachment reaction mechanism is presented. The mechanism is the first theory in literature capable of describing all observed morphological features along length scales. The importance of transport to nucleation rate ratio as the defining property that controls purity and polydispersity is then shown. Using the NaOH derived LBZA, a flexographic printing ink was developed, and proof-of-concept sensors printed. Gas sensing results showed a high response to 200 ppm H2 at 300 °C of 60.2. Through IV measurements and SEM analysis this was shown to be a result of transfer of silver between the electrode and the sensing layer during the printing process. Finally, Investigations into the intense pulsed light treatment of LBZA were conducted. The results show that dehydration at 150 °C prior to exposure is a requirement for successful calcination, producing ZnO quantum dots (QDs) in the process. SEM measurements show mean radii of 1.77-2.02 nm. The QDs show size confinement effects with the exciton blue shifting by 0.105 eV, and exceptionally low defect emission in photoluminescence spectra, indicative of high crystalline quality, and high conductivity. Due to the high crystalline quality and amenity to printing, the IPL ZnO QDs have numerous potential uses ranging from sensing to opto-electronic devices
    • …
    corecore