8,863 research outputs found
Model Diagnostics meets Forecast Evaluation: Goodness-of-Fit, Calibration, and Related Topics
Principled forecast evaluation and model diagnostics are vital in fitting probabilistic models and forecasting outcomes of interest. A common principle is that fitted or predicted distributions ought to be calibrated, ideally in the sense that the outcome is indistinguishable from a random draw from the posited distribution. Much of this thesis is centered on calibration properties of various types of forecasts.
In the first part of the thesis, a simple algorithm for exact multinomial goodness-of-fit tests is proposed. The algorithm computes exact -values based on various test statistics, such as the log-likelihood ratio and Pearson\u27s chi-square. A thorough analysis shows improvement on extant methods. However, the runtime of the algorithm grows exponentially in the number of categories and hence its use is limited.
In the second part, a framework rooted in probability theory is developed, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. Based on a general notion of conditional T-calibration, the thesis introduces population versions of T-reliability diagrams and revisits a score decomposition into measures of miscalibration, discrimination, and uncertainty. Stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, a universal coefficient of determination is introduced that nests and reinterprets the classical in least squares regression.
In the third part, probabilistic top lists are proposed as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The probabilistic top list functional is elicited by strictly consistent evaluation metrics, based on symmetric proper scoring rules, which admit comparison of various types of predictions
Recommended from our members
Co-design As Healing: Exploring The Experiences Of Participants Facing Mental Health Problems
This thesis is an exploration of the healing role of co-design in mental health. Although co-design projects conducted within mental health settings are rising, existing literature tends to focus on the object of design and its outcomes while the experiences of participants per se remain largely unexplored. The guiding research question of this study is not how we design things that improve mental health, but how co-designing, as an act, might do so.
The thesis presents two projects that were organized in collaboration with the mental health charity Islington Mind and the Psychosis Therapy Project (PTP) in London.
The project at Islington Mind used a structured design process inviting participants to design for wellbeing. A case study analysis provides insights on how participants were impacted, summarizing key challenges and opportunities.
The design at PTP worked towards creating a collective brief in an emergent fashion, finally culminating in a board game. The experiences of participants were explored through Interpretative Phenomenological Analysis (IPA), using semi-structured interview data. The analysis served to identify key themes characterising the experience of co-design such as contributing, connecting, thinking and intentioning. In addition, a mixed-methods analysis of questionnaires and interview data exploring participants' wellbeing, showed that all participants who engaged fairly consistently in the project improved after the project ended, although some participants' scores returned to baseline six months later.
Reflecting on both projects, an approach to facilitation within mental health is outlined, detailing how the dimensions of weaving and layered participation, nurturing mattering and facilitating attitudes interlace. This contribution raises awareness of tacit dimensions in the practice of facilitation, articulating the nuances of how to encourage and sustain meaningful and ethical engagement and offering insights into a range of tools. It highlights the importance of remaining reflexive in relation to attitudes and emotions and discusses practical methodological and ethical challenges and ways to resolve them which can be of benefit to researchers embarking on a similar journey.
The thesis also offers detailed insights on how methodologies from different fields were integrated into a whole, arguing for transparency and reflexivity about epistemological assumptions, and how underlying paradigms shift in an interdisciplinary context.
Based on the overall findings, the thesis makes a case for considering design as healing (or a designerly way of healing), highlighting implications at a systems, social and individual level. It makes an original contribution to our understanding of design, highlighting its healing character, and proposes a new way to support mental health. The participants in this study not only had increased their own wellbeing through co-designing, but were also empowered and contributed towards healing the world. Hence, the thesis argues for a unique, holistic perspective of design and mental health, recognizing the interconnectedness of the individual, social and systemic dimensions of the healing processes that are ignited
TkT: Automatic Inference of Timed and Extended Pushdown Automata
To mitigate the cost of manually producing and maintaining models capturing software specifications, specification mining techniques can be exploited to automatically derive up-to-date models that faithfully represent the behavior of software systems. So far, specification mining solutions focused on extracting information about the functional behavior of the system, especially in the form of models that represent the ordering of the operations. Well-known examples are finite state models capturing the usage protocol of software interfaces and temporal rules specifying relations among system events. Although the functional behavior of a software system is a primary aspect of concern, there are several other non-functional characteristics that must be typically addressed jointly with the functional behavior of a software system. Efficiency is one of the most relevant characteristics. In fact, an application delivering the right functionalities inefficiently has a big chance to not satisfy the expectation of its users. Interestingly, the timing behavior is strongly dependent on the functional behavior of a software system. For instance, the timing of an operation depends on the functional complexity and size of the computation that is performed. Consequently, models that combine the functional and timing behaviors, as well as their dependencies, are extremely important to precisely reason on the behavior of software systems. In this paper, we address the challenge of generating models that capture both the functional and timing behavior of a software system from execution traces. The result is the Timed k-Tail (TkT) specification mining technique, which can mine finite state models that capture such an interplay: the functional behavior is represented by the possible order of the events accepted by the transitions, while the timing behavior is represented through clocks and clock constraints of different nature associated with transitions. Our empirical evaluation with several libraries and applications show that TkT can generate accurate models, capable of supporting the identification of timing anomalies due to overloaded environment and performance faults. Furthermore, our study shows that TkT outperforms state-of-the-art techniques in terms of scalability and accuracy of the mined models
How to Be a God
When it comes to questions concerning the nature of Reality, Philosophers and Theologians have the answers.
Philosophers have the answers that can’t be proven right. Theologians have the answers that can’t be proven wrong.
Today’s designers of Massively-Multiplayer Online Role-Playing Games create realities for a living. They can’t spend centuries mulling over the issues: they have to face them head-on. Their practical experiences can indicate which theoretical proposals actually work in practice.
That’s today’s designers. Tomorrow’s will have a whole new set of questions to answer.
The designers of virtual worlds are the literal gods of those realities. Suppose Artificial Intelligence comes through and allows us to create non-player characters as smart as us. What are our responsibilities as gods? How should we, as gods, conduct ourselves?
How should we be gods
Graphical scaffolding for the learning of data wrangling APIs
In order for students across the sciences to avail themselves of modern data streams, they must first know how to wrangle data: how to reshape ill-organised, tabular data into another format, and how to do this programmatically, in languages such as Python and R. Despite the cross-departmental demand and the ubiquity of data wrangling in analytical workflows, the research on how to optimise the instruction of it has been minimal. Although data wrangling as a programming domain presents distinctive challenges - characterised by on-the-fly syntax lookup and code example integration - it also presents opportunities. One such opportunity is how tabular data structures are easily visualised. To leverage the inherent visualisability of data wrangling, this dissertation evaluates three types of graphics that could be employed as scaffolding for novices: subgoal graphics, thumbnail graphics, and parameter graphics. Using a specially built e-learning platform, this dissertation documents a multi-institutional, randomised, and controlled experiment that investigates the pedagogical effects of these. Our results indicate that the graphics are well-received, that subgoal graphics boost the completion rate, and that thumbnail graphics improve navigability within a command menu. We also obtained several non-significant results, and indications that parameter graphics are counter-productive. We will discuss these findings in the context of general scaffolding dilemmas, and how they fit into a wider research programme on data wrangling instruction
Investigative Methods: An NCRM Innovation Collection
This Innovation Collection on investigative methods brings together investigators working in different domains, sectors, and on different topics of interest to help capture the breadth, scope and relevance of investigative practices over 10 substantive chapters. Each of the papers presents a different investigative method or set of methods and, through case studies, attempts to demonstrate their value. All the contributions, in different ways and for different purposes, seek to reconstruct acts, events, practices, biographies and/or milieux, to which the researchers in question lack direct access, but which they want to reconstruct via the traces those phenomena leave behind, traces themselves often produced as part of the phenomena under investigation. These include reports of methods used in investigations on: - The use of force by state actors, including into police violence, military decisions to attack civilians, the provenance of munitions used to attack civilians, and the use and abuse of tear gas; - Networks of far-right discourse, and its links to criminal attacks and state-leveraged misinformation campaigns; - Archives to establish the penal biographies of convicts and the historical practices of democratic petitioning; - Corporate structures and processes that enable tax avoidance and an avoidance of legal responsibilities to workers and the environment. A working principle of the collection is that investigative methods may be considered, alongside creative, qualitative, quantitative, digital, participatory and mixed methods, a distinct yet complementary style of research
Secure authentication and key agreement via abstract multi-agent interaction
Authentication and key agreement are the foundation for secure communication over the Internet. Authenticated Key Exchange (AKE) protocols provide methods for communicating parties to authenticate each other, and establish a shared session key by which they can encrypt messages in the session. Within the category of AKE protocols, symmetric AKE protocols rely on pre-shared master keys for both services. These master keys can be transformed after each session in a key-evolving scheme to provide the property of forward secrecy, whereby the compromise of master keys does not allow for the compromise of past session keys.
This thesis contributes a symmetric AKE protocol named AMI (Authentication via Multi-Agent Interaction). The AMI protocol is a novel formulation of authentication and key agreement as a multi-agent system, where communicating parties are treated as autonomous agents whose behavior within the protocol is governed by private agent models used as the master keys. Parties interact repeatedly using their behavioral models for authentication and for agreeing upon a unique session key per communication session. These models are evolved after each session to provide forward secrecy. The security of the multi-agent interaction process rests upon the difficulty of modeling an agent's decisions from limited observations about its behavior, a long-standing problem in AI research known as opponent modeling. We conjecture that it is difficult to efficiently solve even by a quantum computer, since the problem is fundamentally one of missing information rather than computational hardness.
We show empirically that the AMI protocol achieves high accuracy in correctly identifying legitimate agents while rejecting different adversarial strategies from the security literature. We demonstrate the protocol's resistance to adversarial agents which utilize random, replay, and maximum-likelihood estimation (MLE) strategies to bypass the authentication test. The random strategy chooses actions randomly without attempting to mimic a legitimate agent. The replay strategy replays actions previously observed by a legitimate client. The MLE strategy estimates a legitimate agent model using previously observed interactions, as an attempt to solve the opponent modeling problem.
This thesis also introduces a reinforcement learning approach for efficient multi-agent interaction and authentication. This method trains an authenticating server agent's decision model to take effective probing actions which decrease the number of interactions in a single session required to successfully reject adversarial agents. We empirically evaluate the number of interactions required for a trained server agent to reject an adversarial agent, and show that using the optimized server leads to a much more sample-efficient interaction process than a server agent selecting actions by a uniform-random behavioral policy.
Towards further research on and adoption of the AMI protocol for authenticated key-exchange, this thesis also contributes an open-source application written in Python, PyAMI. PyAMI consists of a multi-agent system where agents run on separate virtual machines, and communicate over low-level network sockets using TCP. The application supports extending the basic client-server setting to a larger multi-agent system for group authentication and key agreement, providing two such architectures for different deployment scenarios
The stuff of strategy: the potential of the material turn in strategy studies
This thesis explores the potential of the material turn in strategy studies to explore how nonhuman ‘things’ contribute to strategy production. Drawing on the ontologies and methodologies of the strategy-as-practice and actor-network theory domains, the empirical research informing this thesis is an immersive mixed methods ethnographic study in a higher education school of art and design conducted over a period of 24 months. The study combines observation and a qualitative interview protocol to build four explorative case study narratives that consider various aspects of material agency in strategy production. Analysis and discussion inform a re-theorising of strategy production that foregrounds the agency of materials beyond that of human intent, providing a counterpoint to prevailing approaches that centre the affordances ‘things’ offer to human action and suggesting instead a novel extension to strategy studies that emphasises emancipatory critique of normative organisational practices and ontologies
At the intersection between machine learning and econometrics: theory and applications
In the present work, we introduce theoretical and application novelties at
the intersection between machine learning and econometrics in social and
health sciences. In particular, Part 1 delves into optimizing the data collection
process in a specific statistical model, commonly used in econometrics,
employing an optimization criterion inspired by machine learning, namely,
the generalization error conditioned on the training input data. In the first
Chapter, we analyze and optimize the trade-off between sample size, the precision
of supervision on a variation of the unbalanced fixed effects panel data
model. In the second Chapter we extend the analysis to the Fixed Effects
GLS (FEGLS) case in order to account for the heterogeneity in the data
associated with different units, for which correlated measurement errors corrupt
distinct observations related to the same unit. In Part 2, we introduce\ud
applications of innovative econometrics and machine learning techniques. In
the third Chapter we propose a novel methodology to explore the effect of
market size on market innovation in the Pharmaceutical industry. Finally, in
the fourth Chapter, we innovate the literature on the economic complexity
of countries through machine learning. The Dissertation contributes to the
literature on machine learning and applied econometrics mainly by: (i) extending
the current framework to novel scenarios and applications (Chapter
1 - Chapter 2); (ii) developing a novel econometric methodology to assess
long-debated issues in literature (Chapter 3); (iii) constructing a novel index
of economic complexity through machine learning (Chapter 4)
Flexographic printed nanogranular LBZA derived ZnO gas sensors: Synthesis, printing and processing
Within this document, investigations of the processes towards the production of a flexographic printed ZnO gas sensor for breath H2 analysis are presented. Initially, a hexamethylenetetramine (HMTA) based, microwave assisted, synthesis method of layered basic zinc acetate (LBZA) nanomaterials was investigated. Using the synthesised LBZA, a dropcast nanogranular ZnO gas sensor was produced. The testing of the sensor showed high sensitivity towards hydrogen with response (Resistanceair/ Resistancegas) to 200 ppm H2 at 328 °C of 7.27. The sensor is highly competitive with non-catalyst surface decorated sensors and sensitive enough to measure current H2 guideline thresholds for carbohydrate malabsorption (Positive test threshold: 20 ppm H2, Predicted response: 1.34). Secondly, a novel LBZA synthesis method was developed, replacing the HMTA by NaOH. This resulted in a large yield improvement, from a [OH-] conversion of 4.08 at% to 71.2 at%. The effects of [OH-]/[Zn2+] ratio, microwave exposure and transport to nucleation rate ratio on purity, length, aspect ratio and polydispersity were investigated in detail. Using classical nucleation theory, analysis of the basal layer charge symmetries, and oriented attachment theory, a dipole-oriented attachment reaction mechanism is presented. The mechanism is the first theory in literature capable of describing all observed morphological features along length scales. The importance of transport to nucleation rate ratio as the defining property that controls purity and polydispersity is then shown. Using the NaOH derived LBZA, a flexographic printing ink was developed, and proof-of-concept sensors printed. Gas sensing results showed a high response to 200 ppm H2 at 300 °C of 60.2. Through IV measurements and SEM analysis this was shown to be a result of transfer of silver between the electrode and the sensing layer during the printing process. Finally, Investigations into the intense pulsed light treatment of LBZA were conducted. The results show that dehydration at 150 °C prior to exposure is a requirement for successful calcination, producing ZnO quantum dots (QDs) in the process. SEM measurements show mean radii of 1.77-2.02 nm. The QDs show size confinement effects with the exciton blue shifting by 0.105 eV, and exceptionally low defect emission in photoluminescence spectra, indicative of high crystalline quality, and high conductivity. Due to the high crystalline quality and amenity to printing, the IPL ZnO QDs have numerous potential uses ranging from sensing to opto-electronic devices
- …