An investigation into the validation of formalised cognitive dimensions

The cognitive dimensions framework is a conceptual framework aimed at characterising features of interactive systems that are strongly influential upon their effective use. As such the framework facilitates the critical assessment and design of a wide variety of information artifacts. Although the framework has proved to be of considerable interest to researchers and practitioners, there has been little research examining how easily the dimensions used by it can be consistently applied. The work reported in this paper addresses this problem by examining an approach to the systematic application of dimensions and assessing its success empirically. The findings demonstrate a relatively successful approach to validating the systematic application of some concepts found in the cognitive dimensions framework.</p

Formalising responsibility modelling for automatic analysis

Modelling the structure of social-technical systems as a basis for informing software system design is a difficult compromise. Formal methods struggle to capture the scale and complexity of the heterogeneous organisations that use technical systems. Conversely, informal approaches lack the rigour needed to inform the software design and construction process or enable automated analysis. We revisit the concept of responsibility modelling, which models social technical systems as a collection of actors who discharge their responsibilities, whilst using and producing resources in the process. Responsibility modelling is formalised as a structured approach for socio-technical system requirements specification and modelling, with well-defined semantics and support for automated structure and validity analysis. The effectiveness of the approach is demonstrated by two case studies of software engineering methodologies

Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments

Calculating the Expected Value of Sample Information using Efficient Nested Monte Carlo: A Tutorial

Objective: The Expected Value of Sample Information (EVSI) quantifies the economic benefit of reducing uncertainty in a health economic model by collecting additional information. This has the potential to improve the allocation of research budgets. Despite this, practical EVSI evaluations are limited, partly due to the computational cost of estimating this value using the "gold-standard" nested simulation methods. Recently, however, Heath et al developed an estimation procedure that reduces the number of simulations required for this "gold-standard" calculation. Up to this point, this new method has been presented in purely technical terms. Study Design: This study presents the practical application of this new method to aid its implementation. We use a worked example to illustrate the key steps of the EVSI estimation procedure before discussing its optimal implementation using a practical health economic model. Methods: The worked example is based on a three parameter linear health economic model. The more realistic model evaluates the cost-effectiveness of a new chemotherapy treatment which aims to reduce the number of side effects experienced by patients. We use a Markov Model structure to evaluate the health economic profile of experiencing side effects. Results: This EVSI estimation method offers accurate estimation within a feasible computation time, seconds compared to days, even for more complex model structures. The EVSI estimation is more accurate if a greater number of nested samples are used, even for a fixed computational cost. Conclusions: This new method reduces the computational cost of estimating the EVSI by nested simulation

What makes industries believe in formal methods

The introduction of formal methods in the design and development departments of an industrial company has far reaching and long lasting consequences. In fact it changes the whole environment of methods, tools and skills that determine the design culture of that company. A decision to replace current design practice by formal methods, therefore, appears a vital one and is not lightly taken. The past has shown that efforts to introduce formal methods in industry has faced a lot of controversy and opposition at various hierarchical levels in companies, resulting in a marginal spread of such methods. This paper revisits the requirements for formal description techniques and identifies some critical success and inhibiting factors associated with the introduction of formal methods in the industrial practice. One of the inhibiting factors is the often encountered lack of appropriateness of the formal model to express and manipulate the design concerns that determine the world of the engineer. This factor motivated our research in the area of architectural and implementation design concepts. The last two sections of this paper report on some results of this research