Global Innovations in Measurement and Evaluation

We researched the latest developments in theory and practice in measurement and evaluation. And we found that new thinking, techniques, and technology are influencing and improving practice. This report highlights 8 developments that we think have the greatest potential to improve evaluation and programme design, and the careful collection and use of data. In it, we seek to inform and inspire—to celebrate what is possible, and encourage wider application of these ideas

An Interactive Visualisation System for Engineering Design using Evolutionary Computing

This thesis describes a system designed to promote collaboration between the human and computer during engineering design tasks. Evolutionary algorithms (in particular the genetic algorithm) can find good solutions to engineering design problems in a small number of iterations, but a review of the interactive evolutionary computing literature reveals that users would benefit from understanding the design space and having the freedom to direct the search. The main objective of this research is to fulfil a dual requirement: the computer should generate data and analyse the design space to identify high performing regions in terms of the quality and robustness of solutions, while at the same time the user should be allowed to interact with the data and use their experience and the information provided to guide the search inside and outside regions already found. To achieve these goals a flexible user interface was developed that links and clarifies the research fields of evolutionary computing, interactive engineering design and multivariate visualisation. A number of accessible visualisation techniques were incorporated into the system. An innovative algorithm based on univariate kernel density estimation is introduced that quickly identifies the relevant clusters in the data from the point of view of the original design variables or a natural coordinate system such as the principal or independent components. The robustness of solutions inside a region can be investigated by novel use of 'negative' genetic algorithm search to find the worst case scenario. New high performance regions can be discovered in further runs of the evolutionary algorithm; penalty functions are used to avoid previously found regions. The clustering procedure was also successfully applied to multiobjective problems and used to force the genetic algorithm to find desired solutions in the trade-off between objectives. The system was evaluated by a small number of users who were asked to solve simulated engineering design scenarios by finding and comparing robust regions in artificial test functions. Empirical comparison with benchmark algorithms was inconclusive but it was shown that even a devoted hybrid algorithm needs help to solve a design task. A critical analysis of the feedback and results suggested modifications to the clustering algorithm and a more practical way to evaluate the robustness of solutions. The system was also shown to experienced engineers working on their real world problems, new solutions were found in pertinent regions of objective space; links to the artefact aided comparison of results. It was confirmed that in practice a lot of design knowledge is encoded into design problems but experienced engineers use subjective knowledge of the problem to make decisions and evaluate the robustness of solutions. So the full potential of the system was seen in its ability to support decision making by supplying a diverse range of alternative design options, thereby enabling knowledge discovery in a wide-ranging number of applications

A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK

There is an increased recognition that learning design drives both student learning experience and quality enhancements of teaching and learning. The Open University UK (OU) has been one of few institutions that have explicitly and systematically captured the designs for learning at a large scale. By applying advanced analytical techniques on large and fine-grained datasets, the OU has been unpacking the complexity of instructional practices, as well as providing conceptual and empirical evidence of how learning design influences student behaviour, satisfaction, and performance. This study discusses the implementation of learning design at the OU in the last ten years, and critically reviews empirical evidence from eight recent large-scale studies that have linked learning design with learning analytics. Four future research themes are identified to support future adoptions of learning design approaches

Engaging stakeholders in research to address water-energy-food (WEF) nexus challenges

The water–energy–food (WEF) nexus has become a popular, and potentially powerful, frame through which to analyse interactions and interdependencies between these three systems. Though the case for transdisciplinary research in this space has been made, the extent of stakeholder engagement in research remains limited with stakeholders most commonly incorporated in research as end-users. Yet, stakeholders interact with nexus issues in a variety of ways, consequently there is much that collaboration might offer to develop nexus research and enhance its application. This paper outlines four aspects of nexus research and considers the value and potential challenges for transdisciplinary research in each. We focus on assessing and visualising nexus systems; understanding governance and capacity building; the importance of scale; and the implications of future change. The paper then proceeds to describe a novel mixed-method study that deeply integrates stakeholder knowledge with insights from multiple disciplines. We argue that mixed-method research designs—in this case orientated around a number of cases studies—are best suited to understanding and addressing real-world nexus challenges, with their inevitable complex, non-linear system characteristics. Moreover, integrating multiple forms of knowledge in the manner described in this paper enables research to assess the potential for, and processes of, scaling-up innovations in the nexus space, to contribute insights to policy and decision making

Supporting decision making process with "Ideal" software agents: what do business executives want?

According to Simon’s (1977) decision making theory, intelligence is the first and most important phase in the decision making process. With the escalation of information resources available to business executives, it is becoming imperative to explore the potential and challenges of using agent-based systems to support the intelligence phase of decision-making. This research examines UK executives’ perceptions of using agent-based support systems and the criteria for design and development of their “ideal” intelligent software agents. The study adopted an inductive approach using focus groups to generate a preliminary set of design criteria of “ideal” agents. It then followed a deductive approach using semi-structured interviews to validate and enhance the criteria. This qualitative research has generated unique insights into executives’ perceptions of the design and use of agent-based support systems. The systematic content analysis of qualitative data led to the proposal and validation of design criteria at three levels. The findings revealed the most desirable criteria for agent based support systems from the end users’ point view. The design criteria can be used not only to guide intelligent agent system design but also system evaluation

Sonification of Network Traffic Flow for Monitoring and Situational Awareness

Maintaining situational awareness of what is happening within a network is challenging, not least because the behaviour happens within computers and communications networks, but also because data traffic speeds and volumes are beyond human ability to process. Visualisation is widely used to present information about the dynamics of network traffic dynamics. Although it provides operators with an overall view and specific information about particular traffic or attacks on the network, it often fails to represent the events in an understandable way. Visualisations require visual attention and so are not well suited to continuous monitoring scenarios in which network administrators must carry out other tasks. Situational awareness is critical and essential for decision-making in the domain of computer network monitoring where it is vital to be able to identify and recognize network environment behaviours.Here we present SoNSTAR (Sonification of Networks for SiTuational AwaReness), a real-time sonification system to be used in the monitoring of computer networks to support the situational awareness of network administrators. SoNSTAR provides an auditory representation of all the TCP/IP protocol traffic within a network based on the different traffic flows between between network hosts. SoNSTAR raises situational awareness levels for computer network defence by allowing operators to achieve better understanding and performance while imposing less workload compared to visual techniques. SoNSTAR identifies the features of network traffic flows by inspecting the status flags of TCP/IP packet headers and mapping traffic events to recorded sounds to generate a soundscape representing the real-time status of the network traffic environment. Listening to the soundscape allows the administrator to recognise anomalous behaviour quickly and without having to continuously watch a computer screen.Comment: 17 pages, 7 figures plus supplemental material in Github repositor

Interactive constraint-based space layout planning

Layout planning is the primordial design activity that determines the characteristics and performance of a building throughout its lifecycle. Due to its iterative nature, there is a growing interest in the automation of space layout planning to enhance the search for optimum design solutions. The approaches for automation range from constraint/heuristics-based to the application of numerical optimisation algorithms. Among these, the use of design constraints to guide the search of the solution space is well regarded due to its ability to model design problems of an applied nature with multiple objectives. Constraint-based approaches also allow interactivity between the designer and layout planning process, which simulates the iterative nature of creative design and can be integrated well with the existing design process. Interactivity also enhances the management of design knowledge through improved processing and visualisation of information. This paper presents a theoretical framework for interactive constraint-based layout optimisation with an implemented prototype for a hospital patient room interior layout. The theoretical framework was developed by analysing existing layout automation methods and interactive approaches through a review of relevant literature. Object-oriented computer programming was used to develop the prototype to demonstrate the proposed approach of interactive layout planning system. The framework augments the iterative design process by facilitating the active participation and sharing of the designer’s knowledge during the aggregation. With regard to the implementation of the framework in large problems, fast evaluation of design solution was found to be necessary to interact with the system in real time. Interactive constraint-based layout optimisation has, therefore, the ability to enhance the search process of optimum design solutions by augmenting the iterative nature of the creative design process

Mixed methods research : creating fusion from the QUAL and QUAN Data Mosaic

There is no single recommended means of discourse for presenting and discussing mixed methods research, with lack of data synthesis and process transparency a frequently cited criticism. This paper addresses the deficiency and explores inventive means of data collection alongside innovative approaches to integrating, analysing and articulating qualitative and quantitative sources. A pragmatic philosophy, supported by theoretical and methodological bricolage is advocated and justified. A panoptic empirical study to elucidate the knowledge sharing influences of middle management in leading UK communication sector operators provides context. A sequential-exploratory and equally weighted QUALQUAN design was selected, incorporating emergent evaluation and integration. Innovative qualitative techniques were adopted, namely STRIKE - STRuctured Interpretation of the Knowledge Environment, photographic analysis and word cloud visualisation, alongside cultural-web focus groups. This facilitated rich, nuanced and multi-textured data capture to aid the instrument fidelity of a quantitative cross-operator survey. Triangulation was undertaken across all sources to assess areas of corroboration, elaboration or dissonance. It is demonstrated that this approach enables a multiplicity of perspectives to build successive deepening of understanding; supports transparency, traceability and synthesis; benefits credibility and validation and provides evidence of methodological robustness. This dynamic approach towards the design, conduct, fusion and presentation of mixed methods research therefore addresses a challenging lacuna: to combine rigour with responsiveness, texture with breadth and communicability with complexity. This can foster reflexivity and sensemaking for the researcher and further, can facilitate understanding, engagement and connection for the audience

Investment Risk Appraisal

Standard financial techniques neglect extreme situations and regards large market shifts as too unlikely to matter. This approach may account for what occurs most of the time in the market, but the picture it presents does not reflect the reality, as the major events happen in the rest of the time and investors are ‘surprised’ by ‘unexpected’ market movements. An alternative fuzzy approach permits fluctuations well beyond the probability type of uncertainty and allows one to make fewer assumptions about the data distribution and market behaviour. Fuzzifying the present value criteria, we suggest a measure of the risk associated with each investment opportunity and estimate the project’s robustness towards market uncertainty. The procedure is applied to thirty-five UK companies and a neural network solution to the fuzzy criterion is provided to facilitate the decision-making process. Finally, we discuss the grounds for classical asset pricing model revision and argue that the demand for relaxed assumptions appeals for another approach to modelling the market environment