Inter-rater reliability for movement pattern analysis (MPA): measuring patterning of behaviors versus discrete behavior counts as indicators of decision-making style

The unique yield of collecting observational data on human movement has received increasing attention in a number of domains, including the study of decision-making style. As such, interest has grown in the nuances of core methodological issues, including the best ways of assessing inter-rater reliability. In this paper we focus on one key topic – the distinction between establishing reliability for the patterning of behaviors as opposed to the computation of raw counts – and suggest that reliability for each be compared empirically rather than determined a priori. We illustrate by assessing inter-rater reliability for key outcome measures derived from movement pattern analysis (MPA), an observational methodology that records body movements as indicators of decision-making style with demonstrated predictive validity. While reliability ranged from moderate to good for raw counts of behaviors reflecting each of two Overall Factors generated within MPA (Assertion and Perspective), inter-rater reliability for patterning (proportional indicators of each factor) was significantly higher and excellent (ICC = 0.89). Furthermore, patterning, as compared to raw counts, provided better prediction of observable decision-making process assessed in the laboratory. These analyses support the utility of using an empirical approach to inform the consideration of measuring patterning versus discrete behavioral counts of behaviors when determining inter-rater reliability of observable behavior. They also speak to the substantial reliability that may be achieved via application of theoretically grounded observational systems such as MPA that reveal thinking and action motivations via visible movement patterns

An Integrated Reliability and Physics-Based Risk Modeling Approach for Assessing Human Spaceflight Systems

This paper presents an integrated reliability and physics-based risk modeling approach for assessing human spaceflight systems. The approach is demonstrated using an example, end-to-end risk assessment of a generic-crewed space transportation system during a reference mission to the International Space Station. The behavior of the system is modeled using analysis techniques from multiple disciplines in order to properly capture the dynamic time- and state- dependent consequences of failures encountered in different mission phases. We discuss how to combine traditional reliability analyses with Monte Carlo simulation methods and physics-based engineering models to produce loss-of- mission and loss-of-crew risk estimates supporting risk-based decision-making and requirement verification. This approach facilitates risk-informed design by providing more realistic representation of system failures and interactions; identifying key risk-driving sensitivities, dependencies, and assumptions; and tracking multiple figures of merit within a single, responsive assessment framework that can readily incorporate evolving design information throughout system development

Reinforcement learning for efficient network penetration testing

Penetration testing (also known as pentesting or PT) is a common practice for actively assessing the defenses of a computer network by planning and executing all possible attacks to discover and exploit existing vulnerabilities. Current penetration testing methods are increasingly becoming non-standard, composite and resource-consuming despite the use of evolving tools. In this paper, we propose and evaluate an AI-based pentesting system which makes use of machine learning techniques, namely reinforcement learning (RL) to learn and reproduce average and complex pentesting activities. The proposed system is named Intelligent Automated Penetration Testing System (IAPTS) consisting of a module that integrates with industrial PT frameworks to enable them to capture information, learn from experience, and reproduce tests in future similar testing cases. IAPTS aims to save human resources while producing much-enhanced results in terms of time consumption, reliability and frequency of testing. IAPTS takes the approach of modeling PT environments and tasks as a partially observed Markov decision process (POMDP) problem which is solved by POMDP-solver. Although the scope of this paper is limited to network infrastructures PT planning and not the entire practice, the obtained results support the hypothesis that RL can enhance PT beyond the capabilities of any human PT expert in terms of time consumed, covered attacking vectors, accuracy and reliability of the outputs. In addition, this work tackles the complex problem of expertise capturing and re-use by allowing the IAPTS learning module to store and re-use PT policies in the same way that a human PT expert would learn but in a more efficient way

Improving water asset management when data are sparse

Ensuring the high of assets in water utilities is critically important and requires continuous improvement. This is due to the need to minimise risk of harm to human health and the environment from contaminated drinking water. Continuous improvement and innovation in water asset management are therefore, necessary and are driven by (i) increased regulatory requirements on serviceability; (ii) high maintenance costs, (iii) higher customer expectations, and (iv) enhanced environmental and health/safety requirements. High quality data on asset failures, maintenance, and operations are key requirements for developing reliability models. However, a literature search revealed that, in practice, there is sometimes limited data in water utilities - particularly for over-ground assets. Perhaps surprisingly, there is often a mismatch between the ambitions of sophisticated reliability tools and the availability of asset data water utilities are able to draw upon to implement them in practice. This research provides models to support decision-making in water utility asset management when there is limited data. Three approaches for assessing asset condition, maintenance effectiveness and selecting maintenance regimes for specific asset groups were developed. Expert elicitation was used to test and apply the developed decision-support tools. A major regional water utility in England was used as a case study to investigate and test the developed approaches. The new approach achieved improved precision in asset condition assessment (Figure 3–3a) - supporting the requirements of the UK Capital Maintenance Planning Common Framework. Critically, the thesis demonstrated that, on occasion, assets were sometimes misallocated by more than 50% between condition grades when using current approaches. Expert opinions were also sought for assessing maintenance effectiveness, and a new approach was tested with over-ground assets. The new approach’s value was demonstrated by the capability to account for finer measurements (as low as 10%) of maintenance effectiveness (Table 4-4). An asset maintenance regime selection approach was developed to support decision-making when data are sparse. The value of the approach is its versatility in selecting different regimes for different asset groups, and specifically accounting for the assets unique performance variables

Measuring situation awareness in complex systems: Comparison of measures study

Situation Awareness (SA) is a distinct critical commodity for teams working in complex industrial systems and its measurement is a key provision in system, procedural and training design efforts. This article describes a study that was undertaken in order to compare three different SA measures (a freeze probe recall approach, a post trial subjective rating approach and a critical incident interview technique) when used to assess participant SA during a military planning task. The results indicate that only the freeze probe recall method produced a statistically significant correlation with performance on the planning task and also that there was no significant correlation between the three methods, which suggests that they were effectively measuring different things during the trials. In conclusion, the findings, whilst raising doubts over the validity of post trial subjective rating and interview-based approaches, offer validation evidence for the use of freeze probe recall approaches to measure SA. The findings are subsequently discussed with regard to their implications for the future measurement of SA in complex collaborative systems

Software reliability and dependability: a roadmap

Shifting the focus from software reliability to user-centred measures of dependability in complete software-based systems. Influencing design practice to facilitate dependability assessment. Propagating awareness of dependability issues and the use of existing, useful methods. Injecting some rigour in the use of process-related evidence for dependability assessment. Better understanding issues of diversity and variation as drivers of dependability. Bev Littlewood is founder-Director of the Centre for Software Reliability, and Professor of Software Engineering at City University, London. Prof Littlewood has worked for many years on problems associated with the modelling and evaluation of the dependability of software-based systems; he has published many papers in international journals and conference proceedings and has edited several books. Much of this work has been carried out in collaborative projects, including the successful EC-funded projects SHIP, PDCS, PDCS2, DeVa. He has been employed as a consultant t

Risk Management in the Arctic Offshore: Wicked Problems Require New Paradigms

Recent project-management literature and high-profile disasters—the financial crisis, the BP Deepwater Horizon oil spill, and the Fukushima nuclear accident—illustrate the flaws of traditional risk models for complex projects. This research examines how various groups with interests in the Arctic offshore define risks. The findings link the wicked problem framework and the emerging paradigm of Project Management of the Second Order (PM-2). Wicked problems are problems that are unstructured, complex, irregular, interactive, adaptive, and novel. The authors synthesize literature on the topic to offer strategies for navigating wicked problems, provide new variables to deconstruct traditional risk models, and integrate objective and subjective schools of risk analysis

The Role of Person-Organization Fit in Organizational Selection Decisions

This paper presents and tests a theoretical model of person-organization fit and organizational selection decisions using data from 35 organizations making hiring decisions. Results suggested that (a) interviewers were able to assess applicants\u27 values with above-chance levels of accuracy, (b) interviewers compare their perceptions of applicants\u27 values with their organizations\u27 values to assess person-organization fit, and (c) it is perceived values congruence and not actual values congruence between applicants and organizations that predicted interviewers\u27 person-organization fit perceptions. Results also suggested that interviewers\u27 person-organization fit assessments had the largest effect on their hiring recommendations even after controlling for competing applicant characteristics (e.g., demographics, human capital), and that interviewers\u27 hiring recommendations had large and significant effects on organizations\u27 hiring decisions (e.g., job offers)